Skip to content

feat(engine): report each call's finish_reason and usage - #116

Open
isarasua wants to merge 2 commits into
mainfrom
isarasua/feat/finish-reason-usage
Open

isarasua wants to merge 2 commits into
mainfrom
isarasua/feat/finish-reason-usage

Conversation

@isarasua

@isarasua isarasua commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Related issue

None. Scoped with the maintainer. Recording the stop reason in a row is a follow-up: #170.

What changed and why

Why: today a probe, scorer or host can't see why a model stopped. A judge or simulated-user reply that hit the token limit reads like any other reply, and a cut-off reply is a classic cause of a wrong verdict: a judge that stops before its <rating>, or a user turn that ends mid-sentence. This adds the extension seam that lets code see it. It also gives each call's token counts without reaching into the outcome builder.

acall_llm's reply now carries two keys describing the call rather than the message:

  • finish_reason: the first choice's, in OpenAI's vocabulary (stop, length, tool_calls, ...), or None when the provider gives none or the response has no list of choices. Data Designer's Anthropic adapter passes stop_reason through, so its stop reasons with an exact OpenAI equivalent are mapped: max_tokens → length, end_turn and stop_sequence → stop, tool_use → tool_calls. Anything else passes through unchanged.
  • usage: {"input_tokens", "output_tokens"}, the numbers _finish_call already records, or None when the provider reported no token count. A provider that reports only a total reads None, not zeros.

Only the real acall_llm returns them, so an extension reads them with .get. docs/extension_api.md and the after_assistant_turn row of AUTHORING_A_PROBE.md say so.

Hosted runs: ActivationResult gains an optional finish_reason (wire key finish_reason, next to usage). The probe then sees both exactly as it would from a configured model, including the Anthropic mapping. Documented in EXTERNAL_PROBE_RUNTIME.md. usersim.testing's hosted parity check passes both to the host as a real host would, so a probe that reads them is compared like for like.

Neither key reaches a transcript or a later request:

  • assistant_message() copies only role, content, tool calls and reasoning.
  • The request conversion (_dicts_to_chat_messages) builds each message from its own fields.

Each of the 24 acall_llm call sites in src/ reads named keys, builds its message with assistant_message, or (health_disclosure's audit) lists the reply's keys in a log line.

How it was verified

At the head, on its own tree:

$ make check-all
All checks passed!
471 files already formatted
$ make test
5266 passed in 114.81s (0:01:54)
$ make health-check HEALTH_LOCALE=en_US PROVIDER_CONFIG=<copy of models_default.toml, see below>
==> simulate  (en_US, ...)
==> eval      (run 1795005622)
==> report    (run 1795005622)
✓ simulate, eval and report all completed

The first commit on its own tree: make check-all clean, and the three changed test files pass.

make health-check with the default models file can't pass on build.nvidia.com today, on main too:

  • openai/gpt-oss-20b (user, api_response and summary) reached end of life there on 2026-11-18 08:00 UTC and returns 410.
  • google/gemma-4-31b-it (assistant) is listed but didn't answer a 16-token request within 60 s.
  • nemotron-3.5-lightning-30b-a3b as evaluator_model returned structured output usersim eval couldn't parse, so eval exits non-zero. Same result on main with the same file.

So the run above used a copy of the file with gpt-oss-20b → nemotron-3.5-lightning-30b-a3b, the assistant → nemotron-3.5-super-vl-120b-a12b and the evaluator → nemotron-3-ultra-550b-a55b, on en_US (the default ja_JP personas aren't downloaded on this machine).

Tests:

  • TestFinishReasonAndUsage (test_llm.py):
    • pass-through;
    • each Anthropic mapping;
    • an unmapped value;
    • Data Designer's real Anthropic parser on a max_tokens response;
    • None cases;
    • a Mock reply without a list of choices;
    • both usage shapes;
    • no stored message and no later request carries either key.
  • test_episode_runtime.py:
    • a host's finish_reason and usage reach the probe (including max_tokens → length);
    • None without them;
    • the wire round-trip and type check;
    • no leak into requests or the stored transcript over a full hosted episode.
  • test_hosted_parity_contract.py:
    • a probe that records each call's finish_reason and usage gives the same values hosted as standalone, including explicit zero counts;
    • the parity kit reads no finish reason from a reply without a list of choices.
  • The two Mock tests fail with the guard removed.
  • Live, through Data Designer on an OpenAI-compatible endpoint (from round 1, a script): three models each returned stop for a normal reply, length with max_tokens=16, and tool_calls for a forced tool call, with token counts. One cut-off reply had empty content, the case length exists to catch. No Anthropic endpoint was available; that path is tested through Data Designer's own parser.

Results impact

None. No probe, prompt, scorer or result column changes. The new keys appear only on acall_llm's reply and, for hosted runs, in the activation payload.

🤖 Generated with Claude Code

@isarasua
isarasua requested a review from a team as a code owner November 2, 2026 14:18

@3mei 3mei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isarasua thank you! This makes sense as an extension seam: today a probe, scorer or host can't see why a model stopped, and a cut-off judge or user reply is a classic cause of a wrong verdict. On that front:

  • Please say that in the PR body; the current "Why" is narrower.
  • The keys never reach results, so a run still can't show that a reply was cut off (I ran the stock probe with every reply reporting max_tokens: nothing in the row). Could you record it, for example on SimulationTrace.extra, in this PR or a follow-up with an issue?

To fix before merge:

  1. acall_llm now raises TypeError: 'Mock' object is not subscriptable for a plain unittest.mock.Mock response, which main accepts. Please guard the choices lookup (isinstance(choices, (list, tuple))) and add a test.
  2. After #140, tests/engine/test_hosted_parity_contract.py:23 imports general_open_ended.generator (now probe.py). Please rebase onto main instead of merging it, and re-run the head checks.
  3. Document the keys in docs/extension_api.md and AUTHORING_A_PROBE.md: they are present on the real acall_llm only, so read them with .get, and build stored messages with assistant_message.
  4. Use register_test_probe in the new tests, and paste the real make test output and the make health-check result.

isarasua and others added 2 commits November 18, 2026 13:15
acall_llm's reply now carries two keys that describe the call rather than
the message, so a caller can tell a cut-off reply from a finished one and
see what the call cost:

- finish_reason: the first choice's, in OpenAI's vocabulary, or None.
  Data Designer's Anthropic adapter passes stop_reason through, so its
  stop reasons with an exact OpenAI equivalent are mapped (max_tokens to
  length, end_turn and stop_sequence to stop, tool_use to tool_calls);
  anything else passes through unchanged.
- usage: the input and output tokens the call already records, or None
  when the provider reported no token count (Data Designer's Usage can
  carry only a total; that is not "0 tokens").

A hosted run's host can record finish_reason on its activation result,
next to usage; the probe then sees both exactly as from a configured
model. usersim.testing's hosted parity check records them too, so a probe
that reads them is compared like for like.

Neither key reaches a transcript or a later request: assistant_message
and the request conversion copy only the message's own fields. Tests pin
both, for configured and hosted models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: isarasua <isarasua@nvidia.com>
…ices

A test double such as Mock(message=..., usage=None) has a `choices`
attribute that is not a list, which `main` accepts; reading its first
item raised. The call and the hosted parity kit now report no finish
reason for it.

Documents both keys where an extension reads them: `acall_llm` in the
extension API and the `after_assistant_turn` hook in the authoring guide.
Only the real `acall_llm` returns them, so a probe reads them with `.get`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: isarasua <isarasua@nvidia.com>
@isarasua

Copy link
Copy Markdown
Contributor Author

@3mei thanks. Rebased onto main (c91d9b2), two commits: the original change, then the fixes below.

  • Why: the body now leads with it. A probe, scorer or host can't see why a model stopped, and a cut-off judge or user reply is a classic cause of a wrong verdict.
  • Recording it in the row: opened Record when a model's reply was cut off, so a run can show it #170 as a follow-up. The loop has no per-call trace today (traces are per check), so where it lands is a design choice: a per-alias count of cut-off replies next to per_model_calls, a warning with the turn, or a trace. The issue asks which you'd prefer.
  1. Mock responses: _finish_reason now reads choices only from a list or tuple, as does the parity kit's _reported_finish_reason. Both are tested, and both tests fail with the guard removed. Mock(message=..., usage=None) behaves as on main.
  2. Rebased rather than merged. The parity test now imports general_open_ended.probe. Head checks re-run on both commits' trees.
  3. Docs: docs/extension_api.md (acall_llm row) and AUTHORING_A_PROBE.md (after_assistant_turn row) describe both keys, say only the real acall_llm returns them so read them with .get, and that assistant_message leaves them out of the stored message.
  4. register_test_probe: the hosted-runtime tests use it. The parity test needs a real conversation loop, which a bare BaseProbe doesn't give (it has no prompts or theme), so it registers a subclass of the open-ended probe under restored_registries, the fixture register_test_probe is built on. make test: 5266 passed. make health-check passed, but not on the default models file: on build.nvidia.com gpt-oss-20b hit end of life today (410), gemma-4-31b-it doesn't answer, and nemotron-3.5-lightning as evaluator returns output usersim eval can't parse (same on main). The PR body has the output and the substitutions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants