Repository navigation
Conversation
3mei
requested changes
Nov 6, 2026
3mei
left a comment
Contributor
There was a problem hiding this comment.
@isarasua thank you! This makes sense as an extension seam: today a probe, scorer or host can't see why a model stopped, and a cut-off judge or user reply is a classic cause of a wrong verdict. On that front:
- Please say that in the PR body; the current "Why" is narrower.
- The keys never reach results, so a run still can't show that a reply was cut off (I ran the stock probe with every reply reporting
max_tokens: nothing in the row). Could you record it, for example onSimulationTrace.extra, in this PR or a follow-up with an issue?
To fix before merge:
acall_llmnow raisesTypeError: 'Mock' object is not subscriptablefor a plainunittest.mock.Mockresponse, whichmainaccepts. Please guard thechoiceslookup (isinstance(choices, (list, tuple))) and add a test.- After #140,
tests/engine/test_hosted_parity_contract.py:23importsgeneral_open_ended.generator(nowprobe.py). Please rebase ontomaininstead of merging it, and re-run the head checks. - Document the keys in
docs/extension_api.mdandAUTHORING_A_PROBE.md: they are present on the realacall_llmonly, so read them with.get, and build stored messages withassistant_message. - Use
register_test_probein the new tests, and paste the realmake testoutput and themake health-checkresult.
acall_llm's reply now carries two keys that describe the call rather than the message, so a caller can tell a cut-off reply from a finished one and see what the call cost: - finish_reason: the first choice's, in OpenAI's vocabulary, or None. Data Designer's Anthropic adapter passes stop_reason through, so its stop reasons with an exact OpenAI equivalent are mapped (max_tokens to length, end_turn and stop_sequence to stop, tool_use to tool_calls); anything else passes through unchanged. - usage: the input and output tokens the call already records, or None when the provider reported no token count (Data Designer's Usage can carry only a total; that is not "0 tokens"). A hosted run's host can record finish_reason on its activation result, next to usage; the probe then sees both exactly as from a configured model. usersim.testing's hosted parity check records them too, so a probe that reads them is compared like for like. Neither key reaches a transcript or a later request: assistant_message and the request conversion copy only the message's own fields. Tests pin both, for configured and hosted models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: isarasua <isarasua@nvidia.com>
…ices A test double such as Mock(message=..., usage=None) has a `choices` attribute that is not a list, which `main` accepts; reading its first item raised. The call and the hosted parity kit now report no finish reason for it. Documents both keys where an extension reads them: `acall_llm` in the extension API and the `after_assistant_turn` hook in the authoring guide. Only the real `acall_llm` returns them, so a probe reads them with `.get`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: isarasua <isarasua@nvidia.com>
isarasua
force-pushed
the
isarasua/feat/finish-reason-usage
branch
from
November 18, 2026 13:09
42949f9 to
c8f9f9c
Compare
Contributor
Author
|
@3mei thanks. Rebased onto
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None. Scoped with the maintainer. Recording the stop reason in a row is a follow-up: #170.
What changed and why
Why: today a probe, scorer or host can't see why a model stopped. A judge or simulated-user reply that hit the token limit reads like any other reply, and a cut-off reply is a classic cause of a wrong verdict: a judge that stops before its
<rating>, or a user turn that ends mid-sentence. This adds the extension seam that lets code see it. It also gives each call's token counts without reaching into the outcome builder.acall_llm's reply now carries two keys describing the call rather than the message:finish_reason: the first choice's, in OpenAI's vocabulary (stop,length,tool_calls, ...), orNonewhen the provider gives none or the response has no list of choices. Data Designer's Anthropic adapter passesstop_reasonthrough, so its stop reasons with an exact OpenAI equivalent are mapped:max_tokens→length,end_turnandstop_sequence→stop,tool_use→tool_calls. Anything else passes through unchanged.usage:{"input_tokens", "output_tokens"}, the numbers_finish_callalready records, orNonewhen the provider reported no token count. A provider that reports only a total readsNone, not zeros.Only the real
acall_llmreturns them, so an extension reads them with.get.docs/extension_api.mdand theafter_assistant_turnrow ofAUTHORING_A_PROBE.mdsay so.Hosted runs:
ActivationResultgains an optionalfinish_reason(wire keyfinish_reason, next tousage). The probe then sees both exactly as it would from a configured model, including the Anthropic mapping. Documented inEXTERNAL_PROBE_RUNTIME.md.usersim.testing's hosted parity check passes both to the host as a real host would, so a probe that reads them is compared like for like.Neither key reaches a transcript or a later request:
assistant_message()copies only role, content, tool calls and reasoning._dicts_to_chat_messages) builds each message from its own fields.Each of the 24
acall_llmcall sites insrc/reads named keys, builds its message withassistant_message, or (health_disclosure's audit) lists the reply's keys in a log line.How it was verified
At the head, on its own tree:
The first commit on its own tree:
make check-allclean, and the three changed test files pass.make health-checkwith the default models file can't pass on build.nvidia.com today, onmaintoo:openai/gpt-oss-20b(user, api_response and summary) reached end of life there on 2026-11-18 08:00 UTC and returns 410.google/gemma-4-31b-it(assistant) is listed but didn't answer a 16-token request within 60 s.nemotron-3.5-lightning-30b-a3basevaluator_modelreturned structured outputusersim evalcouldn't parse, so eval exits non-zero. Same result onmainwith the same file.So the run above used a copy of the file with gpt-oss-20b →
nemotron-3.5-lightning-30b-a3b, the assistant →nemotron-3.5-super-vl-120b-a12band the evaluator →nemotron-3-ultra-550b-a55b, onen_US(the defaultja_JPpersonas aren't downloaded on this machine).Tests:
TestFinishReasonAndUsage(test_llm.py):max_tokensresponse;Nonecases;Mockreply without a list of choices;test_episode_runtime.py:finish_reasonand usage reach the probe (includingmax_tokens→length);Nonewithout them;test_hosted_parity_contract.py:finish_reasonand usage gives the same values hosted as standalone, including explicit zero counts;Mocktests fail with the guard removed.stopfor a normal reply,lengthwithmax_tokens=16, andtool_callsfor a forced tool call, with token counts. One cut-off reply had empty content, the caselengthexists to catch. No Anthropic endpoint was available; that path is tested through Data Designer's own parser.Results impact
None. No probe, prompt, scorer or result column changes. The new keys appear only on
acall_llm's reply and, for hosted runs, in the activation payload.🤖 Generated with Claude Code