Repository navigation
fix: score tool_calling reliably and let rejected openings recover - #171
Merged
Merged
Conversation
The tool_use scorer read whatever JSON its judge returned. A reply that parsed but left out an axis, wrapped the axes in another object, or put text where a score belongs gave null scores with status_proposal true, and the capability report counted the row as a pass. The scorer now checks the reply against the axes' own response model, which also accepts a score written as text, reads axes wrapped in one outer object, and asks the judge once more when a reply falls short. A judgment that fails twice is recorded as an error with no scores and no status_proposal, as a judge call that raises is. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
With skip_if_existing, a stored cell whose envelope matched was kept as long as a rubric axis had a score, even when a scorer such as tool_use had failed on it, so rerunning the same command never repaired it. A cell with a scorer in the error state is now judged again. A scorer that did not apply to the row is not an error. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A judge alias that runs the same model as the run's assistant grades its own replies. usersim eval now prints a warning naming the judge and the model, reading the assistant's model from the run manifest, or from the models file when the run has none. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The tool_use axis told the judge that a real assistant never explains a limitation, so a reply saying it can't search by hashtag, or that a lookup found nothing for a date, scored as a leak. The axis now fails internals the user has no use for: tool, endpoint, parameter and field names, and quoted instructions. Saying in plain words what the assistant can't do or why a request failed, asking for a credential a tool needs, and giving a link a tool returned are not leaks. task_completion counts an action the assistant says it took only when a tool call did it. EVAL_PROMPT_VERSION is v1.4, so stored cells are judged again. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The turn judge failed a reply that said it performed an action no call performed, but passed replies that implied one: a log shown as recorded, an appointment described as arranged after an availability check, a confirmation promised for later. Its rubric now also fails an action presented as done, said or implied, or promised for later, that no call performs; a result read for more than it says, such as a valid username format taken as a free username; and a turn whose calls changed the user's account, data or bookings without the user asking. The simulated user is told that nothing can be attached in the chat, so it shares a photo or file as a link or pastes its text, rather than saying it sent one that never arrives. tool_calling's prompt version is v1.13. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
When a generated opening failed a check (the script check, the probe's opening check or the user-turn judge), the user model was asked again with the same prompt, so a persona that drifted into another language failed every attempt the same way and the row was lost. The next attempt now sees the rejected draft and why it failed. Only the latest rejection is shown, and none is kept in the user's history. Follow-up turns are unchanged. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
English locales allow the occasional borrowed word but not phrases in
another language, and the user and the gate drew that line differently:
a short expression such as "theek hai" in an otherwise English message
could be rejected as Hindi. The user instruction and the gate's language
clause now name the same few borrowed words for en_IN ("yaar", "bhai",
"theek hai") and en_SG ("lah", "shiok", "makan"). Other locales are
unchanged.
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
tool_callingscores trustworthy and stops losing rows to openings that fail the same way three times.tool_usescorer no longer records a judge reply that misses an axis as a pass. It checks the reply against its own response model, asks once more, and records an error otherwise.usersim evaljudges a row again when one of its scorers failed, and warns when a judge is the model under test.architecture_leakingfails exposed internals, not plain statements of what the assistant can't do.task_completioncounts a claimed action only when a tool call did it.en_INanden_SGname the borrowed words their users may write.Changes
fix(evaluator): record atool_usereply missing an axis as an error. Validation uses the axes' response model (a score written as"5"counts, axes wrapped in one outer object are read). Two attempts; a failed judgment has no scores and nostatus_proposal, soscorer_statereads it as an error.fix(evaluator):skip_if_existingjudges a cell again when a scorer recorded an error. A scorer that did not apply to the row is not an error.feat(cli):usersim evalwarns when a judge runs the model the run's assistant ran, read from the run manifest, else from the models file.fix(evaluator): thearchitecture_leakingandtask_completionrubric.EVAL_PROMPT_VERSIONisv1.4, so stored cells are judged again.fix(probes): tool_calling's turn judge rubric and the user prompt's attachment line, inprompts.yamland the compiled defaults.prompt_versionisv1.13; the pinned trajectory id moves with it.fix(engine): the opening's next attempt sees the rejected draft and the reason it failed. Only the latest rejection is shown, and none is kept in the user's history. Follow-up turns are unchanged.fix(engine):en_INanden_SGname the same borrowed words in the user instruction and the gate's language clause. Other locales are unchanged.Docs:
docs/engine/README.md(re-judging, self-judging,max_query_attempts),docs/engine/AUTHORING_A_PROBE.md(opening retries, borrowed words) and both tool_calling READMEs.Verification
Each commit, on its own tree:
make check-all, pre-commit on the commit's files,pytest --collect-only, and the evaluator, tool_calling, simulation, language and eval CLI tests. All passed for c1 to c7.Head:
Live checks, against a custom provider config:
make health-checkwithHEALTH_PROBE=tool_calling HEALTH_LOCALE=en_IN: simulate, eval and report completed, evaluatorfailed=0. The run also showed the new self-judging warning, since that config grades with the assistant's model.tool_usescorer on a fresh tool_calling conversation (--scorers tool_use): every axis scored,status_proposalset, no error.tool_userows without scores counted as passes: 25 before, 0 after (18 rows still get no verdict from that judge after two attempts, and are recorded as errors).architecture_leakingscored 1: 7 before, 0 after. Of the 58 rows scored both times, 4 moved from 1 to 5, 1 from 1 to 3 and 8 from 3 to 5.A test for each bug fix fails without its fix: the re-judging test, and the opening-retry tests (three of four; the fourth covers an accepted first draft, which is unchanged).