Skip to content

fix: score tool_calling reliably and let rejected openings recover - #171

Merged
3mei merged 7 commits into
mainfrom
yev/fix/tool-calling-judges-and-openings
Nov 18, 2026
Merged

3mei merged 7 commits into
mainfrom
yev/fix/tool-calling-judges-and-openings

Conversation

@3mei

@3mei 3mei commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Makes tool_calling scores trustworthy and stops losing rows to openings that fail the same way three times.

  • The tool_use scorer no longer records a judge reply that misses an axis as a pass. It checks the reply against its own response model, asks once more, and records an error otherwise.
  • usersim eval judges a row again when one of its scorers failed, and warns when a judge is the model under test.
  • architecture_leaking fails exposed internals, not plain statements of what the assistant can't do. task_completion counts a claimed action only when a tool call did it.
  • tool_calling's turn judge fails work presented as done that no tool did, results read for more than they say, and changes the user didn't ask for. The simulated user shares files as links or pasted text.
  • A rejected opening goes back to the user model with its reason, and en_IN and en_SG name the borrowed words their users may write.

Changes

  1. fix(evaluator): record a tool_use reply missing an axis as an error. Validation uses the axes' response model (a score written as "5" counts, axes wrapped in one outer object are read). Two attempts; a failed judgment has no scores and no status_proposal, so scorer_state reads it as an error.
  2. fix(evaluator): skip_if_existing judges a cell again when a scorer recorded an error. A scorer that did not apply to the row is not an error.
  3. feat(cli): usersim eval warns when a judge runs the model the run's assistant ran, read from the run manifest, else from the models file.
  4. fix(evaluator): the architecture_leaking and task_completion rubric. EVAL_PROMPT_VERSION is v1.4, so stored cells are judged again.
  5. fix(probes): tool_calling's turn judge rubric and the user prompt's attachment line, in prompts.yaml and the compiled defaults. prompt_version is v1.13; the pinned trajectory id moves with it.
  6. fix(engine): the opening's next attempt sees the rejected draft and the reason it failed. Only the latest rejection is shown, and none is kept in the user's history. Follow-up turns are unchanged.
  7. fix(engine): en_IN and en_SG name the same borrowed words in the user instruction and the gate's language clause. Other locales are unchanged.

Docs: docs/engine/README.md (re-judging, self-judging, max_query_attempts), docs/engine/AUTHORING_A_PROBE.md (opening retries, borrowed words) and both tool_calling READMEs.

Verification

Each commit, on its own tree: make check-all, pre-commit on the commit's files, pytest --collect-only, and the evaluator, tool_calling, simulation, language and eval CLI tests. All passed for c1 to c7.

Head:

$ make test
5263 passed in 64.49s
$ make smoke && make check-license-headers && make check-doc-links && make check-notebooks-clean
all exit 0
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed

Live checks, against a custom provider config:

  • make health-check with HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_IN: simulate, eval and report completed, evaluator failed=0. The run also showed the new self-judging warning, since that config grades with the assistant's model.
  • tool_use scorer on a fresh tool_calling conversation (--scorers tool_use): every axis scored, status_proposal set, no error.
  • Rescoring 79 completed tool_calling conversations across eight locales with the same judge, before and after:
    • tool_use rows without scores counted as passes: 25 before, 0 after (18 rows still get no verdict from that judge after two attempts, and are recorded as errors).
    • architecture_leaking scored 1: 7 before, 0 after. Of the 58 rows scored both times, 4 moved from 1 to 5, 1 from 1 to 3 and 8 from 3 to 5.
    • Rows with a 1 on a critical axis: 14 of 71 scored before, 6 of 61 after.

A test for each bug fix fails without its fix: the re-judging test, and the opening-retry tests (three of four; the fourth covers an accepted first draft, which is unchanged).

3mei added 7 commits November 18, 2026 16:38
The tool_use scorer read whatever JSON its judge returned. A reply that
parsed but left out an axis, wrapped the axes in another object, or put
text where a score belongs gave null scores with status_proposal true,
and the capability report counted the row as a pass.

The scorer now checks the reply against the axes' own response model,
which also accepts a score written as text, reads axes wrapped in one
outer object, and asks the judge once more when a reply falls short. A
judgment that fails twice is recorded as an error with no scores and no
status_proposal, as a judge call that raises is.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
With skip_if_existing, a stored cell whose envelope matched was kept as
long as a rubric axis had a score, even when a scorer such as tool_use
had failed on it, so rerunning the same command never repaired it. A
cell with a scorer in the error state is now judged again. A scorer that
did not apply to the row is not an error.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A judge alias that runs the same model as the run's assistant grades its
own replies. usersim eval now prints a warning naming the judge and the
model, reading the assistant's model from the run manifest, or from the
models file when the run has none.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The tool_use axis told the judge that a real assistant never explains a
limitation, so a reply saying it can't search by hashtag, or that a
lookup found nothing for a date, scored as a leak. The axis now fails
internals the user has no use for: tool, endpoint, parameter and field
names, and quoted instructions. Saying in plain words what the assistant
can't do or why a request failed, asking for a credential a tool needs,
and giving a link a tool returned are not leaks.

task_completion counts an action the assistant says it took only when a
tool call did it. EVAL_PROMPT_VERSION is v1.4, so stored cells are
judged again.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The turn judge failed a reply that said it performed an action no call
performed, but passed replies that implied one: a log shown as recorded,
an appointment described as arranged after an availability check, a
confirmation promised for later. Its rubric now also fails an action
presented as done, said or implied, or promised for later, that no call
performs; a result read for more than it says, such as a valid username
format taken as a free username; and a turn whose calls changed the
user's account, data or bookings without the user asking.

The simulated user is told that nothing can be attached in the chat, so
it shares a photo or file as a link or pastes its text, rather than
saying it sent one that never arrives.

tool_calling's prompt version is v1.13.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
When a generated opening failed a check (the script check, the probe's
opening check or the user-turn judge), the user model was asked again
with the same prompt, so a persona that drifted into another language
failed every attempt the same way and the row was lost. The next attempt
now sees the rejected draft and why it failed. Only the latest rejection
is shown, and none is kept in the user's history. Follow-up turns are
unchanged.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
English locales allow the occasional borrowed word but not phrases in
another language, and the user and the gate drew that line differently:
a short expression such as "theek hai" in an otherwise English message
could be rejected as Hindi. The user instruction and the gate's language
clause now name the same few borrowed words for en_IN ("yaar", "bhai",
"theek hai") and en_SG ("lah", "shiok", "makan"). Other locales are
unchanged.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 18, 2026 21:40
@3mei
3mei merged commit 0cdcbb4 into main Nov 18, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant