Skip to content

fix(probes): keep tool_calling timestamps and IDs consistent - #172

Merged
3mei merged 3 commits into
mainfrom
yev/fix/tool-calling-clock-and-records
Nov 19, 2026
Merged

3mei merged 3 commits into
mainfrom
yev/fix/tool-calling-clock-and-records

Conversation

@3mei

@3mei 3mei commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Related issue

None.

What changed and why

tool_calling's simulated service and its response judge read timestamps from the clock's Unix time, the judge lets a random tool repeat a result, and an ID keeps its record across responses. Every change stays inside the probe. One concern per commit.

  • The clock gives its Unix time. The service and its judge read the clock as a date and a UTC time, so a Unix timestamp in a response had to be worked out from the calendar, and both got it wrong: the service returned timestamps days off, and the judge, counting from the wrong day, failed correct ones and had them rewritten wrong. The clock line now reads Friday, 2026-11-06, 12:04 UTC (Unix time 1793966640) (_format_now), and both prompts count a timestamp from that Unix time, 86,400 seconds a day. The simulated user and the assistant still see the date alone.
  • A random tool may repeat a result. The judge read the rule that a tool returning random or newly generated items returns new ones each time as a requirement, and failed a random fact tool that returned the same fact twice, as a real one can. The simulator's rule still asks for new items; the judge's consistency criterion now lets such a tool return new ones or repeat one.
  • An ID keeps its record. The service sometimes returned an ID from an earlier response with a different record behind it, such as a playlist ID now holding other songs, or a job posting rewritten to match a later search, and the judge caught small edits but not whole swaps. The simulator's rules now say an ID an earlier response returned names the same record in every response, its name, title and other fields as returned; the judge fails an ID given to a different record, or a record whose fields changed in a way time doesn't explain.

The tool_calling prompt version is v1.14. prompts.yaml and the compiled defaults in prompts.py change together.

How it was verified

New and changed tests in tests/probes/tool_calling/test_probe.py fail on the code before each change and pass after it: the clock's Unix time for each user message, and that it is the clock's moment; the Unix time reaching the simulator through execute_tool_call; the timestamp line in both prompts; the judge's line letting a random tool repeat a result; and the ID lines in the simulator's rules and the judge's failure criteria.

On every commit's own tree: make check-all, the pre-commit hooks on the commit's files, pytest --collect-only (5265 to 5266 tests collected) and the tool_calling tests.

At the head:

$ make check-all
All checks passed!
471 files already formatted
$ make test
5266 passed in 75.98s (0:01:15)
$ make smoke
All checks passed.
$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed in 6.85s
7 passed in 1.83s

Live, against a custom provider config, on the head's code: the end-to-end health check with the tool_calling probe, in English, and one tool_calling row simulated and evaluated with --scorers tool_use:

$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_US
✓ simulate, eval and report all completed
$ usersim simulate --probe-mix tool_calling=1.0 --max-turns 4 … && usersim eval --scorers tool_use …
simulation: ok calls 4 rejections 0 prompt v1.14
judge: scored
tool_use: ok (status_proposal=True, error=None)

Results impact

tool_calling conversations from prompt version v1.13 are regenerated at v1.14. Timestamps in simulated responses count from the clock's Unix time, so fewer are wrong and the judge stops rewriting correct ones; the judge no longer fails a random tool for repeating a result; and it now fails an ID that names a different record than before, so a response with a swapped record is rewritten before the assistant reads it. Response-judge rejection counts can move in both directions, and assistant scores with them.

3mei added 3 commits November 19, 2026 11:09
The simulated service and its response judge read the clock as a date
and a UTC time, so a Unix timestamp in a response had to be worked out
from the calendar. Both got it wrong: a service gave timestamps days
off, and the judge, counting from the wrong day, rejected correct ones
and pushed the service onto wrong values. The clock now carries the
Unix time of that moment, and both prompts count timestamps from it.

tool_calling's prompt version is v1.14.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The response judge read "returns new ones each time" as a requirement
and rejected a random fact tool that returned the same fact twice,
though a real one does. The simulator still varies what such a tool
returns; the judge no longer fails a repeat.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The simulated service sometimes returned an ID from an earlier response
with a different record behind it: a playlist ID that now held other
songs, a job posting rewritten to match a later search. The judge caught
small edits but not whole swaps. The simulator is told that an ID names
the same record in every response, and the judge fails one given to a
different record or with fields changed that don't move with time.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 19, 2026 16:26
@3mei
3mei merged commit c297be7 into main Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant