Repository navigation
fix(probes): keep tool_calling timestamps and IDs consistent - #172
Merged
Merged
Conversation
The simulated service and its response judge read the clock as a date and a UTC time, so a Unix timestamp in a response had to be worked out from the calendar. Both got it wrong: a service gave timestamps days off, and the judge, counting from the wrong day, rejected correct ones and pushed the service onto wrong values. The clock now carries the Unix time of that moment, and both prompts count timestamps from it. tool_calling's prompt version is v1.14. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The response judge read "returns new ones each time" as a requirement and rejected a random fact tool that returned the same fact twice, though a real one does. The simulator still varies what such a tool returns; the judge no longer fails a repeat. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The simulated service sometimes returned an ID from an earlier response with a different record behind it: a playlist ID that now held other songs, a job posting rewritten to match a later search. The judge caught small edits but not whole swaps. The simulator is told that an ID names the same record in every response, and the judge fails one given to a different record or with fields changed that don't move with time. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
What changed and why
tool_calling's simulated service and its response judge read timestamps from the clock's Unix time, the judge lets a random tool repeat a result, and an ID keeps its record across responses. Every change stays inside the probe. One concern per commit.Friday, 2026-11-06, 12:04 UTC (Unix time 1793966640)(_format_now), and both prompts count a timestamp from that Unix time, 86,400 seconds a day. The simulated user and the assistant still see the date alone.The
tool_callingprompt version isv1.14.prompts.yamland the compiled defaults inprompts.pychange together.How it was verified
New and changed tests in
tests/probes/tool_calling/test_probe.pyfail on the code before each change and pass after it: the clock's Unix time for each user message, and that it is the clock's moment; the Unix time reaching the simulator throughexecute_tool_call; the timestamp line in both prompts; the judge's line letting a random tool repeat a result; and the ID lines in the simulator's rules and the judge's failure criteria.On every commit's own tree:
make check-all, the pre-commit hooks on the commit's files,pytest --collect-only(5265 to 5266 tests collected) and thetool_callingtests.At the head:
Live, against a custom provider config, on the head's code: the end-to-end health check with the
tool_callingprobe, in English, and onetool_callingrow simulated and evaluated with--scorers tool_use:Results impact
tool_callingconversations from prompt versionv1.13are regenerated atv1.14. Timestamps in simulated responses count from the clock's Unix time, so fewer are wrong and the judge stops rewriting correct ones; the judge no longer fails a random tool for repeating a result; and it now fails an ID that names a different record than before, so a response with a swapped record is rewritten before the assistant reads it. Response-judge rejection counts can move in both directions, and assistant scores with them.