Repository navigation
0.11.0: steps, the end of a case, one budget, jitter, the machine's class, budgets and ledgers in the gate, doctest, and a cheaper library - #26
Merged
Conversation
…d no std::function Under GCC a file of one case and one expectation took 689 ms to compile: std::format's machinery for eq() and std::function for the case's body, each loaded from the standard library's module into every file. Numbers and truth values are now compared and shown by the library (std::cmp_* for integers of either signedness), a case's body is a reference to the callable (a case runs while it is declared), a case declared in a body keeps its name and attributes in the library, and a suite of a name and a body that captures nothing passes a function pointer. The same file takes 265 ms; most of the rest is GCC loading the interface's classes. mcpplibs.os ^0.4.7: a capture no longer fsyncs its file twice per case.
…grams' processes and diagnostic lines; golden redactions lib: step() and exercise() write the step and exercise events; e2e::run writes a step for each command, with its status and the processor time of the programs it ran. at_case_end() registers a check the running case makes before its verdict. The e2e module fails a case whose program left a process holding an output open, and whose programs printed a line matching a project's diagnostic pattern (e2e::diagnostics) that the case did not declare (e2e::expect_diagnostic), or did not print a declared one; under the report policy (MTEST_DIAGNOSTICS=report) the lines are printed and counted as diagnostics.undeclared. golden() applies redactions (paths, durations, sizes, digests) and an update prints what it changed. engine: a case's steps, exercised items and processor time (from its cpu.user and cpu.sys metrics) are read from the report and kept in cases.ndjson. conformance: P5.6.
…class; [budget] and [ledger.*] are judged by the gate run.json's machine is the machine's class (P12.4): the processor's model (mcpplibs.os 0.4.7), else <arch>/<processors>, with processors and memory beside it; it was the host's name. A case's history is taken from runs on a machine of the same class first: hosted runners of one label differ by up to 1.6 times. [budget] case_cpu bounds a case's processor time; a case over it fails the gate unless a [[budget.waiver]] names it with a reason, and a waiver that excuses nothing fails it too. [ledger.<name>] states requirements, the lines of a file or of a command's output; each must be covered by a passing case's clause (evidence = clauses) or a used item of the product's surface (evidence = exercised) over the run and its group, or excused by a [[ledger.<name>.waiver]].
engine: a run makes a budget, a directory of one slot file per job; each worker holds one while it runs programs and states it to them (MTEST_BUDGET, MTEST_BUDGET_HELD), so that the parallel work a program starts takes the slots no worker holds. Inside another runner's budget the engine uses that one. run.json records the budget's slots. mtest run --jitter <ms> [--jitter-seed <n>] states the jitter to the programs (MTEST_JITTER, MTEST_JITTER_SEED) and records it. lib: contention_point() waits a random time up to MTEST_JITTER, the same for the same seed; a case begins at one and a shared object is made at one.
…est coverage spread A run under --coverage reads each .gcda's function records and keeps, per case, the functions whose arc counters ran (<source>#<ident>) beside the files. mtest coverage spread gives how widely the cases share files and functions (the median share): a selection at a finer granularity selects fewer cases only where that share is small.
… the cases near their limit, the budget and the ledgers A run reads a host's builds.ndjson (B4, an edge a build ran or a package the cache served); a case records the time limit applied to it (limit_ms). The Markdown report gains five sections: steps (the slowest steps of the cases), builds (the slowest edges and what the cache served), margin (cases over four fifths of their limit, with the machine's class), budget (cases over [budget] case_cpu) and ledgers; the summary names the machine.
…d main once mcpplibs.mtest.doctest compiles doctest's implementation once and gives a test file its main (test-main): a file includes the header and defines neither DOCTEST_CONFIG_IMPLEMENT nor DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN, whose implementation each file compiled again (1.6 s a file under GCC). Under a runner its reporter lists the registry's cases (a case skipped by its decorator among them), runs exactly a selection, reports a missing identifier, and writes each case's begin, failures and end. The fixture passes the conformance suite, and CI runs it there.
An expectation's run time (1,000,000 eq(), at most 600 ns each; 228 ns measured under GCC dev) and what the protocol adds to a case when a runner keeps each case's output (a copy of the program running 510 cases, at most 0.2 ms a case; 0.075 ms measured, 1.05 to 1.27 s for the 510 under 0.10.0 with mcpplibs.os 0.4.6).
The packages are 0.11.0. spec/protocol.md lists P5.6, P12.4, O11 and O12 and the doctest adapter; docs/config.md [budget] and [ledger.*]; docs/commands.md the budget, jitter, the report's sections, the gate's judgements and coverage spread; the library's guide steps, the end checks, golden redactions and contention points.
A hosted runner measured 0.21 ms a case and an ok image 1,036 ns an expectation; the bounds (1 ms a case, 3,000 ns an expectation) name a return to 0.10.0's order of magnitude, not a slower machine.
This was referenced Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What a test system needs to see the cost of the tests it runs (mcpp SPEC-017 v2.5; mcpp's test and CI design of 2026-11-20, part 1.1 "mtest").
comparisonkeeps its fields.Benchcases with bounds.steps,exercised,cpu_ms,limit_msand reads a host'sbuilds.ndjson.MTEST_DIAGNOSTICS=reportto migrate). Golden comparisons with redactions and an update summary.MTEST_BUDGET,MTEST_BUDGET_HELD).mtest run --jitter MS,contention_point().run.json(processor model via mcpplibs.os 0.4.7); estimates prefer the same class.[budget] case_cpuwith waivers,[ledger.*]requirements met by clauses or exercised items, stale waivers fail.mtest coverage spread.mcpplibs.mtest.doctest, L1): implementation and main compiled once; passes the conformance suite.Test
Local (Linux x86_64, GCC 16.1.0):
mcpp testof every member passes (lib 13 programs, manifest 2, conformance 15 cases, engine 61 cases); the conformance suite passes for the doctest fixture; formatting, version and path checks pass.