Skip to content

0.11.0: steps, the end of a case, one budget, jitter, the machine's class, budgets and ledgers in the gate, doctest, and a cheaper library - #26

Merged
Sunrisepeak merged 12 commits into
mainfrom
release/0.11.0
Nov 20, 2026
Merged

Sunrisepeak merged 12 commits into
mainfrom
release/0.11.0

Conversation

@Sunrisepeak

Copy link
Copy Markdown
Member

Summary

What a test system needs to see the cost of the tests it runs (mcpp SPEC-017 v2.5; mcpp's test and CI design of 2026-11-20, part 1.1 "mtest").

  • Library cost. A GCC test file of one case and one expectation compiled in 689 ms (std::format and std::function loaded into every file); now 265 ms. Numbers and truth values are compared and shown in the library, a case's body is a reference, a nested case keeps its state in the library, a plain suite passes a function pointer. comparison keeps its fields.
  • Protocol cost. 0.075 ms a case under a runner that keeps each case's output (1.05–1.27 s for 510 cases before; the fsync is gone in mcpplibs.os 0.4.7). Both measured by new Bench cases with bounds.
  • P5.6 steps and exercised items; the engine keeps steps, exercised, cpu_ms, limit_ms and reads a host's builds.ndjson.
  • The end of a case: a process left holding an output open, and undeclared or missing diagnostic lines, fail the case (MTEST_DIAGNOSTICS=report to migrate). Golden comparisons with redactions and an update summary.
  • O11 budget: one slot file per job, held by workers and stated to programs (MTEST_BUDGET, MTEST_BUDGET_HELD).
  • O12 jitter: mtest run --jitter MS, contention_point().
  • Machine class in run.json (processor model via mcpplibs.os 0.4.7); estimates prefer the same class.
  • Gate: [budget] case_cpu with waivers, [ledger.*] requirements met by clauses or exercised items, stale waivers fail.
  • Coverage map records functions; mtest coverage spread.
  • Report sections: steps, builds, margin, budget, ledgers; machine in the summary.
  • doctest adapter (mcpplibs.mtest.doctest, L1): implementation and main compiled once; passes the conformance suite.

Test

Local (Linux x86_64, GCC 16.1.0): mcpp test of every member passes (lib 13 programs, manifest 2, conformance 15 cases, engine 61 cases); the conformance suite passes for the doctest fixture; formatting, version and path checks pass.

…d no std::function

Under GCC a file of one case and one expectation took 689 ms to compile:
std::format's machinery for eq() and std::function for the case's body,
each loaded from the standard library's module into every file. Numbers
and truth values are now compared and shown by the library (std::cmp_*
for integers of either signedness), a case's body is a reference to the
callable (a case runs while it is declared), a case declared in a body
keeps its name and attributes in the library, and a suite of a name and a
body that captures nothing passes a function pointer. The same file takes
265 ms; most of the rest is GCC loading the interface's classes.

mcpplibs.os ^0.4.7: a capture no longer fsyncs its file twice per case.
…grams' processes and diagnostic lines; golden redactions

lib: step() and exercise() write the step and exercise events; e2e::run
writes a step for each command, with its status and the processor time of
the programs it ran. at_case_end() registers a check the running case makes
before its verdict. The e2e module fails a case whose program left a
process holding an output open, and whose programs printed a line matching
a project's diagnostic pattern (e2e::diagnostics) that the case did not
declare (e2e::expect_diagnostic), or did not print a declared one; under
the report policy (MTEST_DIAGNOSTICS=report) the lines are printed and
counted as diagnostics.undeclared. golden() applies redactions (paths,
durations, sizes, digests) and an update prints what it changed.

engine: a case's steps, exercised items and processor time (from its
cpu.user and cpu.sys metrics) are read from the report and kept in
cases.ndjson. conformance: P5.6.
…class; [budget] and [ledger.*] are judged by the gate

run.json's machine is the machine's class (P12.4): the processor's model
(mcpplibs.os 0.4.7), else <arch>/<processors>, with processors and memory
beside it; it was the host's name. A case's history is taken from runs on a
machine of the same class first: hosted runners of one label differ by up
to 1.6 times.

[budget] case_cpu bounds a case's processor time; a case over it fails the
gate unless a [[budget.waiver]] names it with a reason, and a waiver that
excuses nothing fails it too. [ledger.<name>] states requirements, the
lines of a file or of a command's output; each must be covered by a
passing case's clause (evidence = clauses) or a used item of the product's
surface (evidence = exercised) over the run and its group, or excused by a
[[ledger.<name>.waiver]].
engine: a run makes a budget, a directory of one slot file per job; each
worker holds one while it runs programs and states it to them
(MTEST_BUDGET, MTEST_BUDGET_HELD), so that the parallel work a program
starts takes the slots no worker holds. Inside another runner's budget the
engine uses that one. run.json records the budget's slots.
mtest run --jitter <ms> [--jitter-seed <n>] states the jitter to the
programs (MTEST_JITTER, MTEST_JITTER_SEED) and records it.

lib: contention_point() waits a random time up to MTEST_JITTER, the same
for the same seed; a case begins at one and a shared object is made at
one.
…est coverage spread

A run under --coverage reads each .gcda's function records and keeps, per
case, the functions whose arc counters ran (<source>#<ident>) beside the
files. mtest coverage spread gives how widely the cases share files and
functions (the median share): a selection at a finer granularity selects
fewer cases only where that share is small.
… the cases near their limit, the budget and the ledgers

A run reads a host's builds.ndjson (B4, an edge a build ran or a package
the cache served); a case records the time limit applied to it
(limit_ms). The Markdown report gains five sections: steps (the slowest
steps of the cases), builds (the slowest edges and what the cache
served), margin (cases over four fifths of their limit, with the
machine's class), budget (cases over [budget] case_cpu) and ledgers; the
summary names the machine.
…d main once

mcpplibs.mtest.doctest compiles doctest's implementation once and gives a
test file its main (test-main): a file includes the header and defines
neither DOCTEST_CONFIG_IMPLEMENT nor DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN,
whose implementation each file compiled again (1.6 s a file under GCC).
Under a runner its reporter lists the registry's cases (a case skipped by
its decorator among them), runs exactly a selection, reports a missing
identifier, and writes each case's begin, failures and end. The fixture
passes the conformance suite, and CI runs it there.
An expectation's run time (1,000,000 eq(), at most 600 ns each; 228 ns
measured under GCC dev) and what the protocol adds to a case when a runner
keeps each case's output (a copy of the program running 510 cases, at
most 0.2 ms a case; 0.075 ms measured, 1.05 to 1.27 s for the 510 under
0.10.0 with mcpplibs.os 0.4.6).
The packages are 0.11.0. spec/protocol.md lists P5.6, P12.4, O11 and O12
and the doctest adapter; docs/config.md [budget] and [ledger.*];
docs/commands.md the budget, jitter, the report's sections, the gate's
judgements and coverage spread; the library's guide steps, the end
checks, golden redactions and contention points.
A hosted runner measured 0.21 ms a case and an ok image 1,036 ns an
expectation; the bounds (1 ms a case, 3,000 ns an expectation) name a
return to 0.10.0's order of magnitude, not a slower machine.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant