Skip to content

Benchmarks: 95% confidence intervals and a long 50-run measurement - #13

Merged
tobiaswuerth merged 2 commits into
mainfrom
benchmark-confidence
Oct 19, 2026
Merged

tobiaswuerth merged 2 commits into
mainfrom
benchmark-confidence

Conversation

@tobiaswuerth

Copy link
Copy Markdown
Owner

Summary

Benchmarks now report how certain each speedup is, and the published matrix comes from one long, fresh run.

Statistics

  • Each repeat already ran both transports back to back, alternating which goes first. The ratio of their durations is one speedup measured under the same conditions, so pairing cancels most of a shared runner's drift between repeats.
  • benchmarks.compare reports the median of the paired speedups and a distribution-free 95% confidence interval for that median from order statistics: the k-th smallest and k-th largest of n runs, with the largest k whose binomial coverage, 1 − 2·P(B < k) for B ~ Binomial(n, ½), stays at least 95%. Deterministic, no assumption about the noise distribution; six runs give the first interval (n = 50 → 18th and 33rd values).
  • The Markdown table shows the interval in parentheses; the matrix shows it in small type under each speedup and says so on its axis.

Workflow

  • 20 repeats on releases, 6 on pull requests (only to check the benchmark works), and a repeats input for manual runs; the job timeout rises to 90 minutes for long runs.
  • Job summaries are written in UTF-8 so Windows runners no longer garble × and –.

Results

  • A manual run with 50 repeats per size on all five platforms, on the 4.3.1 library code, replaces the README matrix, table and raw reports (added to this PR once it finishes).

🤖 Generated with Claude Code

tobiaswuerth and others added 2 commits October 19, 2026 10:28
Each repeat runs both transports back to back, alternating which goes
first, so the ratio of their durations is one speedup measured under
the same conditions on a shared runner. benchmarks.compare now reports
the median of these paired speedups and a distribution-free 95%
confidence interval for it from order statistics: the k-th smallest and
k-th largest of n runs, with the largest k whose binomial coverage stays
at least 95%. Six runs give the first such interval. The table shows it
in parentheses and the matrix in small type under each speedup.

The workflow runs 20 repeats on releases, six on pull requests, which
only check that the benchmark works, and takes the number as an input
on manual runs. Job summaries are written in UTF-8, so Windows no
longer garbles × and –.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the matrix, table and raw reports with a manual run of the
benchmark workflow: py-sharedmemory 4.3.1, Python 3.14.8, NumPy 2.5.4,
50 paired runs per size on each platform. Each cell now shows the median
speedup and its 95% confidence interval. From 1 MB, a pair is 2.8-7.9x
faster than multiprocessing.Queue on Linux, 2.1-3.5x on Windows x64,
1.12x at 1 MB and 3-3.5x from 10 MB on Windows ARM64, and 5.7-27x on
macOS at every size; most Linux and Windows intervals are within a few
percent of their medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tobiaswuerth
tobiaswuerth merged commit ceb27a1 into main Oct 19, 2026
32 checks passed
@tobiaswuerth
tobiaswuerth deleted the benchmark-confidence branch October 19, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant