Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ on:
release:
types: [published]
workflow_dispatch:
inputs:
repeats:
description: Paired runs per payload size; more runs narrow the confidence intervals
default: "20"
pull_request:
paths:
- "benchmarks/*.py"
Expand All @@ -20,12 +24,16 @@ concurrency:

env:
SIZES: "1000 10000 100000 1000000 10000000 100000000"
# Pull requests only check that the benchmark works; six runs give the first 95% interval.
REPEATS: ${{ github.event_name == 'pull_request' && '6' || inputs.repeats || '20' }}
# The tables use × and –, which Windows' default encoding would garble in job summaries.
PYTHONUTF8: "1"

jobs:
benchmark:
name: Benchmark (${{ matrix.label }})
runs-on: ${{ matrix.os }}
timeout-minutes: 30
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
Expand All @@ -47,7 +55,7 @@ jobs:
- name: Benchmark
shell: bash
run: >-
python -m benchmarks.benchmark --sizes $SIZES --repeats 5
python -m benchmarks.benchmark --sizes $SIZES --repeats "$REPEATS"
--label "${{ matrix.label }}" --json "${{ matrix.file }}.json"
- name: Job summary
shell: bash
Expand Down Expand Up @@ -85,7 +93,7 @@ jobs:
done
version=$(python -c "import py_sharedmemory; print(py_sharedmemory.__version__)")
subtitle="GitHub-hosted runners · py-sharedmemory $version · Python 3.14 · NumPy float32"
subtitle="$subtitle · capacity 2 · median of 5 runs · $(date -u +%Y-%m-%d)"
subtitle="$subtitle · capacity 2 · median of $REPEATS paired runs · $(date -u +%Y-%m-%d)"
{
echo "## Speedup over multiprocessing.Queue"
echo
Expand Down
14 changes: 9 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,18 @@ python -m benchmarks.benchmark --sizes 1000000 10000000 --repeats 3 --json resul
python -m benchmarks.compare results.json --image results.png
```

`python -m benchmarks.benchmark --help` lists all options. `benchmarks.compare` prints the
speedups as a Markdown table and draws them as a matrix, one row per report.
`python -m benchmarks.benchmark --help` lists all options. Each repeat runs both transports
back to back, alternating which goes first. `benchmarks.compare` takes the ratio of their
durations in each repeat as one speedup and reports the median with a distribution-free 95%
confidence interval, which needs at least six repeats. It prints a Markdown table and draws
a matrix, one row per report.

The [benchmark workflow](.github/workflows/benchmark.yml) runs on every release, on demand,
and on pull requests that change the benchmarks. It measures Linux, Windows, and macOS
runners, x86-64 and ARM64, from 1 kB to 100 MB. The run summary shows the speedups, and the
`benchmark-summary` artifact holds the reports and the platform matrix. To redraw it from
downloaded reports:
runners, x86-64 and ARM64, from 1 kB to 100 MB, with 20 repeats per size; manual runs take
the number of repeats as an input, and pull requests use six. The run summary shows the
speedups, and the `benchmark-summary` artifact holds the reports and the platform matrix. To
redraw it from downloaded reports:

```sh
python -m benchmarks.compare linux-x64.json linux-arm64.json windows-x64.json \
Expand Down
43 changes: 23 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
**A fast `multiprocessing.Queue` alternative for large Python objects and NumPy arrays.**
Payloads travel between processes through shared memory instead of pipes; the queues
underneath carry only a few bytes of metadata per message. Multi-megabyte batches,
images, and simulation states arrive up to 3.6× faster on Windows and up to 7.4× faster on
images, and simulation states arrive up to 3.5× faster on Windows and up to 7.9× faster on
Linux, on macOS even small messages beat a bounded `multiprocessing.Queue`, and transfers
too large for `multiprocessing.Queue` still work.

Expand Down Expand Up @@ -86,7 +86,7 @@ NumPy arrays, image or video frames, RL rollouts, large `bytes`, or big nested o
[Performance](#performance).

On Linux and Windows, prefer `multiprocessing.Queue` for small messages: up to about
100 kB it is 1.5–4× faster, because handling a shared-memory block for every message
100 kB it is 1.5–3.6× faster, because handling a shared-memory block for every message
costs more than the pipe.

On free-threaded Python (`python3.14t` and later), threads run Python code in parallel.
Expand All @@ -102,33 +102,36 @@ directly. py-sharedmemory delivers an independent copy of each message.

![Speedup of py-sharedmemory over multiprocessing.Queue for each platform and payload size from 1 kB to 100 MB](https://raw.githubusercontent.com/tobiaswuerth/python_shared_memory_queue/main/benchmarks/results/platforms.png)

Speedup over `multiprocessing.Queue`; above 1× means py-sharedmemory is faster.
Speedup over `multiprocessing.Queue`; above 1× means py-sharedmemory is faster. Each cell
is the median of 50 paired runs, with the 95% confidence interval of that median in
parentheses.

| Platform | 1 kB | 10 kB | 100 kB | 1 MB | 10 MB | 100 MB |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Linux x64 | 0.33× | 0.32× | 0.45× | **3.32×** | **5.30×** | **2.80×** |
| Linux arm64 | 0.32× | 0.36× | 0.52× | **2.57×** | **7.42×** | **2.68×** |
| Windows x64 | 0.25× | 0.28× | 0.51× | **2.50×** | **3.07×** | **3.34×** |
| Windows arm64 | 0.30× | 0.31× | 0.63× | **1.11×** | **3.06×** | **3.62×** |
| macOS arm64 | **11.87×** | **22.58×** | **17.02×** | **10.87×** | **5.34×** | **5.42×** |

- **Linux, x86-64 and ARM64:** 2.6–7.4× faster from 1 MB. Up to 100 kB,
`multiprocessing.Queue` is 2–3× faster.
- **Windows x86-64:** 2.5–3.3× faster from 1 MB, and 2–4× slower up to 100 kB.
- **Windows ARM64:** about even at 1 MB, and 3–3.6× faster from 10 MB.
- **macOS (Apple Silicon):** 5–23× faster at every size. macOS has no timed semaphore
| Linux x64 | 0.32× (0.31–0.33) | 0.33× (0.32–0.34) | 0.48× (0.48–0.50) | **3.31×** (3.24–3.38) | **4.92×** (4.82–4.98) | **2.89×** (2.81–3.01) |
| Linux arm64 | 0.34× (0.34–0.35) | 0.36× (0.35–0.37) | 0.55× (0.54–0.57) | **3.06×** (2.95–3.13) | **7.90×** (7.73–8.02) | **2.75×** (2.72–2.80) |
| Windows x64 | 0.28× (0.28–0.29) | 0.30× (0.30–0.31) | 0.55× (0.55–0.56) | **2.09×** (1.91–2.35) | **3.22×** (3.16–3.25) | **3.46×** (3.44–3.50) |
| Windows arm64 | 0.31× (0.31–0.32) | 0.33× (0.32–0.33) | 0.64× (0.62–0.65) | **1.12×** (1.11–1.13) | **3.00×** (2.99–3.02) | **3.50×** (3.46–3.53) |
| macOS arm64 | **11.96×** (10.20–13.04) | **18.04×** (14.98–21.99) | **27.29×** (23.23–29.80) | **23.64×** (21.48–27.08) | **6.61×** (6.09–7.10) | **5.73×** (5.34–6.16) |

- **Linux, x86-64 and ARM64:** 2.8–7.9× faster from 1 MB. Up to 100 kB,
`multiprocessing.Queue` is 1.8–3.1× faster.
- **Windows x86-64:** 2.1–3.5× faster from 1 MB, and 1.8–3.6× slower up to 100 kB.
- **Windows ARM64:** slightly faster at 1 MB, and 3–3.5× faster from 10 MB.
- **macOS (Apple Silicon):** 5.7–27× faster at every size. macOS has no timed semaphore
wait, so a full bounded `multiprocessing.Queue` whose `put` has a timeout, as here,
[polls with sleeps of up to 20 ms](https://github.com/python/cpython/blob/main/Modules/_multiprocessing/semaphore.c).
py-sharedmemory enforces its capacity with a thread condition instead.

Measured by the [benchmark workflow](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/.github/workflows/benchmark.yml)
on GitHub-hosted runners on 2026-10-18: py-sharedmemory 4.3.0, Python 3.14.8, NumPy 2.5.4,
contiguous float32 arrays, `spawn`, capacity 2 for both queues, and the median of 5 runs
that alternate between the two. The runners are shared virtual machines, so compare the
speedups, not absolute throughput. Run-to-run spread is mostly within 20% on Linux and
Windows and larger on macOS, whose speedups dwarf it. See the
on GitHub-hosted runners on 2026-10-19: py-sharedmemory 4.3.1, Python 3.14.8, NumPy 2.5.4,
contiguous float32 arrays, `spawn`, and capacity 2 for both queues. Each run sends the same
messages through both queues back to back, alternating which goes first, and the ratio of
their times is one speedup; pairing them cancels most of the drift of the shared virtual
machines, whose absolute throughput says little. The intervals are distribution-free,
from the order statistics of the 50 speedups. See the
[raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/tree/main/benchmarks/results/platforms).
The workflow reruns on every release.
The workflow reruns with 20 runs on every release.

**Larger messages:** the speedup holds beyond the 100 MB that shared runners can hold. On
a Windows desktop, an earlier version stayed 1.8× faster at 100 MB and 1.5× faster at 1 GB,
Expand Down
107 changes: 89 additions & 18 deletions benchmarks/compare.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
"""Summarize benchmark reports from several platforms as a speedup table and matrix."""
"""Summarize benchmark reports from several platforms as a speedup table and matrix.

Each repeat of a benchmark runs both transports back to back, so the ratio of their
durations is one speedup measured under the same conditions. A cell reports the median of
these paired speedups and a distribution-free 95% confidence interval for that median.
"""

import argparse
import json
import math
import statistics
from pathlib import Path
from typing import cast
from typing import NamedTuple, cast

CONFIDENCE = 0.95

SURFACE = "#fcfcfb"
INK = "#0b0b0b"
Expand All @@ -21,14 +29,43 @@ def size_label(size):
return f"{size} B"


class Speedup(NamedTuple):
median: float
low: float | None # Confidence interval of the median; None with too few runs
high: float | None


def median_interval(values, confidence=CONFIDENCE):
"""A distribution-free confidence interval for the median, from order statistics.

The k-th smallest and k-th largest of n independent values enclose the median with
probability 1 - 2 P(B < k) for B ~ Binomial(n, 1/2); this picks the largest k that keeps
it at least `confidence`. None when there are too few values for any k.
"""
ordered = sorted(values)
count, tail = len(ordered), (1 - confidence) / 2
k, below = 0, 0.0
while True:
below += math.comb(count, k) / 2**count # P(B <= k)
if below > tail:
break
k += 1
return (ordered[k - 1], ordered[count - k]) if k else None


def speedups(report):
"""Shared-memory throughput relative to multiprocessing.Queue per size; None if failed."""
"""Speedup over multiprocessing.Queue per size, or None where either transport failed."""
rows = {(row["transport"], row["size"]): row for row in report["results"]}
result = {}
for size in sorted({size for _, size in rows}):
shared, standard = rows.get(("shared", size)), rows.get(("standard", size))
if shared and standard and shared["status"] == standard["status"] == "ok":
result[size] = shared["median_mib_per_second"] / standard["median_mib_per_second"]
# Both transports sent the same messages in a repeat: the duration ratio is the
# speedup, and pairing cancels most of the runner's drift between repeats.
pairs = zip(standard["durations_seconds"], shared["durations_seconds"], strict=True)
ratios = [standard_time / shared_time for standard_time, shared_time in pairs]
low, high = median_interval(ratios) or (None, None)
result[size] = Speedup(statistics.median(ratios), low, high)
else:
result[size] = None
return result
Expand All @@ -50,10 +87,19 @@ def markdown(reports):
def _cell(values, size):
if size not in values:
return "—"
if values[size] is None:
speedup = values[size]
if speedup is None:
return "failed"
text = f"{values[size]:.2f}×"
return f"**{text}**" if values[size] >= 1 else text
text = f"{speedup.median:.2f}×"
if speedup.median >= 1:
text = f"**{text}**"
if speedup.low is not None:
text += f" ({speedup.low:.2f}–{speedup.high:.2f})"
return text


def _short(value):
return f"{value:.1f}" if value >= 10 else f"{value:.2f}"


def _luminance(color):
Expand Down Expand Up @@ -82,8 +128,9 @@ def plot(reports, output, subtitle=""):

table = [(report["label"], speedups(report)) for report in reports]
sizes = sorted({size for _, values in table for size in values})
cells = [[values.get(size) for size in sizes] for _, values in table]
ratios = np.array(
[[values.get(size) or math.nan for size in sizes] for _, values in table], dtype=float
[[math.nan if cell is None else cell.median for cell in row] for row in cells], dtype=float
)
logs = np.log10(ratios)
limit = max(math.log10(4), float(np.nanmax(np.abs(logs))))
Expand All @@ -92,7 +139,7 @@ def plot(reports, output, subtitle=""):

plt.rcParams.update({"font.size": 11, "font.family": "DejaVu Sans"})
fig, ax = plt.subplots(
figsize=(1.25 * len(sizes) + 3.4, 0.62 * len(table) + 2.0),
figsize=(1.3 * len(sizes) + 3.4, 0.8 * len(table) + 2.0),
layout="constrained",
facecolor=SURFACE,
)
Expand All @@ -101,20 +148,44 @@ def plot(reports, output, subtitle=""):
np.ma.masked_invalid(logs), cmap=cmap, norm=norm, edgecolors=SURFACE, linewidth=2
)
for row, column in np.ndindex(ratios.shape):
if math.isnan(ratios[row, column]):
text, color = ("failed" if sizes[column] in table[row][1] else "—"), SECONDARY_INK
else:
value = ratios[row, column]
text = f"{value:.1f}×" if value >= 10 else f"{value:.2f}×"
rgba = cast(tuple[float, float, float, float], cmap(norm(logs[row, column])))
color = label_ink(to_hex(rgba))
ax.text(column + 0.5, row + 0.5, text, ha="center", va="center", color=color, fontsize=10)
speedup = cells[row][column]
if speedup is None:
text = "failed" if sizes[column] in table[row][1] else "—"
ax.text(column + 0.5, row + 0.5, text, ha="center", va="center", color=SECONDARY_INK)
continue
rgba = cast(tuple[float, float, float, float], cmap(norm(logs[row, column])))
color = label_ink(to_hex(rgba))
middle = row + 0.5 if speedup.low is None else row + 0.38
ax.text(
column + 0.5,
middle,
f"{_short(speedup.median)}×",
ha="center",
va="center",
color=color,
fontsize=10,
)
if speedup.low is not None:
interval = f"{_short(speedup.low)}–{_short(speedup.high)}"
ax.text(
column + 0.5,
row + 0.7,
interval,
ha="center",
va="center",
color=color,
fontsize=7.5,
alpha=0.85,
)

ax.set_xticks(np.arange(len(sizes)) + 0.5, [size_label(size) for size in sizes])
ax.set_yticks(np.arange(len(table)) + 0.5, [label for label, _ in table])
ax.invert_yaxis()
ax.tick_params(length=0, colors=SECONDARY_INK)
ax.set_xlabel("Payload per message", color=SECONDARY_INK)
label = "Payload per message"
if any(cell is not None and cell.low is not None for row in cells for cell in row):
label += " · small figures: 95% confidence interval of the median"
ax.set_xlabel(label, color=SECONDARY_INK)
ax.spines[:].set_visible(False)
ax.set_title(
"py-sharedmemory speedup over multiprocessing.Queue",
Expand Down
Binary file modified benchmarks/results/platforms.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading