Skip to content

docs(data-plane): design parallel precompute (push pipelines) - #854

Draft
zzylol wants to merge 3 commits into
sync/planner-main-1760a69ffrom
feat/parallel-precompute
Draft

zzylol wants to merge 3 commits into
sync/planner-main-1760a69ffrom
feat/parallel-precompute

Conversation

@zzylol

@zzylol zzylol commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Design: docs/design/parallel-precompute.md. This PR contains only the design document and test plan. There is no code and there are no tests yet. It is stacked on #852 and rebased on its sharded ingestion (flush-gated watermark).

For reviewers

  1. Decisions D1–D10 (section "Decisions D1–D10"). Each lists its options, precedents, evidence and conclusion. Experiments E1–E5 decide several of them, and their result slots read pending until the runs report.
  2. Test plan T1–T19, B1–B2 (section "Acceptance and test design"). The tests are written only after it is approved, then the implementation, test first.
  3. Combine rule table: concatenate, merge, ordered-merge (not in v1) and single partition. Install fails for a breaker that has no rule.

Why

The window path folds nothing on arrival. All view work happens at window close, in one batch run per (group, window) over buffered raw samples. That took 0.75–1.3 s per window at 10k series. A single-group plan such as the dashboard gets one core per window. Raw samples are kept for whole windows even when the plan keeps no raw data. Per core, the data plane ingests about 132k samples/s, against 385–443k for VictoriaMetrics.

What the design proposes

  • Push execution on the ingest shards. Each shard folds its morsels into the open window's breakers, and every view of a source runs inline on the shard thread.
  • Seal with the flush. At window close, the seal travels with the coordinator's flush, so each shard seals after exactly the samples admitted before it.
  • Combine and commit. The combine runs on the window-task pool. Per-series states (every real-world PromQL plan in the public corpus) are only concatenated. A single committer commits each group's windows in order.
  • No Planner executor change for today's plans. Today's executor already pushes when it is fed by a channel.

Before / after (dashboard plan, 10k series, 4 shards)

Window path (#852) Push
During the 15 s window raw samples buffered each shard folds its 2.5k series' samples into KLL/Max/Rate/Increase states on arrival
At close one batch run over 150k + 150k samples each shard emits its states (chunked); concatenation; one commit
Raw memory (raw not kept) rate × 15 s × 20 B per metric about one request
Rows / answers — identical (KLL within the existing 5%)

Test plan summary

  • T1–T3: parity of answers and stored rows against the window path and the Planner executor, at K = 1/2/4/8, raw kept and not kept; one morsel reaches every view of its source inline.
  • T4–T7: combine per class (concatenate; merge, including empty and global windows and a 2-key group-by; single partition and window path), and rule derivation with install refusal.
  • T8–T11: windows commit in order; groups that read other groups' views wait for those commits; seal is exact under shard skew and clock-ahead stamps; idle shards seal empty.
  • T12–T16: no raw buffering, late and out-of-order data, backpressure, memory reservations that wait before failing, chunked seal.
  • T17–T19: restart under any shard count or mode, failure isolation, metrics.
  • B1–B2: throughput and per-core rate; newest-window commit lag, compared with the window path at 1k, 10k and 100k series.

The test design was written by the same author as the design. An independent review of the test plan is requested here.

Verification

Not applicable: this PR changes documentation only.

@zzylol zzylol added documentation Improvements or additions to documentation area: data plane Runtime ingestion, storage, and query-serving data paths labels Nov 20, 2026
Design and test plan for computing ingestion-time views on every ingest
shard: refresh groups compiled into push pipelines split at pipeline
breakers, one instance per (shard, window), partial outputs combined at
window close by a rule derived from the plan (concatenate per-series
states, merge mergeable states, single partition otherwise), commits in
window order. Implementation follows after the test plan is approved.
…st plan

Records each decision with options, precedents (push and streaming
engines), corpus evidence (what the Planner puts at ingestion time over
3,731 public queries) and conclusion, with result slots for E1-E5.
Rebased on the sharded ingestion: seal travels with the coordinator's
flush, combine runs on the window-task pool, views of a source run
inline on the shard thread, non-mergeable breakers are a list awaiting
ordered-merge. Test plan revised to T1-T19 and B1-B2.
@zzylol
zzylol force-pushed the feat/parallel-precompute branch from 38a73a7 to 27ae132 Compare November 21, 2026 00:11
…ng until implemented)

Tests from the approved test plan of docs/design/parallel-precompute.md,
written before the implementation:

- T1, T2 in planner_executor_parity.rs: push answers and stored view rows
  against the reference and the window path, K = 1, 2, 4, 8.
- T3-T19 in push_precompute.rs. Each runs its fixture in window mode
  first, then in push mode.
- B1/B2: ingest_bench takes PRECOMPUTE_MODE=push and, when paced,
  reports the newest window's commit lag.

Interface only, no push logic: EngineOptions.precompute_mode
(--precompute-mode, refused at install when push),
precompute_memory_budget_bytes, the test hooks slow_refreshes (the same
shape as the precompute task pool's) and panic_in_combine_of_window_end_ms,
pipelines::{CombineRule, BreakerRule, combine_rule, combine_rules}
(returning "not implemented"), and Snapshot::view_rows for comparing stored
views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: data plane Runtime ingestion, storage, and query-serving data paths documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant