Repository navigation
Conversation
Design and test plan for computing ingestion-time views on every ingest shard: refresh groups compiled into push pipelines split at pipeline breakers, one instance per (shard, window), partial outputs combined at window close by a rule derived from the plan (concatenate per-series states, merge mergeable states, single partition otherwise), commits in window order. Implementation follows after the test plan is approved.
…st plan Records each decision with options, precedents (push and streaming engines), corpus evidence (what the Planner puts at ingestion time over 3,731 public queries) and conclusion, with result slots for E1-E5. Rebased on the sharded ingestion: seal travels with the coordinator's flush, combine runs on the window-task pool, views of a source run inline on the shard thread, non-mergeable breakers are a list awaiting ordered-merge. Test plan revised to T1-T19 and B1-B2.
zzylol
force-pushed
the
feat/parallel-precompute
branch
from
November 21, 2026 00:11
38a73a7 to
27ae132
Compare
…ng until implemented)
Tests from the approved test plan of docs/design/parallel-precompute.md,
written before the implementation:
- T1, T2 in planner_executor_parity.rs: push answers and stored view rows
against the reference and the window path, K = 1, 2, 4, 8.
- T3-T19 in push_precompute.rs. Each runs its fixture in window mode
first, then in push mode.
- B1/B2: ingest_bench takes PRECOMPUTE_MODE=push and, when paced,
reports the newest window's commit lag.
Interface only, no push logic: EngineOptions.precompute_mode
(--precompute-mode, refused at install when push),
precompute_memory_budget_bytes, the test hooks slow_refreshes (the same
shape as the precompute task pool's) and panic_in_combine_of_window_end_ms,
pipelines::{CombineRule, BreakerRule, combine_rule, combine_rules}
(returning "not implemented"), and Snapshot::view_rows for comparing stored
views.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design: docs/design/parallel-precompute.md. This PR contains only the design document and test plan. There is no code and there are no tests yet. It is stacked on #852 and rebased on its sharded ingestion (flush-gated watermark).
For reviewers
Why
The window path folds nothing on arrival. All view work happens at window close, in one batch run per (group, window) over buffered raw samples. That took 0.75–1.3 s per window at 10k series. A single-group plan such as the dashboard gets one core per window. Raw samples are kept for whole windows even when the plan keeps no raw data. Per core, the data plane ingests about 132k samples/s, against 385–443k for VictoriaMetrics.
What the design proposes
Before / after (dashboard plan, 10k series, 4 shards)
Test plan summary
The test design was written by the same author as the design. An independent review of the test plan is requested here.
Verification
Not applicable: this PR changes documentation only.