Skip to content

feat: execute ASAPPlanner main's InstallablePlan through asap-executor - #852

Draft
zzylol wants to merge 42 commits into
mainfrom
sync/planner-main-1760a69f
Draft

zzylol wants to merge 42 commits into
mainfrom
sync/planner-main-1760a69f

Conversation

@zzylol

@zzylol zzylol commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design: docs/design/data-plane-demo.md (the data plane for the comparison with VictoriaMetrics: threads, columnar panes, shared query-time results, range queries, view persistence, metrics, evaluation design).

Supersedes #845. #845 bumps the ASAPPlanner pin only and does not compile: the Planner's plan file, executor API and planning inputs changed under it (66 compile errors). This PR pins ASAPPlanner main 056da657 and runs the plan through the Planner's executor.

What changes for a user

  • control_plane writes plan.json (wire version 15) and, with --report report.json, Stage 3's SelectedPlanReport (cost, predicted usage per node and per query). --plan-selection-settings is replaced by --cost-inputs. --measured-operation-costs and --ignore-accuracy-guarantees are kept.
  • data_plane keeps the raw samples it receives by Remote Write. Planned queries no longer read raw samples from Prometheus. New flags: --run-memory-limit-bytes and --view-retention-margin-ms.
  • /api/v1/status/asap reports ingestion counters (samples, and windows refreshed, failed or skipped) in place of precompute and query_run.
  • The backend declares query_time_results_retained: false, so the Planner selects no view refreshed on first read. The data plane does not refresh such views yet.
  • Planned range queries are answered (/api/v1/query_range), one evaluation per step.
  • Views persist across restarts (--view-dir, default asap-views; --no-view-persistence): one file per refresh group, one record and one fdatasync per window, tied to the plan by a hash; sketch states as asap_sketchlib's ASAPv1 bytes.
  • GET /metrics exports a per-phase CPU breakdown and the data plane's counters in the Prometheus text format.
  • New flags: --query-threads, --max-precompute-backlog, --no-shared-results, --query-time-results-retention-ms, --query-time-results-max-bytes, --view-dir, --no-view-persistence, --result-cache-bytes, --no-result-cache.
  • A Grafana range panel refreshed every 15 s computes only its new steps: final answers are kept in a result cache (below).

How the data plane runs a plan

Step Planner API used
Read and check the plan InstallablePlan::from_json, validate, materialized_views, ingestion_refresh_groups
Store one DataStore: raw samples per scanned metric as (ts, value, $promql_series_identity, labels...), columnar by pane; view rows in a copy-on-write view store with the executor's tiling read
Ingestion per refresh group (W, phase p), one execute_ingestion run per window (e - W, e] once the watermark reaches e; rows appended with the range they cover
Query one compiled query run per root (compile with the views it reads, Scope::Query), rows turned into a PromQL vector; a query-time OneEvaluation node several queries read is compiled on its own and bound as a view slot

The plan's retention counts back from the newest evaluation, but ingestion refreshes windows ahead of the queries it answers. Views and query-time raw samples are therefore kept --view-retention-margin-ms (default 60 s) past their retention. The end-to-end run found this: without the margin, a query at a time 20 s behind ingestion found its oldest windows already dropped.

Threads (before → after)

Before: one engine thread ran Remote Write appends, every closed window's ingestion run and every query in turn. While a window's views were computed, Remote Write queued (503 when full) and queries waited.

After:

Thread Does Publishes
HTTP (tokio) decodes Remote Write, interns series —
ingest-0..N (--ingest-threads, default a quarter of the cores) route and append the samples of the series hashed to each shard, into the shard's own raw data the shard's raw data, in a report after every batch
ingest (coordinator) keeps the watermark over the shards behind a flush, closes windows, has the shards evict panes raw data snapshot (every shard's) after every pass
precompute one execute_ingestion run per closed window, over the raw data as it was when the window closed; appends and syncs view files view snapshot after every window
query-0..N (--query-threads, default one per core) one query run to completion each, over the newest snapshots —

A run's output stream is not Send, so a run never moves between threads; what crosses threads (QueryEngine, RunStore, RawData, ViewStore, Precompute) is checked Send/Sync at compile time. At most --max-precompute-backlog (256) windows wait for precompute; when full, the coordinator waits, the shards' queues fill and Remote Write gets 503 (see below). A query does not wait for queued writes; the event-time barrier does (it answers once its windows are refreshed).

Ingest shards: correctness and backpressure (6d40b3d..8682b64)

With several ingest shards, parity lost 16–20 answers per run (sums and top-k low): windows closed before every shard had appended its samples. The rule is now: the watermark moves only behind a flush, and every pass that took a report or a barrier sends one. A flush means every shard has run what was admitted before it.

  • Reports after a flush are held. A shard that has answered a flush may already run later requests. Its reports are held for the next pass, so they move neither the watermark nor the raw data its windows read. A request reaches its shards whole between two flushes (an admission lock), never split by one.
  • The clock cap is the time the flush was sent. Samples may be stamped ahead of the wall clock, as the feeder does, and the watermark is then capped at the clock. The cap used to come from each shard's report time: a shard working through a backlog reports the current time while it still runs old requests, so the watermark passed samples of requests admitted after the flush. A request admitted later holds samples stamped later. A barrier no longer moves the clock.
  • A window closes once event time is past its end, as in the ASAPQuery precompute engine's window manager. The watermark from samples is one millisecond before the newest sample less the allowed lateness, so samples at a window's end that come in a later request still join it. A barrier counts whole.
  • Barriers. A barrier that comes while a flush is waited for goes to a queue for the next pass, behind its own flush. A single slot dropped a second barrier (HTTP 500), and joining the current pass closed windows without the writes acknowledged before it.
  • A dead thread makes the flush wait stop and fail the waiting barriers instead of hanging.
  • Backpressure. The coordinator counts a shard's samples out of its queue when it applies the shard's report, not when the shard sends it. While the coordinator waits for precompute, the queues fill to --max-queued-samples, Remote Write gets 503, and the reports waiting stay bounded. Before, the shards kept draining into an unbounded channel: no 503, no eviction, unbounded memory.

Borrowed from the ASAPQuery precompute engine (asap-query-engine/src/precompute_engine/): routing by series hash, a cross-worker watermark where workers with no samples do not hold it back, a periodic tick, the close-past-the-end rule, and dropping and counting late samples. Its watermark, unstarted-worker, boundary and late-data tests are ported at the engine level. ASAPQuery closes each worker's own partial windows; here one coordinator closes a window over every shard's raw data, so the cross-shard minimum is reached by the flush.

 struct ShardReport {
     shard: usize,
     raw_data: Arc<RawData>,
     newest_ms: Option<i64>,
     first_seen: Vec<(Arc<str>, i64)>,
-    now_ms: i64,
+    /// Samples of the batch, counted out of the queues when applied.
+    samples: u64,
 }
 enum Command {
     Report(ShardReport),
-    Flushed,
+    /// The shard numbered here has run every message sent before the flush.
+    Flushed(usize),
     AdvanceWatermark { barrier_ms: i64, reply: oneshot::Sender<i64> },
     Stop,
 }
-fn flush(shards: &[Sender<ShardMessage>], shared: &Shared)
+/// Sends a flush under the admission lock; returns the time it was sent.
+fn flush(shards: &[Sender<ShardMessage>], shared: &Shared) -> i64
 impl Ingestion {
-    pub fn observe(&mut self, first_seen: &[(Arc<str>, i64)], now_ms: i64)
+    pub fn observe(&mut self, first_seen: &[(Arc<str>, i64)])
+    /// Sets the clock the watermark is capped at: the last flush's time.
+    pub fn advance_clock(&mut self, now_ms: i64)
 }
 pub struct EngineOptions {
+    /// For tests: every shard but the first waits this long per batch.
+    #[doc(hidden)]
+    pub shard_delay_ms: u64,
 }

The boundary between the coordinator and the precompute is unchanged: PrecomputeJob goes over the bounded jobs channel, and the coordinator blocks on it.

Regression tests (data_plane/tests/ingest_shards.rs, one engine against one shard or several). Each was shown to fail on the old behaviour:

Test Before the fix
a_shard_running_ahead_does_not_close_windows_early (8 shards, 7 slowed by 20 ms) wrong answer at 540 s
a_watermark_capped_at_the_clock_moves_only_behind_a_flush (samples up to 1 s ahead of the clock, 4 shards, 3 slowed by 250 ms) 7–11 samples dropped as late per run
samples_at_a_window_end_in_a_later_request_join_the_window (1 and 8 shards) 32 samples dropped over 8 shards
a_late_sample_is_dropped_on_any_shard_count (from ASAPQuery's test_late_data_drop) — (new coverage)
a_precompute_that_cannot_keep_up_reaches_remote_write 40 of 40 requests admitted, no 503
a_barrier_during_another_barriers_flush_covers_writes_before_it sums low
barriers_during_one_flush_all_answer HTTP 500 for one barrier

Parity (PARITY_INGEST_THREADS=n), on 8682b64's code: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards. ingest_shards: 9/9, 3 runs.

Columnar raw store

Raw samples are held per pane (width: gcd of the refresh groups' widths and phases and 60 s). Each Remote Write request's samples of a pane are one immutable chunk: time and value arrays and a per-row series code into the chunk's series, whose identity JSON is computed once when the series is interned. A pane the watermark has passed is compacted into one chunk. Eviction drops whole panes, and a read never returns a sample at or before the eviction cutoff, so answers match the row store exactly.

Open pane, chosen for query-time Scans: the ingest thread publishes the raw data after every request (copy on write: per-pane chunk lists are copied, chunks are shared), and a read returns one batch per chunk of every pane overlapping the range, the open pane's chunks so far included; only a chunk straddling the range is filtered. This is the simplest option that needs no lock and no copy of the open pane per read; its cost is many small batches when many small requests fall in one open pane.

Shared query-time results (the plan decides)

A OneEvaluation query-time node that several queries' runs reach is computed once per evaluation time T: the first run computes it while concurrent runs wait (single flight), later runs read it from the view store's query-time partition, keyed by (T, node), kept --query-time-results-retention-ms (60 s) behind the newest T, bounded by --query-time-results-max-bytes. Any other node two queries' runs would compute makes the plan refused at install. No executor change: the node is bound as an on-first-read view slot.

Result cache

Before: a range request refreshed 15 s later recomputed every step. After: each planned query's answer at evaluation time T is kept, keyed by (query root, T), once it is final: the run succeeded (every view window it read is stored, and stored windows never change) and, if the root reads raw data at query time, T ≤ the watermark its snapshot was published with. A range request runs only the missing steps; an instant query at T reads a range step at T. Eviction: T more than the plan's longest retention plus --view-retention-margin-ms behind the watermark, then LRU beyond --result-cache-bytes (256 MiB). A run now reads the watermark before the snapshot, which also closes that race for shared query-time results. The steps of one range request read one snapshot, and raw chunks read whole are converted once per request.

examples/range_bench, dashboard workload, 1k series per metric, four 1 h panels at 15 s steps (964 steps), 56-core x86 server:

Request Cache on Cache off
first (cold), views plan 5.0–5.1 s, 176–187 s CPU 5.1 s, 177–180 s CPU
15 s later, views plan 1.2–1.6 s, 1.2–1.6 s CPU (960/964 steps cached) 4.8–5.1 s, 172–175 s CPU
15 s later, raw plan 1.1–1.3 s, 1.1–1.3 s CPU 4.5–4.9 s, 152–172 s CPU

The rest of a cached refresh is JSON encoding of the two 1,000-series matrices (0.4–0.5 s each). Reading one snapshot per request changes cold CPU by about 3 % (within noise): the cold cost is DDSketch merges, not reads. The Planner's cost model does not know about the cache (follow-up in the design doc).

/metrics CPU breakdown

asap_phase_cpu_seconds_total / asap_phase_wall_seconds_total / asap_phase_runs_total by phase: remote_write_decode, append, precompute (per refresh group), query (per query text); thread CPU from CLOCK_THREAD_CPUTIME_ID. Also asap_query_latency_seconds (histogram per query), asap_samples_total, asap_windows_total, asap_samples_queued, asap_precompute_backlog_windows, asap_watermark_lag_seconds, asap_raw_data_bytes, asap_view_bytes, asap_query_time_result_*, asap_view_persisted_bytes_total, asap_view_sync_seconds_total, asap_view_windows_not_persisted_total, process_resident_memory_bytes, and asap_thread_cpu_seconds_total{thread}. scripts/profile.sh writes a flamegraph of a running data plane.

Ingest throughput, before and after

cargo run --release -p data_plane --example ingest_bench -- SERIES 300 PLAN [REQUESTS] [memory]: in process, 300 s of 1 s scrapes sent by Remote Write as fast as accepted, then a barrier at the last sample. Workload random-mixed-seed-862-01; raw keeps raw samples (4 views, 25 windows), views keeps none (7 views, 325 windows). "Refreshed" is samples/s until every window is refreshed; "acknowledged" until every write is accepted. Views are persisted (the default). One run each on a shared host, so the numbers are indicative.

Plan, series Before this PR: refreshed Sync per window: refreshed (acknowledged) Now: refreshed (acknowledged) Now: CPU ingest / precompute; sync time
raw, 1k 387,550 22,953 (473,208) 1,244,680 (2,146,467) 0.12 / 0.21 s; 0 s
raw, 10k 362,043 188,600 (468,328) 413,992 (2,217,223) 0.92 / 2.66 s; 4.0 s
views, 1k 67,883 1,916 (484,228) 25,283 (2,243,669) 0.08 / 5.34 s; 12.0 s
views, 10k 54,173 14,509 (464,644) 11,020 (2,271,512) 0.95 / 61.98 s; 197.9 s
views, 10k, 100 requests per scrape — 20,132 (450,296) 10,570 (2,004,919) 4.15 / 63.22 s; 209.2 s

"Now" includes jemalloc, the in-place Remote Write decode, the interval sync (5 s) and the per-file text dictionary. Writes are acknowledged at 2.0–2.3M samples/s. View-heavy plans are limited by the precompute thread and, at 10k series, by the disk: the views plan writes 6.5 MB per window (2.1 GB per 5 minutes) and syncing it took about 200 s on this shared, nearly full disk.

Ingest threads (same bench, one client, 16 cores). Acknowledged throughput stays at about 2.2–2.3M samples/s at 10k series and 1.7–1.8M at 100k with 1, 2, 4 or 8 shards: the single bench client, which decodes and sends every request itself, is the limit, not the shards. The table is in the design doc's Measurements section.

Remote Write decode (decode_bench, µs per sample): one series per sample (10,000 × 1, the feeder's shape) 1.80 (system allocator) → 1.51 (jemalloc) → 0.38 (in-place decode); 100 samples per series (Prometheus' shape) 0.083 → 0.083 → 0.089.

Bytes per window written (views plan, 1k series): 0.93 MB with a sync per window and label text per row → 0.64 MB with the per-file text dictionary; raw plan 224 KB → 123 KB; views plan, 10k series, 9.4 MB → 6.5 MB.

View persistence

Every sketch kernel is persisted through the executor's sketch() and restore_from_disk (ASAPPlanner #1435): DDSketch, KLL, Count-Min, KMV, HyperLogLog, weighted frequency (Count-Min or Count Sketch, with or without a heap), UnivMon, Hydra-KLL and Hydra-Count-Min (one grid per window, shared by its groups), as asap_sketchlib's ASAPv1 bytes; exact states by their serde form. A UnivMon whose heaps mix key types has no ASAPv1 encoding (asap_sketchlib #265) and is counted in asap_view_windows_not_persisted_total; it is 0 for the benchmark and demo plans. Files are synced every --view-sync-interval-ms (default 5 s, VictoriaMetrics' default -inmemoryDataFlushInterval), so a window is durable within that interval; texts are stored once per file. A file of another plan is renamed *.stale; a torn last record is cut off at load; after a restart, windows whose samples arrived before it stay missing.

Review fixes

Shared query-time results: no orphaned entries or leaked bytes (an entry is created computing, under the map lock), waiters get the computer's failure, results are kept only for times the watermark has passed and expire behind it. Range queries: at most one step per query worker in flight, the first failure or a dropped client stops the rest, steps measured apart from instant queries, the 11,000-step limit as Prometheus applies it. A panicking query run answers with an error and the worker keeps serving. Not done: the precompute test gate stays on the production path (one uncontended lock per window); test helpers are not yet consolidated into tests/common; building MaterializedView values through serde JSON and the per-request work linear in retained panes are recorded as follow-ups in the design doc.

Code deleted, replaced by the Planner

Backend code Replaced by
sds.rs (definitions, keys, build order) materialized_views, ingestion_refresh_groups
precompute.rs pane builds ingestion runs per refresh window
raw_sample_window.rs DataStore::read_raw_data with the executor's Scan/TimeRange
per_sample.rs (per-sample state updates) none yet: views are built from the window's raw samples
query.rs raw fetches from Prometheus, QueryRun panes query runs over the data plane's own store

Their tests are deleted with them (per_sample, precompute, query, query_run, sds, answers).

Tests

  • New planner_executor_parity: 14 public workloads (examples/, plus ASAPPlanner's random-mixed-seed-862 01–06 and four sketch-plan snapshot workloads copied unchanged into data_plane/tests/workloads/), each planned with raw data kept and not kept. The data plane runs over HTTP with --no-forward; the Planner executor runs in process on the same plan and samples. Result after the re-pin and with shared query-time results on: 36 plans (18 workloads, raw data kept and not; two raw-only workloads cannot be planned without raw data), 148 of 148 queries equal, all with rows. Plans with a KLL sketch are compared within 5%: KLL seeds its compaction coin at random, so two builds of one window differ within its error (the dashboard plan's quantile query). The design doc's six example workloads are among them. The same harness plans the mixed workloads 01–03 with range_evaluation and compares each range step with the Planner executor's run at that time: 162 of 162 steps equal.
  • New a_query_behind_the_newest_window_is_still_answered.
  • New: a query is answered from the refreshed views while a refresh is held (concurrency); two queries sharing a OneEvaluation node, run concurrently at two times, compute it once per time and answer as runs that compute it themselves, and a plan that marks that node None is refused (shared_results); a planned range query equals its per-step instant answers and an unplanned one is refused (range_queries); exact, DDSketch and KLL views answer alike after a restart, another plan's files are not loaded, and windows before a restart are not stored empty (view_persistence); each persisted sketch kernel round-trips through bytes, a torn tail is cut off, waiters share the computer's result or failure, range steps are measured apart (unit and range_queries tests); /metrics reports the phases (server); the columnar read returns the row read's rows, pane eviction, and query-time result expiry (unit tests).
  • Adapted to the Planner's current inputs: fixed_interval becomes fixed_interval_at, and values_non_negative becomes column_signs. Two expectations changed with the Planner's selection:
    • The signed-sample DDSketch test now needs ignore_accuracy_guarantees, because there is no guaranteed plan without raw data. It is bounded by the plan's own α (0.1).
    • The joint-quantile test no longer asserts one shared sketch. The Planner now builds two per-series DDSketch chains (their Scans differ only by a region column); answers are still within α.
  • Result cache: planner_executor_parity asks every query (instant and range) twice with the cache on and once with it off; all three must agree with the reference. New result_cache tests: a repeated range runs no cached step; an instant query reads the range step at its time; a step answered before its raw samples arrived runs again (fails if non-final answers are cached); eviction by age and by LRU bytes, view reads by binary search and shared raw chunk conversion (unit tests).
  • cargo fmt --check, cargo clippy --workspace --all-targets -D warnings and cargo test --workspace pass (control_plane 7, data_plane lib 18, concurrency 1, no_forward 3, planner_executor_answers 10, planner_executor_parity 2, range_queries 3, series_ids 2, server 10, shared_results 2, view_persistence 5).
  • End to end: Prometheus 3.5 scraping itself every second sends Remote Write to data_plane --no-forward. Four planned queries (rate, increase, max_over_time, sum(sum_over_time)) all run from tumbling-window views refreshed at ingestion. After about 2 h, at three evaluation times, every answer equals Prometheus's own answer exactly (56 + 56 + 1 + 1 series). RSS stayed at 19 MB.

Not done yet

  • Views refreshed on first read (query-time AcrossEvaluations).
  • A UnivMon view whose heaps mix key types is not persisted (asap_sketchlib feat(query): count_over_time binds to FrequencyEstimate (CMS / CountSketch) #265).
  • At 10k series the views plan writes 6.5 MB per window; on a slow disk, syncing it limits refreshes (a separate sync thread or persisting fewer views would help).
  • Executor follow-ups from the profile: DDSketch merge clones before merging; group tables are not pre-sized.
  • Views are built from each window's raw samples rather than updated per sample; precompute CPU is now the ingest limit for view-heavy plans.
  • The end-to-end Prometheus run above predates this PR's threading changes and was not repeated.
  • A planned query is answered only once the windows it reads are refreshed, so a query at "now" waits for --allowed-lateness-ms.
  • The shards' own throughput limit is not measured: that needs several concurrent senders.
  • The Planner charges every step of a refreshed range; with the result cache only new steps cost.

Pins ASAPPlanner 1760a69f (asap-planner, asap-executor). The plan file is
now wire version 15: a PhysicalASAPDAG with each view's and query-time
Scan's retention_ms, and the query texts.

control_plane: plan_installable returns (InstallablePlan,
SelectedPlanReport); deployment inputs are DeploymentCapabilities and
CostInputs; --cost-inputs, --report; load_plan checks the plan compiles.
The backend declares query_time_results_retained: false (no view
refreshed on first read yet).

data_plane: the stored-summary, precompute, per-sample and raw-window
modules are replaced by the executor. A time-indexed store keeps raw
samples (Remote Write) and view rows (InMemoryDataStore); ingestion runs
one ingestion run per refresh group window; each query is one query run
of its root. Views are kept a margin past their retention so queries
behind ingestion are answered. Raw samples are no longer read from
Prometheus.

Tests: planner_executor_parity compares the data plane over HTTP with the
Planner executor in process on 14 public workloads, raw data kept or not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol added enhancement New feature or request area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration test: end-to-end Process-boundary, differential, conformance, and full-stack tests labels Nov 17, 2026
The deployment capabilities gained counter semantics, state families
not mergeable across windows and excluded candidates; the backend keeps
their defaults. The parity test binds its reference raw tables for
appending, since binding a table whole is now test-support only.
Queries no longer queue behind ingestion on the engine thread. The
engine thread publishes a snapshot of the store after each command;
query-N worker threads (--query-threads, default one per core) each run
one query to completion over the newest snapshot. A run's output stream
is not Send, so a run never moves between threads.

The store is copy on write: materialized views move from the executor's
InMemoryDataStore to a ViewStore whose clones share every batch, with
the same tiling read and retention. Raw data is cloned when a snapshot
still reads it; the next commit replaces it with shared columnar panes.
The engine thread becomes two: the ingest thread stores samples, keeps
the watermark and closes refresh windows; the precompute thread runs each
closed window's ingestion run and publishes the views. A refresh reads the
raw data as it was when its window closed (the snapshot travels with the
job), so it reads the samples the serial engine read. Closed windows not
yet refreshed are bounded (--max-precompute-backlog, default 256): when
the backlog is full the ingest thread waits, its queue fills and Remote
Write gets 503. The backlog is reported on /api/v1/status/asap.

Raw samples are held columnar by pane: each Remote Write request's
samples of a pane become one immutable chunk (time, value and a series
code per row; series identity text computed once per series when it is
interned); a pane the watermark has passed is compacted into one chunk;
eviction drops whole panes and reads never return samples at or before
the eviction cutoff. The pane width is the gcd of the refresh groups'
widths and phases and 60 s. A read returns one batch per chunk of the
panes overlapping the range, the open pane's chunks included, filtering
only chunks that straddle the range.

Tests: a query is answered from the refreshed views while a refresh is
held; pane eviction; the columnar read returns the row-wise read's rows.
…; ingest bench

Threads are named ingest, precompute and query-N. Each phase records its
thread CPU time (CLOCK_THREAD_CPUTIME_ID) and wall time: Remote Write
decode on the HTTP threads, append on the ingest thread, each refresh
group's ingestion runs on the precompute thread, and each planned query's
runs on the workers. GET /metrics exports them in the Prometheus text
format with query latency, sample and window counters, the queued samples
and precompute backlog, the watermark lag, the samples, panes and bytes
held as raw data, the bytes of views, RSS, and each thread's CPU time by
name.

scripts/profile.sh records a perf profile of a running data_plane and
writes a flamegraph. The ingest_bench example feeds Remote Write requests
in process and reports samples per second and CPU per thread.

The README describes the threads, the backlog bound and the metrics.
… per evaluation time

What runs once is the plan's decision, by each query-time node's
Lifetime. A OneEvaluation node that several queries' runs reach is cut
out of them into a run of its own; each query run binds it as a
materialized view slot (an on-first-read descriptor, so asap-executor
is unchanged). Its output at evaluation time T lives in the view store's
query-time partition, keyed by (T, node): the first run that needs it
computes it while concurrent runs wait for it (single flight), later
runs read it, and results are dropped once T is more than
--query-time-results-retention-ms behind the newest evaluation time or
when more than --query-time-results-max-bytes are held. A failed
computation is not kept. A plan that would compute any other node in
several queries' runs is refused at install. --no-shared-results
computes the node in every query's run, as before, for comparison.

/metrics exports the partition's bytes, entries, computes and hits.

Tests: two queries of #509 Example 1 sharing a OneEvaluation node, run
concurrently at two evaluation times, compute it once per time and answer
as runs that compute it themselves; a plan that marks that node None is
refused; the partition's single computation and expiry. The parity test
passes with sharing on (126 of 126 queries equal).
…r step

/api/v1/query_range parses query, start and end (Unix seconds) and step
(seconds or a Prometheus duration), refusing a non-positive step, an end
before the start or more than 11,000 steps with 400 bad_data as
Prometheus does. A planned query is evaluated at every step, the steps in
parallel on the query workers (a shared OneEvaluation node is computed
once per step time), and answered as a Prometheus matrix; it is not
answered if any step is not, and is then forwarded with the reason, or
refused under --no-forward, as an instant query is.

Tests: a planned range query's value at each step equals the instant
answer at that time; an unplanned range query is refused under
--no-forward; the parity harness plans the mixed workloads 01-03 with
range_evaluation over each query's last three evaluations and compares
every step with the Planner executor's query run at that time (162 of 162
steps equal).
Each view has one append-only file in --view-dir (default asap-views;
--no-view-persistence keeps views in memory only): one record per
refreshed window, a little-endian length and a MessagePack payload with
the window's (start, end] and rows, synced before the next window is
refreshed. At startup the records within each view's retention are
loaded back into the view store, and a window every view of its group
already stores is not refreshed again (windows_already_stored). A partial
record at a file's end is ignored. Raw samples and open windows are not
persisted.

Plain values and exact states (ExactAccumulator, IncreaseAccumulator,
which the executor serializes) are stored. Sketch states have no byte
encoding in asap_sketchlib or the executor yet, so a window of a view
holding one is not written; /metrics counts those windows
(asap_view_windows_not_persisted_total) beside the bytes written, the
sync time and the windows loaded.

EngineOptions is no longer Copy: it holds the view directory.

Test: a data plane restarted over the view files of an earlier one
answers a query as before without any sample written again.
docs/design/data-plane-demo.md describes the demo: goals and non-goals,
the ingest, precompute and query-worker threads, columnar panes and the
open pane, backpressure, OneEvaluation results shared per evaluation time
as the plan decides, range queries, view persistence, the /metrics
names, the evaluation design and the rejected alternatives. The README
links it.
The design records why one chain width makes long windows at a low
sample rate no cheaper than raw scans, and the planned levels of chains
(a coarse chain computed from the fine chain's view at ingestion time),
with their cost and the work they need. The alternatives table records
that stored views are bounded, separately from how a window is computed
and from freshness at the open window.
@zzylol zzylol added area: query engine PromQL lowering, execution, result shaping, and fallback behavior feature: range evaluation Range-query steps, windows, timestamps, and temporal evaluation documentation Improvements or additions to documentation labels Nov 20, 2026
…an; safe restarts

Sketch states are stored as asap_sketchlib's ASAPv1 bytes with the
kernel they rebuild (DDSketch, KLL, Count-Min, KMV, Hydra-KLL), read and
rebuilt through the executor kernels' public sketch fields. HyperLogLog,
UnivMon, Count-Min-with-heap and Hydra-Count-Min kernels expose no way
to rebuild their state, so views holding them are still not persisted
and are counted.

Review fixes:
- One file per refresh group, one record per window holding every view
  appended, one fdatasync per window (was one per view and window).
- Each file starts with a header naming the plan (a hash of the plan
  file) and the group; another plan's file is renamed *.stale and never
  loaded, so a changed plan cannot load wrong views.
- A torn or oversized last record is cut off at load, so records
  appended afterwards stay aligned; a garbage length no longer sizes an
  allocation.
- A window is skipped per view: a view that already stores it is not
  appended to again and no duplicate record is written, while the
  group's other views still get it.
- After a restart that loaded views, a declared stream start no longer
  makes the first window complete, so windows whose samples arrived
  before the restart stay missing instead of being stored empty.
- The ingest thread no longer overwrites windows_already_stored.

Tests: each persisted kernel round-trips through bytes (re-encoded
bytes, estimates and merges equal); torn tail and other-plan files;
exact, DDSketch and KLL views answer alike after a restart; another
plan's files are not loaded; windows before a restart are not stored
empty.
…orkers

- Query-time results: an entry is created in a computing state under the
  map lock, so expiry never drops one being computed and its bytes are
  counted only while it is held; a failure reaches the runs that waited
  for it and is not kept; a result is kept only for a time the watermark
  has passed (a client's far-future time is computed per run), and
  results expire behind the watermark, not behind the newest client time.
  The ingest thread publishes the watermark with the raw data.
- Range queries: at most one step per query worker in flight; the first
  unanswered step, or the client going away, drops the remaining steps,
  and a worker skips a step nobody waits for. Steps are measured as
  phase=query_range_step and requests as asap_range_query_latency_seconds,
  apart from instant queries. A range is refused, as Prometheus does,
  only when (end - start) / step exceeds 11,000.
- A panic in a query run answers that query with an error; the worker
  keeps serving.
- Smaller: no String per sample in ingest_at; a chunk's series identity
  dictionary is built on its first read; PublishedStore::run_store
  removed; series_ids doc; the bench reuses metrics::thread_cpu_seconds
  (clock ticks from sysconf), persists views by default and can split a
  scrape into many small requests.

Tests: single computation and expiry behind the watermark; waiters share
the computer's result or failure; range steps measured apart.
Copies, unchanged, ASAPPlanner's chains-10m-5m-3m-10s,
rollup-quantile-nested-labels and topk-by-job-and-global sketch-plan
workloads and the rangeheavy-t5-q05-q09-1h memo example, plus the demo's
dashboard workload, into the parity harness (raw data kept and not).
The dashboard plan's quantile chain is a KLL sketch, which seeds its
compaction coin at random, so a plan with KLL is compared within 5%;
every other plan exactly.
Adds how the precompute and query engines execute six plans of different
shapes, corrected to the implementation (refresh groups in plan order, a
Scan under two groups read by each, OneEvaluation nodes cut when read by
several queries' runs, range steps bounded per worker), and corrects the
design after review: late samples, view retention wording, the new public
surface, query memory beyond the run limit, worker panics, query-time
result retention behind the watermark, range-query bounds and metrics,
per-group view files with one sync per window tied to the plan, restart
behavior, sketch persistence and its gaps, and follow-ups.
The data_plane binary (and the benches) use jemalloc: Remote Write
decoding and query runs allocate many small, short-lived objects across
threads. The write handler's decode-and-intern step moves into
remote_write::decode_samples, and the decode_bench example measures it
per sample for two request shapes.

decode_bench, microseconds per sample (system allocator -> jemalloc):
one series per sample (10,000 x 1) 1.80 -> 1.51; 100 samples per series
(100 x 100) 0.083 -> 0.083.
…es cache

decode_samples walks the decompressed protobuf in place instead of
building a WriteRequest of owned Strings. Each TimeSeries' encoded label
bytes are the key of a per-thread cache of interned series, so a known
series costs one hash and one compare, with no lock and no allocation;
only a new series is parsed, validated (UTF-8, __name__, no duplicate
label) and interned. The samples vector is sized from the body up front.
Errors are those of the full decode.

decode_bench, microseconds per sample (jemalloc, before -> after): one
series per sample (10,000 x 1) 1.51 -> 0.38; 100 samples per series
(100 x 100) 0.083 -> 0.089.

Test: the in-place decode gives the full decode's samples, series and
errors.
…r file

View files are synced every --view-sync-interval-ms (default 5,000:
VictoriaMetrics' default -inmemoryDataFlushInterval as of v1.155.0)
instead of after every window, and the precompute thread also syncs when
no window closes, so a refreshed window is durable within that interval,
the same class of guarantee VictoriaMetrics gives its in-memory data; 0
syncs after every window. The files are synced when the engine stops.

A text (label value or series identity) is stored once per file: each
record carries the texts it adds to the file's dictionary and rows hold
indices. Loading replays the dictionary, and appends after a restart
continue it.

Test: a second restart after windows written since the first reads them
back alike (dictionary continued across restarts).
result_samples parsed every row's series identity JSON into a label map.
The interner now indexes series by identity text, so a row of an input
series takes its labels from the interned series; only a series the
query built (its labels changed) is parsed.
ingest_at looked each sample's metric up in a BTreeSet<String> and a
SipHash map keyed by an Arc<str> clone, and checked every refresh group
for a start. Samples of one metric come together, so the last metric is
checked by pointer first, the per-request metric list holds whether a
Scan reads it and its eviction cutoff (taken at its first routed sample,
as before), and groups are checked for a start only while one has not
started.
inferno stays the default; without it the FlameGraph perl scripts in
FLAMEGRAPH_DIR (default /mydata/bin/FlameGraph-41fee1f) draw the graph,
and the error names whichever tools are missing. Full stacks need
CARGO_PROFILE_RELEASE_DEBUG=true.
…e_from_disk

Pins ASAPPlanner main 056da657, whose executor gives every sketch kernel
one interface: sketch() to read its asap_sketchlib state and
restore_from_disk to rebuild it. All persisted kernels now use it, and
four more persist: HyperLogLog (with its precision), weighted frequency
(Count-Min or Count Sketch, with or without a heap), UnivMon (with its
key-byte counters) and Hydra-Count-Min, whose grid is shared by every
group of a build and is stored once per record and rebuilt once. A
UnivMon whose heaps mix key types has no ASAPv1 encoding; its windows
are counted as not persisted and the refresh goes on.

The design doc records the executor follow-ups the profile found
(DDSketch merge clones before merging; group tables not pre-sized).

Tests: every kernel round-trips through bytes (re-encoded bytes,
readouts and merges equal; a merged UnivMon compared on L2 only, since
its merge breaks heap ties nondeterministically); a shared Hydra-Count-Min
grid is stored once and answers every group as before.
…te path

The precompute thread publishes a refreshed window and queues its rows;
a pool of --persist-threads persist-N threads (default a quarter of the
cores, 1 to 4) encodes, writes and syncs them. Refresh group g goes to
thread g % n, so each file has one writer and keeps window order. The
queue is bounded by bytes (--max-persist-backlog-bytes, default 256 MiB);
only when it is full does the precompute thread wait, exported as
asap_view_persist_wait_seconds_total, with asap_persist_backlog_bytes and
asap_persist_backlog_windows. Each persist thread syncs its files every
--view-sync-interval-ms on its own timer, and writes and syncs everything
queued when the engine stops. An event-time barrier answers once the
windows before it are written, without the precompute thread waiting.

Tests: with every window's write delayed, windows are refreshed and
answer at once, and stopping the engine still writes every queued window;
with three persist threads, every query of a plan with several refresh
groups answers alike after a restart.
…t threads

WORKLOAD plans another workload and feeds every metric it names; PACED=1
sends one scrape per wall-clock second and reports the largest precompute
backlog and persist queue; PERSIST_THREADS sets the persist pool.
A Sample encoded as senders write it (value, tag 0x09 and 8 bytes, then
timestamp, tag 0x10 and a varint, nothing else) is read directly; any
other layout falls back to the generic field reader.

decode_bench, microseconds per sample: 100 samples per series 0.089 ->
0.070; one series per sample 0.38 -> 0.39 (unchanged).

Test: the fast and generic paths read the same samples, and other
layouts (fields swapped, an unknown field, a zero value) go to the
generic path.
zzylol and others added 18 commits November 20, 2026 20:30
Remote Write handlers send each series' samples to one of
--ingest-threads ingest-N shards (by series id; default a quarter of the
cores, at least 1). Each shard owns its raw data and routes and appends
its samples with no lock shared on the hot path, then reports its raw
data to an ingest coordinator. A request is admitted whole or refused
with 503 if any shard it reaches has --max-queued-samples queued.

The coordinator keeps the watermark: it follows the newest sample of any
shard, and before it moves every shard is flushed (runs the samples
queued to it and reports), so a window closes over every sample admitted
before it. Idle shards answer the flush at once and never hold the
watermark; the coordinator also wakes every 100 ms. It closes windows
over the union of the shards' raw data (shared chunk lists, no copy),
publishes the routing (scanned metrics, eviction cutoffs) the shards
route by, and has the shards seal and evict as the watermark moves.
Queries and refreshes read every shard. The precompute thread is
unchanged.

A first version took the watermark as the minimum over shards with
samples queued, without a flush; the parity test caught it losing
samples when requests spread over several shards.

Tests: views over four shards equal views over one, for per-minute
requests; idle shards do not hold the watermark; a full shard refuses the
whole request; parity passes with the default shard count.
A planned query's answer at evaluation time T is kept, keyed by (query
root, T), once it is final: the run succeeded (every view window it read
is stored, and stored windows never change) and, when the root reads raw
data through a Scan, T is at or before the watermark the run's snapshot
was published with. A range request looks up every step and runs only the
missing ones; an instant query at T reads a range step at T. Entries older
than the plan's longest retention plus the view retention margin behind
the watermark are dropped, and the least recently used beyond
--result-cache-bytes (256 MiB); --no-result-cache turns it off.

A run now reads the watermark before the snapshot, so a sample at or
before that watermark is in the snapshot (it was published after the
raw data), for the result cache and the shared query-time results.

Metrics: asap_result_cache_{hits,misses,evictions}_total,
asap_result_cache_{entries,bytes}, and per query
asap_range_query_steps{,_cached}_total. examples/range_bench measures a
dashboard refresh of 1 h ranges at 15 s steps.
Every step of a range request reads the store snapshot taken when the
request starts (raw shards, views and watermark), instead of one each.
Raw chunks a step reads whole are converted to batches once per request
and shared by the other steps (RawReads, sharded by chunk so parallel
steps rarely wait); a view read finds the windows tiling its range by
binary search instead of scanning every stored window.
…d its samples

With several ingest shards, a shard that had answered the coordinator's
flush went on to run later requests while slower shards still ran
earlier ones. Its reports were applied at once, so the watermark moved
past samples the slower shards had yet to append, and windows closed
without them: parity lost 16-20 answers per run (sums and top-k low),
varying run to run; one shard always passed.

- Reports a shard sends after answering a flush are held for the next
  pass, so they move neither this pass's watermark nor the raw data its
  windows read (Flushed now names its shard).
- A request is sent to its shards whole between two flushes: requests
  share an admission lock, a flush takes it alone.
- Test option shard_delay_ms slows every shard but the first. The new
  test a_shard_running_ahead_does_not_close_windows_early (8 shards,
  20 ms delay) compares every window with one shard; it fails without
  the fix (at 540 s) and passes with it.
- PARITY_INGEST_THREADS=n runs the parity test over n shards. With the
  fix: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4
  and 8 shards.
…SAPQuery does

A scraper sends one timestamp over several Remote Write requests. A
window (s, e] closed as soon as a sample at e arrived, so with no
allowed lateness the other series' samples at e, coming in a later
request, were dropped as late. The watermark from samples is now one
millisecond before the newest sample less the allowed lateness; a
barrier still counts whole. This follows the ASAPQuery precompute
engine's window manager (closed_windows waits for the watermark to pass
the window end).

Tests ported from the ASAPQuery precompute engine
(asap-query-engine/src/precompute_engine/), at the engine level over one
and eight ingest shards:
- samples_at_a_window_end_in_a_later_request_join_the_window
  (window_manager.rs boundary rule; fails with the old rule: 32 samples
  dropped over eight shards);
- a_late_sample_is_dropped_on_any_shard_count (worker.rs
  test_late_data_drop);
- idle_shards_do_not_hold_the_watermark now cites
  test_compute_global_watermark_ignores_unstarted.

The slow-disk persistence test writes through 61 s so the window ending
at 60 s closes. The design doc lists what was borrowed and where the
designs differ (one coordinator closes each window over every shard,
reaching the cross-shard minimum by a flush).

Parity: 148/148 queries, 162/162 range steps at 1 and 8 shards.
…e flush time

A feeder stamping samples ahead of the wall clock makes the watermark
follow the clock cap. The cap came from each shard's report time, and
a pass flushed only when the newest sample grew, so:
- a report that moved only the clock (or started a metric's groups)
  moved the watermark without a flush;
- even behind a flush, a shard working through a backlog reported the
  current time while running old requests, so the cap passed samples
  of requests admitted after that flush.
Windows then closed without samples the slow shards had yet to run and
those samples were dropped as late.

Now every pass that took a report, a held report or a barrier flushes
before the watermark moves, and the clock cap is the time the flush was
sent (taken under the admission lock): a request admitted later holds
samples stamped later. Shard reports no longer carry a time;
Ingestion::observe takes only the first samples, Ingestion::advance_clock
sets the cap, and a barrier no longer moves it.

Same rewrite of the coordinator pass, from the review:
- barriers that arrive while a flush is waited for are queued for the
  next pass, behind its own flush (they were closing windows without
  writes acknowledged before them, and a single deferred slot dropped a
  second barrier, whose caller got an error);
- the flush wait times out every tick and stops when an engine thread
  has died, failing the waiting barriers, instead of hanging.

Test a_watermark_capped_at_the_clock_moves_only_behind_a_flush: 64
series stamped up to 1 s ahead of the clock, 4 shards, 3 slowed by
250 ms per batch; asserts no late drops and every 1 s window's count.
Before the fix it dropped 7-11 samples per run; with it, 4/4 runs pass.

Parity: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4
and 8 shards. Precompute boundary unchanged (PrecomputeJob, jobs channel).
…enas

jemalloc's defaults (4 arenas per core, 10 s dirty decay, no background
threads) kept 300-450 MiB of freed pages resident in a data plane answering
a dashboard at 10k series per metric. The data plane now sets
background_thread, dirty_decay_ms 1000, muzzy_decay_ms 0 and narenas 8 at
startup (_RJEM_MALLOC_CONF still overrides them).
…s when appended

A view window rebuilt its series identity and label columns as a fresh
dictionary per window, so every identity was held once per window: about
25% of a 10k-series dashboard's view memory. ViewStore::append now keys
each text column into one values array per view and column, shared by
every window (the executor reads an ordinary dictionary column; entries a
window does not use are never read).

asap_view_bytes summed Batch::memory_bytes, which leaves out the states a
query run has read (the run marks them held outside it), so the gauge fell
by most of the view memory once queries ran. Each window's bytes are now
counted when it is appended, plus each view's shared texts once.
…ith ingest shards

With ingest shards, a shard counted its samples out of its queue as soon
as it sent its report. While the coordinator waited for the precompute
(jobs channel full), the shards kept draining into the unbounded
commands channel: Remote Write never got 503, eviction stopped, and the
queued reports pinned raw-data copies without bound.

Now the coordinator counts a report's samples out of the shard's queue
(and the total) when it applies the report: ShardReport carries the
batch's sample count. While the coordinator waits, samples stay counted,
the queues fill to --max-queued-samples, Remote Write gets 503, and the
reports waiting stay bounded by the queue limit.

Also: when sending a request to a shard fails, the queue counts of the
shards not yet sent to are counted out too (they leaked).

Test a_precompute_that_cannot_keep_up_reaches_remote_write: persist
queue full with 2 s writes, precompute backlog 1, 4 shards, queue limit
2,000; 40 one-minute requests. Before: all 40 admitted. Now: 503.

Interface at the coordinator/precompute boundary: unchanged
(PrecomputeJob over the bounded jobs channel; the coordinator blocks on
it). Only ShardReport gained `samples` and the count-out moved from
serve_shard to the coordinator's apply.

Parity: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4
and 8 shards.
Regression tests for the barrier handling rewritten in 8415cd6, over
four shards with three slowed by 300 ms per batch:
- a_barrier_during_another_barriers_flush_covers_writes_before_it: a
  write and a second barrier come while the first barrier's flush is
  waited for; the second barrier's windows hold that write. Fails when a
  barrier taken during the wait joins the current pass (sums low).
- barriers_during_one_flush_all_answer: two barriers come during one
  flush wait; both answer 200. Fails with a single deferred slot (500).

Also drops a duplicated doc line on Command::Flushed.

Parity on 1c67a13 with these tests: 148/148 queries and 162/162 range
steps, 3 runs each at 1, 4 and 8 shards; ingest_shards 9/9, 3 runs.
… is the limit

ingest_bench at 1, 2, 4 and 8 ingest threads, raw and views plans, 10k
and 100k series: acknowledged throughput stays at about 2.2-2.3M
samples/s (10k) and 1.7-1.8M (100k), bounded by the single in-process
client that decodes and sends every request; shard CPU grows with the
shard count. Measuring the shards' own limit needs several senders.
--precompute-threads (default a quarter of the cores, 1 to 4) computes
closed refresh windows in parallel: windows of different groups, and of
one group, run at once; a group's windows commit in window order through
a reorder buffer; the committer (the former precompute thread) keeps the
view store single-writer and publishes once per pass. The plan's refresh
task graph orders a group after the groups whose views it reads, and runs
a group serially when its sub-DAG holds an operator that may carry state
across windows. Per-group queue wait, commit wait and window-ready time,
and pool busy time, are exported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…y samples and panes

The byte comparison moves the watermark one minute at a time: a watermark
that jumps skips windows past their retention depending on timing, so two
one-thread runs differed. The input check compares samples and panes, not
bytes, since a chunk builds its series identity column on first read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
… skip fsync

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…compute threads

A refresh group whose nodes all compute each series alone (time-series
Scans keeping the series identity, row filters, and summaries with a
PerEntity reduction) and that reads no other group's views has each
window computed as one run per precompute thread, each over the series
whose id falls in its part. The committer merges the parts' rows into
the executor's group key order, so the views are byte-identical to one
run's, and commits the window once every part is in. Any other group is
computed whole; install logs which, and /metrics reports
asap_precompute_window_parts per group.

The 60 s, 10 Hz dashboard workload (one group of twelve per-series
views) is added to the test workloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…e threads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths area: query engine PromQL lowering, execution, result shaping, and fallback behavior documentation Improvements or additions to documentation enhancement New feature or request feature: range evaluation Range-query steps, windows, timestamps, and temporal evaluation integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration test: end-to-end Process-boundary, differential, conformance, and full-stack tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant