Repository navigation
Conversation
Pins ASAPPlanner 1760a69f (asap-planner, asap-executor). The plan file is now wire version 15: a PhysicalASAPDAG with each view's and query-time Scan's retention_ms, and the query texts. control_plane: plan_installable returns (InstallablePlan, SelectedPlanReport); deployment inputs are DeploymentCapabilities and CostInputs; --cost-inputs, --report; load_plan checks the plan compiles. The backend declares query_time_results_retained: false (no view refreshed on first read yet). data_plane: the stored-summary, precompute, per-sample and raw-window modules are replaced by the executor. A time-indexed store keeps raw samples (Remote Write) and view rows (InMemoryDataStore); ingestion runs one ingestion run per refresh group window; each query is one query run of its root. Views are kept a margin past their retention so queries behind ingestion are answered. Raw samples are no longer read from Prometheus. Tests: planner_executor_parity compares the data plane over HTTP with the Planner executor in process on 14 public workloads, raw data kept or not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployment capabilities gained counter semantics, state families not mergeable across windows and excluded candidates; the backend keeps their defaults. The parity test binds its reference raw tables for appending, since binding a table whole is now test-support only.
Queries no longer queue behind ingestion on the engine thread. The engine thread publishes a snapshot of the store after each command; query-N worker threads (--query-threads, default one per core) each run one query to completion over the newest snapshot. A run's output stream is not Send, so a run never moves between threads. The store is copy on write: materialized views move from the executor's InMemoryDataStore to a ViewStore whose clones share every batch, with the same tiling read and retention. Raw data is cloned when a snapshot still reads it; the next commit replaces it with shared columnar panes.
The engine thread becomes two: the ingest thread stores samples, keeps the watermark and closes refresh windows; the precompute thread runs each closed window's ingestion run and publishes the views. A refresh reads the raw data as it was when its window closed (the snapshot travels with the job), so it reads the samples the serial engine read. Closed windows not yet refreshed are bounded (--max-precompute-backlog, default 256): when the backlog is full the ingest thread waits, its queue fills and Remote Write gets 503. The backlog is reported on /api/v1/status/asap. Raw samples are held columnar by pane: each Remote Write request's samples of a pane become one immutable chunk (time, value and a series code per row; series identity text computed once per series when it is interned); a pane the watermark has passed is compacted into one chunk; eviction drops whole panes and reads never return samples at or before the eviction cutoff. The pane width is the gcd of the refresh groups' widths and phases and 60 s. A read returns one batch per chunk of the panes overlapping the range, the open pane's chunks included, filtering only chunks that straddle the range. Tests: a query is answered from the refreshed views while a refresh is held; pane eviction; the columnar read returns the row-wise read's rows.
…; ingest bench Threads are named ingest, precompute and query-N. Each phase records its thread CPU time (CLOCK_THREAD_CPUTIME_ID) and wall time: Remote Write decode on the HTTP threads, append on the ingest thread, each refresh group's ingestion runs on the precompute thread, and each planned query's runs on the workers. GET /metrics exports them in the Prometheus text format with query latency, sample and window counters, the queued samples and precompute backlog, the watermark lag, the samples, panes and bytes held as raw data, the bytes of views, RSS, and each thread's CPU time by name. scripts/profile.sh records a perf profile of a running data_plane and writes a flamegraph. The ingest_bench example feeds Remote Write requests in process and reports samples per second and CPU per thread. The README describes the threads, the backlog bound and the metrics.
… per evaluation time What runs once is the plan's decision, by each query-time node's Lifetime. A OneEvaluation node that several queries' runs reach is cut out of them into a run of its own; each query run binds it as a materialized view slot (an on-first-read descriptor, so asap-executor is unchanged). Its output at evaluation time T lives in the view store's query-time partition, keyed by (T, node): the first run that needs it computes it while concurrent runs wait for it (single flight), later runs read it, and results are dropped once T is more than --query-time-results-retention-ms behind the newest evaluation time or when more than --query-time-results-max-bytes are held. A failed computation is not kept. A plan that would compute any other node in several queries' runs is refused at install. --no-shared-results computes the node in every query's run, as before, for comparison. /metrics exports the partition's bytes, entries, computes and hits. Tests: two queries of #509 Example 1 sharing a OneEvaluation node, run concurrently at two evaluation times, compute it once per time and answer as runs that compute it themselves; a plan that marks that node None is refused; the partition's single computation and expiry. The parity test passes with sharing on (126 of 126 queries equal).
…r step /api/v1/query_range parses query, start and end (Unix seconds) and step (seconds or a Prometheus duration), refusing a non-positive step, an end before the start or more than 11,000 steps with 400 bad_data as Prometheus does. A planned query is evaluated at every step, the steps in parallel on the query workers (a shared OneEvaluation node is computed once per step time), and answered as a Prometheus matrix; it is not answered if any step is not, and is then forwarded with the reason, or refused under --no-forward, as an instant query is. Tests: a planned range query's value at each step equals the instant answer at that time; an unplanned range query is refused under --no-forward; the parity harness plans the mixed workloads 01-03 with range_evaluation over each query's last three evaluations and compares every step with the Planner executor's query run at that time (162 of 162 steps equal).
Each view has one append-only file in --view-dir (default asap-views; --no-view-persistence keeps views in memory only): one record per refreshed window, a little-endian length and a MessagePack payload with the window's (start, end] and rows, synced before the next window is refreshed. At startup the records within each view's retention are loaded back into the view store, and a window every view of its group already stores is not refreshed again (windows_already_stored). A partial record at a file's end is ignored. Raw samples and open windows are not persisted. Plain values and exact states (ExactAccumulator, IncreaseAccumulator, which the executor serializes) are stored. Sketch states have no byte encoding in asap_sketchlib or the executor yet, so a window of a view holding one is not written; /metrics counts those windows (asap_view_windows_not_persisted_total) beside the bytes written, the sync time and the windows loaded. EngineOptions is no longer Copy: it holds the view directory. Test: a data plane restarted over the view files of an earlier one answers a query as before without any sample written again.
docs/design/data-plane-demo.md describes the demo: goals and non-goals, the ingest, precompute and query-worker threads, columnar panes and the open pane, backpressure, OneEvaluation results shared per evaluation time as the plan decides, range queries, view persistence, the /metrics names, the evaluation design and the rejected alternatives. The README links it.
The design records why one chain width makes long windows at a low sample rate no cheaper than raw scans, and the planned levels of chains (a coarse chain computed from the fine chain's view at ingestion time), with their cost and the work they need. The alternatives table records that stored views are bounded, separately from how a window is computed and from freshness at the open window.
…an; safe restarts Sketch states are stored as asap_sketchlib's ASAPv1 bytes with the kernel they rebuild (DDSketch, KLL, Count-Min, KMV, Hydra-KLL), read and rebuilt through the executor kernels' public sketch fields. HyperLogLog, UnivMon, Count-Min-with-heap and Hydra-Count-Min kernels expose no way to rebuild their state, so views holding them are still not persisted and are counted. Review fixes: - One file per refresh group, one record per window holding every view appended, one fdatasync per window (was one per view and window). - Each file starts with a header naming the plan (a hash of the plan file) and the group; another plan's file is renamed *.stale and never loaded, so a changed plan cannot load wrong views. - A torn or oversized last record is cut off at load, so records appended afterwards stay aligned; a garbage length no longer sizes an allocation. - A window is skipped per view: a view that already stores it is not appended to again and no duplicate record is written, while the group's other views still get it. - After a restart that loaded views, a declared stream start no longer makes the first window complete, so windows whose samples arrived before the restart stay missing instead of being stored empty. - The ingest thread no longer overwrites windows_already_stored. Tests: each persisted kernel round-trips through bytes (re-encoded bytes, estimates and merges equal); torn tail and other-plan files; exact, DDSketch and KLL views answer alike after a restart; another plan's files are not loaded; windows before a restart are not stored empty.
…orkers - Query-time results: an entry is created in a computing state under the map lock, so expiry never drops one being computed and its bytes are counted only while it is held; a failure reaches the runs that waited for it and is not kept; a result is kept only for a time the watermark has passed (a client's far-future time is computed per run), and results expire behind the watermark, not behind the newest client time. The ingest thread publishes the watermark with the raw data. - Range queries: at most one step per query worker in flight; the first unanswered step, or the client going away, drops the remaining steps, and a worker skips a step nobody waits for. Steps are measured as phase=query_range_step and requests as asap_range_query_latency_seconds, apart from instant queries. A range is refused, as Prometheus does, only when (end - start) / step exceeds 11,000. - A panic in a query run answers that query with an error; the worker keeps serving. - Smaller: no String per sample in ingest_at; a chunk's series identity dictionary is built on its first read; PublishedStore::run_store removed; series_ids doc; the bench reuses metrics::thread_cpu_seconds (clock ticks from sysconf), persists views by default and can split a scrape into many small requests. Tests: single computation and expiry behind the watermark; waiters share the computer's result or failure; range steps measured apart.
Copies, unchanged, ASAPPlanner's chains-10m-5m-3m-10s, rollup-quantile-nested-labels and topk-by-job-and-global sketch-plan workloads and the rangeheavy-t5-q05-q09-1h memo example, plus the demo's dashboard workload, into the parity harness (raw data kept and not). The dashboard plan's quantile chain is a KLL sketch, which seeds its compaction coin at random, so a plan with KLL is compared within 5%; every other plan exactly.
Adds how the precompute and query engines execute six plans of different shapes, corrected to the implementation (refresh groups in plan order, a Scan under two groups read by each, OneEvaluation nodes cut when read by several queries' runs, range steps bounded per worker), and corrects the design after review: late samples, view retention wording, the new public surface, query memory beyond the run limit, worker panics, query-time result retention behind the watermark, range-query bounds and metrics, per-group view files with one sync per window tied to the plan, restart behavior, sketch persistence and its gaps, and follow-ups.
The data_plane binary (and the benches) use jemalloc: Remote Write decoding and query runs allocate many small, short-lived objects across threads. The write handler's decode-and-intern step moves into remote_write::decode_samples, and the decode_bench example measures it per sample for two request shapes. decode_bench, microseconds per sample (system allocator -> jemalloc): one series per sample (10,000 x 1) 1.80 -> 1.51; 100 samples per series (100 x 100) 0.083 -> 0.083.
…es cache decode_samples walks the decompressed protobuf in place instead of building a WriteRequest of owned Strings. Each TimeSeries' encoded label bytes are the key of a per-thread cache of interned series, so a known series costs one hash and one compare, with no lock and no allocation; only a new series is parsed, validated (UTF-8, __name__, no duplicate label) and interned. The samples vector is sized from the body up front. Errors are those of the full decode. decode_bench, microseconds per sample (jemalloc, before -> after): one series per sample (10,000 x 1) 1.51 -> 0.38; 100 samples per series (100 x 100) 0.083 -> 0.089. Test: the in-place decode gives the full decode's samples, series and errors.
…r file View files are synced every --view-sync-interval-ms (default 5,000: VictoriaMetrics' default -inmemoryDataFlushInterval as of v1.155.0) instead of after every window, and the precompute thread also syncs when no window closes, so a refreshed window is durable within that interval, the same class of guarantee VictoriaMetrics gives its in-memory data; 0 syncs after every window. The files are synced when the engine stops. A text (label value or series identity) is stored once per file: each record carries the texts it adds to the file's dictionary and rows hold indices. Loading replays the dictionary, and appends after a restart continue it. Test: a second restart after windows written since the first reads them back alike (dictionary continued across restarts).
result_samples parsed every row's series identity JSON into a label map. The interner now indexes series by identity text, so a row of an input series takes its labels from the interned series; only a series the query built (its labels changed) is parsed.
ingest_at looked each sample's metric up in a BTreeSet<String> and a SipHash map keyed by an Arc<str> clone, and checked every refresh group for a start. Samples of one metric come together, so the last metric is checked by pointer first, the per-request metric list holds whether a Scan reads it and its eviction cutoff (taken at its first routed sample, as before), and groups are checked for a start only while one has not started.
inferno stays the default; without it the FlameGraph perl scripts in FLAMEGRAPH_DIR (default /mydata/bin/FlameGraph-41fee1f) draw the graph, and the error names whichever tools are missing. Full stacks need CARGO_PROFILE_RELEASE_DEBUG=true.
…e_from_disk Pins ASAPPlanner main 056da657, whose executor gives every sketch kernel one interface: sketch() to read its asap_sketchlib state and restore_from_disk to rebuild it. All persisted kernels now use it, and four more persist: HyperLogLog (with its precision), weighted frequency (Count-Min or Count Sketch, with or without a heap), UnivMon (with its key-byte counters) and Hydra-Count-Min, whose grid is shared by every group of a build and is stored once per record and rebuilt once. A UnivMon whose heaps mix key types has no ASAPv1 encoding; its windows are counted as not persisted and the refresh goes on. The design doc records the executor follow-ups the profile found (DDSketch merge clones before merging; group tables not pre-sized). Tests: every kernel round-trips through bytes (re-encoded bytes, readouts and merges equal; a merged UnivMon compared on L2 only, since its merge breaks heap ties nondeterministically); a shared Hydra-Count-Min grid is stored once and answers every group as before.
…te path The precompute thread publishes a refreshed window and queues its rows; a pool of --persist-threads persist-N threads (default a quarter of the cores, 1 to 4) encodes, writes and syncs them. Refresh group g goes to thread g % n, so each file has one writer and keeps window order. The queue is bounded by bytes (--max-persist-backlog-bytes, default 256 MiB); only when it is full does the precompute thread wait, exported as asap_view_persist_wait_seconds_total, with asap_persist_backlog_bytes and asap_persist_backlog_windows. Each persist thread syncs its files every --view-sync-interval-ms on its own timer, and writes and syncs everything queued when the engine stops. An event-time barrier answers once the windows before it are written, without the precompute thread waiting. Tests: with every window's write delayed, windows are refreshed and answer at once, and stopping the engine still writes every queued window; with three persist threads, every query of a plan with several refresh groups answers alike after a restart.
…t threads WORKLOAD plans another workload and feeds every metric it names; PACED=1 sends one scrape per wall-clock second and reports the largest precompute backlog and persist queue; PERSIST_THREADS sets the persist pool.
A Sample encoded as senders write it (value, tag 0x09 and 8 bytes, then timestamp, tag 0x10 and a varint, nothing else) is read directly; any other layout falls back to the generic field reader. decode_bench, microseconds per sample: 100 samples per series 0.089 -> 0.070; one series per sample 0.38 -> 0.39 (unchanged). Test: the fast and generic paths read the same samples, and other layouts (fields swapped, an unknown field, a zero value) go to the generic path.
Remote Write handlers send each series' samples to one of --ingest-threads ingest-N shards (by series id; default a quarter of the cores, at least 1). Each shard owns its raw data and routes and appends its samples with no lock shared on the hot path, then reports its raw data to an ingest coordinator. A request is admitted whole or refused with 503 if any shard it reaches has --max-queued-samples queued. The coordinator keeps the watermark: it follows the newest sample of any shard, and before it moves every shard is flushed (runs the samples queued to it and reports), so a window closes over every sample admitted before it. Idle shards answer the flush at once and never hold the watermark; the coordinator also wakes every 100 ms. It closes windows over the union of the shards' raw data (shared chunk lists, no copy), publishes the routing (scanned metrics, eviction cutoffs) the shards route by, and has the shards seal and evict as the watermark moves. Queries and refreshes read every shard. The precompute thread is unchanged. A first version took the watermark as the minimum over shards with samples queued, without a flush; the parity test caught it losing samples when requests spread over several shards. Tests: views over four shards equal views over one, for per-minute requests; idle shards do not hold the watermark; a full shard refuses the whole request; parity passes with the default shard count.
A planned query's answer at evaluation time T is kept, keyed by (query
root, T), once it is final: the run succeeded (every view window it read
is stored, and stored windows never change) and, when the root reads raw
data through a Scan, T is at or before the watermark the run's snapshot
was published with. A range request looks up every step and runs only the
missing ones; an instant query at T reads a range step at T. Entries older
than the plan's longest retention plus the view retention margin behind
the watermark are dropped, and the least recently used beyond
--result-cache-bytes (256 MiB); --no-result-cache turns it off.
A run now reads the watermark before the snapshot, so a sample at or
before that watermark is in the snapshot (it was published after the
raw data), for the result cache and the shared query-time results.
Metrics: asap_result_cache_{hits,misses,evictions}_total,
asap_result_cache_{entries,bytes}, and per query
asap_range_query_steps{,_cached}_total. examples/range_bench measures a
dashboard refresh of 1 h ranges at 15 s steps.
Every step of a range request reads the store snapshot taken when the request starts (raw shards, views and watermark), instead of one each. Raw chunks a step reads whole are converted to batches once per request and shared by the other steps (RawReads, sharded by chunk so parallel steps rarely wait); a view read finds the windows tiling its range by binary search instead of scanning every stored window.
…d its samples With several ingest shards, a shard that had answered the coordinator's flush went on to run later requests while slower shards still ran earlier ones. Its reports were applied at once, so the watermark moved past samples the slower shards had yet to append, and windows closed without them: parity lost 16-20 answers per run (sums and top-k low), varying run to run; one shard always passed. - Reports a shard sends after answering a flush are held for the next pass, so they move neither this pass's watermark nor the raw data its windows read (Flushed now names its shard). - A request is sent to its shards whole between two flushes: requests share an admission lock, a flush takes it alone. - Test option shard_delay_ms slows every shard but the first. The new test a_shard_running_ahead_does_not_close_windows_early (8 shards, 20 ms delay) compares every window with one shard; it fails without the fix (at 540 s) and passes with it. - PARITY_INGEST_THREADS=n runs the parity test over n shards. With the fix: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards.
…SAPQuery does A scraper sends one timestamp over several Remote Write requests. A window (s, e] closed as soon as a sample at e arrived, so with no allowed lateness the other series' samples at e, coming in a later request, were dropped as late. The watermark from samples is now one millisecond before the newest sample less the allowed lateness; a barrier still counts whole. This follows the ASAPQuery precompute engine's window manager (closed_windows waits for the watermark to pass the window end). Tests ported from the ASAPQuery precompute engine (asap-query-engine/src/precompute_engine/), at the engine level over one and eight ingest shards: - samples_at_a_window_end_in_a_later_request_join_the_window (window_manager.rs boundary rule; fails with the old rule: 32 samples dropped over eight shards); - a_late_sample_is_dropped_on_any_shard_count (worker.rs test_late_data_drop); - idle_shards_do_not_hold_the_watermark now cites test_compute_global_watermark_ignores_unstarted. The slow-disk persistence test writes through 61 s so the window ending at 60 s closes. The design doc lists what was borrowed and where the designs differ (one coordinator closes each window over every shard, reaching the cross-shard minimum by a flush). Parity: 148/148 queries, 162/162 range steps at 1 and 8 shards.
…e flush time A feeder stamping samples ahead of the wall clock makes the watermark follow the clock cap. The cap came from each shard's report time, and a pass flushed only when the newest sample grew, so: - a report that moved only the clock (or started a metric's groups) moved the watermark without a flush; - even behind a flush, a shard working through a backlog reported the current time while running old requests, so the cap passed samples of requests admitted after that flush. Windows then closed without samples the slow shards had yet to run and those samples were dropped as late. Now every pass that took a report, a held report or a barrier flushes before the watermark moves, and the clock cap is the time the flush was sent (taken under the admission lock): a request admitted later holds samples stamped later. Shard reports no longer carry a time; Ingestion::observe takes only the first samples, Ingestion::advance_clock sets the cap, and a barrier no longer moves it. Same rewrite of the coordinator pass, from the review: - barriers that arrive while a flush is waited for are queued for the next pass, behind its own flush (they were closing windows without writes acknowledged before them, and a single deferred slot dropped a second barrier, whose caller got an error); - the flush wait times out every tick and stops when an engine thread has died, failing the waiting barriers, instead of hanging. Test a_watermark_capped_at_the_clock_moves_only_behind_a_flush: 64 series stamped up to 1 s ahead of the clock, 4 shards, 3 slowed by 250 ms per batch; asserts no late drops and every 1 s window's count. Before the fix it dropped 7-11 samples per run; with it, 4/4 runs pass. Parity: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards. Precompute boundary unchanged (PrecomputeJob, jobs channel).
…enas jemalloc's defaults (4 arenas per core, 10 s dirty decay, no background threads) kept 300-450 MiB of freed pages resident in a data plane answering a dashboard at 10k series per metric. The data plane now sets background_thread, dirty_decay_ms 1000, muzzy_decay_ms 0 and narenas 8 at startup (_RJEM_MALLOC_CONF still overrides them).
…s when appended A view window rebuilt its series identity and label columns as a fresh dictionary per window, so every identity was held once per window: about 25% of a 10k-series dashboard's view memory. ViewStore::append now keys each text column into one values array per view and column, shared by every window (the executor reads an ordinary dictionary column; entries a window does not use are never read). asap_view_bytes summed Batch::memory_bytes, which leaves out the states a query run has read (the run marks them held outside it), so the gauge fell by most of the view memory once queries ran. Each window's bytes are now counted when it is appended, plus each view's shared texts once.
…ith ingest shards With ingest shards, a shard counted its samples out of its queue as soon as it sent its report. While the coordinator waited for the precompute (jobs channel full), the shards kept draining into the unbounded commands channel: Remote Write never got 503, eviction stopped, and the queued reports pinned raw-data copies without bound. Now the coordinator counts a report's samples out of the shard's queue (and the total) when it applies the report: ShardReport carries the batch's sample count. While the coordinator waits, samples stay counted, the queues fill to --max-queued-samples, Remote Write gets 503, and the reports waiting stay bounded by the queue limit. Also: when sending a request to a shard fails, the queue counts of the shards not yet sent to are counted out too (they leaked). Test a_precompute_that_cannot_keep_up_reaches_remote_write: persist queue full with 2 s writes, precompute backlog 1, 4 shards, queue limit 2,000; 40 one-minute requests. Before: all 40 admitted. Now: 503. Interface at the coordinator/precompute boundary: unchanged (PrecomputeJob over the bounded jobs channel; the coordinator blocks on it). Only ShardReport gained `samples` and the count-out moved from serve_shard to the coordinator's apply. Parity: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards.
Regression tests for the barrier handling rewritten in 8415cd6, over four shards with three slowed by 300 ms per batch: - a_barrier_during_another_barriers_flush_covers_writes_before_it: a write and a second barrier come while the first barrier's flush is waited for; the second barrier's windows hold that write. Fails when a barrier taken during the wait joins the current pass (sums low). - barriers_during_one_flush_all_answer: two barriers come during one flush wait; both answer 200. Fails with a single deferred slot (500). Also drops a duplicated doc line on Command::Flushed. Parity on 1c67a13 with these tests: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards; ingest_shards 9/9, 3 runs.
… is the limit ingest_bench at 1, 2, 4 and 8 ingest threads, raw and views plans, 10k and 100k series: acknowledged throughput stays at about 2.2-2.3M samples/s (10k) and 1.7-1.8M (100k), bounded by the single in-process client that decodes and sends every request; shard CPU grows with the shard count. Measuring the shards' own limit needs several senders.
--precompute-threads (default a quarter of the cores, 1 to 4) computes closed refresh windows in parallel: windows of different groups, and of one group, run at once; a group's windows commit in window order through a reorder buffer; the committer (the former precompute thread) keeps the view store single-writer and publishes once per pass. The plan's refresh task graph orders a group after the groups whose views it reads, and runs a group serially when its sub-DAG holds an operator that may carry state across windows. Per-group queue wait, commit wait and window-ready time, and pool busy time, are exported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…y samples and panes The byte comparison moves the watermark one minute at a time: a watermark that jumps skips windows past their retention depending on timing, so two one-thread runs differed. The input check compares samples and panes, not bytes, since a chunk builds its series identity column on first read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
… skip fsync Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…compute threads A refresh group whose nodes all compute each series alone (time-series Scans keeping the series identity, row filters, and summaries with a PerEntity reduction) and that reads no other group's views has each window computed as one run per precompute thread, each over the series whose id falls in its part. The committer merges the parts' rows into the executor's group key order, so the views are byte-identical to one run's, and commits the window once every part is in. Any other group is computed whole; install logs which, and /metrics reports asap_precompute_window_parts per group. The 60 s, 10 Hz dashboard workload (one group of twelve per-series views) is added to the test workloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
…e threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bo6sg4cEjJJhktNtL9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design: docs/design/data-plane-demo.md (the data plane for the comparison with VictoriaMetrics: threads, columnar panes, shared query-time results, range queries, view persistence, metrics, evaluation design).
Supersedes #845. #845 bumps the ASAPPlanner pin only and does not compile: the Planner's plan file, executor API and planning inputs changed under it (66 compile errors). This PR pins ASAPPlanner main
056da657and runs the plan through the Planner's executor.What changes for a user
control_planewritesplan.json(wire version 15) and, with--report report.json, Stage 3'sSelectedPlanReport(cost, predicted usage per node and per query).--plan-selection-settingsis replaced by--cost-inputs.--measured-operation-costsand--ignore-accuracy-guaranteesare kept.data_planekeeps the raw samples it receives by Remote Write. Planned queries no longer read raw samples from Prometheus. New flags:--run-memory-limit-bytesand--view-retention-margin-ms./api/v1/status/asapreportsingestioncounters (samples, and windows refreshed, failed or skipped) in place ofprecomputeandquery_run.query_time_results_retained: false, so the Planner selects no view refreshed on first read. The data plane does not refresh such views yet./api/v1/query_range), one evaluation per step.--view-dir, defaultasap-views;--no-view-persistence): one file per refresh group, one record and onefdatasyncper window, tied to the plan by a hash; sketch states asasap_sketchlib's ASAPv1 bytes.GET /metricsexports a per-phase CPU breakdown and the data plane's counters in the Prometheus text format.--query-threads,--max-precompute-backlog,--no-shared-results,--query-time-results-retention-ms,--query-time-results-max-bytes,--view-dir,--no-view-persistence,--result-cache-bytes,--no-result-cache.How the data plane runs a plan
InstallablePlan::from_json,validate,materialized_views,ingestion_refresh_groupsDataStore: raw samples per scanned metric as(ts, value, $promql_series_identity, labels...), columnar by pane; view rows in a copy-on-write view store with the executor's tiling readW, phasep), oneexecute_ingestionrun per window(e - W, e]once the watermark reachese; rows appended with the range they covercompilewith the views it reads,Scope::Query), rows turned into a PromQL vector; a query-timeOneEvaluationnode several queries read is compiled on its own and bound as a view slotThe plan's retention counts back from the newest evaluation, but ingestion refreshes windows ahead of the queries it answers. Views and query-time raw samples are therefore kept
--view-retention-margin-ms(default 60 s) past their retention. The end-to-end run found this: without the margin, a query at a time 20 s behind ingestion found its oldest windows already dropped.Threads (before → after)
Before: one engine thread ran Remote Write appends, every closed window's ingestion run and every query in turn. While a window's views were computed, Remote Write queued (503 when full) and queries waited.
After:
ingest-0..N(--ingest-threads, default a quarter of the cores)ingest(coordinator)precomputeexecute_ingestionrun per closed window, over the raw data as it was when the window closed; appends and syncs view filesquery-0..N(--query-threads, default one per core)A run's output stream is not
Send, so a run never moves between threads; what crosses threads (QueryEngine,RunStore,RawData,ViewStore,Precompute) is checkedSend/Syncat compile time. At most--max-precompute-backlog(256) windows wait for precompute; when full, the coordinator waits, the shards' queues fill and Remote Write gets 503 (see below). A query does not wait for queued writes; the event-time barrier does (it answers once its windows are refreshed).Ingest shards: correctness and backpressure (6d40b3d..8682b64)
With several ingest shards, parity lost 16–20 answers per run (sums and top-k low): windows closed before every shard had appended its samples. The rule is now: the watermark moves only behind a flush, and every pass that took a report or a barrier sends one. A flush means every shard has run what was admitted before it.
--max-queued-samples, Remote Write gets 503, and the reports waiting stay bounded. Before, the shards kept draining into an unbounded channel: no 503, no eviction, unbounded memory.Borrowed from the ASAPQuery precompute engine (
asap-query-engine/src/precompute_engine/): routing by series hash, a cross-worker watermark where workers with no samples do not hold it back, a periodic tick, the close-past-the-end rule, and dropping and counting late samples. Its watermark, unstarted-worker, boundary and late-data tests are ported at the engine level. ASAPQuery closes each worker's own partial windows; here one coordinator closes a window over every shard's raw data, so the cross-shard minimum is reached by the flush.struct ShardReport { shard: usize, raw_data: Arc<RawData>, newest_ms: Option<i64>, first_seen: Vec<(Arc<str>, i64)>, - now_ms: i64, + /// Samples of the batch, counted out of the queues when applied. + samples: u64, } enum Command { Report(ShardReport), - Flushed, + /// The shard numbered here has run every message sent before the flush. + Flushed(usize), AdvanceWatermark { barrier_ms: i64, reply: oneshot::Sender<i64> }, Stop, } -fn flush(shards: &[Sender<ShardMessage>], shared: &Shared) +/// Sends a flush under the admission lock; returns the time it was sent. +fn flush(shards: &[Sender<ShardMessage>], shared: &Shared) -> i64 impl Ingestion { - pub fn observe(&mut self, first_seen: &[(Arc<str>, i64)], now_ms: i64) + pub fn observe(&mut self, first_seen: &[(Arc<str>, i64)]) + /// Sets the clock the watermark is capped at: the last flush's time. + pub fn advance_clock(&mut self, now_ms: i64) } pub struct EngineOptions { + /// For tests: every shard but the first waits this long per batch. + #[doc(hidden)] + pub shard_delay_ms: u64, }The boundary between the coordinator and the precompute is unchanged:
PrecomputeJobgoes over the bounded jobs channel, and the coordinator blocks on it.Regression tests (
data_plane/tests/ingest_shards.rs, one engine against one shard or several). Each was shown to fail on the old behaviour:a_shard_running_ahead_does_not_close_windows_early(8 shards, 7 slowed by 20 ms)a_watermark_capped_at_the_clock_moves_only_behind_a_flush(samples up to 1 s ahead of the clock, 4 shards, 3 slowed by 250 ms)samples_at_a_window_end_in_a_later_request_join_the_window(1 and 8 shards)a_late_sample_is_dropped_on_any_shard_count(from ASAPQuery'stest_late_data_drop)a_precompute_that_cannot_keep_up_reaches_remote_writea_barrier_during_another_barriers_flush_covers_writes_before_itbarriers_during_one_flush_all_answerParity (
PARITY_INGEST_THREADS=n), on 8682b64's code: 148/148 queries and 162/162 range steps, 3 runs each at 1, 4 and 8 shards.ingest_shards: 9/9, 3 runs.Columnar raw store
Raw samples are held per pane (width: gcd of the refresh groups' widths and phases and 60 s). Each Remote Write request's samples of a pane are one immutable chunk: time and value arrays and a per-row series code into the chunk's series, whose identity JSON is computed once when the series is interned. A pane the watermark has passed is compacted into one chunk. Eviction drops whole panes, and a read never returns a sample at or before the eviction cutoff, so answers match the row store exactly.
Open pane, chosen for query-time
Scans: the ingest thread publishes the raw data after every request (copy on write: per-pane chunk lists are copied, chunks are shared), and a read returns one batch per chunk of every pane overlapping the range, the open pane's chunks so far included; only a chunk straddling the range is filtered. This is the simplest option that needs no lock and no copy of the open pane per read; its cost is many small batches when many small requests fall in one open pane.Shared query-time results (the plan decides)
A
OneEvaluationquery-time node that several queries' runs reach is computed once per evaluation time T: the first run computes it while concurrent runs wait (single flight), later runs read it from the view store's query-time partition, keyed by (T, node), kept--query-time-results-retention-ms(60 s) behind the newest T, bounded by--query-time-results-max-bytes. Any other node two queries' runs would compute makes the plan refused at install. No executor change: the node is bound as an on-first-read view slot.Result cache
Before: a range request refreshed 15 s later recomputed every step. After: each planned query's answer at evaluation time T is kept, keyed by (query root, T), once it is final: the run succeeded (every view window it read is stored, and stored windows never change) and, if the root reads raw data at query time, T ≤ the watermark its snapshot was published with. A range request runs only the missing steps; an instant query at T reads a range step at T. Eviction: T more than the plan's longest retention plus
--view-retention-margin-msbehind the watermark, then LRU beyond--result-cache-bytes(256 MiB). A run now reads the watermark before the snapshot, which also closes that race for shared query-time results. The steps of one range request read one snapshot, and raw chunks read whole are converted once per request.examples/range_bench, dashboard workload, 1k series per metric, four 1 h panels at 15 s steps (964 steps), 56-core x86 server:The rest of a cached refresh is JSON encoding of the two 1,000-series matrices (0.4–0.5 s each). Reading one snapshot per request changes cold CPU by about 3 % (within noise): the cold cost is DDSketch merges, not reads. The Planner's cost model does not know about the cache (follow-up in the design doc).
/metricsCPU breakdownasap_phase_cpu_seconds_total/asap_phase_wall_seconds_total/asap_phase_runs_totalbyphase:remote_write_decode,append,precompute(per refresh group),query(per query text); thread CPU fromCLOCK_THREAD_CPUTIME_ID. Alsoasap_query_latency_seconds(histogram per query),asap_samples_total,asap_windows_total,asap_samples_queued,asap_precompute_backlog_windows,asap_watermark_lag_seconds,asap_raw_data_bytes,asap_view_bytes,asap_query_time_result_*,asap_view_persisted_bytes_total,asap_view_sync_seconds_total,asap_view_windows_not_persisted_total,process_resident_memory_bytes, andasap_thread_cpu_seconds_total{thread}.scripts/profile.shwrites a flamegraph of a running data plane.Ingest throughput, before and after
cargo run --release -p data_plane --example ingest_bench -- SERIES 300 PLAN [REQUESTS] [memory]: in process, 300 s of 1 s scrapes sent by Remote Write as fast as accepted, then a barrier at the last sample. Workloadrandom-mixed-seed-862-01;rawkeeps raw samples (4 views, 25 windows),viewskeeps none (7 views, 325 windows). "Refreshed" is samples/s until every window is refreshed; "acknowledged" until every write is accepted. Views are persisted (the default). One run each on a shared host, so the numbers are indicative."Now" includes jemalloc, the in-place Remote Write decode, the interval sync (5 s) and the per-file text dictionary. Writes are acknowledged at 2.0–2.3M samples/s. View-heavy plans are limited by the precompute thread and, at 10k series, by the disk: the views plan writes 6.5 MB per window (2.1 GB per 5 minutes) and syncing it took about 200 s on this shared, nearly full disk.
Ingest threads (same bench, one client, 16 cores). Acknowledged throughput stays at about 2.2–2.3M samples/s at 10k series and 1.7–1.8M at 100k with 1, 2, 4 or 8 shards: the single bench client, which decodes and sends every request itself, is the limit, not the shards. The table is in the design doc's Measurements section.
Remote Write decode (
decode_bench, µs per sample): one series per sample (10,000 × 1, the feeder's shape) 1.80 (system allocator) → 1.51 (jemalloc) → 0.38 (in-place decode); 100 samples per series (Prometheus' shape) 0.083 → 0.083 → 0.089.Bytes per window written (views plan, 1k series): 0.93 MB with a sync per window and label text per row → 0.64 MB with the per-file text dictionary; raw plan 224 KB → 123 KB; views plan, 10k series, 9.4 MB → 6.5 MB.
View persistence
Every sketch kernel is persisted through the executor's
sketch()andrestore_from_disk(ASAPPlanner #1435): DDSketch, KLL, Count-Min, KMV, HyperLogLog, weighted frequency (Count-Min or Count Sketch, with or without a heap), UnivMon, Hydra-KLL and Hydra-Count-Min (one grid per window, shared by its groups), asasap_sketchlib's ASAPv1 bytes; exact states by their serde form. A UnivMon whose heaps mix key types has no ASAPv1 encoding (asap_sketchlib#265) and is counted inasap_view_windows_not_persisted_total; it is 0 for the benchmark and demo plans. Files are synced every--view-sync-interval-ms(default 5 s, VictoriaMetrics' default-inmemoryDataFlushInterval), so a window is durable within that interval; texts are stored once per file. A file of another plan is renamed*.stale; a torn last record is cut off at load; after a restart, windows whose samples arrived before it stay missing.Review fixes
Shared query-time results: no orphaned entries or leaked bytes (an entry is created computing, under the map lock), waiters get the computer's failure, results are kept only for times the watermark has passed and expire behind it. Range queries: at most one step per query worker in flight, the first failure or a dropped client stops the rest, steps measured apart from instant queries, the 11,000-step limit as Prometheus applies it. A panicking query run answers with an error and the worker keeps serving. Not done: the precompute test gate stays on the production path (one uncontended lock per window); test helpers are not yet consolidated into
tests/common; buildingMaterializedViewvalues through serde JSON and the per-request work linear in retained panes are recorded as follow-ups in the design doc.Code deleted, replaced by the Planner
sds.rs(definitions, keys, build order)materialized_views,ingestion_refresh_groupsprecompute.rspane buildsraw_sample_window.rsDataStore::read_raw_datawith the executor'sScan/TimeRangeper_sample.rs(per-sample state updates)query.rsraw fetches from Prometheus,QueryRunpanesTheir tests are deleted with them (
per_sample,precompute,query,query_run,sds,answers).Tests
planner_executor_parity: 14 public workloads (examples/, plus ASAPPlanner'srandom-mixed-seed-86201–06 and four sketch-plan snapshot workloads copied unchanged intodata_plane/tests/workloads/), each planned with raw data kept and not kept. The data plane runs over HTTP with--no-forward; the Planner executor runs in process on the same plan and samples. Result after the re-pin and with shared query-time results on: 36 plans (18 workloads, raw data kept and not; two raw-only workloads cannot be planned without raw data), 148 of 148 queries equal, all with rows. Plans with a KLL sketch are compared within 5%: KLL seeds its compaction coin at random, so two builds of one window differ within its error (the dashboard plan's quantile query). The design doc's six example workloads are among them. The same harness plans the mixed workloads 01–03 withrange_evaluationand compares each range step with the Planner executor's run at that time: 162 of 162 steps equal.a_query_behind_the_newest_window_is_still_answered.concurrency); two queries sharing aOneEvaluationnode, run concurrently at two times, compute it once per time and answer as runs that compute it themselves, and a plan that marks that nodeNoneis refused (shared_results); a planned range query equals its per-step instant answers and an unplanned one is refused (range_queries); exact, DDSketch and KLL views answer alike after a restart, another plan's files are not loaded, and windows before a restart are not stored empty (view_persistence); each persisted sketch kernel round-trips through bytes, a torn tail is cut off, waiters share the computer's result or failure, range steps are measured apart (unit andrange_queriestests);/metricsreports the phases (server); the columnar read returns the row read's rows, pane eviction, and query-time result expiry (unit tests).fixed_intervalbecomesfixed_interval_at, andvalues_non_negativebecomescolumn_signs. Two expectations changed with the Planner's selection:ignore_accuracy_guarantees, because there is no guaranteed plan without raw data. It is bounded by the plan's own α (0.1).Scans differ only by aregioncolumn); answers are still within α.planner_executor_parityasks every query (instant and range) twice with the cache on and once with it off; all three must agree with the reference. Newresult_cachetests: a repeated range runs no cached step; an instant query reads the range step at its time; a step answered before its raw samples arrived runs again (fails if non-final answers are cached); eviction by age and by LRU bytes, view reads by binary search and shared raw chunk conversion (unit tests).cargo fmt --check,cargo clippy --workspace --all-targets -D warningsandcargo test --workspacepass (control_plane 7, data_plane lib 18, concurrency 1, no_forward 3, planner_executor_answers 10, planner_executor_parity 2, range_queries 3, series_ids 2, server 10, shared_results 2, view_persistence 5).data_plane --no-forward. Four planned queries (rate,increase,max_over_time,sum(sum_over_time)) all run from tumbling-window views refreshed at ingestion. After about 2 h, at three evaluation times, every answer equals Prometheus's own answer exactly (56 + 56 + 1 + 1 series). RSS stayed at 19 MB.Not done yet
AcrossEvaluations).asap_sketchlibfeat(query): count_over_time binds to FrequencyEstimate (CMS / CountSketch) #265).--allowed-lateness-ms.