Repository navigation
docs(query-engine): design engine boundaries for ASAPPlanner plans - #935
milindsrivastava1997 wants to merge 26 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y decisions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ispatcher Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ngest waits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…isions 34-39 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dPlan decisions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…solve functions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
left a comment
There was a problem hiding this comment.
Overall the interfaces LGTM! Minor changes commented.
I can see when we actually implement these, there will be some missing fields.
| put(SummaryKey, SummaryState); get(plan, summary, range); evict_before(plan, summary, ts); | ||
| earliest_timestamp_per_summary() -> Map<summary, ts>; // was get_earliest_timestamp_per_aggregation_id | ||
| } | ||
| struct SummaryKey { plan: PlanId, summary: PhysicalASAPNodeId, window: (i64, i64) } |
There was a problem hiding this comment.
What does summarykey mean? SummaryId?
There was a problem hiding this comment.
It was the key of one stored window, which was confusing. I split it into SummaryId { plan, node } and SummaryWindowKey { summary, window }.
There was a problem hiding this comment.
SummaryId-> ViewId(plan (plan is a sub-DAG), node (what is node))?
There was a problem hiding this comment.
Reworked; see the view store model (decisions 43–44). It is now three concepts: ViewDefinitionId (the physical computation), ViewInputDataScope (the data that went in), ViewInstance (the state). Happy to do a call.
There was a problem hiding this comment.
Store keys are now (ViewDefinitionId, ViewInputDataScope), not plan and node. The definition id should be a physical view id from ASAPPlanner: ProjectASAP/ASAPPlanner#1429 (decision 44).
|
|
||
| ```rust | ||
| // main: the deployment owns the stores and injects them (decision 8). | ||
| let summaries = Arc::new(InMemorySummaryStore::new()); |
There was a problem hiding this comment.
Can we change the store definition to be "SummaryStore" and "RawDataStore", and make them internally decide whether in memory, or have the on disk part?
There was a problem hiding this comment.
Agreed. The traits are the boundary, and whether data is in memory or on disk is up to the implementation (decision 8). Caveat: a durable store would need summary ids that survive a restart.
There was a problem hiding this comment.
Caveat: a durable store would need summary ids that survive a restart. need explain of this.
There was a problem hiding this comment.
That caveat is gone. A physical ViewDefinitionId is the same across restarts, so a durable store can key recovered instances by it. Recovery itself is deferred (decision 8).
|
|
||
| trait RawDataStore { append(Source, SampleBatch); read(Source, range); evict(retention, now); } | ||
|
|
||
| enum SampleBatch { |
There was a problem hiding this comment.
Can we reuse the term from current ASAPPlanner? If the SampleBatch refers to data sources?
There was a problem hiding this comment.
Renamed to RawDataBatch, after ASAPPlanner's read_raw_data. It holds raw input only (decision 13).
There was a problem hiding this comment.
Do we need a table definition here? I think no.
There was a problem hiding this comment.
Agreed, removed. v1 only has TimeSeries.
…ind traits; generic ingest handler Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
…executor value types Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
… pipeline (I9) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
… (I11), T7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
| put(SummaryWindowKey, SummaryState); get(SummaryId, range); evict_before(SummaryId, ts); | ||
| timestamp_range_per_summary() -> Map<SummaryId, (earliest, latest)>; // was get_earliest_timestamp_per_aggregation_id | ||
| } | ||
| struct SummaryId { plan: PlanId, node: PhysicalASAPNodeId } // one summary |
There was a problem hiding this comment.
SummaryId will be changed to ViewId.
There was a problem hiding this comment.
ViewId, or a view's definition, should be a sub-DAG (this is defined as a node, mostly a node should be ok? or a DAG data structure in ASAPPlanner?)
There was a problem hiding this comment.
Add comment of "plan id here is for retrieving the whole sub-DAG from a specific plan, given "node""
There was a problem hiding this comment.
Renamed to ViewDefinitionId. ASAPPlanner's current definition_id already hashes the view's sub-DAG down to its Scans, but leaves out θ and the window width, so we asked for a physical id: ProjectASAP/ASAPPlanner#1429.
| enum NotAnswerableReason { Unmatched, NotReady, EngineError(QueryEngineError) } | ||
|
|
||
| trait SummaryStore { | ||
| put(SummaryWindowKey, SummaryState); get(SummaryId, range); evict_before(SummaryId, ts); |
There was a problem hiding this comment.
We miss an indexing definition for SummaryStore, what "keys" we use for indexing?
Should we store index by view (SDS, output from one operator), or store by group-by keys (can be output from multiple operators)
There was a problem hiding this comment.
"CapabilityMatched" how can we serve this?
There was a problem hiding this comment.
The store is now opaque to how the state is structured. It is keyed by (definition, input data scope), and indexing is up to the implementation (decisions 8 and 43).
There was a problem hiding this comment.
Not in v1. It needs the ASAPPlanner API: ProjectASAP/ASAPPlanner#1430 (T2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
…stance); wait for ASAPPlanner physical view id Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
…capability matching Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014qhwq4jq8TFcKGchhF7Hfb
|
|
||
| struct ViewDefinitionId { // decision 44 | ||
| definition_id: DefinitionId, // ASAPPlanner's logical id | ||
| physical: PhysicalProperties, // from ASAPPlanner: θ, pane width |
There was a problem hiding this comment.
Please see the comment in ASAPPlanner #1429
There was a problem hiding this comment.
proposed:
ViewDefinitionId = (plan id, definition id)
plan Id = a version of physical DAG plan stored
definition_id: DefinitionId, // ASAPPlanner's logical id refering to root / sub-DAG of a view
We should be clear when defining terms for phyiscal and logical, e.g., regarding views, schema/coverage, plans (physical, logical).
There was a problem hiding this comment.
See #935 (comment): views are keyed by (PlanId, PhysicalASAPNodeId, window).
| constraints: SelectionConstraints, // from the plan's selection, e.g. label matchers | ||
| interval: (i64, i64), // the exact pane (start, end] | ||
| } | ||
| struct ViewInstanceKey { definition: ViewDefinitionId, scope: ViewInputDataScope } |
There was a problem hiding this comment.
The view definition (ASAPPlanner's logical id) contains the input data already, because the e,g, Scan nodes are defined inside already.
There was a problem hiding this comment.
So scope: ViewInputDataScope seems not needed
There was a problem hiding this comment.
See #935 (comment): views are keyed by (PlanId, PhysicalASAPNodeId, window).
| definition_id: DefinitionId, // ASAPPlanner's logical id | ||
| physical: PhysicalProperties, // from ASAPPlanner: θ, pane width | ||
| } | ||
| struct ViewInputDataScope { // the concrete data that went in |
There was a problem hiding this comment.
This is defined as data source related nodes in definition id 's DAG already
There was a problem hiding this comment.
- interval may be needed or inside the constraints?
- Constraints: need to think about how to define the constraints for easy capability-matching-based query serving, not in v1.
There was a problem hiding this comment.
See #935 (comment): views are keyed by (PlanId, PhysicalASAPNodeId, window).
| trait ViewStore { // in-memory or on disk; indexing is the implementation's (decision 8) | ||
| put(ViewInstanceKey, ViewInstance); // once; never rewritten (I16) | ||
| get(&ViewInstanceKey) -> Option<ViewInstance>; | ||
| instances(&ViewDefinitionId, &SelectionConstraints, interval) -> Vec<(ViewInputDataScope, ViewInstance)>; |
There was a problem hiding this comment.
Need a concrete algorithm here on
SelectionConstraintsdefinition- how to match
SelectionConstraintsto the views. timestamp_range_per_view() -> Map<ViewDefinitionId, (earliest, latest)>;this may not be needed
There was a problem hiding this comment.
Question (will raise issue): should a view be bounded batch, or unbounded streaming?
There was a problem hiding this comment.
See #935 (comment): views are keyed by (PlanId, PhysicalASAPNodeId, window).
|
View identity, logical vs physical ids: findings and proposal (replies to the threads on lines 126, 128, 132 and 139, and ProjectASAP/ASAPPlanner#1429) How ASAPPlanner layers identity today (checked against
Lifted selection: walking down from a Findings:
Proposal for the data plane (v1):
Proposed for ASAPPlanner (none blocks v1): reframe #1429 as "a plan-independent id at the logical-candidate level (the sub-DAG with its θ and window widths), for reusing views across plans", and document Also:
I'll update the doc (terms, interface sketch, decisions 38, 43–44, step order) once we agree. |
Design doc for the traits and boundaries ASAPQuery needs in order to run plans from ASAPPlanner (
InstallablePlan) and retire asap-planner-rs. Adapters, fallback clients and ingest decoders stay; the precompute engine andSimpleEnginemove behind newIngestEngine/QueryEnginetraits.Covers interfaces before → after, the new interfaces, recorded decisions, tabled questions (T1–T4) and open questions (Q3). It is limited to boundaries; engine implementations are out of scope.
The doc is the source of truth and will be updated as decisions change.
🤖 Generated with Claude Code