Repository navigation
Add projectionHint to SG Parquet mode - #7206
SungJin1212 wants to merge 11 commits into
Conversation
| f.Float64Var(&cfg.TokenBucketBytesLimiter.TouchedChunksTokenFactor, "blocks-storage.bucket-store.token-bucket-bytes-limiter.touched-chunks-token-factor", 1, "Multiplication factor used for touched chunks token") | ||
| f.IntVar(&cfg.MatchersCacheMaxItems, "blocks-storage.bucket-store.matchers-cache-max-items", 0, "Maximum number of entries in the regex matchers cache. 0 to disable.") | ||
| cfg.ParquetShardCache.RegisterFlagsWithPrefix("blocks-storage.bucket-store.", f) | ||
| f.BoolVar(&cfg.HonorProjectionHints, "blocks-storage.bucket-store.honor-projection-hints", false, "[Experimental] If enabled, Store Gateway will honor projection hints and only materialize requested labels. It is only effect when `-blocks-storage.bucket-store.bucket-store-type` is a parquet.") |
There was a problem hiding this comment.
Just to double check, if I want to actually use parquet store gateway projection, I need to also honor projection hints in Querier (but not enable parquet querier)?
There was a problem hiding this comment.
yeah, even if the Parquet Querier is disabled (using the standard Querier), the Parquet Store Gateway relies on these hints to perform projection pushdown (Querier send the projection hints).
There was a problem hiding this comment.
Seems that even if we don't enable honor projection hints in querier, it still propagates projection hints to store gateway. I don't see you enable that in the integration test.
| storageHints.ProjectionLabels = queryHints.ProjectionLabels | ||
|
|
||
| // Reset projection hints if not all parquet shard have the hash column. | ||
| if !allParquetBlocksHaveHashColumn(shards) { |
There was a problem hiding this comment.
We should only project if it is projection include. Add that check first can avoid checking all blocks of hash column when unnecessary
| vec := aggResult.(model.Vector) | ||
| require.Len(t, vec, 1) | ||
| require.Equal(t, model.LabelValue("series_1"), vec[0].Metric["series_1"]) | ||
| require.Equal(t, expectedVector1[0].Value, vec[0].Value) |
There was a problem hiding this comment.
I want to avoid adding more test cases to this test.
I think overall the test is very flaky and it keeps failing for different reasons. I would prefer separate tests for parquet store gateway mode tbh.
| storageHints.ProjectionInclude = false | ||
| storageHints.ProjectionLabels = nil | ||
| } | ||
| } |
There was a problem hiding this comment.
I would also reset the projection hints if it is not projection include. Same as what we do in querier.
Line 455 in 28ad1ac
If we don't do this then it can do wrong below as we only add series hash if it is projection include. In general it is hard to estimate the cost of projection not include and it can be more expensive
903d00e to
d9638cc
Compare
|
@yeya24 |
6e47284 to
1c9539b
Compare
9f5b844 to
d860aff
Compare
8b10151 to
f033993
Compare
b20e2e2 to
a2b1d29
Compare
b45fb3f to
30437f2
Compare
6792a7b to
bac13d7
Compare
sandy2008
left a comment
There was a problem hiding this comment.
Thanks. The earlier review points look addressed. I checked that the new TestQuerier_ProjectionHints* and TestParquetBucketStore_* cases fail without the querier and store-gateway changes, and that the generated config reference matches. There's one problem with how the V1 fallback works across multiple store-gateways (inline). The test (arm64) failure is an unrelated port collision in pkg/util/tls/test.
| lbls := make([]string, 0, len(queryHints.ProjectionLabels)+1) | ||
| storageHints.ProjectionLabels = append(lbls, queryHints.ProjectionLabels...) | ||
|
|
||
| if !allParquetBlocksHaveHashColumn(shards) { |
There was a problem hiding this comment.
This fallback only sees the blocks this gateway was asked for. The querier sends the same projection hints to every store-gateway, so if one gateway holds a V1 block and another holds only V2 blocks, the first returns full label sets and the second returns the projected labels plus s_series_hash.
blocksStoreQuerier merges the responses with storage.NewMergeSeriesSet(…, storage.ChainedSeriesMerge), which only merges identical label sets. The same series therefore comes back twice, and overlapping samples (e.g. a source block and its compacted block both still in the bucket) are no longer deduplicated. I reproduced the split with buildSelectHints plus that merge: V1-only shards → no projection, V2-only → [job s_series_hash], 2 series after the merge.
The direct parquet queryable avoids this by deciding once for all selected blocks before fan-out (allParquetBlocksHaveHashColumn over the bucket-index blocks in parquet_queryable.go). queryWithConsistencyCheck already has those blocks (knownBlocks, with Parquet.Version), so the hints could be reset there when any selected block is V1.
There was a problem hiding this comment.
It makes sense, I applied it to the latest commit. Thanks.
2b82175 to
6ec078d
Compare
sandy2008
left a comment
There was a problem hiding this comment.
Thanks, re-checked at 6ec078d. The querier now decides once per query from the converter marks of all selected blocks and resets the hints for every store-gateway, so the mixed V1/V2 duplicates are gone. The new mixed-version Select case and TestStoreGatewaySelectHints both fail with that reset removed, and TestStoreGatewaySelectHints also catches the hints being modified in place instead of copied. Two suggestions inline: the rollout order in the flag docs, and a fixture tweak that removes a small flake in the new integration test.
| f.IntVar(&cfg.MatchersCacheMaxItems, "blocks-storage.bucket-store.matchers-cache-max-items", 0, "Maximum number of entries in the regex matchers cache. 0 to disable.") | ||
| f.IntVar(&cfg.ParquetQueryConcurrency, "blocks-storage.bucket-store.parquet-query-concurrency", 4, "Maximum number of concurrent goroutines per query applied at each level of parquet processing: shard querying, row group processing, and column materialization. Note: this limit is applied independently at each level, so the total goroutines per query can grow multiplicatively (up to N^3 in the worst case).") | ||
| cfg.ParquetShardCache.RegisterFlagsWithPrefix("blocks-storage.bucket-store.", f) | ||
| f.BoolVar(&cfg.HonorProjectionHints, "blocks-storage.bucket-store.honor-projection-hints", false, "[Experimental] If enabled, Store Gateway will honor projection hints and only materialize requested labels. It only takes effect when `-blocks-storage.bucket-store.bucket-store-type` is parquet, `-querier.honor-projection-hints` is enabled and the querier uses the Thanos engine (`-querier.thanos-engine`) with the `projection` optimizer (`-querier.optimizers`).") |
There was a problem hiding this comment.
This is the same mixed-response problem as before, only caused by config skew this time. During the rolling restart that turns this on, queriers already running with -querier.honor-projection-hints=true send the hints to every store-gateway. The restarted ones return {job="a", s_series_hash="..."}, the others return the full label set, and the merge keeps both. One projectingStoreGatewayClientMock plus one plain mock over two V2 blocks gives 2 series. Queriers older than this PR forward the hints even with that flag off, so they need upgrading first. Could the description spell out the order? It could also mention the bucket index, since the bucket-scan finder never sets Block.Parquet, so without it projection never applies. Something like this, plus make doc:
| f.BoolVar(&cfg.HonorProjectionHints, "blocks-storage.bucket-store.honor-projection-hints", false, "[Experimental] If enabled, Store Gateway will honor projection hints and only materialize requested labels. It only takes effect when `-blocks-storage.bucket-store.bucket-store-type` is parquet, `-querier.honor-projection-hints` is enabled and the querier uses the Thanos engine (`-querier.thanos-engine`) with the `projection` optimizer (`-querier.optimizers`).") | |
| f.BoolVar(&cfg.HonorProjectionHints, "blocks-storage.bucket-store.honor-projection-hints", false, "[Experimental] If enabled, Store Gateway will honor projection hints and only materialize requested labels. It only takes effect when `-blocks-storage.bucket-store.bucket-store-type` is parquet, `-querier.honor-projection-hints` is enabled, the querier uses the Thanos engine (`-querier.thanos-engine`) with the `projection` optimizer (`-querier.optimizers`), and every queried block has a parquet converter mark of version 2 or later in the bucket index. Enable it on all store-gateways before enabling `-querier.honor-projection-hints`, and disable it in the reverse order, otherwise the same series can come back from different store-gateways with different label sets and be counted twice.") |
The -querier.honor-projection-hints help (querier.go:157, outside the diff) still says projection only works with the Parquet Queryable, so it could be updated in the same pass.
| // Time range: [Now - 24h] to [Now - 20h] | ||
| start := now.Add(-24 * time.Hour) | ||
| end := now.Add(-20 * time.Hour) |
There was a problem hiding this comment.
The seed comes from time.Now().Unix(), and each series gets 100 one-minute samples from a random start inside this 4h range. Sometimes no series has a sample in the 5m before end, and then the NotEmpty check fails whatever the projection does. I replayed CreateBlock's placement over 200k consecutive seeds: it happens for about 0.35% of them (0.4% for the rate query), so roughly one run in 250. Seed 140 is one, with its last sample about 55 minutes before end. Keeping the block no longer than the samples removes it (0 of 200k), and the 100-minute block is still converted with -compactor.block-ranges=1ms,12h:
| // Time range: [Now - 24h] to [Now - 20h] | |
| start := now.Add(-24 * time.Hour) | |
| end := now.Add(-20 * time.Hour) | |
| // Each series gets 100 samples one minute apart from a random start, so keep the block no | |
| // longer than that, or the instant queries at end can find no sample within the lookback. | |
| start := now.Add(-24 * time.Hour) | |
| end := start.Add(100 * time.Minute) |
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
…flake - Describe in the store-gateway and querier flag help that projection needs the bucket index (converter mark version 2 or later), and that store-gateways must enable it before queriers and disable it after them, so the same series is not returned with different label sets. - Update the querier flag help, which still said projection only works with the Parquet Queryable. - Keep the integration test block no longer than its 100 one-minute samples so the instant queries at end always find a sample within the lookback. Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
6ec078d to
00eefee
Compare
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
|
@sandy2008 |
Replace the converter mark version and bucket index internals in the flag help with the bucket index flag and the Cortex version (v1.21.0) whose parquet converter writes the series hash column, and note that blocks converted earlier are not converted again. Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
Keep the conditions in the querier flag help and refer to it from the store-gateway flag, mention that hints come from the Thanos engine projection optimizer, and require all store-gateways to stay in parquet mode while the querier flag is enabled. Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
This PR adds
-blocks-storage.bucket-store.honor-projection-hintsCLI flag. If enabled, Store Gateway in Parquet mode will honor projection hints and only materialize requested labels.The benchmark results
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]