Repository navigation
Conversation
Push resolved the tenant's matching limits_per_label_set entries for every series in every write request, before appending any sample. The result is read only when an append fails with errMaxSeriesPerLabelSetLimitExceeded, so on the success path the slice was allocated and thrown away. LimitsPerLabelSetsForSeries sizes that slice to the tenant's whole label-set list, so each series costs one allocation proportional to the number of configured label sets. Resolve the label sets inside the one failure branch that reads them, and drop the value from handleAppendFailure. Limit enforcement is unchanged: it runs through AssertMaxSeriesPerLabelSet, which resolves the label sets itself. Signed-off-by: Ben Ye <benye@amazon.com>
yeya24
added a commit
to yeya24/cortex
that referenced
this pull request
Nov 20, 2026
Signed-off-by: Ben Ye <benye@amazon.com>
yeya24
force-pushed
the
ingester-lazy-labelset-limits-upstream
branch
from
November 20, 2026 04:59
7387f40 to
a4cb2cb
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
(*Ingester).Pushresolves the tenant's matchinglimits_per_label_setentries for every series in every write request, before appending any sample. The result is only read when an append fails witherrMaxSeriesPerLabelSetLimitExceeded, so when the append succeeds the slice is allocated and thrown away.LimitsPerLabelSetsForSeriessizes that slice to the tenant's whole label-set list, so every series pays one allocation that grows with the number of configured label sets.This PR resolves the label sets inside the failure branch that reads them, and removes the parameter from
handleAppendFailure.Unchanged:
AssertMaxSeriesPerLabelSet, which resolves the label sets itself.formatMaxSeriesPerLabelSetErrorbuilds it from the error value, not from the matched slice.The only difference: the discarded-per-labelset metrics now use the limits in effect when the append fails rather than when the series was looked up. These can differ only if the runtime config reloads during a single push.
Why: in one of our clusters, a tenant with 27 label sets (monitoring only,
max_series: 0) madeLimitsPerLabelSetsForSeriesthe largest ingester allocator, about 65% of all bytes allocated.A benchmark with 27 label sets, pushing 100 existing series per request:
TestIngesterPerLabelsetLimitExceededcovers the moved code. It fails if the lookup in the failure branch returns no label sets.Which issue(s) this PR fixes:
None
Checklist
CHANGELOG.mdupdated (n/a: internal allocation change, no user-facing behaviour change) - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags (n/a)