Skip to content

Ingester: resolve per-labelset limits only on append failure - #7955

Open
yeya24 wants to merge 1 commit into
cortexproject:masterfrom
yeya24:ingester-lazy-labelset-limits-upstream
Open

yeya24 wants to merge 1 commit into
cortexproject:masterfrom
yeya24:ingester-lazy-labelset-limits-upstream

Conversation

@yeya24

@yeya24 yeya24 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does:

(*Ingester).Push resolves the tenant's matching limits_per_label_set entries for every series in every write request, before appending any sample. The result is only read when an append fails with errMaxSeriesPerLabelSetLimitExceeded, so when the append succeeds the slice is allocated and thrown away. LimitsPerLabelSetsForSeries sizes that slice to the tenant's whole label-set list, so every series pays one allocation that grows with the number of configured label sets.

This PR resolves the label sets inside the failure branch that reads them, and removes the parameter from handleAppendFailure.

Unchanged:

  • Limit enforcement. It runs through AssertMaxSeriesPerLabelSet, which resolves the label sets itself.
  • The returned error. formatMaxSeriesPerLabelSetError builds it from the error value, not from the matched slice.

The only difference: the discarded-per-labelset metrics now use the limits in effect when the append fails rather than when the series was looked up. These can differ only if the runtime config reloads during a single push.

Why: in one of our clusters, a tenant with 27 label sets (monitoring only, max_series: 0) made LimitsPerLabelSetsForSeries the largest ingester allocator, about 65% of all bytes allocated.

A benchmark with 27 label sets, pushing 100 existing series per request:

B/op allocs/op ns/op
before 159,295 113 ~186k
after 2,360 12 ~78k

TestIngesterPerLabelsetLimitExceeded covers the moved code. It fails if the lookup in the failure branch returns no label sets.

Which issue(s) this PR fixes:
None

Checklist

  • Tests updated (existing tests cover the moved code)
  • Documentation added (n/a)
  • CHANGELOG.md updated (n/a: internal allocation change, no user-facing behaviour change) - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags (n/a)

Push resolved the tenant's matching limits_per_label_set entries for
every series in every write request, before appending any sample. The
result is read only when an append fails with
errMaxSeriesPerLabelSetLimitExceeded, so on the success path the slice
was allocated and thrown away. LimitsPerLabelSetsForSeries sizes that
slice to the tenant's whole label-set list, so each series costs one
allocation proportional to the number of configured label sets.

Resolve the label sets inside the one failure branch that reads them,
and drop the value from handleAppendFailure. Limit enforcement is
unchanged: it runs through AssertMaxSeriesPerLabelSet, which resolves
the label sets itself.

Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24
yeya24 requested a review from a team as a code owner November 20, 2026 04:54
@yeya24
yeya24 requested a review from danielblando November 20, 2026 04:54
yeya24 added a commit to yeya24/cortex that referenced this pull request Nov 20, 2026
Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24
yeya24 force-pushed the ingester-lazy-labelset-limits-upstream branch from 7387f40 to a4cb2cb Compare November 20, 2026 04:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant