Repository navigation
Conversation
NewQueryTripperware sends /api/v1/query and /api/v1/query_range through a
roundTripper that rebuilds the outgoing request with the codec and injects the
request metadata into the headers of the request it builds. Every other
endpoint it handles, /api/v1/series, /api/v1/labels,
/api/v1/label/{name}/values, /api/v1/metadata, /api/v1/read,
/api/v1/query_exemplars, /api/v1/format_query and /api/v1/parse_query, is
forwarded with the request as it was received, and nothing injects the metadata
into its headers.
HTTPHeaderMiddleware puts the request ID in the request context only, so for
those endpoints the querier found no request ID header to recover it from, and
its own HTTPHeaderMiddleware generated no replacement because a metadata map
was already in the context. The gRPC calls the querier made to ingesters and
store-gateways then carried an empty request ID, so a request to one of those
endpoints could not be followed across components.
Inject the metadata on the pass-through branch, into a clone of the request so
that the request owned by the server is left unmodified.
InjectMetadataIntoHTTPRequestHeaders now sets each header instead of appending
to it. The receiving side builds its map from the first value of each header,
in frontend_processor.go and scheduler_processor.go, so a value the client sent
for the same header would otherwise shadow the one the query frontend put in
the request context. The metadata map is a map[string]string, so no key it
holds carries more than one value, and the function has no other caller.
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
NewQueryTripperwareroutes/api/v1/queryand/api/v1/query_rangethrough aroundTripperwhoseDorebuilds the outgoing request with the codec and callsrequestmeta.InjectMetadataIntoHTTPRequestHeaders. Every other endpoint it handles(
/api/v1/series,/api/v1/labels,/api/v1/label/{name}/values,/api/v1/metadata,/api/v1/read,/api/v1/query_exemplars,/api/v1/format_query,/api/v1/parse_query)falls through to
next.RoundTrip(r)with the request as it was received, and nothinginjects the metadata.
api.HTTPHeaderMiddleware.injectRequestContextputs the request ID (generated, or takenfrom
-api.request-id-header) into the request context only, never into the headers. Sofor the pass-through endpoints the querier's
ContextWithRequestMetadataMapFromHeadersfinds no
x-cortex-request-id, and the querier's ownHTTPHeaderMiddlewaredoes notgenerate a replacement because a metadata map is already in the context. The gRPC calls
the querier then makes to ingesters and store-gateways carry an empty request ID in
x-request-metadata-propagation-string, so a request to one of these endpoints cannot becorrelated across components. I noticed this on a cluster where a handful of
/api/v1/seriesrequests reached the store-gateways with an empty request ID while every/api/v1/queryand/api/v1/query_rangerequest carried one.The fix injects the metadata on the pass-through branch, into a clone of the request so
the request owned by the HTTP server is not modified (a
RoundTrippershould not mutatethe request it is given, and the inbound
Headermap is shared with it).It also changes
InjectMetadataIntoHTTPRequestHeadersfromHeader.AddtoHeader.Set.frontend_processor.goandscheduler_processor.gobuild their header map fromh.Values[0], so withAdda value the client sent for the same header shadows the onethe query frontend put in the context. The metadata map is a
map[string]string, so nokey it holds is meant to carry more than one value, and
roundTripper.Dowas thefunction's only non-test caller.
One case I deliberately left alone: when both middleware lists are empty
NewQueryTripperwarereturnsnextunchanged, so nothing is injected for any endpoint.That path is unreachable in Cortex's own wiring, because
queryrange.Middlewaresalwaysreturns at least
NewLimitsMiddleware. I can wrapnextthere too if you would ratherhave the guard.
Which issue(s) this PR fixes:
None, found while tracing a request through the read path.
Checklist
CHANGELOG.mdupdateddocs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags — no new flagsThis change was drafted with the help of an AI coding assistant. I have reviewed and
verified every line, the reasoning about the header semantics, and the tests.