Repository navigation
Fragment large TsBlocks of query result responses between java session and server - #19263
Wei-hao-Li wants to merge 12 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19263 +/- ##
============================================
+ Coverage 49.22% 49.24% +0.02%
- Complexity 717 718 +1
============================================
Files 5646 5648 +2
Lines 416368 416693 +325
Branches 54966 55011 +45
============================================
+ Hits 204959 205216 +257
- Misses 211409 211477 +68 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JackieTien97
left a comment
There was a problem hiding this comment.
Please address the memory accounting and read-size quota regressions introduced by fragmentation, and apply the payload limit to the Fast LAST cache-hit paths. The inline comments describe the affected paths.
…d quota by whole blocks - SharedTsBlockQueue.remove(retainMemory) keeps the reservation of a block that outlives the RPC in PendingResultFragment; freeRetainedMemory() releases it and close/abort free the buffered and handed-over memory together. - LocalSourceHandle delegates the retain/release to its queue, so single-node deployments keep fragmented transfers accounted in the memory pool. - OperationQuota only exposes addReadResult(long bytes); responses charge a block whose fragmentation starts in them by its totalLength, so a fragmented result is charged exactly once against the read quota.
…result The cache-hit branches of executeFastLastDataQueryForOneDeviceV2 and executeFastLastDataQueryForOnePrefixPath serialized the whole result into one response, so an oversized cached value could still exceed the Thrift frame limit. The cached result is now registered as a MaterializedResultExecution and sent through the regular fragmented transfer: fetchResultsV2 continues from its pending fragment, and the serialized bytes are reserved from the query memory pool (best effort) until the last fragment is sent or the query is cleaned up.
Conflicts in i18n message classes are additive; kept the constants from both sides (fragmented query result messages and the incoming buffer/frame-size messages).
respondWithFragmentedResult is only entered when the cached result exceeds the payload bound, so the materialized transfer always starts with a fragment; replace the defensive finish branch with checkState.
Conflicts in the JDBC result-set construction: keep the upstream resource cleanup (close the query operation when result-set creation fails) together with the offset/totalLength arguments added for fragmented results.
JackieTien97
left a comment
There was a problem hiding this comment.
I re-reviewed head 3a83908 and also ran it on a real cluster (1 ConfigNode, 2 DataNodes, mpp_data_exchange_max_payload_size_in_bytes=131072). Tree queries through Session and JDBC, table queries through JDBC, raw / last / aggregation queries, both Fast LAST RPCs, and both the local and the remote result path returned the expected rows. A session that does not announce chunking still receives whole blocks.
Previous round:
- Read-size quota: addressed. A block whose fragmentation starts in an execute response is charged once by its
totalLength(checked by reading the code, not with a throttling test). - Fast LAST cache hits: addressed. Both RPCs fragment an oversized cached result, and the reservation is released after the last fragment. One side effect is noted inline.
- Pending fragments on the local result path: partly addressed. Blocks other than the last one stay reserved now, the last block does not. See the
[P1]comment.
Inline comments are tagged by priority. [P1] and the [P2] items are the ones I would like to see addressed before merging.
One more item that has no line to attach to: the PR description is empty. This PR changes the client RPC protocol, so please describe it with a concrete example: the new fields, how a client opts in, what a fragmented transfer looks like, and which setting bounds the payload.
| bufferRetainedSizeInBytes); | ||
| bufferRetainedSizeInBytes = 0; | ||
| } | ||
| freeAllRetainedMemory(); |
There was a problem hiding this comment.
[P1] The reservation of the last block is still dropped before the client has fetched it
This follows up on the earlier comment about the local result path. Blocks that are not the last one now stay reserved until freeRetainedMemory(). The last block of a result does not, and "the whole result is one oversized block" is exactly that case.
remove(true) moves the bytes to pendingRetainedSizeInBytes and then calls sinkChannel.checkAndInvokeOnFinished(). Once the queue is empty and the producer has called setNoMoreTsBlocks(), that runs ISinkChannelListenerImpl.onFinish -> ShuffleSinkHandle.close() -> LocalSinkChannel.close() -> queue.close(), and this line frees the pending bytes together with the buffered ones. If the producer finishes later, the same happens at that moment. Either way the reservation is gone while PendingResultFragment still holds the whole serialized block.
The free cannot simply be skipped here. The queue's reservation is registered under the upstream fragment instance, and that account is force-deregistered when the instance finishes, so bytes still reserved at that point would be reported as a MemoryLeakException and never returned to the pool.
Measured on a single DataNode with the payload bound at 128 KiB and a query that returns one row with a 3 MB TEXT value: after the execute response the DataNode still held 2,868,953 unsent bytes in the pending fragment, and the query memory pool showed 0 reserved bytes. The same query read through a remote result handle (SourceHandle) kept its reservation until the last fragment was sent.
The two new tests in LocalSourceHandleTest build the queue without a sink channel, so they do not reach this path. To reproduce at unit level: create the sink with MPPDataExchangeManager.createShuffleSinkHandle and the result handle with createLocalSourceHandleForFragment (as MPPDataExchangeManagerTest#testCreateLocalSinkHandle does) on a real MemoryPool, send one block, call setNoMoreTsBlocks(), then getSerializedTsBlock(true). MemoryPool#getReservedBytes() is 0 right after that call.
Suggestion: account for a pending fragment on the coordinator side instead of in the upstream queue. MaterializedResultExecution.reserveResult() already does this for the Fast LAST case: reserve the serialized length under an account owned by the query when a block becomes pending, and free it in PendingResultFragment.clear(). That behaves the same for local, remote and in-memory result handles, and retainMemory / freeRetainedMemory could be removed again. It would also reserve what is actually held: in the remote run above the retained amount was 375,067 bytes for a 3,000,025-byte pending block, because the exchange layer reserves the TsBlock size estimate.
| public String getQueryId() { | ||
| return Long.toString(queryId); |
There was a problem hiding this comment.
[P2] SHOW QUERIES reports a wrong DataNodeId while a fragmented Fast LAST transfer is pending
getQueryId() returns the numeric client query id here, while every other execution returns a QueryId string such as 20261119_050035_03054_1. ShowQueriesOperator#buildTsBlock derives the DataNodeId for all rows from the first execution in the list (queryExecutions.get(0).getQueryId().split("_"), last element parsed as an int). When this execution is first, every row gets the query id as its DataNodeId.
Observed on a node whose DataNodeId is 1, between the first response of executeFastLastDataQueryForOneDeviceV2 and the last fetch:
QueryId | DataNodeId | Statement
3058 | 3058 | UNKNOWN
20261119_050035_03054_1 | 3058 | show queries
The id comes from a process-wide counter that write requests advance as well, so on a long-running node it can exceed Integer.MAX_VALUE. Integer.parseInt then throws and SHOW QUERIES fails.
Please give this execution a real query id (Coordinator#createQueryId()), which can also serve as the memory pool key, and consider returning a meaningful getExecuteSQL() so that the row is not shown as UNKNOWN.
| QueryDataSetUtils.ResultChunk chunk = | ||
| fetchResultChunk( | ||
| queryExecution, | ||
| req.fetchSize, | ||
| req.isSetOffset() ? req.getOffset() : 0, | ||
| req.isSetOffset() ? config.getMppDataExchangeMaxPayloadSizeInBytes() : 0); |
There was a problem hiding this comment.
[P3] A continuation request without offset silently drops the rest of a pending block
The execute paths decide by the session flag (getMaxResultPayloadSize(clientSession)), but this path decides by req.isSetOffset(). If a fragment is pending and the fetch request carries no offset, fetchResultChunk takes the legacy branch, which never looks at the pending fragment. The response is a plain success and the pending bytes are skipped.
Observed with a raw Thrift client on a session that enabled chunking: the execute response carried the first 131,072 bytes of a 3,000,025-byte block with moreData=true. A following fetchResultsV2 without offset returned status=200, hasResultSet=false, moreData=false.
The Java clients in this PR always set offset, so this needs an inconsistent client. It is still worth failing loudly, because other language clients will implement this protocol later. Either use the session flag here as well, or reject a request without offset while a fragment is pending.
| boolean finished = | ||
| SELECT_RESULT.apply(resp, execution, fetchSize, getMaxResultPayloadSize(clientSession)); | ||
| // the cached result exceeds the payload bound, so the first response is always a fragment | ||
| checkState( | ||
| !finished, | ||
| DataNodeMiscMessages | ||
| .EXCEPTION_THE_FIRST_RESPONSE_OF_A_FRAGMENTED_FAST_LAST_RESULT_MUST_NOT_BE_COMPLETE_DEF121D5); |
There was a problem hiding this comment.
[P3] The payload bound is read twice, and the assertion can fail after a hot reload
canFitInOneResponse reads mpp_data_exchange_max_payload_size_in_bytes to decide that the cached result is too large, and this call reads it again. The setting is hot-reloadable. If it grows in between, the whole block fits, finished is true, and checkState fails a query that could simply have been answered.
In executeFastLastDataQueryForOnePrefixPath the catch block then returns the error without cleaning up the execution registered above, so it stays in the Coordinator until the statement or the session is closed. executeFastLastDataQueryForOneDeviceV2 does clean up in its finally.
This comes from reading the code, I did not reproduce the race. Reading the bound once and passing it down, and handling finished like the other paths (resp.setMoreData(!finished) plus cleanup), would remove both problems.
| res.add(slice(byteBuffer, blockStart, remainingPayloadSize)); | ||
| pendingFragment.reset(byteBuffer, blockStart + remainingPayloadSize); | ||
| return ResultChunk.fragment( | ||
| res, blockStart + remainingPayloadSize, blockStart == 0 ? blockSize : -1, false); |
There was a problem hiding this comment.
[P3] Offsets are computed from the buffer position, not from the start of the block
blockStart + remainingPayloadSize is what the client receives as offset, and totalLength is only sent when blockStart == 0. The client expects offset to equal the number of bytes it has accumulated and requires totalLength on the first fragment, so both are only correct for a buffer whose position is 0.
That holds for every result handle today: the local and in-memory handles serialize into a fresh buffer, and the MPP exchange client uses the deep-copy transport. A buffer with a non-zero position would break the transfer with Missing totalLength or an offset mismatch. Slicing the block once before use (byteBuffer = byteBuffer.slice()) makes all offsets block-relative regardless of where the buffer comes from.
| if (nextOffset != buffer.size()) { | ||
| throw new IllegalArgumentException( | ||
| String.format( | ||
| RpcMessages | ||
| .EXCEPTION_CHUNKED_RESULT_FRAGMENT_OFFSET_ARG_NOT_GREATER_THAN_ACCUMULATED_LENGTH_ARG_A0F032E3, | ||
| nextOffset, | ||
| buffer.size())); |
There was a problem hiding this comment.
[nit] The message does not match the check
The check rejects any nextOffset that differs from the accumulated length, but the message says the offset "is not greater than" it, which is misleading when the offset is larger. Something like Fragment offset %d does not match the accumulated length %d covers both cases. The constant name and the zh text need to follow.
| public static final String EXCEPTION_DATA_TYPE_ARG_IS_NOT_SUPPORTED_31213160 = | ||
| "Data type %s is not supported."; | ||
|
|
||
| public static final String EXCEPTION_CHUNKED_RESULT_EMPTY_FRAGMENT_03A0EF39 = |
There was a problem hiding this comment.
[nit] Constant names do not follow the i18n naming rule
AGENTS.md derives the name from the full English text (<PREFIX>_<NORMALIZED>_<HASH>). The hashes of the new constants are right, but six of the seven names abbreviate the text. This one would be EXCEPTION_EMPTY_FRAGMENT_RECEIVED_WHEN_ASSEMBLING_A_CHUNKED_QUERY_RESULT_03A0EF39. The same applies to the other three here and to the two EXCEPTION_RESULT_FRAGMENT_... constants in DataNodeQueryMessages.
| * bounded-payload chunks via {@link QueryDataSetUtils} and the client side ({@code | ||
| * IoTDBRpcDataSet}) assembles them back into the original rows. | ||
| */ | ||
| public class ResultChunkingEndToEndTest { |
There was a problem hiding this comment.
[P2] Please add integration tests that go through a real DataNode
This test and IoTDBRpcDataSetFragmentTest exercise the two ends against mocks, so the RPC entry points are not covered. Codecov reports 13.5% patch coverage for ClientRPCServiceImpl.
An IT with mpp_data_exchange_max_payload_size_in_bytes at its minimum (131072) keeps the data small. Useful cases:
- a row larger than the bound, through Session and JDBC, in tree and table model
- several oversized rows mixed with normal ones and a small
fetchSize - both Fast LAST RPCs on a cached oversized value
- closing the result set while a fragment is pending
- a multi-DataNode run where the coordinator reads the result through a remote handle
I ran these by hand against this head (all except the table model through a table Session) and they pass, so this is about keeping them from regressing.
| // Do not fetch new blocks in the same response, otherwise the response would carry the | ||
| // tail of the big block plus up to fetchSize new rows, whose row count is not accounted | ||
| // for. One response either continues a fragmented block or delivers new blocks. | ||
| return ResultChunk.complete(res, !queryExecution.hasNextResult()); |
There was a problem hiding this comment.
[P3, optional] Ordinary blocks are split at the payload boundary, and the tail travels alone
A block that does not fit into the remaining payload is fragmented even when it is smaller than the bound, and because of this early return the next response carries only its tail. For a result made of ordinary blocks this roughly doubles the number of round trips, and the client copies every split block once more to reassemble it.
Measured with the bound at 128 KiB and fetchSize=200000 on 200,000 INT64 rows (52 blocks of at most 65,549 bytes, 3.2 MB in total): a session without chunking needs 1 RPC, a chunking session needs 35, and 17 of them carry nothing but a tail.
With the default 4 MiB bound this only shows up when one response would exceed 4 MiB (large fetch sizes or wide rows), so it is not a blocker. A possible refinement: when a block is not larger than the bound and the response already has data, carry the whole block over to the next response instead of cutting it.
- Account a pending fragment under a query-owned memory account instead of the result handles: SharedTsBlockQueue.close() could release the last block's reservation (sink finish -> LocalSinkChannel.close -> queue.close) while the fragment still held it, and the queue's account is force-deregistered with the upstream fragment instance. The handle-level retainMemory/freeRetainedMemory API is removed again; PendingResultFragment reserves the serialized length on its first reset and releases it on clear. - Give the Fast LAST materialized execution a real QueryId from Coordinator#createQueryId(), so SHOW QUERIES parses the DataNodeId correctly, and return a readable SQL from getExecuteSQL(). - Decide fragmentation in fetchResultsV2 by the session capability, so a continuation without offset fails the offset check instead of dropping the tail; read the payload bound once and pass it down; slice the block before computing offsets so they are block-relative; clean up the registered execution when the Fast LAST transfer fails. - Rename the new i18n constants after their full English text and fix the offset mismatch message. - Add integration tests: IoTDBSessionResultChunkingIT (tree model, standalone and cluster) and IoTDBTableResultChunkingIT (table model).
|



No description provided.