Skip to content

Fragment large TsBlocks of query result responses between java session and server - #19263

Open
Wei-hao-Li wants to merge 12 commits into
masterfrom
fragmentCSRPC
Open

Wei-hao-Li wants to merge 12 commits into
masterfrom
fragmentCSRPC

Conversation

@Wei-hao-Li

Copy link
Copy Markdown
Collaborator

No description provided.

Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.19222% with 139 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.24%. Comparing base (720da6c) to head (3a83908).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
.../db/protocol/thrift/impl/ClientRPCServiceImpl.java 13.54% 83 Missing ⚠️
...ne/plan/execution/MaterializedResultExecution.java 59.75% 33 Missing ⚠️
...ain/java/org/apache/iotdb/rpc/IoTDBRpcDataSet.java 91.52% 5 Missing ⚠️
...ache/iotdb/db/protocol/session/IClientSession.java 25.00% 3 Missing ⚠️
...java/org/apache/iotdb/jdbc/IoTDBJDBCResultSet.java 60.00% 2 Missing ⚠️
...va/org/apache/iotdb/session/SessionConnection.java 84.61% 2 Missing ⚠️
...yengine/execution/exchange/SharedTsBlockQueue.java 92.00% 2 Missing ⚠️
.../apache/iotdb/db/queryengine/plan/Coordinator.java 0.00% 2 Missing ⚠️
.../db/queryengine/plan/execution/QueryExecution.java 66.66% 2 Missing ⚠️
...java/org/apache/iotdb/isession/SessionDataSet.java 83.33% 1 Missing ⚠️
... and 4 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19263      +/-   ##
============================================
+ Coverage     49.22%   49.24%   +0.02%     
- Complexity      717      718       +1     
============================================
  Files          5646     5648       +2     
  Lines        416368   416693     +325     
  Branches      54966    55011      +45     
============================================
+ Hits         204959   205216     +257     
- Misses       211409   211477      +68     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the memory accounting and read-size quota regressions introduced by fragmentation, and apply the payload limit to the Fast LAST cache-hit paths. The inline comments describe the affected paths.

…d quota by whole blocks

- SharedTsBlockQueue.remove(retainMemory) keeps the reservation of a block that
  outlives the RPC in PendingResultFragment; freeRetainedMemory() releases it and
  close/abort free the buffered and handed-over memory together.
- LocalSourceHandle delegates the retain/release to its queue, so single-node
  deployments keep fragmented transfers accounted in the memory pool.
- OperationQuota only exposes addReadResult(long bytes); responses charge a block
  whose fragmentation starts in them by its totalLength, so a fragmented result
  is charged exactly once against the read quota.
…result

The cache-hit branches of executeFastLastDataQueryForOneDeviceV2 and
executeFastLastDataQueryForOnePrefixPath serialized the whole result into one
response, so an oversized cached value could still exceed the Thrift frame
limit. The cached result is now registered as a MaterializedResultExecution and
sent through the regular fragmented transfer: fetchResultsV2 continues from its
pending fragment, and the serialized bytes are reserved from the query memory
pool (best effort) until the last fragment is sent or the query is cleaned up.
Conflicts in i18n message classes are additive; kept the constants from both
sides (fragmented query result messages and the incoming buffer/frame-size
messages).
respondWithFragmentedResult is only entered when the cached result exceeds the
payload bound, so the materialized transfer always starts with a fragment;
replace the defensive finish branch with checkState.
Conflicts in the JDBC result-set construction: keep the upstream resource
cleanup (close the query operation when result-set creation fails) together
with the offset/totalLength arguments added for fragmented results.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed head 3a83908 and also ran it on a real cluster (1 ConfigNode, 2 DataNodes, mpp_data_exchange_max_payload_size_in_bytes=131072). Tree queries through Session and JDBC, table queries through JDBC, raw / last / aggregation queries, both Fast LAST RPCs, and both the local and the remote result path returned the expected rows. A session that does not announce chunking still receives whole blocks.

Previous round:

  • Read-size quota: addressed. A block whose fragmentation starts in an execute response is charged once by its totalLength (checked by reading the code, not with a throttling test).
  • Fast LAST cache hits: addressed. Both RPCs fragment an oversized cached result, and the reservation is released after the last fragment. One side effect is noted inline.
  • Pending fragments on the local result path: partly addressed. Blocks other than the last one stay reserved now, the last block does not. See the [P1] comment.

Inline comments are tagged by priority. [P1] and the [P2] items are the ones I would like to see addressed before merging.

One more item that has no line to attach to: the PR description is empty. This PR changes the client RPC protocol, so please describe it with a concrete example: the new fields, how a client opts in, what a fragmented transfer looks like, and which setting bounds the payload.

bufferRetainedSizeInBytes);
bufferRetainedSizeInBytes = 0;
}
freeAllRetainedMemory();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] The reservation of the last block is still dropped before the client has fetched it

This follows up on the earlier comment about the local result path. Blocks that are not the last one now stay reserved until freeRetainedMemory(). The last block of a result does not, and "the whole result is one oversized block" is exactly that case.

remove(true) moves the bytes to pendingRetainedSizeInBytes and then calls sinkChannel.checkAndInvokeOnFinished(). Once the queue is empty and the producer has called setNoMoreTsBlocks(), that runs ISinkChannelListenerImpl.onFinish -> ShuffleSinkHandle.close() -> LocalSinkChannel.close() -> queue.close(), and this line frees the pending bytes together with the buffered ones. If the producer finishes later, the same happens at that moment. Either way the reservation is gone while PendingResultFragment still holds the whole serialized block.

The free cannot simply be skipped here. The queue's reservation is registered under the upstream fragment instance, and that account is force-deregistered when the instance finishes, so bytes still reserved at that point would be reported as a MemoryLeakException and never returned to the pool.

Measured on a single DataNode with the payload bound at 128 KiB and a query that returns one row with a 3 MB TEXT value: after the execute response the DataNode still held 2,868,953 unsent bytes in the pending fragment, and the query memory pool showed 0 reserved bytes. The same query read through a remote result handle (SourceHandle) kept its reservation until the last fragment was sent.

The two new tests in LocalSourceHandleTest build the queue without a sink channel, so they do not reach this path. To reproduce at unit level: create the sink with MPPDataExchangeManager.createShuffleSinkHandle and the result handle with createLocalSourceHandleForFragment (as MPPDataExchangeManagerTest#testCreateLocalSinkHandle does) on a real MemoryPool, send one block, call setNoMoreTsBlocks(), then getSerializedTsBlock(true). MemoryPool#getReservedBytes() is 0 right after that call.

Suggestion: account for a pending fragment on the coordinator side instead of in the upstream queue. MaterializedResultExecution.reserveResult() already does this for the Fast LAST case: reserve the serialized length under an account owned by the query when a block becomes pending, and free it in PendingResultFragment.clear(). That behaves the same for local, remote and in-memory result handles, and retainMemory / freeRetainedMemory could be removed again. It would also reserve what is actually held: in the remote run above the retained amount was 375,067 bytes for a 3,000,025-byte pending block, because the exchange layer reserves the TsBlock size estimate.

Comment on lines +232 to +233
public String getQueryId() {
return Long.toString(queryId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] SHOW QUERIES reports a wrong DataNodeId while a fragmented Fast LAST transfer is pending

getQueryId() returns the numeric client query id here, while every other execution returns a QueryId string such as 20261119_050035_03054_1. ShowQueriesOperator#buildTsBlock derives the DataNodeId for all rows from the first execution in the list (queryExecutions.get(0).getQueryId().split("_"), last element parsed as an int). When this execution is first, every row gets the query id as its DataNodeId.

Observed on a node whose DataNodeId is 1, between the first response of executeFastLastDataQueryForOneDeviceV2 and the last fetch:

QueryId                 | DataNodeId | Statement
3058                    | 3058       | UNKNOWN
20261119_050035_03054_1 | 3058       | show queries

The id comes from a process-wide counter that write requests advance as well, so on a long-running node it can exceed Integer.MAX_VALUE. Integer.parseInt then throws and SHOW QUERIES fails.

Please give this execution a real query id (Coordinator#createQueryId()), which can also serve as the memory pool key, and consider returning a meaningful getExecuteSQL() so that the row is not shown as UNKNOWN.

Comment on lines +1720 to +1725
QueryDataSetUtils.ResultChunk chunk =
fetchResultChunk(
queryExecution,
req.fetchSize,
req.isSetOffset() ? req.getOffset() : 0,
req.isSetOffset() ? config.getMppDataExchangeMaxPayloadSizeInBytes() : 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] A continuation request without offset silently drops the rest of a pending block

The execute paths decide by the session flag (getMaxResultPayloadSize(clientSession)), but this path decides by req.isSetOffset(). If a fragment is pending and the fetch request carries no offset, fetchResultChunk takes the legacy branch, which never looks at the pending fragment. The response is a plain success and the pending bytes are skipped.

Observed with a raw Thrift client on a session that enabled chunking: the execute response carried the first 131,072 bytes of a 3,000,025-byte block with moreData=true. A following fetchResultsV2 without offset returned status=200, hasResultSet=false, moreData=false.

The Java clients in this PR always set offset, so this needs an inconsistent client. It is still worth failing loudly, because other language clients will implement this protocol later. Either use the session flag here as well, or reject a request without offset while a fragment is pending.

Comment on lines +433 to +439
boolean finished =
SELECT_RESULT.apply(resp, execution, fetchSize, getMaxResultPayloadSize(clientSession));
// the cached result exceeds the payload bound, so the first response is always a fragment
checkState(
!finished,
DataNodeMiscMessages
.EXCEPTION_THE_FIRST_RESPONSE_OF_A_FRAGMENTED_FAST_LAST_RESULT_MUST_NOT_BE_COMPLETE_DEF121D5);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] The payload bound is read twice, and the assertion can fail after a hot reload

canFitInOneResponse reads mpp_data_exchange_max_payload_size_in_bytes to decide that the cached result is too large, and this call reads it again. The setting is hot-reloadable. If it grows in between, the whole block fits, finished is true, and checkState fails a query that could simply have been answered.

In executeFastLastDataQueryForOnePrefixPath the catch block then returns the error without cleaning up the execution registered above, so it stays in the Coordinator until the statement or the session is closed. executeFastLastDataQueryForOneDeviceV2 does clean up in its finally.

This comes from reading the code, I did not reproduce the race. Reading the bound once and passing it down, and handling finished like the other paths (resp.setMoreData(!finished) plus cleanup), would remove both problems.

Comment on lines +337 to +340
res.add(slice(byteBuffer, blockStart, remainingPayloadSize));
pendingFragment.reset(byteBuffer, blockStart + remainingPayloadSize);
return ResultChunk.fragment(
res, blockStart + remainingPayloadSize, blockStart == 0 ? blockSize : -1, false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Offsets are computed from the buffer position, not from the start of the block

blockStart + remainingPayloadSize is what the client receives as offset, and totalLength is only sent when blockStart == 0. The client expects offset to equal the number of bytes it has accumulated and requires totalLength on the first fragment, so both are only correct for a buffer whose position is 0.

That holds for every result handle today: the local and in-memory handles serialize into a fresh buffer, and the MPP exchange client uses the deep-copy transport. A buffer with a non-zero position would break the transfer with Missing totalLength or an offset mismatch. Slicing the block once before use (byteBuffer = byteBuffer.slice()) makes all offsets block-relative regardless of where the buffer comes from.

Comment on lines +426 to +432
if (nextOffset != buffer.size()) {
throw new IllegalArgumentException(
String.format(
RpcMessages
.EXCEPTION_CHUNKED_RESULT_FRAGMENT_OFFSET_ARG_NOT_GREATER_THAN_ACCUMULATED_LENGTH_ARG_A0F032E3,
nextOffset,
buffer.size()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The message does not match the check

The check rejects any nextOffset that differs from the accumulated length, but the message says the offset "is not greater than" it, which is misleading when the offset is larger. Something like Fragment offset %d does not match the accumulated length %d covers both cases. The constant name and the zh text need to follow.

public static final String EXCEPTION_DATA_TYPE_ARG_IS_NOT_SUPPORTED_31213160 =
"Data type %s is not supported.";

public static final String EXCEPTION_CHUNKED_RESULT_EMPTY_FRAGMENT_03A0EF39 =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Constant names do not follow the i18n naming rule

AGENTS.md derives the name from the full English text (<PREFIX>_<NORMALIZED>_<HASH>). The hashes of the new constants are right, but six of the seven names abbreviate the text. This one would be EXCEPTION_EMPTY_FRAGMENT_RECEIVED_WHEN_ASSEMBLING_A_CHUNKED_QUERY_RESULT_03A0EF39. The same applies to the other three here and to the two EXCEPTION_RESULT_FRAGMENT_... constants in DataNodeQueryMessages.

* bounded-payload chunks via {@link QueryDataSetUtils} and the client side ({@code
* IoTDBRpcDataSet}) assembles them back into the original rows.
*/
public class ResultChunkingEndToEndTest {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Please add integration tests that go through a real DataNode

This test and IoTDBRpcDataSetFragmentTest exercise the two ends against mocks, so the RPC entry points are not covered. Codecov reports 13.5% patch coverage for ClientRPCServiceImpl.

An IT with mpp_data_exchange_max_payload_size_in_bytes at its minimum (131072) keeps the data small. Useful cases:

  • a row larger than the bound, through Session and JDBC, in tree and table model
  • several oversized rows mixed with normal ones and a small fetchSize
  • both Fast LAST RPCs on a cached oversized value
  • closing the result set while a fragment is pending
  • a multi-DataNode run where the coordinator reads the result through a remote handle

I ran these by hand against this head (all except the table model through a table Session) and they pass, so this is about keeping them from regressing.

Comment on lines +302 to +305
// Do not fetch new blocks in the same response, otherwise the response would carry the
// tail of the big block plus up to fetchSize new rows, whose row count is not accounted
// for. One response either continues a fragmented block or delivers new blocks.
return ResultChunk.complete(res, !queryExecution.hasNextResult());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3, optional] Ordinary blocks are split at the payload boundary, and the tail travels alone

A block that does not fit into the remaining payload is fragmented even when it is smaller than the bound, and because of this early return the next response carries only its tail. For a result made of ordinary blocks this roughly doubles the number of round trips, and the client copies every split block once more to reassemble it.

Measured with the bound at 128 KiB and fetchSize=200000 on 200,000 INT64 rows (52 blocks of at most 65,549 bytes, 3.2 MB in total): a session without chunking needs 1 RPC, a chunking session needs 35, and 17 of them carry nothing but a tail.

With the default 4 MiB bound this only shows up when one response would exceed 4 MiB (large fetch sizes or wide rows), so it is not a blocker. A possible refinement: when a block is not larger than the bound and the response already has data, carry the whole block over to the next response instead of cutting it.

- Account a pending fragment under a query-owned memory account instead of the
  result handles: SharedTsBlockQueue.close() could release the last block's
  reservation (sink finish -> LocalSinkChannel.close -> queue.close) while the
  fragment still held it, and the queue's account is force-deregistered with the
  upstream fragment instance. The handle-level retainMemory/freeRetainedMemory
  API is removed again; PendingResultFragment reserves the serialized length on
  its first reset and releases it on clear.
- Give the Fast LAST materialized execution a real QueryId from
  Coordinator#createQueryId(), so SHOW QUERIES parses the DataNodeId correctly,
  and return a readable SQL from getExecuteSQL().
- Decide fragmentation in fetchResultsV2 by the session capability, so a
  continuation without offset fails the offset check instead of dropping the
  tail; read the payload bound once and pass it down; slice the block before
  computing offsets so they are block-relative; clean up the registered
  execution when the Fast LAST transfer fails.
- Rename the new i18n constants after their full English text and fix the
  offset mismatch message.
- Add integration tests: IoTDBSessionResultChunkingIT (tree model, standalone
  and cluster) and IoTDBTableResultChunkingIT (table model).
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants