Repository navigation
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces cross-query reuse for immutable Iceberg metadata I/O by adding a bounded metadata-content cache in FileIO and persisting parsed snapshot manifest-list entries on Snapshot instances to avoid repeated parsing across scans.
Changes:
- Add
MetadataCache+MetadataCacheOptionsand integrate it intoFileIOviaReadFileCached()/NewCachedInputFile(), with Java-compatible property names and defaults. - Persist snapshot manifest parsing results via a
Snapshot-owned shared cache state and returnstd::span<const ManifestFile>fromSnapshotCache. - Wire cache enablement through catalogs/registry and add unit tests + build system registration.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/iceberg/update/expire_snapshots.cc | Adjust manifest collection to work with the new span<const ManifestFile> cache surface. |
| src/iceberg/test/metadata_cache_test.cc | Adds coverage for cache reuse, eviction, invalidation, and concurrency behavior. |
| src/iceberg/test/meson.build | Registers the new metadata cache test with Meson. |
| src/iceberg/test/CMakeLists.txt | Registers the new metadata cache test with CMake. |
| src/iceberg/table_metadata.cc | Switches metadata JSON reads to go through the cached read path. |
| src/iceberg/snapshot.h | Adds snapshot-shared cache state and tightens manifest spans to const. |
| src/iceberg/snapshot.cc | Implements snapshot-shared manifest-list parsing/cache coalescing. |
| src/iceberg/metadata_cache.h | Introduces the public cache/options API and property contract. |
| src/iceberg/metadata_cache.cc | Implements the bounded, expiring, coalescing metadata cache. |
| src/iceberg/meson.build | Adds metadata_cache.cc and installs metadata_cache.h. |
| src/iceberg/manifest/manifest_reader.cc | Enables metadata-content caching for manifests and manifest lists via reader options. |
| src/iceberg/file_reader.h | Adds ReaderOptions::cache_content to control metadata caching. |
| src/iceberg/file_io.h | Adds cache configuration and cached read/input APIs to FileIO. |
| src/iceberg/file_io.cc | Implements cached reads, cached input wrapping, and cache configuration plumbing. |
| src/iceberg/file_io_registry.cc | Configures metadata caching during registry-based FileIO creation. |
| src/iceberg/CMakeLists.txt | Adds metadata_cache.cc to the build. |
| src/iceberg/catalog/sql/sql_catalog.cc | Wires cache configuration through SQL catalog properties when enabled. |
| src/iceberg/catalog/memory/in_memory_catalog.cc | Wires cache configuration through in-memory catalog properties when enabled. |
| src/iceberg/avro/avro_reader.cc | Passes caching intent down to Arrow input opening. |
| src/iceberg/arrow/arrow_io.cc | Adds an opt-in cached-input path when cache is enabled + requested. |
| src/iceberg/arrow/arrow_io_internal.h | Extends Arrow input open helper signature to accept cache intent. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/iceberg/file_reader.h:104
ReaderOptionsnow exposescache_content, but not all reader implementations appear to honor it. For example,parquet_reader.ccstill callsarrow::OpenArrowInputStream(options.io, options.path, options.length)without passingoptions.cache_content, so enablingcache_contentwould have no effect for Parquet reads. If the intent is that this flag controls whether the underlyingFileIOusesNewCachedInputFile, it should be forwarded consistently by readers that open Arrow input streams.
/// \brief FileIO instance to open the file.
std::shared_ptr<class FileIO> io;
/// \brief Cache this immutable metadata file's content when FileIO caching is enabled.
bool cache_content = false;
/// \brief The projection schema to read from the file. This field is required.
std::shared_ptr<class Schema> projection;
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/iceberg/metadata_cache.cc:168
- MetadataCacheOptions::expiration_interval_ms is only validated for non-negativity. Extremely large values can overflow/behave unexpectedly when converted to std::chrono::milliseconds in IsExpired(), leading to incorrect eviction behavior. Consider rejecting values larger than std::chrono::milliseconds::max().count().
ICEBERG_PRECHECK(options.expiration_interval_ms >= 0,
"Metadata cache expiration interval must not be negative: {}",
options.expiration_interval_ms);
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/iceberg/snapshot.h:422
cache_datais eagerly allocated for everySnapshotinstance and the manifest-list parse results appear to be retained for the lifetime of theSnapshot(no bounding/eviction). For tables with many snapshots and large manifest lists, this can cause unbounded memory retention across queries. Consider lazy-initializing the cache state on first use and/or adding a bounded/clearable policy similar toMetadataCache.
/// Internal lazy state shared by Snapshot copies so manifest lists are parsed once.
mutable std::shared_ptr<internal::SnapshotCacheData> cache_data =
internal::MakeSnapshotCacheData();
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/iceberg/metadata_cache.cc:45
- ParseBoolean only accepts exact "true"/"false" and rejects common case variants (e.g., "TRUE"), which can break configuration coming from Java-style properties sources. Since this is meant to be Java-compatible, parse booleans case-insensitively.
Result<bool> ParseBoolean(std::string_view key, const std::string& value) {
if (value == "true") {
return true;
}
if (value == "false") {
src/iceberg/file_io.cc:213
- NewCachedInputFile checks the cache using the caller-provided file_location, but CachedInputFile later uses input_file_->location() as the cache key. If an InputFile implementation normalizes/aliases locations, this inconsistency can cause missed cache hits and duplicate entries; use the InputFile's location consistently.
if (auto cached = cache->GetIfPresent(file_location)) {
return std::make_unique<CachedInputFile>(std::move(input_file), std::move(cache),
static_cast<int64_t>(cached->size()));
}
src/iceberg/file_io.cc:281
- ClearMetadataCache currently clears only the byte-content MetadataCache, but it does not reset the per-FileIO SnapshotCacheData (parsed manifest-list entries). That means manifest-list parsing cache can grow without any clear/invalidation path even when callers explicitly clear metadata caches.
void FileIO::ClearMetadataCache() {
auto cache = GetMetadataCache();
if (cache != nullptr) {
cache->Clear();
}
d70ca60 to
433113f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.
Suppressed comments (3)
src/iceberg/file_io.cc:210
NewCachedInputFilechecks the cache usingfile_location, butCachedInputFilelater usesinput_file_->location()(location_) as the cache key. If aFileIOnormalizes/rewrites locations (e.g., resolves paths, strips schemes, redirects), this can cause systematic cache misses and duplicate cache entries. Consider usinginput_file->location()consistently for (1) theGetIfPresentlookup and (2) the location passed into cached loads, while still preserving the originalfile_locationfor error messages if desired.
Result<std::unique_ptr<InputFile>> FileIO::NewCachedInputFile(
std::string file_location, std::optional<size_t> length) {
std::unique_ptr<InputFile> input_file;
if (length.has_value()) {
ICEBERG_ASSIGN_OR_RAISE(input_file, NewInputFile(file_location, *length));
} else {
ICEBERG_ASSIGN_OR_RAISE(input_file, NewInputFile(file_location));
}
auto cache = GetMetadataCache();
if (cache == nullptr || !cache->options().enabled) {
return input_file;
}
if (auto cached = cache->GetIfPresent(file_location)) {
return std::make_unique<CachedInputFile>(std::move(input_file), std::move(cache),
static_cast<int64_t>(cached->size()));
}
src/iceberg/metadata_cache.cc:145
- The eviction condition uses
options_.max_total_bytes - total_bytes_onsize_t, which can underflow iftotal_bytes_ever exceedsmax_total_bytes(even transiently due to future changes). A safer, more self-documenting check iswhile (!lru_.empty() && (incoming_bytes + total_bytes_ > options_.max_total_bytes))(with overflow-safe ordering), which avoids unsigned underflow and makes the invariant explicit.
void EvictToFit(size_t incoming_bytes) {
while (!lru_.empty() && incoming_bytes > options_.max_total_bytes - total_bytes_) {
auto it = entries_.find(lru_.back());
if (it == entries_.end()) {
lru_.pop_back();
} else {
EraseLoaded(it);
}
}
}
src/iceberg/arrow/arrow_io.cc:535
- In this function, the
if (cache_content)branch is reached only whencache_contentis true andio->MetadataCacheEnabled()is true (due to the earlier guard). Theelse if (length.has_value()) { ... } else { ... }branch is therefore dead code in this path and can be simplified to reduce cognitive overhead (e.g., remove theelsecases here and keep the non-cached path entirely in the earlier branch).
int64_t size;
std::unique_ptr<InputFile> input_file;
if (cache_content) {
ICEBERG_ASSIGN_OR_RAISE(input_file, io->NewCachedInputFile(path, length));
} else if (length.has_value()) {
ICEBERG_ASSIGN_OR_RAISE(input_file, io->NewInputFile(path, *length));
} else {
ICEBERG_ASSIGN_OR_RAISE(input_file, io->NewInputFile(path));
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/iceberg/file_io.cc:230
NewCachedInputFileonly skips wrapping whensize > max_content_length, but a file can also be larger thanmax_total_bytes. In that caseCachedInputFile::Open()will still read the entire file into memory viaMetadataCache::Get()(and then drop it from the cache), which can cause large transient allocations and defeats the intent ofmax_total_bytesas a bound. Consider also bypassing the cached wrapper whensize > max_total_bytes.
if (std::cmp_greater(size, cache->options().max_content_length)) {
return input_file;
}
return std::make_unique<CachedInputFile>(std::move(input_file), std::move(cache), size);
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Leave a comment to not make it stale. |
88752d5 to
4ce6ed5
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
6 open issues
ReserveForUnitWeight()usesstd::is_same_v, which requires<type_traits>to be included by… · New Give zero-length cache entries a positive weight Ifinput->GetSize()fails, the function returns early without closinginput, which can leak… · NewkMaxCacheWeightis stored assize_tbut represents anint64_tboundary; on 32-bit platforms… · New Check the cache before resolving the backing file Validate cached input length before calling virtual method
Review effort: Lite
52a6848 to
687667b
Compare
|
@wgtmac @shangxinli The cache is now owned per table and previous comments should be resolved. Please review again. |
WZhuo
left a comment
There was a problem hiding this comment.
Thanks for working on this! My main concern is where the cache lives. Right now it is attached to each Table handle, so it starts empty on every LoadTable, and there is no process-wide memory bound (see inline comments).
I think one ContentCache per Catalog would work better:
- Memory management: a single, catalog-wide budget instead of "number of tables ×
max-total-bytes". - Reuse: loading, refreshing and committing tables from the same catalog all share the same cache, so repeated scans across
LoadTablecalls actually benefit. This would also remove the need forReuseManifestCache. - Per-table opt-out: whether to use the cache becomes a parameter when the catalog builds the table, so a specific table load can bypass the cache when needed.
This would also simplify the plumbing: catalogs would no longer need to forward io_properties on every Table::Make / StagedTable::Make call.
| ICEBERG_ASSIGN_OR_RAISE(auto options, | ||
| internal::ContentCacheOptions::FromProperties(properties)); | ||
| if (options.enabled) { | ||
| content_cache_ = std::make_shared<internal::ContentCache>(options); |
There was a problem hiding this comment.
The cache is attached to a single Table handle, so every LoadTable call starts with an empty cache. Many engines call LoadTable once per query, so with that usage pattern the "reuse across repeated scans" this PR is aiming for will rarely kick in.
There is also no process-wide memory limit: the total footprint is effectively "number of tables × 100 MiB". For comparison, Java keys the content cache by FileIO (which is shared by tables loaded from the same catalog), keeps at most 8 FileIO caches (iceberg.io.manifest.cache.fileio-max), and holds them as soft values so they can be reclaimed under memory pressure.
There was a problem hiding this comment.
I added a per process limit with default value 8 like Java. The cache is also held as soft values now.
|
@WZhuo We can have both a catalog cache and a table cache which provides caching at different layers. We use the catalog cache when loading data from a remote catalog and a table cache when from the filesystem. I don't think they should be mixed into one. |
Keep bounded content-cache state on Table and share it with manifest readers, scans, and compatible refresh and commit handles. Configure independent table budgets from catalog and table properties while keeping FileIO and ReaderOptions unchanged. Reuse weighted LruCache for binary content and bounded normalized aliases. Skip empty and oversized files, validate lengths before storage access, and close Arrow size probes on every path while preserving errors.
Add braces required by clang-tidy and retain the intentional positional ReaderOptions compatibility check with a targeted suppression. Co-authored-by: Codex <codex@openai.com>
Treat cache configuration as a best-effort read optimization, simplify its configuration and I/O plumbing, and invalidate deleted manifests without warming the cache during expiration cleanup. Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
9c65802 to
43adf80
Compare
Co-authored-by: Codex <codex@openai.com>


Summary
Repeated scans currently reread immutable manifests and manifest lists. Add an opt-in content cache for each loaded
Table, shared by its manifest readers and scan builders, with a process-wide limit on retained cache states.FileIO. A process-wide LRU retains at most eight cache states by default. Table handles and manifest I/O adapters share a cache handle that holds a weak reference to the retained state. Compatible refreshes and committed table handles share their source table's warm cache.internal::LruCachefor weighted eviction of binarystd::vector<std::byte>content. Look up by file location and use the canonical location returned by storage when it differs.GetFileInfowhen available. Close fallback probes on both success and failure.Configuration and lifetime
Process-wide cache count
The default is eight retained manifest caches per process. Set
ICEBERG_IO_MANIFEST_CACHE_LIMIT=16in the application's launch environment to raise it to 16. The registry reads this setting once at initialization. The value must be a positive integer that fits inint32_t. An invalid value logs a warning and uses the default.For runtime changes, call
Table::SetManifestCacheLimit(...). The setter overrides the launch value.Include
iceberg/table.hand check the returnedStatus. Lowering the count immediately evicts the least recently used caches, even if their tables are still loaded.Per-table cache settings
Each independently loaded table gets its own byte budget. Enable caching and set the byte limits through the table's
FileIO::properties().io.manifest.cache-enabledfalsetrueio.manifest.cache.max-total-bytes104857600(100 MiB)67108864(64 MiB)io.manifest.cache.max-content-length8388608(8 MiB)4194304(4 MiB)For a REST catalog, add these string-valued keys to the client configuration passed to
RestCatalog::Make. For a table-specific override, return the keys in the REST load-table response'sconfigmap. REST merges those values into the table'sFileIO. If you inject a customFileIO, return the same keys from itsproperties()method. The process-wide count is set separately throughTable::SetManifestCacheLimit.Eviction and refresh
The next read after eviction recreates the state on demand, and destroying the last cache handle releases any retained state. Active readers and input files can still hold bytes after eviction, and concurrent load buffers are outside the retained byte budget.
Refreshes preserve warm caches across S3 access key, secret key, session token, and token expiry changes when the table UUID, cache options, and remaining storage properties match. This also covers S3 configurations that use the default endpoint. Refresh installs the new
FileIOeven when the metadata location is unchanged, so cache misses use the new credentials. A different endpoint, region,io-impl, or per-table byte limit starts a new cache. A replacementFileIOwith no comparable storage properties or S3 credentials also starts a new cache.Disabled caches ignore tuning properties. Invalid limits log a warning and disable the cache so publishing new metadata can still succeed. Immutable content leaves the cache through weighted eviction, manifest cleanup, or
Table::ClearManifestCache(). Files above either byte limit bypass caching. Time-based expiration and coalescing concurrent loads are outside the scope of this PR.The library guide (
mkdocs/docs/library.md) documents the settings, cache lifetime, and refresh behavior.Test coverage
Local validation
58080f8e) passed a macOS C++23 Debug core build,util_test(including all 39 manifest-cache tests), and pre-commit on all PR files.Before
c159911autil_test,arrow_test,manifest_test, andtable_update_test.