Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions mkdocs/docs/library.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,40 @@ follows:
`iceberg::avro::RegisterAll()`.
- Parquet readers and writers are registered by
`iceberg::parquet::RegisterAll()`.

## Manifest content caching

Tables can cache immutable manifest and manifest-list bytes. Per-table settings come
exclusively from `FileIO::properties()`; table metadata properties do not enable
or override the cache. REST catalogs supply their merged catalog and table I/O
configuration through the table's `FileIO`. Applications that inject a `FileIO`
must expose the desired settings through its `properties()` method.

| Property | Default |
|---|---|
| `io.manifest.cache-enabled` | `false` |
| `io.manifest.cache.max-total-bytes` | `104857600` (100 MiB) |
| `io.manifest.cache.max-content-length` | `8388608` (8 MiB) |

Independent table loads have separate caches and byte budgets. A process-wide
LRU registry retains at most eight cache states by default. To change the limit
at launch, set `ICEBERG_IO_MANIFEST_CACHE_LIMIT` to a positive integer that fits
in `int32_t`. For example, `ICEBERG_IO_MANIFEST_CACHE_LIMIT=16` allows up to 16
retained caches per process. The registry reads this setting once when it
initializes and uses the default if the setting is missing or invalid.

`Table::SetManifestCacheLimit(max_caches)` overrides the initial limit for the
entire process. Lowering the limit immediately evicts the least recently used
cache states.

Per-table handles and manifest I/O adapters retain only weak references to cache
states. Cache states are created on first use, and an evicted table recreates an
empty cache on its next read. Active readers and input files can still hold bytes
after eviction, and concurrent load buffers are outside the retained byte budget.
Call `Table::ClearManifestCache()` to release a table's retained content explicitly.

Compatible refreshes and committed table handles share their source table's cache.
S3 access keys, secrets, session tokens, and token expiry can rotate without losing
warm content when the remaining storage configuration matches. Changes to cache
limits or storage configuration, such as an endpoint or region, start a new cache.
Data files and table metadata JSON continue to use storage I/O directly.
1 change: 1 addition & 0 deletions src/iceberg/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ set(ICEBERG_SOURCES
catalog/memory/in_memory_catalog.cc
catalog/session_catalog.cc
catalog/session_context.cc
content_cache_internal.cc
data_table_scan.cc
delete_file_index.cc
deletes/dv_util.cc
Expand Down
32 changes: 28 additions & 4 deletions src/iceberg/arrow/arrow_io.cc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <limits>
#include <mutex>
#include <optional>
#include <utility>
#include <vector>

#include <arrow/buffer.h>
Expand Down Expand Up @@ -426,9 +427,30 @@ class ArrowInputFile : public InputFile {
if (file_size_.has_value()) {
return *file_size_;
}
::arrow::fs::FileInfo file_info(path_, ::arrow::fs::FileType::File);
ICEBERG_ARROW_ASSIGN_OR_RETURN(auto input, fs_->OpenInputFile(file_info));
ICEBERG_ARROW_ASSIGN_OR_RETURN(auto size, input->GetSize());
ICEBERG_ARROW_ASSIGN_OR_RETURN(auto info, fs_->GetFileInfo(path_));
if (info.type() == ::arrow::fs::FileType::NotFound) {
return IOError("File does not exist: {}", location_);
}
if (info.type() != ::arrow::fs::FileType::File) {
return IOError("Not a regular file: {}", location_);
}
if (info.size() != ::arrow::fs::kNoSize) {
file_size_ = info.size();
return *file_size_;
}
// Some filesystems report a file without a size.
ICEBERG_ARROW_ASSIGN_OR_RETURN(auto input, fs_->OpenInputFile(info));
auto size_result = input->GetSize();
auto close_status = input->Close();
if (!size_result.ok() && !close_status.ok()) {
auto status = size_result.status().WithMessage(
size_result.status().message(),
"; additionally failed to close file: ", close_status.ToString());
ICEBERG_ARROW_RETURN_NOT_OK(status);
}
ICEBERG_ARROW_ASSIGN_OR_RETURN(auto size, std::move(size_result));
ICEBERG_ARROW_RETURN_NOT_OK(close_status);
file_size_ = size;
Comment thread
manuzhang marked this conversation as resolved.
Resolved
return size;
}

Expand All @@ -445,7 +467,9 @@ class ArrowInputFile : public InputFile {
std::shared_ptr<::arrow::fs::FileSystem> fs_;
std::string location_;
std::string path_;
std::optional<int64_t> file_size_;
// Set by Size() so a later Open() can skip the size lookup. Not synchronized: like
// other InputFile handles, this one is not shared across threads.
mutable std::optional<int64_t> file_size_;
};

class ArrowOutputFile : public OutputFile {
Expand Down
1 change: 1 addition & 0 deletions src/iceberg/catalog/memory/in_memory_catalog.cc
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,7 @@ Result<std::shared_ptr<InMemoryCatalog>> InMemoryCatalog::Make(
const std::string& name, const std::shared_ptr<FileIO>& file_io,
const std::string& warehouse_location,
const std::unordered_map<std::string, std::string>& properties) {
ICEBERG_PRECHECK(file_io != nullptr, "InMemoryCatalog requires a non-null FileIO");
std::shared_ptr<MetricsReporter> reporter;
auto it = properties.find(std::string(kMetricsReporterImpl));
if (it != properties.end() && !it->second.empty() &&
Expand Down
4 changes: 3 additions & 1 deletion src/iceberg/catalog/rest/rest_table.cc
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,15 @@ Result<std::shared_ptr<RestTable>> RestTable::Make(
std::shared_ptr<Catalog> catalog, std::string full_name,
std::shared_ptr<MetricsReporter> reporter, RestScanContext rest_context,
std::unique_ptr<Labels> labels) {
ICEBERG_PRECHECK(io != nullptr, "FileIO cannot be null");
if (metadata == nullptr) {
return InvalidArgument("Metadata cannot be null");
}
return std::shared_ptr<RestTable>(new RestTable(
auto table = std::shared_ptr<RestTable>(new RestTable(
std::move(identifier), std::move(metadata), std::move(metadata_location),
std::move(io), std::move(catalog), std::move(full_name), std::move(reporter),
std::move(rest_context), std::move(labels)));
return table;
}

Result<std::unique_ptr<DataTableScanBuilder>> RestTable::NewScan() const {
Expand Down
Loading