Skip to content

GH-55347: [C++][Parquet][Python] Support the FINITE_VECTOR logical type - #55348

Draft
rok wants to merge 2 commits into
apache:mainfrom
rok:parquet-vector-logical-type
Draft

rok wants to merge 2 commits into
apache:mainfrom
rok:parquet-vector-logical-type

Conversation

@rok

@rok rok commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

apache/parquet-format#624 added the FINITE_VECTOR(num_elements) logical type (merged, not yet in a format release): a canonical 3-level LIST whose non-null values each hold exactly num_elements required, finite primitive elements. Arrow's FixedSizeList maps onto it directly. parquet-java counterpart: apache/parquet-java#3876.

What changes are included in this PR?

  • Format: add FiniteVectorType to parquet.thrift and regenerate with thrift 0.23.0 (same version as before). The generated diff is almost entirely renumbered temporaries and can be skimmed.
  • Schema: LogicalType::FiniteVector(n) (n > 0, converted type LIST). Constructing an annotated GroupNode validates the canonical structure and the element type (BOOLEAN, INT32/INT64, FLOAT, DOUBLE, FLOAT16 or DECIMAL), both when building schemas and when reading files.
  • Read: FINITE_VECTOR(n) is read as fixed_size_list<element: T not null>[n]. A nullable element field is restored from the stored Arrow schema.
  • Write: experimental ArrowWriterProperties::Builder::enable_finite_vector_logical_type(), off by default. Eligible FixedSizeList columns are written as FINITE_VECTOR, all others as LIST as before. Writing fails with Invalid if a non-null vector contains a null, NaN or ±Inf element.
  • Reader fix: a non-nullable fixed-size list under a null struct was rejected as a wrong-sized list. It now reads correctly.
  • Python: use_finite_vector_logical_type on write_table, ParquetWriter and the dataset write options.
  • Docs: FINITE_VECTOR added to the C++ Parquet logical type table.

Are these changes tested?

Yes, in schema_test.cc, arrow_schema_test.cc, arrow_reader_writer_test.cc and test_data_types.py: logical type properties and thrift round-trip, structure validation, schema mapping in both directions, data round-trips (nulls, nesting, extension types), fallback to LIST, write errors, and reading a file with a wrong-sized vector.

Are there any user-facing changes?

  • New opt-in, experimental writer option in C++ and Python. Default behaviour is unchanged.
  • Files containing FINITE_VECTOR are now read as FixedSizeList.

Released Arrow versions cannot open these files ("Logical type Undefined cannot be applied to group node") instead of falling back to ConvertedType.LIST. That is a separate forward-compatibility problem with unknown group logical types and is not addressed here.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Copilot AI balanced review requested due to automatic review settings October 30, 2026 22:27
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55347 has been automatically assigned in GitHub to PR creator.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Required vectors fail to read under null parent structs, while insufficient element-type filtering causes schema-conversion failures, exponential conversion work, and unvalidated NaN/Inf values in VECTOR files.

2 open issues
What changed in this PR

Adds experimental support for the proposed Parquet VECTOR logical type, mapping it to Arrow FixedSizeList in C++ and Python while keeping VECTOR writing disabled by default.

Changes:

  • Adds VECTOR serialization, logical-type APIs, and canonical schema validation.
  • Implements FixedSizeList mapping and opt-in writing with element validation.
  • Exposes Python writer and dataset options, with documentation and round-trip tests.
File Description
python/​pyarrow/​tests/​parquet/​test_data_types.py Tests VECTOR round-trips, fallbacks, validation, and dataset writing.
python/​pyarrow/​parquet/​core.py Exposes and documents the Python writer option.
python/​pyarrow/​includes/​libparquet.pxd Declares VECTOR and C++ writer-property bindings.
python/​pyarrow/​_parquet.pyx Connects Python options to C++ and names VECTOR metadata.
python/​pyarrow/​_parquet.pxd Extends the writer-properties helper declaration.
python/​pyarrow/​_dataset_parquet.pyx Adds VECTOR to dataset write options.
docs/​source/​cpp/​parquet.rst Documents VECTOR mapping and experimental writing.
cpp/​src/​parquet/​types.h Declares the VECTOR logical-type API.
cpp/​src/​parquet/​types.cc Implements VECTOR properties and serialization.
cpp/​src/​parquet/​schema.h Exposes VECTOR element eligibility validation.
cpp/​src/​parquet/​schema.cc Validates canonical VECTOR group structure.
cpp/​src/​parquet/​schema_test.cc Tests logical-type properties, structure, and serialization.
cpp/​src/​parquet/​properties.h Adds the opt-in C++ writer setting.
cpp/​src/​parquet/​parquet.thrift Defines VECTOR in the format schema.
cpp/​src/​parquet/​arrow/​writer.cc Aligns element nullability and validates written values.
cpp/​src/​parquet/​arrow/​schema.cc Maps VECTOR and FixedSizeList schemas.
cpp/​src/​parquet/​arrow/​arrow_schema_test.cc Tests schema mapping, defaults, and fallbacks.
cpp/​src/​parquet/​arrow/​arrow_reader_writer_test.cc Tests nested round-trips and invalid values or lengths.
cpp/​src/​generated/​parquet_types.tcc Adds generated VECTOR protocol serialization.
cpp/​src/​generated/​parquet_types.h Adds generated VECTOR declarations.
cpp/​src/​generated/​parquet_types.cpp Adds generated VECTOR implementations.

Review effort: Balanced

Comment thread cpp/src/parquet/arrow/schema.cc Resolved
Comment thread cpp/src/parquet/arrow/schema.cc Resolved
@rok
rok force-pushed the parquet-vector-logical-type branch 2 times, most recently from 714d287 to 8acc8ae Compare October 31, 2026 17:50
@rok rok changed the title GH-55347: [C++][Parquet][Python] Support the proposed VECTOR logical type GH-55347: [C++][Parquet][Python] Support the proposed FINITE_VECTOR logical type Oct 31, 2026
@rok
rok force-pushed the parquet-vector-logical-type branch 2 times, most recently from d888630 to 31e7a59 Compare November 1, 2026 14:23

@MashrafeeAryan MashrafeeAryan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just had few comments on it Rok. Overall, it looks good to me.

Comment thread cpp/src/parquet/arrow/writer.cc Outdated
if (root_field != nullptr) {
ARROW_ASSIGN_OR_RAISE(
std::shared_ptr<ArrayData> prepared,
MakeFiniteVectorElementsRequired(array_to_write->data(), *root_field));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don’t have any FINITE_VECTOR fields, can we skip this helper? We could check the schema once before the chunk loop. If there are no finite vectors, we can skip the helper instead of having it walk the nested arrays in every chunk.

I am sure how much time this would save. Could we benchmark it?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: each column now checks once, before the chunk loop, whether its schema subtree has a FINITE_VECTOR, and skips the helper otherwise. I didn't benchmark it. The helper only walks the type tree (not the values) and returns the input unchanged when there's nothing to do, so I'd expect the difference to be in the noise.

Comment thread cpp/src/parquet/schema.cc
field_name_to_idx_.clear();
auto field_idx = 0;
for (NodePtr& field : fields_) {
field->SetParent(this);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are setting the new group as the child’s parent with field->SetParent(this) before checking if the group is valid. If CheckFiniteVectorStructure throws an error, the parent group goes away. Then the child now points to a parent that no longer exists. I believe if we move the check before setting the parent, it would be great

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I moved the check ahead of the SetParent loop, so a rejected group is never left as the parent of its fields. Also added a test that the child's parent stays null after the throw.

rok added 2 commits November 19, 2026 18:58
…cal type

Add the FINITE_VECTOR logical type from parquet-format
(apache/parquet-format#624): a canonical 3-level LIST whose non-null
values contain exactly num_elements required, finite primitive elements.

- Add FiniteVectorType to parquet.thrift and regenerate with thrift
  0.23.0.
- Add LogicalType::FiniteVector / FiniteVectorLogicalType, mapped to
  ConvertedType LIST, and validate the canonical structure and allowed
  element types when a FINITE_VECTOR-annotated GroupNode is created.
- Read FINITE_VECTOR groups as fixed_size_list<element: T not null>[n],
  restoring a nullable element field from the stored Arrow schema.
- Fix FixedSizeListReader so that a non-nullable fixed-size list under a
  null ancestor (e.g. a required FINITE_VECTOR inside a nullable struct)
  is padded instead of rejected as a wrong-sized list.
- Add the experimental ArrowWriterProperties option
  enable_finite_vector_logical_type() (default off) to write eligible
  FixedSizeList columns as FINITE_VECTOR. Eligibility is decided from
  the Arrow value type (extension storage unwrapped, dictionaries
  excluded); ineligible ones are written as LIST. Writing fails if a
  non-null vector has a null or non-finite element.
- Expose use_finite_vector_logical_type in pyarrow's write_table,
  ParquetWriter and dataset write options.
- Check the FINITE_VECTOR structure before the fields are given the new
  group as their parent, so a rejected group is never left as a parent.
- Decide once per column whether it contains a FINITE_VECTOR and skip
  MakeFiniteVectorElementsRequired otherwise.
- FINITE_VECTOR is now part of parquet-format (not yet released): update
  the pyarrow docstring and point the FixedSizeList docs footnote to it.
@rok
rok force-pushed the parquet-vector-logical-type branch from 31e7a59 to 67f89f6 Compare November 19, 2026 18:15
@rok rok changed the title GH-55347: [C++][Parquet][Python] Support the proposed FINITE_VECTOR logical type GH-55347: [C++][Parquet][Python] Support the FINITE_VECTOR logical type Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants