Skip to content

Find partition source columns the same way as the writer - #4457

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/partition-source-under-dotted-struct
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/partition-source-under-dotted-struct

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: _determine_partitions reads the partition source from the input by its full name in the current table schema, split on dots, while the writer follows the table's struct path with the input's own names (task_schema, from the Arrow field IDs or the name mapping). That gives two bugs:

  • A dot in a name on the path to a nested source fails the write. With a struct a.b and an identity partition on its child c, append raises KeyError: 'Field "a" does not exist in schema', because the split (from Fix support for writing to nested field partition #2204) looks for a column a.
  • When the input has field IDs from before a rename, the partition values come from another column than the one the writer stores, so filtered scans miss the rows:
schema = Schema(NestedField(1, "a", StringType()), NestedField(2, "b", StringType()))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="a_part"))
tbl = catalog.create_table("db.t", schema=schema, partition_spec=spec)
df = pa.Table.from_pylist([{"a": "x", "b": "y"}], schema=tbl.schema().as_arrow())  # field IDs 1 and 2
for old, new in [("a", "tmp"), ("b", "a"), ("tmp", "b")]:
    with tbl.update_schema() as update:
        update.rename_column(old, new)
tbl.append(df)
tbl.scan().to_arrow().to_pylist()  # [{'b': 'x', 'a': 'y'}]
[task.file.partition for task in tbl.scan().plan_files()]  # [Record[y]], expected [Record[x]]
tbl.scan(row_filter="b = 'x'").to_arrow().to_pylist()  # [], expected the row

Fix: find the source the same way as the writer, following its struct path in the table schema with each name taken from the input schema, and pass the names to Arrow literally. How the writer matches columns is unchanged. An input without the partition source's field ID now raises a ValueError, instead of a KeyError or a wrong partition value.

Java: SparkWriteBuilder aligns the input schema's IDs with the table, and SparkWrite reuses a PartitionKey whose StructTransform reads each row through schema.accessorForField(sourceId) of that input schema.

Not covered here: dynamic partition overwrite still has #4196 (open fix #4204), and the row filter converter (_ConvertToArrowExpression._get_field_name) also splits names, so delete rewrites and filtered scans of add_files files with dotted struct names can still raise ArrowInvalid.

Are these changes tested?

Yes, test_append_to_identity_partitioned_table_with_dotted_names, test_append_with_field_ids_after_swapping_column_names and test_partition_values_follow_writer_struct_path in tests/catalog/test_catalog_behaviors.py, and test_partition_for_nested_field_with_dotted_names in tests/io/test_pyarrow.py. On main the dotted-name tests fail with the KeyError above and the swap test with partition y instead of x; all pass with this change.

Are there any user-facing changes?

Yes, writes to tables partitioned on a nested field with a dot in a name on its path work, and partition values match the written data when the input has field IDs from before a rename.

Please add the changelog label.

This pull request and its description were written by Isaac.

_determine_partitions read the partition source from the input by its full
name in the table schema, split on dots, while the writer finds each column
by following the table's struct path with the input's own names. A dotted
struct name raised KeyError, and an input with field IDs from before a
rename got its partition values from another column, so filtered scans
missed rows. Follow the writer's path and names instead.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@geruh, you wrote _get_field_from_arrow_table in #2204. Could you take a look when you have a chance? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant