Skip to content

Keep the properties dict passed to create_table unchanged - #4456

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/create-table-keep-caller-properties
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/create-table-keep-caller-properties

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: new_table_metadata reads the requested format version with properties.pop("format-version", ...) on the dict passed to create_table, so the caller's dict loses the key. For example, reusing a v1 properties dict with the SQL catalog creates the second table as v2:

properties = {"format-version": "1", "owner": "etl"}
catalog.create_table("db.events_2024", schema, properties=properties).format_version  # 1
catalog.create_table("db.events_2025", schema, properties=properties).format_version  # 2
properties  # {'owner': 'etl'}

create_table_transaction removes the key before the commit, and create_table_if_not_exists also removes it when the table exists. The pop is from #411, to keep format-version out of the persisted table properties.

Fix: pop from a copy. The Glue and DynamoDB create_table wrote the caller's dict to the Glue parameters and the DynamoDB item after this call, so they relied on the pop; they now write the new metadata's properties, as Glue's commit_table does. That also turns the values into strings there: Glue stores True as true, and DynamoDB accepts non-string values instead of failing with ParamValidationError.

Java: the BaseMetastoreCatalog table builder copies the properties into its own map, and TableMetadata.newTableMetadata reads the version with PropertyUtil.propertyAsInt and persists a new filtered map, so the caller's map is never changed.

  • Java's persistedProperties leaves out all reserved properties (uuid, snapshot-count, ...); new_table_metadata still only leaves out format-version.

Are these changes tested?

Yes, test_create_v1_tables_with_shared_properties in tests/catalog/test_catalog_behaviors.py, test_new_table_metadata_does_not_change_properties in tests/table/test_metadata.py and test_create_v1_table in tests/catalog/test_dynamodb.py fail on main (the dict loses format-version, or the versions are [1, 2, 2]) and pass with this change. test_create_v1_table and test_create_table_s3tables in tests/catalog/test_glue.py now check the exact Glue parameters.

Are there any user-facing changes?

Yes, create_table no longer changes the passed properties dict, and the Glue and DynamoDB catalogs write the property values as strings when creating a table.

Please add the changelog label.

This pull request and its description were written by Isaac.

new_table_metadata popped format-version from the caller's properties
dict, so reusing one dict gave the requested version only to the first
table. Pop from a copy, and make the Glue and DynamoDB create paths,
which relied on the pop, write the new metadata's properties instead.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, you reviewed #4133, which pruned removed table properties from Glue parameters. @Ankitraj-builds, you wrote it. Could you take a look when you have a chance? Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant