Repository navigation
Deprecate implicit non-spec type changes in update_column - #4410
rahulsmahadev wants to merge 2 commits into
Conversation
update_column applied string <-> binary and fixed(16) -> uuid silently, using the read-side promote(). Check the spec's promotion list instead. Those three pairs now warn with a DeprecationWarning, and are rejected in a later release unless allow_incompatible_changes=True is set. Co-authored-by: Isaac <no-reply@databricks.com>
|
@geruh, could you review the compatibility transition for these previously accepted schema changes? @Anurag-S1ngh, tagging you as the latest author of |
Co-authored-by: Isaac <no-reply@databricks.com>
| other_field_id: int | None = None | ||
|
|
||
|
|
||
| def _is_valid_promotion(from_type: IcebergType, to_type: IcebergType) -> bool: |
There was a problem hiding this comment.
I'm wondering if we can re-use promote:
iceberg-python/pyiceberg/schema.py
Line 1671 in 9955502
| if isinstance(from_type, StringType) and isinstance(to_type, BinaryType): | ||
| return True | ||
| if isinstance(from_type, BinaryType) and isinstance(to_type, StringType): | ||
| return True |
There was a problem hiding this comment.
This was an earlier discussion: apache/iceberg#5151
I would just allow this for now. We should have one single place to do this, for example, promote() as suggested above.
Same for the UUID. I believe we support this because of older versions of PyArrow, that didn't support UUIDs yet. This makes it a bit more lenient to still convert it into a UUID.
Let's encode these edge cases as comments in the code, so the LLM doesn't trip over it again.
Rationale for this change
update_columnchecks a type change withpromote(), the read-side rule that the Avro and Parquet readers use. That rule is more lenient than the table spec: it also acceptsstring -> binary,binary -> stringandfixed(16) -> uuid. So a populatedstringcolumn can be changed tobinarywithout any warning, and the commit succeeds:The spec's "Valid primitive type promotions" table does not list these pairs. Java's
SchemaUpdate.updateColumnrejects them throughTypeUtil.isPromotionAllowed, which allows equal types,int -> long,float -> doubleand decimal widening with the same scale.This PR validates primitive type changes using Java's v1/v2 promotion rules, while keeping PyIceberg's existing
unknown -> primitivebehaviour. It does not add v3date -> timestampordate -> timestamp_nssupport, and it does not broaden unknown-type conversions. Validation runs whenfield_typeis supplied andallow_incompatible_changesis false.promote()is unchanged, because the Avro resolver, the Arrow projection, theunion_by_nameno-op check and append compatibility (_check_schema_compatible) still rely on its leniency.Behaviour change: without
allow_incompatible_changes=True,update_columnwarns before applyingstring -> binary,binary -> stringorfixed[16] -> uuid. The warning is aDeprecationWarningthat says the change is deprecated in 0.13.0 and will be removed in 0.14.0. A subsequent release will require the explicit opt-in. Every other invalid change still raisesValidationError, as onmain.allow_incompatible_changes=Trueapplies these three changes without a warning.union_by_namewarns when an incoming UUID field replaces an existingfixed[16]field, including nested struct fields, list elements and map values. String/binary unions remain unchanged, becauseunion_by_namechecks them withpromote()before it callsupdate_column. An incomingfixed[16]field over an existing UUID field also remains a no-op.Precedent: Fokko added the string <-> binary allowed-update tests in apache/iceberg#8374 (September 2023), and #207 only moved that test file. #986 added
test_table_scan_default_to_large_types, which changed a string column to binary withupdate_column. #1669 later changed that test to create the column as binary, so the update became a no-op.Alternative for maintainers: reject these three changes immediately instead of warning first. #3613 rejected decimal promotions that change the scale without a deprecation cycle, and Fokko approved it. This PR chooses a warning cycle, because these changes were tested as supported (apache/iceberg#8374), and AGENTS.md asks for a deprecation cycle before breaking public API. Choosing immediate rejection means replacing the warning branch in
update_columnwith aValidationError.Other engines: in a local DuckDB 2.0.0
iceberg_scanreproduction, promoting a populated string column to binary made a non-ASCII value fail to read, and changed the four-character string\x41to the single byteA. PyIceberg continued to read the original UTF-8 bytes.Are these changes tested?
Yes.
tests/test_schema.pycovers the following.test_update_column_rejects_invalid_promotions: the exact error message for each invalid change (for exampleint -> float, a decimal scale change andfixed[15] -> uuid), unchanged table metadata after the rejections, and the valid promotions committed and read back after a reload.test_update_column_warns_for_non_spec_promotions: the three deprecated changes warn with the exact message, commit, and read back after a reload.test_update_column_no_warning_with_allow_incompatible_changes: the explicit opt-in applies the same three changes without a warning.test_union_by_name_fixed_uuid_and_string_binary:fixed(16)and UUID in both directions, string/binary unchanged, and a list element.test_update_column_promotes_unknown_to_primitive_only: the existingunknownbehaviour, on a v3 table.On
main, the three warning tests fail withDID NOT WARNand the union case fails because no warning is emitted. With this change they all pass. Intests/integration/test_rest_schema.py,test_allowed_updatesandtest_disallowed_updatesexpect the deprecation warning for string <-> binary, andtest_alter_map_keystill opts in withallow_incompatible_changes=Trueto reach the map key check.Are there any user-facing changes?
Yes. Without
allow_incompatible_changes=True,update_columnemits aDeprecationWarningbefore applyingstring -> binary,binary -> stringorfixed[16] -> uuid. Without this opt-in,union_by_namewarns when incoming UUID fields replace existing fixed[16] fields. String/binary unions and incoming fixed[16] fields over existing UUID fields remain type no-ops. The deprecated implicit changes are scheduled to require the explicit opt-in in 0.14.0. Please add thechangeloglabel.This pull request and its description were written by Isaac.