Repository navigation
Conversation
Fokko
left a comment
There was a problem hiding this comment.
Thanks @sohailshk, this is a nice fix and the PR description is excellent, the repro makes the gap very clear :)
One thing to address before this can merge, the required integration job is red because a test codified the old behavior:
-
tests/integration/test_writes/test_partitioned_writes.py:884
test_unsupported_transformexpects a ValueError when appending large_binary to a truncate(2)-partitioned binary column. With the new cast the append succeeds, so it fails with DID NOT RAISE. Could you update the test to assert the append now works (or drop it in favor of the new unit tests)? -
pyiceberg/io/pyarrow.py
_cast_to_transform_input_type: rest of the logic looks right to me. Small note: an int64 or uint64 column with out-of-range values on an int source now raises ArrowInvalid (a ValueError subclass) from the cast. That seems like the right outcome, but if you agree it might be worth a line in the PR description.
Everything else is green (lint, unit tests on 3.11-3.14, windows, s3/gcs/adls), only the required integration check fails because of the test above.
Rationale for this change
Table.appendaccepts any Arrow column that converts to the Iceberg column type, but the partition transforms of pyiceberg-core only handle the Arrow type that matches the Iceberg type exactly. On a partitioned table these appends fail with aValueErrorfrom pyiceberg-core (FeatureUnsupported => Unsupported data type for bucket transformorUnexpected => Should not call internally for unsupported data type), although the same data appends fine to an unpartitioned table or an identity-partitioned one.This is easy to hit:
pyarrow.csv.read_csvandpyarrow.json.read_jsoninfertimestamp[s], which breaks the common spec of a table partitioned by day on the event time. DuckDBTIMESTAMP_S,TIMESTAMP_MSandSMALLINTcome out astimestamp[s],timestamp[ms]andint16, and polars produceslarge_binary.The other affected combinations of Arrow type and transform:
int8,int16,uint8,uint16(anduint32or in-rangeuint64on a long column) forbucketandtruncatetimestamp[s]andtimestamp[ms], with or without a time zone, forbucket,dayandhour, and foridentitywith time zone+00:00large_binaryandbinary_viewfortruncateThe fix
_determine_partitionsnow casts the source column of each partition field to the Arrow type of its Iceberg type (int32/int64, microsecond timestamps, binary) before applying the transform. Only the partition computation changes; the data that is written is unchanged (narrow integers are already widened on write since #2799).large_binaryis cast tobinaryfortruncateonly (apache/iceberg-rust#3119 tracksLargeBinarysupport in the Rust truncate kernel), sobucketandidentityonlarge_binarybehave as before. Nanosecond timestamps and dictionary-encoded columns are left alone (#4340 covers dictionary).Partition values are computed on the Iceberg-typed value (the spec hashes
intaslong, and timestamps are microseconds), which are the values Java computes; the unit test checks every value against the Python transforms.Are these changes tested?
Yes, in tests/io/test_pyarrow.py:
test_determine_partitions_with_arrow_types_narrower_than_the_iceberg_typechecks the partition values of each type and transform pair against the Python transformstest_determine_partitions_keeps_large_binary_for_bucketchecks thatbucketonlarge_binaryis not casttest_append_arrow_types_narrower_than_the_iceberg_type_to_partitioned_tableappends to a table partitioned by bucket, day and truncate and checks the scan round tripThe first and the last fail on main (
Unsupported data type for bucket transform: Int16) and pass now.pytest tests/io/test_pyarrow.py tests/table/test_partitioning.py tests/test_transforms.py(without the network-bound S3/GCS/ADLS/retry/ssl tests): 557 passed, 1 skipped.Are there any user-facing changes?
Partitioned tables accept these Arrow column types on append and overwrite.