Skip to content

Fix partitioned appends of narrow integer, s/ms timestamp and large_binary columns - #4421

Open
sohailshk wants to merge 1 commit into
apache:mainfrom
sohailshk:fix/partition-arrow-types
Open

sohailshk wants to merge 1 commit into
apache:mainfrom
sohailshk:fix/partition-arrow-types

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

Rationale for this change

Table.append accepts any Arrow column that converts to the Iceberg column type, but the partition transforms of pyiceberg-core only handle the Arrow type that matches the Iceberg type exactly. On a partitioned table these appends fail with a ValueError from pyiceberg-core (FeatureUnsupported => Unsupported data type for bucket transform or Unexpected => Should not call internally for unsupported data type), although the same data appends fine to an unpartitioned table or an identity-partitioned one.

This is easy to hit: pyarrow.csv.read_csv and pyarrow.json.read_json infer timestamp[s], which breaks the common spec of a table partitioned by day on the event time. DuckDB TIMESTAMP_S, TIMESTAMP_MS and SMALLINT come out as timestamp[s], timestamp[ms] and int16, and polars produces large_binary.

import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import DayTransform
from pyiceberg.types import NestedField, TimestampType

catalog = SqlCatalog("c", uri="sqlite:///:memory:", warehouse="file:///tmp/wh")
catalog.create_namespace("n")
schema = Schema(NestedField(1, "ts", TimestampType(), required=False))
spec = PartitionSpec(PartitionField(1, 1000, DayTransform(), "ts_day"))
table = catalog.create_table("n.t", schema, partition_spec=spec)
table.append(pa.table({"ts": pa.array([1, 2], pa.timestamp("s"))}))
# ValueError: Unexpected => Should not call internally for unsupported data type Timestamp(Second, None)

The other affected combinations of Arrow type and transform:

  • int8, int16, uint8, uint16 (and uint32 or in-range uint64 on a long column) for bucket and truncate
  • timestamp[s] and timestamp[ms], with or without a time zone, for bucket, day and hour, and for identity with time zone +00:00
  • large_binary and binary_view for truncate

The fix

_determine_partitions now casts the source column of each partition field to the Arrow type of its Iceberg type (int32/int64, microsecond timestamps, binary) before applying the transform. Only the partition computation changes; the data that is written is unchanged (narrow integers are already widened on write since #2799). large_binary is cast to binary for truncate only (apache/iceberg-rust#3119 tracks LargeBinary support in the Rust truncate kernel), so bucket and identity on large_binary behave as before. Nanosecond timestamps and dictionary-encoded columns are left alone (#4340 covers dictionary).

Partition values are computed on the Iceberg-typed value (the spec hashes int as long, and timestamps are microseconds), which are the values Java computes; the unit test checks every value against the Python transforms.

Are these changes tested?

Yes, in tests/io/test_pyarrow.py:

  • test_determine_partitions_with_arrow_types_narrower_than_the_iceberg_type checks the partition values of each type and transform pair against the Python transforms
  • test_determine_partitions_keeps_large_binary_for_bucket checks that bucket on large_binary is not cast
  • test_append_arrow_types_narrower_than_the_iceberg_type_to_partitioned_table appends to a table partitioned by bucket, day and truncate and checks the scan round trip

The first and the last fail on main (Unsupported data type for bucket transform: Int16) and pass now. pytest tests/io/test_pyarrow.py tests/table/test_partitioning.py tests/test_transforms.py (without the network-bound S3/GCS/ADLS/retry/ssl tests): 557 passed, 1 skipped.

Are there any user-facing changes?

Partitioned tables accept these Arrow column types on append and overwrite.

@Fokko
Fokko self-requested a review November 18, 2026 21:30

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sohailshk, this is a nice fix and the PR description is excellent, the repro makes the gap very clear :)

One thing to address before this can merge, the required integration job is red because a test codified the old behavior:

  • tests/integration/test_writes/test_partitioned_writes.py:884 test_unsupported_transform expects a ValueError when appending large_binary to a truncate(2)-partitioned binary column. With the new cast the append succeeds, so it fails with DID NOT RAISE. Could you update the test to assert the append now works (or drop it in favor of the new unit tests)?

  • pyiceberg/io/pyarrow.py _cast_to_transform_input_type: rest of the logic looks right to me. Small note: an int64 or uint64 column with out-of-range values on an int source now raises ArrowInvalid (a ValueError subclass) from the cast. That seems like the right outcome, but if you agree it might be worth a line in the PR description.

Everything else is green (lint, unit tests on 3.11-3.14, windows, s3/gcs/adls), only the required integration check fails because of the test above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants