Repository navigation
Conversation
_HasIds previously required every column to have an ID. A file with IDs on only some columns, such as a Databricks UniForm table with row-tracking columns, fell back to name mapping and failed when physical names differed from the mapped names. Match Java's ParquetSchemaUtil.hasIds (any) and prune fields without IDs, including incomplete nested types.
There was a problem hiding this comment.
Thanks @linzch3! The read side matches Java's ParquetSchemaUtil.hasIds, and the new tests fail on main and pass here.
On the write-path change you mention in the description, I'd recommend keeping the old behavior, since it can lose data silently:
df = pq.read_table(path_of_a_pyiceberg_data_file).append_column("country", pa.array(["NL"]))
tbl.append(df) # main: ValueError "PyArrow table contains more columns: country"; here: succeeds, country is lostEven after union_by_name, as that error suggests, the append writes None for country, where main writes "NL". One option: appends keep the old rule (match by name or reject), and only reads of data files use the new one.
A couple of smaller suggestions:
- A column without an ID is still type-converted before pruning, so an unsupported type such as
time32[ms]fails the read; pruning first would avoid that. - Removing the element-ID guard in
_ConvertToIceberg.list, or the key-ID check inmap, still passes the tests, so a test for each could help.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @linzch3! Checked out the branch and ran tests/io/test_pyarrow_visitor.py locally: all 63 pass, and the UniForm-style example with physical names that differ from the mapping is nicely covered.
Read side matching Java's ParquetSchemaUtil.hasIds makes sense to me, and I confirmed _ConvertToIcebergWithoutIDs inherits the pruning logic but always returns -1 IDs, so the create-table path is unchanged.
I agree with @rahulsmahadev on the write-path concern: silently dropping a column on append loses data without any signal, so I'd keep the old reject behavior there and only use the prune-on-read rule.
Nit: in _HasIds.map the any(...) means a map where only the key has an ID passes the check and the value is then dropped in _ConvertToIceberg.map. That is consistent with the new policy, just worth keeping in mind when reporting errors on the write path, since the user will not see why a column disappeared.
Closes #4417
Rationale for this change
pyarrow_to_schemaonly used field IDs when every column had one (_HasIdsusedall). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns.I changed
_HasIdsto match Java'sParquetSchemaUtil.hasIds(any), and_ConvertToIcebergnow drops fields without IDs, including lists or maps whose element, key, or value is missing an ID.Are these changes tested?
The issue's example is a unit test in
tests/io/test_pyarrow_visitor.py, with and without a name mapping. I also added a nested case: a map missing child IDs is dropped, and a struct keeps only the children that have IDs. Existing_HasIdstests for partial IDs now expectTrue. Two schema-compatibility tests that rebuild a nested field fromas_arrow()now keep field IDs on that field, so they still test a missing nested column rather than the new prune path.pytest tests/io/test_pyarrow_visitor.pyand the related schema-compatibility tests passed.make lintpassed.Are there any user-facing changes?
Files that already have IDs on every column are unchanged. Files with IDs on only some columns are now read by ID, and columns without IDs are ignored, even if a name mapping is present. That also applies to schema-compatibility checks on write: an extra column without an ID is dropped rather than rejected. I can change that if you'd rather keep the old write-path behavior.
Happy to adjust the approach if you'd prefer something else.