Skip to content

Fix strict metrics evaluation on nested columns - #4090

Closed
anishmehta24 wants to merge 1 commit into
apache:mainfrom
anishmehta24:fix/strict-metrics-nested-columns
Closed

anishmehta24 wants to merge 1 commit into
apache:mainfrom
anishmehta24:fix/strict-metrics-nested-columns

Conversation

@anishmehta24

Copy link
Copy Markdown

Rationale for this change

_StrictMetricsEvaluator raises when a predicate references a nested column, so delete() and overwrite() with such a filter fail during planning:

schema = pa.schema([pa.field("id", pa.int64()), pa.field("s", pa.struct([pa.field("x", pa.int64())]))])
tbl = catalog.create_table("default.t", schema=schema)
tbl.append(pa.table({"id": [1, 2], "s": [{"x": 1}, {"x": 5}]}, schema=schema))
tbl.delete("s.x != 5")
# ValueError: Cannot find field, might be nested or missing: 3

NotEqualTo, In and NotIn look up the field in the top-level struct before checking bounds, so they fail even on files that pyiceberg writes itself, which have no nested bounds. The range and equality predicates fail once the file has bounds for the nested column, for example files written by Spark. Java fixed this in apache/iceberg#11261: for a nested column, strict evaluation returns ROWS_MIGHT_NOT_MATCH because nested null counts are not reliable. This change does the same. It also matters for #3810, which starts writing nested bounds from pyiceberg. After it, the delete above rewrites the file and leaves [{'id': 2, 's': {'x': 5}}].

Are these changes tested?

Yes. test_strict_nested_column in tests/expressions/test_evaluator.py covers nested columns with and without bounds, and it fails on main with the ValueError above. The evaluator, visitor and residual tests pass.

Are there any user-facing changes?

Yes. delete() and overwrite() with a filter on a nested struct field no longer raise.

Written with help from Claude.

Return ROWS_MIGHT_NOT_MATCH for nested columns instead of raising, as Java does since apache/iceberg#11261.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 01:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 8, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants