Skip to content

Read a list or map column referenced by a null filter - #4400

Merged
Fokko merged 2 commits into
apache:mainfrom
gedaliakliger:fix/4399-null-filter-list-map
Nov 17, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
gedaliakliger:fix/4399-null-filter-list-map

Conversation

@gedaliakliger

Copy link
Copy Markdown
Contributor

Closes #4399

Rationale for this change

A scan whose row_filter tests a list or map column for null (tags is null, props is not null) fails in to_arrow() with ValueError: Cannot explicitly project List or Map types.

ArrowScan._projected_field_ids leaves list and map IDs out of the projected schema's IDs, because prune_columns rebuilds those columns from their nested fields, but it adds the filter's field IDs as they are. A null test on a list or map column adds the column's own ID, and prune_columns rejects it. This change projects the nested fields of such a column instead, the same way a selected list or map column is read.

Are these changes tested?

Yes. test_projection_null_filter_on_list_and_map in tests/io/test_pyarrow.py runs IsNull and NotNull on a list, a map and a list of structs, with the column selected and with only id selected. It fails on main with the ValueError above and passes with this change.

Ran locally: pytest tests/io/test_pyarrow.py -m unit, and the ruff, ruff-format and mypy hooks on the two changed files. The rest of the suite and the integration tests are left to CI.

Are there any user-facing changes?

IS NULL and IS NOT NULL filters on list and map columns return the matching rows instead of raising.

A scan filter that tests a list or map column for null added the column's own
field ID to the projected IDs, and prune_columns rejects a list or map selected
by its own ID. Project the column's nested fields instead.
Comment thread tests/io/test_pyarrow.py Outdated
assert repr(result_table.schema) == "id: int32 not null"


def test_projection_null_filter_on_list_and_map(tmpdir: str) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test is hard to read, and it only tests a specific part of the code. How about having something similar as reported in the issue?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced it with a scan-level test that follows the issue: list, map and list-of-struct columns, is null and is not null, through a catalog. 9a74899

@Fokko
Fokko added this pull request to the merge queue Nov 17, 2026
@Fokko

Fokko commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Thanks @gedaliakliger 🙌

Merged via the queue into apache:main with commit 46d7835 Nov 17, 2026
20 checks passed
@gedaliakliger
gedaliakliger deleted the fix/4399-null-filter-list-map branch November 18, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

row_filter with IS NULL / IS NOT NULL on a list or map column raises "Cannot explicitly project List or Map types"

2 participants