Skip to content

fix(arrow): filter absent nullable columns in ordered comparisons - #3508

Open
mattfaltyn wants to merge 4 commits into
apache:mainfrom
mattfaltyn:fix/missing-null-comparisons
Open

mattfaltyn wants to merge 4 commits into
apache:mainfrom
mattfaltyn:fix/missing-null-comparisons

Conversation

@mattfaltyn

@mattfaltyn mattfaltyn commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3355.

What changes are included in this PR?

< and <= keep every row when an optional column is absent from a Parquet file, even though the same column stored as NULL matches no rows. Treat the absent column as NULL when it has no initial default or non-null identity partition value.

Required-field validation and existing default/partition-constant handling are preserved. The broader evaluation of non-null missing values remains in #3302; this fix can land independently.

Are these changes tested?

Local validation passed: 7 focused tests; 2,513 workspace nextest tests (1 pre-existing ignored test); 96 doctests (19 ignored examples); and 41 Python tests (1 expected failure). Workspace build, Clippy, documentation, formatting, TOML, dependency/license checks, MSRV, no-default-feature build, and all 10 public API baselines passed. API comparisons used a path-safe equivalent of the Make recipe for a checkout path containing a space. Live HuggingFace tests were gated off without credentials.

All 25 CI checks passed. The audit reproduction passes all 32 comparison configurations, including pruning switches and projected/filter-only columns; 16 failed before the fix.

AI Disclosure

Codex assisted with investigation, implementation, tests, and this description. Review should focus on preserving required-field errors and default/identity-partition behavior while changing only the known-NULL comparison fallback. Non-null missing-value filtering and null-partition/default precedence remain outside this fix.

@mattfaltyn
mattfaltyn marked this pull request as ready for review October 22, 2026 13:17
}))
}

fn missing_field_is_null(&self, reference: &BoundReference) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not_in and not_starts_with still return build_always_true() for a missing nullable column. On this branch b NOT IN ('x', 'y') and b NOT STARTS WITH 'x' give 3 rows when b is missing from the file and 0 when it's stored as NULL. Same mismatch as <, so maybe worth covering here, or a follow-up?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this! Both predicates now treat an absent nullable column like a stored NULL. I added Parquet regression coverage, and the local Rust and Python suites passed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ran my probe on 90649d6. NOT IN and NOT STARTS WITH now give 0 rows for the missing column, same as the stored-NULL file. NOT (b < 'm'), IS NULL and IS NOT NULL match too. LGTM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Row filter keeps every row for < and <= on a null column that is missing from the data file

2 participants