Skip to content

Populate nan_value_counts when writing parquet files - #4412

Closed
AP24110011695 wants to merge 1 commit into
apache:mainfrom
AP24110011695:fix/nan-value-counts-writer
Closed

AP24110011695 wants to merge 1 commit into
apache:mainfrom
AP24110011695:fix/nan-value-counts-writer

Conversation

@AP24110011695

Copy link
Copy Markdown
Contributor

What's changed

Implements design A from #4396: data_file_statistics_from_parquet_metadata returned an empty nan_value_counts dict unconditionally, so data files written by the pyarrow writer never reported NaN counts — and NaN-aware scan pruning stayed silently unavailable for them (_InclusiveMetricsEvaluator degrades to ROWS_MIGHT_MATCH without nan counts, for is_nan, not_nan and range predicates via _contains_nans_only).

pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no nan_count/has_nan_count), so the writer now counts them from the written table itself:

  • ParquetFormatWriter.write() accumulates pa.compute.is_nan(...) sums per top-level float/double field (is_nan is null on nulls and sum skips nulls — matching the spec's NaN-count semantics of counting non-null NaN values)
  • the counts flow through a new optional table_nan_value_counts parameter of data_file_statistics_from_parquet_metadata
  • the metadata-only caller (parquet_file_to_data_file, used by add_files) keeps its previous behavior — no NaN counts from metadata-only input

Nested float fields keep the previous (empty) behavior for now; the metrics plan and evaluator keys make them a natural follow-up once the top-level path is settled.

Test

test_nan_value_counts_are_written (mixed values + null) and test_nan_value_counts_are_written_across_row_groups (all-NaN, two write() calls) — both verified failing-first on main (nan_value_counts == {}) and passing with the change; the second also pins the pruning win (not_nan now prunes an all-NaN file via _contains_nans_only). Full tests/io/test_pyarrow_stats.py (22) green; the pre-existing azure-fixture failures in test_pyarrow.py are unrelated (missing optional dep in this environment).

AI disclosure: drafted with AI assistance, verified locally by running the failing-first cycle.

data_file_statistics_from_parquet_metadata returned an empty
nan_value_counts dict unconditionally, so data files written by the
pyarrow writer never reported NaN counts and NaN-aware scan pruning
stayed unavailable for them — the metrics evaluator degrades to
ROWS_MIGHT_MATCH without nan counts (_InclusiveMetricsEvaluator
visit_is_nan / _contains_nans_only).

pyarrow's statistics API does not expose NaN counts, so the writer now
counts them from the written table itself (pa.compute.is_nan summed per
top-level float/double field, excluding nulls per the spec's NaN-count
semantics) and passes them through a new optional
table_nan_value_counts parameter. The metadata-only caller
(parquet_file_to_data_file) keeps its previous behavior.

Refs apache#4396.

Signed-off-by: Ayush Kumar Saha <225407166+AP24110011695@users.noreply.github.com>

@rahulsmahadev rahulsmahadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AP24110011695! #4401 merged today with the same change (NaN counts taken from the data before writing, for top-level float and double columns), including tests for repeated write() calls, the none metrics mode and pruning, so this now conflicts with main. Is there a case here that #4401 doesn't cover?

@AP24110011695

Copy link
Copy Markdown
Contributor Author

Good question, and the answer is no — #4401 covers everything this PR did and a bit more. Same writer scope (top-level float/double, counts from the data before writing, accumulated across repeated write() calls), plus the none-metrics-mode handling I had missed entirely.

For the record, #4396 was opened on Nov 16 (the NaN counts were never accumulated by the writer — the metrics evaluator's nan-based pruning was silently dead for pyiceberg-written files), and #4401 picking it up the next morning and merging is exactly how that should have played out. My branch raced it and lost, and it now conflicts with main as you noted.

Closing this one — thanks for checking rather than letting a conflicting duplicate sit. The open thread on #4396 is the add_files path, which #4401 explicitly left for a separate change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants