Skip to content

ORC: Fix IS NULL on fields under a null parent struct - #19065

Open
mhatipog wants to merge 8 commits into
apache:mainfrom
mhatipog:orc-metrics-ancestor-null-counts
Open

mhatipog wants to merge 8 commits into
apache:mainfrom
mhatipog:orc-metrics-ancestor-null-counts

Conversation

@mhatipog

Copy link
Copy Markdown

Fixes #19064.

ORC only stores a struct field where all of its ancestors are present, and hasNull() covers only the field's own nulls, so a field that is null because its parent is null was treated as non-null:

  • OrcMetrics now derives the null count as rowCount - numberOfValues for fields outside lists and maps, instead of 0 when hasNull() is false. Trino's Iceberg writer computes it the same way.
  • ExpressionToSearchArgument no longer pushes isNull to ORC for a field with an optional ancestor struct. notNull and fields inside lists and maps are unchanged.

Both are needed: with only the metrics change, ORC still skips the row group. Files written before this change keep their metrics until they are rewritten.

Testing: new cases in TestMetrics (Parquet and ORC) and TestExpressionToSearchArgument, which fail without the change, plus the updated expectation in testNestedPrimitives. I also checked the repro from the issue on 1.12.0 with Spark 4.0: the row is returned and deleted.

AI tools helped draft the change and tests; I reviewed and tested them.

@RussellSpitzer

Copy link
Copy Markdown
Member

Thanks for finding this @mhatipog , Are you an ORC user? We are discussing removing ORC support for V4 tables so i was wondering if you had a vested interest in keeping orc around. Last we checked we didn't have any real maintainers

@github-actions github-actions Bot added the API label Nov 20, 2026
@mhatipog

Copy link
Copy Markdown
Author

Thanks Russell! I'm a data engineer working on data platform migrations and cross-engine compatibility. I came across this issue while investigating data correctness across formats.

I don't have a particular dependency on ORC, but I think preserving reliable reads of existing ORC data is important, especially during migrations.

I'd also be interested in contributing further to the ORC integration if maintainership is an area where help is needed.

@RussellSpitzer

Copy link
Copy Markdown
Member

We do need it but I was mostly interested in if you had a production ORC use case. We were hoping to drop support for ORC unless we could find some real sign that users were interested in ORC in the future.

@mhatipog

Copy link
Copy Markdown
Author

Understood, no production ORC use case on my side. I'll focus on getting the correctness fix merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ORC: IS NULL on a struct field misses rows whose parent struct is null

2 participants