Repository navigation
Conversation
|
Thanks for finding this @mhatipog , Are you an ORC user? We are discussing removing ORC support for V4 tables so i was wondering if you had a vested interest in keeping orc around. Last we checked we didn't have any real maintainers |
|
Thanks Russell! I'm a data engineer working on data platform migrations and cross-engine compatibility. I came across this issue while investigating data correctness across formats. I don't have a particular dependency on ORC, but I think preserving reliable reads of existing ORC data is important, especially during migrations. I'd also be interested in contributing further to the ORC integration if maintainership is an area where help is needed. |
|
We do need it but I was mostly interested in if you had a production ORC use case. We were hoping to drop support for ORC unless we could find some real sign that users were interested in ORC in the future. |
|
Understood, no production ORC use case on my side. I'll focus on getting the correctness fix merged. |
Fixes #19064.
ORC only stores a struct field where all of its ancestors are present, and
hasNull()covers only the field's own nulls, so a field that is null because its parent is null was treated as non-null:OrcMetricsnow derives the null count asrowCount - numberOfValuesfor fields outside lists and maps, instead of 0 whenhasNull()is false. Trino's Iceberg writer computes it the same way.ExpressionToSearchArgumentno longer pushesisNullto ORC for a field with an optional ancestor struct.notNulland fields inside lists and maps are unchanged.Both are needed: with only the metrics change, ORC still skips the row group. Files written before this change keep their metrics until they are rewritten.
Testing: new cases in
TestMetrics(Parquet and ORC) andTestExpressionToSearchArgument, which fail without the change, plus the updated expectation intestNestedPrimitives. I also checked the repro from the issue on 1.12.0 with Spark 4.0: the row is returned and deleted.AI tools helped draft the change and tests; I reviewed and tested them.