Motivation
Make Hive leverage bulk skip when implementing probe decode for Parquet, similarly to https://issues.apache.org/jira/browse/HIVE-22731, which was about ORC.
Problem
ValuesReader.skip(int n) ships with a naive default:
public void skip(int n) {
for (int i = 0; i < n; i++) skip();
}
For dictionary-encoded columns (the common case), each skip() bottoms
out in RunLengthBitPackingHybridDecoder.readInt() — a mode switch,
array-index arithmetic, and a value the caller immediately discards.
Any filter-then-skip path (column-index row ranges, hash-join probe
filtering, runtime filters) pays this cost per skipped row.
Proposal
- Add
RunLengthBitPackingHybridDecoder.skipInts(int n) — re-use
readNext() per run, then advance currentCount by
min(n, currentCount) instead of walking every value through
readInt().
- Override
skip(int) on DictionaryValuesReader and
RunLengthBitPackingHybridValuesReader to call decoder.skipInts(n).
Component(s)
Core
Motivation
Make Hive leverage bulk skip when implementing probe decode for Parquet, similarly to https://issues.apache.org/jira/browse/HIVE-22731, which was about ORC.
Problem
ValuesReader.skip(int n)ships with a naive default:For dictionary-encoded columns (the common case), each
skip()bottomsout in
RunLengthBitPackingHybridDecoder.readInt()— a mode switch,array-index arithmetic, and a value the caller immediately discards.
Any filter-then-skip path (column-index row ranges, hash-join probe
filtering, runtime filters) pays this cost per skipped row.
Proposal
RunLengthBitPackingHybridDecoder.skipInts(int n)— re-usereadNext()per run, then advancecurrentCountbymin(n, currentCount)instead of walking every value throughreadInt().skip(int)onDictionaryValuesReaderandRunLengthBitPackingHybridValuesReaderto calldecoder.skipInts(n).Component(s)
Core