Repository navigation
Parquet: Read footer with a single tail read for range-readable streams - #18724
gtrettenero wants to merge 1 commit into
Conversation
Parquet reads the footer length at the end of the file and then seeks backward to read the footer. For object store streams this costs two sequential requests, and the backward seek aborts the first stream so the second request also opens a new connection. When the stream supports RangeReadable, read the end of the file once with readTail and parse the footer from it using Parquet's own readFooter, then open the reader with the pre-read footer. A second tail read is only needed when the footer is larger than 64 KiB. Other streams use the existing path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| class ParquetIO { | ||
| // number of bytes read from the end of a file to get the footer in a single request; this matches | ||
| // the default footer read size used by the Arrow C++ Parquet reader | ||
| @VisibleForTesting static final int FOOTER_READ_SIZE = 64 * 1024; |
There was a problem hiding this comment.
Open question for reviewers: 64 KiB matches the Arrow C++ default footer read size and covers typical footers, but very wide schemas (roughly 1,000+ columns) have larger footers and need a second tail read. That case is no worse than today (two requests either way), just no better. Would you prefer a larger default (e.g. 256 KiB), a read property to tune it, or keeping it as is?
Adds ParquetFooterReadBenchmark to compare ParquetFileReader.open (seek-based footer read) with ParquetIO.open (tail read) from apache#18724, using a simulated object store, a local S3-compatible server (Testcontainers), or a real bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I put a JMH benchmark for this change in a separate PR to keep this one focused: gtrettenero#2. The results, setup, and reproduction steps are in that PR's description. Through the real |
Summary
Parquet reads the footer by seeking to the end of the file for the footer length and magic, then seeking backward to read the footer (
ParquetFileReader.readFooter). ForFileIOstreams backed by object stores this is two sequential GET requests per file. ForS3InputStream, the backward seek also aborts the first, partially read HTTP stream, so the second request opens a new connection and pays a new TLS handshake.This change reads the footer with a single
RangeReadable.readTailcall when the stream supports it, which covers S3, GCS, and ADLS:readTailof exactlyfooter length + 8bytes.ParquetFileReader.readFooter(InputFile, ParquetReadOptions, SeekableInputStream)over an in-memory stream of those bytes. Parquet still validates length, magic, and footer position, applies the read options' metadata filter (row-group range), and handles encrypted footers.ParquetFileReader(InputFile, ParquetMetadata, ParquetReadOptions, SeekableInputStream)constructor.ParquetFileReader(InputFile, ParquetReadOptions, SeekableInputStream)already does exactly this internally (this(file, readFooter(file, options, f, true), options, f)); only the source of the footer bytes changes.RangeReadablekeep the current path unchanged: Hadoop streams,AesGcmInputStream, and in-memory streams.Because
ReadBuilderalways enables vectored IO, andParquetRangeReadableInputStreamAdapterservesreadVectoredwith positioned reads, the sequential stream is no longer used at all when reading throughReadConf. Each file now costs one tail request plus row-group range requests, instead of two sequential footer requests plus an aborted connection plus the row-group requests.This complements #17284 (eager fetch of files ≤ 1 MiB) for files above the eager-fetch threshold. Below the threshold the stream is the in-memory
EagerInputStream, so the tail read costs no extra request.Motivation
We found this while profiling a Spark
rewrite_data_filesworkload on S3 that reads many small and medium files. In sampled executor thread dumps, task threads spent a large share of wall time blocked inReadConf.newReader→ParquetFileReader.readFooter→S3InputStream.openStream. The time split between the two footer requests, and there were TLS handshake frames from the reopened connection. The cost is per file, so it dominates when files are small and numerous.Testing
New
TestParquetFooterRead:footer length + 8.Parquet.readof a file above the eager-fetch threshold makes no sequential reads; row groups use positioned reads.is not a Parquet fileerror, and the stream is closed.Existing tests:
./gradlew :iceberg-parquet:check: 784 tests, plus checkstyle, error-prone, and spotless.:iceberg-data:test --tests '*Parquet*': 146 tests.Notes for reviewers
RangeReadable.readTailreturning exactly the requested bytes when the request is no larger than the file. A short read raisesEOFException. Like the existing path andEagerInputFile, it trustsInputFile.getLength(), which may come from manifest metadata.ParquetUtil.fileMetrics, which reads only the footer (used byTableMigrationUtil), could useParquetIO.opentoo. I kept this PR focused on the read path.AI disclosure: this change was drafted with AI assistance (Claude Code). That covered the analysis of the read path, the implementation, and the test scaffolding. The areas above are the ones that most deserve careful review.
🤖 Generated with Claude Code