Skip to content

Parquet: Read footer with a single tail read for range-readable streams - #18724

Open
gtrettenero wants to merge 1 commit into
apache:mainfrom
gtrettenero:parquet-footer-tail-read
Open

gtrettenero wants to merge 1 commit into
apache:mainfrom
gtrettenero:parquet-footer-tail-read

Conversation

@gtrettenero

Copy link
Copy Markdown
Contributor

Summary

Parquet reads the footer by seeking to the end of the file for the footer length and magic, then seeking backward to read the footer (ParquetFileReader.readFooter). For FileIO streams backed by object stores this is two sequential GET requests per file. For S3InputStream, the backward seek also aborts the first, partially read HTTP stream, so the second request opens a new connection and pays a new TLS handshake.

This change reads the footer with a single RangeReadable.readTail call when the stream supports it, which covers S3, GCS, and ADLS:

  • Read the last 64 KiB of the file, or the whole file if it is smaller. 64 KiB matches the default footer read size of the Arrow C++ Parquet reader.
  • If the footer is larger than that, issue one more readTail of exactly footer length + 8 bytes.
  • Parse the footer with Parquet's own public ParquetFileReader.readFooter(InputFile, ParquetReadOptions, SeekableInputStream) over an in-memory stream of those bytes. Parquet still validates length, magic, and footer position, applies the read options' metadata filter (row-group range), and handles encrypted footers.
  • Open the reader with the existing ParquetFileReader(InputFile, ParquetMetadata, ParquetReadOptions, SeekableInputStream) constructor. ParquetFileReader(InputFile, ParquetReadOptions, SeekableInputStream) already does exactly this internally (this(file, readFooter(file, options, f, true), options, f)); only the source of the footer bytes changes.
  • Streams that are not RangeReadable keep the current path unchanged: Hadoop streams, AesGcmInputStream, and in-memory streams.

Because ReadBuilder always enables vectored IO, and ParquetRangeReadableInputStreamAdapter serves readVectored with positioned reads, the sequential stream is no longer used at all when reading through ReadConf. Each file now costs one tail request plus row-group range requests, instead of two sequential footer requests plus an aborted connection plus the row-group requests.

This complements #17284 (eager fetch of files ≤ 1 MiB) for files above the eager-fetch threshold. Below the threshold the stream is the in-memory EagerInputStream, so the tail read costs no extra request.

Motivation

We found this while profiling a Spark rewrite_data_files workload on S3 that reads many small and medium files. In sampled executor thread dumps, task threads spent a large share of wall time blocked in ReadConf.newReader → ParquetFileReader.readFooter → S3InputStream.openStream. The time split between the two footer requests, and there were TLS handshake frames from the reopened connection. The cost is per file, so it dominates when files are small and numerous.

Testing

New TestParquetFooterRead:

  • A small footer is read with a single tail read and no seek-and-read on the stream.
  • A footer larger than 64 KiB (2,000-column file) uses exactly two tail reads: 64 KiB, then footer length + 8.
  • An end-to-end Parquet.read of a file above the eager-fetch threshold makes no sequential reads; row groups use positioned reads.
  • Streams without range reads fall back to the existing path.
  • A non-Parquet file still fails with Parquet's is not a Parquet file error, and the stream is closed.

Existing tests:

  • ./gradlew :iceberg-parquet:check: 784 tests, plus checkstyle, error-prone, and spotless.
  • :iceberg-data:test --tests '*Parquet*': 146 tests.

Notes for reviewers

  • The 64 KiB read size is a constant, not a read property. I'm happy to make it configurable if preferred.
  • The change relies on each RangeReadable.readTail returning exactly the requested bytes when the request is no larger than the file. A short read raises EOFException. Like the existing path and EagerInputFile, it trusts InputFile.getLength(), which may come from manifest metadata.
  • Follow-up: ParquetUtil.fileMetrics, which reads only the footer (used by TableMigrationUtil), could use ParquetIO.open too. I kept this PR focused on the read path.

AI disclosure: this change was drafted with AI assistance (Claude Code). That covered the analysis of the read path, the implementation, and the test scaffolding. The areas above are the ones that most deserve careful review.

🤖 Generated with Claude Code

Parquet reads the footer length at the end of the file and then seeks
backward to read the footer. For object store streams this costs two
sequential requests, and the backward seek aborts the first stream so the
second request also opens a new connection.

When the stream supports RangeReadable, read the end of the file once with
readTail and parse the footer from it using Parquet's own readFooter, then
open the reader with the pre-read footer. A second tail read is only needed
when the footer is larger than 64 KiB. Other streams use the existing path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gtrettenero
gtrettenero marked this pull request as ready for review October 31, 2026 10:41
class ParquetIO {
// number of bytes read from the end of a file to get the footer in a single request; this matches
// the default footer read size used by the Arrow C++ Parquet reader
@VisibleForTesting static final int FOOTER_READ_SIZE = 64 * 1024;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open question for reviewers: 64 KiB matches the Arrow C++ default footer read size and covers typical footers, but very wide schemas (roughly 1,000+ columns) have larger footers and need a second tail read. That case is no worse than today (two requests either way), just no better. Would you prefer a larger default (e.g. 256 KiB), a read property to tune it, or keeping it as is?

gtrettenero pushed a commit to gtrettenero/iceberg that referenced this pull request Oct 31, 2026
Adds ParquetFooterReadBenchmark to compare ParquetFileReader.open (seek-based
footer read) with ParquetIO.open (tail read) from apache#18724, using
a simulated object store, a local S3-compatible server (Testcontainers), or a
real bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gtrettenero

Copy link
Copy Markdown
Contributor Author

I put a JMH benchmark for this change in a separate PR to keep this one focused: gtrettenero#2. The results, setup, and reproduction steps are in that PR's description.

Through the real S3FileIO against a local S3-compatible server, the change cuts GetObject requests per file from 2 to 1 for the footer and from 3 to 2 for a full row-group read. It makes no difference for footers larger than 64 KiB; see the question on FOOTER_READ_SIZE.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant