Benchmark the S3 filesystem against s3fs and decide on a standalone package #1315

Summary

Decide, from measurements, whether PyAthena's S3 filesystem (S3FileSystem and AioS3FileSystem) has an advantage over s3fs that justifies offering it as an s3fs alternative, and whether it should then be developed and published as a standalone package that PyAthena depends on.

Background

The filesystem is built on the synchronous boto3 client and thread pools; s3fs is built on aiobotocore. The two differ in three ways that matter to users:

1. Dependencies. s3fs 2026.9.0 requires aiobotocore>=2.19,<4 and pins fsspec to its own version (>=2026.9.0,<2026.9.1). aiobotocore 3.10.0 requires botocore>=1.43.115,<1.43.118, while the latest boto3 is 1.43.118. PyAthena requires only boto3>=1.43.31, botocore>=1.43.31, fsspec>=2026.9.0, and tenacity. The filesystem will not depend on aiobotocore (Separate an S3 core from the fsspec adapter in pyathena.filesystem #1053).
2. Concurrency model. One PyAthena file reads its ranges and uploads its parts in parallel through an S3Executor sized by max_workers; the aio filesystem runs the blocking boto3 calls in its own thread pools (Size the asyncio filesystem's parallel requests by max_workers instead of the default executor #1159). s3fs runs every request as a coroutine on fsspec's IO loop, so batch operations over many objects (cat, get, put, rm, copy) run with batch_size concurrency. The synchronous S3FileSystem inherits fsspec's sequential loops for these batch operations (copy and get in pyathena/filesystem/s3.py fall through to AbstractFileSystem), and AioS3FileSystem gets its batch concurrency from AsyncFileSystem with threads.
3. Design. Typed models (S3Object, S3Core, S3Path, S3PathPairing), per-operation request parameters, error translation, and the write session (Give multipart writes a single owner of their parts and an explicit state #1211) are PyAthena's own; the s3fs API surface is kept for migration (Improve s3fs / fsspec API parity for the built-in S3FileSystem #722).

The filesystem suite of the benchmark project measures one-object reads, writes, and copies through the two PyAthena filesystems only. The 4.2.0 run found the aio paths at the sync throughput for writes and copies, but nothing in the history compares either with s3fs, and the suite has no scenario with many objects, where s3fs's coroutine concurrency should matter most.

Plan

1. Add s3fs to the filesystem suite as a third API, in its own benchmark environment because aiobotocore pins botocore: Add s3fs as an API of the filesystem benchmark suite #1316.
2. Add many-object scenarios to the suite: batch reads, writes, deletes, listing, and metadata fan-out over hundreds of small objects: Add many-object scenarios to the filesystem benchmark suite #1317.
3. Measure the extended suite on one c8gn.2xlarge Spot host and record a history entry with the comparison and a recommendation: Measure the filesystem suite against s3fs on c8gn.2xlarge #1318.
4. Decide from the results. Expected outcomes, to be confirmed or refuted by step 3:
   • One large object: PyAthena's parallel range reads should match or beat s3fs's single-stream open().read(); writes and server-side copies should be close, since both are multipart.
   • Many small objects: s3fs is likely ahead for the synchronous filesystem, which runs its batch operations sequentially, and may be ahead of the aio filesystem at high concurrency, where threads and the connection pool cap PyAthena. If so, parallel batch operations for S3FileSystem are the first improvement to file.
   • Process cost: peak RSS, thread count, and CPU time per transferred byte for each implementation.
5. Standalone package, only if step 4 finds an advantage worth offering to users without Athena. The design question is filed separately after the measurement; the inventory below is what it has to resolve.

Coupling inventory for step 5

pyathena.filesystem is 10,971 lines in 11 modules with 374 commits since 2022-09. Its imports from the rest of PyAthena:

PyAthena modules that import the filesystem: the pandas, Polars, Arrow, and S3FS result sets, pyathena.aio.s3fs.cursor, and the pyathena.pandas / pyathena.polars packages, which register the s3 and s3a protocols. A standalone package would register through fsspec's fsspec.specs entry point instead, and pyathena.filesystem would stay as a compatibility re-export for 4.x.

fsspec ships no benchmark (pytest-benchmark is a test dependency of fsspec without a test that uses it), and s3fs none either, so the comparison has to come from the suite here. fsspec does ship the conformance tests fsspec.tests.abstract (copy, get, put, mv, open, pipe), which s3fs runs in s3fs/tests/derived; running them against PyAthena's filesystem is a precondition of step 5, since an s3fs alternative is judged by that contract.

A uv workspace member in this repository (as benchmarks/ is) published as its own distribution would keep the AWS test infrastructure, CI, and docs shared; a separate repository is a later option if the release cadence diverges.

Decision criteria

• A measured throughput, memory, or CPU advantage in at least one common scenario, or parity with the dependency advantage above, supports the alternative.
• Parity everywhere with higher process cost means the filesystem stays an internal part of PyAthena, and step 5 is not filed.