You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix the filesystem's integration with PolarsCursor, the fsspec registry, pickling, and connection= #1339
Four defects where the filesystem meets the rest of PyAthena and fsspec's registry, reproduced offline at 766d79d.
Defects
PolarsCursor's whole-CSV read uses the fsspec registry. It opens the result through fsspec.open(self.output_location, "rb", connection=..., max_workers=..., ...) (polars/result_set.py:608-623, :874). After the re-registration that docs/filesystem.md recommends for using another s3 implementation (fsspec.register_implementation("s3", s3fs.S3FileSystem, clobber=True)), that class receives connection and the other options as unknown storage options, and s3fs ignores them, so the read uses ambient credentials and region instead of the connection's. docs/polars.md says this read "uses PyAthena's S3FileSystem". The pandas result set builds S3FileSystem directly.
fsspec.filesystem("s3", asynchronous=True) returns the synchronous S3FileSystem after import pyathena.pandas or pyathena.polars, which register only that class (filesystem/__init__.py:36-44). It accepts the flag through **kwargs and stores it in storage_options; a caller expecting an async filesystem fails later with AttributeError or blocks.
S3FileSystem(connection=...) and its read files cannot be pickled: fsspec stores every constructor argument in storage_options, including the Connection, whose boto3 clients do not pickle (AttributeError: Can't get local object 'lazy_call.<locals>._handler'). Dask, multiprocessing and Ray ship filesystems and open files this way; a filesystem built from s3fs-style arguments pickles. This is not documented.
connection= silently ignores the client arguments.S3FileSystem(connection=conn, endpoint_url=..., config_kwargs={"max_pool_connections": 64}, anon=True) uses the connection's client, endpoint and pool of 10 (s3.py:217-235); only requester_pays is honored. The docstring hints at this, docs/filesystem.md does not.
Fix direction
Build S3FileSystem directly in the Polars whole-CSV read unless the user gave storage_options, and correct docs/polars.md; decide whether the registration covers only the synchronous class (and document it) or rejects asynchronous=True; make connection= filesystems picklable (for example by reducing to the connection's picklable arguments) or document the limit; raise TypeError when connection= is combined with client-building arguments.
Item 1: the Polars whole-CSV read builds S3FileSystem directly unless storage_options is given; docs/polars.md is corrected.
Item 2: S3FileSystem rejects asynchronous=True with ValueError pointing to AioS3FileSystem; the docs state that the registration covers the synchronous class only.
Activity
laughingman7743 commentedon Nov 6, 2026
Decisions (2026-11-07):
S3FileSystemdirectly unlessstorage_optionsis given;docs/polars.mdis corrected.S3FileSystemrejectsasynchronous=TruewithValueErrorpointing toAioS3FileSystem; the docs state that the registration covers the synchronous class only.connection=withclient=/retry_config=injection. This issue is closed by the PR for items 1 and 2.Read whole Polars CSV results with PyAthena's filesystem (GH-1339)
Document that the registered S3FileSystem ignores asynchronous=True (G…
Read whole Polars CSV results with PyAthena's filesystem (GH-1339)
Document that the registered S3FileSystem ignores asynchronous=True (G…
Correct the premise in the Polars registry test docstring (GH-1339)
Inject the filesystem's S3 client with client= and retry_config= (GH-…
Inject the filesystem's S3 client with client= and retry_config= (GH-…