Skip to content

Own the listing cache key scheme and invalidation in one cache object #1384

Open

Description

@laughingman7743

Part of #1341 (decision 11 of the audit).

Current state (29cc1d3)

  1. The listing cache keys are spread over about nine methods: the path (HeadObject/HeadBucket results), "" (buckets), (path, "/") and (path, "") (listings), and (path, "lookups") (lookups with parameters).
  2. _invalidate_path (s3.py:2586) evicts four keys for each of the four version-query spellings of each ancestor.
  3. AioS3FileSystem shares the sync dircache (s3_async.py:161) and delegates invalidation.
  4. 81 references to dircache in tests/pyathena/filesystem; Audit the filesystem tests changed since the #1239 audit #1328 left these rewrites to Settle the response models, S3Object, and the listing cache #1341.

Decision

One S3ListingCache over fs.dircache, shared by both filesystems, owns the key scheme, the lookups with parameters, eviction and invalidation. fs.dircache stays fsspec's DirCache, so use_listings_cache, listings_expiry_time, max_paths and empty-cache pickling keep working, and the dircache keys stay private.

It keeps today's behavior: the four version-query spellings, explicit version_id, trailing-slash and empty-segment variants, eviction of the bucket listing only when it lists the bucket, uncached null versions, no caching for prefix/token listings, per-parameter lookups with their own timestamps and hashed SSE-C keys, and the _ls_from_cache override.

Tests that read dircache keys are rewritten in the PR, keeping their request, expiry and invalidation evidence. The interface is settled before #1340.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions