Skip to content

Keep the identity of listed keys through traversal, fan-outs, and invalidation #1381

Open

Description

@laughingman7743

Part of #1341 (decisions 7, 8 and the key="" part of 10 of the audit). Defect 23 of #1327 and the items deferred from #1370.

Start after #1344 merges, since both rewrite the pairing and the fan-outs of rm, chmod and copy/mv.

Current state (29cc1d3)

  1. Listed names are re-parsed as paths. With the literal keys a/x and a/x?versionId=null in an unversioned bucket, find("bucket/a") names the second bucket/a/x?versionId=null, and:
    • chmod(recursive=True) updates a/x twice (once with VersionId=null) and never the literal key;
    • rm(recursive=True) deletes a/x and a/x version null, and leaves the literal key;
    • copy(recursive=True) raises ValueError: Cannot copy to a versioned file. after copying a/x;
    • mv(recursive=True) raises ValueError: Cannot move several paths to the same destination. before copying;
    • fsspec's du counts a/x once.
  2. Traversal and invalidation. With the key a/dir?versionId=v1/z, find("bucket/a") lists it but find("bucket/a", maxdepth=2) does not: _find_levels (s3.py:930) re-parses the directory name. _invalidate_path (s3.py:2594) can take part of such a key for a version and skip the parent's listing.
  3. bucket//. S3Path.parse("b//") is the key / with is_bucket True. find and recursive chmod list the bucket, and the recursive chmod sends no PutBucketAcl; rm_file("bucket//") deletes the key /.
  4. key="". S3Path("b", "") prints b and re-parses with key None.

Decision

  1. Filesystem operations carry the bucket, key and version ID of the listed entries through traversal, pairing, the fan-outs, lookups and invalidation, as a private sequence of typed paths; they never re-parse a listed name. Public find() and expand_path() still return strings.
  2. A key that looks like a version path cannot be addressed by a string path. This stays a documented limit, as in s3fs; S3Core with an explicit S3Path reaches it. No escape syntax.
  3. bucket// keeps its current meaning (the key / for single-object operations, the bucket for listings), documented, including that a recursive chmod of it does not change the bucket ACL.
  4. S3Path(bucket, "") normalizes the key to None.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions