Skip to content

Send the authorization parameters of s3_additional_kwargs with every filesystem request #1372

Closed

Description

@laughingman7743

Part of #1342 (item 1, and the copy/mv input in this comment). Completes what remains of #1333.

Decision

Every request of the filesystem sends the authorization parameters of s3_additional_kwargs: RequestPayer, ExpectedBucketOwner, and SSECustomer*. Listings are included. Each operation receives only the parameters it accepts. The other parameters (SSE for writes, ContentType, ...) keep reaching only open, pipe_file, and put_file.

Current state (29cc1d3, offline)

  1. Filesystem-wide parameters: s3_additional_kwargs reaches open, pipe_file and put_file only. It does not reach touch (s3.py:1489), cat_file (s3.py:2030), cp_file/copy/mv, setxattr, rm_file/rm, info, or the listings, sync or aio. So cat(p) fails on a requester-pays or SSE-C object where open(p).read() works.
  2. copy/mv expansion: copy("s3://b/a", "s3://b/c", RequestPayer="requester") sends HeadObject(a), HeadObject(c) and ListObjectsV2(c/) without RequestPayer, sync and aio. With the authorization names on the filesystem, the expansion and pairing requests carry them too. Per-call listing parameters stay out of scope; the reasons are in the linked comment.
  3. rm_file: drops per-call RequestPayer/ExpectedBucketOwner (s3.py:1136), which rm forwards.
  4. mkdir: reads only acl and region_name; ObjectLockEnabledForBucket and ExpectedBucketOwner are dropped (s3.py:1384).
  5. touch(Body=b"x"): writes the data although touch creates an empty object.
  6. touch(truncate=False): the existence check self.exists(path) (s3.py:1482) sends no lookup parameters and treats a prefix-only key as an object, unlike _object_exists (Keep the caller's parameters and the open-time pin in S3File #1334).

Compatibility

Rejecting Body in touch is breaking (release note). Sending the authorization names to more requests changes requests, not results, except where they previously failed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions