Repository navigation
Pass the source's SSE-C and RequestPayer parameters to the lookups of copies and setxattr #1333
Description
Part of #1327 (defect 1 of the audit report).
Summary
Copies and metadata replacement look up the source object without the authorization parameters that the call carries, so an SSE-C source, or a requester-pays source with a per-call RequestPayer, cannot be copied: the lookup fails before CopyObject is sent.
Scenario
fs.cp_file(src, dst,
CopySourceSSECustomerAlgorithm="AES256", CopySourceSSECustomerKey=key,
SSECustomerAlgorithm="AES256", SSECustomerKey=key)Expected: the copy succeeds. S3Core already maps these parameters to the source lookups (S3Core._copy_source_params), and plan_multipart_copy uses that mapping.
Actual: HeadObject is sent with only Bucket and Key; S3 answers 400 for an SSE-C object, which surfaces as OSError(EINVAL). Whether it fails depends on the cache: a cached parent listing answers info() without a request.
Affected paths (at 766d79d)
S3FileSystem._copy_file:info1 = self.info(path1)(s3.py:1604), and the same inAioS3FileSystem._copy_file(s3_async.py:576).S3FileSystem._move_pairs:self._head_object(source)(s3.py:1440) for the conflict check ofmv.mv's delete of the copied sources receives none of the call's parameters (s3.py:1398).setxattr:head = self.metadata(path)(s3.py:2132) without the lookup parameters ofcopy_kwargs.
The existing test test_copy_object_with_multipart_upload_sse_c_source calls _copy_object_with_multipart_upload directly and skips the lookup.
Fix direction
Make the source-lookup mapping a public S3Core method next to the destination-side operation_params, and pass its result to every source lookup above, in both filesystems; pass DeleteObjects-accepted parameters to mv's delete. Reproduced offline with botocore's Stubber: the HeadObject request of cp_file, setxattr, and cp_file(..., RequestPayer="requester") carried only Bucket and Key.
Activity
laughingman7743 commentedon Nov 6, 2026
Re-checked at 2a60ecc with an offline recording client: all four paths still send the source lookups with only
BucketandKey(cp_filewithCopySourceSSECustomer*orRequestPayer,mv's conflict HeadObject and its DeleteObjects,setxattrwith SSE-Ccopy_kwargs).Plan (PR-A; #1334 follows as a stacked PR-B)
S3Core._copy_source_paramsbecomes the publicS3Core.copy_source_params, next tooperation_params._copy_file(syncinfo()and aio_info()), the conflict lookup of_move_pairs, and the HeadObject ofsetxattrsend the lookup parameters of that mapping (RequestPayer,ExpectedBucketOwnerfromExpectedSourceBucketOwner,SSECustomer*fromCopySourceSSECustomer*).mv's DeleteObjects of the copied sources receives the parameters of that mapping that DeleteObjects accepts (RequestPayer, andExpectedBucketOwnerfromExpectedSourceBucketOwner), in both filesystems.Note that the path expansion of copy and mv sends no source parameters (
Order the aio move test and name every expansion request in the docs (G…
Send the copy parameters to the expansion and pairing of copy and mv (G…
Revert the parameters of the expansion and pairing of copy and mv (GH…
laughingman7743 commentedon Nov 7, 2026
Scope update: #1362 fixes the source lookups of
cp_file/copy/mv/setxattr,mv's conflict lookup, andmv's deletion.The requests that
copyandmvwith string paths send to expand and pair their paths still send no parameters. Acopy/mvfrom an SSE-C or requester-pays source can therefore still fail there; lists of both sources and destinations avoid those requests. An attempt to forward the parameters there was reverted after review, and the design moves to #1342 (input).#1362 therefore says "Part of #1333", and this issue stays open until that part is resolved.
Merge pull request #1362 from pyathena-dev/fix/1333-copy-source-lookups
laughingman7743 commentedon Nov 7, 2026
The remaining part of this issue moves to #1372. That covers the expansion and pairing requests of
copy/mv, which send no source parameters. #1372 makes every request send the authorization parameters ofs3_additional_kwargs, filtered per operation, listings included. #1362 fixed the source lookups.