Skip to content

Consolidate the S3 response models and settle the core results and parameter precedence #1383

Open

Description

@laughingman7743

Part of #1341 (decisions 3 to 6 of the audit).

Current state (29cc1d3)

  1. Response models in two styles and two modules: frozen dataclasses in s3_core.py, property classes over raw dicts in s3_object.py. Checksum properties: 10 on parts, 4 on S3PutObject/S3CompleteMultipartUpload, none on S3Metadata. S3ObjectVersion.name is bucket/key.
  2. S3Metadata is falsy without user metadata and compares equal to {}, the documented mapping behavior of fs.metadata().
  3. copy_object, delete_object and replace_object_metadata return None.
  4. touch() returns S3PutObject.to_dict(), keyed by private attribute names (_etag, ...). s3fs returns the raw PutObject response.
  5. S3Core methods follow four **params precedence rules: sent as given with TypeError on a duplicate; the method's fields win (get_object, put_object, the multipart methods, list_object_annotations); params win over retained fields but TypeError for the fields the copy sets (replace_object_metadata); params win over the path (generate_presigned_url).

Decision

  1. One module of models in one style, re-exported under the current names, with constructors that still accept response dictionaries. Checksum properties per operation as botocore defines them. S3ObjectVersion gains a typed path; its name is unchanged. Pickles of the old property classes need not load (release note).
  2. S3Metadata stays the result of S3Core.head_object and fs.metadata(); its truth value and equality are documented.
  3. copy_object and replace_object_metadata return an S3CopyObject (destination version ID, source version ID, ETag, last modified, checksums); delete_object returns an S3DeleteObject (version ID, delete marker).
  4. touch() returns the raw PutObject response, as s3fs (breaking).
  5. One **params rule: fields that a method derives from its arguments raise TypeError when also given in params; defaults that the method fills in (retained metadata, request_kwargs) yield to params. The PR tabulates each method's derived fields (including CopySourceRange, MultipartUpload, checksum and pagination fields), keeps documented fallbacks such as range_=None with Range=, and normalizes the writer's internal calls (s3_writer.py:219) that forward checksum parameters (breaking where a method's field now raises or a parameter no longer wins).

The compatible consolidation (1-3) can land first; 4 and 5 go in the same release. Coordinate with #1373, which changes the append precedence over S3Object.to_api_repr().

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions