Skip to content

fix(streaming): return empty bytes when reading at the end of an S3 object - #8511

Open
chancesong10 wants to merge 2 commits into
aws-powertools:developfrom
chancesong10:fix/streaming-read-at-end-of-object
Open

chancesong10 wants to merge 2 commits into
aws-powertools:developfrom
chancesong10:fix/streaming-read-at-end-of-object

Conversation

@chancesong10

Copy link
Copy Markdown

Issue number: closes #8510

Summary

Changes

_S3SeekableIO opens each stream with Range: bytes=<position>-, and S3 answers 416 InvalidRange when that position is at or past the end of the object, which includes any range on an empty object.

  • raw_stream catches ClientError with code InvalidRange and uses an empty PowertoolsStreamingBody, so reads at the end return b"" like a regular file. Any other error is still raised. This keeps one GetObject per stream, with no extra HeadObject to check the size first.
  • close() only closes _raw_stream if a stream is already open. Before, it went through the raw_stream property, so closing an unread object made a GetObject just to close it, and closing at the end of the object raised.

PowertoolsStreamingBody is now imported at runtime rather than only under TYPE_CHECKING, since it's used to build the empty body.

Tests added to tests/functional/streaming/_boto3/test_s3_seekable_io.py:

  • test_read_at_end_of_object_returns_empty_bytes
  • test_read_empty_object_returns_empty_bytes, covering read() and iteration
  • test_raw_stream_raises_other_client_errors (NoSuchKey still raises)
  • test_context_manager_at_end_of_object

test_closed no longer stubs a get_object. It now checks that closing an unread object makes no request. The three new behaviour tests and the updated test_closed fail on develop.

User experience

Before:

obj = S3Object(bucket="my-bucket", key="data.txt")
obj.seek(0, io.SEEK_END)
obj.read()        # ClientError: InvalidRange

S3Object(bucket="my-bucket", key="photos/").read()  # empty object: ClientError: InvalidRange

with S3Object(bucket="my-bucket", key="data.txt") as obj:
    obj.seek(0, io.SEEK_END)
# ClientError: InvalidRange, raised by close() on exit

After, these return b"", b"", and exit cleanly, the same as io.BytesIO. Closing an unread object no longer calls GetObject.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@chancesong10
chancesong10 requested a review from a team as a code owner October 1, 2026 06:43
@chancesong10
chancesong10 requested a review from svozza October 1, 2026 06:43
@boring-cyborg

boring-cyborg Bot commented Oct 1, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 1, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 2, 2026
@sonarqubecloud

sonarqubecloud Bot commented Oct 2, 2026

Copy link
Copy Markdown

@codecov

codecov Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.92%. Comparing base (b1af7e8) to head (164497c).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8511      +/-   ##
===========================================
+ Coverage    96.88%   96.92%   +0.04%     
===========================================
  Files          317      317              
  Lines        15899    15910      +11     
  Branches      1395     1398       +3     
===========================================
+ Hits         15403    15421      +18     
+ Misses         361      353       -8     
- Partials       135      136       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes 100-499 lines, ignoring generated files. streaming tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: S3Object raises InvalidRange when reading at the end of an object or reading an empty object

2 participants