Skip to content

feat(rs-client): Add automatic resumable uploads - #700

Merged
lcian merged 34 commits into
mainfrom
lcian/feat/rust-automatic-resumable-uploads
Nov 3, 2026
Merged

lcian merged 34 commits into
mainfrom
lcian/feat/rust-automatic-resumable-uploads

Conversation

@lcian

@lcian lcian commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Changelog Entry

  • Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write
    • Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.
    • Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).
    • The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

Details

Session::put now uses resumable uploads automatically for eligible uploads, matching the Python client (#667). An upload is eligible if:

  • It's a buffer or a path to a regular file (put, put_path)
  • Its input size, before client compression, is at least the resumable threshold (default 256 MiB)
  • It's sent standalone. Uploads queued through many() keep their existing batch or direct routing and ignore the threshold.

Open file handles (put_file), streams, generic readers, non-regular paths and inputs of unknown size use a direct upload.
The API and behavior mirrors the Python client:

let usecase = Usecase::from_static("debug_files")
    .with_resumable_threshold_bytes(Some(256 * 1024 * 1024)); // None disables; Some(0) includes every supported payload

session.put(payload).resumable_threshold_bytes(None).send().await?; // per-request override

ClientBuilder::new(url).spool_dir("/var/tmp/objectstore"); // optional

Path and reader lifetime. Each attempt opens an independent reader. Source paths must remain readable, unchanged, and must not be renamed, removed, or replaced until the upload completes. Compressed payloads spool to named temporary files and retries reopen them. The HTTP layer may retain old readers after the upload returns or is dropped, delaying handle closure and temporary-storage reclamation.

For reviewers

  • The Error variants ResumableUploadUnavailable, UploadOffsetMismatch and the chunk-granularity variant are no longer feature-gated. Since Error is #[non_exhaustive] (feat(rs-client)!: Mark Error as non-exhaustive #706), this is additive.
  • Most of the logic is in put.rs (routing), put/payload.rs (preparation and readers) and put/automatic.rs (recovery loop).

Closes FS-561

Resume eligible buffer and file uploads from authoritative server progress, with configurable thresholds and temporary-file compression. Keep automatic uploads available without the manual protocol feature.
@codecov

This comment has been minimized.

@lcian lcian changed the title feat(rust): Add automatic resumable uploads feat(rust-client): Add automatic resumable uploads Oct 21, 2026
lcian and others added 3 commits October 21, 2026 14:48
Separate payload preparation and reader lifetime from protocol recovery. Stop and join each request reader before returning its result, report local IO errors directly, and keep progress probes in a dedicated retry loop.
Unify direct fallbacks, make sizing optional, and limit resumables to standalone uploads.
# Conflicts:
#	Cargo.lock
#	clients/rust/src/error.rs

Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
@sentry-junior sentry-junior Bot changed the title feat(rust-client): Add automatic resumable uploads feat(rs-client): Add automatic resumable uploads Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Semver Impact of This PR

This PR lands in these workspaces:

  • clients/rust: 🟡 Minor (new features)
📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


New Features

  • Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write by lcian with sentry-junior in #700
    • Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.
    • Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).
    • The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

🤖 This preview updates automatically when you update the PR.

sentry-junior Bot and others added 7 commits October 23, 2026 11:30
…ated docs

Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
Keep both test dependencies and support gzip when spooling resumable payloads.
Limit automatic resumables to buffers and stable paths.

Reopen source paths and named spool files per attempt to remove reader coordination.
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread clients/rust/src/put.rs Resolved
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread clients/rust/src/put/payload.rs Resolved
@lcian
lcian marked this pull request as ready for review October 28, 2026 16:03
@lcian
lcian requested a review from a team as a code owner October 28, 2026 16:03
@linear-code

This comment has been minimized.

Comment thread clients/rust/src/resumable/mod.rs Resolved
Comment thread clients/rust/src/resumable/api.rs Resolved
Comment thread clients/rust/src/resumable/mod.rs Resolved
Comment thread clients/rust/src/client.rs Resolved
Comment thread clients/rust/src/client.rs Resolved
Comment thread clients/rust/src/put/payload.rs Resolved
Comment thread clients/rust/src/put/payload.rs Resolved
Comment thread clients/rust/src/put/payload.rs Resolved
Comment thread clients/rust/src/put.rs Resolved
Comment thread clients/rust/src/put/automatic.rs Resolved
@lcian
lcian marked this pull request as draft October 29, 2026 12:25
@lcian
lcian marked this pull request as ready for review October 29, 2026 12:26
lcian added a commit that referenced this pull request Oct 29, 2026
Fixes the changelog preview added in #737, which fails at startup on
every PR ([run on
#700](https://github.com/getsentry/objectstore/actions/runs/40873702019):
"likely failed because of a workflow file issue").

Craft's reusable preview job requests `contents: read`, `pull-requests:
write` and `statuses: write`. A called workflow can't get more
permissions than its caller, and our caller only granted the first two.
This adds `statuses: write`, matching the old per-client preview
workflows.

The preview runs from the base branch (`pull_request_target`), so it
starts working on other PRs once this merges.

#skip-changelog

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Lorenzo Cian <17258265+lcian@users.noreply.github.com>

@jan-auer jan-auer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved provided the below comments are addressed.

Comment thread clients/rust/src/lib.rs Resolved
Comment thread clients/rust/src/lib.rs Resolved
Comment thread clients/rust/src/put.rs Resolved
Comment thread clients/rust/src/put.rs Resolved
Comment thread clients/rust/src/put.rs Resolved
Comment thread clients/rust/src/put/payload.rs Resolved
Comment thread clients/rust/src/put.rs Resolved
lcian added 2 commits October 30, 2026 12:05
Keep optional resumable exports together and move the dead-code allowance into the module.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c6b8bef. Configure here.

Comment thread clients/rust/src/put/automatic.rs Resolved
lcian added 4 commits October 30, 2026 13:48
…746)

Compressed resumable uploads now spool to anonymous temporary files and
read each retry at explicit offsets. This avoids leaving named spool
files behind after a process crash and lets an earlier HTTP request
finish reading without affecting a retry's position.

Stacked on #700. Reads run on Tokio's blocking pool in 4 KiB chunks,
using Unix positional reads or Windows `seek_read`. Each chunk is
zero-initialized and converted to `Bytes` without copying its contents.
Source-path uploads continue opening independent readers.

Regression coverage checks overlapping readers, cancellation with a
pending read, and premature EOF. Client tests pass with all features and
with default features disabled on macOS; the Windows implementation has
not been exercised on Windows.
Comment thread clients/rust/src/put.rs Resolved
@lcian
lcian enabled auto-merge (squash) November 3, 2026 09:55
@lcian
lcian merged commit dfe0a58 into main Nov 3, 2026
26 checks passed
@lcian
lcian deleted the lcian/feat/rust-automatic-resumable-uploads branch November 3, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants