feat: async chunk upload, dynamic part sizing for multi-TB uploads, and RUFH compliance improvements

.gitignore

@@ -180,3 +180,4 @@ SFTP_STORAGE_ANALYSIS.md
 NFS_LOCKING.md
 RELEASE-REVIEW.md
 ASYNC_REVIEW.md
+ASYNC_REVIEW_2.md

AGENTS.md

@@ -211,6 +211,7 @@ When updating the IETF protocol implementation for a new draft revision, follow
 ### 4. Conformity Test Suite Maintenance & Subagent Isolation
 Whenever a new draft revision of the RUFH specification is published, the repository's Python conformity test suite (`scripts/rufh_conformity_test.py`) MUST be reviewed and updated by a separate, dedicated subagent.
 - **Strict Isolation Rule**: The subagent tasked with updating `scripts/rufh_conformity_test.py` MUST ONLY consult the official IETF specification document (and RFC 9530) and MUST NOT inspect the Java server implementation code under `src/main/java/`. This ensures the conformity test suite remains an independent, unbiased specification benchmark.
+- **Default Multi-Backend Execution**: When executing the conformity test suite (`scripts/rufh_conformity_test.py`), tests MUST be run by default against all three supported storage backend types (Disk: `/test/api/upload`, S3: `/test-s3/api/upload`, and Azure Blob: `/test-azure/api/upload`) as documented in [`docs/CONFORMITY_TESTING.md`](docs/CONFORMITY_TESTING.md).
 
 ### 5. Conformity Test Suite Audit — Repeatable Procedure
 Use this procedure to audit `scripts/rufh_conformity_test.py` against the current (or a new) specification revision. The goal is to identify untested MUST/SHOULD/MAY requirements and produce an actionable improvement report.

CHANGELOG.md

@@ -7,7 +7,7 @@ All notable changes to this project will be documented in this file.
 ### New
 
 - **NFS- & SMB-Safe Lease Locking (`LeaseFileLockingService` & `LeaseFileMutex`)**: Added distributed, container-safe filesystem locking using atomic sibling mutex directories (`<UploadId>.mutex/`), in-place expired lock takeover, and TTL-based JSON lease files with background heartbeat renewal and ownership fencing. Operates reliably across multi-server replicas on NFS (v3/v4), AWS EFS, Azure Files, Windows SMB/CIFS, and local disks without requiring Redis, ZooKeeper, or OS-level `FileLock` daemons. Comprehensive guide and legacy opt-out instructions available in `docs/DISK_BASED_LOCKING.md`.
-- **S3-Compatible Storage & Distributed Locking**: Added native S3 storage support via `S3StorageService` (MinIO SDK), distributed locking via `S3LockingService` (S3 conditional writes with TTL leases and interrupt signals for multi-replica container deployments), S3-native concatenation via `S3ConcatenationService`, and complete documentation in `docs/S3_STORAGE.md`.
+- **S3-Compatible Storage & Distributed Locking**: Added native S3 storage support via `S3StorageService`, distributed locking via `S3LockingService` (S3 conditional writes with TTL leases and interrupt signals for multi-replica container deployments), S3-native concatenation via `S3ConcatenationService`, configured entirely via standard S3 connection parameters, and complete documentation in `docs/S3_STORAGE.md`.
 - **Azure Blob Storage & Distributed Leases**: Added native Azure Blob Storage support via `AzureBlobStorageService` (Block Blob staging with streaming appends, sub-threshold buffering, truncation, and deduplication), distributed locking via `AzureBlobLockingService` (Azure Blob Leases with auto-renewal, JVM interruption, cross-replica `.stop` signals, and clean shutdown), zero-copy server-side concatenation via `AzureBlobConcatenationService` (`stageBlockFromUrl`), and comprehensive documentation in `docs/AZURE_BLOB_STORAGE.md`.
 - **IETF Resumable Uploads for HTTP (RUFH) Protocol**: Implemented full support for the official IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload-12`).
   - **Dual Protocol Auto-Detection**: Added transparent protocol routing in `TusFileUploadService` supporting both legacy `TUS_1_0_0` (`Tus-Resumable: 1.0.0`) and `RUFH` (`ProtocolVersion.RUFH`) clients concurrently on the same endpoint.
@@ -16,21 +16,27 @@ All notable changes to this project will be documented in this file.
   - **Dedicated Compliance Test Suites**: Added comprehensive, spec-quoted end-to-end tests using a dedicated Python script `scripts/rufh_conformity_test.py` with documentation on how to run the tests in `docs/CONFORMITY_TESTING.md`.
   - **User Migration & Interim Responses Documentation**: Added `docs/MIGRATION.md` and `docs/INTERIM_RESPONSES.md` detailing migration strategies, HTTP 104 status frames under IETF RUFH, Tomcat/Servlet container limitations, cached reflection optimizations, and Spring Boot Tomcat Valve integration.
 - **Server-Side Upload Completion Listeners (`UploadCompletionListener`)**: Added a functional interface callback mechanism allowing developers to register post-upload listeners via `withUploadCompletionListener(UploadCompletionListener)` or `addUploadCompletionListener(UploadCompletionListener)`. Listeners receive the completed `UploadInfo` and `TusFileUploadService` instance after lock release, allowing immediate byte streaming and deletion without contention. Added helper overloads `TusFileUploadService.getUploadedBytes(UploadInfo)` and `TusFileUploadService.deleteUpload(UploadInfo)`.
+- **Unified Lock Wait & Cloud Drain Timeout**: Added `TusFileUploadService.withLockWaitTimeout(Duration)` (default 60 seconds) to configure the maximum wait duration for requests contending for an active upload lock. Automatically derives the background cloud upload chunk drain timeout (`lockWaitTimeout - 5 seconds`, default 55s) and the retry polling budget (`lockWaitTimeout / 200ms`, default 300 retries).
 - **JSON Serialization**: Support storing `UploadInfo` objects as JSON files in the storage backend using `TusFileUploadService.withJsonSerialization(true)`.
 
 ### Changed
 - **Default Disk-Based Locking**: `TusFileUploadService.withStoragePath(String)` now defaults to `LeaseFileLockingService` instead of `DiskLockingService` for out-of-the-box Kubernetes, container, and shared network storage compatibility. See `docs/DISK_BASED_LOCKING.md` for legacy opt-out instructions.
-- **Calibrated Retry Budget**: Extended `TusFileUploadService` lock acquisition retry budget to 8.0 seconds (40 retries x 200ms) to ensure reliable contention resolution over network storage.
+- **Unified Lock Wait Budget & Cloud Chunk Drain Calibration**: Replaced unreleased `withMaxLockRetries` with `withLockWaitTimeout(Duration)` (default 60s / 300 retries), synchronizing the lock wait timeout budget across storage backends and automatically deriving the cloud chunk drain timeout (`lockWaitTimeout - 5s`, default 55s) to guarantee in-flight chunks are cleanly flushed to cloud storage before releasing upload locks.
 - **Absolute Base URL & Location Header Support**: Extended `withUploadUri(String)` to accept absolute base URLs (e.g. `https://upload.example.com/files`), returning full URLs in `Location` response headers for upload creation across both Tus 1.0.0 and RUFH protocols while preserving backward compatibility for relative paths.
 - **`process()` Return Value (`UploadInfo`)**: `TusFileUploadService.process(...)` now returns the created or updated `UploadInfo` instance (or `null` on errors or `OPTIONS` preflight requests), enabling applications to track and store upload IDs directly into user sessions or database repositories.
 - **Jackson Bundled in Compile Scope**: Promoted Jackson dependencies (`jackson-databind`, `jackson-annotations`, `jackson-core`) to `compile` scope, eliminating `NoClassDefFoundError` when enabling JSON serialization or using cloud storage.
 - **Lock-Holding Stream Lifecycle**: `TusFileUploadService.getUploadedBytes(...)` now retains the upload lock until the returned stream is closed, preventing concurrent modifications or deletions from corrupting data mid-stream.
+- **S3 Locking Optimization & Non-CAS Fallback**: Optimized `S3LockingService` and `S3UploadLock` by skipping the redundant `isLockExpired` pre-check on happy-path acquisitions (saving 1 remote GET round-trip) and using S3 Multi-Object Delete to delete `.lock` and `.stop` objects in a single batch round-trip on lock release. Made read-after-write verification jitter bounds configurable via `AbstractLeaseLockingService.setJitter(minMs, maxMs)` and `S3LockingService.withJitter(minMs, maxMs)`. Added automatic non-CAS fallback when `If-None-Match: *` returns `HTTP 501 Not Implemented` (enabling seamless out-of-the-box support for Backblaze B2 and Ceph RGW) along with `withS3ConditionalWritesSupported(boolean)` override. Streamlined `S3LockingService` constructors to minimal required set and documented zero-jitter throughput optimization for pure AWS S3 / Cloudflare R2 deployments in `docs/S3_STORAGE.md`.
 
 ### Fixed
 - **RFC 9110 Media Type Matching & MIME Parameter Tolerance**: Enhanced `Content-Type` header validation in `ContentTypeValidator`, `PostContentTypeValidator`, and `RufhAppendValidator` using RFC 9110 §8.3 compliant media-type parsing (`Utils.isMediaType`). Media type matching now tolerates MIME parameters (such as `;charset=UTF-8` automatically appended by Spring Boot `CharacterEncodingFilter`, servlet wrappers, proxies, or HTTP clients), whitespace variations, and case-insensitivity without incorrectly rejecting valid requests with `406 Not Acceptable`.
 - **Clear Content-Length on Error Responses**: Cleared `Content-Length` response header prior to invoking `HttpServletResponse.sendError(...)` during exception handling, resolving buffer conflicts and exceptions in Undertow and other servlet containers ([#40](https://github.com/tomdesair/tus-java-server/issues/40)).
 - **Prevent Disk Truncate Underflow**: Guarded file truncate logic in `DiskStorageService` against underflow when removing bytes (`Math.max(0L, file.size() - byteCount)`).
 - **File Channel Leak Prevention in FileBasedLock**: Guaranteed `FileChannel` is closed immediately upon lock acquisition errors to avoid file descriptor leaks.
+- **Cloud Upload Pause & Drain Timeout Resilience**: Integrated `AsyncChunkUploader` default drain timeout (55 seconds, calibrated to `lockWaitTimeout - 5s`) with error-resilient recovery logic across `AzureBlobStorageService` and `S3StorageService`. Confirmed uploaded chunks and staged Azure blocks are committed and metadata (`UploadInfo` offset) is persisted to storage before throwing stream or drain exceptions, ensuring paused uploads (such as Uppy client pause/resume) preserve their progress and resume from the exact byte offset instead of restarting from 0.
+- **Azure Blob Storage Upload Cancellation Lease Safety**: Checked lock blob lease state before calling `deleteIfExists()` in `AzureBlobStorageService.terminateUpload()`. When an active lease is held on the lock blob by an ongoing request (e.g. `DELETE` cancellation), attempting deletion without specifying the lease ID triggered an Azure SDK error log (`HTTP 412 LeaseIdMissing`). Skipping deletion for actively leased blobs prevents this error and allows normal lease release upon request completion.
+- **Lock Contention Stream Interruption Handling**: Handled `IOException` from `InterruptibleInputStream` across upload request handlers (`CorePatchRequestHandler`, `RufhAppendPatchRequestHandler`, `RufhCreationPostRequestHandler`, `CreationWithUploadPostRequestHandler`). When an upload stream is interrupted by the locking service watchdog during lock contention (such as concurrent `HEAD` progress checks or `DELETE` cancellation requests), the storage backend commits all buffered bytes received so far and updates the offset in storage. Request handlers now reload the refreshed `UploadInfo` and return a clean HTTP 204/201 response with the updated offset rather than bubbling an unhandled `IOException` / HTTP 500 to the servlet container.
+- **S3 Server-Side Part Composition & Native Multipart Copy**: Implemented `S3ServerSideComposeHelper` to resolve AWS S3 header rejection during server-side part composition. MinIO Java SDK 9.0.3's `composeObject` delegates to `UploadPartCopy` with an empty body placeholder that automatically injects `Content-MD5` and `Content-Type` headers, which Amazon AWS S3 strictly forbids on part copy requests and rejects with HTTP 400 (`The specified header is not valid in this context`). `S3ServerSideComposeHelper` executes native S3 multipart copy requests directly with SigV4 signing omitting `Content-MD5`, allowing fast zero-bandwidth server-side composition on both AWS S3 and MinIO/Ceph clusters without external AWS SDK dependencies. Added reflection-free constructors to `S3StorageService` and `S3LockingService` accepting connection parameters directly, with `eu-central-1` as the default fallback region, while preserving multi-tier streaming concatenation fallbacks.
 
 ### Breaking
 - **Downloads**: In order to support both the Tus protocol and RUFH protocol, the unofficial download extension will not return a HTTP status code `204` for uploads that are still in progress and will not contain the response header `Tus-Resumable`. Removed the `UploadInProgressException` class.

README.md

@@ -181,7 +181,7 @@ After creating the object, you can configure it using the following methods:
 | `withStoragePath(String)` | `${java.io.tmpdir}/tus` | Path on the filesystem or shared drive where uploaded bytes and metadata are stored when using `DiskStorageService`. |
 | `withSupportedProtocolVersions(ProtocolVersion)` | `ProtocolVersion.AUTO` | Configures protocol handling: `AUTO` (header-based auto-detection), `TUS_1_0_0` (Tus 1.0.0 only), or `RUFH` (IETF draft-12 only). |
 | `withMaxUploadSize(Long)` | `Long.MAX_VALUE` | Maximum allowed total upload size in bytes per upload resource. |
-| `withMaxLockRetries(int)` | `40` | Maximum lock acquisition retries during lock contention resolution (200ms sleep, resulting in an 8.0s timeout budget). |
+| `withLockWaitTimeout(Duration)` | `Duration.ofSeconds(60)` | Maximum duration a request waits to acquire an upload lock held by an in-flight transfer (retrying every 200ms, resulting in 300 retries). Automatically configures cloud background chunk drain timeout to 5s less than this value (default 55s). |
 | `withChunkedTransferDecoding(Boolean)` | `false` | Enables manual chunked HTTP decoding for servlet containers that do not decode chunked requests natively. |
 | `withThreadLocalCache(Boolean)` | `false` | Enables in-memory thread-local caching of upload request data to reduce storage backend I/O load. |
 | `withUploadExpirationPeriod(Long)` | `null` (disabled) | Expiration period in milliseconds after which incomplete/expired uploads become eligible for cleanup. |
@@ -191,11 +191,12 @@ After creating the object, you can configure it using the following methods:
 | `disableTusExtension(String)` | None | Disables a built-in extension (`creation`, `checksum`, `expiration`, `concatenation`, `termination`, `download`, `cors`). |
 | `withUploadIdFactory(UploadIdFactory)` | `UuidUploadIdFactory` | Custom ID generator for upload resources (e.g., `UuidUploadIdFactory` or `TimeBasedUploadIdFactory`). |
 | `withUploadCompletionListener(UploadCompletionListener)` | None | Registers a callback invoked immediately when an upload finishes transferring all bytes and is completed. |
+| `withCloudUploadThreadPoolSize(int)` | `10` | Maximum number of worker threads used for asynchronous background chunk uploading in cloud storage backends (S3, Azure). |
 | `withJsonSerialization()` | Java serialization | Enables JSON serialization for upload metadata (`UploadInfo`). Jackson is bundled by default. |
 | `withUploadStorageService(UploadStorageService)` | `DiskStorageService` | Configures custom or cloud storage backend (`DiskStorageService`, `S3StorageService`, `AzureBlobStorageService`). |
 | `withUploadLockingService(UploadLockingService)` | `LeaseFileLockingService` | Configures custom or cloud locking backend (`LeaseFileLockingService`, `S3LockingService`, `AzureBlobLockingService`). |
 
-The library provides filesystem-based storage (`DiskStorageService` / `LeaseFileLockingService`), S3-compatible object storage (`S3StorageService` / `S3LockingService`), and Azure Blob Storage (`AzureBlobStorageService` / `AzureBlobLockingService`). See the **[Disk & Network Storage Locking Guide](docs/DISK_BASED_LOCKING.md)**, **[S3 Storage Guide](docs/S3_STORAGE.md)**, and **[Azure Blob Storage Guide](docs/AZURE_BLOB_STORAGE.md)** for detailed instructions on multi-replica container deployments in Kubernetes, post-upload processing, and legacy locking opt-out.
+The library provides filesystem-based storage (`DiskStorageService` / `LeaseFileLockingService`), S3-compatible object storage (`S3StorageService` / `S3LockingService`), and Azure Blob Storage (`AzureBlobStorageService` / `AzureBlobLockingService`). Cloud backends feature an asynchronous 3-slot chunk pipeline (`AsyncChunkUploader`) that overlaps client payload streaming with cloud staging in the background. See the **[Disk & Network Storage Locking Guide](docs/DISK_BASED_LOCKING.md)**, **[S3 Storage Guide](docs/S3_STORAGE.md)**, and **[Azure Blob Storage Guide](docs/AZURE_BLOB_STORAGE.md)** for detailed instructions on multi-replica container deployments in Kubernetes, post-upload processing, and legacy locking opt-out.
 
 ### 2. Receiving a Resumable Upload
 To process an upload request you have to pass the current `jakarta.servlet.http.HttpServletRequest` and `jakarta.servlet.http.HttpServletResponse` objects to the `me.desair.tus.server.TusFileUploadService.process()` method. Typical places were you can do this are inside Servlets, Filters or REST API Controllers.
@@ -409,10 +410,10 @@ public TomcatServletWebServerFactory tomcatFactory(TusFileUploadService tusFileU
 
 ## Compatible Client Implementations & Conformity Testing
 This server implementation has been tested with:
-- **Tus 1.0.0 Clients**: Tested with [Uppy](https://uppy.io/) and `tus-js-client`.
-- **IETF Resumable Uploads Clients & Conformity Tests**: The implementation has been thoroughly tested with our own built-in RUFH conformity test suite (`scripts/rufh_conformity_test.py`) validating compliance with draft-12 of the RUFH protocol specification and RFC 9530 HTTP Digests, as well as the community [RUFH conformity tests from the IETF hackathon](https://github.com/tus/ietf-hackathon).
+- **Tus 1.0.0 Clients & Conformity Tests**: Tested with [Uppy](https://uppy.io/), `tus-js-client`, and our built-in Tus v1.0.0 conformity test suite (`scripts/tus_conformity_test.py`) validating all core protocol mechanisms and protocol extensions (creation, creation-with-upload, checksum, termination, concatenation, and expiration) across Disk, S3, and Azure Blob backends.
+- **IETF Resumable Uploads Clients & Conformity Tests**: The implementation has been thoroughly tested with our built-in RUFH conformity test suite (`scripts/rufh_conformity_test.py`) validating compliance with draft-12 of the RUFH protocol specification and RFC 9530 HTTP Digests, as well as the community [RUFH conformity tests from the IETF hackathon](https://github.com/tus/ietf-hackathon).
 
-For detailed instructions on running our native conformity test suite and interpreting results, see the **[Conformity Testing Guide (docs/CONFORMITY_TESTING.md)](docs/CONFORMITY_TESTING.md)**.
+For detailed instructions on running our native conformity test suites across all storage backends (Disk, S3, Azure Blob) and interpreting results, see the **[Conformity Testing Guide (docs/CONFORMITY_TESTING.md)](docs/CONFORMITY_TESTING.md)**.
 
 This repository also contains comprehensive automated integration test suites (`ITTusFileUploadService`, `RufhProtocolCreationTest`, `RufhProtocolAppendTest`, `RufhProtocolHeadTest`, `RufhProtocolCancellationTest`) validating both protocol specifications.