Unbounded memory growth: upload and download routes ignore stream backpressure (v9.7.0, v9.8.0) #274

Summary

Both proxy routes read from the faster side without waiting for the slower side, so the difference accumulates in process memory as Buffers:

• Download (GET /download/:cacheEntryId): a slow restore client makes the server read the entire object from storage into the socket write queue. N slow clients on a large entry hold roughly N × entry size.
• Upload (PUT /upload/:uploadId?blockid=…): the block body is read off the socket in full, regardless of how fast the storage adapter consumes it.

With direct downloads disabled, a handful of slow restores of one large entry is enough to OOM-kill the server. Every in-flight upload and restore then fails with connection reset by peer, and the runners' retries tend to OOM the fresh process again.

Affected: v9.7.0 and v9.8.0. The code for both paths is unchanged between them and current dev. h3 1.15.11.

Configuration

• STORAGE_DRIVER=s3 (reproduced against MinIO), DB_DRIVER=sqlite
• ENABLE_DIRECT_DOWNLOADS=false, so every restore goes through /download/:id
• Official image with its default command (node --expose-gc …), container memory limit 4 GiB

Reproduction (MinIO only)

1. Start MinIO and ghcr.io/falcondev-oss/github-actions-cache-server:9.7.0 with the configuration above and --memory 4g.
2. Save one ~1.3 GiB entry with @actions/cache and restore it once so it is merged.
3. Restore it with N concurrent clients whose download bandwidth is limited to ~15 MiB/s. That is the single-stream proxy restore rate reported in Proxy download route ignores Range; @actions/cache restores are single-stream against this server #263. We used a toxiproxy bandwidth toxic between the clients and the server.
4. Sample process.memoryUsage() inside the server.

How much is buffered depends on the client's read rate, not the concurrency. The memory is released once the slow clients finish or disconnect.

On the upload side, a single 1331 MiB save from a fast client peaked at 741 MB RSS and 489 MB arrayBuffers (baseline 217 MB / 0.1 MB). The regression test below isolates the route: while the storage adapter is busy, a 512 MiB block PUT is read off the socket in full (536,871,021 bytes). Once the route stops using getRequestWebStream, only 131,072 bytes are read.

Expected vs observed

Expected: memory per in-flight request is bounded by stream highWaterMarks and socket buffers (KiB to low MiB), and TCP flow control paces the faster side.

Observed: memory per in-flight request grows to the size of the object (download) or the block (upload).

Cause

• Download, routes/download/[cacheEntryId].ts:29 (v9.7.0): await sendStream(event, Readable.toWeb(stream) as ReadableStream). In h3 1.15.11, sendStream writes web streams through stream.pipeTo(new WritableStream({ write(chunk) { event.node.res.write(chunk) } })). It ignores write()'s return value, so the pipe never waits for drain (h3/dist/index.mjs:857-875). A Node Readable would take the .pipe() branch, which does respect backpressure.
• Upload, routes/devstoreaccount1/upload/[uploadId].put.ts:40 (v9.7.0): getRequestWebStream(event). In h3 1.15.11 this builds a ReadableStream whose start does req.on('data', chunk => controller.enqueue(chunk)). It has no pull and never pauses the request (h3/dist/index.mjs:476-508). Storage.uploadPart then wraps it in Readable.fromWeb (lib/storage.ts:227-246).

Related

• Increased memory consumption #182 and Memory requirements/leak? #135: same symptom, both closed without a root cause.
• Proxy download route ignores Range; @actions/cache restores are single-stream against this server #263: slow single-stream restores.
• Cache restore hangs forever when parts are large (64MiB) — merge abort deadlocks pumpPartsToStreams, client sees "Premature close" #247: restore hang with a different cause.
• feat(download): serve HTTP Range on the proxy route #270 replaces sendStream with stream.pipeline and fixes the download half, though it is framed as an abort/lease leak. Nothing covers the upload half yet; a PR with a regression test follows.

Proposed fix

• Download: feat(download): serve HTTP Range on the proxy route #270's pipeline(download.stream, event.node.res).
• Upload: pass event.node.req to Storage.uploadPart (typed Readable), so the adapter's reads pace the client.