Skip to content

Honor a default source's expectedLength in TypeScript streams - #8050

Merged
jasnell merged 7 commits into
mainfrom
jasnell/ts-streams-default-expected-length
Oct 28, 2026
Merged

jasnell merged 7 commits into
mainfrom
jasnell/ts-streams-default-expected-length

Conversation

@jasnell

@jasnell jasnell commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Under typescript_implemented_streams, the default controller ignored
the expectedLength its underlying source declared: only byte sources,
a TransformStream's readable (through an internal setter) and native
streams reported a length. A body built from a default source that
declared its length was therefore sent with chunked encoding, where
the C++ implementation sends a Content-Length.

The default controller now caches the converted source's
expectedLength, as the byte controller does, and TransformStream
passes its transformer's through the source it builds for its
readable, which retires the internal setter.

Body consumption already treated a TransformStream readable's
declaration as an exact total (readable ledger #21); that now covers
any default source that declares one. A source that delivers more
than it declared rejects text(), arrayBuffer() and the other
consumers with a RangeError and is cancelled with it, where the C++
implementation returns everything. The divergence is intentional:
the declaration is the exact-total contract that byte and native
sources already enforce.

typescript_implemented_streams is still experimental, with no enable
date, so the change needs no compatibility flag.

Fixes: #8040

@jasnell
jasnell requested review from a team and guybedford October 27, 2026 23:03
@jasnell
jasnell requested review from a team as code owners October 27, 2026 23:03
@clanky-the-clanker

clanky-the-clanker Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

💬 Approved with comments — The approver's follow-ups (underflow enforcement and the draining reader's length after a read) are fixed. The only remaining finding is an INFO about a comment that overstates how closely the C++ implementation matches on underflow.

Findings: 1 info

Not opened by the correctness review: 1 changed file it was meant to read
  • src/per_isolate/webstreams/types.d.ts

The decision is advisory: this comment never approves, blocks or requests changes on the pull request.


📝 Documentation: 💬 1 info

One comment overstates how closely the C++ implementation matches on underflow.

Recommendation: 💬 Optional Nits

  • [INFO] The comment says a declared-length breach cancels the stream "as the C++ AllReader does", but for underflow the C++ implementation returns the short body (see the expectedLengthUnderflow test). Limit the comparison to overflow. (src/per_isolate/webstreams/readable.ts:4584)
View full review

The comment at src/per_isolate/webstreams/readable.ts:4581-4585 says a declared-length breach names its cause and cancels the stream "as the C++ AllReader does". That holds for overflow but not for underflow: the C++ implementation returns the short body, as the new expectedLengthUnderflow test records. Limit the comparison to overflow, or remove it.

The earlier INFO about the #expectedLength comment is resolved, now that consumption enforces both overflow and underflow.


✅ All other checks passed

🔒 Security: ✅ LGTM

Body consumption now rejects a body that delivers more or fewer bytes than it declared, and cancels the stream. A partly read stream, or a fork made after a read, reports no length, so a body cannot advertise a Content-Length larger than what is left to send. The internal setControllerExpectedLength export is removed.


📖 AGENTS.md: ✅ LGTM

The AGENTS.md ledger updates match the tests and configuration changes.


📋 Code Quality: ✅ LGTM

  • Underflow: collectChunks now rejects a stream that closes short of its declared expectedLength and cancels it, matching the overflow check.
  • Draining reader: its expectedLength getter uses getReadableStreamExpectedLength, so it agrees with the C++ bridge after a read or a fork.
  • Native streams: they remain exempt from the read/fork guard.
  • Writable sink and TransformStream: both changes behave as before.

⚡ Performance: ✅ LGTM

The default controller reads expectedLength once and caches it. The underflow check runs once, at end of stream. The writable controller no longer allocates three closures per instance.


Reviewed commit: 30509c11 — same changes as 57dd4310, so this is that commit's review (no new review was run; line numbers are as of 57dd4310)

@ask-bonk

ask-bonk Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Since last review: 0 resolved, 0 still open, 0 new.
LGTM!

Not re-run: api-compat, tests, docs, compat-flags, design-simplicity, jsg-gc, kj-style, memory-safety (no author changes in their files since the last review)

Specialist findings not posted (1)
  • correctness#1 [WARNING] src/per_isolate/webstreams/readable.ts:1781: A default source's declared total is never enforced while it is streamed as a Request/Response body. For example, a source with expectedLength: 10 that enqueues a five-byte Uint8Array then closes … — The current diff rejects a declared-length underflow at EOF in collectChunks(), and the added integration regression covers both default and TransformStream bodies.

Reviewed commit: 30509c11 · github run

Comment thread src/per_isolate/webstreams/readable.ts
Comment thread src/per_isolate/webstreams/writable.ts
@jasnell
jasnell force-pushed the jasnell/ts-streams-default-expected-length branch from b1f9752 to 86a01dc Compare October 28, 2026 14:19

@ry ry left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. A few follow-ups inline. Ledgers #36 and #37 are C++ bugs, both from ReadableStreamJsController::tryGetLength returning expectedLength regardless of encoding or reads; worth tracking issues.

Comment thread src/tests/streams/readable/expected-length.js Outdated
Comment thread src/workerd/api/tests/js-rpc-streams-crossflag-test.js
Comment thread src/tests/streams/readable/integration-body-failures.js
Comment thread src/per_isolate/webstreams/readable.ts
Comment thread src/per_isolate/webstreams/readable.ts
@jasnell
jasnell force-pushed the jasnell/ts-streams-default-expected-length branch from 86a01dc to 57dd431 Compare October 28, 2026 20:38
@jasnell
jasnell enabled auto-merge October 28, 2026 20:40
Under typescript_implemented_streams, the default controller ignored
the expectedLength its underlying source declared: only byte sources,
a TransformStream's readable (through an internal setter) and native
streams reported a length. A body built from a default source that
declared its length was therefore sent with chunked encoding, where
the C++ implementation sends a Content-Length.

The default controller now caches the converted source's
expectedLength, as the byte controller does, and TransformStream
passes its transformer's through the source it builds for its
readable, which retires the internal setter.

Body consumption already treated a TransformStream readable's
declaration as an exact total (readable ledger #21); that now covers
any default source that declares one. A source that delivers more
than it declared rejects text(), arrayBuffer() and the other
consumers with a RangeError and is cancelled with it, where the C++
implementation returns everything. The divergence is intentional:
the declaration is the exact-total contract that byte and native
sources already enforce.

typescript_implemented_streams is still experimental, with no enable
date, so the change needs no compatibility flag.

Fixes: #8040
A queued TypeScript stream reported the expectedLength its source
declared to the C++ bridge, which uses it as a body's Content-Length,
even after part of the stream had been read. The declaration is the
stream's total, and the controller, which tee branches share, does
not count what a stream has already delivered. So a tee branch made
after a read, or the copy detach() makes of one (as new Request()
does with a body), sent the full declared length as its
Content-Length, and the request failed with "Network connection
lost." when the body fell short. A partly read stream handed to JS
RPC, KV put() or R2 put() announced its full total the same way.

A queued stream now reports its declared length only while nothing
has been read from it or, for a tee branch or detached copy, before
the fork. makeShell() records whether the stream it forks had been
read, and getReadableStreamExpectedLength() checks that record along
with whether the stream itself has been read. Native streams are
unchanged: their forks are fresh streams over sources that count
what is left, and KV, R2 and RPC reject a partly read native stream.

This leaves three intentional divergences from the C++
implementation, pinned in the streams suites and the
cross-implementation RPC test:

- A fork made before any read keeps the declared length in
  TypeScript. C++ tee branches never carry a length.
- A partly read stream reports no length in TypeScript: RPC and KV
  send it chunked, and R2 put() rejects it for want of a known
  length. C++ still reports the declared total, so the transfer
  fails.
- A FixedLengthStream branch teed after a read goes out chunked in
  TypeScript. C++ carries what the branch has left.

The identity suite gains a SELF echo to observe a FixedLengthStream's
Content-Length.

typescript_implemented_streams is still experimental, with no enable
date, so the change needs no compatibility flag.
A response whose body declares expectedLength and carries
Content-Encoding: gzip is compressed on the way out, so its
uncompressed declaration is not a valid Content-Length. The C++
bridge already reports no length for a queued TypeScript stream
whose body is encoded, as for a FixedLengthStream, but only a C++
unit test checked it.

The readable and readable-byte suites now serve such a response
through SELF from a default source, a TransformStream readable and a
byte source. Under TypeScript it goes out chunked and decodes in
full. The C++ implementation keeps the uncompressed length as the
Content-Length, so the compressed body falls short of it and the
reader fails with "Network connection lost." That is a pre-existing
C++ defect, pinned as a divergence (readable ledger #37,
readable-byte ledger #46) rather than reproduced.

Refs: #8040 (comment)
A default source or TransformStream readable whose declared
expectedLength is sent as the Content-Length, but which delivers
less than it declared, fails the transfer: as a request body the
receiver's read rejects with "Network connection lost.", and as a
response body the client's does. The transport enforces the framing
it was given, so a short body is never taken for a complete one. The
C++ implementation behaves the same, so this is pinned as parity.

The readable suite's SELF service serves the short response at
/short, and the body helper behind the /gzip response now takes the
declared length and the text to deliver.

The parity note in the suite's AGENTS.md also now lists ledger #37
among the integration divergences.
sinkReleasedOnceEnded flaked under typescript_implemented_streams
(the write-rejects case, on macOS and Linux CI; seen on unrelated
branches too): a stream that had ended still kept its underlying
sink alive. The write, close and abort algorithms were closures
over the sink, and ClearAlgorithms dropped them, but once V8 had
JIT-compiled them their context, and the sink in it, could stay
reachable. With Maglev tiering up early (--invocation-count-for-
maglev=5 --no-turbofan --minimum-invocations-after-ic-update=2) the
test fails every run under TypeScript and passes under C++.

The controller now holds the sink and its uncurried hooks in fields
that ClearAlgorithms clears, and calls them from private methods,
so no closure captures the sink. The test passes every run with
those flags.
sinkReleasedOnceEnded checks that an ended stream releases its sink,
but it ran before V8 had optimized the TypeScript sink algorithms in
most runs, so it rarely caught closures over the sink kept alive by
optimized code. Both main cells now pass
--invocation-count-for-maglev=5 and
--minimum-invocations-after-ic-update=2, so the algorithms are
optimized by the time gc.js runs. Under them the closure-based
algorithms fail the test in 30 of 36 runs across the three variants;
the field-based ones pass all 36, and the C++ cell passes too.

A test that warms the algorithms inside gc.js does not reproduce the
retention, nor does gc.js alone in a cell of its own with the same
flags: it takes the state the rest of the suite leaves behind, so
the flags go on the full cells.
Body consumption under typescript_implemented_streams treated a
default source's or TransformStream readable's declared
expectedLength as an exact total only on overflow: a stream closing
short of its declaration resolved with what had arrived. It now
rejects text(), arrayBuffer() and the other consumers with a
RangeError, as byte and native sources already do by erroring at
close(), and as the transport does by failing a short body. The
stream has already closed, so there is nothing to cancel. Readable
ledger #21 now covers both directions; C++ consumption still ignores
the declaration.

ReadableStreamDrainingReader's expectedLength reported the source's
declaration even after the stream had been read, disagreeing with
the length the C++ bridge sees. It now asks
getReadableStreamExpectedLength(), so a partly read stream, or a
fork made after a read, reports none.

The C++ sides of readable ledgers #36 and #37 (readable-byte #45 and
#46) are now marked as defects: ReadableStreamJsController's
tryGetLength() reports the declared total regardless of reads or
encoding.
@jasnell
jasnell force-pushed the jasnell/ts-streams-default-expected-length branch from 57dd431 to 30509c1 Compare October 28, 2026 21:01
@jasnell
jasnell merged commit 9792d03 into main Oct 28, 2026
25 checks passed
@jasnell
jasnell deleted the jasnell/ts-streams-default-expected-length branch October 28, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report — Runtime APIs: expectedLength on a default ReadableStream source no longer sets Content-Length with typescript_implemented_streams

2 participants