[simin-yuan]

Reference implementation for #2051.

timeout= bounds a single socket read. A server that sends : ping comments or ping events resets
that timer without ever moving the response forward, so a stalled stream and a slow one are
indistinguishable to the caller.

This adds an opt-in guard that counts content instead of bytes: if no content event arrives for N
seconds, the stream raises APITimeoutError, even though keepalives are still arriving.

from anthropic.lib.streaming._idle_timeout import guard_stream

stream = client.messages.create(..., stream=True)
guard_stream(stream, seconds=30)
for event in stream:          # APITimeoutError if the next 30 s carry no content
    ...

It is not re-exported from anthropic.lib.streaming in this PR — one explicit import instead of
expanding the public surface before the parameter shape is settled (see "Not in this PR").

Where it sits

The guard wraps the response's byte stream rather than the parsed events, because the SDK discards
keepalive frames while parsing — by the time an event reaches a caller, the information that a
keepalive arrived is already gone. Comment lines and ping events do not count as content; error
frames do, since the caller can act on them.

Two limits, stated rather than hidden: the check runs when a chunk arrives, so a server that stops
sending bytes entirely is still the transport read timeout's job; and a frame split across two chunks
is reassembled in a bounded buffer before being classified. Lines end at CR, LF or CRLF, the same
rule SSEDecoder applies when it calls bytes.splitlines(), and the buffer is capped at 64 KiB, so
a server that never terminates a line cannot grow it without bound.

Tests

tests/lib/streaming/test_idle_timeout.py, 22 tests, 22 passed locally (Python 3.10.20):

• a comment-only stream (: ping, the frame LiteLLM's proxy writes) raises on the guard's own clock,
  not at the end of the body;
• a ping-event-only stream does the same;
• 20 content frames spaced under the window are all delivered, none truncated;
• a keepalive split across two chunks is still classified as a keepalive;
• an error frame counts as content;
• an async twin of the comment-keepalive test, plus end-to-end runs through Anthropic and
  AsyncAnthropic with a mock transport;
• unguarded behaviour is unchanged;
• guarding a consumed response raises, and a non-positive window raises;
• a stream whose frames are terminated with a bare CR: keepalives still do not reset the clock,
  content does, on both the sync and async paths;
• 1000 CR-only keepalive frames leave nothing pending: _tail is empty after every frame;
• a CRLF split across two chunks (chunk 1 ends CR, chunk 2 starts LF): the stray LF reads as an extra
  blank line, which is a keepalive either way, and a content line that ends the chunk is counted at once;
• a data: line spread over three chunks is recognised as content;
• an unterminated line past the cap is counted as content and the buffer is dropped.

Not in this PR

Plumbing idle_timeout= into messages.create(stream=True) as a first-class parameter: it needs a
decision about whether the window belongs on the client, on the call, or on the stream, and about how
it composes with timeout=. Once that is settled the guard moves inside Stream.__stream__ and the
public helper goes away.

[mangeshraut712]

Solid guard design — keepalives resetting a read timer while content never arrives is a real blind spot, and the mid-frame tail is a nice touch. One issue in _KeepaliveTracker.has_content: it only splits on \n, but the SSE spec (WHATWG) defines CR, LF and CRLF all as valid line terminators.

Consequences for a server that terminates lines with a bare \r:

1. The frame never completes a split(b"\n") line, so _tail is never emptied — each chunk is appended until the stream ends, so an otherwise idle-but-alive stream grows _tail without bound (memory leak on long keepalive streams).
2. A data: line carrying real content, terminated with \r, is never parsed as content, so has_content returns False even though the response is moving forward. If the gap between two CR-terminated frames exceeds seconds, the guard raises a false APITimeoutError on a healthy stream.

Suggest splitting on a bytes regex that accepts \r\n?/\n (or normalizing \r to \n before split) and a test with CR-only keepalive and content frames. Also worth a guard test with a chunk ending exactly at a frame boundary mid-data: payload with the tail non-empty across >2 chunks, since _tail + chunk concatenation is what the parser depends on.

[simin-yuan]

Thanks — you're right, and I checked it against the SDK's own decoder before changing anything.

What the decoder actually does. SSEDecoder does not split lines chunk by chunk: it accumulates bytes up to an event boundary (\r\r, \n\n, \r\n\r\n) and then calls chunk.splitlines(). On bytes, splitlines() breaks on CR, LF and CRLF and nothing else, which is exactly what the comment above iter_bytes says. There is no standalone line splitter in _streaming.py to import, so the tracker now spells out the same rule: rb"\r\n|\r|\n".

CRLF across a chunk boundary. A chunk that ends in a bare CR is held back instead of being treated as a finished line, because the next chunk may start with LF and together they are one terminator. Ending the line at the CR would make the incoming LF read as an extra blank line — and a blank line ends an SSE event, so the guard would be parsing a different stream from the decoder.

Buffer bound. Classification needs the whole line only because a data: payload has to be parsed to tell {"type": "ping"} from content, and keepalives are a few dozen bytes. So I took the second option from your review: the pending line is capped at 64 KiB, and anything longer counts as content and is dropped. Worst case that is one missed check on a pathological stream, against the unbounded _tail that was the leak.

Tests (tests/lib/streaming/test_idle_timeout.py, 22 total now):

• CR-only comment, ping-event and content frames — keepalives do not reset the clock, content does (sync and async paths);
• 1000 CR-only keepalive frames — _tail stays at 1 byte;
• CRLF split across two chunks (chunk 1 ends CR, chunk 2 starts LF) — one terminator, no invented blank line, and a content line split that way is still content;
• a data: line spread over three chunks — recognised as content;
• an unterminated line past the cap — the buffer stays bounded and the line counts as content.

Four of the new tests fail against the previous implementation: the sync and async false timeouts, the unbounded _tail, and the uncapped pending line.

Full suite on this machine (pytest tests/): 12549 passed, 166 failed, 27 errors — the same 166 failures and 27 errors before my change, all POSIX-only tests (os.geteuid, os.mkfifo, executable bits, symlinks) running on Windows; the delta is +10 passed. ./scripts/lint passes ruff and the dependency-cap check, and pyright's 15 errors are those same Windows-only os attributes in unrelated test files.

Commit: simin-yuan@cf8bcad

Thanks for catching the CR case — the tail growth would only surface against a server that uses it, which is exactly the case that never gets tested.

— Simin Yuan

[simin-yuan]

Follow-up in simin-yuan@6bec8e2: I removed the held-back trailing CR. The tracker classifies line by line and an empty line is always a keepalive, so a CRLF split across chunks cannot change the result — while holding the CR delayed counting a CR-terminated content line until the next chunk. Tests updated accordingly; the split-content test now fails on cf8bcad and passes here. Single-file run: 22 passed on Python 3.10.

Since this is my first contribution, the workflows need a maintainer's approval to run — thanks in advance.

— Simin Yuan

[simin-yuan]

@mangeshraut712 thanks for the review. Bare CR is handled now: lines end at CR, LF or CRLF (cf8bcad0), and a CR-terminated line counts as soon as it arrives (6bec8e22), with CR-only tests for keepalive, content and the tail buffer.