Skip to content

Keep a caller's headersTimeout and bodyTimeout in the fetch patch - #121

Open
Vivswan Shah (Vivswan) wants to merge 1 commit into
microsoft:mainfrom
Vivswan:fetch-patch-keeps-caller-timeouts
Open

Vivswan Shah (Vivswan) wants to merge 1 commit into
microsoft:mainfrom
Vivswan:fetch-patch-keeps-caller-timeouts

Conversation

@Vivswan

Copy link
Copy Markdown

What this changes

extension:  fetch(url, { dispatcher: new Agent({ headersTimeout: 36_000_000, bodyTimeout: 36_000_000 }) })

before:     createFetchPatch -> getAgentOptions keeps allowH2, connect.ca, requestTls.ca, proxyTls.ca
            -> new undici.Agent({ allowH2, connect: { ca } })            <- 300 s idle defaults again
after:      -> new undici.Agent({ allowH2, connect: { ca }, headersTimeout, bodyTimeout })

Why

undici's Agent and ProxyAgent default headersTimeout and bodyTimeout to 300 s. A caller that streams for longer than that with pauses (an LLM reasoning for minutes between chunks) raises both on its dispatcher. With http.proxySupport at its default override, or system certificates on, the fetch patch replaces that dispatcher and the timeouts do not come along, so the stream is closed at 300 s whatever the caller asked for. Extensions cannot work around it from their side.

How

  • getAgentOptions also reads headersTimeout and bodyTimeout from the caller's Agent or ProxyAgent options (both symbol paths) and returns them as timeouts.
  • getAgent / createAgent and getProxyAgent / createProxyAgent take the timeouts and pass them to the undici constructors. The WebSocket ProxyDispatcher threads the same values.
  • Caching is unchanged: agents are cached per original dispatcher, and the timeouts are a property of that dispatcher.

Proof

  • npm run compile typechecks.
  • Observed from Vivswan/litellm-vscode-chat#343: with the current patch a stream silent for 320 s dies at 301 s with UND_ERR_BODY_TIMEOUT; the extension has since moved to https.request, which the http patch handles without an idle clock, so this change is a courtesy for other callers of the fetch patch.

The fetch patch replaces the caller's dispatcher with its own Agent or ProxyAgent and carried over only allowH2, connect.ca, requestTls.ca, and proxyTls.ca. A caller that raised headersTimeout or bodyTimeout on its dispatcher, to let a long-running stream outlive undici's 300 s idle defaults, lost them: the replacement fell back to the defaults and the stream died at 300 s whatever the caller asked for. Carry both timeouts through getAgentOptions into createAgent and createProxyAgent, for the fetch patch and for the WebSocket dispatcher.

Seen from an extension in Vivswan/litellm-vscode-chat#343.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant