Skip to content

fix(net): escalate the backoff when GOAWAY redirects loop between peers - #5959

Closed
kkroo wants to merge 1 commit into
moq-dev:mainfrom
Blockcast:fix/goaway-redirect-loop-backoff
Closed

kkroo wants to merge 1 commit into
moq-dev:mainfrom
Blockcast:fix/goaway-redirect-loop-backoff

Conversation

@kkroo

@kkroo kkroo commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Problem

Reload damps a GOAWAY redirect only when it arrives within delay.initial of connecting. When two relays each redirect to the other after a session has outlived that delay, every hop looks healthy. The backoff resets after each migration, so the client keeps bouncing between the two relays with no delay and never reports an error. Measured on main with mock relays (10 ms initial, 40 ms sessions): 24 dials in 1 s, alternating between the two URLs, and error stays undefined.

Approach

While migrations are chained, Reload remembers which endpoints redirected it somewhere else. If a redirect points back to one of them, that hop goes through #retry without the healthy-session reset. The delay therefore escalates up to delay.max, and delay.timeout still ends the sequence, with the error redirect loop. The redirect itself is still honoured; it is only slowed down.

The remembered set is cleared when a session ends without a GOAWAY, or when a new sequence starts (URL change, disable/re-enable, page hide).

Two cases are deliberately left unchanged:

Impact

  • No public API or wire change.
  • Behaviour: a redirect back to an endpoint that already moved the client elsewhere in the same chain now waits out the jittered backoff before dialing. A legitimate A→B→A rolling restart therefore pays one escalating delay. The draining session keeps serving during that wait, up to its handover cap.

Alternatives

  • End the sequence on a loop. Rejected: a recoverable fleet misconfiguration would become a permanent outage.
  • A count or time-window limit on migrations. Rejected: it adds a new knob, while reusing delay keeps a single backoff policy.

Follow-ups

  • Nothing resets the chain while sessions keep ending by GOAWAY. A fleet that cycles A→B→A over days escalates to delay.max and stays there. Is it worth resetting the chain after a session that lived for some long interval?
  • The Rust moq-tokio reconnect loop may have the same shape. Not checked here.

Verification

Four tests added to js/net/src/connection/migrate.test.ts:

  • Mutual redirect (A↔B): at most 6 dials in 1 s, against 24 on main.
  • Loop with delay.timeout: gives up with redirect loop.
  • Control, one-way move: migrates within 500 ms with a 1 s initial.
  • Control, empty-URI GOAWAYs: three in a row on healthy sessions each redial within 150 ms.

Results:

  • bun test (js/net): 1771 pass, 0 fail.
  • migrate.test.ts: 10/10 on each of 5 runs.
  • tsc --noEmit and biome check: clean. The one warning is already on main.
  • Mutation checks: with the loop guard disabled, the first two tests fail. Counting same-place GOAWAYs as a loop makes the empty-URI control fail.

🤖 Generated with Claude Code

Two relays that each redirect to the other once a session outlives
`delay.initial` bounced a Reload between them indefinitely: every hop
looked healthy, so the immediate-redirect damping never applied and the
backoff reset after each migration. Returning to an endpoint that already
sent us elsewhere in the current run of migrations now goes through the
backoff without the healthy reset, so the delay escalates to `max` and
`delay.timeout` still applies. One-way moves and GOAWAYs back to the same
place (a restart) are unaffected.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: HangPlayerEngineer <hangplayerengineer@paperclip.blockcast.net>
@kixelated

Copy link
Copy Markdown
Collaborator

Thanks @kkroo for tracking down this loop and for the clear write-up! Maintainers can't push to branches on organization-owned forks, so this continues in #6115. Your commit is carried over unchanged, so you stay its author. On top of it, that PR merges main, moves the tests onto the mock clock, and resets the redirect chain once a session lives at least delay.max, which answers your follow-up about long-lived fleets.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator

Superseded by #6115, which landed with your commit preserved. Thanks @kkroo!

(Written by Claude Opus 5.5)

@kixelated kixelated closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants