[shawnburke]

Summary

Two bugs in manageStream (agent/server/grpctunnel/tunnel_client.go):

1. Backoff never reset after a healthy stream. backoff went back to minBackoff only on the retired and Unauthenticated paths. A stream that served traffic for hours and then ended (GOAWAY max_age, EOF) kept the backoff of earlier, unrelated failures, so after a few routine drops every reconnect waited about 30s. runOneStream now also returns established (the handshake completed), and manageStream resets to minBackoff when it is true. Repeated failures to open still back off exponentially.
2. Re-registration stampede, and established-stream drops counted as open failures.
   • The threshold check and the reset were not atomic, so when N workers failed together several passed the check before any reset it. The reset is now claimed with CompareAndSwap, so one threshold crossing forces one re-registration.
   • consecFailures grew on any stream end. Only streams that failed to open now count, so a simultaneous drop of 5+ established slots no longer forces re-registration.

Prod evidence

Slot 3 nextBackoff over 2026-10-22 to 2026-10-24, each failure hours apart with a successful "Tunnel stream re-established" in between: 1s -> 2s -> 4s -> 8s -> 16s.

2026-10-23T18:49:11: "Repeated open failures; forcing re-registration" from 5 workers, followed by five "Registered with Cortex API" lines within 11ms.

TDD

Commits: (1) jittered becomes a var so tests can skip the wait, no behavior change; (2) failing tests; (3) the fix.

Tests drive manageStream workers directly and reuse fakeTunnelService, startFakeServer, newTestClient and waitFor. The backoff test reads nextBackoff from the existing "stream ended; will retry" log via zaptest/observer.

Red (commit 2, before the fix)

=== RUN   TestManageStream_BackoffResetsAfterEstablishedStream
        	            	expected: []time.Duration{1000000000, 2000000000, 4000000000, 1000000000}
        	            	actual  : []time.Duration{1000000000, 2000000000, 4000000000, 8000000000}
        	            	--- Expected
--- FAIL: TestManageStream_BackoffResetsAfterEstablishedStream (0.02s)
=== RUN   TestManageStream_ForcedReregisterIsSingleFlight
        	            	expected: 1
        	            	actual  : 4
--- FAIL: TestManageStream_ForcedReregisterIsSingleFlight (0.01s)
=== RUN   TestManageStream_EstablishedDropsDoNotForceReregister
--- FAIL: TestManageStream_EstablishedDropsDoNotForceReregister (0.02s)
=== RUN   TestManageStream_OpenFailuresForceReregisterAtThreshold
=== RUN   TestManageStream_OpenFailuresForceReregisterAtThreshold/4_failures
=== RUN   TestManageStream_OpenFailuresForceReregisterAtThreshold/5_failures
--- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold (0.02s)
    --- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold/4_failures (0.01s)
    --- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold/5_failures (0.01s)
FAIL
FAIL	github.com/cortexapps/axon/server/grpctunnel	0.109s
FAIL

Green (commit 3)

--- PASS: TestManageStream_BackoffResetsAfterEstablishedStream (0.03s)
--- PASS: TestManageStream_ForcedReregisterIsSingleFlight (0.01s)
--- PASS: TestManageStream_EstablishedDropsDoNotForceReregister (0.03s)
--- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold (0.02s)
    --- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold/4_failures (0.01s)
    --- PASS: TestManageStream_OpenFailuresForceReregisterAtThreshold/5_failures (0.01s)
PASS
ok  	github.com/cortexapps/axon/server/grpctunnel	1.133s

Also run: go test -race ./server/grpctunnel/ (ok, 3 runs), the new tests with -race -count=20 (ok), go vet ./server/grpctunnel/ (clean).

🤖 Generated with Claude Code