Skip to content

test: retry requests refused while a TCP endpoint comes online - #218

Merged
wesnel merged 1 commit into
mainfrom
wgn/flaky-tcp-endpoint-tests
Nov 13, 2026
Merged

wesnel merged 1 commit into
mainfrom
wgn/flaky-tcp-endpoint-tests

Conversation

@wesnel

@wesnel wesnel commented Nov 11, 2026

Copy link
Copy Markdown
Member

The online tests request each endpoint right after creating it, so they retry while ngrok brings it online. Until now they retried only a 404 or 5xx response. An HTTP endpoint that isn't live yet answers 404, so it was retried. A TCP endpoint refuses the connection instead, and a refused connection has no response, so the request failed straight away. That's what happened to "listen and forward multipass" in a #215 CI run (connect ECONNREFUSED to its TCP endpoint), while the same code passed in another run.

retry-config.mjs now also retries a refused connection, with the same delays. Every test that makes requests uses this one config.

Validation

  • A new offline test, retry-config.spec.mjs, refuses the first connection and then starts listening. It checks that the config retries until it gets the answer. Before the fix it failed with connect ECONNREFUSED and no retry, the same error as the CI failure. After the fix it passed three runs out of three.
  • I couldn't reproduce the online failure itself without an authtoken. CI's online suite is where it shows.

Weak spots, assumptions and decisions

  • Assumption: a new TCP endpoint refuses connections only briefly, well within the existing retries (2, 4 and 6 s). If routing takes longer, these tests still fail, and the delays would need to grow.
  • How often the failure happened is unknown. Of the failed runs checked, the deps: bump ngrok to 0.19.0 #215 run is the only one with this error, and most older logs have expired.
  • A refused connection is now retried for every request the tests make, HTTP endpoints included. A test that breaks its local upstream server would take up to 12 s longer to fail.
  • The new test takes about 2 s, the first retry's delay, because it uses the real config rather than a copy with shorter delays.

Reverting

Two-way door. Only test code changes. Reverting brings back the occasional failure.

The online tests send requests to an endpoint right after creating it.
An HTTP endpoint that ngrok has not routed yet answers 404, which
retry-config.mjs retries. A TCP endpoint refuses the connection instead,
and the retry condition gave up on any error without a response, so a
test that requested its TCP endpoint too soon failed with ECONNREFUSED,
as "listen and forward multipass" did on one CI run while passing on
another of the same code.

Retry a refused connection with the same delays. A new test refuses the
first connection and listens from then on, and checks the config retries
until it gets the answer.
@wesnel
wesnel added this pull request to stack #219 November 11, 2026 23:58
@wesnel
wesnel marked this pull request as ready for review November 12, 2026 00:02
@joshbottelberghe joshbottelberghe self-assigned this Nov 12, 2026

@joshbottelberghe joshbottelberghe left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@wesnel
wesnel merged commit ee42133 into main Nov 13, 2026
46 checks passed
@wesnel
wesnel deleted the wgn/flaky-tcp-endpoint-tests branch November 13, 2026 01:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants