Skip to content

Fix PinghubWebSocketTests port collisions - #26088

Draft
jkmassel wants to merge 1 commit into
trunkfrom
jkmassel/deflake-pinghub-websocket-tests
Draft

jkmassel wants to merge 1 commit into
trunkfrom
jkmassel/deflake-pinghub-websocket-tests

Conversation

@jkmassel

Copy link
Copy Markdown
Contributor

Fixes PinghubWebSocketTests.testReceiveManyMessage (62% reliable over the last 7 days in Test Engine, labeled flaky). The same bug affects every test in the class; testReceiveManyMessage is just the one that loses most often on CI.

Root Cause

PinghubServer picked a random port in 9000...9999 and retried up to 5 times if WebSocketServer.start returned an error. But Starscream's start creates an NWListener and returns nil right away — the bind happens asynchronously — so a taken port is never reported and the retry loop never retries:

nw_listener_socket_inbox_create_socket bind(83, ::.9201) ... failed [48: Address already in use]
nw_read_request_report [C147] Receive failed with error "Connection reset by peer"

The port is usually taken by a listener from an earlier test. PinghubServer never stops its listener (Starscream 4.0.8 has no API for it), so listeners pile up for the life of the test process. When a new server lands on one of their ports, the client connects to the stale listener, the connection gets reset, and the test waits out its 20s timeout.

Reproduced locally by running PinghubWebSocketTests 30× in one process: 11/150 executions failed, and 22 of the 150 servers hit Address already in use.

Separately, connect() only waited for the client to see the upgrade response. The server appends the connection to clients just after sending that response, on its own queue, so a broadcast could in principle race it (and clients was read and written from two threads).

Fix

  • Ask the OS for a free port. unusedPort() binds a dual-stack socket to port 0, reads the assigned port with getsockname, and releases it for the server. Leaked listeners still hold their ports, so the OS won't hand those out.
  • Wait for both ends in connect(). The server fulfills a "registered" expectation once it has the connection, so broadcast never runs against an empty clients list.
  • Keep clients on the main thread. Server events hop to main before touching the list.

Test-only; PinghubClient is unchanged.

Test plan

  • Local: PinghubWebSocketTests × 100 iterations in one process (Xcode 27.0, iOS 27.0 simulator) — 500/500 passed, 0 Address already in use, 2.8s total (vs 190s with the timeouts)
  • CI: stress builds, each on its own ref at this commit; check Test Engine first-attempt executions

The test server picked a random port in 9000-9999 and relied on
WebSocketServer.start to report a taken port, but start returns before
the NWListener binds, so the retry loop never retried. Listeners from
earlier tests are never stopped (Starscream has no API for it), so a
collision sent the client to a dead listener and the test timed out.

Ask the OS for a free port instead, wait for the server to register the
connection before broadcasting, and keep the server's client list on the
main thread.
@jkmassel jkmassel added this to the 27.4 milestone Sep 25, 2026
@jkmassel jkmassel added the Testing Unit and UI Tests and Tooling label Sep 25, 2026
@jkmassel jkmassel self-assigned this Sep 25, 2026
@dangermattic

Copy link
Copy Markdown
Collaborator
1 Message
📖 This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

@wpmobilebot

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.
App NameJetpack
ConfigurationRelease-Alpha
Build Number34704
VersionPR #26088
Bundle IDcom.jetpack.alpha
Commit26eeaea
Installation URL1ls4su5oia43o
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@wpmobilebot

Copy link
Copy Markdown
Contributor
App Icon📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.
App NameWordPress
ConfigurationRelease-Alpha
Build Number34704
VersionPR #26088
Bundle IDorg.wordpress.alpha
Commit26eeaea
Installation URL0g2g8tkomip5g
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Unit and UI Tests and Tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants