[aszarama]

What

The relay reflector now waits up to 30 seconds for the broker to write a request. Before, it closed any connection that had no complete request one second after the connection opened. Under load, the broker often writes later than that, so the reflector reset requests that were already on their way.

Each of the agent's HTTP servers now has a fixed read bound. There is no setting to change.

Server	Read bound
Relay reflector	30s (was 1s)
Webhook server	30s (was 1s)
Main server	1s, except 30s for a /cortex-api/ request body

Why

ReadTimeout counts from when a connection is accepted, not from the first byte. The broker opens a connection and writes the request only when its event loop gets to it. That loop stalls for two to three seconds each time it sends a large (13 MB) response over the websocket. Any connection the broker opened just before a stall is closed before its request arrives.

The broker sees this as socket hang up (ECONNRESET), or EPIPE for a body it is still writing. It retries after 500ms, up to three times. In one 17-minute production export from a single pod on v0.3.4, that was about 8,000 retries, 440 for every 1,000 relayed requests, and 31 requests failed after three retries. In the v0.3.9 export, which has #168, it is 132 to 180 retries per 1,000 requests. #168 fixed idle keep-alive connections but not new ones.

The reset also hits the relay dispatcher's health probe. The dispatcher calls /__axon/info through the broker, which reaches the agent through the reflector, like other traffic. The same export has about 4,000 failed /__axon/info attempts. A failed probe makes the dispatcher mark the agent as not responding. With no responsive agent, callers get 555, even for low-volume requests unrelated to the large responses.

30 seconds covers every stall in the v0.3.4 export and 1,875 of 1,880 in a 75-minute, four-pod export on v0.3.9. It is also less than the relay dispatcher's 60-second request timeout.

How

Review focus: http_server.go (WithReadTimeout) and cortex_api_proxy_handler.go. The reflector and webhook changes are one line each.

• New WithReadTimeout server option. A server without it keeps the one-second default.
• The reflector and the webhook server use 30s. Webhook senders are outside the agent's network and can upload large payloads slowly.
• The main server keeps 1s for its own endpoints. The /cortex-api/ proxy moves its read deadline to 30s with http.ResponseController. responseRecorder gets an Unwrap method, so the controller can reach the real connection.
• ReadHeaderTimeout stays unset, so net/http uses ReadTimeout for headers too. In a test, a 30s header-only bound with the 1s ReadTimeout still failed a PUT body that arrived 1.5s late.
• Not in this PR: the broker stalls themselves, and the websocket reconnects seen in the same logs. Deployed agents get the fix only after they upgrade.

Verification

• New tests pass: a 30s server serves a request written 1.5s after connecting and a PUT body written 1.5s after its headers; a default server drops a request written 1.5s late; the /cortex-api/ proxy forwards a body that arrives 1.5s late.
• Through the real reflector, I opened 50 connections and blocked a Node 20 client for 1.5 seconds before it wrote the requests. With 1s, all 50 reset. With 30s, all 50 were served. This ran on an earlier commit that set the value with an environment variable.
• go test ./... in agent/ passes on the latest commit, except TestLoopbackOnlyRefusesOffHostConnection. It dials the machine's LAN address, and it fails the same way on unmodified v0.3.9 on my machine.
• Not run: a full agent against a broker server, or a production relay with this build. The drop in health-probe failures and 555s is expected, not measured.

[shawnburke]

we should deal with this at the callsite rather than globally - see comment

[shawnburke]

I don't think this is the right approach. There are three different instances of httpServer:

• Axon stuff (that is all fast)
• Webhooks (its own instance and port)
• Reflector

The second two need configurable timeouts as a startup WithReadTimeout option kind of thing so they should just have longer timeouts. This is an inheritance of defaults problem

The tougher case is that the cortex-api passthrough is on the main server so it picks up the 1s default. We can raise that with a thing like _ = http.NewResponseController(w).SetReadDeadline(time.Now().Add(apiProxyReadTimeout)) in the cortex_api_handler code.

[shawnburke]

Good - 30s is still small do we want these configurable?