Repository navigation
tweak: Request the minimal workflow record and stop renewing on any 4xx lookup reply [NOTASK] - #1209
Conversation
…up reply The token watcher only reads the workflow status, but fetched the full describe record, whose params, config and error report can exceed the 4 MiB pairing cap and close the socket for every user (1009). Ask Platform for attributes=minimal, which older Platforms ignore. Treat any 4xx reply other than 408/429 (e.g. 413) as a definite answer and stop tracking the request, instead of renewing it for up to the max token lifetime. Timeouts, I/O errors and 5xx still renew (fail-open). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Verdict: COMMENT. The two changes each make sense alone, but together they can cause a fail-closed outage on Platforms that reject the new query parameter. I can't check that from this repo, so a human should confirm it before merge.
The risk: attributes=minimal combined with "any 4xx revokes"
TowerClient.workflowDescribeEndpointnow always sends?attributes=minimal. The PR says the Platform side that supportsminimalis still "in progress", and that older Platforms "ignore the parameter".isDefiniteRejectionnow treats every 4xx except 408/429 as a definite answer.onLookupErrorthen stops renewing the token, which revokes image access for a pipeline that may still be running.
On Platform, attributes on the workflow endpoints is (as far as I know) bound to an enum (WorkflowQueryAttribute). In Micronaut, a query value that doesn't convert to the enum normally gets a 400 Bad Request; it is not dropped. If a deployed Platform does that for minimal, here is what happens:
- Every watcher lookup against that Platform returns 400.
- Before this PR, 400 was fail-open: the token renews and Wave logs a warning. After it, 400 is a definite rejection: the watcher stops tracking at the next check.
- Every container token issued through that Platform expires at its current TTL, while its workflow may still be running. Self-hosted Enterprise installs on older versions would be hit first, and the two-repo rollout makes them the likely case. The PR says the repos "can be released in either order", which assumes the ignore behaviour.
Please confirm one of these before merging:
- Current and supported older Platform versions really ignore an unknown
attributesvalue. A quickcurl '<platform>/api/workflow/<id>?attributes=minimal'against the oldest supported release would settle it. Or - Keep 400 transient for now: put it in
TRANSIENT_CLIENT_CODES, or limit the definite set to[401, 403, 404, 410, 413]. Broaden it once theminimalPlatform release is the floor.
The rest of the 4xx widening is harder to justify as "retrying would get the same answer". Two examples:
412, which Wave raises itself inTowerConnector.parseTokenswhen a refresh succeeds but has no JWT cookie.- A 400 from the refresh endpoint.
Both now revoke a running pipeline's access. That trades fail-open for fail-closed on more cases than the 413 this PR targets. It may be what you intend, but it's worth saying so explicitly.
Otherwise fine
- The URI change is correct:
?attributes=minimal&workspaceId=…, and the tests cover both forms.describeWorkflowis only called from the watcher, which only readsstatus, so the smaller record is enough. - Keeping 408 transient is right.
TowerConnectorturns its own timeouts into 408 (line 185) and I/O errors into 503. - The test updates cover the new 400/413 behaviour and the 408/429/500/503/timeout fail-open cases. I didn't run the test suite.
Approval is withheld only because the 400 question depends on Platform behaviour this PR doesn't include. The Wave-side code has no defect.
Problem
The container token watcher calls
GET /workflow/{id}through the Platform pairing socket only to readworkflow.status, but receives the full record (params, configText, errorReport, progress). On 2026-10-22 one completed workflow's record exceeded the 4 MiB pairing cap: Wave closed the shared socket with 1009, the lookup timed out, and the watcher logged "could not confirm workflow status ... renewing anyway". It then retried on every renewal for up towave.tokens.cache.max-duration(2d), closing the socket for all users each time.Write-up: https://claude.ai/artifact/Cf24jN8vRNat6yZXBqWh5i
Change
TowerClient.workflowDescribeEndpointaddsattributes=minimal(plusworkspaceIdwhen present). The watcher is the only caller ofdescribeWorkflowand it reads only the status.ContainerRequestServiceImpl.isDefiniteRejectiontreats any 4xx reply except 408 and 429 as a definite answer. Before, only 401, 403 and 404 counted. Wave now stops tracking the request, as it does for an inactive workflow, and logs once at INFO. 408 is excluded because the connector also uses it for its own timeouts. Timeouts, I/O errors and 5xx still renew (fail-open), as before.Companion Platform changes (in progress)
attributes=minimaland omits the large fields. Older Platforms ignore the parameter and return the full record, so it is always safe to send.The two repos can be released in either order.
Tests
TowerClientTest(describe URI with and without workspaceId) andContainerRequestServiceImplTest(400/413 stop tracking; 408, 429, 503 and timeout still renew) pass locally.🤖 Generated with Claude Code