Skip to content

tweak: Request the minimal workflow record and stop renewing on any 4xx lookup reply [NOTASK] - #1209

Merged
pditommaso merged 2 commits into
masterfrom
fix/workflow-lookup-minimal
Oct 27, 2026
Merged

pditommaso merged 2 commits into
masterfrom
fix/workflow-lookup-minimal

Conversation

@pditommaso

Copy link
Copy Markdown
Collaborator

Problem

The container token watcher calls GET /workflow/{id} through the Platform pairing socket only to read workflow.status, but receives the full record (params, configText, errorReport, progress). On 2026-10-22 one completed workflow's record exceeded the 4 MiB pairing cap: Wave closed the shared socket with 1009, the lookup timed out, and the watcher logged "could not confirm workflow status ... renewing anyway". It then retried on every renewal for up to wave.tokens.cache.max-duration (2d), closing the socket for all users each time.

Write-up: https://claude.ai/artifact/Cf24jN8vRNat6yZXBqWh5i

Change

  • TowerClient.workflowDescribeEndpoint adds attributes=minimal (plus workspaceId when present). The watcher is the only caller of describeWorkflow and it reads only the status.
  • ContainerRequestServiceImpl.isDefiniteRejection treats any 4xx reply except 408 and 429 as a definite answer. Before, only 401, 403 and 404 counted. Wave now stops tracking the request, as it does for an inactive workflow, and logs once at INFO. 408 is excluded because the connector also uses it for its own timeouts. Timeouts, I/O errors and 5xx still renew (fail-open), as before.

Companion Platform changes (in progress)

  • Platform's describe honours attributes=minimal and omits the large fields. Older Platforms ignore the parameter and return the full record, so it is always safe to send.
  • Platform replies HTTP 413 instead of sending a pairing message over the cap. With this PR, Wave stops retrying when it gets that 413.

The two repos can be released in either order.

Tests

TowerClientTest (describe URI with and without workspaceId) and ContainerRequestServiceImplTest (400/413 stop tracking; 408, 429, 503 and timeout still renew) pass locally.

🤖 Generated with Claude Code

…up reply

The token watcher only reads the workflow status, but fetched the full describe
record, whose params, config and error report can exceed the 4 MiB pairing cap
and close the socket for every user (1009). Ask Platform for attributes=minimal,
which older Platforms ignore.

Treat any 4xx reply other than 408/429 (e.g. 413) as a definite answer and stop
tracking the request, instead of renewing it for up to the max token lifetime.
Timeouts, I/O errors and 5xx still renew (fail-open).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pditommaso
pditommaso marked this pull request as ready for review October 23, 2026 10:07

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: COMMENT. The two changes each make sense alone, but together they can cause a fail-closed outage on Platforms that reject the new query parameter. I can't check that from this repo, so a human should confirm it before merge.

The risk: attributes=minimal combined with "any 4xx revokes"

  • TowerClient.workflowDescribeEndpoint now always sends ?attributes=minimal. The PR says the Platform side that supports minimal is still "in progress", and that older Platforms "ignore the parameter".
  • isDefiniteRejection now treats every 4xx except 408/429 as a definite answer. onLookupError then stops renewing the token, which revokes image access for a pipeline that may still be running.

On Platform, attributes on the workflow endpoints is (as far as I know) bound to an enum (WorkflowQueryAttribute). In Micronaut, a query value that doesn't convert to the enum normally gets a 400 Bad Request; it is not dropped. If a deployed Platform does that for minimal, here is what happens:

  1. Every watcher lookup against that Platform returns 400.
  2. Before this PR, 400 was fail-open: the token renews and Wave logs a warning. After it, 400 is a definite rejection: the watcher stops tracking at the next check.
  3. Every container token issued through that Platform expires at its current TTL, while its workflow may still be running. Self-hosted Enterprise installs on older versions would be hit first, and the two-repo rollout makes them the likely case. The PR says the repos "can be released in either order", which assumes the ignore behaviour.

Please confirm one of these before merging:

  • Current and supported older Platform versions really ignore an unknown attributes value. A quick curl '<platform>/api/workflow/<id>?attributes=minimal' against the oldest supported release would settle it. Or
  • Keep 400 transient for now: put it in TRANSIENT_CLIENT_CODES, or limit the definite set to [401, 403, 404, 410, 413]. Broaden it once the minimal Platform release is the floor.

The rest of the 4xx widening is harder to justify as "retrying would get the same answer". Two examples:

  • 412, which Wave raises itself in TowerConnector.parseTokens when a refresh succeeds but has no JWT cookie.
  • A 400 from the refresh endpoint.

Both now revoke a running pipeline's access. That trades fail-open for fail-closed on more cases than the 413 this PR targets. It may be what you intend, but it's worth saying so explicitly.

Otherwise fine

  • The URI change is correct: ?attributes=minimal&workspaceId=…, and the tests cover both forms. describeWorkflow is only called from the watcher, which only reads status, so the smaller record is enough.
  • Keeping 408 transient is right. TowerConnector turns its own timeouts into 408 (line 185) and I/O errors into 503.
  • The test updates cover the new 400/413 behaviour and the 408/429/500/503/timeout fail-open cases. I didn't run the test suite.

Approval is withheld only because the 400 question depends on Platform behaviour this PR doesn't include. The Wave-side code has no defect.

@munishchouhan munishchouhan changed the title Request the minimal workflow record and stop renewing on any 4xx lookup reply tweak: Request the minimal workflow record and stop renewing on any 4xx lookup reply [NOTASK] Oct 27, 2026
@pditommaso
pditommaso merged commit 5e01600 into master Oct 27, 2026
6 of 7 checks passed
@pditommaso
pditommaso deleted the fix/workflow-lookup-minimal branch October 27, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants