Skip to content

Fix PR preview capacity admission race - #349

Merged
kfiramar merged 2 commits into
mainfrom
codex/fix-preview-capacity-race
Jul 15, 2026
Merged

kfiramar merged 2 commits into
mainfrom
codex/fix-preview-capacity-race

Conversation

@kfiramar

@kfiramar kfiramar commented Jul 15, 2026 •

Copy link
Copy Markdown
Collaborator
  • Fix PR preview capacity admission race (e3cfb56)

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3cfb56c86

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return 0

active_prs = self.active_prs_with_label()
active_previews = self.capacity_consuming_preview_states()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Persist a starting state before admitting capacity

With this switch to capacity_consuming_preview_states(), admission no longer observes PRs that have the label but have not yet written a state file. In the start() path I inspected, no starting state is saved before envctl import/envctl start; the first save happens later in failure/success branches. If two label or synchronize jobs run at the same time, both can read zero capacity-consuming previews and both pass ENVCTL_PREVIEW_MAX_OTHER_ACTIVE, oversubscribing the runner; the previous label-based query would count the other labeled PR.

Useful? React with 👍 / 👎.

r"^ENVCTL_[A-Z][A-Z0-9_]*_ENV__[A-Z_][A-Z0-9_]*$"
)
FAILED_START_STATUSES = frozenset({"public_route_unhealthy", "start_failed"})
CAPACITY_CONSUMING_PREVIEW_STATUSES = frozenset({"running", "starting", "qa_user_failed", "public_route_failed"})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Count failed stops as capacity-consuming

stop() records status = "stop_failed" when envctl stop returns nonzero while leaving the state’s project intact, which is exactly a case where the runtime may still be alive. Because this allow-list omits stop_failed, later starts ignore that preview in max_other_active_previews even if the PR still has the deploy label, such as after a failed manual stop, so the controller can admit more previews onto an already occupied runner.

Useful? React with 👍 / 👎.

r"^ENVCTL_[A-Z][A-Z0-9_]*_ENV__[A-Z_][A-Z0-9_]*$"
)
FAILED_START_STATUSES = frozenset({"public_route_unhealthy", "start_failed"})
CAPACITY_CONSUMING_PREVIEW_STATUSES = frozenset({"running", "starting", "qa_user_failed", "public_route_failed"})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include failed-start cleanup states in capacity

When envctl start fails or the public-route health check fails, the controller saves start_failed/public_route_unhealthy before attempting cleanup; if cleanup_failed_start_preview() returns nonzero, that state remains while the partially started runtime may still be alive and the label is left for retry. Since this capacity allow-list excludes both statuses, later PRs ignore that occupied runtime and can pass ENVCTL_PREVIEW_MAX_OTHER_ACTIVE even though the earlier start did not clean up.

Useful? React with 👍 / 👎.

@kfiramar
kfiramar merged commit 6f07541 into main Jul 15, 2026
3 checks passed
@kfiramar
kfiramar deleted the codex/fix-preview-capacity-race branch July 15, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant