Repository navigation
Fix PR preview capacity admission race - #349
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e3cfb56c86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return 0 | ||
|
|
||
| active_prs = self.active_prs_with_label() | ||
| active_previews = self.capacity_consuming_preview_states() |
There was a problem hiding this comment.
Persist a starting state before admitting capacity
With this switch to capacity_consuming_preview_states(), admission no longer observes PRs that have the label but have not yet written a state file. In the start() path I inspected, no starting state is saved before envctl import/envctl start; the first save happens later in failure/success branches. If two label or synchronize jobs run at the same time, both can read zero capacity-consuming previews and both pass ENVCTL_PREVIEW_MAX_OTHER_ACTIVE, oversubscribing the runner; the previous label-based query would count the other labeled PR.
Useful? React with 👍 / 👎.
| r"^ENVCTL_[A-Z][A-Z0-9_]*_ENV__[A-Z_][A-Z0-9_]*$" | ||
| ) | ||
| FAILED_START_STATUSES = frozenset({"public_route_unhealthy", "start_failed"}) | ||
| CAPACITY_CONSUMING_PREVIEW_STATUSES = frozenset({"running", "starting", "qa_user_failed", "public_route_failed"}) |
There was a problem hiding this comment.
Count failed stops as capacity-consuming
stop() records status = "stop_failed" when envctl stop returns nonzero while leaving the state’s project intact, which is exactly a case where the runtime may still be alive. Because this allow-list omits stop_failed, later starts ignore that preview in max_other_active_previews even if the PR still has the deploy label, such as after a failed manual stop, so the controller can admit more previews onto an already occupied runner.
Useful? React with 👍 / 👎.
| r"^ENVCTL_[A-Z][A-Z0-9_]*_ENV__[A-Z_][A-Z0-9_]*$" | ||
| ) | ||
| FAILED_START_STATUSES = frozenset({"public_route_unhealthy", "start_failed"}) | ||
| CAPACITY_CONSUMING_PREVIEW_STATUSES = frozenset({"running", "starting", "qa_user_failed", "public_route_failed"}) |
There was a problem hiding this comment.
Include failed-start cleanup states in capacity
When envctl start fails or the public-route health check fails, the controller saves start_failed/public_route_unhealthy before attempting cleanup; if cleanup_failed_start_preview() returns nonzero, that state remains while the partially started runtime may still be alive and the label is left for retry. Since this capacity allow-list excludes both statuses, later PRs ignore that occupied runtime and can pass ENVCTL_PREVIEW_MAX_OTHER_ACTIVE even though the earlier start did not clean up.
Useful? React with 👍 / 👎.
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.