Repository navigation
feat(worktree): supervise runs in systemd scopes and release automatically - #178
Conversation
Quest-Claim: 4465a18959f4b93c
…cally `quest worktree run <id> -- <command>` runs a job in a checkout inside a systemd user scope named once per run. A handshake proves the shell is in that scope's v2 cgroup before the supervisor lets the command start, so a dead supervisor can never start it. When the command exits, descendants left in the scope are killed, and once the scope is empty and no other run remains, the scratch declared at `create` is pruned. Runs are recorded before their scope exists. Every registry operation reconciles runs whose owner (PID plus Linux start time) exited and whose scope is empty, so a crashed supervisor's checkout is released only after its orphans stop. A run that cannot be inspected or confirmed stopped stays recorded and the checkout stays reserved, with the reason noted. `release` refuses while runs are live. Non-Linux hosts fail explicitly. The registry lock now names its owner in a file inside the lock directory and is taken by an atomic rename, so the lock of a process that exited on this host is reclaimed by unlinking that exact owner file, never a later owner's lock. Also: scratch is declared at `create` instead of `release`, a scratch symlink to another worktree is unlinked rather than refused, and CI starts a systemd user manager for the real-scope fixtures (orphan cleanup, cancellation, crash reconciliation). The quest now covers only `quest worktree remove`; the runner and agent quests no longer wait on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
resolve('/', '.') is a drive root on Windows, so '.' passed the check
there and was refused only by the tracked-file check. Test the path's own
components instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/quest-start outcomeThis lands the first slice of process ownership: Local This PR stays a draft. Four decisions are listed under Decisions needed in the description, each with a recommendation already implemented: what triggers a release, killing leftover descendants, unblocking the runner and agent quests, and the CI user manager. (Written by Claude Opus 5.5) |
Drop links to the deleted local-gate quest and add stop and stale-failure clearing to the remaining worktree scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfcac77340
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Count recent admissions against free memory as well as load, define the spacing band, require atomic minute-long admission records, and pin down the claim directory, clock-step, and Ctrl-C rules. The Ctrl-C fix covers `quest worktree run` only if #178 has landed. The quest is now [M]. Also fix the seed test path in the inbox-accuracy quest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before Linux 5.14 a scope's cgroup has no cgroup.kill, so leftover descendants were never killed and the supervisor waited on a populated scope forever. run now refuses such hosts up front, and kill() ignores a missing cgroup.kill only once the cgroup itself is gone, without ever creating the file. Retitle the worktree quest [M] and point the Ctrl-C follow-up at #180. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#178 landed, so the gate headroom quest fixes the double Ctrl-C in both commands unconditionally, and the worktree quest links to it instead of the planning PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(quest): plan gate admission with headroom Follow-up of the contention measurements in #177: admit at once while the host has room, fix the two-window spacing, a stable claim directory, backward clock steps, and forwarding Ctrl-C once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): address review of the gate headroom plan Count recent admissions against free memory as well as load, define the spacing band, require atomic minute-long admission records, and pin down the claim directory, clock-step, and Ctrl-C rules. The Ctrl-C fix covers `quest worktree run` only if #178 has landed. The quest is now [M]. Also fix the seed test path in the inbox-accuracy quest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): count unexpired gate records and survive large clock steps Derive the recent-admission count from unexpired records with a record-then-count step instead of slot numbers, which expiry leaves with holes. Rewrite a future-dated record's time to now rather than discarding it, so a large backward clock step still spaces admissions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): elect one gate admission at a time under a lock Record-then-count prevents over-admission but can livelock when a whole burst withdraws together. Suggest serializing prune, count, and record under a short host-wide lock reclaimed from exited owners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): share the Ctrl-C fix with quest worktree run #178 landed, so the gate headroom quest fixes the double Ctrl-C in both commands unconditionally, and the worktree quest links to it instead of the planning PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): state gate admission as requirements and hazards Four review rounds kept finding edge cases in the prescribed records, lock, and clock-step mechanisms. Keep the decisions and the three-band rule, state what the admission mechanism must guarantee, and list every review finding as a hazard the implementation must test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): require a claim location stable across login sessions /run/user/<uid> comes and goes with logind sessions, so choosing by its existence can split callers again. State the requirement and the hazards instead of a directory rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): keep forwarding SIGHUP from the gate A SIGHUP sent with kill to Quest alone cannot be told apart from a terminal hangup, so skip forwarding only SIGINT in the foreground group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
First slice of Worktree process ownership, following kixelated/quest#127. Commands in a
quest worktreecheckout can now run supervised, and the checkout is released automatically once they stop.What changed
quest worktree run <id> -- <command>runs a job in the checkout inside a systemd user scope. Each run gets its own scope name (quest-worktree-<uuid>.scope), never reused, which avoids the slot-reuse races kixelated/quest#128 kept hitting. A short shell handshake checks that it runs in that scope's v2 cgroup, then waits for the supervisor to acknowledge before itexecs the command. A supervisor that dies never acknowledges, so the command never starts.cgroup.kill, and the supervisor waits forcgroup.eventsto report the scope empty. Once no other run remains in the checkout, the scratch declared atcreateis pruned. If the checkout was released,runrestores it first, through admission. The first signal goes to the command, and a second one kills the whole scope.statuslists the scope, whichsystemctl --user stopends). A run that can't be inspected or confirmed stopped stays recorded, and the checkout stays reserved, with the reason infailure. If an automatic release fails, it is recorded the same way, and a successful command exits 1.releaserefuses while supervised runs are live.create(--scratch), so release needs no caller.releaseno longer takes--scratch.cgroup.killin the caller's unified cgroup, checked before any restore), a systemd 254+ user manager, or unified cgroup v2.kill()treats a missingcgroup.killas success only once the cgroup itself is gone, so a populated scope is never waited on forever.quest worktree remove(maintainer request),quest worktree stop, and clearing a stale inspection failure. Own-PC runner and Agent lifecycle integration move it from Required to Related, since the supervision they needed has landed.Measurements
On this repository, a fresh checkout is 2.9 MB. It grows to 394 MB after
bun install(root andcloudflare/) andbun run buildinsiderun, and is back to 2.9 MB oncerunreleases it. A suspended job keeps that same released footprint until it is restored.Validation
nix develop -c just checkandnix develop -c just testpass locally: 354 CLI tests, 306 Worker tests, and the lifecycle checks.run. A crashed owner with a live scope is retained, then released once the scope empties. An uninspectable run and a run that can't be confirmed stopped are both treated as live. A failed automatic release is reported and keeps the checkout reserved. A command that never started keeps its scratch. A live, foreign-host, dead, or empty registry lock is each handled correctly, and a symlink to another worktree is unlinked. On non-Linux hosts,runfails explicitly. A cgroup withoutcgroup.kill, or a hybrid hierarchy, is refused, andkill()throws on a live group withoutcgroup.killbut not on a removed one (Codex P1 r4342588607).test/worktree.run.test.ts, Linux only, which fail if no user manager is present) check thatrunforwards argument and status values (including a literal$b) and kills a detached orphan, that SIGTERM cancellation with a live descendant exits 143 and releases, and that after the supervisor is SIGKILLed the run is retained untilsystemctl --user stopempties the scope, then released. All three passed 5 of 5 repeated runs, with no leftover scopes.local-gate.mdquest;just checkandjust testpass again on the merge.checkand the explicit non-Linux refusal on Windows: run 39679519210. The first dispatch found that.passedchildPathon Windows, whereresolve("/", ".")is a drive root. That was pre-existing from kixelated/quest#127, and only the tracked-file check stopped it; it is fixed in c5efae0.Decisions
Settled on the orchestrator's recommendation under the maintainer's standing instruction to apply its recommendations (2026-10-21); each is what's implemented.
cgroup.kill) as soon as the command exits./quest/m0/worktree.mdunder Related, not Required, sinceremovedoesn't block them.Suggested follow-ups
quest worktree remove: aquest worktree stop <id>to end a crashed supervisor's leftover processes, and clearing a stale inspection-failure reason on the next successful inspection.quest gateand planned in kixelated/quest#180, not fixed here.🤖 Generated with Claude Code
(Written by Claude Opus 5.5)