Skip to content

feat(worktree): supervise runs in systemd scopes and release automatically - #178

Merged
kixelated merged 5 commits into
mainfrom
quest/m0/worktree
Oct 21, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m0/worktree

Conversation

@kixelated

@kixelated kixelated commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

First slice of Worktree process ownership, following kixelated/quest#127. Commands in a quest worktree checkout can now run supervised, and the checkout is released automatically once they stop.

What changed

  • quest worktree run <id> -- <command> runs a job in the checkout inside a systemd user scope. Each run gets its own scope name (quest-worktree-<uuid>.scope), never reused, which avoids the slot-reuse races kixelated/quest#128 kept hitting. A short shell handshake checks that it runs in that scope's v2 cgroup, then waits for the supervisor to acknowledge before it execs the command. A supervisor that dies never acknowledges, so the command never starts.
  • When the command exits, any descendants left in the scope are killed through cgroup.kill, and the supervisor waits for cgroup.events to report the scope empty. Once no other run remains in the checkout, the scratch declared at create is pruned. If the checkout was released, run restores it first, through admission. The first signal goes to the command, and a second one kills the whole scope.
  • Crash reconciliation: each run is recorded, along with its owner (PID plus Linux start time), before its scope exists. Every registry operation drops runs whose owner has exited and whose scope is empty, then releases the checkout. A crashed supervisor's orphans keep the scratch until they stop (status lists the scope, which systemctl --user stop ends). A run that can't be inspected or confirmed stopped stays recorded, and the checkout stays reserved, with the reason in failure. If an automatic release fails, it is recorded the same way, and a successful command exits 1.
  • release refuses while supervised runs are live.
  • Scratch is declared at create (--scratch), so release needs no caller. release no longer takes --scratch.
  • Stale registry lock: the lock directory holds a single file named after its owner and is created with an atomic rename. If that owner has exited on this host, its lock is reclaimed by unlinking that exact file, so a later owner's lock is never removed. Live locks are never stolen, and neither are locks held from another host.
  • Folded in from kixelated/quest#127: a scratch symlink that points to another worktree is unlinked rather than refused (lstat on the final component).
  • Hosts without the backend fail explicitly: non-Linux hosts, and hosts without Linux 5.14+ (cgroup.kill in the caller's unified cgroup, checked before any restore), a systemd 254+ user manager, or unified cgroup v2. kill() treats a missing cgroup.kill as success only once the cgroup itself is gone, so a populated scope is never waited on forever.
  • CI starts a systemd user manager (linger) so the real-scope fixtures run.
  • The quest file now covers only what remains: quest worktree remove (maintainer request), quest worktree stop, and clearing a stale inspection failure. Own-PC runner and Agent lifecycle integration move it from Required to Related, since the supervision they needed has landed.

Measurements

On this repository, a fresh checkout is 2.9 MB. It grows to 394 MB after bun install (root and cloudflare/) and bun run build inside run, and is back to 2.9 MB once run releases it. A suspended job keeps that same released footprint until it is restored.

Validation

  • nix develop -c just check and nix develop -c just test pass locally: 354 CLI tests, 306 Worker tests, and the lifecycle checks.
  • Unit tests, with fake scopes, cover these cases. Concurrent runs keep scratch until the last one stops, and a released checkout is restored by run. A crashed owner with a live scope is retained, then released once the scope empties. An uninspectable run and a run that can't be confirmed stopped are both treated as live. A failed automatic release is reported and keeps the checkout reserved. A command that never started keeps its scratch. A live, foreign-host, dead, or empty registry lock is each handled correctly, and a symlink to another worktree is unlinked. On non-Linux hosts, run fails explicitly. A cgroup without cgroup.kill, or a hybrid hierarchy, is refused, and kill() throws on a live group without cgroup.kill but not on a removed one (Codex P1 r4342588607).
  • Real systemd fixtures (test/worktree.run.test.ts, Linux only, which fail if no user manager is present) check that run forwards argument and status values (including a literal $b) and kills a detached orphan, that SIGTERM cancellation with a live descendant exits 143 and releases, and that after the supervisor is SIGKILLed the run is retained until systemctl --user stop empties the scope, then released. All three passed 5 of 5 repeated runs, with no leftover scopes.
  • Merged main (kixelated/quest#175, kixelated/quest#177, kixelated/quest#179), dropping the links to the deleted local-gate.md quest; just check and just test pass again on the merge.
  • I dispatched the Check workflow on this draft. On c5efae0, all five jobs pass, including the systemd fixtures under linger in check and the explicit non-Linux refusal on Windows: run 39679519210. The first dispatch found that . passed childPath on Windows, where resolve("/", ".") is a drive root. That was pre-existing from kixelated/quest#127, and only the tracked-file check stopped it; it is fixed in c5efae0.

Decisions

Settled on the orchestrator's recommendation under the maintainer's standing instruction to apply its recommendations (2026-10-21); each is what's implemented.

  1. ✅ Automatic release trigger: scratch is pruned when the last supervised run in a checkout stops, so a whole job belongs in one run.
  2. ✅ Leftover descendants: SIGKILLed through the scope (cgroup.kill) as soon as the command exits.
  3. ✅ Runner and agent-lifecycle quests: list /quest/m0/worktree.md under Related, not Required, since remove doesn't block them.
  4. ✅ CI user manager: linger is enabled in the Check job so the real-scope fixtures run in CI.

Suggested follow-ups

  • Added to the quest's remaining scope, alongside quest worktree remove: a quest worktree stop <id> to end a crashed supervisor's leftover processes, and clearing a stale inspection-failure reason on the next successful inspection.
  • Terminal Ctrl-C reaches the command twice, once from the terminal and once forwarded. It is shared with quest gate and planned in kixelated/quest#180, not fixed here.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 21, 2026 14:06
Quest-Claim: 4465a18959f4b93c
…cally

`quest worktree run <id> -- <command>` runs a job in a checkout inside a
systemd user scope named once per run. A handshake proves the shell is in
that scope's v2 cgroup before the supervisor lets the command start, so a
dead supervisor can never start it. When the command exits, descendants
left in the scope are killed, and once the scope is empty and no other run
remains, the scratch declared at `create` is pruned.

Runs are recorded before their scope exists. Every registry operation
reconciles runs whose owner (PID plus Linux start time) exited and whose
scope is empty, so a crashed supervisor's checkout is released only after
its orphans stop. A run that cannot be inspected or confirmed stopped stays
recorded and the checkout stays reserved, with the reason noted. `release`
refuses while runs are live. Non-Linux hosts fail explicitly.

The registry lock now names its owner in a file inside the lock directory
and is taken by an atomic rename, so the lock of a process that exited on
this host is reclaimed by unlinking that exact owner file, never a later
owner's lock.

Also: scratch is declared at `create` instead of `release`, a scratch
symlink to another worktree is unlinked rather than refused, and CI starts
a systemd user manager for the real-scope fixtures (orphan cleanup,
cancellation, crash reconciliation).

The quest now covers only `quest worktree remove`; the runner and agent
quests no longer wait on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Own worktree processes and release checkouts automatically feat(worktree): supervise runs in systemd scopes and release automatically Oct 21, 2026
resolve('/', '.') is a drive root on Windows, so '.' passed the check
there and was refused only by the tracked-file check. Test the path's own
components instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Owner Author

/quest-start outcome

This lands the first slice of process ownership: quest worktree run supervises each job in its own systemd user scope, kills orphans when the command exits, and prunes the scratch declared at create once the checkout's last run stops. Crashed supervisors are reconciled once their scope is empty, and the registry lock of an exited process is reclaimed. The #127 symlink follow-up is folded in. quest worktree remove (maintainer request) is recorded as the quest's remaining scope.

Local just check and just test pass. A dispatched Check run passes on c5efae0 across all five jobs: https://github.com/kixelated/quest/actions/runs/39679519210

This PR stays a draft. Four decisions are listed under Decisions needed in the description, each with a recommendation already implemented: what triggers a release, killing leftover descendants, unblocking the runner and agent quests, and the CI user manager.

(Written by Claude Opus 5.5)

Drop links to the deleted local-gate quest and add stop and stale-failure clearing to the remaining worktree scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 21, 2026 22:18
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T22:30:53.153820Z 6fc0924 New commits
🔒 Security Review ✅ Completed 2026-10-21T22:31:26.133880Z 6fc0924 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bfcac77340

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli/supervise.ts Outdated
kixelated added a commit that referenced this pull request Oct 21, 2026
Count recent admissions against free memory as well as load, define the
spacing band, require atomic minute-long admission records, and pin down
the claim directory, clock-step, and Ctrl-C rules. The Ctrl-C fix covers
`quest worktree run` only if #178 has landed. The quest is
now [M]. Also fix the seed test path in the inbox-accuracy quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before Linux 5.14 a scope's cgroup has no cgroup.kill, so leftover descendants were never killed and the supervisor waited on a populated scope forever. run now refuses such hosts up front, and kill() ignores a missing cgroup.kill only once the cgroup itself is gone, without ever creating the file. Retitle the worktree quest [M] and point the Ctrl-C follow-up at #180.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated merged commit e04029d into main Oct 21, 2026
14 checks passed
@kixelated
kixelated deleted the quest/m0/worktree branch October 21, 2026 22:38
kixelated added a commit that referenced this pull request Oct 21, 2026
#178 landed, so the gate headroom quest fixes the double
Ctrl-C in both commands unconditionally, and the worktree quest links to
it instead of the planning PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 21, 2026
* docs(quest): plan gate admission with headroom

Follow-up of the contention measurements in #177: admit at once while the
host has room, fix the two-window spacing, a stable claim directory,
backward clock steps, and forwarding Ctrl-C once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): address review of the gate headroom plan

Count recent admissions against free memory as well as load, define the
spacing band, require atomic minute-long admission records, and pin down
the claim directory, clock-step, and Ctrl-C rules. The Ctrl-C fix covers
`quest worktree run` only if #178 has landed. The quest is
now [M]. Also fix the seed test path in the inbox-accuracy quest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): count unexpired gate records and survive large clock steps

Derive the recent-admission count from unexpired records with a
record-then-count step instead of slot numbers, which expiry leaves with
holes. Rewrite a future-dated record's time to now rather than discarding
it, so a large backward clock step still spaces admissions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): elect one gate admission at a time under a lock

Record-then-count prevents over-admission but can livelock when a whole
burst withdraws together. Suggest serializing prune, count, and record
under a short host-wide lock reclaimed from exited owners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): share the Ctrl-C fix with quest worktree run

#178 landed, so the gate headroom quest fixes the double
Ctrl-C in both commands unconditionally, and the worktree quest links to
it instead of the planning PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): state gate admission as requirements and hazards

Four review rounds kept finding edge cases in the prescribed records,
lock, and clock-step mechanisms. Keep the decisions and the three-band
rule, state what the admission mechanism must guarantee, and list every
review finding as a hazard the implementation must test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): require a claim location stable across login sessions

/run/user/<uid> comes and goes with logind sessions, so choosing by its
existence can split callers again. State the requirement and the
hazards instead of a directory rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(quest): keep forwarding SIGHUP from the gate

A SIGHUP sent with kill to Quest alone cannot be told apart from a
terminal hangup, so skip forwarding only SIGINT in the foreground group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant