Skip to content

feat(runner): run kix jobs on your own machines with quest runner - #188

Merged
kixelated merged 12 commits into
mainfrom
quest/m0/kix/runner
Oct 22, 2026
Merged

kixelated merged 12 commits into
mainfrom
quest/m0/kix/runner

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Implements quest/m0/kix/runner.md and deletes it. The job protocol, hub, and publisher were already on the Worker, but nothing ran jobs. This adds the runner and the Worker pieces it needs. The checks that need live machines move into the kix epic's end-to-end rehearsal, so sandbox, opencode, planning, merge-job, activity, and the m1 quests that only needed this code unblock.

Summary

quest runner (src/runner/, outside the core):

  • quest runner login makes a token on the machine and prints a /runners?name=…&platform=…&key=<sha256> link. Once the signed-in owner approves it, the runner polls GET /api/runner and saves its login. The token itself never leaves the machine.
  • quest runner account add [--console] [--label] runs claude auth login in a new CLAUDE_CONFIG_DIR under the runner's home, then reads claude auth status --json to get the account's kind and a default label. Account ids are random and stored on disk, so they stay the same across restarts. There are also account list and account remove.
  • quest runner [--slots N] holds the WebSocket and reconnects with backoff. It sends a coalesced status whenever anything changes, and an assignment is acknowledged by listing it. A repeated assign counts as one attempt. It honors cancel pushes and the stop/cancel actions in status replies. Requests that are unanswered when the connection drops are resent with the same id. If the hub closes with 4000 ("replaced"), the runner exits and clears its account-reset timer. On SIGINT or SIGTERM it publishes running work, then sends attempts: [] so those jobs move to another runner right away.
  • Each job gets its own worktree, made from a bare clone per repository that fetches the commit by SHA. Operations on a clone queue behind one another, so jobs in several slots never race on it. The agent runs under nix develop --accept-flake-config --no-write-lock-file --command (bootstrap jobs run without a devShell). The checkout is kept between attempts while the branch head still matches, so a paused session resumes where it was. remote.origin.pushurl is set to an invalid value so a push from the checkout fails.
  • Publication compares the working tree to a local commit that matches the branch head. It uses a temporary index, so the agent's own index and commits don't matter, untracked files are included, and ignored files are not. It sends changed files as base64 with their exec and symlink modes; submodules are skipped and logged. A 202 reply pauses the attempt. If kix replies 409 with a newer head, as it does after an earlier reply was lost, the runner fetches that head and checks that it descends from the last head it knew. It then continues from that head and resends only what it lacks.
  • Claude Code adapter: the unmodified binary, run as claude -p --output-format stream-json --verbose with --session-id or --resume. The kix tools (checkpoint, ask, deliver, and context on issue and pull request jobs) are served as a localhost HTTP MCP server with a bearer token per attempt. Its config file is written mode 0600 in the account's home, so the token stays out of ps. --strict-mcp-config and --disallowedTools AskUserQuestion are passed, so every question goes through kix. Account limits are read from a rejected rate_limit_event, assistant rate_limit/billing_error flags, or the text of the result. The adapter interface (src/runner/adapter.ts) is small enough for opencode to fit.
  • Outcomes: deliver is the only path to completed. ask and publication decisions pause the job without sending a result. A turn that ends without delivering is failed, and work in progress is published first in every case. limited keeps the account out of its status reports until the reported reset (or one hour when none is reported), and the runner never switches accounts. cancelled publishes the partial work. Untrusted jobs are refused, and so is an assignment carrying an answer without approvedAt.
  • The prompt includes the bundled guide and the workflow's skill text in full, rather than a slash command the CLI might read as plain text. It also includes the root AGENTS.md and lists the tracked nested AGENTS.md files, which the agent reads for the directories it works in. A system prompt maps the parts of the workflow that assume push, gh, or quest start onto the kix tools. Plan-issue and iterate jobs are told to read their target with context first.
  • Raw output is uploaded as output chunks of at most 1 MiB with contiguous seq, split on character boundaries. Chunks are batched at 256 KiB or 60 s, and uploading stops on truncated.
  • The flake's new runner package wraps quest with QUEST_RUNNER_CLAUDE set to nixpkgs' claude-code. It is allowed through an unfree predicate for that one package. The default quest package is unchanged.

Worker:

  • Runners and accounts: registerRunner (the same hash approved again returns the same runner; another owner's hash is refused), moveAccount, GET /api/runner, and a /runners page linked from the inbox. The page shows each runner's online state and accounts, the account preference order across runners with Up/Down controls (which re-dispatch), and the form that approves a login link. Form posts check the origin.
  • GET /api/runner/context?job=…&attempt=…, fenced to the current attempt, reads the job's target live as the App with the requester's current authority. An issue returns its text and comments. A pull request returns its text, comments, reviews, unresolved review threads, and the checks at its head.
  • A publication on a stale head gets 409 { error, head }.
  • Answer approval: job_questions.approvedAt (migration 0017, after main's 0016_recovery). The requester's own answers are approved as given. Another maintainer's answer leaves the job waiting and sends an approval push. Only the requester can POST …/jobs/<job>/approve it, or answer the question themselves instead. The inbox shows the requester the pending answer with an approve button above the usual form. Everyone else sees the job waiting. The planning decision record uses approved answers only.

Validation

  • just check and just test pass (429 CLI tests with 2 skipped, and 330 Worker tests, after merging main). test/runner.test.ts runs a fake claude and nix against a fake hub. It covers deliver, ask and resume, cancel, limits, failures, untrusted jobs, stop, a repeated assign, shutdown, and reconnect resends. This round adds tests for:

    • a checkpoint whose reply is lost, followed by a delivery;
    • the context tool on a planning job;
    • refusing an unapproved answer;
    • clearing the reset timer on replacement;
    • three jobs prepared concurrently on one repository (this test fails without the queue);
    • nested AGENTS.md listing.

    Worker tests cover approval through the store and the inbox forms, the context endpoint for issues, pull requests, and quests, and the stale-head reply. cloudflare/test/runners.test.ts covers the runner pages.

  • The login test now skips on Windows, like the other runner tests, since quest runner refuses to run there. That was the native Windows CI failure.

  • Run against the real Claude Code 2.1.305 with a Max login, on 2026-10-21: the session listed only mcp__kix__ask and mcp__kix__checkpoint (no AskUserQuestion), called both, and a --resume with the answer continued the session. rate_limit_event arrives as {rate_limit_info: {status, resetsAt}}, which is what the parser expects. Not rerun this round.

  • nix build .#runner builds with claude-code 2.1.291. Nix also reads flakes from worktrees of a bare repository.

Limitations

  • Not yet checked live (the kix epic's rehearsal now covers these): Keychain isolation of CLAUDE_CONFIG_DIR on macOS; whether GitHub serves a fork-network commit by SHA from the upstream URL, which a reassigned job and the lost-reply reconciliation rely on; a real rejected rate_limit_event; the context GraphQL query against real GitHub; and the whole flow against kix.quest.
  • The agent runs with --permission-mode bypassPermissions and the user's environment, including the SSH agent and git credentials, minus GH_TOKEN, GITHUB_TOKEN, and harness variables. That holds for trusted revisions only, and only for answers the runner's owner gave or approved, until the job sandbox lands. Issue and pull request text from context is still untrusted input to that agent; the sandbox quest already plans to contain prompt injection from it.
  • The checkouts of jobs that were fenced out, or abandoned and never came back, stay on disk.
  • The per-clone queue is in-process. Two quest runner processes sharing one runner home are not supported: kix replaces the older connection.

Decisions

The maintainer settled these on 2026-10-22, each as recommended:

  1. Sign-in flow: keep the link carrying the token's hash, approved on /runners, rather than an RFC 8628 device-code flow with a server-side pending table. It needs no new table and the token never leaves the machine. A phished link connects the attacker's machine, which the page warns about; a device code has the same risk.
  2. Questions: keep the kix ask MCP tool with AskUserQuestion disabled, rather than intercepting AskUserQuestion through the SDK's canUseTool control protocol. A kix question pauses the job for hours, so the turn ends either way, and opencode can use the same tools.
  3. Permissions: run agents with --permission-mode bypassPermissions until the job sandbox ships and replaces it; the sandbox quest now records this. Routing permission prompts to kix would pause the job on every tool call.
  4. Limit without a reset time: the account sits out one hour (LIMIT_FALLBACK_MS), rather than being reported usable at once and taking the next job straight into the limit.
  5. Pinning Claude Code: through the separate unfree runner flake package, which lags upstream (2.1.291 against 2.1.305 today). Without Nix, claude comes from QUEST_RUNNER_CLAUDE or PATH.
  6. The quest: quest rm deletes quest/m0/kix/runner.md, which held only live checks. The kix epic's end-to-end rehearsal now runs a Linux and a macOS runner and lists those checks (Keychain isolation, fetching a fork commit by SHA, a real rejected rate_limit_event), so the quests it blocked unblock.
  7. Answers from other maintainers (from Codex's security review): until the job sandbox ships, a job resumes with an answer from a maintainer other than the runner's owner only after the runner's owner approves it. This is recorded in sandbox.md, and shipping the sandbox lifts the gate.

Choices made while addressing the review, for the reviewer to check:

  • Target context: a read-only context tool backed by a live kix read, rather than a snapshot sent with each assignment. A job resumed after a long pause sees new reviews and checks, nothing is stored in D1, and the assignment, which kix resends until acknowledged, stays small.
  • Lost publication reply: the runner reconciles from the head kix reports, rather than kix matching a retried request against the one that landed. That needs no new column and covers a commit that landed with its outcome unrecorded too.
  • Approval scope: the gate covers all three answer kinds, including an account choice, so the rule has no exceptions. The requester can answer in place of a pending answer, so there is no separate decline action.
  • Nested AGENTS.md: listed by path for the agent to read, not inlined, to keep prompts bounded in large repositories.
  • Start over and the gate (merging kixelated/quest#190): Start over needs no approval, since the new job runs the requester's own request and no other maintainer's text reaches the agent; its answer is recorded as approved. As with answers, the requester can start over in place of another maintainer's answer that awaits approval, and no one else can. The decision record follows that PR's retry chain and reads approved answers only.

Suggested follow-ups

  • Reuse quest worktree in the job adapter, as kixelated/quest#186 had planned in the deleted quest: run for supervision, scratch release, and disk admission, remove once a job's checkout is finished (including jobs fenced out or never reassigned here), and stop for a crashed supervisor's leftover processes.
  • quest runner service install to write a systemd user unit or launchd agent.
  • Notify only the job's requester for an approval push. Today it goes to every project member with a device, like other decisions.

🤖 Generated with Claude Code

(written by Claude Opus 5.5)

kixelated and others added 5 commits October 21, 2026 22:39
Quest-Claim: 7f8d2d05130c3865
Checkpoint: quest runner login/account/run, the Claude Code adapter over
claude -p stream-json with kix tools as an HTTP MCP server, job checkouts and
publication, raw output upload, and the Worker's runner page, login approval,
and account preference order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Run jobs on your own machines with quest runner feat(runner): run kix jobs on your own machines with quest runner Oct 22, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: quest runner (login, accounts, and the job loop), the Claude Code adapter, and the Worker's /runners page are implemented. just check and just test pass, and the adapter was run against the real Claude Code with a Max login: kix tools only, ask, then resume. This stays a draft for the six decisions in the description. The main one is whether to delete runner.md and fold its live checks into the kix epic's rehearsal, so sandbox, opencode, and planning unblock (recommended). Still unverified: macOS Keychain isolation, GitHub serving fork commits by SHA, and a live run against kix.quest.

(written by Claude Opus 5.5)

kixelated and others added 2 commits October 22, 2026 08:50
Delete quest/m0/kix/runner.md, which held only checks that need live
machines, so the quests it blocked unblock. The kix epic's end-to-end
rehearsal now covers them on a Linux and a macOS runner, and the sandbox
quest records that it replaces bypassPermissions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 22, 2026 15:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T18:09:14.833628Z 2cb9ac0 Manual request
🔒 Security Review ✅ Completed 2026-10-22T16:08:13.193734Z 61317ab New commits

Security findings

Advisory findings (1)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f79b90976

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/runner/workspace.ts Resolved
Comment thread src/runner/runner.ts Resolved
# Conflicts:
#	quest/m0/kix/runner.md
#	quest/m0/worktree-agents.md
#	src/cli/main.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61317ab3be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/runner/prompt.ts Resolved
Comment thread src/runner/job.ts Resolved
Comment thread src/runner/prompt.ts Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 61317ab3be

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread src/runner/cli.ts Resolved
kixelated and others added 4 commits October 22, 2026 09:27
Until the job sandbox ships, a job resumes with an answer from a maintainer
other than its requester (the runner's owner) only once the requester approves
it in the inbox; the runner also refuses an answer without approvedAt.

- Publication: a stale head gets 409 { error, head }, and the runner continues
  from that head, so a lost reply no longer fails the attempt.
- Plan and iterate jobs read their issue or pull request (reviews, unresolved
  threads, checks) through a read-only kix context tool.
- Serialize operations on each repository's bare clone across slots.
- Clear the account-reset timer when another runner replaces this one.
- Point the agent to nested AGENTS.md files.
- Skip the login test on Windows, where runners do not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renumber the answer approval migration to 0017, after main's
0016_recovery, so the Worker applies both in order. The decision record
follows main's retry chain and reads approved answers only. Start over
records an approved answer, and the requester can start over in place of
another maintainer's answer that awaits approval, as they can answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging after main moved under this PR:

  • Merged origin/main twice (no rebase): first for kixelated/quest#190, then for kixelated/quest#183 and kixelated/quest#194.
  • Renumbered this PR's migration to 0017_answer_approval.sql, after feat(kix): recover jobs from lost and blocked publications #190's 0016_recovery.sql. The Worker tests read the migrations directory in order, so they apply both.
  • Conflicts resolved keeping both sides: the decision record follows the retry chain from feat(kix): recover jobs from lost and blocked publications #190 and reads approved answers only. The publication docs cover both the 409 { error, head } reply and Start over. The inbox history keeps the shared Reply component and shows "Started over from the job's base." The epic README drops the runner, recovery, and epic-branches links, since all three quests are finished. main.ts takes questBranch from core, as main now does.
  • Start over and the approval gate: Start over needs no approval, because the new job runs the requester's own request and no other maintainer's text reaches the agent. Its answer is recorded as approved. As with answers, only the requester can start over in place of another maintainer's answer that awaits approval. A new test in cloudflare/test/publish.test.ts covers this.
  • just check and just test pass locally (430 CLI tests with 2 skipped, 333 Worker tests). quest check --base origin/main passes from a base build, and CI is green on 64e038083bff69bfa82d6afdcb76cf577c5bf13a.

Enabling squash auto-merge on that head.

(written by Claude Opus 5.5)

@kixelated
kixelated merged commit 2cb9ac0 into main Oct 22, 2026
14 checks passed
@kixelated
kixelated deleted the quest/m0/kix/runner branch October 22, 2026 17:33
kixelated added a commit that referenced this pull request Oct 22, 2026
Main's runner (#188) gives kix iterate jobs the iterate skill, which now takes and releases the PR with quest iterate and quest release. Runners hold no GitHub credentials and kix already owns the job's PR, so the system prompt tells the job to skip those steps, as it does for quest start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

Requesting a post-merge review: the rework after your last review of 61317ab (runner-owner approval gate, read-only context tool, lost-publish-reply catch-up, per-repository clone lock, and Start over under the gate) merged without one.

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2cb9ac04f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/runner/workspace.ts
}

/** Each bare clone's queue: jobs on one repository share its config, objects, and worktree list. */
const queues = new Map<string, Promise<void>>();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent runner processes from sharing one worktree

When a second quest runner starts with the same default runner home, the hub replaces the first connection and can immediately reassign its acknowledged jobs while the first process's agents are still stopping for up to STOP_GRACE_MS. This Map serializes repository operations only within one JavaScript process, so the two processes can concurrently reuse and modify the same job worktree and bare repository, mixing or losing the new attempt's changes. Enforce a process-wide lock for the runner home (or use an inter-process lock covering repository/worktree ownership) before accepting jobs.

Useful? React with 👍 / 👎.

? `issue(number: $number) { number title state body author { login } ${comments} }`
: `pullRequest(number: $number) { number title state body isDraft author { login } baseRefName headRefName headRefOid ${comments}
reviews(last: 50) { nodes { author { login } state body submittedAt } }
reviewThreads(first: 100) { nodes { isResolved isOutdated path line comments(first: 50) { nodes { author { login } body createdAt } } } }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Paginate unresolved review threads in runner context

For a pull request with more than 100 review threads, this fetches only the first page and filters unresolved threads afterward, so a newer unresolved thread beyond that page is silently omitted. Because context is the iteration agent's sole source of review feedback, it can deliver without addressing those comments; follow pageInfo cursors until all threads have been examined.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant