[scuttletoby]

Fork tests can run its Linux jobs on this repository's own machines (the fleet). Each run picks per job, and the repository variable CI_RUNNERS controls it: auto (also unset), fleet or github. Until a fleet host runs the dispatcher, auto behaves as today: everything is GitHub-hosted, with a note that says why.

Runbook, host setup, owner steps, a real-runner check and the full failure table: .github/fork-fleet.md.

Design: no runner waits for work

The first version had a standing self-hosted runner and a job-started hook that refused untrusted jobs. Security lens s1 showed that this hook is a pre-job step: when it fails, later if: always() / failure() steps still run. A stranger's pull request that edits the workflow to name the fleet label could then run code on the host, and rewrite the runner's .env to remove the hook. The fix removes the cause: an untrusted job never finds a runner at all.

1. Route is now the last step of changes (no extra job). It calls no API and reads no secret. In auto, a pool counts only when its dispatcher's heartbeat (a repository variable, at most 180 s old) reports a free slot. For each fleet job it picks two labels: the pool's label, and a label made for this run (<pool label>-<job key>-<16 random hex>); each job's runs-on adds the attempt (-a<attempt>). Untrusted runs go GitHub-hosted.

2. The dispatcher (fork-fleet.py dispatch) is the root of the design. It runs as root on the fleet host, from a root-owned copy of main's script and config. It starts a one-job (JIT) runner only when all of these hold:

   • the queued job asks for exactly that label pair;
   • GitHub's own record of the run passes the trust rule, with each pull request read from pulls/<n>;
   • the job's attempt, and the attempt in its label, are the run's attempt it read;
   • no other queued job, of any run, could take that runner.

   The launcher (fork-fleet-launch.sh) runs that runner from a fresh hard-link copy of a root-owned, unconfigured runner template. It runs as the runner user of a free slot (one user per slot, no unit shares one), in a transient systemd unit (whole process group killed on stop, 1 h at most). HOME and TMPDIR are inside the copy, which is 0700 to that user. Before a user is reused, every process of it is killed and its old copies are removed. Each unit has its own /tmp, /var/tmp, /dev/shm and /run, no /home, and a network namespace of its own with internet only (round 3). Its environment comes from a root-only file that is deleted once the runner has started: the JIT config, the run and attempt it was made for, HOME, TMPDIR and the hook path. A job that names a fleet label by itself, from a changed workflow or a fork's pull request, stays queued with no runner.

3. The hook (fork-fleet-hook.sh → gate, root-owned) is the second line. If the job is not the run and attempt its runner was made for, or if it breaks the trust rule, the hook kills the job's Runner.Worker (SIGKILL). The job then fails with no later step run, always() ones included. A gate that crashes refuses the job.

4. Bounded waits. fleet_watch (a job in each run that sent a job to the fleet) cancels the run with an error naming the job when a fleet job waits 10 minutes (fleet mode: 2 hours). The stateless sweep (every 10 min) re-runs such a run once, GitHub-hosted, in auto/github. In fleet mode it re-runs nothing, and it cancels with an error any trusted run whose fleet job waited 2 hours. A re-run of failed jobs has no watch of its own, so this bounds it. The sweep never touches an untrusted run.

Why this design: the hook alone cannot be made safe, because a failed pre-job step does not stop the job. Killing the worker does stop it, but it still depends on the hook's environment. With a standing runner, a job could rewrite that environment for the next job. With one-job runners made only for a trusted run's own labels, untrusted code needs both a runner it can never get and a gate it cannot change. The PC pool needs no standing connection either: its runners start on demand.

Runner source this relies on (actions/runner main, read 2026-11-12)

• src/Runner.Worker/JobExtension.cs ~303-315: the hook path is read from the worker's process environment, and the hook is a pre-job step with condition always(). Container setup is added after it, so the hook runs first.
• src/Runner.Worker/JobHookProvider.cs:74: the hook's environment starts empty and gets the process environment plus GITHUB_*/RUNNER_*, not the workflow's env:. So FORK_FLEET_RUN/FORK_FLEET_ATTEMPT from the root-only file can be trusted.
• src/Runner.Listener/JobDispatcher.cs:1179-1219: when the worker dies, the listener marks the job failed, and no later step runs.
• src/Runner.Listener/CommandSettings.cs:112-138: ACTIONS_RUNNER_INPUT_JITCONFIG is read from the environment and then deleted, so the JIT config is never in argv.
• src/Runner.Listener/Program.cs:172-194: the runner loads .env from its own directory and lets it override its environment. That is why the launcher refuses a template with .env, .path, .runner or .credentials, and uses each copy for one job only.

Owner decision: upstream sync on GitHub-hosted runners only. (2026-11-12) The upstream sync (fork-follow.yml, with fork-checks.yml) and the sweep (fork-fleet.yml) run only on GitHub-hosted runners, never fleet or self-hosted ones, whatever CI_RUNNERS says. Enforced by test_the_upstream_sync_runs_on_github_hosted_runners_only: every job's runs-on must be a fixed GitHub-hosted image (no expression, no list), and the release that fork-follow.yml calls passes a ref, so its route never runs. Changing one runs-on in fork-follow.yml to fleet-vm-t3code, [fleet-vm-t3code], self-hosted or a ${{ vars.CI_RUNNERS … }} expression turns it red. The push run that the sync's move of main starts is GitHub-hosted too (round 3, item 3): test_the_upstream_syncs_push_run_is_hosted_in_every_mode.

Fix round 3 (lens comments S-1 and S-2)

1. Network (major, S-1). The vm pool runs on VM106, the owner's workspace, and a job shared its network: the host's loopback services (an unauthenticated web GUI on 127.0.0.1, T3 on 3773), the LAN, Tailscale and the Docker bridges. Now each runner user has a network namespace of its own (fork-fleet-<user>), with its own loopback and one veth out through the host. The nft table inet fork_fleet refuses the host itself (input chain), RFC 1918, CGNAT 100.64.0.0/10, link-local, loopback, the other reserved ranges and all of IPv6, and masquerades the rest. The unit's IPAddressDeny refuses the same ranges again in the kernel, should the table ever go. The jobs get a resolv.conf with public nameservers (the host's resolver is on its loopback). On a host with Docker, the launcher adds the two accept rules Docker's DROP forward policy needs, once each. The launcher (re)makes the namespace and the table at every start, under the launch lock, and starts no runner when anything fails (no nft, IPv4 forwarding off, no nameserver, an ip or nft error). The address range (NET_BASE, default 10.204, a /30 per user from its uid) and the resolv.conf path are settings in launch.conf; two users whose uids would share a /30 are refused. New verb check-net URL HOST:PORT... proves it: it runs a probe as a free runner user, in the very sandbox a runner gets. The URL must answer, and every target must be closed from inside. It refuses a target the host itself cannot reach, because that refusal would prove nothing. Host step 6 runs it against every IPv4 service the host listens on, from each of its addresses (loopback, LAN, Tailscale, Docker bridges). That is wider than a fixed list, and it covers a GUI port like 8384 whenever that port listens. The "sandboxed runner VM" wording in fork-fleet.json and the runbook now says what is true.
2. Root's /dev/shm delete (major, S-2). Root's find … -exec rm -rf in the shared /dev/shm could be led by a symlink that another runner user swapped in, to delete any file. The delete is gone. Each unit now has its own /run, /dev/shm, /tmp and /var/tmp (memory file systems that go with the unit), its own IPC (PrivateIPC), no /home (ProtectHome) and no view of other users' processes (ProtectProc=invisible). So nothing is left there to clean. Root's other rm -rf (prune) removes a copy only once no unit can write it; a comment there says why that is safe.
3. The upstream sync's push run (major, S-2; S-1's question). The daily sync moves main with the owner's token, so the push run after it passed the trust rule and could run upstream's code on the fleet. Now fork-follow-push.sh advance first pushes the tag fork-follow/synced/<sha> for the commit it moves main to (upstream_sync.tag_prefix in the config). It moves main only when that tag push worked, and a re-run tags a commit an earlier run moved. The trust rule (made_by_the_sync, in route, dispatcher and gate) asks the repository anonymously (git ls-remote --exit-code, with no git config of whoever runs it) whether the pushed commit has that tag. A tagged commit, a short sha, a git failure or a timeout is untrusted, so GitHub-hosted in every mode. Why upstream cannot fake it: only a holder of push rights here (the owner, the sync's token) can make or remove a tag in this repository. Nothing in a commit (message, author, files) or upstream (branches, tags, pull requests) changes that. And a tag can only ever send a run GitHub-hosted.
4. Minors.
   • Tests now fail when the "own group only" check or the launch.lock lock is removed.
   • A pool its dispatcher closed or reports down (a new actions/runner release, a host error), or a stale heartbeat, is a ::notice:: on the run page. A pool that was never set up stays a log line.
   • Host steps 2 and 3 start with set -euo pipefail. Step 2 checks the new template (sha256, unconfigured, version file) before it replaces anything, and the old one stays until then. Step 3 fetches all four files and checks them (bash -n, the gate's refusal outside a runner) before it installs any. It writes SOURCE last.

Residual: once synced, upstream's code is in main. The owner's later pull requests and pushes build on it, so they run it on the fleet. Sync tags accumulate, one per sync. Keep them about 30 days, because a run can be re-run for that long.

Red → green:

• 16 guards broken by hand, one at a time. Each was red in the unprivileged suites and green once restored:
  • the input chain and the private ranges;
  • the namespace property, the flock and the group check;
  • the uid collision, ip_forward and the resolv nameserver;
  • Docker's -C answer;
  • refused() skipping the sync check;
  • a git failure or exit 0 counted as "not synced";
  • the tag push removed, or removed on the already-moved path;
  • the closed-pool notice;
  • the tag pushed after main.
• The input chain and the private ranges are also red in LauncherAsRoot.
• The old fork-follow-push.sh fails the 3 new sync tests.
• The /dev/shm swap race deletes root's victim file on the old launcher, 5 of 5 runs; the new launcher has no such delete.
• The new suites cannot run against the old launcher or the old fork-fleet.py at all (the new settings and made_by_the_sync do not exist there). So the mutations are the per-guard proof.

Real systemd, in a throwaway privileged ubuntu:24.04 container (removed after):

• A real start unit had private /tmp, /var/tmp, /dev/shm and /run, no /home, no view of root's process and no SysV shm. The system was read-only, and only its copy, HOME and TMPDIR were writable.
• Its resolv.conf had the public nameservers, and its only links were lo and its veth.
• https://github.com answered 200. Closed from inside: the host's 127.0.0.1:3773, the host's own address, the Docker gateway, a LAN address and a Tailscale address.
• With the nft table deleted, a control connection from outside the unit was open while the unit still found it closed. So IPAddressDeny holds on its own.
• check-net exited 0 and made the table again. It refused a Tailscale target the container itself could not reach.

Fix round 2 (lens comments S-1 and S-2)

1. One user per slot (major, both lenses). All runners shared one user and home, so a compromised trusted job could read the other slot's start config and tokens, change how it starts, or plant files in its home or /dev/shm for a later main job to save to the cache. Now: RUNNER_USERS lists one user per slot; the launcher refuses two slots that are one user, root, a user in another group or a lingering one, and picks a user no loaded unit runs as. HOME and TMPDIR are in the copy (0700 to its user) and go with it. Before a user is reused, the launcher kills every process of it (kill -1 as that user, repeated until pgrep finds none), removes its /dev/shm files and deletes its old copies; it refuses to start if anything is left. (Round 3 replaced the /dev/shm cleanup with a private /dev/shm per unit.) Root tests (LauncherAsRoot, a new step in the workflows job, run with sudo) use two real users: one cannot read the other's copy, home or config, and a file in the home or /dev/shm and a detached process are gone for the user's next job.
2. Re-run after a cancel (both lenses). Root fix: the attempt is in the label. Each routed runs-on adds -a<github.run_attempt>, and the dispatcher starts a runner only when the label's attempt is the run's. A job of attempt 2 then can never be given an idle runner made for attempt 1. Chosen over stopping runners fast because it is structural and does not depend on timing (the idle runner is still stopped, as cleanup).
3. One host fails, two dispatchers. One lock per pool (--lock-dir, dispatch-<pool>.lock; the old --lock is gone and no prefix of it matches). A host error leaves only that pool alone for the pass (no cleanup, no start, heartbeat free: 0 with the reason); the other pools go on.
4. Free-slot count. A test fails when queued, not yet started jobs are left out of the free count.
5. Runbook steps 1-2 safe to run twice. Step 1 keeps existing users and adds each cron/at deny line once. Step 2 stops the dispatcher, swaps the template with mv -T via .old, writes .fork-fleet-version, and starts it again. The launcher checks that TEMPLATE and WORK_ROOT are on one filesystem (cp -al) and says so. The intended SC2064 has its disable comment.
6. Runner self-update: fail closed. The dispatcher reads the newest actions/runner release once an hour; the launcher's ready compares it with the template's .fork-fleet-version. An older template, or none, starts no runner on that pool: the heartbeat is free: 0 with a why telling the owner to refresh the template (host step 2), so auto goes GitHub-hosted. Cleanup goes on.

Also fixed (lens minors)

• s2-1, re-runs: runs-on also checks vars.CI_RUNNERS. A re-run follows the setting as it is now (github → hosted, auto → hosted for a re-run of failed jobs, fleet → the run's labels while it is still fleet). The watch reads its own attempt, and the sweep reads the latest attempt of every run.
• s2-2: route is a step of changes; the route job is gone.
• s2-3: a truncated or garbled reply (http.client.IncompleteRead, ValueError and the like) is a warning and never turns a run red, in watch, sweep and dispatch (route calls no API).
• s2-4 / s1: a runner-writable .env can no longer turn off the hook (see the design).
• s1, sweep trust: the sweep uses the dispatcher's rule on GitHub's record and leaves a stranger's run alone.
• s1, no silent 24 h: fleet_watch and the sweep (above).
• s1, a person's cancel stands: now tested, and removing it turns a test red.
• Doc: in fleet mode, shards that only print a skip run GitHub-hosted.

Found while testing, and fixed:

• the launcher's JIT config check was a bash regex error that refused every config;
• WORK_ROOT must be 0711 so the runner user can reach its copy;
• the launcher's EXIT trap named a local, so every start exited 1 under set -u and left the environment file behind;
• the dispatcher could make a runner for attempt 1 while the owner's re-run had moved the job to attempt 2, and the gate would then kill a trusted job. It now skips such a job for one pass;
• the dispatcher's lock directory under /run is created after a boot.

Owner steps (none done here: no settings, variables, secrets, runners or repositories were changed)

# Setting (optional; unset = auto):
gh variable set CI_RUNNERS -R scuttletoby/t3code --body auto
# The dispatcher's token: fine-grained PAT, resource owner scuttletoby, only scuttletoby/t3code, repository
# permissions Administration: Read and write, Actions: Read-only, Variables: Read and write,
# Pull requests: Read-only. It is stored on the fleet host only (root, 0600), not as a repository secret.
# The first design's route token is no longer used:
gh secret delete FLEET_RUNNERS_TOKEN -R scuttletoby/t3code   # if it was set
# Approval for outside contributors must stay all_external_contributors:
gh api repos/scuttletoby/t3code/actions/permissions/fork-pr-contributor-approval --jq .approval_policy

Host setup on VM106 is in .github/fork-fleet.md as paste-ready root commands:

• one runner user per slot (fork-runner-1, fork-runner-2): no home, no sudo, no docker group, no linger, no cron or at;
• what the runners' network needs (step 1b): nftables, IPv4 forwarding and the jobs' resolv.conf with public nameservers;
• a root-owned runner template, checked by sha256, with its version in .fork-fleet-version, swapped with the dispatcher stopped and only once the new one is checked;
• the four files from a pinned main commit;
• /etc/fork-fleet/launch.conf and the token;
• the dispatcher unit with --pool vm;
• step 6: check-net from inside a runner, against every service the host listens on.

The pc pool stays off until a launch-pc ssh wrapper exists. Whether the PC joins is an owner decision.

Real-runner check before the fleet takes jobs: the steps are in the runbook. They need a throwaway private repository under the owner's account (the owner's call; none was created) and check:

• a refused job's always() step never runs;
• the environment file is gone after the start;
• the runner user cannot write the template;
• a bare fleet label finds no runner;
• the owner's re-run gets a runner for attempt 2;
• a cancelled run's idle runner never takes the re-run's job;
• the two slots run as different users that cannot read each other's copy, and nothing a job leaves reaches the next one.

Failure table: what happens when this runs again, stops half-way, or meets existing/old/empty state?

The full table (about 50 rows) is in the runbook. All state lives in GitHub (runs, jobs, runner registrations, heartbeat variables) and in systemd on the host. The dispatcher keeps only pacing in memory.

Part	Situation	What happens
Route	CI_RUNNERS misspelled	changes fails with a clear message: red. Unset or empty means auto.
Route	No, old or garbled heartbeat	That pool counts no slot. auto: GitHub-hosted with a note. fleet: queue on vm.
Route	Untrusted run, release run, run from before this PR	GitHub-hosted.
Route	A push to main of a commit tagged fork-follow/synced/<sha>, or the tag cannot be read	Untrusted: GitHub-hosted in every mode; the dispatcher and the gate refuse it too.
Sync	Stops between its tag and main	The tag names a commit main does not have yet: harmless. The next sync tags again and moves main.
Sync	The tag push is refused	main is not moved; the run fails and says why.
Route	Re-run of failed jobs / of all jobs	Follows the setting as it is now (see s2-1). For all jobs, route runs again: fleet only if the owner re-ran it.
Dispatcher	Crash after generate-jitconfig, before start	An offline registration with no unit: the next pass deletes it, and the pass after starts the job again (at most 3 tries per job).
Dispatcher	Crash after start	The next pass sees the unit and starts nothing twice.
Dispatcher	Launcher killed half-way through the copy	No unit: the registration is deleted, prune removes the half copy and any environment file, and the job is started again.
Dispatcher	Restart or reboot	It reads everything again. The lock directory is made again. A running job fails as a lost runner (auto: the sweep re-runs it hosted), and a queued job gets a new runner.
Dispatcher	Two dispatchers for one pool / for different pools	The second fails on that pool's lock / both run.
Dispatcher	One pool's host fails	Only that pool is skipped for the pass, with a free: 0 heartbeat that says why.
Dispatcher	Template older than the newest runner release, or no version file	No start on that pool, cleanup goes on, heartbeat free: 0 says to refresh the template.
Dispatcher	The job was cancelled before its runner took it	The runner is stopped after 3 passes, or at once if another queued job asks for all its labels. A re-run's job never can (its label names its attempt).
Dispatcher	Re-run between reading the run and its jobs	Skipped for one pass, then a runner is made for the new attempt.
Dispatcher	A stranger's queued job could take the runner	No launch. The trusted job falls back (auto: hosted after 10 min; fleet: an error at 2 h).
Dispatcher	Down	Heartbeats go stale. auto: hosted. fleet: the watch reports at 2 h.
Launcher	Template configured, a symlink or writable by others; conf or hook writable by others; bad input; a name that ran already	It refuses before it starts anything.
Launcher	systemd-run fails	The environment file is removed on exit, and the copy goes at the next prune.
Launcher	A job left processes or files in its home	All killed or removed before that user's next job; if anything stays, it refuses. Its /tmp, /var/tmp, /dev/shm and /run went with its unit.
Launcher	The runners' network cannot be made (no nft, IPv4 forwarding off, no nameserver, an ip/nft error)	It refuses to start a runner and says why.
Launcher	The nft table is deleted while a runner runs	IPAddressDeny still refuses the private ranges in the kernel; the next start makes the table again.
Launcher	RUNNER_USERS shares a user, or TEMPLATE/WORK_ROOT on two filesystems	It refuses, with a message.
Host setup	Steps 1-2 run again	Users kept, deny lines added once; the template swap happens with the dispatcher stopped.
Hook	Wrong run or attempt, or an untrusted run	Runner.Worker is killed: no step runs, always() included.
Hook	Broken installed config, or a runner the dispatcher did not start	Refused (fails closed).
Hook	No Runner.Worker above the hook (a renamed worker)	It can only exit 1, and says so. The real-runner check covers this.
Watch	Truncated or garbled reply	A warning, then it tries again. Past the bound it stops watching, never red. The sweep is the fallback.
Sweep	Stops after the cancel, before the re-run	The next sweep re-runs it (the job waited 10 min and never started).
Sweep	Runs again on its own re-run	That run is untrusted (github-actions), so it is left alone: one re-run at most.
Sweep	A person cancels; branch moved; a real test failure	Stands; not re-run; stays red.
Old state	Old copies or environment files	prune removes them on every pass.
Old state	First design's token or a standing runner	Unused; delete them. The fork has no runners today.
Old state	Installed copy older than main	The old rule applies until re-installed. /opt/fork-fleet/SOURCE names the commit.
Empty state	No heartbeat variables yet	The dispatcher creates them on its first pass.

Tests

.github/scripts/fork-fleet.test.py has 140 tests (4 of them, LauncherAsRoot, only as root: a separate sudo step in the workflows job, which now installs nftables for the network test), run by the workflows job:

• config;
• the trust rule from the environment and from GitHub's record;
• the gate;
• route in every mode;
• watch;
• sweep;
• the dispatcher against a fake GitHub and a fake host;
• the launcher, run for real against fake systemd-run/systemctl/pgrep/setpriv/ip/nft, and as root with two real users and real namespaces;
• the sync check, against fake and real git, and fork-follow.test.sh for the tag the sync pushes;
• the wiring of every workflow against the config.

GateProcess models the s1 blocker. A process named Runner.Worker runs the installed hook, then an always() step that writes a marker. A refused job leaves no marker, in these cases:

• a fork's pull request;
• an old runner seeing a re-run;
• a runner the dispatcher did not start;
• a crashing gate.

A mutant gate that only exits 1 does leave the marker.

Fix round 2 mutation check: 18 more guards, each broken by hand, red, then green once restored. They cover the label-attempt check, queued jobs in the free count, the per-pool skip on a host error, the closed path for an outdated template, per-pool locks, distinct runner users, the busy-user check, the process kill, the /dev/shm cleanup (red in LauncherAsRoot), removing old copies, the same-filesystem check, the version compare, the version file on start, HOME/TMPDIR in the copy, the 0700 copy, the -a<attempt> in runs-on, the root test step, and the owner decision (one fork-follow.yml or sweep runs-on changed to the fleet label).

Mutation check: 15 mutations, each red, then green once restored (details in the fix-round comment). They cover:

• a person's cancel;
• the SIGKILL;
• the run and attempt check, in the gate, the dispatcher and runs-on;
• the sweep's and the dispatcher's trust rule;
• the reply errors;
• the launcher's input and template checks, and its three bugs;
• both fleet-mode sweep cases.

Locally:

• actionlint is clean on fork-*.yml;
• node .github/scripts/fork-ci-tests.cjs passes, and so do fork-contracts.test.py and fork-release-workflow.test.py;
• fork-tests-drift.cjs matches the record;
• bash -n passes on both new scripts;
• gitleaks finds nothing in the diff;
• shellcheck 0.11.0 (pinned by sha256) is clean on the launcher, the hook and fork-follow-push.sh;
• LauncherAsRoot passes as root in a throwaway ubuntu:24.04 container;
• all fork suites pass with GH_TOKEN/GITHUB_TOKEN unset and an empty GH_CONFIG_DIR.

Round 3, at c562e0143e: fork-fleet.test.py 136 OK and 4 root-only skipped, fork-follow.test.sh all passed, fork-ci-tests.cjs 0, drift check matches, actionlint 1.7.12 clean, oxfmt clean, LauncherAsRoot 4 OK in the container. No CI: GitHub Actions is disabled on this repository ("Actions has been disabled for this repository"), so this head has no checks. Nothing was re-enabled.

General vs project

• General: fork-fleet.py (route, watch, sweep, dispatcher, gate), fork-fleet-launch.sh and the hook. They fit any repository and any pools.
• Project: everything in .github/fork-fleet.json: the repository, actors, branches, pools (labels, slots, heartbeat variables, launch commands), jobs (shards, gates, memory) and the time limits.

Follow-up

Plan row 9 is not in this PR: the config path as an input, a hosted: false option, at most one heavy job per run on the PC, a runner pool by draft state, and the SkyRPG fleet requirements as tests. This PR is already about 2,300 changed lines, so row 9 will be a stacked follow-up PR on this branch.

Open for the owner

• Merge, then the host setup and the real-runner check, in that order. Nothing runs on the fleet before the dispatcher does.
• Pull requests by anyone but scuttletoby (bots included) always run GitHub-hosted. Allowing them would be a change to trusted. Upstream sync, and the push run its move of main starts, stay GitHub-hosted by owner decision (above).
• Upstream's code reaches the fleet once it is in main: the owner's later pull requests and pushes build on it.
• Sync tags fork-follow/synced/<sha> accumulate, one per sync. Keep them about 30 days (the re-run window).
• Host step 6 (check-net) runs again after any network change on the host and whenever a new service listens.
• When actions/runner has a new release, the fleet stops starting runners until host step 2 is run again with it (fail closed).
• fleet mode runs up to 16 Linux jobs two at a time on vm. A slow full run can come near the 2-hour bound. auto is meant as the everyday mode.
• The sweep adds 144 short runs a day to the Actions list. They cost no minutes, because the repository is public.

Round cap: merged with minors (round 5, both lenses SAFE)

Lenses r5: #13 (comment) and #13 (comment) (no blocker, no major). Minors carried to one follow-up PR:

• .github/fork-fleet.md:341 runbook step 2: [ -x run.sh ] && ... under set -e does not stop the step; a broken runner is swapped in (fails closed, fleet down until step 2 runs again).
• .github/fork-fleet.md:318 step 1b: a rerun after a runner started always warns "10.204 is in use"; ignores a later NET_BASE.
• fork-fleet-launch.sh:200 + runbook step 6: the home connection's public address is neither blocked nor probed (router admin page / port forwards). Optional extra deny list + probe.
• .github/fork-fleet.md:415 step 6 lists IPv4 listeners only (ss -4), missing dual-stack ones.
• fork-fleet-launch.sh:290: three 1 GB RAM tmpfs per runner, no runner memory cap; dispatcher's memory check does not count them.

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2) · read at: f2d6e15
covered: the whole diff against main 8b0ad2c (fork-fleet.py route/gate/capacity/sweep, fork-fleet.json, fork-fleet.md, fork-fleet.yml, every fork-tests.yml change, fork-ci-tests.txt), plus what it calls into: fork-release.yml's workflow_call into fork-tests.yml (it passes no secrets), the changes job outputs, fork-ci-result's needs, and GitHub's docs on job hooks and default variables.
checks:

• Outside-fork PR hard rule: holds. route sends any untrusted run GitHub-hosted. The only real line is the job-started hook, because a fork PR controls both its workflow file and the fork-fleet.py that route runs. The hook gets the default variables, which a workflow's env: cannot override (GitHub docs: "the assignment is ignored"), plus GitHub's own event file. A stranger's fork PR fails on GITHUB_ACTOR. The owner re-running it keeps actor = stranger and head.repo = the fork, so it is refused. A maintainer push to a fork branch has actor = owner but head.repo = the fork: refused. A workflow_call from a release or another workflow gets the caller's GITHUB_WORKFLOW_REF: refused. pull_request_target cannot be added by a PR (it reads the base file). A bare "self-hosted" label lands on vm106 and is refused. A non-zero hook exit fails the job before any step (docs), and the gate fails closed on a crash or a missing config. FLEET_RUNNERS_TOKEN reaches only the hosted route job. A fork PR gets no secrets, and a release workflow_call passes none. Fleet jobs read no secrets (tested).
• Sweep acting on others' runs: it can only cancel a stranger's own run that hard-codes a fleet label. Re-running one is blocked by current(): head_branch is the fork's branch name, so the base branch sha never matches. A re-run keeps the original actor's privileges anyway. No way to act on the owner's runs from outside.
• auto fallback: no token, an API error, a re-run, a busy or offline runner, a memory shortage and a lost runner all end hosted or get re-run once. The "lost runner = no failed step" signal relies on GitHub's real step conclusions, which can only be checked on a live runner.
• fleet never silently hosted: correct for trusted runs. The one exception is test_server shards whose gate is false: they run hosted, but they are no-op shards, so this is harmless and arguably right.
• General and upstream: fork-fleet.py holds no repository or project facts (all in fork-fleet.json). Only fork-* files change; ci.yml and release.yml are untouched.
• State question: unset variable = auto (tested). A typo = red route (tested; a typo also reds the release workflow_call's tests, which is consistent). A pre-PR run re-run uses its old workflow, so hosted. A sweep that stops half-way is finished by the next one (stateless, tested). Two sweeps queue in one concurrency group, and attempt 2 is ignored. A runner missing from fork-fleet.json fails capacity, which the sweep rescues. A stale installed gate still checks the trust rule only, so it stays safe. A gate with no config fails closed (tested with -I). The setting changing between attempts: see finding 1.
• Tests: scratch clone at the head, 51/51 OK. I broke 33 guards by hand (each actor, head/base repo, opener, push branch/repo, workflow ref, repository, other events, no event, gate crash-open, route trust skip, auto re-run, busy/offline idle, fleet leftovers, typo=auto, capacity unknown/timeout, the sweep's attempt, branch-moved, real-failure, fleet-mode, stuck-threshold and hosted-job guards, the runs-on attempt check, macOS to fleet, a secret in a routed job, sudo without the self-hosted check, the capacity step removed, a wrong shard key). 31 went red. Two stayed green: the gate using {} for an unreadable event (equivalent: {} is refused anyway), and the "a person's cancel stands" rule at fork-fleet.py:350 (no test has a fleet job failing and then a person cancelling; a test gap only). Clone deleted.
  finding: minor .github/workflows/fork-tests.yml:280 (and the 6 other routed runs-on lines) + .github/scripts/fork-fleet.py:384. Attempt 1 ran with CI_RUNNERS=fleet, then vm106 goes down; the owner sets CI_RUNNERS=github (or auto) and clicks "Re-run failed jobs". route is not re-run, and its attempt-1 output mode=='fleet' still wins in runs-on, so the jobs queue on fleet-vm106-t3code again despite the setting being github. The sweep skips run_attempt != 1, so nothing cancels them: they wait up to GitHub's 24 h limit. The failure table's "switching fleet -> github: the sweep cancels and re-runs" holds only for attempt 1. Fix: also require the current setting in the expression, e.g. (... || (needs.route.outputs.mode == 'fleet' && vars.CI_RUNNERS == 'fleet')), and add a wiring test. (high)
  finding: minor .github/workflows/fork-tests.yml:153 The route job is a new GitHub-hosted job hop on every run's critical path, even in github mode (one more runner start, ~15-30 s, before any Linux job). Its logic needs only the changes outputs and a sparse checkout, so it could be a step in the changes job, which saves a queue and start-up on every run. The owner puts CI speed first. (medium)
  finding: minor .github/scripts/fork-fleet.py:195 TROUBLE leaves out http.client.HTTPException (IncompleteRead, BadStatusLine are not OSError). A truncated runner-list response during route crashes it, so the run is red in auto, which goes against "auto never makes a run red". Add http.client.HTTPException to TROUBLE. (medium)
  finding: minor .github/fork-fleet.md (job-started hook section) The hook script is root-owned, but the line that enables it (ACTIONS_RUNNER_HOOK_JOB_STARTED in the runner directory's .env) is writable by the runner user, and vm106 is not ephemeral. So one compromised trusted job (for example a bad upstream npm dependency in the owner's own PR) can drop the hook for every later job, including strangers' fork PRs that name the label. Defence in depth: make .env root-owned (or set the variable in the systemd unit), or make vm106 JIT/ephemeral like the PC. (medium)

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1) · read at: f2d6e15
covered: the whole diff against main (fork-fleet.py route/gate/capacity/sweep, fork-fleet.json, fork-fleet.yml, the fork-tests.yml route job + routed runs-on + capacity step + no-sudo apt branches, fork-fleet.md runbook and hook install, the 51 tests). It also covers what this calls into: actions/runner main (JobExtension.cs, JobHookProvider.cs, ScriptHandler.cs, GitHubContext.cs, StepsRunner.cs, JobRunner.cs), read to see when the job-started hook runs and what env it gets.
checks: ran fork-fleet.test.py in a scratch clone at the head: 51 OK. Then I broke guards by hand, one at a time. Each of these turned the tests red: dropping the actor or triggering_actor check, the head-repo check, the PR-author check, the push-branch check, the workflow check, letting other events through, the gate allowing an unreadable event, the gate crashing open, route skipping trust, route ignoring the attempt in auto, the sweep re-running attempt 2+, the sweep ignoring a moved branch, the sweep re-running real failures, an unknown runner passing capacity, capacity skipped everywhere, lint's runs-on without the attempt guard, a secret+sudo in a routed job, and an extra job hard-coding a fleet label. Survived (no test failed): removing the "cancelled by a person stands" return in rescue_reason. Making capacity never time out hangs the suite instead of failing it.
Hook env: the runner builds the hook's env from an empty dict plus the github context, so workflow/job env: cannot spoof GITHUB_ACTOR, GITHUB_TRIGGERING_ACTOR, GITHUB_EVENT_NAME, GITHUB_REF or GITHUB_EVENT_PATH. Actor vs triggering_actor vs head repo vs PR author is right: a stranger's re-run, a fork PR the owner edited, and a same-repo-head PR a stranger opened are all refused. Route reads only env (no title/branch interpolation). Fork PRs get no FLEET_RUNNERS_TOKEN. No upstream file is touched. The script has no project names (all facts in fork-fleet.json).
State question: variable unset -> auto (tested). Typo -> route red. A failed route -> routed jobs fall to ubuntu-24.04 and fork-ci-result is red. Re-run failed jobs in auto -> hosted (attempt guard tested). Re-run of a pre-PR run -> the old workflow file, hosted. Two sweeps -> concurrency group, and attempt != 1 is skipped. A sweep that stops half-way -> the next one finds the cancelled run (the not-finished-cancelling path is tested). Runner lost mid-job -> no-failed-step rescue. Runner name not in fork-fleet.json -> capacity fails at once: re-run hosted in auto, red in fleet. Branch deleted -> 404 -> no re-run. Fleet mode goes hosted only for untrusted runs and for test_server shards whose gate is false (they only print a skip; the doc says "every job that runs goes to the fleet").
finding: blocker .github/fork-fleet.md:105-124 + fork-fleet.py:157 gate. A refused job-started hook does NOT stop the job. In actions/runner (JobExtension.cs ~303-312) the hook is just a pre-job step with condition always(). StepsRunner then evaluates each later step's own if:, so a step with if: always() or if: failure() still runs after the gate exits 1. Inputs: an outside contributor (any account that does not need fork-PR approval, or an approval the owner gives without spotting a workflow edit) opens a PR whose fork-tests.yml adds a job with runs-on: [self-hosted] or [fleet-vm106-t3code] / [fleet-pc-t3code] and a step with if: always(). Result: the gate prints "may not run on a fleet runner" and the step runs the stranger's code on vm106, or on the owner's PC as the runner user. On the always-on vm106 it can rewrite the runner directory's .env (the hook line lives there, owned by the runner user), stay behind (it can drop RUNNER_TRACKING_ID to escape orphan cleanup), and read later trusted jobs' tokens. This repo already shows the effect: test_server's "Detect transfer budget report" step (fork-tests.yml:807-809, if: always()) would also run after a refusal. So the failure-table row "the gate refuses it: red, and nothing runs on the fleet" is wrong. No test can catch this, because it is runner behaviour. Fix direction: on refusal, the hook must abort the worker itself (for example, kill its Runner.Worker parent so no later step is scheduled), proven with a livelike run where an if: always() step does not run. Also make the fleet runners one-job/ephemeral, keep the hook and .env out of the runner user's reach, and as an owner step set "Require approval for all external contributors" for fork PR workflows. (high: read in runner source; not run on a real runner)
finding: minor .github/scripts/fork-fleet.py:383-417 sweep. The sweep cancels and re-runs runs without applying the trust rule: it selects only by a fleet label on a job. Inputs: a stranger's fork PR hard-codes a fleet label, and its job queues 10 min (pc offline or vm106 busy). The sweep cancels it, which is harmless, then checks current() against the base repo's branch of the same name as the fork branch, so a re-run is unlikely but not ruled out. Then GitHub re-runs a stranger's run with the repo's token, without approval. Filter runs by event, head repository and actor (the refused() rule from run fields) before acting. (medium)
finding: minor .github/workflows/fork-fleet.yml:16-19 auto fallback. In auto, the only bound on a fleet job queued for a runner that went away is this cron sweep. GitHub delays or drops scheduled runs under load, and disables schedules in public repos after 60 days without activity. Inputs: the sweep is disabled or broken, and route picked a runner that then went offline. Result: the job sits queued up to GitHub's 24 h limit, silently (timeout-minutes does not count queue time). Consider an in-run bound, or have the sweep's own failure page the owner. (medium)
finding: minor .github/scripts/fork-fleet.test.py rescue_reason. No test pins "a run a person cancelled while its fleet job ran stands": removing that return at fork-fleet.py:351 leaves all 51 green. Inputs: a fleet job fails with no failed step, then a person cancels the run. Result: the sweep re-runs it against the person's cancel. (high)

[scuttletoby]

not a lens: fix round r1

New head: eaf13202f9062257474076ec0b3ac89378398370. Before it comes ce8b61eb4, which merges fork main (919556a1a, #12: outside PRs run nothing) into this branch. For about a minute, ce8b61eb4 alone was the head, because a commit failed before a push. The PR stays a draft. The PR body and .github/fork-fleet.md have the new design, the owner steps and the failure table.

Per finding

Finding	Result
s1 blocker: a failed job-started hook still lets always()/failure() steps run, and a runner-writable .env can remove the hook	Fixed at the root. No standing runner exists. A root dispatcher makes a one-job JIT runner only for a queued job that asks for exactly a pool label plus the run's own random label, of a run whose GitHub record passes the trust rule, when no other queued job could take that runner. The runner runs from a fresh copy of a root-owned template, as a user of its own; its environment comes from a root-only file, removed after the start. The hook is the second line, and it now SIGKILLs Runner.Worker. Proved by GateProcess.test_the_hook_ends_a_refused_job_before_its_always_steps: a refused job leaves no always() marker, and a mutant gate that only exits 1 leaves one.
s2-1: a re-run keeps the first attempt's routing	Fixed. runs-on also checks vars.CI_RUNNERS. The watch reads its own attempt, and the sweep reads the latest attempt of every run. The dispatcher makes each runner for one run and attempt, and the gate checks both.
s2-2: fold route into an existing job	Done: Route is the last step of changes.
s2-3: a truncated or invalid reply must not turn auto red	Fixed. TROUBLE includes http.client.HTTPException (IncompleteRead), ValueError, KeyError and TypeError. Tested for watch, sweep and dispatch. Route calls no API now.
s2-4 / s1: runner-writable .env	Covered by the root-owned config. The launcher refuses a template with .env, .path, .runner or .credentials, and uses each copy once.
s1: the sweep must use the trust rule	Fixed. run_refused reads GitHub's record of the run plus pulls/<n>. A stranger's run is never cancelled or re-run.
s1: auto must not hang on one cron	Fixed. fleet_watch in each fleet run cancels with an error after 10 min (fleet: 2 h). The sweep is the fallback, and in fleet mode it now cancels, with an error, a run past 2 h, because a re-run of failed jobs has no watch.
s1: test for "a person's cancel stands"	Added: Sweep.test_a_persons_cancel_stands.
Doc: skip-only shards in fleet mode	The doc now says they run GitHub-hosted.
Plan row 9	Not here. The diff is already about 2,300 lines, so it will be a stacked follow-up PR.

Found while testing, and fixed:

• the launcher's JIT check was a bash regex error that refused every config;
• WORK_ROOT had to be 0711 so the runner user can reach its copy;
• the launcher's EXIT trap named a local, so every start exited 1 under set -u and left the environment file behind;
• the dispatcher could make a runner for attempt 1 after the owner's re-run had moved the job to attempt 2, and the gate would then kill a trusted job. It now waits a pass;
• the lock directory under /run is created after a boot.

Red → green evidence

Each mutation was applied to the real file, the suite was run (red), the file was restored, and the suite was run again (green):

ok  a person's cancel stands (s1 minor, fork-fleet.py rescue_reason): mutated -> FAILED (failures=1); restored -> OK
ok  the hook kills the worker (s1 blocker, stop_job): mutated -> FAILED (failures=2); restored -> OK
ok  the gate checks the run its runner was made for: mutated -> FAILED (failures=3); restored -> OK
ok  the sweep applies the trust rule (s1 minor): mutated -> FAILED (failures=6); restored -> OK
ok  the dispatcher applies the trust rule (blocker root): mutated -> FAILED (failures=4); restored -> OK
ok  a garbled reply is never red (s2 minor 3): mutated -> FAILED (errors=2); restored -> OK
ok  a re-run follows the setting (s2 minor 1, runs-on): mutated -> FAILED (failures=3); restored -> OK
ok  the launcher takes only its three input lines: mutated -> FAILED (failures=2); restored -> OK
ok  the launcher refuses a configured template (.env): mutated -> FAILED (failures=5); restored -> OK
ok  launcher bug 1: the JIT check was a bash regex error: mutated -> FAILED (failures=2); restored -> OK
ok  launcher bug 2: WORK_ROOT must let RUNNER_USER through (0711): mutated -> FAILED (failures=1); restored -> OK
ok  launcher bug 3: the EXIT trap named a local: mutated -> FAILED (failures=2); restored -> OK
ok  the dispatcher makes a runner only for the attempt it read: mutated -> FAILED (failures=1); restored -> OK
ok  fleet mode: the sweep reports a long wait (s1 minor: no silent 24 h): mutated -> FAILED (failures=1); restored -> OK
ok  fleet mode: the sweep cancels a run past fleet_stuck_seconds: mutated -> FAILED (failures=1); restored -> OK

Suites at the new head, all green:

• fork-fleet.test.py: 104 tests, OK;
• fork-contracts.test.py and fork-release-workflow.test.py;
• node .github/scripts/fork-ci-tests.cjs (rc 0);
• fork-tests-drift.cjs: matches the record;
• actionlint on fork-*.yml;
• bash -n on both new scripts;
• gitleaks on the staged diff.

shellcheck is not installed here, so it did not run.

Not proved here: a real runner. The runbook's real-runner check needs a throwaway private repository under the owner's account, and none was created. It runs on VM106 next to the real install and checks:

• a refused job's always() step never runs;
• the environment file is gone after the start;
• the runner user cannot write the template;
• a bare label finds no runner;
• a re-run gets a runner for attempt 2.

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1-r2) · read at: eaf1320
covered: the fix diff f2d6e15..eaf1320 minus the merge of main 919556a (git diff 919556a eaf1320 over the 10 changed files). I read in full: fork-fleet.py (route, watch, sweep, dispatch, Host, gate, worker_above, stop_job, refused, run_refused), fork-fleet-launch.sh, fork-fleet-hook.sh, fork-fleet.json, fork-fleet.yml, every fork-tests.yml change (Route step, fleet_watch, 7 routed runs-on lines, no-sudo branches, fork-ci-result), fork-release.yml's permissions, the runbook .github/fork-fleet.md (host steps, owner steps, real-runner check, failure table), and the GateProcess/Dispatch/Launcher/Wiring tests. I also read what this reaches: .github/actions/setup-ci-deps (the ~/.vite-plus cache) and fork-release.yml's build job, which uses it.
checks:

• Hard rule (stranger never runs on the fleet): holds by design. The dispatcher starts a runner only for a job that asks for exactly a pool label plus a well-formed nonce label (fleet_job), and only for a run that GitHub's own record says is trusted (run_refused: actor, triggering_actor, event, workflow path, head repo = run head repo = base, and PR author from pulls/; a fork run lists no PRs, so it is refused). A fork PR whose workflow names a fleet label, or copies a run's public nonce label, finds no runner, or it blocks the launch through the rival check. If it wins the race for a fresh runner, the gate refuses it on made_for (run id/attempt from the root-only env file) before any step. Before the hook, only the job message and action downloads touch the host; no stranger code runs. Re-runs by others, workflow_run, labels and comments are all refused by the same rule. fork: outside PRs run nothing, one checkout without actions/checkout, contract tests #12 and all_external_contributors come on top.
• GitHub input reaching root: none reaches a shell, path, unit name or file. The unit or dir name is the pool label from config plus int(job id). Labels are regex-checked before generate-jitconfig. The JIT config is checked against [A-Za-z0-9+/=] in both Python and bash, and run and attempt are int()ed. The launcher accepts only its three stdin keys, reads launch.conf as data and never sources it, and names reach systemctl only after name_ok. prune runs rm -rf --one-file-system only on top-level names that root created in a 0711 root dir. Job names and labels go only to the journal.
• Hook kill: stop_job SIGKILLs the nearest Runner.Worker ancestor. hook.sh execs python -I, so the worker is the direct parent. A crashing gate is killed too. The GateProcess model shows that no always() marker is written, and the mutant that only exits 1 is caught. This is runner behaviour; I did not prove it on a real runner (the runbook check 2 covers it). One residual case: if the installed fork-fleet.py cannot even start (a syntax error, or GNU install replacing it as a hook starts), python exits 2 without the kill. Only a runner made for a trusted run can hit this.
• Escape between jobs: the template is safe. The copies hard-link root-owned go-w files, the subdirs stay root's, and the launcher refuses .env/.runner/.credentials. The hook path comes from the root-only env file, so a job cannot drop it (this fixes S-2 fork: follow each stable upstream release with the patch series #4's hook part). The unit has KillMode=control-group, RuntimeMaxSec, NoNewPrivileges and PrivateTmp; with no linger, cron or at, nothing outlives the unit. Jobs are not separated from each other: see finding 1.
• Races: the dispatcher's attempt race is fixed (it skips a job whose run_attempt differs). A second queued job that could take the runner is a rival, so nothing starts. The case left open is GitHub handing an idle leftover runner the next attempt's job: see finding 2.
• Round-1 findings: S-1 blocker fixed at the root (no standing runner, dispatcher trust, gate kill). S-1 sweep trust fixed (run_refused, 7 tests red when removed). S-1 cron-only bound fixed (fleet_watch). S-1 "a person's cancel stands" test added. S-2 fork: the server and CLI update from the desktop app's repository #1 re-run routing fixed in runs-on (vars.CI_RUNNERS checked, wiring test). S-2 fork: release workflow for v<upstream>-fork.N on GitHub-hosted runners #2: route is now a step of changes. S-2 fork: CI check that refuses database schema and migration changes #3: TROUBLE has HTTPException, ValueError, KeyError and TypeError. S-2 fork: follow each stable upstream release with the patch series #4: the hook part is fixed, but the "one compromised trusted job reaches later jobs" part is not: finding 1.
• auto never red or stuck: an unclear heartbeat goes hosted. A job queued 10 min means watch cancels, then the sweep re-runs hosted (github-actions is untrusted). A gate-killed or lost runner has no failed step, so it is re-run hosted. The watch has API trouble handling and gives up quietly after bound+2 min. The only lasting red is when both watch and sweep are dead. fleet never silently hosted: untrusted runs and skip-only test_server shards go hosted with a notice (documented). A re-run of failed jobs after a switch auto->fleet goes hosted (attempt differs), which is right.
• State question: unset variable = auto. A typo makes the Route step red. Dispatcher restarted mid-launch: an offline registration with no unit is deleted, prune removes the half copy and the env file, and it retries (MAX_TRIES 3, in memory, so it resets on restart; harmless). After a reboot, /run is recreated (lock dir and RUN_DIR), offline registrations are deleted, and copies are pruned. A leftover unit is counted in active and its name is skipped. A leftover runner user is fine; useradd fails on a re-run (finding 5). A missing, world-readable or empty token file refuses at start, Restart=always retries every 30 s, and auto goes hosted. An expired token is a 401 (HTTPError is in TROUBLE), so each pass warns, heartbeats go stale, auto goes hosted, and fleet errors after 2 h. Two dispatchers: flock on /run/fork-fleet/dispatch.lock, but finding 3 covers two pools. A run from before this PR, re-run, uses its own workflow: hosted. Empty history: no heartbeat, so hosted. The sweep is stateless and serialised.
• Runbook as root: steps 3-5 are safe and idempotent. The token goes in with umask 077 and read -s, launch.conf is chmod 600, and the template is sha256-checked and extracted with --no-same-owner + go-w. Steps 1-2 are not idempotent or atomic: finding 5.
• Tests (scratch clone at the head, GH_TOKEN/GITHUB_TOKEN unset): fork-fleet.test.py 104 OK; fork-contracts and fork-release-workflow tests pass. actionlint 1.7.12 on fork-tests/fork-fleet/fork-release: clean. shellcheck on launch.sh and hook.sh: only SC2064 at launch.sh:107, which is intended (expand now). I broke 41 guards by hand, one at a time, and 40 went red: fleet_job label count, rival check, dispatcher trust, PR head vs run head, the no-PR fork run, gate made_for, made_for attempt, the kill, the crash path, the attempt-race skip, the at-once idle stop, MAX_TRIES, token perms, lock, JIT check, deleting a live registration, GENERIC in could_take, started_mb in fits, route trust, sweep trust, and in the launcher: conf/hook/template ownership, the template .env refusal, root RUNNER_USER, name_ok in start and stop, dir reuse, the trap, the hook line, RuntimeMaxSec, KillMode, PrivateTmp/NNP, the copy and RUN_DIR modes, extra input keys, prune of live units, hook -I, runs-on ignoring github, and the watch's cancel. The one that stayed green is finding 4. The clone is clean afterwards and was deleted.
  finding: major .github/scripts/fork-fleet-launch.sh:117 (--uid "$RUNNER_USER") + .github/fork-fleet.md:196-198 + fork-fleet.json slots 2. Every one-job runner runs as the same user fork-runner, with one persistent $HOME, and two run at once. Inputs: the owner's own unmerged PR pulls in a bad dependency (a postinstall), and that is a trusted run, so it lands on the fleet. (a) While a push-to-main job runs in the other slot, the PR job reads /proc//environ of that job's processes (same uid; Yama does not limit environ reads). It gets ACTIONS_RUNTIME_TOKEN, which can write main-scope Actions cache, and also the JIT config of a runner that is starting: run.sh, run-helper and the Listener's initial environ keep ACTIONS_RUNNER_INPUT_JITCONFIG, because the runner's delete changes only its .NET view. With that config it can take the main job onto its own runner. (b) Without any concurrency, it plants files under ~/.vite-plus, which setup-ci-deps:36-41 caches by os/arch/version key only. A later main job on a key miss saves the planted tree to main scope, and fork-release.yml's Linux build job (signing secrets, setup-ci-deps at :196) restores that key and runs vp from it. GitHub-hosted runners keep PR scope away from main scope; this design removes that wall. WORK_ROOT 0711 and the 0700 copies do not separate jobs of one uid. The stranger rule is not broken. Fix: one uid per unit (systemd DynamicUser=yes, or a pool of users with exactly one unit per user at a time); HOME (and XDG/tool caches) inside the copy so prune deletes it; drop the "home persists" row, or mark it as a deliberate risk the owner accepts. (medium: same-uid environ access is standard Linux; the cache chain comes from reading setup-ci-deps and fork-release, not from a run)
  finding: minor .github/scripts/fork-fleet.py:672-679 + the routed runs-on lines in fork-tests.yml. Inputs: fleet mode; the owner cancels a run while a fleet job's runner is still coming up; then, within about a minute, clicks "Re-run failed jobs". The attempt-2 job keeps attempt 1's labels (route is not re-run, so it has the same nonce). The attempt-1 runner, now online and idle, matches it, and GitHub assigns it within seconds, before the dispatcher's next 20 s pass can stop it. The gate sees FORK_FLEET_ATTEMPT 1 != 2 and kills the job, so the run is red. fork-fleet.md:431 ("stopped at once if another queued job could take it") overstates this. Fix: put github.run_attempt into the job's own label (for example, build the nonce label in runs-on as -a, and match the attempt in fleet_job), so that a runner of one attempt can never match another attempt. (medium)
  finding: minor .github/scripts/fork-fleet.py:653 (units for every pool) + :789. One dispatcher serving several pools (the default with no --pool, and the json says pc's dispatcher runs on the always-on host): a HostError from one pool (pc: ssh to a PC that is off) aborts the whole pass. The vm pool then gets no launches and no heartbeat, so auto goes all hosted and fleet waits 2 h, then errors. The alternative, a second dispatcher with --pool pc, fails on the shared default --lock /run/fork-fleet/dispatch.lock. test_a_host_that_cannot_answer_stops_the_pass pins today's behaviour. This is not live while the runbook runs --pool vm only. Fix: catch HostError per pool (skip that pool, beacon free=0) or default the lock per pool, before pc joins. (high)
  finding: minor .github/scripts/fork-fleet.py:743. Test gap: dropping "- pending[...]" from the heartbeat's free count leaves all 104 tests green. Inputs: trusted fleet jobs are queued and waiting (no slot or memory). Result: the heartbeat still reports free slots, auto routes more jobs to a full pool, and they wait 10 min, get cancelled and re-run. Add a heartbeat test with pending jobs. (high)
  finding: minor .github/fork-fleet.md:215-232. Host steps 1-2 run again: useradd fails on an existing fork-runner, and the cron/at.deny line is appended each time. Step 2's template refresh (rm -rf then mv) races a dispatcher in the middle of cp -al: you get a half copy and a failed runner, which is retried. Fix: id fork-runner || useradd ..., grep -qx before appending, and stop the dispatcher around step 2 (or use an atomic rename -T swap). (high)

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2-r2) · read at: eaf1320
covered: the fix round's own changes (git diff 919556a..eaf1320, which leaves out what the #12 merge brought). Read in full: fork-fleet.py (route, watch, sweep, dispatch, gate), fork-fleet-launch.sh, fork-fleet-hook.sh, fork-fleet.json, fork-fleet.yml, every fork-tests.yml change (Route step, fleet_watch, the 7 routed runs-on lines, the no-sudo apt branches, fork-ci-result), the fork-release.yml permissions, the runbook .github/fork-fleet.md (host steps, owner steps, real-runner check, failure table) and the GateProcess/Launcher/Dispatch/Wiring tests. Round-1 S-1 and S-2 findings checked one by one.
checks:

• Hard rule (strangers never reach the fleet): holds. The dispatcher starts a runner only for a job asking for exactly {pool label, pool-label-key-16hex}, of a run whose GitHub record passes refused() (fork PR runs list no pull request: refused; each PR is read from pulls/, head must equal the run's head repo, opener/actor/triggering actor must be trusted). A bare fleet label, self-hosted, a group-only runs-on or a copied nonce label from the public jobs API is either not matched or counted as a rival, so no launch. A stranger's job that queues after the launch and races onto the runner is killed by the gate (made_for fails: wrong run). A stranger opening a PR from the owner's own branch: actor/opener check refuses it, and the owner's run on that branch is refused too (fail-closed, DoS only). workflow_call/release, workflow_run, schedule, sweep re-runs (github-actions) and bots: refused. Re-runs by others: strangers cannot re-run here.
• Root attack surface: no GitHub-controlled string reaches a shell, path, unit name or file as root. The unit/runner name is label + int(job id), runner names from the API must fullmatch label-digits, generate-jitconfig labels are only the validated pair, the JIT config is base64-checked in Python and again in the launcher, run/attempt are ints, launch.conf is parsed (never sourced). Job names only reach the journal. prune runs rm -rf --one-file-system as root on runner-written trees; GNU rm does not follow symlinks, so it stays inside. The token file (root 0600 in a 0700 dir), the template (root-owned, hard links the runner user cannot write) and root (no sudo, NoNewPrivileges, PrivateTmp, KillMode=control-group, RuntimeMaxSec) are out of a job's reach. Not out of reach: the other runner and later jobs (finding 1).
• Gate kill: the hook execs python under Runner.Worker, and worker_above finds it by comm. SIGKILL ends the worker before any step, always() included. GateProcess shows it, and the mutant that only exits 1 leaves the marker. Not proven on a real runner, which is what the runbook's real-runner check is for. Do it before the fleet takes jobs.
• Round-1 findings: S-1 blocker fixed at the root (no standing runner; trust is decided from GitHub's record, outside any job). S-2 fork: the server and CLI update from the desktop app's repository #1 (re-run routing) fixed (vars.CI_RUNNERS in runs-on, mutation red). fork: release workflow for v<upstream>-fork.N on GitHub-hosted runners #2 done (Route is a step of changes). fork: CI check that refuses database schema and migration changes #3 fixed (http.client.HTTPException in TROUBLE). fork: follow each stable upstream release with the patch series #4 (.env writable) only partly fixed (finding 1). S-1 sweep trust, auto bound (fleet_watch) and the person's-cancel test: fixed and tested.
• auto: no heartbeat, a stale or garbled one, a dispatcher that is down, a job that never gets a runner (rival, MAX_TRIES, memory) all end GitHub-hosted, or cancelled at 10 min by the watch and re-run hosted by the sweep. Never red for good. fleet: a trusted job is never hosted silently: route crash = red, refused/blocked = 2 h error by the watch or the sweep; skip-only shards and untrusted runs hosted, as documented.
• State question: variable unset = auto, typo = red (tested). Dispatcher restarted mid-launch: offline registration deleted, half copy and env file pruned, relaunched. Reboot: /run lock dir and RUN_DIR re-created, copies pruned, lost job rescued in auto. A leftover unit or registration not matching label-digits is ignored (the first design's standing runner must be deleted by hand, as the doc says). Token missing or wrong mode = refuses to start; expired = every pass warns and the heartbeat goes stale = hosted. Two dispatchers = second fails on flock. A run from before this PR re-run = its old workflow, hosted. The re-run race between reading the run and its jobs is handled (tested). Runbook as root: safe, nothing prints a secret, and every step re-runs harmlessly (useradd just fails, cron.deny gains a duplicate line).
• Tests in a scratch clone at the head with GH_TOKEN/GITHUB_TOKEN unset: fork-fleet.test.py 104 OK (5.5 s). Broke 40 guards by hand, one at a time, file restored after each: 39 red. Red: dispatch trust skip, exact label pair, rival check, attempt check, gate made_for, gate refused, no SIGKILL, fork PR with no pulls, PR head compare, could_take GENERIC/case, idle stop at once, sweep trust, route trust, push branch, triggering actor, PR opener, token mode, flock mode, gate crash-open, pending count, MAX_TRIES, person's cancel, watch fleet bound, launcher (.env allowed, template find check, RUN_DIR 0700, EXIT trap, NoNewPrivileges, PrivateTmp, KillMode, RuntimeMaxSec, uid 0 allowed, JIT charset, dir-exists, conf owner, prune deleting live copies), runs-on (vars fleet check, github check). Green: worker_above matching comm.startswith('Runner') (equivalent: the nearest ancestor is still Runner.Worker).
• actionlint 1.7.12 on fork-*.yml: clean. shellcheck 0.10.0 on both scripts: one SC2064 at fork-fleet-launch.sh:107, deliberate (expanded now on purpose); add a disable comment if shellcheck ever gates. Clone deleted.
  finding: minor .github/scripts/fork-fleet-launch.sh:109,117 + .github/fork-fleet.md:74-75,196-198,215 All runners run as one uid (fork-runner), and the copy's top directory is that uid's (chown + 0700), so "a user of its own" is not true across runners. Inputs: vm has 2 slots; a trusted job runs a malicious dependency (an owner dependency-bump or upstream-sync PR) while the other slot's runner is starting. The job can open WORK_ROOT/fleet-vm-t3code- by name (0711 hides only the listing; job ids are public). There it can write .env (the listener reads it on start, and again after a self-update restart), swap run.sh/bin, or read /proc//environ, where the JIT config stays in the initial env block after the runner unsets it. Each of these runs that runner without the gate, or as itself. $HOME (rustup, caches) and /dev/shm also carry the compromise into every later fleet job, pushes to main included. The dispatcher still keeps strangers out (a stranger would also need to race onto that runner), so this is defence in depth. It is round-1 S-2 fork: follow each stable upstream release with the patch series #4 again: one job can still change another job's gate. Fix: one uid per runner (a fixed fork-runner-1..N, one per slot and never two units at once, or DynamicUser with the copy as its state directory), a per-job HOME (e.g. HOME=/home in the env file, or ProtectHome=tmpfs), and a real-runner check step where one job tries to write another runner's copy. (medium-high)
  finding: minor .github/scripts/fork-fleet.py:672-679 (+ gate made_for at :236-242) Fleet mode, a runner left over from a cancelled attempt. Inputs: CI_RUNNERS=fleet. The owner cancels a run while the runner made for job X (attempt 1) is up but has not taken it, then clicks "Re-run failed jobs" within about a minute. Attempt 2's job has the same two labels (in fleet mode runs-on keeps attempt 1's labels), and GitHub can give it to the idle attempt-1 runner before a pass stops it (idle_passes x 20 s, or the next pass at best). The gate then kills a trusted job: red, and fleet mode never rescues. Fix: stop the unit and delete the registration as soon as the runner's own job (its id is in the name) is completed or cancelled. Read actions/jobs/ instead of waiting for idle passes. (medium: depends on GitHub's assignment timing)
  finding: minor .github/fork-fleet.md:193-195 + 323-392 Runner self-update in the hard-linked copy is assumed, not checked. Inputs: the template falls one actions/runner release behind (every few weeks). Each JIT runner then updates itself before its job, inside a copy whose bin/, externals/ and top-level files are root-owned hard links. The real-runner check uses a fresh template, so it never runs this path. If the updater cannot rename or overwrite those files, every fleet job loses its runner: auto pays 10 min plus a full hosted re-run on every run, and fleet goes red, until someone refreshes the template. Even when the update works, every job pays for a runner download. Fix: add a step to the real-runner check with a template one release behind, and have the dispatcher report free=0 (or warn) when the template is older than the newest release. (low-medium)

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2-r4) · read at: 0514b66
covered: fix diff eaf1320..0514b66 in full (1d5e892, 0514b66) and the round-2 lens comments; read in full fork-fleet.py (dispatcher), fork-fleet-launch.sh, the job hook, fork-fleet.json, fork-fleet.yml, the runs-on/route changes in fork-tests.yml, fork-follow.yml + fork-follow-push.sh (sweep), and the runbook .github/fork-fleet.md. Ran in a scratch clone with GH_TOKEN/GITHUB_TOKEN unset and an empty GH_CONFIG_DIR: fork-fleet.test.py 119 OK (2 root-only skipped), contracts + release-workflow tests OK, fork-ci-tests.cjs 0 and drift check matches, actionlint 1.7.12 clean (fork-tests/fleet/release), shellcheck 0.11.0 clean (launcher, hook). LauncherAsRoot 2/2 OK inside a throwaway docker ubuntu:24.04 (never as root on VM106); CI at this head all green incl. "Workflow lint and upstream drift". Broke 27 guards by hand one at a time: 24 went red (runs-on -a, dispatch label attempt check, NONCE, pending count in the heartbeat, host failure stops only its pool, per-pool locks, outdated template closes the pool, no start before the first release read, gate attempt, PR opener trust, launcher ready x2, free_user, reset_user kill, copy check, HOME/TMPDIR, duplicate uid, linger, ProtectSystem, RemoveIPC, home 0700, copy top dir 0700, same-filesystem check, template version in start); SHM cleanup removal red only in the root test; 2 stayed green (see minor below).
checks: all 6 round-2 items fixed at the root and tested: (1) one user per slot, killed + wiped + reused only after reset; (2) runner label carries the attempt, dispatch refuses a mismatched attempt; (3) one lock per pool, a host failure skips only its pool; (4) the free-slot count includes queued/pending jobs; (5) runbook steps 1-2 re-runnable (users kept, deny lines once, mv -T swap); (6) outdated template closes the pool instead of self-updating. Hard rule holds: outside-fork PRs, other actors, other events and other repos are refused before any fleet label is used; no GitHub-controlled field (labels, run names, branch names, API fields) reaches a shell, path, file or unit name as root (checked against the allowlists/regexes). State question: dispatcher restart mid-launch -> pending count + prune recover; SIGKILL on the 120 s launcher timeout leaves the env file, which prune removes later; reboot (/run gone) -> users reset before reuse; leftover user/unit -> reset_user kills and wipes or dies; old launch.conf with RUNNER_USER fails closed with a reason; variable unset -> auto; two dispatchers -> per-pool flock; a pre-PR run re-run -> no fleet label, hosted. Notes, not findings: the 1 h cache of the newest runner release leaves a window where an older runner may self-update; in fleet mode every actions/runner release turns runs red after 2 h until the owner refreshes the template (documented, not silent).
finding: major .github/scripts/fork-fleet-launch.sh:164 two slot users, each running a trusted job with a malicious dependency: user B makes /dev/shm/bdir/sub mode 777, user A's job leaves a file in it; at A's next reset_user, B swaps sub for a symlink between find and rm -> root's rm -rf --one-file-system (find -exec ... {} +) follows it and deletes an arbitrary file/tree outside /dev/shm (e.g. under /etc or /home/toby); protected_symlinks does not help because bdir is not sticky. Reproduced in a container with this exact line: root deleted /etc/victimdir/victim at round 3. This is a cross-job escape to root-level deletion. Fix: give each unit a private /dev/shm (e.g. TemporaryFileSystem=/dev/shm or PrivateIPC+tmpfs) so no cleanup is needed, or run the cleanup as the runner user (setpriv --reuid) so root never rm's inside a directory another user controls; add a LauncherAsRoot test (high)
finding: major .github/scripts/fork-follow-push.sh:104 + .github/scripts/fork-fleet.py:179 if FORK_PUSH_TOKEN belongs to scuttletoby (a trusted actor), the daily sync's force-push of main triggers a fork-tests.yml push run with actor = triggering actor = scuttletoby on refs/heads/main -> refused() returns None and upstream's new, unreviewed code and dependencies run on the VM106 fleet every day, against the owner decision "upstream sync on GitHub-hosted runners only". test_the_upstream_sync_runs_on_github_hosted_runners_only (fork-fleet.test.py:1765) checks only the sync workflows' runs-on lines, not the push run they cause. Fix: push with an identity not in trusted.actors (GitHub App / machine user) or make the trust rule refuse sync pushes; add a test (medium: depends on whose PAT FORK_PUSH_TOKEN is; fork-follow has not run yet)
finding: minor .github/fork-fleet.json:29 + .github/fork-fleet.md:245 the vm pool is documented as "a sandboxed always-on runner VM", but VM106 is the owner's T3 workspace; units have no network/process limits (no IPAddressDeny, ProtectProc, ProtectHome), so a job can reach T3 on 0.0.0.0:3773, 127.0.0.1 services (4300, 3308, 8384, ...), docker bridges and the LAN 10.82.0.0/24 (PVE, VM111). /home/toby is 750 so files are safe; fix the doc or add IPAddressDeny=any + allowlist and ProtectProc=invisible (medium)
finding: minor .github/fork-fleet.md:271-299 host steps 2-3 have no set -e (or && chain): a failed curl or sha256sum -c (:274) does not stop the step, so an unverified or empty tarball is swapped in with mv -T and the old template is deleted; step 3 can install mixed versions and still writes SOURCE (:299), so the Old state check trusts it. Add set -euo pipefail at the top of each paste block (high)
finding: minor .github/scripts/fork-fleet-launch.sh:124 and :78 test gaps: breaking the check_users "only its own group" check, or removing the launch.lock flock in setup(), leaves every test green (root and non-root); add a test for each (high)

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1-r4) · read at: 0514b66
covered: the fix diff eaf1320..0514b66 in full (1d5e892 + the oxfmt commit 0514b66). I read in full: fork-fleet.py (refused, run_refused, made_for, worker_above, stop_job, gate, route, watch, sweep, Host, fleet_job, could_take, newest_runner, dispatch_pass, the locks, cmd_dispatch), fork-fleet-launch.sh, fork-fleet-hook.sh, fork-fleet.json, fork-fleet.yml, every routed runs-on line and the Route/fleet_watch/workflows jobs in fork-tests.yml, the runs-on lines of fork-follow.yml, fork-checks.yml and fork-release.yml, the runbook .github/fork-fleet.md (design, races, host steps 1-5), and the LauncherAsRoot, Wiring and upstream-sync tests. I also read actions/runner main JobHookProvider.cs and ScriptHandler.cs to check that the hook's environment cannot come from the workflow's env:.
checks:

• Round-2 findings, one by one. (1) One user per slot: fixed at the root. free_user picks a user that no loaded fork-fleet-* unit runs as, under launch.lock. reset_user kills every process of that uid, empties its /dev/shm files and removes its old copies before reuse. The copy's top dir, HOME and TMPDIR are 0700 and owned by the user. ProtectSystem=strict with ReadWritePaths=, PrivateTmp and RemoveIPC. The vite-plus cache chain is gone because HOME now dies with the copy. (2) Re-run after a cancel: fixed structurally. runs-on adds -a<run_attempt>, fleet_job needs it, and the dispatcher skips a label whose attempt is not the run's. (3) Per-pool locks (dispatch-.lock): fixed. A HostError now marks only its own pool down, with heartbeat free 0 and a why. (4) Queued jobs count against free slots, and that is now tested. (5) Runbook steps 1-2 are safe to run twice. Step 2 stops the dispatcher around an mv -T swap, and running copies keep their hard-linked inodes. (6) Self-update fails closed. ready compares .fork-fleet-version with the newest release, read hourly; with no release read yet, nothing starts. The pool is then closed (free 0, why), so auto goes hosted and fleet waits, then errors.
• Hard rule (strangers never run on the fleet): holds. Nothing in this round weakens the dispatcher's trust check (GitHub's run record, pulls/, a fork run lists no PR), the exact two-label match, the rival check or the gate's made_for. In actions/runner the hook's environment starts as an empty dictionary plus the runner's github/runner contexts (JobHookProvider.cs:74, ScriptHandler.cs:290-299), so a workflow env: cannot fake FORK_FLEET_RUN/ATTEMPT. A copied label is now also tied to one attempt.
• GitHub input reaching root: none. Names are the pool label plus int(job id). Labels match the NONCE fullmatch, and the attempt is int()ed. The release tag goes through VERSION in Python and again in bash before ready. User names come only from root's launch.conf (regex-checked). systemctl show User is compared by name, the same string systemd-run --uid was given.
• Escape between jobs: closed for files, processes, shm and the template (see above, all tested as root). Not closed for the network: see the major finding.
• auto/fleet: auto never red. A closed or down pool, or one whose release read failed, beacons free 0, so jobs go hosted; the 10-min watch plus the sweep stay as before. fleet is never silently hosted: a closed pool means jobs wait, then the 2 h error.
• Owner decision: fork-follow.yml, fork-checks.yml and fork-fleet.yml are fixed GitHub-hosted labels, enforced by test_the_upstream_sync_runs_on_github_hosted_runners_only (red when I made one runs-on self-hosted). Note: the sync pushes main with FORK_PUSH_TOKEN (a scuttletoby token), so the Fork tests run that push starts is a trusted push to main, and in auto/fleet it may use the fleet. The decision as written covers only the sync's own jobs. Please confirm that this is what was meant.
• State question: Dispatcher killed mid-start: the half copy is root's and gets pruned, the env file is in 0700 RUN_DIR and gets pruned, the offline registration is deleted, and the user is reset on next use. Reboot: /run and the locks come back, copies are pruned, /dev/shm is empty. A leftover unit stays busy (its user too) until it is gone. Template missing a version file, older, or on another filesystem: refused, fail closed. Release API down at start: no runner starts. Two dispatchers on one pool: the second fails on its pool lock. Two pools on one host serialise on launch.lock and share the users safely. Variable unset = auto. A run from before this PR, re-run: old workflow, hosted. One gap I could not test: RUNNER_USERS with fewer users than slots is not checked against slots. free_user then dies, and the pool flaps to free 0 (doc-only, harmless).
• Tests (scratch worktree at the head, GH_TOKEN/GITHUB_TOKEN unset, GH_CONFIG_DIR empty): fork-fleet.test.py 119 OK (2 root-only skipped); fork-contracts and fork-release-workflow pass. LauncherAsRoot in a throwaway docker ubuntu:24.04 (--init): 2 OK. Without --init, the container's PID 1 does not reap zombies, pgrep sees them and both tests fail. That is a container artifact, not a launcher bug. CI at this head is green, including Workflow lint. shellcheck 0.10.0 on the launcher and hook: clean. actionlint 1.7.7 on the fork-.yml files: only main's own SC2153 infos and macos-26 (unknown to 1.7.7); nothing from this PR. I broke 29 guards by hand, one at a time, and 28 went red. Red: the attempt-label skip, NONCE without -a, runs-on without the attempt, a HostError aborting the pass, one shared lock, pending dropped from free, a closed pool still launching, no release read = OK, ready skipped, a down pool beaconing free; in the launcher: ready always OK, the version compare, the missing version file, free_user always taking the first user, no kill loop, no shm cleanup (root suite only), no prune in reset, duplicate uid, linger, uid 0, copy 0755, HOME not the user's, HOME/TMPDIR not in the env file, no ProtectSystem/ReadWritePaths, no RemoveIPC, no same-filesystem check, a busy user handed out, and fork-follow on self-hosted. Green: minor finding 1. The worktree is clean at the head.
  finding: major .github/fork-fleet.md:245 (host setup on VM106) + .github/scripts/fork-fleet-launch.sh:234-237 (the unit has no network limit). Inputs: a trusted fleet job runs a bad dependency, such as a postinstall in the owner's own PR, the same precondition as round 2's major. The job shares VM106's network namespace, so as fork-runner-N it can reach every loopback and LAN service of the owner's workspace. On VM106 today, 127.0.0.1:8384 is toby's Syncthing GUI with no GUI user/password set (its config.xml has no ). Any local client gets that GUI's CSRF token by loading it, then drives the REST API. It can add a device and share a folder under /home/toby out, to read the gh token or the Claude logins, or receive one into toby's home, such as ~/.claude settings with hooks, which is code as toby. Also reachable: 0.0.0.0:3773 (T3), 127.0.0.1:4300/3308/631, the LAN 10.82.0.0/24 (the PVE host, VM111) and the docker bridges. The containment section stops at the runner user, and fork-fleet.json's own _doc calls the vm pool "a sandboxed always-on runner VM", but the runbook installs it on VM106, where the agents and owner tokens live. IPAddressDeny=localhost is not a fix by itself, because tests need their own localhost servers. Fix: run the vm pool on a VM that holds nothing else, as the json says (or one VM or netns per job). Add IPAddressDeny for 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and link-local to the unit (DNS goes through 127.0.0.53, so this costs nothing). Add a runbook check that no loopback service on the fleet host accepts an unauthenticated local user. For VM106 meanwhile: set a Syncthing GUI password. (medium-high: Syncthing's no-auth localhost behaviour is documented and I read the config, but I did not probe the service)
  finding: minor .github/scripts/fork-fleet-launch.sh:124. Test gap: deleting the "in its own group only" check leaves both suites green (unprivileged 119, LauncherAsRoot 2). Inputs: a runner user that is also in docker (root on the host) or adm. Result: no test notices that the launcher would hand that user a job. Add a Launcher test with a fake id -G that reports a second group, and expect a refusal. (high)
  finding: minor .github/scripts/fork-fleet.py:378. A closed or down pool's reason is a plain log line in the Route step. Inputs: a new actions/runner release (every few weeks) closes the pool, fail closed by design. Result: in auto, every run quietly goes hosted until the owner reads a Route log or the host journal; nothing shows on the run page. Make it a ::notice:: (or a line in the step summary), at least when the why is the template version. (high)

[scuttletoby]

lens: security+correctness S-2 · reviewer: opus high (t3lens-13-s2-r5) · read at: c562e01
covered: the fix diff 0514b66..c562e01 in full (one commit, 8 files) and the round-4 lens comments S-1 and S-2. Read in full: fork-fleet-launch.sh (all verbs, with the new net_up, docker_rules, resolv_target, sandbox, check_net with its PROBE, reset_user, prune_now, check_users and setup), fork-follow-push.sh, fork-fleet-hook.sh, the fork-fleet.json upstream_sync and pool entries, and these parts of fork-fleet.py: refused, made_by_the_sync, facts_of_run, run_refused, gate, stop_job, route and check_config. Also the fork-tests.yml change, the fork-follow.yml push steps, and the runbook .github/fork-fleet.md (the trust rule, containment and network sections, races, and host steps 1 to 6). From the tests: UpstreamSync, the new Launcher network, check-net, group and lock tests, LauncherAsRoot (the shm race and check_net_proves_a_runners_network), and the new fork-follow.test.sh cases.
checks:

• Round-4 items, one by one. (1) Fleet jobs could reach the host's services, LAN and tailnet: fixed at the root. Each runner user gets its own network namespace with its own loopback, so the host's loopback services, abstract sockets and services that start listening later are out of reach by construction. Its single veth goes out through the nft table inet fork_fleet. The input hook rejects every packet from ffh-* to the host itself, in both IPv4 and IPv6, so the host's LAN, docker0/br-* and tailscale0 addresses and the gateway are all closed. The forward chain rejects RFC 1918, CGNAT, link-local, loopback, multicast and the reserved ranges, drops all IPv6, and lets nothing new open into ffh-*. Docker's published ports are DNATed to 172.x and so are still rejected in forward, and IPAddressDeny also covers the original 10.204.x.1. Runner A cannot reach runner B (10/8). The unit's IPAddressDeny (the same ranges plus ::/0, with localhost meaning its own lo) is a second, independent layer: if the table is flushed (for example a restart of nftables.service with "flush ruleset"), a running job still reaches no private address, and loses masquerade too. TemporaryFileSystem=/run, /tmp, /var/tmp and /dev/shm plus ProtectHome hide every filesystem socket: docker.sock, tailscaled.sock, dbus, /run/user. On the host no listening Unix socket exists outside /run, /tmp, /home and /dev/shm (checked with ss -xl). (2) The /dev/shm swap race: fixed at the root. Root no longer deletes anything in /dev/shm, each unit has its own /dev/shm, and the real race test (40 rounds) stays in LauncherAsRoot. (3) The sync push running on the fleet: fixed for that run. The sync tags fork-follow/synced/ before it moves main, and refused() asks with an anonymous ls-remote --exit-code. Only exit 2 (no such ref) counts as trusted; a found tag, exit 128, a timeout, no git, or a non-40-hex or short sha all refuse. The dispatcher gets head_sha from GitHub's run record. A tag can only be made by a push-rights holder, and a stray tag only ever sends a run GitHub-hosted. Faking the tag onto another commit makes no difference, because the name carries the sha and only the name is asked about. If the tag is pushed but the main push fails, the tag points at a commit that is not main, which is harmless. Fleet mode with a refused run is hosted, not queued. The runbook now says plainly what this does not cover: the owner's later PRs and pushes, built on the synced main, run upstream code on the fleet, so the network sandbox is the real protection. Minors: the own-group check and the launch lock now have tests (both go red when broken), a closed pool is a ::notice::, the json/runbook no longer say "sandboxed VM", and runbook steps 1b-3 run in set -euo pipefail subshells (but see finding 1).
• Hard rule (strangers never reach the fleet, its tokens or the host): holds. Nothing in this round weakens the trust checks in route, the dispatcher or the gate. The gate's ls-remote runs inside the sandbox, anonymously (no HOME, no global or system git config), and on any failure it kills the worker. In auto mode the sweep then re-runs the job hosted.
• Fail closed: net_up dies through must() on any nft, ip or iptables failure, on ip_forward != 1, on a resolv.conf with no nameserver or one that others can write, and on a DOCKER-USER check with no clear answer (exit other than 0 or 1). If the namespace file is missing, systemd cannot start the unit.
• State question. Launcher run again: a namespace and veth that are already there are kept, addresses and routes are replaced, and the table is swapped in one nft -f transaction (delete and re-create, conntrack keeps running jobs' flows). DOCKER-USER rules are added once (tested). Half-built: a namespace without its veth gets the veth. A host veth whose peer was left in an orphaned namespace makes the next "ip -n ... addr replace" fail, so the start dies, which is fail closed. After a reboot, /run/netns and the table are gone and every start re-makes them before the unit. A changed NET_BASE or uid leaves a stale address or veth that still matches ffh-*, which is harmless. check-net holds launch.lock for its whole probe, so no start can reset its user mid-probe. Runbook: step 2, if stopped between the two mv's, leaves no live template (the launcher fails closed); a re-run re-checks and swaps, and restarts the dispatcher. Step 3 writes SOURCE last. fork-follow-push.sh advance: a re-run after the move re-makes a missing tag (tested). Stopped after the tag but before main moved, a re-run sees main == prev and pushes the backup and the tag again, both no-ops, then moves main. With empty state (no synced tags yet) the sync works as before. Pre-PR runs have no fleet labels, so they stay hosted.
• Tests, in a scratch clone at the head with GH_TOKEN/GITHUB_TOKEN unset and an empty GH_CONFIG_DIR: fork-fleet.test.py 136 OK (4 root-only skipped), and fork-follow.test.sh all passed. I did not run LauncherAsRoot locally, because the capacity check on this host said NOT OK (load and swap). In fork CI at this head (run 43538442982, reopened) every check passed, including "Workflow lint and upstream drift" (actionlint 1.7.12) and "Fleet launcher as root", where LauncherAsRoot ran 4 tests OK with real nft and namespaces. No local actionlint or shellcheck was available. I broke 30 guards by hand, one at a time, and 29 went red: the tag refusal, git error = trusted, push skipping the sync check, head_sha missing from facts_of_run, a short-sha regex, the notice, the tag before the move, the tag on a re-run, the input reject, the IPv6 drop, the ct drop, CGNAT left out, IPAddressDeny gone, ::/0 gone, NetworkNamespacePath gone, the tmpfs gone, ProtectHome, ProtectProc, the ip_forward check, an nft failure ignored, net_up skipped, the /30 collision check, the group check, flock, an iptables -C exit 4 accepted, a nameserver-less resolv.conf, a probe failure ignored, check-net's host-reach check, and check-net without its namespace. One stayed green: a failed "ip link add" veth ignored. The next must ip addr replace on the missing device dies anyway, so it still fails closed; not a finding. Not covered by any test: the real systemd side of the sandbox (IPAddressDeny, TemporaryFileSystem plus BindReadOnlyPaths of resolv.conf into the tmpfs /run). Host step 6 (check-net with the real systemd-run and props) is the proof of that on the host. The clone stayed clean.
  finding: minor .github/fork-fleet.md:341 (host step 2). Under set -e, only the last command of an && list stops the shell. So [ -x "$new/run.sh" ] && [ -x "$new/bin/Runner.Listener" ] && [ version ] with no run.sh, or no Runner.Listener, goes on: the old template is deleted and the broken one is swapped in. Reproduced: a subshell with set -euo pipefail and no run.sh reached the swap. The launcher still refuses a template without run.sh (fail closed, but the fleet is down until step 2 is run again), and it does not check Runner.Listener. Put each test on a line of its own, or end the line with || exit 1. Same section, line 318: after the first runner start, the host has 10.204.x.0/30 routes on ffh-, so running step 1b again always prints "10.204 is in use here: pick another NET_BASE". That contradicts "safe to run again"; ignore dev ffh- in that check. (high)
  finding: minor .github/scripts/fork-fleet-launch.sh:200 + .github/fork-fleet.md:419. Public addresses are allowed, which includes the home router's own WAN address. Inputs: a trusted job running a bad dependency connects to the router's public IP. Masquerade sends it from the host's LAN address, and many home routers serve their admin UI or DNS on their WAN address to LAN clients (NAT reflection). So a LAN device stays reachable outside the private ranges. Step 6 probes only this host's own listeners, so it would not show this. Fix: an optional EXTRA_DENY setting in launch.conf (the WAN address, any other public address that leads back home) that adds to both the nft set and IPAddressDeny, plus a step-6 line that probes WAN:80/443. (medium: depends on the router; not probed)

[scuttletoby]

lens: security+correctness S-1 · reviewer: opus high (t3lens-13-s1-r5) · read at: c562e01
covered: the fix diff 0514b66..c562e01 in full (one commit, 8 files) and the round-4 lens comments S-1 (6854360796) and S-2 (6854189577). Read in full: fork-fleet-launch.sh (all verbs: conf parsing, setup/lock, ready, check_users, free_user, reset_user, net_up with the nft table, docker_rules, resolv_target, sandbox, start, the check-net PROBE and check_net, prune/prune_now), fork-fleet-hook.sh, fork-follow-push.sh (all of it, incl. tag_synced and cmd_advance), in fork-fleet.py refused, made_by_the_sync, facts_of_run, gate, the route notice and check_config's upstream_sync check, fork-fleet.json, the fork-tests.yml change, the new UpstreamSync/Launcher/LauncherAsRoot tests, the fork-follow.test.sh additions, and the runbook .github/fork-fleet.md (trust rule, containment, network, races, host steps 1-6).
checks:

• Round-4 findings, each fixed at the root. (1) Network: every runner user gets its own netns (own loopback, no abstract sockets of the host). The nft table inet fork_fleet rejects anything from ffh-* to the host itself (input hook, any address, any protocol, any port, so also a service that starts listening later), rejects RFC 1918, CGNAT, link-local, loopback, multicast and reserved ranges on forward, drops all IPv6, and drops new inbound flows into a namespace. The unit's IPAddressDeny repeats the IPv4 ranges plus ::/0 in the kernel. Docker bridges (172.17-19/16), the LAN (10.82.0.0/24), Tailscale (100.64/10 and fd7a: over IPv6), MagicDNS 100.100.100.100 and the other slot's /30 are all covered. Docker's published ports DNAT to 172.x, which the forward chain then rejects. /run, /dev/shm, /tmp and /var/tmp are the unit's own tmpfs; ProtectHome, ProtectProc=invisible, PrivateIPC, NoNewPrivileges and ProtectSystem=strict stay. On VM106 I found no listening unix socket outside /run, /tmp, /home and /dev/shm. (2) /dev/shm race: gone at the root. reset_user no longer deletes in any directory a runner user can write. prune_now runs rm -rf only on a copy whose unit is gone, and no process of that user can still write it: the cgroup is killed, and other units see it read-only. (3) Sync push: fork-follow-push.sh tags fork-follow/synced/ before it moves main, refuses to move main when the tag push fails, and tags again on the "already" path. refused() asks for the tag anonymously, with a fixed env (PATH, no global or system git config, so a runner user's ~/.gitconfig insteadOf cannot point it elsewhere), a 30 s timeout, and a full-sha check. Exit 0 means a sync commit (hosted), exit 2 means not one, and anything else (timeout, 128, OSError) refuses. The dispatcher sees the same through facts_of_run's head_sha. Can it be faked or fail open? A tag on another commit, or a stray tag, only sends a run hosted. Only push-right holders can make a tag. ls-remote's tail match can only add matches. A short or 64-hex sha refuses. If the tag push works and the main push fails, nothing runs. If the tag push fails, main stays put. Nothing deletes fork-follow/synced tags. The candidate branch and tag pushes are already refused (only refs/heads/main). Remaining gap, which the runbook names: after a sync, the owner's next PR or push to main runs that upstream code on the fleet. It is now inside the netns sandbox. Minors fixed: own-group check and launch lock are tested (both red when broken), a closed pool is a ::notice::, the json no longer says "sandboxed VM", and runbook steps 1b-3 run in set -euo pipefail subshells.
• Fail closed: start runs net_up after the lock, reset_user and before systemd-run. Every nft, ip netns, veth, addr and link step goes through must(), and ip_forward must be 1. A missing or bogus /run/netns entry makes ip netns add or systemd's NetworkNamespacePath fail, so the unit does not start. A table removed while a unit runs leaves IPAddressDeny in place (VM106 has no public IPv4 of its own, so that covers every host address). A DOCKER-USER chain that does not answer kills the start. The table is replaced in one nft transaction. Note, not a finding: IPAddressDeny is silently ignored on a host without cgroup-v2 BPF. Ubuntu 24.04 has it, so the nft table stays the first line.
• State question. Launcher run again: net_up keeps what is right, replaces addresses and the route, and reapplies the table. Stopped half-way: before the unit starts, nothing runs. A half copy is pruned. A namespace without its veth is rebuilt, and a veth whose peer is missing dies. A uid that changed leaves an old ffh-* that is still covered by the wildcard rules. After a reboot, /run, the namespaces, the table and the DOCKER-USER rules are all rebuilt on the first start. check-net holds the launch lock for its whole probe. fork-follow-push.sh run again: an existing tag on the same commit is a no-op, an existing tag on another commit refuses before main moves, and a re-run after the move retags. Runbook: step 2 interrupted between the two mv calls leaves no live template and a stopped dispatcher (closed), and a re-run only removes .old after the new template has been verified. Step 3 interrupted: mixed files. The new fork-fleet.py requires upstream_sync in the json, so an old json makes it fail closed. A running old dispatcher keeps the old rule until it restarts, but the gate (new file, run per job) checks the tag. Step 1b re-run: see the minor below.
• Tests (scratch clone at the head, GH_TOKEN and GITHUB_TOKEN unset, empty GH_CONFIG_DIR): fork-fleet.test.py 136 OK (4 root-only skipped), fork-follow.test.sh all passed. LauncherAsRoot in a throwaway ubuntu:24.04 container (--init, NET_ADMIN and SYS_ADMIN): 4/4 OK, including the real-namespace check-net. In that container, deleting the input reject rule turned 3 probes red, and deleting the forward private rule turned 2 red. Then I broke 42 guards by hand, one at a time: 41 went red. These include the netns prop, IPAddressDeny (and its ::/0), each nft rule, CGNAT and link-local, nft and netns without must, net_up dropped from start, the ip_forward check, the uid-network clash, the docker no-answer case, the tmpfs and the shm tmpfs, ProtectHome, ProtectProc, PrivateIPC, the resolv owner check and bind, check-net's host-reach check, rc and sandbox, the lock, the own group, the made_by_the_sync call, each of its fail-open paths (error, exception, exit 0, sha check, timeout, env), facts_of_run's sha, the notice, and in fork-follow-push the tag before main, the tag after main, the already-path tag, a non-fatal tag push and a changed prefix. Stayed green: the default route without must, which fails closed (no route means no network). shellcheck 0.9.0 is clean on the launcher, the hook and fork-follow-push.sh. CI at this head (restarted run 43538442982) is all green, including Workflow lint and upstream drift (actionlint 1.7.12 and LauncherAsRoot 4 OK).
  finding: minor .github/fork-fleet.md:415 Host step 6 builds its targets with ss -4ltnH, which lists no dual-stack listener. On VM106 today, xrdp listens on *:3389 (and sshd on [::]:22 as well as 0.0.0.0:22). 10.82.0.10:3389 is reachable over IPv4 from the host, but check-net is never asked to prove it closed. The input chain still blocks it, so this is only a gap in the proof. Use ss -ltnH and map * and [::] listeners to every IPv4 address, as is done for 0.0.0.0. (high)
  finding: minor .github/scripts/fork-fleet-launch.sh:200 + .github/fork-fleet.md:408 The refused ranges are by address, so a job can still reach the home connection's own public IPv4 (the router's WAN side). Through NAT loopback that can lead back to port-forwarded LAN services, or, on some routers, to the admin UI as seen from the LAN. Neither the table nor step 6 covers it. Add the public address (and the router's admin ports) to step 6's targets, or note it as an accepted limit. (medium-low: depends on the router, not probed)
  finding: minor .github/fork-fleet.md:318 Step 1b, "safe to run again": once the launcher has run, ip -4 route | grep '^10\.204\.' matches the runners' own /30 routes, so every re-run prints "10.204 is in use here: pick another NET_BASE". It also runs before step 4 and checks only 10.204, not a NET_BASE set there. Exclude dev ffh-* routes, or check the NET_BASE from launch.conf. (high)
  finding: minor .github/scripts/fork-fleet-launch.sh:290 /tmp, /var/tmp and /dev/shm became RAM tmpfs (1G each, /run unsized) on a unit with no MemoryMax. A job that writes to /tmp instead of TMPDIR (Chromium, some test tools) can hold up to about 3 GB of RAM per slot beyond the memory_mb that fits reserved, on VM106 where RAM is tight. Add MemoryMax (or MemoryHigh) to the unit near memory_mb, or smaller tmpfs sizes. (medium)