Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/fork-ci-tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ fork-ci/actions/pr-queue/lens.test.py
fork-ci/actions/pr-queue/pr_queue.test.py
fork-ci/actions/pr-queue/tested-base/tested_base.test.py
fork-kit/common/tests/machine_file.test.py
fork-kit/desk/tests/install.test.py
fork-kit/golive/tests/check.test.py
fork-kit/golive/tests/contract.test.py
fork-kit/golive/tests/journal.test.py
Expand Down
5 changes: 3 additions & 2 deletions .github/fork-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -642,8 +642,9 @@ beside the walls (one kit: the one those walls read), so `desk.conf` needs no
`LAUNCH` line; an old one that names another launcher makes the agent refuse
to run (exit 2) until the line is gone. The agent fixes the kit when it starts,
in its real path: a link above it that an upgrade repoints changes nothing
until the agent restarts, so restart it after an upgrade. Then, once per
desktop:
until the agent restarts, so restart it after an upgrade. The desk kit
([fork-kit/desk](../fork-kit/desk/README.md)) does all of this, with a PASS/FAIL
check and an uninstall; by hand, once per desktop:

```sh
install -d -m 755 /etc/fork-fleet
Expand Down
1 change: 1 addition & 0 deletions fork-kit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Every kit follows the same pattern:

| Kit | What |
| ---------------------------- | ------------------------------------------------------------------------------------------------------- |
| [desk](desk/README.md) | A desktop's side of a fork fleet desk pool: runner users, walls, desk agent; one folder (Linux, WSL2) |
| [gh-apps](gh-apps/README.md) | GitHub App tokens for agents through `gh-as`, without giving agents the keys (Linux) |
| [golive](golive/README.md) | Move a T3 boot service to a new build with a cold copy first; core and `check` so far (Linux) |
| [heavy](heavy/README.md) | `heavy <command>`: run builds and tests in a capped slice, stop what they leave behind (Linux, systemd) |
Expand Down
7 changes: 7 additions & 0 deletions fork-kit/common/machine.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,12 @@
"backupDir": "/srv/example/golive/backups",
"stateDir": "/srv/example/golive/state",
"freeFloorMb": 2048
},
"desk": {
"pool": "pc",
"host": "fleet-host.example",
"hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
"slots": 2,
"reserveMb": 4096
}
}
97 changes: 97 additions & 0 deletions fork-kit/desk/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# desk: a desktop's side of a fork fleet desk pool

Turns a Linux machine (or WSL2 on Windows) into one pool of the fork fleet: the desk agent dials out to the
always-on host over ssh, and the dispatcher there starts one-job runners on this machine through it. Everything
the fleet runbook ([.github/fork-fleet.md](../../.github/fork-fleet.md), "The desktop's side") lists by hand,
done by one installer, with a PASS/FAIL check and an uninstall.

## The machine file

The facts of this machine go in the `desk` section of `/etc/fork-kit/machine.json` (root-owned; see
[common](../common/README.md)). Made-up example:

```json
{
"version": 1,
"desk": {
"pool": "pc",
"host": "fleet-host.example",
"hostKey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...",
"slots": 2,
"reserveMb": 4096
}
}
```

`pool` must be a pool in the fleet config that is installed (the release's own has `pc`; a project's
`--fleet-config` has its own pools); `host` and `hostKey` are the always-on host and its ssh host key
(on that host: `cat /etc/ssh/ssh_host_ed25519_key.pub`, without the comment). Optional: `port` (22), `user` (`fleet-desk`), `slots` (1, at most 8; one
runner user each), `reserveMb` (memory the launcher keeps free for the owner), `netBase` (`10.N`, the runners' network,
default `10.204`: pick another when this machine's network already uses it).

## Install, check, uninstall

As root, from the fork's protected release branch:

```sh
sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA [--fleet-config /path/to/project/fork-fleet.json]
sh bootstrap.sh --repo OWNER/REPO --sha RELEASE_SHA --release-fleet-config # back to the release's own
/opt/fork-fleet/desk-install check # every line PASS, then ALL PASS
/opt/fork-fleet/desk-install uninstall
```

`--fleet-config` is the project's fleet config (its pools); without it the release's own is used. It must be in a
folder only root (or you) can write; a link is followed, and the folders of both the link and the file it names are
checked. The project's is kept in `/etc/fork-fleet/fleet-config.json` and the choice is
recorded before any file is swapped, so a later install of a new release keeps it, even after a stopped run;
`--release-fleet-config` goes back to the release's own. Install prints the desk key's public half: add it on
the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by
the refresh timer; `check` says FAIL for `template` until it is there.

`check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the
desk key's modes, the project's fleet config against the installed one, each unit against the release's (header plus
`/opt/fork-fleet/units/UNIT`), a restart a stopped install still owes, systemd drop-ins that change the kit's units,
every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings
files against the machine file, the runner users (each must exist, have no login shell, a locked password and no
line in `cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4
forwarding.

A `fork-runner-N` user made by hand before the install is used, but only when it is as locked as one the kit makes
(no login shell, locked password, not in an allow file): otherwise install refuses before changing anything and
names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`). One install or uninstall runs at a time (a lock on
`/opt/fork-fleet`); a second one is refused.

## Runs again, stops half-way, old or empty state

- **Again:** files, settings and units that did not change are not rewritten (a file whose mode or owner was
changed is put back from the release), and the agent is not stopped or
restarted (a restart would end a job it is starting). After a change it is restarted. The agent reads its settings
only when it starts, so before the first change the record gets a restart-pending mark, cleared after the
restart: a run stopped in between restarts the agent on the next run. The mark is also set before a new fleet
config choice is saved, so a run stopped between saving the choice and the swap fails `check` until the next run.
- **Stopped half-way:** the next install ends it. A new release is staged in `/opt/fork-fleet/.kit-stage-*` and
checked first (gate loads with the fleet config, the pool is in it, `bash -n`, the walls know this OS); a release
that fails is never installed. Files are renamed in one by one while the agent and the refresh timer are stopped;
`SOURCE` is written
last, so `check` reports a half-done swap. Users are recorded in `/etc/fork-fleet/desk-kit.json` before they are
made, so uninstall removes them even after a stop.
- **Old record:** a record that names a project's fleet config while neither `/etc/fork-fleet/fleet-config.json`
nor the installed copy is there is refused with the fix (`--fleet-config FILE` or `--release-fleet-config`).
- **Old install:** a launcher, `launch.conf`, `desk.conf`, desk key, unit, or a systemd drop-in for one of the kit's
units (`UNIT.d/*.conf`, also the dash-prefix forms) that this kit did not write is refused before anything changes.
The type-wide `service.d/` and `timer.d/` folders are not checked: they hold the distribution's defaults for every
unit (refusing them would refuse a stock system), and only root can write them. Remove the older (by hand) install first, the runbook's steps in reverse.
- **Empty:** `check` before an install says FAIL and writes nothing; `uninstall` with no record does nothing.

Uninstall stops every runner (through the launcher, then `systemctl stop` for any left). While one still runs it
stops there and changes nothing else: the `fork_fleet` nft table, users and record stay; run it again. Then it
removes every `fork-fleet-*` namespace and `ffh-*` link, the nft table, the kit's units, the files of its manifest,
the template and runner copies, its settings and key, the users it made, and `cron.deny`/`at.deny` when the kit
made them (otherwise the lines it added, for every runner user, also one that was there before). The record goes last. Anything else in its folders (a file the kit
did not make, a drop-in) is kept and named. IPv4 forwarding stays on until the next reboot.

## Why one flat folder

The other kits keep one folder per version behind one link. The desk agent refuses a launcher or walls file whose
folder path has a link in it (so nobody can point it elsewhere), and every fleet default names `/opt/fork-fleet`, so
this kit installs flat there: changed files are renamed in, a manifest records each hash.
71 changes: 71 additions & 0 deletions fork-kit/desk/bootstrap.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/bin/sh
# desk bootstrap: installs the desk kit (a desktop's side of a fork fleet desk pool) of one release of this fork.
# Run as root on the desktop (on WSL2: inside the distro), after the owner wrote the machine file's `desk` section:
#
# sh desk-bootstrap.sh --repo FORK_OWNER/FORK --sha RELEASE_SHA [--fleet-config FILE | --release-fleet-config]
#
# --fleet-config: the fleet config of the project whose jobs this desktop runs (that repository's file, at the commit
# its always-on host uses); without it, the release's own .github/fork-fleet.json (the fork's CI), or the project's
# from the last install. --release-fleet-config: back to the release's own after a project's was used.
# It refuses a sha that is not a commit on the fork's protected release branch (ci-release), downloads exactly that
# commit, prints the sha and the hash of what it downloaded, and runs its installer (safe to run again).
# Needs: curl, tar, python3. Linux with systemd only (WSL2 with systemd on).
set -eu
umask 022
PATH=/usr/sbin:/usr/bin:/sbin:/bin
export PATH
unset TMPDIR CURL_HOME SSL_CERT_FILE SSL_CERT_DIR || true

die() { echo "desk-bootstrap: $*" >&2; exit 1; }

repo='' sha='' branch='ci-release' fleet='' release_fleet='' api='https://api.github.com'
codeload='https://codeload.github.com'
while [ $# -gt 0 ]; do
case "$1" in
--repo | --sha | --release-branch | --fleet-config) [ $# -ge 2 ] || die "$1 needs a value" ;;
esac
case "$1" in
--repo) repo="$2"; shift 2 ;;
--sha) sha="$2"; shift 2 ;;
--release-branch) branch="$2"; shift 2 ;;
--fleet-config) fleet="$2"; shift 2 ;;
--release-fleet-config) release_fleet=1; shift ;;
*) die "unknown option $1 (see the top of this file)" ;;
esac
done
printf '%s\n' "$repo" | grep -Eqx '[A-Za-z0-9][A-Za-z0-9-]{0,38}/[A-Za-z0-9._-]{1,100}' || die "--repo must be OWNER/NAME"
printf '%s\n' "$sha" | grep -Eqx '[0-9a-f]{40}' || die "--sha must be the full 40-character release commit"
printf '%s\n' "$branch" | grep -Eqx '[A-Za-z0-9._/-]{1,100}' || die "bad --release-branch"
[ -z "$fleet" ] || [ -f "$fleet" ] || die "--fleet-config $fleet: no such file"
[ -z "$fleet" ] || [ -z "$release_fleet" ] || die "--fleet-config and --release-fleet-config: one or the other"
[ "$(id -u)" = 0 ] || die "run this as root"

work="$(mktemp -d /tmp/desk-bootstrap.XXXXXXXX)"
trap 'rm -rf "${work:?}"' EXIT
trap 'exit 1' INT TERM HUP
fetch() { curl -fsSL --proto '=https' --tlsv1.2 --max-redirs 0 -H 'Accept: application/vnd.github+json' -o "$2" "$1"; }

fetch "$api/repos/$repo/branches/$branch" "$work/branch.json" || die "cannot read branch $branch of $repo"
fetch "$api/repos/$repo/compare/$sha...$branch" "$work/compare.json" || die "cannot compare $sha with $branch"
python3 -I -B - "$work/branch.json" "$work/compare.json" "$sha" <<'PY' || die "$sha is not a release: it must be a commit on the protected branch $branch"
import json, sys
branch, compare = (json.load(open(p, encoding='utf-8')) for p in sys.argv[1:3])
# base_commit is always the sha asked for; merge_base_commit is the newest commit both share: it is the sha itself
# only when the sha is on the branch
ok = (branch.get('protected') is True and compare.get('status') in ('ahead', 'identical')
and (compare.get('merge_base_commit') or {}).get('sha') == sys.argv[3])
sys.exit(0 if ok else 1)
PY

# codeload.github.com answers a commit tarball directly (no redirect)
fetch "$codeload/$repo/tar.gz/$sha" "$work/kit.tar.gz" || die "cannot download $repo at $sha"
mkdir "$work/src"
tar -xzf "$work/kit.tar.gz" -C "$work/src" --no-same-owner --no-same-permissions
top="$(find "$work/src" -mindepth 1 -maxdepth 1 -type d)"
[ "$(printf '%s\n' "$top" | wc -l)" = 1 ] && [ -n "$top" ] || die "the download does not have one top folder"
[ -f "$top/fork-kit/desk/sbin/desk-install" ] || die "release $sha has no fork-kit/desk (is it a release with the kit?)"
echo "desk-bootstrap: release $sha of $repo, download sha256 $(sha256sum "$work/kit.tar.gz" | cut -d' ' -f1)"
set -- install --source "$top" --sha "$sha"
[ -z "$fleet" ] || set -- "$@" --fleet-config "$fleet"
[ -z "$release_fleet" ] || set -- "$@" --release-fleet-config
python3 -I -B "$top/fork-kit/desk/sbin/desk-install" "$@"
Loading
Loading