Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/fork-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -729,6 +729,55 @@ how often a desktop asks: the key belongs to the pool's own desktop, one stream
per pool runs at a time, and each is cut off at its deadline, so the most it
costs is one file's reads and upload at a time.

The desktop's side is the same script's `run`, hourly from
`t3code-fleet-seed.timer` (the desk kit installs both). It takes only the
names its own installed fleet config lists for its pool, and holds them to
its own `every_hours` and `max_age_hours` and the smaller of its own and the
host's `max_mb`: a host cannot make a desktop take more. Each new seed is
unpacked with Python's tar data filter and a limit on what it would make on
disk (each member's size plus 4 KiB for it and for each folder it makes, all
within `max_mb`; a size below 0, or a size on a member that is not a file, is
refused; at most 200,000 members and folders; no member more than 64
folders deep or with a name over 1 KiB) and on memory (members are let go as
they are unpacked; the extended pax or long-name headers before one member, and
the global ones of the whole tar, at most 1 MiB; sparse members are refused, so
make a seed without `--sparse`; a hard link must name a file unpacked before
it, and a link whose path is already there, or that cannot be made, is refused,
never copied from its target, which tarfile would look for in the rest of the
stream), checked against the trailer (its size
and date must be the list's too), checked again for links that lead out of
it, made readable by every job (a cache's 0600 files too), and swapped in by a
rename; ssh's stderr and the list are read with a limit. A seed that fails in
any way is named in the log and the run goes on with the next; old copies
are pruned at the start and the end of every run (so a run killed by its unit's
memory or time limit cannot pile them up). A run fetches for at most two hours
in all (the unit allows three), each seed at most 30 minutes. A seed torn again
and again is tried again after about 1, 2, 4, 8, 16 and then every 24 hours (a
link too slow for it does not spend every hour on it; a fetch that kills the
run counts as torn; the times are approximate: the timer's 10 minute random
delay and a 5 minute margin can make a wait up to an hour longer), and at once
when the host offers another copy of it. One torn on every try for more than a
day makes the run exit 1, so `desk-install check` shows FAIL until it is
fetched (`/var/lib/fork-fleet/seeds/.torn.json` names it and why); so does a
run that cannot write that file (a full disk). A host that gives no list (down,
an older kit, a half-done install) means no new seeds: each one here stays until
its age drops it, and the torn record is left as it is. A desktop whose own fleet config is broken or lacks its pool fetches
nothing and exits 2 (`desk-install check` shows the seed service's last
result), but still drops each seed past the `max_age_hours` it was fetched
with. Jobs see
`/var/lib/fork-fleet/seeds/NAME` read-only (the launcher's `ALLOW_PATHS`;
the unpacking and old copies beside it, `.new` and `.v`, are visible too and
hold the same data), copy what they need from it when it is there, and run
cold when it is not. A replaced copy is kept 24 hours for a job still reading
it and removed by the next hourly prune, so it lives up to about 25 hours;
there is no free-space check before unpacking (a full disk ends that seed's
fetch and keeps the old one). **Disk budget per seed:** up to `max_mb` x
(25 / `every_hours`, rounded up, + 2): the copy in use, the copies replaced in
the last 25 hours, and one being unpacked (`every_hours` 1 and `max_mb` 4096:
up to about 108 GiB; `every_hours` 6: about 28 GiB). Size `max_mb` and `every_hours` to the
desktop's free disk. The state table (a torn download, the host down,
clocks that disagree, a run that stopped half-way) is in the script's header.

## Owner steps on the repository

Run these from a machine where `gh` is logged in as the owner. None of them
Expand Down
629 changes: 628 additions & 1 deletion .github/scripts/fork-fleet-desk-seed.py

Large diffs are not rendered by default.

835 changes: 828 additions & 7 deletions .github/scripts/fork-fleet-desk-seed.test.py

Large diffs are not rendered by default.

46 changes: 46 additions & 0 deletions .github/scripts/fork-fleet-seed.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Fork only: a desk pool's warm caches, the desktop's side (fork-fleet-desk-seed.py run; .github/fork-fleet.md,
# "Seeds"). The desk kit installs it as /etc/systemd/system/t3code-fleet-seed.service with its timer; not as
# fork-fleet-*, the runner units' prefix. Safe to start by hand: a second run while one runs exits at once (a lock).
#
# Root, so the seeds are root's and no job can change one; sandboxed: it may write only the seeds' folder
# (StateDirectory makes it, 0755; new copies are unpacked inside it, so a rename never crosses a mount), reads the
# desk link's settings and key in /etc/fork-fleet, reaches the network (ssh to the always-on host only, with the
# agent's pinned host key) and has a memory, CPU and task limit.
# The sandbox lines below are safety-critical: they bound what a tar that slips past Python's data filter (some
# releases have known bypasses; the script also checks every link after unpacking) could reach. Do not loosen them.
[Unit]
Description=t3code fork fleet: warm caches for a desk pool's jobs
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 -I /opt/fork-fleet/fork-fleet-desk-seed.py run
TimeoutStartSec=3h
Nice=10
IOSchedulingClass=idle
UMask=0022
StateDirectory=fork-fleet/seeds
StateDirectoryMode=0755
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
NoNewPrivileges=yes
RestrictSUIDSGID=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
CapabilityBoundingSet=CAP_DAC_OVERRIDE
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
LockPersonality=yes
RestrictRealtime=yes
SystemCallArchitectures=native
MemoryMax=512M
MemorySwapMax=0
CPUQuota=100%
TasksMax=32
13 changes: 13 additions & 0 deletions .github/scripts/fork-fleet-seed.timer
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Fork only: runs t3code-fleet-seed.service every hour, at a random minute in its first 10, and after a wake or a
# boot when a run was missed (a desktop that slept). A run with nothing new asks the always-on host for one short
# list and fetches nothing; each seed's every_hours (fork-fleet.json) paces the downloads.
[Unit]
Description=t3code fork fleet: hourly warm caches for a desk pool

[Timer]
OnCalendar=hourly
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target
31 changes: 20 additions & 11 deletions fork-kit/desk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,14 +50,14 @@ recorded before any file is swapped, so a later install of a new release keeps i
the always-on host with `fork-fleet-desk-key.py add POOL`. The first runner template is made in the background by
the refresh timer; `check` says FAIL for `template` until it is there.

`check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the
desk key's modes and root owner (not a link), the project's fleet config against the installed one, each unit against
the release's (header plus `/opt/fork-fleet/units/UNIT`, mode and root owner), a restart a stopped install still owes, systemd drop-ins that change the kit's units,
every wall of this OS (`fork-fleet-walls.py describe`), the agent's settings, the units, the template, the settings
files against the machine file (text, mode and root owner), the runner users (each must exist, have no login shell,
a locked password, a uid other than 0, its own group only, no home that is there (`/nonexistent`) and no line in
`cron.allow`/`at.allow`; and it names a user the kit made that a smaller `slots` no longer uses) and IPv4
forwarding.
`check` only reads: the installed files against their manifest (text, root owner and mode), the folders' and the desk
key's modes and root owner (not a link), the project's fleet config against the installed one, each unit against the
release's (header plus `/opt/fork-fleet/units/UNIT`, mode and root owner), a restart a stopped install still owes,
systemd drop-ins that change the kit's units, every wall of this OS (`fork-fleet-walls.py describe`), the agent's
settings, the units (and the seed service's last result), the template, the settings files against the machine file
(text, mode and root owner), the runner users (each must exist, have no login shell, a locked password, a uid other
than 0, its own group only, no home that is there (`/nonexistent`) and no line in `cron.allow`/`at.allow`; and it
names a user the kit made that a smaller `slots` no longer uses) and IPv4 forwarding.

A `fork-runner-N` user made by hand before the install is used, but only when it passes the same checks (no login
shell, locked password, uid not 0, its own group only, no home that is there, not in an allow file; the launcher
Expand All @@ -67,6 +67,14 @@ names the fix (`usermod -s /usr/sbin/nologin`, `passwd -l`, `gpasswd -d`, `userm
hand setup) is refused the same way, not followed or fixed. One install or uninstall runs at a time (a lock on
`/etc/fork-fleet`, which only root can open); a second one is refused.

The seed timer (`t3code-fleet-seed.timer`) fetches the pool's warm caches, if its fleet config lists any, into
`/var/lib/fork-fleet/seeds`, which jobs see read-only ([fork-fleet.md](../../.github/fork-fleet.md), "Seeds").
A torn seed is tried again after about 1, 2, 4, 8, 16, then 24 hours (up to an hour later: the timer's random delay),
or at once when the host offers another copy; one torn on every try for more than a day makes each run that reads the
host's list exit 1, so `check` shows the seed service's last run as failed (a run that cannot write the record, a full
disk, exits 1 too). `/var/lib/fork-fleet/seeds/.torn.json` names it and the host's reason (jobs can read that file
too); a run without the host's list leaves it as it is.

## Windows (WSL2)

[windows/start-desk.ps1](windows/start-desk.ps1) does the Windows side and then this kit inside one WSL2 distro.
Expand Down Expand Up @@ -135,9 +143,10 @@ parts run in the tests on a fake root.
Uninstall stops every runner (through the launcher, then `systemctl stop` for any left). While one still runs it
stops there and changes nothing else: the `fork_fleet` nft table, users and record stay; run it again. Then it
removes every `fork-fleet-*` namespace and `ffh-*` link, the nft table, the kit's units, the files of its manifest,
the template and runner copies, its settings and key, the users it made, and `cron.deny`/`at.deny` when the kit
made them (otherwise the lines it added, for every runner user, also one that was there before). The record goes last. Anything else in its folders (a file the kit
did not make, a drop-in) is kept and named. IPv4 forwarding stays on until the next reboot.
the template, runner copies and seeds, its settings and key, the users it made, and `cron.deny`/`at.deny` when the kit
made them (otherwise the lines it added, for every runner user, also one that was there before). The record goes
last. Anything else in its folders (a file the kit did not make, a drop-in) is kept and named. IPv4 forwarding
stays on until the next reboot.

## Why one flat folder

Expand Down
29 changes: 19 additions & 10 deletions fork-kit/desk/sbin/desk-install
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
install reads the `desk` section of the machine file, makes the runner users, stages the fleet files of one release
(launcher, walls, desk agent, gate hook and the fleet config), checks the staged copy, stops the agent, moves each
file into /opt/fork-fleet, writes MANIFEST.sha256 and then SOURCE (last), writes the settings in /etc/fork-fleet and
the units, makes the desk key once, and starts the agent and the refresh timer (restarting the agent only when
something changed). Safe to run again and to stop at any step: a stopped run leaves SOURCE naming the old release
(check says what differs) and the next run ends it. uninstall removes only what the kit made, its record last.
the units, makes the desk key once, and starts the agent and the refresh and seed timers (restarting the agent only
when something changed). Safe to run again and to stop at any step: a stopped run leaves SOURCE naming the old
release (check says what differs) and the next run ends it. uninstall removes only what the kit made, its record last.
"""
import argparse, contextlib, fcntl, hashlib, importlib.util, json, os, pwd, re, shutil, stat, subprocess, sys, tempfile

Expand All @@ -34,15 +34,20 @@ FILES = {'fork-fleet-launch.sh': '.github/scripts/fork-fleet-launch.sh',
'fork-fleet-walls.py': '.github/scripts/fork-fleet-walls.py',
'fork-fleet-desk-agent.py': '.github/scripts/fork-fleet-desk-agent.py',
'fork-fleet-desk-wire.py': '.github/scripts/fork-fleet-desk-wire.py',
'fork-fleet-desk-seed.py': '.github/scripts/fork-fleet-desk-seed.py',
'hook.sh': '.github/scripts/fork-fleet-hook.sh', 'fork-fleet.py': '.github/scripts/fork-fleet.py',
'fork-fleet.json': '.github/fork-fleet.json', 'desk-install': 'fork-kit/desk/sbin/desk-install',
'machine_file.py': 'fork-kit/common/machine_file.py', 'README.md': 'fork-kit/desk/README.md',
'free_contract.py': 'fork-kit/common/free_contract.py', 'free-memory': 'fork-kit/common/free/memory'}
WALLS = '.github/scripts/fork-fleet-walls' # copied whole: every OS's walls file and fixture
UNITS = {'fork-fleet-desk-agent.service': '.github/scripts/fork-fleet-desk-agent.service',
't3code-fleet-refresh.service': '.github/scripts/fork-fleet-refresh.service',
't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer'}
't3code-fleet-refresh.timer': '.github/scripts/fork-fleet-refresh.timer',
't3code-fleet-seed.service': '.github/scripts/fork-fleet-seed.service',
't3code-fleet-seed.timer': '.github/scripts/fork-fleet-seed.timer'}
AGENT, TIMER, REFRESH = 'fork-fleet-desk-agent.service', 't3code-fleet-refresh.timer', 't3code-fleet-refresh.service'
SEED_TIMER, SEED = 't3code-fleet-seed.timer', 't3code-fleet-seed.service'
SEEDS = WORK + '/seeds' # the warm caches (fork-fleet-desk-seed.py); jobs see them read-only (ALLOW_PATHS)
FOREIGN = (OPT + '/fork-fleet-launch.sh', ETC + '/launch.conf', ETC + '/desk.conf', KEY,
*(f'{UNIT_DIR}/{u}' for u in UNITS))
CMD = {k: [k] for k in ('useradd', 'userdel', 'loginctl', 'systemctl', 'sysctl', 'ssh-keygen', 'ip', 'nft', 'bash',
Expand Down Expand Up @@ -351,7 +356,7 @@ def read_manifest():

def swap(tmp, manifest, sha, before_change):
"""Each changed file renamed into /opt/fork-fleet (the agent stopped meanwhile, so no verb runs from half-old,
half-new files, and the refresh timer and service stopped too: refresh runs the launcher); files of the old
half-new files, and the refresh and seed timers and services stopped too: they run its files); files of the old
release that are gone, removed. Unchanged files (same text, mode and owner root) are left as they are; one whose
mode or owner was changed is replaced by the staged copy. True when anything changed."""
def same(rel):
Expand All @@ -363,7 +368,7 @@ def swap(tmp, manifest, sha, before_change):
gone = sorted(set(read_manifest()) - set(manifest))
if changed or gone:
before_change()
run('systemctl', 'stop', TIMER, REFRESH, AGENT, check=False)
run('systemctl', 'stop', TIMER, REFRESH, SEED_TIMER, SEED, AGENT, check=False)
for rel in changed:
os.makedirs(os.path.dirname(p(f'{OPT}/{rel}')), mode=0o755, exist_ok=True)
os.replace(os.path.join(tmp, rel), p(f'{OPT}/{rel}'))
Expand Down Expand Up @@ -414,7 +419,7 @@ def make_users(users, state):

def settings(m, users):
known = (m['host'] if m['port'] == 22 else f'[{m["host"]}]:{m["port"]}') + ' ' + m['hostKey']
files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\n'
files = {ETC + '/launch.conf': (f'RUNNER_USERS={",".join(users)}\nALLOW_PATHS={SEEDS}\n'
+ (f'RESERVE_MB={m["reserveMb"]}\n' if m['reserveMb'] is not None else '')
+ (f'NET_BASE={m["netBase"]}\n' if m['netBase'] else '')
+ (f'FREE_COMMAND={m["freeCommand"]}\n' if m['freeCommand'] else ''), 0o600),
Expand Down Expand Up @@ -558,7 +563,7 @@ def install(a, stop_after=None, stack=None):
if r.returncode:
raise Refused(f'the desk agent refuses its settings (not started): {r.stderr.strip()[:300]}')
run('systemctl', 'daemon-reload')
run('systemctl', 'enable', '--now', TIMER)
run('systemctl', 'enable', '--now', TIMER, SEED_TIMER)
if not os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')):
run(p(OPT + '/fork-fleet-launch.sh'), 'prune', check=False) # makes the launch lock refresh opens
run('systemctl', 'start', '--no-block', REFRESH) # the first runner template, in the background
Expand Down Expand Up @@ -601,10 +606,13 @@ def check():
row(False, 'walls', f'exit {walls.returncode}: {walls.stderr.strip()[:200]}')
r = py(p(OPT + '/fork-fleet-desk-agent.py'), 'check')
row(r.returncode == 0, 'agent-settings', (r.stdout or r.stderr).strip()[:200])
for unit in (AGENT, TIMER):
for unit in (AGENT, TIMER, SEED_TIMER):
on = run('systemctl', 'is-active', '--quiet', unit, check=False).returncode == 0
row(on and run('systemctl', 'is-enabled', '--quiet', unit, check=False).returncode == 0, 'unit',
f'{unit} enabled and active')
last = run('systemctl', 'show', '-p', 'Result', '--value', SEED, check=False).stdout.strip() or 'unknown'
row(last == 'success', 'unit', f'{SEED} last run: {last}' + ('' if last == 'success' else
f' (journalctl -u {SEED} says why)'))
row(os.path.exists(p(OPT + '/template/runner-template/.fork-fleet-version')), 'template',
'the runner template is there (the refresh timer makes it)')
try:
Expand Down Expand Up @@ -740,7 +748,7 @@ def uninstall(stack=None):
say(f'no {STATE}: nothing here was installed by this kit (nothing changed)')
return 0
lock(stack if stack is not None else contextlib.ExitStack()) # the record is in /etc/fork-fleet: it is there
for unit in (AGENT, TIMER, REFRESH):
for unit in (AGENT, TIMER, REFRESH, SEED_TIMER, SEED):
run('systemctl', 'disable', '--now', unit, check=False)
launcher = p(OPT + '/fork-fleet-launch.sh')
if os.path.exists(launcher):
Expand Down Expand Up @@ -772,6 +780,7 @@ def uninstall(stack=None):
*(n for n in os.listdir(p(OPT)) if n.startswith('.kit-stage-'))] if os.path.isdir(p(OPT)) else []:
remove(f'{OPT}/{rel}')
remove(WORK + '/runners')
remove(SEEDS)
for u in state['createdUsers']:
if user_exists(u):
run('userdel', u)
Expand Down
Loading
Loading