Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/fork-fleet.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@
},
"pools": [
{
"_doc": "The owner's PC: its dispatcher runs on the always-on host and starts runners on the PC over ssh.",
"_doc": "The owner's PC, a desk pool: its desk agent dials out to the always-on host, whose dispatcher for this pool starts runners on the PC through that link (fork-fleet.md, The desk link).",
"name": "pc",
"label": "fleet-pc-t3code",
"slots": 1,
"beacon": "FLEET_BEACON_PC",
"launch": ["/opt/fork-fleet/launch-pc"],
"launch": ["python3", "-I", "/opt/fork-fleet/fork-fleet-desk.py", "launch", "pc"],
"waits": false
},
{
Expand Down
36 changes: 30 additions & 6 deletions .github/fork-fleet.md
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,7 @@ done
scripts/fork-fleet-hook.sh:hook.sh:755 scripts/fork-fleet-launch.sh:fork-fleet-launch.sh:755
scripts/fork-fleet-desk.py:fork-fleet-desk.py:644 scripts/fork-fleet-desk-wire.py:fork-fleet-desk-wire.py:644
scripts/fork-fleet-desk-seed.py:fork-fleet-desk-seed.py:644
scripts/fork-fleet-desk-key.py:fork-fleet-desk-key.py:644
scripts/fork-fleet-refresh.service:fork-fleet-refresh.service:644
scripts/fork-fleet-refresh.timer:fork-fleet-refresh.timer:644
scripts/fork-fleet-job-paths.txt:fork-fleet-job-paths.txt:644"
Expand Down Expand Up @@ -572,10 +573,15 @@ Do not name the dispatcher's unit `fork-fleet-*`: that prefix is for runner
units. Runner units are separate transient units, so restarting the dispatcher
leaves running jobs alone.

**The `pc` pool** stays off until `/opt/fork-fleet/launch-pc` exists: a wrapper
that runs `fork-fleet-launch.sh` on the PC over ssh. Run the dispatcher with
`--pool vm` until then. Without its dispatcher, `pc` has no heartbeat, and
`route` never counts it. Whether and how the PC joins is an owner decision.
**The `pc` pool** is a desk pool: the owner's PC, through the desk link below.
It stays off until its own dispatcher unit runs (`t3code-fleet-dispatcher-pc`,
"One dispatcher per desk pool") and the PC's desk key is added: without its
dispatcher, `pc` has no heartbeat and `route` never counts it. While the PC
sleeps or is off, `launch` exits 3 at once and its jobs go elsewhere. A host
that ran an older `pc` dispatcher on `/opt/fork-fleet/launch-pc` first stops
that dispatcher and lets its runners finish (this config's cleanup skips a pool
that is down, so it never stops or prunes them); then it may delete the
wrapper: nothing reads it.

### The desk link (a desktop pool)

Expand All @@ -585,7 +591,7 @@ out to the always-on host over ssh, as a dedicated unprivileged user (default
pool. The pool's dispatcher on the always-on host sends the launcher's verbs
through that one session (`fork-fleet-desk.py`, its wire format
`fork-fleet-desk-wire.py`). Once per always-on host, as root (step 3 installs
both files):
these files and the key tool `fork-fleet-desk-key.py`):

```sh
# The desk-link user: a real shell (sshd runs the forced command through it, so nologin would close the link) and no
Expand All @@ -596,7 +602,7 @@ usermod -p '*' fleet-desk
echo 'd /run/fork-fleet-desk 0710 fleet-desk fleet-desk -' > /etc/tmpfiles.d/fork-fleet-desk.conf
systemd-tmpfiles --create /etc/tmpfiles.d/fork-fleet-desk.conf
# Per desk pool: the desktop's public key, as the desk-link user, with the key tool fork-fleet-desk-key.py
# (row D4 of the desktop-pool design; install it beside the others).
# (step 3 installs it).
sudo -u fleet-desk -H /usr/bin/python3 -I /opt/fork-fleet/fork-fleet-desk-key.py --config /opt/fork-fleet/fork-fleet.json add POOL < desk-POOL.pub
```

Expand All @@ -610,6 +616,24 @@ Another user name: use it in all three places above, and give the dispatcher
tmpfiles line; the desk-link user must be in it, and so must the dispatcher's
user when that is not root.

A desk pool's entry in a project's fleet config (made-up names; `seeds` is
optional, see "Seeds" below):

```json
{
"name": "desktop",
"label": "fleet-desktop-example",
"slots": 2,
"beacon": "FLEET_BEACON_DESKTOP",
"launch": ["python3", "-I", "/opt/fork-fleet/fork-fleet-desk.py", "launch", "desktop"],
"waits": false
}
```

`waits: false`: a desktop may sleep, so a job never waits for it. `slots` at
most the desktop's own `slots` (its machine file). The pool's name is the one in
the desktop's machine file and in the key line.

`serve` keeps one session per pool (a lock), listens on a 0660 socket in
`/run/fork-fleet-desk` (it lets in only root, its own user and the socket
group), and pings the agent; `launch` fails at once (exit 3) when no session is
Expand Down
7 changes: 5 additions & 2 deletions .github/scripts/fork-fleet-desk-key.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,10 @@ def setUp(self):
self.home = os.path.join(self.tmp, 'home')
os.mkdir(self.home)
cfg = dk.ff.load()
cfg['pools'] = cfg['pools'] + [desk('desk'), desk('desk2')]
# only this test's desk pools: the release's own desk pool (pc) is left out by name, never by the function
# under test; vm stays, so a non-desk pool is always there to be refused
cfg['pools'] = [p for p in cfg['pools'] if p['name'] != 'pc'] + [desk('desk'), desk('desk2')]
assert 'vm' in [p['name'] for p in cfg['pools']]
self.config = os.path.join(self.tmp, 'fleet.json')
with open(self.config, 'w') as f:
json.dump(cfg, f)
Expand Down Expand Up @@ -151,7 +154,7 @@ def test_refuses_anything_but_one_valid_ed25519_public_key(self):
self.assertIn('private key', self.run_tool('add', 'desk', stdin=PRIVATE)[1])

def test_refuses_a_pool_it_does_not_know(self):
for pool in ('nope', 'vm', 'pc', '../desk', 'desk "x"'):
for pool in ('nope', 'vm', '../desk', 'desk "x"'):
with self.subTest(pool):
code, out = self.run_tool('add', pool, stdin=PUB1)
self.assertEqual(code, REFUSED, out)
Expand Down
6 changes: 4 additions & 2 deletions .github/scripts/fork-fleet-refresh.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -967,12 +967,14 @@ def test_the_service_cannot_write_run_dir(self):

def test_the_service_writes_the_templates_own_folder_only(self):
"""The writable folder is the default template's own folder, and none of the files root runs (the launcher,
the dispatcher, its trust config, the job hook, the pc wrapper) is in it or under it."""
the dispatcher, its trust config, the job hook, the desk link and its wire format, seed and key tools) is in it
or under it."""
service = unit('fork-fleet-refresh.service')['Service']
rw = [p.lstrip('-') for p in service['ReadWritePaths'].split()]
self.assertEqual(rw, [os.path.dirname(default_template())])
root_dir = os.path.dirname(service['ExecStart'].split()[0])
for f in ('fork-fleet-launch.sh', 'fork-fleet.py', 'fork-fleet.json', 'hook.sh', 'launch-pc', 'SOURCE'):
for f in ('fork-fleet-launch.sh', 'fork-fleet.py', 'fork-fleet.json', 'hook.sh', 'fork-fleet-desk.py',
'fork-fleet-desk-wire.py', 'fork-fleet-desk-seed.py', 'fork-fleet-desk-key.py', 'SOURCE'):
path = f'{root_dir}/{f}'
self.assertFalse(any(path == w or path.startswith(w.rstrip('/') + '/') for w in rw), path)
self.assertNotIn('AF_UNIX', service['RestrictAddressFamilies'].split())
Expand Down
Loading
Loading