Skip to content

fork fleet: seed verb and seeds in the fleet config (D11) - #236

Draft
scuttletoby wants to merge 7 commits into
fork/desk-d10-free-contractfrom
fork/desk-d11-seed
Draft

scuttletoby wants to merge 7 commits into
fork/desk-d10-free-contractfrom
fork/desk-d11-seed

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Row D11 of the desktop pool plan, part 1 of 2: the host side of seeds, the warm caches for a desk pool's jobs. It adds seeds to a pool's fleet config entry and a seeds / seed NAME request on the desk link's forced command. Stacked on #230 (D10). Critical: it adds a second request to the desk link's forced command.

The D11 row came to 518 non-test lines, over the 400 limit, so it is split:

  • this PR: the config, the host's offer and the forced command's divert (183 non-test lines)
  • D11b, stacked on this one: the desktop fetching, its timer, the install kit and jobs reading the seeds

Why

A job's copy starts empty, so every job on a desktop downloads its tools and dependencies again. That makes it slow and wastes the PC's network. With seeds, the always-on host offers ready-made caches and the desktop keeps a recent copy that its jobs start from.

  • Config: pools[].seeds is a list of {"name", "every_hours", "max_age_hours", "max_mb"}. check_config refuses bad names, values that are not positive whole numbers (bools too), max_age_hours below every_hours, max_mb above 16384, missing fields and duplicate names.
  • Host offer: the new fork-fleet-desk-seed.py (standard library only) answers two requests.
    • seeds prints the pool's list as one JSON object: for each seed, bytes and made when it is offered now, and why when it is not.
    • seed NAME streams /var/lib/fork-fleet/seed-files/POOL/NAME.tar (a folder per pool: two pools never share a
      file), followed by a trailer line on stderr: seed NAME BYTES SHA256 MADE.
    • The fleet makes nothing itself. The project's own timer or job writes the tar (temp file and rename).
  • Forced command: fork-fleet-desk.py serve POOL already runs as the desk-link user for the desk key. When SSH_ORIGINAL_COMMAND starts with seed (so seeds, seed NAME, and also seeds x, seed, seedsx), it hands the request to the seed module instead of serving a session; the module refuses anything but seeds and seed NAME with exit 2. Every other request keeps the old path.

Safety (critical)

  • No paths from the desktop: only a name that is in the pool's own seeds list is served. The name never becomes part of a path unless it is on that list, and names match [a-z0-9_-] (the POOL pattern). seed ../m2, seed /etc/passwd, seed M2, seed m2 x, seeds x and the like all exit 2 and send nothing (tested, also through the real forced command: seeds x, seed, seedsx and seeds exit 2 and make no socket). A command that does not start with seed keeps the old path (a session request).
  • What may be sent: the file is opened with O_NOFOLLOW. It must be a plain file that no group or other user may write, in a pool folder and a files folder (not links) that no group or other user may write, at most max_mb, younger than max_age_hours, and not dated more than 5 minutes in the future. Otherwise it exits 4 and sends nothing. The folders' owner is not checked, only their mode (documented).
  • A changed file is caught: at most the checked size is read. A file changed in place while it was sent (size, mtime or ctime differ) exits 4, and the desktop then throws its copy away (D11b). A same-size rewrite with its old date put back is caught by ctime. A file replaced by a rename while it is sent is fine: the open file then has no links left (st_nlink 0), so only its ctime (changed by the unlink) is not compared, and the old whole file is sent (tested). The trailer's SHA-256 is of exactly the bytes sent.
  • Never as root, and a fleet config that will not load exits 2.
  • One stream at a time per pool: a lock in the desk link's own socket folder (POOL.seed.lock, under safe_dir). A second stream exits 3, so a misbehaving desktop cannot open many. A desktop that takes less than 64 KiB in 120 s, or not the whole file within 600 s plus one second per 256 KiB, is cut off (exit 1), so it cannot hold the lock for long. A lock file that cannot be opened or that another user owns is a clear refusal (exit 1, tested).
  • The docs say: never put a secret in a seed, because every job of the pool reads it.

What happens when this runs again, stops half-way, or meets existing, old or empty state?

Case What happens
Runs again Both requests are read-only on the host, apart from the lock file, which is reused. Asking twice gives the same answer while the file is unchanged.
Stops half-way When the ssh session dies mid-stream, the desktop gets no trailer or a short stream and keeps its old copy (D11b). The lock is an flock, so it goes with the process, and the next stream gets it (tested).
Existing / old state A file older than max_age_hours is not offered. A file replaced by rename while being sent is fine, because the open fd still reads the old whole file. A file written in place is refused. A fleet config without seeds means an empty list. Older desktops that never ask are unaffected.
Empty state A seed not made yet shows why: not made yet in the list, and seed NAME exits 4. A pool with no seeds lists [].

General vs project

General. The verb, the config check and the offer name no project. Which caches a pool has, and how the host makes each tar, are the project's own config and its own timer. The doc example (gradle) is only an example.

Upstream fit

Only the fork-only .github/scripts/fork-fleet* files, .github/fork-fleet.md and the fork CI test list change. Nothing in upstream T3 changes.

Tests

.github/scripts/fork-fleet-desk-seed.test.py (new, in .github/fork-ci-tests.txt). It uses temp folders only: no root, no ssh, no network.

  • Config: the seeds check, with a good list, an empty list, max_mb at its cap and 11 bad ones.
  • Host:
    • The list says what is offered, and why not.
    • A seed streams whole, with the right trailer.
    • Nine refused commands send nothing.
    • Nine files that may not be sent: missing, a link, group-writable, too big, too old, dated in the future, a folder, a pool folder others may write, and a files folder others may write. Each must be refused for its own reason, read from the pool's folder.
    • A second stream is refused, and the lock goes with its holder.
    • A file changed while it is sent is refused, and no more than its checked size is sent; a same-size rewrite with its old date put back is refused.
    • A file replaced by rename while it is sent is sent whole, with its trailer.
    • A lock file of another user is refused.
    • A desktop that stops reading lets the lock go; one that does not finish by the deadline is cut off; one that reads slowly but steadily is kept.
    • Two pools never share a seed file.
    • It never runs as root, and a bad config is refused.
    • The forced command, run as a real process from a copy of the host's install folder, diverts seeds, seed NAME and every other seed-prefixed command. It serves no session for them.

The existing fork-fleet-desk.test.py and fork-fleet.test.py were run too.

Local run (round 2):

  • .github/scripts/fork-fleet-desk-seed.test.py: 16 OK
  • .github/scripts/fork-fleet-desk.test.py: 42 OK
  • .github/scripts/fork-fleet.test.py: 201 OK (7 skipped)
  • Round 1: the seed tests run against the round-0 code fail 11 times (per-pool path, future date, cut length, ctime rewrite, the seed/seeds divert, the stall deadline). Some of these fail on the round-0 code only because the per-pool path changed where the test puts the tar.
  • Round 2: the seed tests run against the round-1 code fail 5 times (rename while sent, the deadline, a slow steady reader, the stall message, the max_mb cap). Mutations of the round-2 code: without the future-date check, without the lock-owner check, and with round-0's path (no pool folder), the matching tests fail (the future-dated case now says why it fails: "not made yet" instead of "dated in the future").

Red-checks

Each rule was broken in a copy, one at a time, and the named test failed (all 11 red):

  • only a listed name is sent -> test_other_commands_and_names_are_refused_and_send_nothing
  • a link is never followed (O_NOFOLLOW) -> test_a_file_that_may_not_be_sent_is_exit_4
  • a file others may write is not sent -> the same test
  • a file older than max_age_hours is not sent -> the same test
  • a file changed while it is sent is refused -> test_a_file_changed_while_sent_is_exit_4
  • one stream per pool -> test_a_second_stream_for_the_pool_is_exit_3
  • a lock of another user is refused (round 2) -> test_a_lock_of_another_user_is_refused
  • a file dated in the future is not sent (round 2) -> test_a_file_that_may_not_be_sent_is_exit_4
  • never as root -> test_never_as_root_and_a_bad_config_is_exit_2
  • config: max_age_hours at least every_hours -> Config
  • the forced command diverts seed requests -> test_the_forced_command_diverts_seed_requests

The first run found one gap: with O_NOFOLLOW removed the link case still passed, because the link's target was
group-writable and refused for that reason. The test now makes the target 0644, so only the link can be the reason.

🤖 Generated with Claude Code

A desk pool may list warm caches ("seeds") in its fleet config entry. The
always-on host's forced command answers a desktop's `seeds` (the pool's list)
and `seed NAME` (that one ready-made tar, with a size and SHA-256 trailer) in
an ssh session beside the agent's. Only names of the list, never a path;
exit 3 while another stream for the pool runs, 4 when the file may not be
sent. The desktop's fetcher follows in the next change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Review round 1 at 57e1cb5: two independent lenses (security, correctness), both SAFE WITH NITS. Nothing serves a path the desktop picks; the existing serve request is unchanged; 9 seed + 42 desk tests pass. Fix round to follow.

  1. Shared across pools (fork-fleet-desk-seed.py:33,58): the file path is FILES/NAME.tar with no pool, so a second desk pool b listing gradle streams the tar pool A's timer made. Use FILES/POOL/NAME.tar or document the sharing (fork-fleet.md:704 understates it).
  2. A file dated in the future is offered forever (:70): now - mtime is negative, so the age check never fails. Proven: a 10-years-ahead mtime with max_age_hours 48 is sent with exit 0; it should be exit 4.
  3. The stream is not capped at the checked size (:111,123): send() reads to EOF, so a file growing in place streams past max_mb and only then exits 4. Read at most st.st_size.
  4. The changed-file check skips ctime (:111): a same-size rewrite with the old mtime put back is sent with exit 0. Add st_ctime_ns.
  5. PR body overstates refusals (fork-fleet-desk.py:412-414): seed, seeds x, serve x exit 2 only when offer() is called directly (the test does). Through the forced command they fall through to serve (old behaviour, not a regression; a session lock is made, exit 1). Either divert every seed/seeds-prefixed command, or correct the body; D11b must treat a session reply to seeds as "no seeds".
  6. No deadline on send (:101-106): a desktop that stops reading holds the pool's seed lock until TCP gives up (no ClientAliveInterval by default); other seed requests get exit 3 meanwhile.
  7. Nits: the lock file's owner is not checked (:95; a root-owned leftover gives exit 1 with a terse OSError instead of the clear refusal serve gives); the seed folder's owner is not checked, only its mode (:59-61).

State question: runs again (read-only, lock reused), killed mid-stream (flock dies, no trailer), broken pipe (exit 1, files closed), missing/broken config (exit 2), pool not in config (exit 2), no seeds (empty list), missing folder/tar ("not made yet", exit 4), missing socket folder (made): all handled.

scuttletoby and others added 4 commits November 15, 2026 20:46
- seeds live in FILES/POOL/NAME.tar: pools never share a file
- a file dated more than five minutes in the future is refused (exit 4)
- the stream stops at the size that was checked; ctime and inode join the
  changed-file check
- the forced command sends every command that starts with `seed` to the seed
  handler, so `seeds x` and `seed` exit 2 and never reach a session
- a desktop that takes nothing for STALL seconds is cut off (exit 1) and the
  pool's lock let go
- the lock file must be this user's; a clear refusal otherwise

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ool (D11)

The lock check after a stall sent 1 MB into a pipe nobody read until the
call returned, and the second pool reused the first one's label and beacon,
which the config check refuses. The pool test also puts a tar at the old
shared path, so the old code fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Fix round 1 (commits 7026afd and 099f438, the second a test-only fix; D10's round 1 merged in; head 099f438). Tests: desk-seed 12 OK, desk 42 OK.

  1. Shared across pools: fixed (7026afd). The path is now FILES/POOL/NAME.tar (a folder per pool); D11b and the runbook follow. Test: test_pools_never_share_a_seed_file.
  2. A file dated in the future: fixed (7026afd). More than 5 minutes ahead is refused (exit 4, "dated in the future"). Test: a case in test_a_file_that_may_not_be_sent_is_exit_4.
  3. Stream not capped: fixed (7026afd). send() reads at most the checked size. Test: test_a_file_changed_while_sent_is_exit_4 checks the cut length.
  4. ctime: fixed (7026afd). The changed-file check compares size, mtime, ctime and inode. Test: test_a_same_size_rewrite_with_its_old_date_put_back_is_exit_4.
  5. Forced-command divert: fixed (7026afd). Every command starting with seed goes to the seed module, so seeds x, seed, seedsx, seeds exit 2 and make no socket. Test: test_the_forced_command_diverts_seed_requests. Body corrected.
  6. No send deadline: fixed (7026afd). A desktop that takes nothing for 120 s is cut off (exit 1) and the lock goes. Test: test_a_desktop_that_stops_reading_lets_the_lock_go.
  7. Nits: lock owner: fixed (an open error or another owner is a clear refusal, exit 1). Seed folder owner: documented (only the mode is checked; fork-fleet.md Seeds).

Fails before the fix: the seed tests run against the round-0 code give 10 failures and 1 error (every test above). Some fail there partly because the per-pool path moved the tar; test_pools_never_share_a_seed_file also puts a tar at the old shared path, so it fails on the old code for the real reason.

PR body updated. Still draft.

@scuttletoby

Copy link
Copy Markdown
Owner Author

Re-check lens 236s r1 (security) at 099f438: SAFE WITH NITS

Earlier findings: 1 pools sharing a seed FIXED; 2 future-dated file FIXED, but its test passes on round-0 (round-0 never reads files/POOL/, so it answers exit 4 anyway); 3 stream cap FIXED; 4 ctime FIXED; 5 forced-command divert FIXED; 6 send deadline FIXED; 7 nits FIXED.

New:

  1. Regression from fix 4 (availability): fork-fleet-desk-seed.py:123 compares st_ctime_ns on the open fd; a temp-file-and-rename over NAME.tar (the documented way to refresh a seed) changes the old inode's ctime on unlink, so a stream in progress ends with exit 4 and the desktop discards it. Large seeds with a frequent refresh may never get through. Fix idea: when after.st_nlink == 0 the inode was replaced, not written in place; ignore ctime then. Failing test (class Host):
def test_a_rename_while_sent_is_fine(self):
    old = self.make('m2', {'a': b'x' * 3000}); send = seed.send
    def renaming(fd, out, size):
        self.make('m2', {'a': b'y' * 3000})
        return send(fd, out, size)
    with mock.patch.object(seed, 'send', renaming):
        code, out, err = self.offer('seed m2')
    self.assertEqual((code, out), (0, old), err)
  1. Nit: the per-write alarm (seed.py:153) lets a 1-byte-per-119 s reader hold the pool's seed lock; an overall deadline from max_mb and a minimum rate closes it (only that pool's key can do this).
  2. Nit: the st_ino check compares fstat of the same fd, so it never differs; the PR body's "inode differ" claim is misleading.
  3. Nit: max_mb has no upper bound (fork-fleet.py:145).

Checked safe: exact-key parsing of seed NAME with [a-z0-9_-] names, traversal/extra args/newlines exit 2, O_NOFOLLOW + lstat on folders, a pool reads only its own folder, the launch path unchanged. State cases (rerun, killed mid-stream, broken pipe, module missing mid-update, broken/missing config, old config without seeds, missing tar, foreign lock) each handled. Seed tests 12 OK, desk tests 42 OK.

@scuttletoby

Copy link
Copy Markdown
Owner Author

Re-check lens 236c r1 (correctness) at 099f438: SAFE WITH NITS

Earlier findings 1-7 FIXED (round-0 seed tests: 10 failures + 1 error; each fix also removed in a copy to see its test catch it). Exceptions: the future-date case passes on round-0 too (round-0 never reads the per-pool path), though it does catch removing the check at head; the lock-owner check (fork-fleet-desk-seed.py:109) has no test (if False: leaves everything green). The old serve POOL request takes the unchanged path byte for byte. Tests: desk-seed 12 OK, desk 42 OK; fork-fleet.test.py has the same 11 failures as the base branch (launcher/network/systemd), so none come from this PR.

New:

  1. Regression (should fix before merge), same as lens 236s r1 finding 1: the ctime compare (:123) refuses a seed replaced by rename mid-stream with exit 4, contradicting the PR body ("a file replaced by rename while being sent is fine") and the docstring. Fix: skip ctime when after.st_nlink == 0.
  2. Nit: the stall alarm restarts per 1 MiB write, not per byte moved (:153), so a desktop reading slower than about 8.7 KB/s is dropped while still reading; the docs say "takes nothing for two minutes". Reword or alarm per smaller piece.
  3. Nit: no test for the lock-owner refusal (:109-110).
  4. Note for D11b: a host on the older kit ignores SSH_ORIGINAL_COMMAND and answers a seeds request with a serve session; a host stopped half-way through install gives exit 1 (fork-fleet-desk.py:426). D11b must treat a non-JSON reply or exit 1 as "no seeds".

State question: rerun (lock reused :105), killed (flock released, no trailer, desktop keeps its copy), broken pipe (fork-fleet-desk.py:427), missing/broken config or unknown pool (:51-56), missing tar or empty cache (:69), old socket folder (safe_dir fork-fleet-desk.py:59-75), foreign lock (:107, :110), FIFO/folder/link (:67, :72), truncated in place (:124). Old configs without seeds give an empty list (fork-fleet.py:142); seeds: null is refused.

scuttletoby and others added 2 commits November 15, 2026 22:49
…) (D11)

- A seed replaced by rename while it streams is sent whole: when the open
  file has no links left (st_nlink 0) it was replaced, not rewritten, so
  its ctime is not compared; size and mtime still are. The st_ino compare
  (fstat of the same fd, it never differed) is gone.
- The stall wait restarts per 64 KiB piece written, not per 1 MiB, and a
  stream has an overall deadline: 600 s plus one second per 256 KiB.
- max_mb is at most 16384 (fork-fleet.py SEED_MAX_MB).
- Tests: the lens rename test, the lock-owner refusal, every refused file
  must say its own reason (the future-dated case now fails on round-0's
  path), the deadline, and a slow but steady reader that is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

Fix round 2 at 3fb8617 (D10 round 2 merged in first)

Lens 236s r1 (security):

  1. Regression, MUST: a seed replaced by rename while it streams. Fixed in 3fb8617.
    • When the open file has no links left (st_nlink 0), it was replaced, not rewritten. Its ctime is then not compared; size and mtime still are.
    • The lens test test_a_rename_while_sent_is_fine sends the old file whole, with its trailer. It fails on round 1.
  2. Nit: the per-write alarm lets a very slow reader hold the lock. Fixed in 3fb8617.
    • Each stream now has an overall deadline: 600 s plus one second per 256 KiB.
    • The stall wait restarts per 64 KiB written, not per 1 MiB.
    • Tests: test_a_slow_desktop_meets_the_deadline, and test_a_desktop_reading_slowly_but_steadily_is_not_dropped. Both fail on round 1.
  3. Nit: the st_ino check never differs. Fixed: the check is removed, and the PR body no longer claims an inode compare.
  4. Nit: max_mb had no upper bound. Fixed: at most 16384 (fork-fleet.py SEED_MAX_MB). There is a Config test case for it.
  5. Earlier note: the future-date test passed on round 0. Fixed.
    • Every refused file must now say its own reason (assertIn(name, err)), and the tar is read from the pool's folder.
    • With round 0's path, the future-dated case fails ("not made yet" instead of "dated in the future"). With the future-date check removed, it fails too.

Lens 236c r1 (correctness):

  1. Regression: the rename case. This is the same as security finding 1 above: fixed.
  2. Nit: the stall alarm restarts per 1 MiB, but the docs say "takes nothing for two minutes". Fixed: the alarm is per 64 KiB. The docs (fork-fleet.md, the module header, the PR body) now say "less than 64 KiB in two minutes, or not the whole file by the deadline".
  3. Nit: no test for the lock-owner refusal. Fixed: test_a_lock_of_another_user_is_refused. With the check replaced by if False:, it fails.
  4. Note for D11b: an older host's seeds reply. Handled in D11b (fork fleet: a desktop fetches its desk pool's seeds (D11b) #237): a non-JSON reply or exit 1 will mean "no seeds".

State question:

  • Runs again: nothing new is stored, and the lock file is reused.
  • Stops half-way:
    • A rename mid-stream now completes.
    • A killed stream gives no trailer, and the desktop keeps its old copy.
    • A slow or stalled desktop gives up the lock within the deadline.
  • Old state: a fleet config with max_mb above 16384 is now refused by check_config. No shipped config has one.

Local run (round 2):

Test Result
desk-seed 16 OK
desk 42 OK
desk-agent 31 OK
desk-key 20 OK
fork-fleet 201 OK (7 skipped)

Red-check: the new seed tests run against round-1 code give 3 failures and 2 errors.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant