[Japabu]

Problem

toyos-metal waited on every ssh for as long as it ran. Its ssh line set only ConnectTimeout: no ServerAliveInterval/ServerAliveCountMax, and nothing bounded the remote command. On 2026-11-11 an AMD-bench boot's sudo -n /usr/sbin/reboot did not return. The machine had rebooted, booted ToyOS and was back in Ubuntu within minutes (its uptime said so when probed), but the session's close never reached the development host. ssh sat on the connection for 75 minutes until the orchestrator killed it. Only TCP's own keepalive would otherwise have ended it, and that starts after 7200 s idle on the development host (inferred from net.inet.tcp.keepidle, not observed). Every wait needs a failure timeout; these had none.

Resulting behavior

• Every ssh carries ServerAliveInterval=10 and ServerAliveCountMax=3. A machine that stops answering ends the connection with ssh's own 255 within about 40 s (four unanswered intervals). That is earlier than any command's bound and distinguishable from it; a compile-time assertion holds it under the shortest bound.
• Every command over ssh runs through one bounded wait. A command still running at its failure bound is killed and refused as Expired ("… had not ended after N s, its failure bound, so it was ended"), with exit 2 like every refusal of the loop. The bounds (Job::secs):
  • 120 s for a command that moves no image's bytes;
  • a settle gets 180 s more, udevd's own bound;
  • a write or the raw log read gets its bytes at 32 KiB/s more. That is under a fifth of the slowest flash over the bench links (153 MB in 832.7 s of dd, 184 kB/s).
  • Each write now prints its dd seconds against its bound.
  • The feed's copy and both output pipes run on their own threads, so the wait is on the command's exit and not on a pipe.
• Only a reboot the machine started is ridden when ssh fails. ssh(1) exits 255 both when the connection fails before the command ran (a refused connect, a failed key exchange: reboot was never sent) and when it drops after. So the reboot's line is echo rebooting && sudo -n '/usr/sbin/reboot':
  • a 255 that follows the machine's rebooting line is ridden as before, and the loop says ssh failed itself once the machine had started the reboot (…);
  • a 255 without that line is the loop's own refusal (Remote, exit 2), as on main.
• A boot is believed only once the firmware's list says it took BootNext. UEFI 2.10 §3.1.2: "The boot manager removes the BootNext variable before transferring control to the BootNext boot option."
  • The list read as the machine comes back is refused as Untaken (exit 2) while it still holds BootNext, before any log is read.
  • The BootNext is then deleted (efibootmgr --delete-bootnext), so no later reboot of the bench boots the entry once.
  • That check replaces --fall-through's own BootNext check. The same list is what the order check after the readback compares, so there is one efibootmgr read fewer.
• The down phase is counted from the reboot's ask, so a ridden connection's seconds stay inside Readback::ran_within's window; entry ends at the ask.
• A --delete-bootnext that fails after an Untaken prints its own refusal beside the one the loop answers with.
• --boot-internal, which rides nothing, still refuses a failed reboot.

Nothing a readback holds (boot.txt, verdict.txt) or the command line changes.

Consequential decisions

• Ride a 255 rather than refuse it, but only after the machine said it started the reboot. Refusing every dropped reboot would throw away boots that happened (an AMD-bench flash alone has taken 14 minutes). Riding every 255 would turn a failed connect into "the machine did not go down" (exit 1, the machine's failure) or worse. The rebooting line is the one fact that tells the two apart, and Untaken checks the rest after the ride.
• One bound per command, not one per run. A transfer's bound grows with its bytes; a fixed bound short enough to matter would cut a real flash over a slow link.

Checks

Evidence: ToyOSOrg/evidence@b0f828aef618 (paths masked). The logs answering review round 1 live under r2/ and was taken at the commit itself (01e092da8, clean tree). The first round's logs at 2a2d4325e stay at the branch's root.

The change is one file of toyos-metal, so the checks are that crate's:

• cargo test -p toyos-metal: 77 passed, exit 0 (log). New tests:
  • a_command_that_outlasts_its_bound_is_ended_and_refused: a sleep 30 under a 1 s bound is refused as Expired within 20 s;
  • a_bounded_command_is_fed_its_slice_and_answers_whole: the feed's slice, both streams and the status come back;
  • only_a_reboot_the_machine_started_is_ridden_when_ssh_fails: 255 after rebooting is ridden; 255 without it, and 1 after it, are refusals that are not the boot's; the reboot's line is pinned;
  • a_machine_still_holding_bootnext_never_booted_its_entry.
  • The argv test pins both options and the --delete-bootnext line.
• Round 2 (636b71537; the head db7f1e57f differs from it only in an issue record): the failed-delete print. cargo test -p toyos-metal passed 77 and clippy exited 0 (dir).
• cargo clippy -p toyos-metal --all-targets with the workspace's adopted lints and -D warnings: exit 0 (log).
• Negative controls. Each is a checked patch applied, built, its test run and reverted in one run (log, patches). Each of these fails its test (exit 101), and the restored tree passes:
  • dropping the two options;
  • waiting without the bound;
  • not killing at the bound;
  • refusing a reboot's 255 after rebooting;
  • riding a 255 without it;
  • ignoring a BootNext still set.
• The development host's OpenSSH (10.5p1) parses both options (ssh -G, which connects to nothing) (log). The UEFI text the check rests on: excerpt. The flash and log-read rates the transfer bounds rest on: excerpt.

Metal. Both benches went down on 2026-11-11, so metal is no longer this PR's landing condition.

• Shown on the AMD bench. Its two flashes of A metal row declares its benches, a boot names its bench, and judges name the one they judged #1254's boot:testcases (testcases and testcases-watchdog) ran through this branch at 01e092da8 before the benches went down (the loop's lines, masked). Both exited 0, with no Expired and no Untaken. Each reboot was echo rebooting && sudo -n '/usr/sbin/reboot' and was answered cleanly. Every piece printed its dd seconds against its bound; the largest share any piece took was 17.4 s of 1208 s. Those two readbacks were deleted before a judge read them, so wall_clock_is_firmwares was not judged on either.
• Owed on metal, recorded in issues/a-boots-readback-is-lost-to-one-ssh-that-times-out.md:
  • the T14's two flashes of boot:testcases. Each toyos-metal log of main must show:
    • an exit of 0 or 1, never 2, with no Expired and no Untaken;
    • the reboot line, then either a clean answer or ssh failed itself once the machine had started the reboot (…); it is ridden, then a readback;
    • dd took X s of its Y s bound for every piece, with the smallest margin recorded;
  • on both benches, the judge of wall_clock_is_firmwares, which must not be red on its window: down is counted from the reboot's ask.

The dropped-connection race itself cannot be provoked on demand; the host tests and the specification cover it.

Not run: the full --ci host (one crate changed; the merge queue runs it), any guest row (nothing a guest runs changed).

Risks

• Untaken is new on a normal boot's path. If a bench firmware kept BootNext after booting the entry, every run there would be refused by name (exit 2), loudly and before judging. Both benches' --fall-through runs passed the check this replaces, and the UEFI text says the boot manager removes it.
• A command bound that a real link outruns would refuse a good run as Expired. The floor is under a fifth of the slowest observed flash rate, and every write now prints its margin.
• A 255 whose rebooting line was lost with the connection is refused (exit 2) although the reboot may have run. That costs a boot and is said as the loop's failure, not the machine's.

🤖 Generated with Claude Code

[Japabu]

Review of #1255 at 2a2d432 (on f79df95)

Architecture. One owner and one wait: every ssh goes through Target::ssh into bounded, and no other Command::new("ssh") remains. A dropped connection ends as ssh's own 255 through ServerAlive*. A ridden reboot is checked against a line of the listing the loop already reads, so nothing new is added for it. The change also deletes --fall-through's duplicate check. It merges cleanly with current main and with #1254. Net: +237/−39 in tests/metal/src/lib.rs, about 120 lines of production code and 75 of tests.

Bounds. The settle bound comes from udevd's own 180 s. The floor rests on a measurement: in the orchestrator's local log amd-q-cmos-amd2-1.log, 153,255,936 bytes were written in dd 832.7 s, which is 184 kB/s. A fifth of that is 36.8 kB/s, and 32 KiB/s (32.8 kB/s) is below it, so the claim holds. COMMAND_SECS = 120 s and ALIVE_* = 10 s × (3+1) are judgments, and the compile-time assertion ties them together. Log cats are at most 1 MiB each (logkeeper's MAX_LOG_BYTES), which takes 32 s at the floor and fits under 120 s.

Untaken. I checked UEFI 2.10 §3.1.2 against the archived page: "initiate a boot of the boot option specified by the BootNext variable … on the next boot, and only on the next boot. The boot manager removes the BootNext variable before transferring control…". "Only on the next boot" also covers an entry that fails to load, which is the --fall-through case. So a compliant firmware never shows BootNext after taking it, on either path. No ToyOS code writes BootNext: the loader reads BootCurrent/BootOrder, and writes only its floor variable and MemoryOverwriteRequestControl. On a compliant firmware Untaken cannot misfire, and a non-compliant one is refused by name with exit 2. On the normal path it catches a reboot that never happened only if the machine reads as down and then answers. Blocker 1 covers the case where the machine never reads as down.

Blockers

1. tests/metal/src/lib.rs:1048-1055 (reboot_asked), :1207 (ride_the_reboot), :224 (about_the_boot): a reboot that never reached the machine is reported as a failed boot.
   • ssh(1) exits 255 for any error of its own, including a failed connect or authentication, so in that case reboot was never sent.
   • If the machine stays reachable, the ride waits 120 s on "go down" and refuses Silent { what: "go down" }. about_the_boot() marks that as the boot's failure, so toyos-metal exits 1 ("the machine is broken").
   • The printed line says ssh failed "as it does where the machine went down first", which is not established.
   • On the base, the same event was Remote, exit 2. main.rs's contract exists to keep the loop's failures apart from the machine's.
   • The PR's claim that "a reboot that never reached the machine … cannot be judged as the boot" is false for this path. A brief link flap at the moment of the reboot is enough to trigger it, and the bench links have dropped before.
   • Fix: after a ridden 255, refuse a machine that never goes down as a loop failure that names ssh's stderr, not as Silent. For example, reboot() reports that it rode and ride_the_reboot maps that Silent. The printed line should stop asserting a cause. Cheapest check: a unit test over that mapping; wait_on already takes closures.
2. tests/metal/src/lib.rs:45-49: the ALIVE_SECS comment is false within this change. With bounded, a dropped connection without these options ends at its command's bound (120 s, or a write's byte allowance), not after TCP's two hours. What the options buy now is ssh's own 255 within about 40 s, earlier than that bound and distinguishable from it. That is what lets reboot be ridden and lets a transfer fail fast. Say that, briefly.
3. The commit message states an observation that did not happen. It says the reboot "did not return for two hours", and the PR body adds that ssh sat "until TCP's own keepalive ended it". The orchestrator's record of that run says the reboot ssh hung for 75 min and was then killed. The two hours is an inference from keepidle. Correct the commit message; correcting the PR body is the orchestrator's job.

Suggestions

• Untaken returns with BootNext still naming the entry, so the bench's next reboot, for any reason, boots that image once. Consider efibootmgr --delete-bootnext before refusing.
• A ridden 255 measures down from the start of the ride, up to about 40 s after the reboot was asked. That shifts Readback::ran_within's window by the same amount, which is harmless under BENCH_CLOCK_APART_SECS (300 s). Measuring down from the ask would keep its doc ("from the reboot that started it") exact.
• Put a masked excerpt of the 184 kB/s flash (piece sizes and dd seconds) in the evidence. The comment rests on it.
• Negative controls. The five are adequate for what they target: the argv, the bound, the kill, the 255 ride and taken's logic. The call sites (run/fall_through calling taken and reboot()) and Job::secs's byte allowance are covered only by inspection. That is acceptable, except that no test reaches blocker 1's path.

Metal (LAND-pending-metal once the blockers are fixed)

For each of the four flashes (testcases and testcases-watchdog on each bench), the flasher log should show:

• the flasher's revision is this branch;
• toyos-metal exits 0 or 1, never 2;
• no Expired and no Untaken;
• the reboot either answered or printed ssh failed itself (…), and a readback followed;
• each AMD piece's dd seconds against its bound of 120 + bytes/32768, with the margin recorded.

A normal-path taken that passes on both firmwares is the measured confirmation of deletion for a loadable entry. The dropped-connection race stays covered by the host tests and the specification.

SEND BACK

[Japabu]

Follow-up review of #1255 at 01e092d (over 2a2d432, on f79df95)

Scope. This round checks the round-1 blockers and the code changed for them. Interdiff: +75/−38 in tests/metal/src/lib.rs.

• About 45 lines are production: the reboot's rebooting line, DeleteBootNext, took_bootnext, asked_at, and the dd line.
• The rest is tests and doc comments.
• It merges cleanly with main be76e6f and with A metal row declares its benches, a boot names its bench, and judges name the one they judged #1254 at 3074570.

Round-1 blockers

1. Resolved.

   • Target::remote (tests/metal/src/lib.rs:705-711) sends echo rebooting && sudo -n '/usr/sbin/reboot'.
   • reboot_asked (:1062-1070) rides a 255 only when stdout holds that line.
   • Every other exit goes to answer, which refuses it as Remote. about_the_boot excludes Remote, so the loop exits 2, as on main.
   • --boot-internal (:1773) runs the reboot through as_root, so it still refuses every non-zero exit.

   Can a reboot that happened be refused? Only if the line is lost after the shell wrote it.

   • echo is a builtin, and bash and dash flush it before sudo forks.
   • Before the network goes, sudo must exec and pass PAM, systemctl only enqueues the reboot job, and systemd must stop the session's and the network's units. sshd forwards the line well before all that.
   • ssh writes channel data to its stdout as it arrives, and drained reads that pipe until it closes. ServerAlive's 255 comes 30–40 s later.
   • So only a line still in flight when the link drops is lost. That costs one boot, reported as the loop's failure, which the PR states as a risk.

   Can a reboot that did not happen be ridden? Three things must hold at once: the line reached this host, ssh then failed itself, and sudo -n reboot failed on the machine.

   • If only sudo or reboot fails, the shell's status comes back and is refused as Remote.
   • If only the link fails, the reboot still runs, since it needs no link.
   • A 255 because the session was ended by a signal after the line is the shutdown itself.
   • That leaves an independent link failure coinciding with a failed reboot. If the outage lasts long enough to read as down, the ride ends in Untaken: exit 2, with BootNext deleted. Only a machine that reads as up throughout ends in Silent.
   • That residual is acceptable. It is far narrower than round 1's failed connect, which was ridden on every occurrence.

   only_a_reboot_the_machine_started_is_ridden_when_ssh_fails pins all four combinations and the line itself. The mutants unstarted-255-ridden and reboot-255-refused each fail it (r2/mutations/mutants.log, at 01e092d on a clean tree).

2. Resolved. The comment at :45-50 now says what the options buy under bounded: the connection ends within about 40 s, before the command's bound and distinguishable from it. It is accurate and short.

3. Resolved. The commit message says the reboot's ssh hung for 75 minutes until it was killed. It gives the 7200 s keepidle only as what "would have" ended it.

Suggestions taken

• took_bootnext (:1233-1237) deletes BootNext only where taken refuses.
  • On --fall-through that runs after the entry's own deletion. That is harmless, since it deletes the variable and not an entry.
  • The argv test pins --delete-bootnext.
• asked_at is taken just before reboot() in both run and fall_through.
  • entry now ends at the ask, so ran_within's window (back + down + 1 before back_at) begins at the ask. That is the earliest a boot can start.
  • Phases' docs match.
• The dd line prints Job::Write(o).secs(piece.len). That is the bound run_as_root applied, since fed is piece.len, and it prints only on a real write.
• r2/bounds/slowest-flash.txt carries the masked excerpt that the floor rests on.

Suggestion

• tests/metal/src/lib.rs:1235: a failed --delete-bootnext is dropped silently after its run: line was printed, so the log reads as if the bench were left clean. Printing the error costs one line. Not blocking.

Evidence

The checks ran at 01e092d on a clean tree:

• cargo test -p toyos-metal: 77 passed;
• clippy with the workspace's lints: exit 0;
• all six mutants fail, and the restored tree passes.

That is sufficient for this round. Nothing else needs to run.

Metal (LAND-pending-metal)

These are the queued boot:testcases flashes, two on each bench, run through this head. Each of the four flasher logs must show:

• toyos-metal at 01e092d, exiting 0 or 1 and never 2, with no Expired and no Untaken;
• run: echo rebooting && sudo -n '/usr/sbin/reboot', then either a clean answer or ssh failed itself once the machine had started the reboot (…); it is ridden, then a readback;
• dd took X s of its Y s bound for every piece, with the smallest margin recorded, the AMD bench's especially;
• wall_clock_is_firmwares not red on its window. It is a TESTCASES row on every bench and reads ran_within, so it exercises down counted from the ask.

The two AMD flashes at 2a2d432 are local logs only, so they are not evidence here.

LAND (pending metal)

[Japabu]

Orchestrator verification, landing under the benches-down rule (metal unavailable): review LAND pending metal. Since the reviewed 01e092d: the review's optional suggestion (a failed BootNext deletion is now printed, not discarded: tests/metal/src/lib.rs +4/−2, read by me; cargo test -p toyos-metal 77 passed, clippy 0 per the author) and the owed-on-metal record. The AMD half ran before the benches hung (flasher logs published as evidence); the T14 half and the judge are owed, as recorded.