[Japabu]

Problem

Nothing in the native write path could make one write durable by itself. The block protocol, the NVMe driver, the USB mass-storage driver and bcachefs knew a write and a flush of the whole device, and nothing else. So each bcachefs journal entry was written between two flushes (flush, entry, flush), and an entry that names itself between three.

Behavior

• A write can be asked onto the medium by itself. BlockDev::write_durable returns once its blocks are on the medium, and makes nothing else durable.
• A journal entry is one flush and one forced write. The flush before it still makes durable everything the entry depends on. An entry naming itself is forced headless and then its header block forced, with no flush between. A commit of names alone, with nothing written to the disk since the last flush, is one forced write and no flush: the file server's cache already asks the disk for no flush when nothing was written.
• NVMe sets FUA in the Write command (NVM Command Set 1.0 §3.2.6, Command Dword 12 bit 30). Every controller takes the bit, so every NVMe session offers it.
• USB mass storage sets FUA in WRITE(10) (SBC-3 §5.32, byte 1 bit 3) where the device's mode parameters set DPOFUA, which the driver now asks once, with MODE SENSE(6), as a disk first comes up.
• A device that cannot force a write falls back to write-then-flush, and its log says so. It is one rule at two sites. A served disk: the session's open says whether its device forces writes, and the session's client (Served::write_shared_durable) writes and flushes when it does not; a server refuses a forced write it did not offer. Every other device (RAM, the crash model, wrappers that do not forward): write_durable's default. The driver's bring-up line says which: a forced write is a Write with FUA (NVMe), DPOFUA set, so a forced write is a WRITE(10) with FUA or no DPOFUA, so a forced write is a write and a flush (USB).

Decisions worth a reviewer's eye

• A forced write acknowledged is kept by nobody. The server records it under no loss count and the client keeps no copy to issue again, because no reset can take it. The price is a rule: a forced write never goes over a write of the same client that no flush has covered, or a reissue after a loss would put the older bytes back. The block client enforces it at run time (r1): it panics rather than send a forced write over a kept one. The journal holds it because each attempt at an entry starts with its own flush, which covers what a refused attempt left written; a test holds that flush inside the retry.
• A reconnect adopts the device's new answer for forced writes (r1). It is held to the partition's length and GUID only, so a driver that reads DPOFUA differently after a restart does not cost the volume.
• The cache runs a durable write in the flush's place in line. One flush or durable write at a time, so no flush carries an older copy of the same block beside it.
• MODE SENSE is new traffic to every stick at bring-up. It asks for all pages (page code 3Fh) with 192 bytes allocated and reads only the 4-byte header, which it believes only if its MODE DATA LENGTH covers it. A stick that refuses it is served without FUA. A stick that breaks on it would climb the recovery ladder like any broken command; none has been tried (see Risks). Why that page and length, and that the pair is my recollection and not a checked source, is in the evidence README.
• osbench_disk (first commit) is the metal row nvme: the controller's own idle power states, by APST #1131 introduced: files, nvme and persist alone. It lands here because this change is measured with it and the base arm needs it. Only a pass that names it runs it.
• The manual's storage page gains the decision (a write may be asked onto the medium by itself; the fallback).

Checks

The change was checked at 8a8ff27b9; the review's fixes at the head, fea712a16.

check	result
cargo run -- --ci host at fea712a16	exit 1: 110 of 111 steps green. The one red is llvm::tests::the_caller_s_environment_reaches_neither_the_build_nor_the_key, red on every macOS host (issues/the-llvm-environment-test-breaks-git-on-macos.md).
cargo run -- --build-only at fea712a16	exit 0
guest rows at 8a8ff27b9: nvme_disk_boots, the_data_partition_beside_the_stick_is_designated, the_log_lands_on_the_boot_stick	all PASS, run 42918253433
the same rows at fea712a16	all PASS, run 42928628188. QEMU's stick answered: serving it as disk 0: DPOFUA set, so a forced write is a WRITE(10) with FUA.
control, committed	mutate-fua-goes-out-unforced reds model::tests::a_forced_write_answered_done_stays_on_the_medium
control, in the test	a device that leaves the entry in its cache loses a committed create
control, by hand	the journal's entry written with a plain write: 15 of 19 crash tests red
control, by hand (r1)	the journal's flush hoisted out of its retry: exactly an_entry_written_again_after_a_refusal_follows_a_flush_of_its_own red

What the host tests establish:

• Protocol. The exhaustive model (every ordering of client, server, device, reset, server death and device error) gains forced writes and the law that one answered done stays on the medium until a later write replaces it.
• Journal. A commit asks the device for exactly Flush, Durable, and after data Write, Flush, Durable. The crash tests judge every state both while a forced entry is still torn and once it is on the medium.
• Drivers. NVMe's command word, WRITE(10)'s byte and the DPOFUA read are unit tests; a model stick that sets DPOFUA is sent FUA, and one that does not refuses it and is offered without.

Evidence: https://github.com/ToyOSOrg/evidence/blob/5d1ed034f23eab59d4e5513ffa9cc26b5092288d/README.md, and for the review's fixes r1/README.md at the same commit.

Metal: judged, and this PR is not ready

The A/B ran to its end: four boots of main with the row, this PR at 8a8ff27b9 and #1131 on it, per bench, interleaved, under rules fixed before any boot. Evidence: https://github.com/ToyOSOrg/evidence/blob/7efbe1295aa54370b213719b4d4f460e7a103554/ab/README.md

• AMD: a regression, rule F2 fails. Every durable row is slower, in every boot, with no overlap: persist.write_4k_fsync 2.36 ms (2.24..2.46) on main against 4.00 ms (4.00..4.01) here; append_4k_fsync +66%, create_fsync_unlink +71%, write_4k_fsync_beside_1m +55%, seq_write_fsync 266 against 212 MB/s. On this drive a forced write costs about 1.6 ms more than the write and Flush it replaced. Whether the 4.00 ms is the drive's or a cadence on the host is not established; no boot timed single commands.
• T14: no gain, and nothing resolved. The drive's slow mode decided every row. This PR drew it in 4 of 4 boots and main in 2 of 4. Slow against slow the two are equal. Forced writes in the fast mode, read off nvme: the controller's own idle power states, by APST #1131's boots, take 0.93..1.08 ms against main's fast boots at 0.97 and 1.12 ms: no faster than a write and a Flush.
• The prediction, one Flush fewer per durable operation, is true of the commands and false of the time.

So this PR should not land as it is. The options and a recommendation are with the orchestrator.

Risks

• No real stick has answered the MODE SENSE or taken a WRITE(10) with FUA. The benches boot off NVMe, so no bench row reaches usbd's disk path; issues/no-bench-row-drives-usb-storage-since-the-benches-boot-off-nvme.md records it. The guest row passed, but a green run keeps no serial log, so which of the two lines QEMU's stick produced is not known.
• The section numbers for MODE SENSE(6) and the mode parameter header are cited by clause title, not number: I could not check them against the standards' text from here. The NVMe and WRITE(10) citations follow the numbering the drivers already use.
• The T14's durable writes run slow in some boots (issues/toyos-durable-writes-on-the-t14s-nvme-run-six-times-slower-in-some-boots.md), which can confound an A/B there. The rules compare mode by mode.
• The superblock's write-then-flush is unchanged; only the journal uses the forced write.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1202 at b8e7fe921 (base f9b5d7f12)

Verdict: SEND BACK. Three blockers, none in the durability ordering itself, which I traced and found sound.

Read: all three commits, the full diff, the changed files and their callers, the handoff, and the evidence at 97b210c8. Nothing was built or run.

Architecture

The ownership is right. The device's driver knows whether it can force a write and says so in the open's answer. The server refuses what it did not offer. The session's client (Served) picks forced or write-then-flush. The cache gives a durable write the flush's place in line. The journal only calls BlockDev::write_durable. Nothing touches the kernel or toyos-abi, and there is no new dependency.

Not keeping an acknowledged forced write is the correct design, not a shortcut. I checked the alternative of keeping it like any write: after a loss, the reissued headless copy would sit over the forced header until its own reissue, so a crash in that window loses a committed entry.

Size: +1037/−225 over 31 files. About 415 lines are in test-only files and roughly 150 more in in-file test modules, leaving about +250 net production lines.

Durability: what I checked

• The flush before the entry is enough. The entry's bytes are fixed at close, the flush is asked after that and awaited before the forced write, so an entry on the medium implies everything it names is durable. The old first flush already had to guarantee this, because the old entry write could reach the medium at any time. Nothing reads the old trailing flush's side effect: journal.written, pending.durable and alloc.durable_through depend on the entry alone.
• Header last is forced headless, then the header block forced. The body is durable before the header sector exists, as before.
• No forced write goes over a kept write today. A refused forced write leaves its blocks dirty in the cache, and any cache flush then writes them plain, so the journal's blocks are in fact written another way. What saves it is that each retry attempt starts with a flush inside the reissue closure, and the cache's elision only skips the disk flush when flushed has caught up, which it does only on success. I walked header-last retries, multi-run entries, another task's dirty-limit flush between attempts, and a failed disk flush, and found no path that breaks it.
• Crash model and controls. Watched judges each forced write both torn and on the medium. The three controls are real: mutate-fua-goes-out-unforced fails its named test in host/ci-host.log, the in-test forces=false arm loses the committed create, and the hand patch reds 15 of 19 (exit 101 in its log).

BLOCKERS

1. lib/blockring/src/client.rs:28-32, 314-317, 339 — the forced-over-kept rule is enforced only by a test model. To the orchestrator's question: yes, the client should enforce it, at run time.

   • Consequence of a violation: after a loss the older kept bytes are reissued over the forced ones, and a later flush answers Durable over them. It is silent, and it happens on the reset path nobody watches.
   • Why the client: only it holds the kept set. The rule is prose in four places (lib/blockdev/src/lib.rs:76, the client header, Served::write_shared_durable, mount/mod.rs:1447) and a panic in the crash model. Root prefers a run-time check to a test, and internal invariants fail loudly.
   • Cheapest fix: when a fua write reaches the front of the outbox, no reissue is pending, so every kept write is in acked. Refuse loudly if one overlaps. Dropping superseded kept writes instead does not work, because the cache merges adjacent dirty blocks into one run and overlaps can be partial.
   • lib/bcachefs/src/engine/mount/mod.rs:1447-1448 gives an incomplete reason: "written no other way" is false below the mount, as above. The comment should say each attempt's flush covers what a failed attempt left.
   • Evidence gap: no test holds the flush inside the retry. Hoisting it out of the reissue closure keeps [Flush, Durable] and every crash test green. The client check closes this; otherwise a device double that refuses the first forced write and expects Flush, Durable, Flush, Durable does.

2. system/diskserver/src/session.rs:191-193 and system/diskserver/src/disk.rs:197-198 — both comments are false. They say a reconnect may answer fua differently. Session::reconnect (session.rs:385) refuses any open that differs from the first with Error::Protocol, and Opened's equality now includes fua, so fua never changes over a Served's life.

   • Consequence: a stick whose MODE SENSE is refused once (a unit attention, say) and answered after a usbd restart, or the reverse, now fails its reconnect and the volume is gone.
   • Fix: decide which is meant. Either compare blocks and unique only and adopt the new answer, with a test that flips it; or keep the refusal and delete both comments.
   • system/diskserver/src/disk/tests.rs:79-80: the Script arm that refuses an unoffered forced write is reached by no test.

3. issues/no-bench-row-drives-usb-storage-since-the-benches-boot-off-nvme.md:34-37, 46 — the new gap has no exit. The bullet sits under a lead-in about judgements the boot stick lost, which it is not, and the Exit names serving, a reset and a tone. Discharging the Exit as written closes the record with DPOFUA and the FUA write still unread. The Exit should name them, and the record should say what a stick that sets DPOFUA and then refuses or ignores FUA does: the journal stops writing, or durability is silently lost.

SUGGESTIONS

• lib/scsi/src/lib.rs:108: dpofua believes byte 2 of any four delivered bytes. The module header says everything a device answers is checked. Refusing a header whose MODE DATA LENGTH is under 3 costs one comparison. The bounds are otherwise fine: delivered is the smaller of the controller's count and the CSW's, the buffer is a page, and stale bytes are never read as a header.
• lib/scsi/src/lib.rs:67-70, 100: "so a device with pages to send has room" is not the constraint, since only the header is read. Why page 3Fh and 192 bytes rather than the header alone should be stated, and its provenance kept in the evidence. Orchestrator: please confirm where that pair came from.
• tests/toyos.rs:3323: print the stick's bring-up line after the marker, as the designation row prints its line. Every run of the row then records which path QEMU's stick took.
• lib/blockdev/src/lib.rs:75-76: "nothing else made durable by it" reads as a guarantee the default (write, then flush) breaks. "Nothing else is promised durable" is the contract.
• PR description, for the orchestrator: "the fallback is write_durable's default" and "in one place" are loose. A served disk without FUA takes the explicit branch in Served::write_shared_durable; the trait default serves RAM, model and wrapper devices. It is one rule at two sites.

Spec conformance

The bit positions in code are right as I know the standards. I did not re-read the standards' text in this session, and neither did the implementer.

Item	Code	Verdict
NVMe Write FUA	Command Dword 12 bit 30 (1 << 30), count zero-based in bits 15:0	correct; no capability bit gates it, so offering it always is right
WRITE(10) FUA	byte 1 bit 3 (0x08)	correct
MODE SENSE(6) CDB	1A 00 3F 00 C0 00: current values, all pages, 192 allocated	correct
DPOFUA	bit 4 of byte 2 of the 6-byte mode parameter header	correct

The two citations by clause title are honest and findable, which is better than an unchecked number. The numbers should be filled in by someone holding SPC-4 and SBC-3. SBC-3 §5.32 reuses the file's existing citation, which I could not check either.

Protocol and fallback

• Validation: Request::decode takes the flag only on a write and refuses any other bit and any flag on a read or flush. take refuses an unoffered forced write unissued, after the read-only check. Opened::decode refuses a flag byte that is neither 0 nor 1.
• Wire compatibility: Opened grows to 25 bytes with an exact-length decode, and a request's sixth word goes from reserved to flags. Every consumer is in this workspace and lands together; nothing under rust/library uses it.
• Fallback: explicit and logged by both drivers. BlockDev wrappers that do not forward write_durable (the installer's) take write-then-flush, which is correct.

osbench_disk

Fine to carry. It is named_only, so no unfiltered or nightly pass takes it, and the word osbench does not select it. It reuses benchcase.toml, whose three jobs exist, and the suite's judge and bound, with the bound's reuse stated. Its first run is the staged A/B.

Evidence

• Host: host/ci-host.log at 8a8ff27b9 shows every new test ok and the single llvm red. I accept that red as independent, since it is recorded and no file here is near it.
• Guest: the log shows 8a8ff27b9 checked out and three PASS. Only nvme_disk_boots reaches a forced write, on QEMU's NVMe. The stick rows show MODE SENSE did not break QEMU's stick and nothing more, as the PR says.
• Head: b8e7fe921 changes one module comment, so the results carry. The "doc tests ran again at the head" claim has no log, and needs none.

What the USB path still needs

No real stick has seen either new command. In order of cost:

1. Which line QEMU's stick produced: the print suggested above, then one dispatch of the_log_lands_on_the_boot_stick.
2. MODE SENSE on real sticks: one bench boot of any image with a stick plugged into a port that is not the boot disk, repeated for each stick at hand. The readback must show serving it as disk N: ... with either ending, and no a Bulk-Only round trip broke or the ladder climbs before it. This needs no new row and can ride the staged A/B boots. A stick that hangs on this command is never served, because the command runs before the disk arrives on every bring-up. I would not land before this.
3. WRITE(10) with FUA accepted: this needs a device that sets DPOFUA (bridges usually do, plain sticks rarely) carrying a bcachefs volume. No row serves a non-boot stick's volume, so it stays owed under blocker 3's exit.
4. FUA actually honoured: only a power cut after an acknowledged commit shows it. That is evidence to record if ever taken, not a gate.

[Japabu]

Closing by the orchestrator's decision: measured and declined. The journal keeps write-plus-Flush, and the forced-write plumbing, which had no other user, does not land. The branch is deleted; its last head was fea712a16, and the measured image was 8a8ff27b9.

Result (osbench_disk, four boots an arm on each bench, interleaved, rules fixed before any boot; median of boots' p50 with the range over boots). Evidence: https://github.com/ToyOSOrg/evidence/blob/7efbe1295aa54370b213719b4d4f460e7a103554/ab/README.md

AMD, every boot of each arm agreeing, no overlap:

row	main (c9d6eff6f)	forced (8a8ff27b9)	Linux
persist.write_4k_fsync	2.36 ms (2.24-2.46)	4.00 ms (4.00-4.01)	3.58 ms
persist.append_4k_fsync	2.41 ms (2.32-2.51)	4.00 ms (4.00-4.00)	3.84 ms
persist.create_fsync_unlink	2.62 ms (2.54-2.93)	4.47 ms (4.25-4.64)	3.32 ms
persist.write_4k_fsync_beside_1m	3.03 ms (2.95-3.17)	4.69 ms (4.63-4.83)	4.18 ms
persist.seq_write_fsync	266 MB/s (243-274)	212 MB/s (204-223)	502 MB/s

T14: the drive's slow mode decided every row. Slow against slow the arms were equal (5.66-6.24 ms against 5.78 and 5.92 ms); the forced arm had no fast boot. Forced writes in the fast mode, read off #1131's boots: append_4k_fsync 0.93-1.08 ms, against 0.97 and 1.12 ms for a write and a Flush.

No faster at best, 66% slower at worst.

Not established: whether the AMD's 4.00 ms (one boot: min 3.71, p50 4.01, p90 4.25) is the drive's cost of a forced write or a cadence on ToyOS's side. No boot timed single commands.

What survives is in #1215: the osbench_disk row, and a record of this result so it is not retried blind. Review r1's durability trace and its three blockers were addressed at fea712a16 and go with the branch.