osbench: persist and the disk's cached rows run on ToyOS's DATA, the bench's NVMe

issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md

@@ -42,6 +42,9 @@ boots' 13 s, so the same write stalls on metal even where it completes.
 The server also refuses nothing on the way: every request answered until the
 allocator aborted the process, and the client saw `the file server is gone`.
 
+The same write is now `osbench persist`'s `fs.disk.seq_write.*`, which
+reaches the in-memory volume on a boot whose DATA is not designated.
+
 **Exit**: the server's memory for a 64 MiB file written to the in-memory volume
 is bounded by the file's size plus a stated overhead, shown by the same write
 under QEMU with its peak in the kernel's exit record; or the server refuses

issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md

@@ -0,0 +1,31 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# A clean fsync on DATA writes the superblock and flushes the disk
+
+An fsync of a file on DATA syncs the whole volume
+(`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`), and the
+volume's sync, `bcachefs::Mounted::sync` (`lib/bcachefs/src/fs.rs`), commits
+unconditionally. It flushes the cache, writes the superblock, and flushes
+again. The file server's cache issues a device flush only where it owes one,
+after a write it made. The superblock write makes one owed every time, so an
+fsync with nothing new to make durable still costs a block write and an NVMe
+Flush.
+
+Measured on the AMD bench (2026-10-28, `osbench persist` at PR #1097's
+c8f3fbfb0, DATA on its designated NVMe partition): `persist.fsync_clean`
+p50 1,233,821 ns under ToyOS, against 56,649 and 69,703 ns under the
+bench's Linux on the same NVMe. That is about the cost of one Flush on that
+disk, and `persist.write_4k_fsync`, which owes one, reads 2.9 ms on ToyOS
+and 3.0 to 3.6 ms on Linux.
+
+**Exit**: a sync of a volume with nothing uncommitted writes and flushes
+nothing, shown by counting the requests a clean sync issues on a host disk.
+`persist.fsync_clean` on the AMD bench then reads within the file server's
+round trip, tens of µs and not a Flush's ms.
+
+**Owner**: the DATA format (`lib/bcachefs/src/fs.rs`) and the file server's
+DATA volume (`system/fileserver/src/data.rs`).

issues/a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.md

@@ -0,0 +1,30 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# A created, fsynced and unlinked file costs ToyOS more than Linux
+
+`persist.create_fsync_unlink` creates a file on the persistent filesystem,
+writes 4 KiB to it, `sync_all`s it, closes and removes it. At PR #1097's
+landing head (069cbbeb8, 2026-10-28, one ToyOS boot and two Linux runs per
+bench, DATA and Linux's home on the same NVMe) it read:
+
+| bench | ToyOS | Linux |
+|---|---|---|
+| T14 | 1,168,401 ns | 953,976 and 966,415 ns |
+| AMD | 3,533,628 ns | 3,257,728 and 3,283,416 ns |
+
+That is 8 to 22% dearer, where the plain durable writes
+(`write_4k_fsync`, `append_4k_fsync`) are at or near Linux's on both
+benches. What the difference is made of is unmeasured. The create and the
+unlink are file server requests, and the fsync commits the whole volume
+with its superblock.
+
+**Exit**: the row within Linux's on both benches, or its difference
+measured to its parts and justified, in a labelled current-state
+comparison.
+
+**Owner**: the file server's DATA volume (`system/fileserver/src/data.rs`)
+and the DATA format (`lib/bcachefs/src/fs.rs`).

issues/a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call.md

@@ -0,0 +1,36 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# A file server request costs three to twelve times a kernel file call
+
+Every file operation on DATA is a request to the DATA file server process.
+On the AMD bench (2026-10-28, `osbench` at PR #1097's c8f3fbfb0, DATA on its
+NVMe), that boot's cached rows on DATA against its kernel `/tmp`, and the
+bench's Linux page cache on the same NVMe:
+
+| row, p50 | ToyOS DATA | ToyOS `/tmp` | DATA / `/tmp` | Linux home |
+|---|---|---|---|---|
+| `read_4k_at` (seek and 4 KiB read) | 16,903 ns | 1,413 ns | 12.0x | 1,463–2,375 ns |
+| `stat` | 10,530 ns | 2,596 ns | 4.1x | 1,453–2,794 ns |
+| `open_close` | 18,686 ns | 6,091 ns | 3.1x | 3,056–4,400 ns |
+| `rename` | 59,154 ns | 6,242 ns | 9.5x | 9,308–10,895 ns |
+| `seq_read.1048576` | 6,401 MB/s | 8,041 MB/s | | 11,295–11,403 MB/s |
+
+Four of these rows the disk does not set. On 2026-10-24's in-memory DATA the
+same rows read within about 5%: `read_4k_at` 16,822, `stat` 10,450 and
+`rename` 58,002 ns, and `seq_read.1048576` 6,080 MB/s. `open_close` read
+14,178 ns there, so it is 32% dearer on the NVMe, and what that costs is
+unattributed. Both tables are in PR #1097's evidence. What the round trip
+spends its time on, on metal, is unmeasured.
+`issues/a-cached-read-from-a-file-server-is-measured-only-under-tcg.md`
+splits it under TCG.
+
+**Exit**: the cached `fs.disk.*` rows within Linux's on both benches, or
+each row's remaining difference measured to its parts and justified, in a
+labelled current-state comparison.
+
+**Owner**: the file server (`system/fileserver/`) and its transport
+(`sdk/toyos/src/fs.rs`).

issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md

@@ -0,0 +1,33 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# A sequential read of a file as large as DATA's clean cache reads the disk
+
+The file server's cache (`system/fileserver/src/cache.rs`) keeps
+`CLEAN_LIMIT` clean blocks, 16,384 of 4 KiB (64 MiB), and evicts them in
+the order they arrived. A hit does not move a block. So once a file and the
+volume's metadata together pass that limit, a sequential read evicts, at
+each miss, the block it is about to read next, and every pass misses every
+block.
+
+Measured on the AMD bench (2026-10-28, `osbench persist` at PR #1097's
+c8f3fbfb0, DATA on its NVMe): `fs.disk.seq_read.4096`, a 64 MiB file read in
+4 KiB calls, p50 158 MB/s, min 158 and max 426, against 4,960 and 5,054 MB/s
+under the bench's Linux. Four of the five rounds ran at about a third of the
+fifth.
+
+The cause, shown on the host by a probe of the cache over a counting disk
+(PR #1097's evidence, `probe.log`): blocks written in 4 KiB calls, flushed,
+then read in order five times. With 16 blocks over the limit (16,400), each
+pass made 16,400 disk requests. With 16 blocks under it (16,368), each pass
+made none.
+
+**Exit**: a sequential read of a file larger than the clean limit keeps
+hitting what the cache can hold, rather than missing everything. A host test
+of the cache shows it, by counting the disk reads of repeated passes over
+`CLEAN_LIMIT + 16` blocks.
+
+**Owner**: the file server's cache (`system/fileserver/src/cache.rs`).

issues/no-published-run-compares-toyos-with-linux-on-one-machine.md

@@ -36,10 +36,10 @@ its groups:
 
 `tests/guest/osbench/README.md` is how a comparison is run: both halves from
 one revision, interleaved runs, Linux as installed for the baseline, and
-Linux under ToyOS's frequency request as a labelled experiment. `persist` is
-not yet one disk: ToyOS writes the boot stick's LOG volume, and moves to the
-bench's NVMe once ToyOS designates its DATA partition itself. The README
-says why the AMD bench's Linux keeps time on the HPET.
+Linux under ToyOS's frequency request as a labelled experiment. `persist`
+runs on the bench's NVMe under both: the `osbench` row refuses a boot whose
+DATA is not the NVMe's TOYOS-DATA partition. The README says why the AMD
+bench's Linux keeps time on the HPET.
 
 `internet.*`, HTTPS to Cloudflare's speed test, is exploratory evidence of a
 path neither system owns, run only when a pass names its row: no automated

issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md

@@ -0,0 +1,32 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# ToyOS's durable writes on the T14's NVMe took 1 to 6 ms across two boots
+
+On the T14 (SK hynix BC501, volatile write cache present, so ToyOS's NVMe
+driver issues Flush), two boots of PR #1097, both off the internal NVMe with
+DATA on it, read these p50s. Linux is two runs on ext4 on the same NVMe in
+each validation:
+
+| row | ToyOS, c8f3fbfb0 | ToyOS, 069cbbeb8 | Linux (both validations) |
+|---|---|---|---|
+| `persist.fsync_clean` | 3,019,602 ns | 464,505 ns | 22,970 to 23,012 ns |
+| `persist.write_4k_fsync` | 6,125,947 ns | 1,099,737 ns | 811,313 to 1,035,997 ns |
+| `persist.append_4k_fsync` | 6,182,133 ns | 1,012,768 ns | 796,449 to 1,200,332 ns |
+| `persist.seq_write_fsync` | 162 MB/s | 347 MB/s | 527 to 598 MB/s |
+
+No change to the NVMe driver, the block service, the file server or the
+DATA format lies between the two heads; the second rebased onto #1095's
+boot path, and both booted off the NVMe. Why one boot's durable writes cost
+six times the other's is unknown. The clean fsync's remaining 20x, a
+superblock write and a Flush where Linux's clean fsync takes 23 µs, is
+`issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md`.
+
+**Exit**: repeated boots on the T14 read ToyOS's `persist.write_4k_fsync`
+within Linux's, or the boots that do not are explained by a measured cause.
+
+**Owner**: the NVMe driver (`system/nvme/`) and the DATA volume's commit
+(`lib/bcachefs/src/fs.rs`).

tests/guest/osbench/README.md

@@ -134,14 +134,49 @@ CPU's TSC read behind another's and no step exceeded the 1.2 µs of a read.
 A Linux forced onto the TSC (`tsc=reliable`) has those checks switched off,
 so it is no baseline.
 
-## The disk is not yet one disk
-
-`persist` writes each system's persistent filesystem. On Linux that is the
-runner's home, ext4 on the bench's NVMe. On ToyOS it is still the boot
-stick's LOG volume, FAT32 over USB 2.0, because ToyOS's DATA is the bench's
-partition only once ToyOS has designated it, and ToyOS cannot yet do that
-itself. So `persist` compares two media. Its lines name theirs, and
-`osbench-compare` refuses a verdict on any row whose media differ.
+## Both write the bench's NVMe
+
+`persist` and `fs.disk` run on each system's persistent filesystem on the
+bench's NVMe. On Linux that is the runner's home, ext4 on the bench's root
+partition. On ToyOS it is DATA: the bench's TOYOS-DATA partition, its fourth,
+named `ToyOS data`, which ToyOS designates itself. It is served through the
+DATA file server and the NVMe driver. ToyOS boots off the same NVMe, and its
+log volume is there too, so the log's writes share the device with
+`persist` (`toyos-metal`'s internal install). The `osbench` row refuses a
+boot whose DATA was anything else, and refuses the boot that formats a
+newly designated DATA as a run, saying how long the format took.
+
+What makes the disk rows one comparison, and what stays each system's own:
+
+- **One disk.** Both write the bench's one NVMe namespace: Linux its root
+  partition, ToyOS its fourth. They are different regions of one device,
+  through each system's own driver. Each row's `medium` names it, and the
+  compare refuses a verdict where the two systems' media differ. On Linux
+  the medium is read off sysfs. On ToyOS it is a constant, `nvme`, since a
+  program cannot ask what holds `/home`: only the `osbench` row's refusal
+  makes it true.
+- **Each system's own filesystem.** ext4, and ToyOS's DATA format behind
+  its file server, are part of what is compared. Linux's mount and
+  superblock options are in `fs.disk.mount`. The free space they write into
+  differs: Linux's root holds its installation, and ToyOS's DATA is a volume
+  of its own.
+- **One durability.** A `persist` row's write counts once `sync_all` or
+  `sync_data` returns. Both systems answer that after the device's own
+  cache has been flushed. On Linux, ext4 does it by its default barriers: a
+  `nobarrier` would show in `fs.disk.mount`'s superblock options, and the
+  device's write-cache mode is in `served_by` (`write cache write back`).
+  On ToyOS, the NVMe driver issues Flush where the controller has a volatile
+  write cache, which its boot log says. ToyOS's fsync makes its whole volume
+  durable, not just the file
+  (`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`).
+- **One procedure for the cache.** Every row reads only files it wrote in its
+  own run, and neither system's cache is dropped before it. What then
+  reaches the disk is each system's own write-back and cache policy, and
+  part of what is compared. On ToyOS the DATA cache writes back once 32 MiB
+  is dirty and syncs the volume 2 s after an unsynced write, so the 64 MiB
+  `fs.disk.seq_write.*` rounds write the disk. Its cache keeps 64 MiB clean,
+  evicting in insertion order, and the 4 KiB sequential read misses
+  (`issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md`).
 
 ## What is not a comparison
 

tests/guest/osbench/compare/src/lib.rs

@@ -291,7 +291,7 @@ pub const SETUP: [&str; 13] = [
     "platform_profile",
     "smt_active",
     "fs.mem.served_by",
-    "fs.served.served_by",
+    "fs.disk.served_by",
     "persist.served_by",
 ];
 

tests/guest/osbench/src/files.rs

@@ -1,12 +1,13 @@
 //! Files, through std's `fs` on both systems: the same operations under a
-//! filesystem each keeps in memory (`mem`), and on ToyOS under `/home`
-//! (`served`). Persistent storage is `storage`'s.
+//! filesystem each keeps in memory (`mem`) and under the one on the bench's
+//! disk (`disk`, which `storage` runs beside its durable writes). Nothing here
+//! asks for durability: each system's cache and write-back decide what reaches
+//! the disk, and that is part of what the rows measure.
 //!
 //! `mem` is Linux's `/dev/shm` through the VFS, and ToyOS's `/tmp`, which the
-//! kernel serves through syscalls. `served` is ToyOS's DATA file server,
-//! through IPC per request: on the volume the bench's DATA partition holds
-//! where it holds one of ours, and in memory where it does not (the file
-//! server says which in the log).
+//! kernel serves through syscalls. `disk` is each system's
+//! [`crate::sys::persistent_dir`]: Linux's VFS and page cache over the
+//! runner's home, and ToyOS's DATA file server, through IPC per request.
 
 use std::fs::{self, File, OpenOptions};
 use std::io::{Read, Seek, SeekFrom, Write};
@@ -44,17 +45,6 @@ pub fn memory() -> Target {
     Target { name: "mem", dir: dir.into(), served_by: served_by.into(), medium: Some("memory".into()) }
 }
 
-/// Every directory `files` measures under.
-pub fn targets() -> Vec<Target> {
-    let mut targets = vec![memory()];
-    if cfg!(target_os = "toyos") {
-        // Memory or the DATA partition's volume: the program cannot tell which.
-        let served_by = "the DATA file server, through IPC".into();
-        targets.push(Target { name: "served", dir: "/home".into(), served_by, medium: None });
-    }
-    targets
-}
-
 /// The persistent filesystem's directory, or `None` where the system has none.
 pub fn disk() -> Option<Target> {
     crate::sys::persistent_dir().map(|(dir, served_by, medium)| Target { name: "disk", dir, served_by, medium: Some(medium) })

tests/guest/osbench/src/main.rs

@@ -126,9 +126,7 @@ fn fair() {
 }
 
 fn files() {
-    for target in files::targets() {
-        files::suite(&target);
-    }
+    files::suite(&files::memory());
 }
 
 /// Every group the machine alone runs, in the order `all` and the ToyOS job

tests/guest/osbench/src/storage.rs

@@ -1,14 +1,15 @@
-//! Persistent storage, each system on the medium a measured boot has, so
-//! **not like for like** until ToyOS's DATA is the bench's partition: what
-//! these say is what a durable write costs there, each line naming it.
+//! The bench's disk, through each system's persistent filesystem
+//! ([`crate::files::disk`]): durable writes first, each made durable by
+//! `sync_all` or `sync_data` before the next, then `files`' cached operations
+//! under the same directory as `fs.disk.*`. Every line names what served it
+//! and its medium.
 //!
-//! On a ToyOS benchcase boot that is the boot USB stick's `TOYOS-LOG`
-//! partition, FAT32, through the LOG file server and usbd: the stick's other
-//! volumes are read-only, and DATA is the bench's partition only once ToyOS
-//! has designated it. On Linux it is a directory in the runner's home, on
-//! whatever disk and filesystem hold it. Every file here is removed at the
-//! end; the most alive at once is under 10 MiB, inside what the log volume
-//! leaves free.
+//! On Linux that is the runner's home, on whatever filesystem and disk hold
+//! it. On ToyOS it is `/home`, on the DATA volume: the bcachefs crate's
+//! format, through the DATA file server, the block service and the disk's
+//! driver. A ToyOS boot whose DATA is in memory measures memory here, which
+//! the `osbench` metal row refuses (`tests/toyos.rs`). Every file here is
+//! removed at the end.
 
 use std::fs::{self, File, OpenOptions};
 use std::io::{Read, Seek, SeekFrom, Write};
@@ -23,6 +24,11 @@ pub fn suite() {
     let Some(disk) = files::disk() else {
         return refuse("persist", "no persistent directory to write in");
     };
+    durable(&disk);
+    files::suite(&disk);
+}
+
+fn durable(disk: &files::Target) {
     let scratch = match Scratch::new(&disk.dir) {
         Ok(s) => s,
         Err(e) => return refuse("persist", format!("no scratch directory: {e}")),