From 950b18afb1bd2764af0c064ba29bebfe9777ad95 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 16:21:41 +0100 Subject: [PATCH 1/7] osbench: persist and the disk's cached rows run on ToyOS's DATA, the bench's NVMe #1085 lets ToyOS designate its own DATA, and both benches' fourth partitions are designated and mounted. So the follow-up staged in #1079 lands. ToyOS's persistent filesystem is DATA's /home (bcachefs crate's format, through the DATA file server and the NVMe driver), medium nvme. persist runs its durable rows there, then files' cached operations as fs.disk.*, on both systems: Linux on its home, ext4 on the same NVMe. A row whose media differ still has no verdict, and these now agree. The osbench row refuses a boot whose DATA was not a TOYOS-DATA partition the NVMe driver serves, mounted and not absent. It also refuses the boot that formats a newly designated DATA as a run, stating the format's duration. ToyOS's files group keeps /tmp alone: /home is fs.disk now. The compare's setup reads fs.disk.served_by. The README states the disk rows' rules: - one NVMe for both; - each system's own filesystem, Linux's mount options recorded; - one durability: a sync returns after the device's cache is flushed, ToyOS's fsync covering its whole volume; - one cache state: every row's files are written in its run and read cached on both. #1085 already judges nvme_leaves_the_benchs_disk_alone against a designated DATA, so that part of the staged patch is superseded and dropped. Nothing here depends on #1095 (ToyOS booting off the internal NVMe); the judge finds DATA by the NVMe driver's partition line, wherever the boot came from. Co-Authored-By: Claude Opus 5.5 --- ...lume-runs-its-file-server-out-of-memory.md | 3 + ...ompares-toyos-with-linux-on-one-machine.md | 8 +-- tests/guest/osbench/README.md | 39 ++++++++++--- tests/guest/osbench/compare/src/lib.rs | 2 +- tests/guest/osbench/src/files.rs | 26 +++------ tests/guest/osbench/src/main.rs | 4 +- tests/guest/osbench/src/storage.rs | 26 +++++---- tests/guest/osbench/src/sys/toyos.rs | 10 ++-- tests/images/benchcase.toml | 4 +- tests/toyos.rs | 56 ++++++++++++++++++- 10 files changed, 125 insertions(+), 53 deletions(-) diff --git a/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md b/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md index d7cc2276bf4..73293ed0c9b 100644 --- a/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md +++ b/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md @@ -42,6 +42,9 @@ boots' 13 s, so the same write stalls on metal even where it completes. The server also refuses nothing on the way: every request answered until the allocator aborted the process, and the client saw `the file server is gone`. +The same write is now `osbench persist`'s `fs.disk.seq_write.*`, which +reaches the in-memory volume on a boot whose DATA is not designated. + **Exit**: the server's memory for a 64 MiB file written to the in-memory volume is bounded by the file's size plus a stated overhead, shown by the same write under QEMU with its peak in the kernel's exit record; or the server refuses diff --git a/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md b/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md index 4ce5bfc1134..723a6c4f1cd 100644 --- a/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md +++ b/issues/no-published-run-compares-toyos-with-linux-on-one-machine.md @@ -36,10 +36,10 @@ its groups: `tests/guest/osbench/README.md` is how a comparison is run: both halves from one revision, interleaved runs, Linux as installed for the baseline, and -Linux under ToyOS's frequency request as a labelled experiment. `persist` is -not yet one disk: ToyOS writes the boot stick's LOG volume, and moves to the -bench's NVMe once ToyOS designates its DATA partition itself. The README -says why the AMD bench's Linux keeps time on the HPET. +Linux under ToyOS's frequency request as a labelled experiment. `persist` +runs on the bench's NVMe under both: the `osbench` row refuses a boot whose +DATA is not the NVMe's TOYOS-DATA partition. The README says why the AMD +bench's Linux keeps time on the HPET. `internet.*`, HTTPS to Cloudflare's speed test, is exploratory evidence of a path neither system owns, run only when a pass names its row: no automated diff --git a/tests/guest/osbench/README.md b/tests/guest/osbench/README.md index a08e43896d3..8d0626f8157 100644 --- a/tests/guest/osbench/README.md +++ b/tests/guest/osbench/README.md @@ -134,14 +134,37 @@ CPU's TSC read behind another's and no step exceeded the 1.2 µs of a read. A Linux forced onto the TSC (`tsc=reliable`) has those checks switched off, so it is no baseline. -## The disk is not yet one disk - -`persist` writes each system's persistent filesystem. On Linux that is the -runner's home, ext4 on the bench's NVMe. On ToyOS it is still the boot -stick's LOG volume, FAT32 over USB 2.0, because ToyOS's DATA is the bench's -partition only once ToyOS has designated it, and ToyOS cannot yet do that -itself. So `persist` compares two media. Its lines name theirs, and -`osbench-compare` refuses a verdict on any row whose media differ. +## Both write the bench's NVMe + +`persist` and `fs.disk` run on each system's persistent filesystem on the +bench's NVMe. On Linux that is the runner's home, ext4 on the bench's root +partition. On ToyOS it is DATA: the bench's TOYOS-DATA partition, its fourth, +named `ToyOS data`, which ToyOS designates itself. It is served through the +DATA file server and the NVMe driver. The `osbench` row refuses a boot whose +DATA was anything else, and refuses the boot that formats a newly +designated DATA as a run, saying how long the format took. Every row's +`served_by` and `medium` name the filesystem and the disk it measured. + +What makes the disk rows one comparison, and what stays each system's own: + +- **One disk.** Both write the bench's one NVMe namespace: Linux its root + partition, ToyOS its fourth. They are different regions of one device, + through each system's own driver. Each row's `medium` is `nvme` on both, + and the compare refuses a verdict on any row where it is not. +- **Each system's own filesystem.** ext4, and ToyOS's DATA format behind + its file server, are part of what is compared. Linux's mount options are + in `fs.disk.mount`. The free space they write into differs: Linux's root + holds its installation, and ToyOS's DATA is a volume of its own. +- **One durability.** A `persist` row's write counts once `sync_all` or + `sync_data` returns. Both systems answer that after the device's own + cache has been flushed: Linux's ext4 by its default barriers, which a + `nobarrier` in `fs.disk.mount` would show, and ToyOS's NVMe driver by + issuing Flush where the controller has a volatile write cache, which its + boot log says. ToyOS's fsync makes its whole volume durable, not just the + file (`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`). +- **One cache state.** Every row writes its own files in its run, and its + reads find them in each system's cache. `persist.seq_read_cached` and + `fs.disk.*` are cached reads on both, and no row reads the disk cold. ## What is not a comparison diff --git a/tests/guest/osbench/compare/src/lib.rs b/tests/guest/osbench/compare/src/lib.rs index feea0194dac..4ed7516f55f 100644 --- a/tests/guest/osbench/compare/src/lib.rs +++ b/tests/guest/osbench/compare/src/lib.rs @@ -291,7 +291,7 @@ pub const SETUP: [&str; 13] = [ "platform_profile", "smt_active", "fs.mem.served_by", - "fs.served.served_by", + "fs.disk.served_by", "persist.served_by", ]; diff --git a/tests/guest/osbench/src/files.rs b/tests/guest/osbench/src/files.rs index 401a9c57066..1a018901c2b 100644 --- a/tests/guest/osbench/src/files.rs +++ b/tests/guest/osbench/src/files.rs @@ -1,12 +1,12 @@ //! Files, through std's `fs` on both systems: the same operations under a -//! filesystem each keeps in memory (`mem`), and on ToyOS under `/home` -//! (`served`). Persistent storage is `storage`'s. +//! filesystem each keeps in memory (`mem`) and under the one on the bench's +//! disk (`disk`, which `storage` runs beside its durable writes). Nothing here +//! asks for durability, so on the disk these are the cached path. //! //! `mem` is Linux's `/dev/shm` through the VFS, and ToyOS's `/tmp`, which the -//! kernel serves through syscalls. `served` is ToyOS's DATA file server, -//! through IPC per request: on the volume the bench's DATA partition holds -//! where it holds one of ours, and in memory where it does not (the file -//! server says which in the log). +//! kernel serves through syscalls. `disk` is each system's +//! [`crate::sys::persistent_dir`]: Linux's VFS and page cache over the +//! runner's home, and ToyOS's DATA file server, through IPC per request. use std::fs::{self, File, OpenOptions}; use std::io::{Read, Seek, SeekFrom, Write}; @@ -44,17 +44,6 @@ pub fn memory() -> Target { Target { name: "mem", dir: dir.into(), served_by: served_by.into(), medium: Some("memory".into()) } } -/// Every directory `files` measures under. -pub fn targets() -> Vec { - let mut targets = vec![memory()]; - if cfg!(target_os = "toyos") { - // Memory or the DATA partition's volume: the program cannot tell which. - let served_by = "the DATA file server, through IPC".into(); - targets.push(Target { name: "served", dir: "/home".into(), served_by, medium: None }); - } - targets -} - /// The persistent filesystem's directory, or `None` where the system has none. pub fn disk() -> Option { crate::sys::persistent_dir().map(|(dir, served_by, medium)| Target { name: "disk", dir, served_by, medium: Some(medium) }) @@ -103,7 +92,8 @@ fn page_at(i: usize, pages: usize) -> u64 { } /// Every benchmark under `target`, each line naming what served it and its -/// medium. Each one's comment is its warm-up and repetitions. +/// medium. Each +/// one's comment is its warm-up and repetitions. pub fn suite(target: &Target) { condition(&format!("fs.{}.served_by", target.name), &target.served_by); #[cfg(target_os = "linux")] diff --git a/tests/guest/osbench/src/main.rs b/tests/guest/osbench/src/main.rs index c95ac65b47e..9e88b2f3654 100644 --- a/tests/guest/osbench/src/main.rs +++ b/tests/guest/osbench/src/main.rs @@ -126,9 +126,7 @@ fn fair() { } fn files() { - for target in files::targets() { - files::suite(&target); - } + files::suite(&files::memory()); } /// Every group the machine alone runs, in the order `all` and the ToyOS job diff --git a/tests/guest/osbench/src/storage.rs b/tests/guest/osbench/src/storage.rs index af1d628a39e..d5b7d371c27 100644 --- a/tests/guest/osbench/src/storage.rs +++ b/tests/guest/osbench/src/storage.rs @@ -1,14 +1,15 @@ -//! Persistent storage, each system on the medium a measured boot has, so -//! **not like for like** until ToyOS's DATA is the bench's partition: what -//! these say is what a durable write costs there, each line naming it. +//! The bench's disk, through each system's persistent filesystem +//! ([`crate::files::disk`]): durable writes first, each made durable by +//! `sync_all` or `sync_data` before the next, then `files`' cached operations +//! under the same directory as `fs.disk.*`. Every line names what served it +//! and its medium. //! -//! On a ToyOS benchcase boot that is the boot USB stick's `TOYOS-LOG` -//! partition, FAT32, through the LOG file server and usbd: the stick's other -//! volumes are read-only, and DATA is the bench's partition only once ToyOS -//! has designated it. On Linux it is a directory in the runner's home, on -//! whatever disk and filesystem hold it. Every file here is removed at the -//! end; the most alive at once is under 10 MiB, inside what the log volume -//! leaves free. +//! On Linux that is the runner's home, on whatever filesystem and disk hold +//! it. On ToyOS it is `/home`, on the DATA volume: the bcachefs crate's +//! format, through the DATA file server, the block service and the disk's +//! driver. A ToyOS boot whose DATA is in memory measures memory here, which +//! the `osbench` metal row refuses (`tests/toyos.rs`). Every file here is +//! removed at the end. use std::fs::{self, File, OpenOptions}; use std::io::{Read, Seek, SeekFrom, Write}; @@ -23,6 +24,11 @@ pub fn suite() { let Some(disk) = files::disk() else { return refuse("persist", "no persistent directory to write in"); }; + durable(&disk); + files::suite(&disk); +} + +fn durable(disk: &files::Target) { let scratch = match Scratch::new(&disk.dir) { Ok(s) => s, Err(e) => return refuse("persist", format!("no scratch directory: {e}")), diff --git a/tests/guest/osbench/src/sys/toyos.rs b/tests/guest/osbench/src/sys/toyos.rs index 5cb088aef03..fb2e0123842 100644 --- a/tests/guest/osbench/src/sys/toyos.rs +++ b/tests/guest/osbench/src/sys/toyos.rs @@ -388,11 +388,13 @@ pub fn map_anon_4k(len: usize) -> *mut u8 { map_anon(len) } -/// The one writable volume a benchcase boot certainly has on a disk, what -/// serves it, and its medium. +/// DATA's `/home`, what serves it, and its medium. A program cannot ask +/// which disk DATA is on, or whether it is memory instead; the `osbench` row +/// refuses a boot whose DATA was not the NVMe's partition, so that is the +/// medium a measured run's lines name. pub fn persistent_dir() -> Option<(String, String, String)> { - let served_by = "boot USB stick TOYOS-LOG partition, FAT32, LOG file server over usbd mass storage"; - Some(("/log".into(), served_by.into(), "usb".into())) + let served_by = "DATA volume, bcachefs crate's format, DATA file server over the block service"; + Some(("/home".into(), served_by.into(), "nvme".into())) } /// What a transfer row reports of the stack beneath it, as netstack's `inspect` answers it: diff --git a/tests/images/benchcase.toml b/tests/images/benchcase.toml index 5eea324c793..4cd1394e7d0 100644 --- a/tests/images/benchcase.toml +++ b/tests/images/benchcase.toml @@ -5,8 +5,8 @@ # ACPI server, and netstack for `osbench lan` and `osbench internet`, each its # own metal row (`tests/toyos.rs`): no compositor or sound, so little but the # measurement asks for a CPU. The stick has no DATA partition, so DATA is the -# bench's TOYOS-DATA partition where ToyOS has designated it, and memory where -# it has not; the log volume is what `osbench persist` writes. +# bench's: the TOYOS-DATA partition on its NVMe, which `osbench persist` +# writes, and which the `osbench` row refuses a boot to be without. # # The runner's capability carries `jobs`, for the fairness run that starts each # workload as a job of its own; `counters`, `trace` and `roster`, so `osbench diff --git a/tests/toyos.rs b/tests/toyos.rs index 09fb62989e8..c228add7987 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -1145,10 +1145,19 @@ const METAL: &[(&str, metal::Metal)] = &[ ( // The numbers compared with the same program on the bench's own Linux. // Evidence and not a verdict: they are read off the log, and the row - // holds only that every group ran to its end. The T14 and the AMD bench. - // One boot runs the whole suite, so three readbacks are three runs. + // holds only that every group ran to its end on the bench's NVMe, on + // a boot that did not spend itself formatting DATA. The T14 and the + // AMD bench. One boot runs the whole suite, so three readbacks are + // three runs. "osbench", - metal::Metal { arms: BENCHCASE, judge: |b| osbench_ran(b[0], &OSBENCH_GROUPS) }, + metal::Metal { + arms: BENCHCASE, + judge: |b| { + not_the_format_boot(b[0])?; + data_on_the_nvme(b[0])?; + osbench_ran(b[0], &OSBENCH_GROUPS) + }, + }, ), ( // `osbench lan` against the `osbench-peer` the other bench's Linux runs, @@ -1770,6 +1779,47 @@ fn osbench_ran(boot: &metal::Readback, jobs: &[&str]) -> Result<(), String> { Ok(()) } +/// The boot that formats a freshly designated DATA partition is no measured +/// run: its start went to the format, and the volume it leaves is new. It +/// says how long the format took. +fn not_the_format_boot(boot: &metal::Readback) -> Result<(), String> { + const FORMATTING: &str = "fileserver: block 0 designates this partition for ToyOS; formatting it"; + const SERVING: &str = "fileserver: Data serving "; + let log = boot.log(); + let text = log.text(); + let Some(began) = text.lines().find(|l| l.contains(FORMATTING)) else { + return Ok(()); + }; + let ended = text.lines().find(|l| l.contains(SERVING)); + let secs = ended.and_then(since_boot).zip(since_boot(began)).map(|(end, start)| end - start); + let took = secs.map_or("its end unlogged".to_string(), |s| format!("{s:.3} s")); + Err(format!("this boot formatted DATA off its designation ({took}), and the format's boot is no measured run")) +} + +/// DATA, which `osbench persist` writes, was a volume on a TOYOS-DATA +/// partition the NVMe driver serves, mounted, and not memory standing in for +/// a partition that is not ours. The file server refuses DATA where the block +/// service offers two TOYOS-DATA partitions, so a mount beside the NVMe's +/// partition is that partition's. +fn data_on_the_nvme(boot: &metal::Readback) -> Result<(), String> { + let log = boot.log(); + let text = log.text(); + let served = format!(" of type {} at block ", toyos_gpt::Guid::TOYOS_DATA); + if !text.lines().any(|l| l.contains("nvme] nvme ") && l.contains(&served)) { + return Err("the NVMe driver served no TOYOS-DATA partition, so persist measured no disk of the bench's".into()); + } + if !text.contains("fileserver: mounted the DATA volume") || text.contains("it is absent this boot") { + let why = text.lines().find(|l| l.contains("fileserver: the DATA")).unwrap_or("the file server said nothing of DATA"); + return Err(format!("DATA was not the NVMe's partition, so persist measured no disk: {why}")); + } + Ok(()) +} + +/// A log line's seconds since boot: `[2026-10-24 14:17:17 12.664 fileserver]`. +fn since_boot(line: &str) -> Option { + line.strip_prefix('[')?.split_whitespace().nth(2)?.parse().ok() +} + /// A timing boot was in ACPI mode before its first job and stayed there, as a /// shipped image and the bench's Linux are. In legacy mode the T14's firmware /// stops every CPU every 2.2 s, and its time would be measured as ToyOS's From 4895e1caefa868e2f6b8f1dbd8ded423f9991457 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 17:05:36 +0100 Subject: [PATCH 2/7] issues: a clean fsync on DATA writes the superblock and flushes the disk #1097's AMD validation reads persist.fsync_clean at 1.23 ms under ToyOS against 57-70 us under Linux on the same NVMe. Mounted::sync writes the superblock on every commit, so the cache owes a Flush even where nothing changed. #1067's handoff named this open and unfiled. Co-Authored-By: Claude Opus 5.5 --- ...tes-the-superblock-and-flushes-the-disk.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md diff --git a/issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md b/issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md new file mode 100644 index 00000000000..45d793b892a --- /dev/null +++ b/issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md @@ -0,0 +1,31 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A clean fsync on DATA writes the superblock and flushes the disk + +An fsync of a file on DATA syncs the whole volume +(`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`), and the +volume's sync, `bcachefs::Mounted::sync` (`lib/bcachefs/src/fs.rs`), commits +unconditionally. It flushes the cache, writes the superblock, and flushes +again. The file server's cache issues a device flush only where it owes one, +after a write it made. The superblock write makes one owed every time, so an +fsync with nothing new to make durable still costs a block write and an NVMe +Flush. + +Measured on the AMD bench (2026-10-28, `osbench persist` at PR #1097's +c8f3fbfb0, DATA on its designated NVMe partition): `persist.fsync_clean` +p50 1,233,821 ns under ToyOS, against 56,649 and 69,703 ns under the +bench's Linux on the same NVMe. That is about the cost of one Flush on that +disk, and `persist.write_4k_fsync`, which owes one, reads 2.9 ms on ToyOS +and 3.0 to 3.6 ms on Linux. + +**Exit**: a sync of a volume with nothing uncommitted writes and flushes +nothing, shown by counting the requests a clean sync issues on a host disk. +`persist.fsync_clean` on the AMD bench then reads within the file server's +round trip, tens of µs and not a Flush's ms. + +**Owner**: the DATA format (`lib/bcachefs/src/fs.rs`) and the file server's +DATA volume (`system/fileserver/src/data.rs`). From b4121ccc0e66ddefca996fb2cd48d6913c184929 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 17:23:19 +0100 Subject: [PATCH 3/7] review r1: the disk rules say what is true, the judge is hostdisk's, and the rows' debts are filed Rebased on #1095 (370d7d9f6). - B1: Linux's fs.disk.mount now carries the superblock options too, where ext4's nobarrier would be, and served_by names the device's write-cache mode (queue/write_cache). - B2: the README states the cache rule as procedure. Every row reads only files it wrote in its run, and neither cache is dropped. ToyOS's write rows include its 32 MiB and 2 s write-back. The 4 KiB sequential read's 158 MB/s is filed: a host probe of the cache shows a file 16 blocks over CLEAN_LIMIT missing every block on every pass (16,400 requests) and one 16 under missing none, the insertion-order eviction. - B3: the README says the compare refuses differing media, and that ToyOS's medium is a constant the osbench row alone makes true. - B4: the rewrap. - The osbench row's judge is hostdisk::left_alone with the loader's volumes: Mounted passes, Formatted is the format boot (its duration stated), and Foreign or memory is refused. The second parser goes. - The README says ToyOS's log volume shares the NVMe, and the image comment says the image, not the stick, has no DATA partition. - issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md: the cached fs.disk rows' 4-12x, the round trip, against Linux and ToyOS's own /tmp, with parity as its exit. Co-Authored-By: Claude Opus 5.5 --- ...uest-costs-ten-times-a-kernel-file-call.md | 34 +++++++++++++ ...rge-as-datas-clean-cache-reads-the-disk.md | 33 +++++++++++++ tests/guest/osbench/README.md | 46 ++++++++++------- tests/guest/osbench/src/files.rs | 6 +-- tests/guest/osbench/src/sys/linux.rs | 24 +++++---- tests/images/benchcase.toml | 12 ++--- tests/toyos.rs | 49 +++++++------------ 7 files changed, 137 insertions(+), 67 deletions(-) create mode 100644 issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md create mode 100644 issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md diff --git a/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md b/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md new file mode 100644 index 00000000000..c15b49f2052 --- /dev/null +++ b/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md @@ -0,0 +1,34 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A file server request costs ten times a kernel file call + +Every file operation on DATA is a request to the DATA file server process. +On the AMD bench (2026-10-28, `osbench persist` at PR #1097's c8f3fbfb0, +DATA on its NVMe) that round trip sets the cached rows, against the bench's +Linux page cache on the same NVMe and ToyOS's own kernel `/tmp`: + +| row, p50 | ToyOS DATA | ToyOS `/tmp` | Linux home | +|---|---|---|---| +| `read_4k_at` (seek and 4 KiB read) | 16,903 ns | 1,323 ns | 1,463–2,375 ns | +| `stat` | 10,530 ns | 2,615 ns | 1,453–2,794 ns | +| `open_close` | 18,686 ns | 6,072 ns | 3,056–4,400 ns | +| `rename` | 59,154 ns | 6,082 ns | 9,308–10,895 ns | +| `seq_read.1048576` | 6,401 MB/s | 7,868 MB/s | 11,295–11,403 MB/s | + +The disk adds nothing to these. The same rows on 2026-10-24's in-memory DATA +read 16,822, 10,450, 14,178 and 58,002 ns and 6,080 MB/s. The `/tmp` column +is that boot's kernel tmpfs (the orchestrator's full-final session; both +tables are in PR #1097's evidence). What the round trip spends its time on, +on metal, is unmeasured. `issues/a-cached-read-from-a-file-server-is-measured-only-under-tcg.md` +splits it under TCG. + +**Exit**: the cached `fs.disk.*` rows within Linux's on both benches, or +each row's remaining difference measured to its parts and justified, in a +labelled current-state comparison. + +**Owner**: the file server (`system/fileserver/`) and its transport +(`sdk/toyos/src/fs.rs`). diff --git a/issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md b/issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md new file mode 100644 index 00000000000..4bbdbc4686c --- /dev/null +++ b/issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md @@ -0,0 +1,33 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A sequential read of a file as large as DATA's clean cache reads the disk + +The file server's cache (`system/fileserver/src/cache.rs`) keeps +`CLEAN_LIMIT` clean blocks, 16,384 of 4 KiB (64 MiB), and evicts them in +the order they arrived. A hit does not move a block. So once a file and the +volume's metadata together pass that limit, a sequential read evicts, at +each miss, the block it is about to read next, and every pass misses every +block. + +Measured on the AMD bench (2026-10-28, `osbench persist` at PR #1097's +c8f3fbfb0, DATA on its NVMe): `fs.disk.seq_read.4096`, a 64 MiB file read in +4 KiB calls, p50 158 MB/s, min 158 and max 426, against 4,960 and 5,054 MB/s +under the bench's Linux. Four of the five rounds ran at about a third of the +fifth. + +The cause, shown on the host by a probe of the cache over a counting disk +(PR #1097's evidence, `probe.log`): blocks written in 4 KiB calls, flushed, +then read in order five times. With 16 blocks over the limit (16,400), each +pass made 16,400 disk requests. With 16 blocks under it (16,368), each pass +made none. + +**Exit**: a sequential read of a file larger than the clean limit keeps +hitting what the cache can hold, rather than missing everything. A host test +of the cache shows it, by counting the disk reads of repeated passes over +`CLEAN_LIMIT + 16` blocks. + +**Owner**: the file server's cache (`system/fileserver/src/cache.rs`). diff --git a/tests/guest/osbench/README.md b/tests/guest/osbench/README.md index 8d0626f8157..7ecd8e7617c 100644 --- a/tests/guest/osbench/README.md +++ b/tests/guest/osbench/README.md @@ -140,31 +140,43 @@ so it is no baseline. bench's NVMe. On Linux that is the runner's home, ext4 on the bench's root partition. On ToyOS it is DATA: the bench's TOYOS-DATA partition, its fourth, named `ToyOS data`, which ToyOS designates itself. It is served through the -DATA file server and the NVMe driver. The `osbench` row refuses a boot whose -DATA was anything else, and refuses the boot that formats a newly -designated DATA as a run, saying how long the format took. Every row's -`served_by` and `medium` name the filesystem and the disk it measured. +DATA file server and the NVMe driver. ToyOS boots off the same NVMe, and its +log volume is there too, so the log's writes share the device with +`persist` (`toyos-metal`'s internal install). The `osbench` row refuses a +boot whose DATA was anything else, and refuses the boot that formats a +newly designated DATA as a run, saying how long the format took. What makes the disk rows one comparison, and what stays each system's own: - **One disk.** Both write the bench's one NVMe namespace: Linux its root partition, ToyOS its fourth. They are different regions of one device, - through each system's own driver. Each row's `medium` is `nvme` on both, - and the compare refuses a verdict on any row where it is not. + through each system's own driver. Each row's `medium` names it, and the + compare refuses a verdict where the two systems' media differ. On Linux + the medium is read off sysfs. On ToyOS it is a constant, `nvme`, since a + program cannot ask what holds `/home`: only the `osbench` row's refusal + makes it true. - **Each system's own filesystem.** ext4, and ToyOS's DATA format behind - its file server, are part of what is compared. Linux's mount options are - in `fs.disk.mount`. The free space they write into differs: Linux's root - holds its installation, and ToyOS's DATA is a volume of its own. + its file server, are part of what is compared. Linux's mount and + superblock options are in `fs.disk.mount`. The free space they write into + differs: Linux's root holds its installation, and ToyOS's DATA is a volume + of its own. - **One durability.** A `persist` row's write counts once `sync_all` or `sync_data` returns. Both systems answer that after the device's own - cache has been flushed: Linux's ext4 by its default barriers, which a - `nobarrier` in `fs.disk.mount` would show, and ToyOS's NVMe driver by - issuing Flush where the controller has a volatile write cache, which its - boot log says. ToyOS's fsync makes its whole volume durable, not just the - file (`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`). -- **One cache state.** Every row writes its own files in its run, and its - reads find them in each system's cache. `persist.seq_read_cached` and - `fs.disk.*` are cached reads on both, and no row reads the disk cold. + cache has been flushed. On Linux, ext4 does it by its default barriers: a + `nobarrier` would show in `fs.disk.mount`'s superblock options, and the + device's write-cache mode is in `served_by` (`write cache write back`). + On ToyOS, the NVMe driver issues Flush where the controller has a volatile + write cache, which its boot log says. ToyOS's fsync makes its whole volume + durable, not just the file + (`issues/an-fsync-of-a-served-file-syncs-its-whole-volume.md`). +- **One procedure for the cache.** Every row reads only files it wrote in its + own run, and neither system's cache is dropped before it. What then + reaches the disk is each system's own write-back and cache policy, and + part of what is compared. On ToyOS the DATA cache writes back once 32 MiB + is dirty and syncs the volume 2 s after an unsynced write, so the 64 MiB + `fs.disk.seq_write.*` rounds write the disk. Its cache keeps 64 MiB clean, + evicting in insertion order, and the 4 KiB sequential read misses + (`issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md`). ## What is not a comparison diff --git a/tests/guest/osbench/src/files.rs b/tests/guest/osbench/src/files.rs index 1a018901c2b..d7414c5c3d3 100644 --- a/tests/guest/osbench/src/files.rs +++ b/tests/guest/osbench/src/files.rs @@ -1,7 +1,8 @@ //! Files, through std's `fs` on both systems: the same operations under a //! filesystem each keeps in memory (`mem`) and under the one on the bench's //! disk (`disk`, which `storage` runs beside its durable writes). Nothing here -//! asks for durability, so on the disk these are the cached path. +//! asks for durability: each system's cache and write-back decide what reaches +//! the disk, and that is part of what the rows measure. //! //! `mem` is Linux's `/dev/shm` through the VFS, and ToyOS's `/tmp`, which the //! kernel serves through syscalls. `disk` is each system's @@ -92,8 +93,7 @@ fn page_at(i: usize, pages: usize) -> u64 { } /// Every benchmark under `target`, each line naming what served it and its -/// medium. Each -/// one's comment is its warm-up and repetitions. +/// medium. Each one's comment is its warm-up and repetitions. pub fn suite(target: &Target) { condition(&format!("fs.{}.served_by", target.name), &target.served_by); #[cfg(target_os = "linux")] diff --git a/tests/guest/osbench/src/sys/linux.rs b/tests/guest/osbench/src/sys/linux.rs index 91c38cec919..45188760ccc 100644 --- a/tests/guest/osbench/src/sys/linux.rs +++ b/tests/guest/osbench/src/sys/linux.rs @@ -496,8 +496,9 @@ pub mod ipc { } /// The mount holding `path`, off `/proc/self/mountinfo`: its filesystem -/// type, its source and its options. -fn mount_of(path: &str) -> Option<(String, String, String)> { +/// type, its source, its mount's options and its superblock's, where a +/// filesystem's own options such as ext4's `nobarrier` are. +fn mount_of(path: &str) -> Option<(String, String, String, String)> { let mounts = fs::read_to_string("/proc/self/mountinfo").ok()?; // `id parent major:minor root mountpoint options... - fstype source superoptions` mounts @@ -509,16 +510,18 @@ fn mount_of(path: &str) -> Option<(String, String, String)> { let options = left.next()?; let mut right = right.split_whitespace(); let (fstype, source) = (right.next()?, right.next()?); + let superblock = right.next().unwrap_or_default(); let under = path == point || point == "/" || path.starts_with(&format!("{point}/")); - under.then(|| (point.len(), (fstype.to_string(), source.to_string(), options.to_string()))) + under.then(|| (point.len(), (fstype.to_string(), source.to_string(), options.to_string(), superblock.to_string()))) }) .max_by_key(|(len, _)| *len) .map(|(_, mount)| mount) } -/// How `dir` is mounted, for a condition: its filesystem and options. +/// How `dir` is mounted, for a condition: its filesystem, its mount's +/// options and its superblock's. pub fn mount_options(dir: &str) -> String { - mount_of(dir).map_or_else(|| "unknown".into(), |(fstype, _, options)| format!("{fstype} {options}")) + mount_of(dir).map_or_else(|| "unknown".into(), |(fstype, _, options, superblock)| format!("{fstype} {options} {superblock}")) } /// What a transfer row reports of the stack beneath it: the kernel's TCP segment counts and @@ -631,13 +634,15 @@ fn tcp_info(stream: &std::net::TcpStream) -> Result { ]) } -/// The runner's home, the filesystem and disk model holding it, as `ext4, -/// NVMe disk, model `, and the disk's kind as a medium: `nvme`, `usb`, +/// The runner's home, the filesystem and disk holding it, as `ext4, NVMe +/// disk, model , write cache write back` (the block layer's +/// `queue/write_cache`: whether a sync must flush the device's cache), and +/// the disk's kind as a medium: `nvme`, `usb`, /// `sata`, `mmc`, `virtio`, `other`, or `none` for no block device. The path /// itself names the user and is never printed. pub fn persistent_dir() -> Option<(String, String, String)> { let home = std::env::var("HOME").ok()?; - let (fstype, source, _) = mount_of(&home)?; + let (fstype, source, _, _) = mount_of(&home)?; // btrfs reports an anonymous `0:NN` that names no block device; its // source device's own number does. let sysfs = |devno: String| fs::canonicalize(format!("/sys/dev/block/{devno}")).ok(); @@ -652,6 +657,7 @@ pub fn persistent_dir() -> Option<(String, String, String)> { }; let (disk, mapped) = backing_disk(device); let model = read(&disk.join("device/model").to_string_lossy()).unwrap_or_else(|| "unknown".into()); + let cache = read(&disk.join("queue/write_cache").to_string_lossy()).unwrap_or_else(|| "unknown".into()); let name = disk.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default(); // A SCSI disk is a USB one where sysfs puts it under a USB device. let usb = disk.to_string_lossy().contains("/usb"); @@ -663,7 +669,7 @@ pub fn persistent_dir() -> Option<(String, String, String)> { let through = if mapped { " through device-mapper" } else { "" }; // The device's own name is never printed: an installer or a user chose // a mapper's (`luks-`, a volume group named for the host). - Some((home, format!("{fstype}{through}, {kind} disk, model {model}"), medium.into())) + Some((home, format!("{fstype}{through}, {kind} disk, model {model}, write cache {cache}"), medium.into())) } /// The `major:minor` of the filesystem holding `path`, off mountinfo's third diff --git a/tests/images/benchcase.toml b/tests/images/benchcase.toml index 4cd1394e7d0..e172d1a076a 100644 --- a/tests/images/benchcase.toml +++ b/tests/images/benchcase.toml @@ -1,12 +1,12 @@ # The boot `osbench` measures ToyOS on, for a comparison with the same program # on the bench's own Linux (`tests/guest/osbench`). # -# Only what a job list needs to run and to reach the log on the boot stick, the -# ACPI server, and netstack for `osbench lan` and `osbench internet`, each its -# own metal row (`tests/toyos.rs`): no compositor or sound, so little but the -# measurement asks for a CPU. The stick has no DATA partition, so DATA is the -# bench's: the TOYOS-DATA partition on its NVMe, which `osbench persist` -# writes, and which the `osbench` row refuses a boot to be without. +# Only what a job list needs to run and to reach the log, the ACPI server, and +# netstack for `osbench lan` and `osbench internet`, each its own metal row +# (`tests/toyos.rs`): no compositor or sound, so little but the measurement +# asks for a CPU. The image has no DATA partition, so DATA is the bench's: the +# TOYOS-DATA partition on its NVMe, which `osbench persist` writes, and which +# the `osbench` row refuses a boot to be without. # # The runner's capability carries `jobs`, for the fairness run that starts each # workload as a job of its own; `counters`, `trace` and `roster`, so `osbench diff --git a/tests/toyos.rs b/tests/toyos.rs index c228add7987..de07bbd109f 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -1153,7 +1153,6 @@ const METAL: &[(&str, metal::Metal)] = &[ metal::Metal { arms: BENCHCASE, judge: |b| { - not_the_format_boot(b[0])?; data_on_the_nvme(b[0])?; osbench_ran(b[0], &OSBENCH_GROUPS) }, @@ -1779,40 +1778,26 @@ fn osbench_ran(boot: &metal::Readback, jobs: &[&str]) -> Result<(), String> { Ok(()) } -/// The boot that formats a freshly designated DATA partition is no measured -/// run: its start went to the format, and the volume it leaves is new. It -/// says how long the format took. -fn not_the_format_boot(boot: &metal::Readback) -> Result<(), String> { - const FORMATTING: &str = "fileserver: block 0 designates this partition for ToyOS; formatting it"; - const SERVING: &str = "fileserver: Data serving "; - let log = boot.log(); - let text = log.text(); - let Some(began) = text.lines().find(|l| l.contains(FORMATTING)) else { - return Ok(()); - }; - let ended = text.lines().find(|l| l.contains(SERVING)); - let secs = ended.and_then(since_boot).zip(since_boot(began)).map(|(end, start)| end - start); - let took = secs.map_or("its end unlogged".to_string(), |s| format!("{s:.3} s")); - Err(format!("this boot formatted DATA off its designation ({took}), and the format's boot is no measured run")) -} - -/// DATA, which `osbench persist` writes, was a volume on a TOYOS-DATA -/// partition the NVMe driver serves, mounted, and not memory standing in for -/// a partition that is not ours. The file server refuses DATA where the block -/// service offers two TOYOS-DATA partitions, so a mount beside the NVMe's -/// partition is that partition's. +/// DATA, which `osbench persist` writes, was a volume of ours on the bench's +/// NVMe, mounted (`hostdisk::left_alone`, which also refuses DATA from +/// memory). The boot that formatted it is no measured run: its start went to +/// the format, and the volume it leaves is new; it says how long that took. fn data_on_the_nvme(boot: &metal::Readback) -> Result<(), String> { let log = boot.log(); - let text = log.text(); - let served = format!(" of type {} at block ", toyos_gpt::Guid::TOYOS_DATA); - if !text.lines().any(|l| l.contains("nvme] nvme ") && l.contains(&served)) { - return Err("the NVMe driver served no TOYOS-DATA partition, so persist measured no disk of the bench's".into()); - } - if !text.contains("fileserver: mounted the DATA volume") || text.contains("it is absent this boot") { - let why = text.lines().find(|l| l.contains("fileserver: the DATA")).unwrap_or("the file server said nothing of DATA"); - return Err(format!("DATA was not the NVMe's partition, so persist measured no disk: {why}")); + let ours = common::hostdisk::loader_named(&boot.loader())?; + match common::hostdisk::left_alone(&log, &ours)?.1 { + common::hostdisk::Data::Mounted => Ok(()), + common::hostdisk::Data::Formatted => { + let text = log.text(); + let at = |said: &str| text.lines().find(|l| l.contains(said)).and_then(since_boot); + let took = at("formatting it").zip(at("fileserver: Data serving ")).map(|(start, end)| end - start); + let took = took.map_or("its end unlogged".to_string(), |s| format!("{s:.3} s")); + Err(format!("this boot formatted DATA off its designation ({took}), and the format's boot is no measured run")) + } + common::hostdisk::Data::Foreign => { + Err("DATA on the bench's NVMe was no volume of ours, so persist measured no disk".into()) + } } - Ok(()) } /// A log line's seconds since boot: `[2026-10-24 14:17:17 12.664 fileserver]`. From 069cbbeb878d5df3015ec636f786613194643304 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 17:35:45 +0100 Subject: [PATCH 4/7] issues: a durable write on the T14's NVMe costs ToyOS six times Linux's From #1097's T14 validation: a clean fsync costs ToyOS 3.0 ms, so its Flush on that disk costs about that, against Linux's whole write and fsync at 0.8-1.0 ms. The AMD's Samsung shows no such gap. Co-Authored-By: Claude Opus 5.5 --- ...-t14s-nvme-costs-toyos-six-times-linuxs.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md diff --git a/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md b/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md new file mode 100644 index 00000000000..7fdcc1bff35 --- /dev/null +++ b/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md @@ -0,0 +1,35 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A durable write on the T14's NVMe costs ToyOS six times Linux's + +On the T14 (SK hynix BC501, volatile write cache present, so ToyOS's NVMe +driver issues Flush), the 2026-10-28 validation of PR #1097 at c8f3fbfb0 +read these p50s. ToyOS is one boot, with DATA on the NVMe; Linux is two runs +on ext4 on the same NVMe: + +| row | ToyOS | Linux | +|---|---|---| +| `persist.fsync_clean` | 3,019,602 ns | 22,971 and 23,002 ns | +| `persist.write_4k_fsync` | 6,125,947 ns | 811,313 and 1,009,786 ns | +| `persist.append_4k_fsync` | 6,182,133 ns | 796,449 and 1,040,764 ns | +| `persist.seq_write_fsync` | 162 MB/s | 587 and 598 MB/s | + +A clean fsync on ToyOS is one superblock write and one Flush +(`issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md`), +so a Flush on that disk costs ToyOS about 3 ms. Linux's whole write and +fsync, its journal commit included, costs 0.8 to 1.0 ms. On the AMD bench's +Samsung the two systems' durable writes cost about the same (2.9 against 3.0 +to 3.6 ms). So the gap is that disk's under ToyOS, not durability's. The +cause is unmeasured. It could be the commit's two flushes, the drive's +power state between requests, or the driver's completion path. + +**Exit**: `persist.write_4k_fsync` and `persist.fsync_clean` on the T14 +within Linux's, or the remaining difference measured to its parts and +justified, in a labelled current-state comparison. + +**Owner**: the NVMe driver (`system/nvme/`) and the DATA volume's commit +(`lib/bcachefs/src/fs.rs`). From d6fc8be014d8c4c1249eba705982e248fe198c25 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 18:16:52 +0100 Subject: [PATCH 5/7] issues: the T14's durable writes took 1 to 6 ms across two boots The landing head's T14 validation (069cbbeb8) read ToyOS's write_4k_fsync at 1.10 ms against Linux's 1.01-1.04 ms, where c8f3fbfb0's read 6.1 ms. No storage-path code changed between them and both booted off the NVMe, so the record states both boots and an unknown cause instead of a sixfold gap. Co-Authored-By: Claude Opus 5.5 --- ...-t14s-nvme-costs-toyos-six-times-linuxs.md | 35 ------------------- ...4s-nvme-took-1-to-6-ms-across-two-boots.md | 34 ++++++++++++++++++ 2 files changed, 34 insertions(+), 35 deletions(-) delete mode 100644 issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md create mode 100644 issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md diff --git a/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md b/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md deleted file mode 100644 index 7fdcc1bff35..00000000000 --- a/issues/a-durable-write-on-the-t14s-nvme-costs-toyos-six-times-linuxs.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-28 ---- - -# A durable write on the T14's NVMe costs ToyOS six times Linux's - -On the T14 (SK hynix BC501, volatile write cache present, so ToyOS's NVMe -driver issues Flush), the 2026-10-28 validation of PR #1097 at c8f3fbfb0 -read these p50s. ToyOS is one boot, with DATA on the NVMe; Linux is two runs -on ext4 on the same NVMe: - -| row | ToyOS | Linux | -|---|---|---| -| `persist.fsync_clean` | 3,019,602 ns | 22,971 and 23,002 ns | -| `persist.write_4k_fsync` | 6,125,947 ns | 811,313 and 1,009,786 ns | -| `persist.append_4k_fsync` | 6,182,133 ns | 796,449 and 1,040,764 ns | -| `persist.seq_write_fsync` | 162 MB/s | 587 and 598 MB/s | - -A clean fsync on ToyOS is one superblock write and one Flush -(`issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md`), -so a Flush on that disk costs ToyOS about 3 ms. Linux's whole write and -fsync, its journal commit included, costs 0.8 to 1.0 ms. On the AMD bench's -Samsung the two systems' durable writes cost about the same (2.9 against 3.0 -to 3.6 ms). So the gap is that disk's under ToyOS, not durability's. The -cause is unmeasured. It could be the commit's two flushes, the drive's -power state between requests, or the driver's completion path. - -**Exit**: `persist.write_4k_fsync` and `persist.fsync_clean` on the T14 -within Linux's, or the remaining difference measured to its parts and -justified, in a labelled current-state comparison. - -**Owner**: the NVMe driver (`system/nvme/`) and the DATA volume's commit -(`lib/bcachefs/src/fs.rs`). diff --git a/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md b/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md new file mode 100644 index 00000000000..1d94ba12fb7 --- /dev/null +++ b/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md @@ -0,0 +1,34 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# ToyOS's durable writes on the T14's NVMe took 1 to 6 ms across two boots + +On the T14 (SK hynix BC501, volatile write cache present, so ToyOS's NVMe +driver issues Flush), two boots of PR #1097, both off the internal NVMe with +DATA on it, read these p50s. Linux is two runs on ext4 on the same NVMe in +each validation: + +| row | ToyOS, c8f3fbfb0 | ToyOS, 069cbbeb8 | Linux (both validations) | +|---|---|---|---| +| `persist.fsync_clean` | 3,019,602 ns | 464,505 ns | 22,970 to 23,012 ns | +| `persist.write_4k_fsync` | 6,125,947 ns | 1,099,737 ns | 811,313 to 1,035,997 ns | +| `persist.append_4k_fsync` | 6,182,133 ns | 1,012,768 ns | 796,449 to 1,200,332 ns | +| `persist.seq_write_fsync` | 162 MB/s | 347 MB/s | 527 to 598 MB/s | + +No change to the NVMe driver, the block service, the file server or the +DATA format lies between the two heads; the second rebased onto #1095's +boot path, and both booted off the NVMe. Why one boot's durable writes cost +six times the other's is unknown. The first followed DATA's format by less +than a day, so the drive's own state is a candidate, an inference. The clean +fsync's remaining 20x, a superblock write and a Flush where Linux issues +neither, is +`issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md`. + +**Exit**: repeated boots on the T14 read ToyOS's `persist.write_4k_fsync` +within Linux's, or the boots that do not are explained by a measured cause. + +**Owner**: the NVMe driver (`system/nvme/`) and the DATA volume's commit +(`lib/bcachefs/src/fs.rs`). From ef94653fcd8092301a93aa791829144eef488d39 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 18:33:03 +0100 Subject: [PATCH 6/7] review r2: the records say what was measured - N1: the T14 durable-write record drops "followed DATA's format by less than a day", which both boots did, and says Linux's clean fsync takes 23 us where it claimed Linux issues no write or Flush, which nothing measured. - N2: the file-server record is "three to twelve times", the ratios its table shows. Its /tmp column is the same boot's. "The disk adds nothing" holds for four rows, and open_close's 32% on the NVMe is stated as unattributed. Co-Authored-By: Claude Opus 5.5 --- ...uest-costs-ten-times-a-kernel-file-call.md | 34 ------------------ ...hree-to-twelve-times-a-kernel-file-call.md | 36 +++++++++++++++++++ ...4s-nvme-took-1-to-6-ms-across-two-boots.md | 6 ++-- 3 files changed, 38 insertions(+), 38 deletions(-) delete mode 100644 issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md create mode 100644 issues/a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call.md diff --git a/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md b/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md deleted file mode 100644 index c15b49f2052..00000000000 --- a/issues/a-file-server-request-costs-ten-times-a-kernel-file-call.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-28 ---- - -# A file server request costs ten times a kernel file call - -Every file operation on DATA is a request to the DATA file server process. -On the AMD bench (2026-10-28, `osbench persist` at PR #1097's c8f3fbfb0, -DATA on its NVMe) that round trip sets the cached rows, against the bench's -Linux page cache on the same NVMe and ToyOS's own kernel `/tmp`: - -| row, p50 | ToyOS DATA | ToyOS `/tmp` | Linux home | -|---|---|---|---| -| `read_4k_at` (seek and 4 KiB read) | 16,903 ns | 1,323 ns | 1,463–2,375 ns | -| `stat` | 10,530 ns | 2,615 ns | 1,453–2,794 ns | -| `open_close` | 18,686 ns | 6,072 ns | 3,056–4,400 ns | -| `rename` | 59,154 ns | 6,082 ns | 9,308–10,895 ns | -| `seq_read.1048576` | 6,401 MB/s | 7,868 MB/s | 11,295–11,403 MB/s | - -The disk adds nothing to these. The same rows on 2026-10-24's in-memory DATA -read 16,822, 10,450, 14,178 and 58,002 ns and 6,080 MB/s. The `/tmp` column -is that boot's kernel tmpfs (the orchestrator's full-final session; both -tables are in PR #1097's evidence). What the round trip spends its time on, -on metal, is unmeasured. `issues/a-cached-read-from-a-file-server-is-measured-only-under-tcg.md` -splits it under TCG. - -**Exit**: the cached `fs.disk.*` rows within Linux's on both benches, or -each row's remaining difference measured to its parts and justified, in a -labelled current-state comparison. - -**Owner**: the file server (`system/fileserver/`) and its transport -(`sdk/toyos/src/fs.rs`). diff --git a/issues/a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call.md b/issues/a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call.md new file mode 100644 index 00000000000..d1c22062002 --- /dev/null +++ b/issues/a-file-server-request-costs-three-to-twelve-times-a-kernel-file-call.md @@ -0,0 +1,36 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A file server request costs three to twelve times a kernel file call + +Every file operation on DATA is a request to the DATA file server process. +On the AMD bench (2026-10-28, `osbench` at PR #1097's c8f3fbfb0, DATA on its +NVMe), that boot's cached rows on DATA against its kernel `/tmp`, and the +bench's Linux page cache on the same NVMe: + +| row, p50 | ToyOS DATA | ToyOS `/tmp` | DATA / `/tmp` | Linux home | +|---|---|---|---|---| +| `read_4k_at` (seek and 4 KiB read) | 16,903 ns | 1,413 ns | 12.0x | 1,463–2,375 ns | +| `stat` | 10,530 ns | 2,596 ns | 4.1x | 1,453–2,794 ns | +| `open_close` | 18,686 ns | 6,091 ns | 3.1x | 3,056–4,400 ns | +| `rename` | 59,154 ns | 6,242 ns | 9.5x | 9,308–10,895 ns | +| `seq_read.1048576` | 6,401 MB/s | 8,041 MB/s | | 11,295–11,403 MB/s | + +Four of these rows the disk does not set. On 2026-10-24's in-memory DATA the +same rows read within about 5%: `read_4k_at` 16,822, `stat` 10,450 and +`rename` 58,002 ns, and `seq_read.1048576` 6,080 MB/s. `open_close` read +14,178 ns there, so it is 32% dearer on the NVMe, and what that costs is +unattributed. Both tables are in PR #1097's evidence. What the round trip +spends its time on, on metal, is unmeasured. +`issues/a-cached-read-from-a-file-server-is-measured-only-under-tcg.md` +splits it under TCG. + +**Exit**: the cached `fs.disk.*` rows within Linux's on both benches, or +each row's remaining difference measured to its parts and justified, in a +labelled current-state comparison. + +**Owner**: the file server (`system/fileserver/`) and its transport +(`sdk/toyos/src/fs.rs`). diff --git a/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md b/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md index 1d94ba12fb7..fb0341082f4 100644 --- a/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md +++ b/issues/toyos-durable-writes-on-the-t14s-nvme-took-1-to-6-ms-across-two-boots.md @@ -21,10 +21,8 @@ each validation: No change to the NVMe driver, the block service, the file server or the DATA format lies between the two heads; the second rebased onto #1095's boot path, and both booted off the NVMe. Why one boot's durable writes cost -six times the other's is unknown. The first followed DATA's format by less -than a day, so the drive's own state is a candidate, an inference. The clean -fsync's remaining 20x, a superblock write and a Flush where Linux issues -neither, is +six times the other's is unknown. The clean fsync's remaining 20x, a +superblock write and a Flush where Linux's clean fsync takes 23 µs, is `issues/a-clean-fsync-on-data-writes-the-superblock-and-flushes-the-disk.md`. **Exit**: repeated boots on the T14 read ToyOS's `persist.write_4k_fsync` From ddc26a3eacb704e5d1962a366782c4d6e3fdc206 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 18:34:12 +0100 Subject: [PATCH 7/7] issues: a created, fsynced and unlinked file costs ToyOS more than Linux The last disk row ToyOS loses at #1097's landing head without a record: 8-22% dearer on both benches, where the plain durable writes are at parity. Co-Authored-By: Claude Opus 5.5 --- ...linked-file-costs-toyos-more-than-linux.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 issues/a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.md diff --git a/issues/a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.md b/issues/a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.md new file mode 100644 index 00000000000..7a0e32776a4 --- /dev/null +++ b/issues/a-created-fsynced-and-unlinked-file-costs-toyos-more-than-linux.md @@ -0,0 +1,30 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A created, fsynced and unlinked file costs ToyOS more than Linux + +`persist.create_fsync_unlink` creates a file on the persistent filesystem, +writes 4 KiB to it, `sync_all`s it, closes and removes it. At PR #1097's +landing head (069cbbeb8, 2026-10-28, one ToyOS boot and two Linux runs per +bench, DATA and Linux's home on the same NVMe) it read: + +| bench | ToyOS | Linux | +|---|---|---| +| T14 | 1,168,401 ns | 953,976 and 966,415 ns | +| AMD | 3,533,628 ns | 3,257,728 and 3,283,416 ns | + +That is 8 to 22% dearer, where the plain durable writes +(`write_4k_fsync`, `append_4k_fsync`) are at or near Linux's on both +benches. What the difference is made of is unmeasured. The create and the +unlink are file server requests, and the fsync commits the whole volume +with its superblock. + +**Exit**: the row within Linux's on both benches, or its difference +measured to its parts and justified, in a labelled current-state +comparison. + +**Owner**: the file server's DATA volume (`system/fileserver/src/data.rs`) +and the DATA format (`lib/bcachefs/src/fs.rs`).