fileserver: a volume's cache keeps an eighth of physical memory clean

issues/a-32-mib-anonymous-reserve-on-the-amd-bench-costs-twice-as-much-in-some-boots.md

@@ -0,0 +1,43 @@
+---
+status: open
+kind: finding
+opened: 2026-10-31
+---
+
+# A 32 MiB anonymous reserve on the AMD bench costs twice as much in some boots
+
+osbench's `mem.anon.map` times one `vm_reserve` of 32 MiB alone, with no
+page touched (`tests/guest/osbench/src/system.rs`, `anon_memory`). On the
+AMD bench (Ryzen 7 4800U) its p50 is either about 200 ns/MiB (6.4 µs a
+reserve) or 376–388 ns/MiB. Twelve `benchcase` boots of six trees
+(2026-10-28 to 2026-10-31, `ToyOSOrg/evidence` branch `pr-1099`
+ec19d8d3db4b4ebf01804c738dd17eaebbc202b4, `r4/anon-spawn.txt`) split:
+
+- Nine boots read 198–215, including four of ddc26a3ea, two of b3f9af23a
+  and one of f40791833.
+- Three boots read 376–388: two of 34409c9c2 and one of f40791833. That
+  last is the same tree as one of the fast boots.
+
+So the mode is per boot, not per change. The same boots' `mem.anon.touch`,
+`unmap` and `total` show no split. No other row separates the slow boots
+from all nine fast ones, apart from `work.build_shaped.j1` at 0.96x
+(`r4/slow-boots-comove.txt`). The mode does not follow the settled clock
+either: 4291 MHz in a fast boot and 4267 in a slow one. f40791833's slow
+boot and its fast one log the same firmware memory map and the same
+`pmm:` frame counts.
+
+What a reserve does in the slow mode is not measured. It takes the space's
+lock, places the range and builds the range's tables
+(`AddressSpace::reserve`, `kernel/src/mm/space.rs`): 1/512 of the range,
+here 16 leaf tables
+(`issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md`).
+
+A comparison of `mem.anon.map` between two AMD boots can read 1.9x that
+neither tree caused, as one did on PR #1099.
+
+**Exit**: a probe that splits a reserve's time into the lock, the placement
+and the table build (with its frame allocations and zeroing), in one boot of
+each mode, which names the slow mode's cause. Then fix it, or fold the cause
+into `space.rs`.
+
+**Owner**: the kernel's address spaces (`kernel/src/mm/space.rs`).

issues/a-sequential-read-of-a-file-as-large-as-datas-clean-cache-reads-the-disk.md

@@ -6,12 +6,14 @@ opened: 2026-10-28
 
 # A sequential read of a file as large as DATA's clean cache reads the disk
 
-The file server's cache (`system/fileserver/src/cache.rs`) keeps
-`CLEAN_LIMIT` clean blocks, 16,384 of 4 KiB (64 MiB), and evicts them in
-the order they arrived. A hit does not move a block. So once a file and the
-volume's metadata together pass that limit, a sequential read evicts, at
-each miss, the block it is about to read next, and every pass misses every
-block.
+The file server's cache (`system/fileserver/src/cache.rs`) keeps clean
+blocks up to the limit the server gives it, and evicts them in the order
+they arrived. A hit does not move a block. So once a file and the volume's
+metadata together pass that limit, a sequential read evicts, at each miss,
+the block it is about to read next, and every pass misses every block.
+The limit was a fixed 16,384 blocks (64 MiB) when this was measured; it is
+now most of an eighth of physical memory for DATA (#1099), so the 64 MiB
+file below fits, and a file larger than that limit falls off the same cliff.
 
 Measured on the AMD bench (2026-10-28, `osbench persist` at PR #1097's
 c8f3fbfb0, DATA on its NVMe): `fs.disk.seq_read.4096`, a 64 MiB file read in
@@ -28,6 +30,6 @@ made none.
 **Exit**: a sequential read of a file larger than the clean limit keeps
 hitting what the cache can hold, rather than missing everything. A host test
 of the cache shows it, by counting the disk reads of repeated passes over
-`CLEAN_LIMIT + 16` blocks.
+16 blocks more than the cache's limit.
 
 **Owner**: the file server's cache (`system/fileserver/src/cache.rs`).

issues/metalprobes-usb-read-is-answered-from-fileservers-cache.md

@@ -10,8 +10,8 @@ opened: 2026-09-27
 and times reading it back as "a cache miss the stick has to answer" — true
 while the kernel's write-back dropped a closed file from its cache. `/log` is
 served by a file server now, whose block cache
-(`system/fileserver/src/cache.rs`) keeps clean blocks until it holds
-`CLEAN_LIMIT` of them and drops nothing at a close. So the timed read is the
+(`system/fileserver/src/cache.rs`) keeps up to 16 MiB of clean blocks and
+drops nothing at a close. So the timed read is the
 file server's memory, and the `usbread` span is no longer the stick's.
 
 **Exit**: the timed read is one the stick answers — a file the file server has

issues/no-physical-memory-fairness.md

@@ -100,7 +100,9 @@ memory is counted to whichever job made it: a client's own connection is
 its job's, but a pipe a server makes and hands out (fileserver's stream
 pipes, netstack's bells) is the server's, which the server bounds. With no
 per-job limit a cap is still the only thing between one client and the
-machine.
+machine. The file servers' block caches hold an eighth between them, and no
+other job gets it back
+(`issues/the-file-servers-clean-blocks-are-given-back-to-no-other-job.md`).
 
 **Promoted to `defect` 2026-08-25** (finding-lifecycle ruling: isolation
 findings are security-adjacent and are promoted, never folded). One job

issues/the-file-servers-clean-blocks-are-given-back-to-no-other-job.md

@@ -0,0 +1,39 @@
+---
+status: open
+kind: defect
+opened: 2026-10-29
+---
+
+# The file servers' clean blocks are given back to no other job
+
+Each file server's block cache (`system/fileserver/src/cache.rs`) keeps clean
+blocks up to the limit its server gives it (`system/fileserver/src/main.rs`:
+`SMALL_CLEAN_BYTES` for log and boot, `data_clean_limit` for DATA). Together the DATA, log and boot servers keep at most an
+eighth of physical memory clean: 16 MiB each for log and boot, the rest of
+the eighth for DATA. Reads fill it, and so do writes, since a flushed block
+stays cached as clean. Nothing drops a clean block for another job's sake: a
+block leaves only to make room for another in the same cache, or when the
+server ends. So with every cache full, an ordinary job can be refused memory
+while an eighth of the machine holds blocks that could be read again from
+the disk.
+
+The eighth is the bound netstack and the compositor take for want of a
+kernel memory limit (`issues/no-physical-memory-fairness.md`); it is an
+interim, not a sizing.
+
+A clean block the allocator refuses, or whose place in the eviction order it
+refuses, is not kept: a read is still answered, and a flushed block is already
+on the disk. Four allocations still have no refusal path, so a server whose job is out of memory can still
+abort at them:
+- the block map's node, a small one every few inserts of a clean or a dirty
+  block;
+- a dirty block;
+- a fetch's staging buffer;
+- a flush's list of the dirty blocks and its buffer for one run.
+
+**Exit**: the file servers drop clean blocks when the memory service reports
+pressure, and `CLEAN_SHARE` is deleted.
+
+**Owner**: the file server (`system/fileserver/`) and the memory track
+(`issues/anonymous-memory-swaps-to-an-encrypted-partition.md`'s memory
+service).

sdk/toyos/src/system.rs

@@ -32,6 +32,14 @@ pub fn sysinfo(buf: &mut [u8; SYSINFO_HEADER_SIZE]) -> usize {
     syscall::sysinfo(toyos_abi::handle::HANDLE_INVALID, buf)
 }
 
+/// The machine's physical memory in bytes, as the kernel reports it.
+pub fn memory_total() -> u64 {
+    let mut buf = [0u8; SYSINFO_HEADER_SIZE];
+    let n = sysinfo(&mut buf);
+    assert!(n >= SYSINFO_HEADER_SIZE, "sysinfo returned {n} bytes");
+    syscall::SysinfoHeader::decode(&buf).memory_total
+}
+
 // Powering the machine off used to be a free function here, over a syscall that
 // took no argument. It is [`crate::syscap::SysCap::shutdown`] now: it is an
 // authority over the whole machine, and every one of those is a bit on a

system/compositor/src/session.rs

@@ -352,7 +352,7 @@ impl Session {
         let apps = launcher_apps();
         let desk = Desk::new(mode_rect(mode), scale, apps.len());
         let font = shapes.font(&desk);
-        let total_mem = total_memory();
+        let total_mem = toyos::system::memory_total();
         let max_windows =
             toyos_desktop::max_windows(total_mem, desk.screen, MAX_WINDOW_SLOTS as usize);
         println!(
@@ -2167,14 +2167,6 @@ fn blit(screen: &Screen, back: &Framebuffer, regions: &[Rect]) -> u64 {
     pixels
 }
 
-/// Total physical memory, as the kernel reports it.
-fn total_memory() -> u64 {
-    let mut buf = [0u8; system::SYSINFO_HEADER_SIZE];
-    let n = system::sysinfo(&mut buf);
-    assert!(n >= system::SYSINFO_HEADER_SIZE, "sysinfo returned {n} bytes");
-    toyos_abi::syscall::SysinfoHeader::decode(&buf).memory_total
-}
-
 /// The mode a window toggles into when it is maximized by button or chord.
 fn toggled(mode: WindowMode) -> WindowMode {
     if mode == WindowMode::Normal {

system/fileserver/src/cache.rs

@@ -8,8 +8,13 @@
 //! blocks flushes first, and is refused whole when that flush is, so memory
 //! bounds the write-back and not the other way round.
 //!
-//! **Clean blocks are kept up to [`CLEAN_LIMIT`]** and the oldest-read goes
-//! first; a dirty block is never dropped.
+//! **Clean blocks are kept up to the limit the cache is made with** and the
+//! oldest-read goes first; a dirty block is never dropped. A block read in
+//! is kept only if the allocator gives it and its place in the eviction order
+//! room: refused, the read is answered from the disk's bytes and nothing is
+//! cached. A block a flush cleans whose place is refused is dropped, since the
+//! disk holds it. The block map's node, a small allocation every few inserts,
+//! is the one part of keeping a block read in that still cannot be refused.
 //!
 //! **Several tasks may use it at once**, and the disk never has two requests
 //! for one block in flight when either is a write, which the block service's
@@ -44,9 +49,6 @@ impl Disk for diskserver::disk::Served {
     }
 }
 
-/// Clean blocks kept: 64 MiB.
-pub const CLEAN_LIMIT: usize = 16 * 1024;
-
 /// Dirty blocks held before the cache flushes by itself: 32 MiB.
 pub const DIRTY_LIMIT: usize = 8 * 1024;
 
@@ -69,6 +71,8 @@ struct Fetch {
 
 #[derive(Default)]
 struct Inner {
+    /// The most clean blocks kept.
+    clean_limit: usize,
     slots: BTreeMap<u64, Slot>,
     /// Clean blocks in the order they were read or cleaned; stale entries are
     /// skipped when popped.
@@ -104,8 +108,9 @@ pub struct Counts {
 }
 
 impl<D: Disk> Cache<D> {
-    pub fn new(disk: D) -> Self {
-        Self { disk, inner: RefCell::new(Inner { unflushed: true, ..Inner::default() }) }
+    /// A cache over `disk` keeping at most `clean_limit` clean blocks.
+    pub fn new(disk: D, clean_limit: usize) -> Self {
+        Self { disk, inner: RefCell::new(Inner { clean_limit, unflushed: true, ..Inner::default() }) }
     }
 
     /// The disk, once nothing else holds the cache: what it holds is what the
@@ -171,8 +176,12 @@ impl<D: Disk> Cache<D> {
                     Some(slot) => put(i + k, &slot.data),
                     None => {
                         put(i + k, data);
-                        inner.slots.insert(at, Slot { data: Box::new(*data), dirty: false, version: 0 });
-                        inner.order.push_back(at);
+                        if inner.order.try_reserve(1).is_ok() {
+                            if let Some(data) = boxed(data) {
+                                inner.slots.insert(at, Slot { data, dirty: false, version: 0 });
+                                inner.order.push_back(at);
+                            }
+                        }
                     }
                 }
             }
@@ -202,6 +211,14 @@ impl<D: Disk> Cache<D> {
     }
 }
 
+/// `data` in a block of its own, or `None` where the allocator has no room.
+fn boxed(data: &Block) -> Option<Box<Block>> {
+    let mut bytes = Vec::new();
+    bytes.try_reserve_exact(BLOCK).ok()?;
+    bytes.extend_from_slice(data);
+    Some(bytes.into_boxed_slice().try_into().expect("a block's bytes"))
+}
+
 /// The one flush running, until this is dropped: when it finishes, is refused
 /// or is abandoned.
 struct Flushing<'a, D: Disk>(&'a Cache<D>);
@@ -298,7 +315,13 @@ impl<D: Disk> BlockDev for Cache<D> {
                 if slot.version == version {
                     slot.dirty = false;
                     inner.dirty -= 1;
-                    inner.order.push_back(block);
+                    // The disk holds it now, so a block whose place in the
+                    // eviction order is refused is dropped rather than kept.
+                    if inner.order.try_reserve(1).is_ok() {
+                        inner.order.push_back(block);
+                    } else {
+                        inner.slots.remove(&block);
+                    }
                 }
             }
             i += run;
@@ -313,17 +336,17 @@ impl<D: Disk> BlockDev for Cache<D> {
 }
 
 impl Inner {
-    /// Drop the oldest clean blocks past [`CLEAN_LIMIT`].
+    /// Drop the oldest clean blocks past the limit.
     fn evict(&mut self) {
-        while self.slots.len() - self.dirty > CLEAN_LIMIT {
+        while self.slots.len() - self.dirty > self.clean_limit {
             let Some(block) = self.order.pop_front() else { break };
             if self.slots.get(&block).is_some_and(|s| !s.dirty) {
                 self.slots.remove(&block);
             }
         }
         // A queue of mostly stale entries is compacted, so it stays the size
         // of what it orders.
-        if self.order.len() > 4 * (CLEAN_LIMIT + DIRTY_LIMIT) {
+        if self.order.len() > 4 * (self.clean_limit + DIRTY_LIMIT) {
             let slots = &self.slots;
             self.order.retain(|b| slots.get(b).is_some_and(|s| !s.dirty));
         }
@@ -381,6 +404,9 @@ mod tests {
     use crate::ram::Ram;
     use toyos_blockdev::{block_on_ready, Executor};
 
+    /// The clean blocks every cache here keeps.
+    const CLEAN_LIMIT: usize = 4096;
+
     /// A disk that counts what reaches it, and refuses its flushes while
     /// `refusing_flush` is set.
     struct Counting {
@@ -427,7 +453,7 @@ mod tests {
     impl Disk for Counting {}
 
     fn cache() -> Cache<Counting> {
-        Cache::new(Counting::new(1024))
+        Cache::new(Counting::new(1024), CLEAN_LIMIT)
     }
 
     fn read_one<D: Disk>(c: &Cache<D>, block: u64) -> Result<Block, Error> {
@@ -505,7 +531,7 @@ mod tests {
 
     #[test]
     fn clean_blocks_are_bounded_and_dirty_ones_are_kept() {
-        let c = Cache::new(Counting::new((CLEAN_LIMIT + 64) as u64));
+        let c = Cache::new(Counting::new((CLEAN_LIMIT + 64) as u64), CLEAN_LIMIT);
         block_on_ready(c.write(0, &[[1; BLOCK]])).unwrap();
         for b in 1..(CLEAN_LIMIT as u64 + 32) {
             read_one(&c, b).unwrap();
@@ -530,7 +556,7 @@ mod tests {
     #[test]
     fn every_evicted_clean_block_reads_back_off_the_disk_byte_for_byte() {
         let blocks = (CLEAN_LIMIT + 2048) as u64;
-        let c = Cache::new(Counting::new(blocks));
+        let c = Cache::new(Counting::new(blocks), CLEAN_LIMIT);
         for b in 0..blocks {
             block_on_ready(c.write(b, &[pattern(b)])).unwrap();
         }
@@ -574,7 +600,7 @@ mod tests {
     #[test]
     fn at_the_dirty_limit_a_refused_flush_refuses_the_write_and_the_cache_grows_no_larger() {
         let limit = DIRTY_LIMIT as u64;
-        let c = Cache::new(Refusing { ram: Ram::new(limit + 16), refusing: Cell::new(false) });
+        let c = Cache::new(Refusing { ram: Ram::new(limit + 16), refusing: Cell::new(false) }, CLEAN_LIMIT);
         for b in 0..limit {
             block_on_ready(c.write(b, &[[1; BLOCK]])).unwrap();
         }
@@ -686,7 +712,7 @@ mod tests {
 
     #[test]
     fn a_block_written_while_a_flush_carries_its_older_copy_stays_dirty() {
-        let c = Cache::new(Held::new(16));
+        let c = Cache::new(Held::new(16), CLEAN_LIMIT);
         block_on_ready(c.write(3, &[[1; BLOCK]])).unwrap();
         c.disk.open.set(false);
         let exec = Executor::new();
@@ -705,7 +731,7 @@ mod tests {
     /// whose landing order nothing fixes.
     #[test]
     fn a_flush_asked_during_another_waits_and_then_writes_the_newer_copy() {
-        let c = Cache::new(Held::new(16));
+        let c = Cache::new(Held::new(16), CLEAN_LIMIT);
         block_on_ready(c.write(3, &[[1; BLOCK]])).unwrap();
         c.disk.open.set(false);
         let exec = Executor::new();
@@ -727,7 +753,7 @@ mod tests {
     /// block, and the cache keeps the write's copy, not the fetch's.
     #[test]
     fn a_flush_waits_for_a_fetch_of_its_block_and_the_write_is_kept() {
-        let c = Cache::new(Held::new(16));
+        let c = Cache::new(Held::new(16), CLEAN_LIMIT);
         block_on_ready(c.disk.ram.write(5, &[[1; BLOCK]])).unwrap();
         c.disk.open.set(false);
         let exec = Executor::new();