diff --git a/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md b/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md deleted file mode 100644 index f5ea23ef061..00000000000 --- a/issues/a-64-mib-write-to-the-in-memory-data-volume-runs-its-file-server-out-of-memory.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -status: open -kind: defect -severity: [data-loss] -opened: 2026-10-21 ---- - -# A 64 MiB write to the in-memory DATA volume runs its file server out of memory - -On a boot with no DATA partition the DATA file server formats a volume in -memory (`system/fileserver/src/main.rs`, `RAM_BLOCKS`: 1 GiB, sparse). Writing -one 64 MiB file to it through std (`osbench files`, `fs.served.seq_write.4096`: -`File::create`, then 16,384 writes of 4 KiB) ends the server: - -``` -fileserver error] memory allocation of 4096 bytes failed -cpu0 kernel] exit: fileserver pid=4 code=134 cpu=11326ms peak=3724MB allocs=1863 ... 63=1864 ... -supervisor] supervisor: fileserver data (pid 4) ended; started again as pid 11 -``` - -QEMU TCG, 4 vCPUs, 4 GiB of memory, the benchcase image (`tests/images/benchcase.toml`) -built from `wt/toyos-bench2`. Before it, the same process had served a 16 MiB -file, 2,000 4 KiB writes into it and a few thousand small creates and stats. -The restarted server died the same way within its first 88 requests of the -1 MiB-call variant of the same write (`peak=3724MB`). The 1 GiB volume and a -cache bounded for clean blocks account for at most a little over 1 GiB; the -server's heap reached every byte the machine had. - -The growth is not isolated. The server's own allocations per write are a -block (`cache.write` of a dirty block, `data.rs`'s `write`) and an allocation -in the volume's metadata per page grown (`resolve_or_alloc_block` then; -`Mounted::allocate` now allocates a write's pages at once); a -metadata update that writes a whole btree node through the unbounded dirty -side of the cache per page would come to gigabytes for 16,384 pages, but that -is an inference from the code, not a measurement. - -On metal the server survived: all six osbench boots of round 2's -`e5573b7fe` (three each on the T14 and the AMD bench, both with more memory -than the 4 GiB guest) wrote the file and exited 0. One did not do it cleanly: -on the T14's first boot a `fs.served.seq_write.1048576` round of the same -write ran under 1 MB/s (`min=0`), and `files` took 80 s against the other -boots' 13 s, so the same write stalls on metal even where it completes. - -The server also refuses nothing on the way: every request answered until the -allocator aborted the process, and the client saw `the file server is gone`. - -The same write is now `osbench persist`'s `fs.disk.seq_write.*`, which -reaches the in-memory volume on a boot whose DATA is not designated. - -**Exit**: the server's memory for a 64 MiB file written to the in-memory volume -is bounded by the file's size plus a stated overhead, shown by the same write -under QEMU with its peak in the kernel's exit record; or the server refuses -the write that would exceed what it may hold, by name. - -Owner: the file server (`system/fileserver/`). diff --git a/issues/console-line-atomicity-loses-five-of-a-thousand-lines-on-ci.md b/issues/console-line-atomicity-loses-five-of-a-thousand-lines-on-ci.md deleted file mode 100644 index cdecf908335..00000000000 --- a/issues/console-line-atomicity-loses-five-of-a-thousand-lines-on-ci.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -status: open -kind: defect -severity: [data-loss] -opened: 2026-08-20 ---- - -# `console_line_atomicity` loses five of a thousand lines on CI, one guest per machine - -First sighting on the CI instrument: PR #166 run `32364721784`, `guest (10)`, -`writer A declared 1000 whole lines and the capture carries 995`, `ALONE: -GREEN` in the same job. The diff it rode on is an issues-and-prose audit, so -the tree is not a suspect. - -CI runs one guest per machine with `--jobs 1`, so whatever loses five of a -writer's thousand lines there is not host contention between suites — which -sharpens the question the test was built to ask rather than settling it: the -loss is inside one guest's own console path. - -Split out of `issues/parallel-tests-red-under-other-suites.md`, whose -rate table was never this test's — CI's single-guest-per-machine shards rule -out the contention shape that file is about. - -**Exit condition.** The lost lines' cause is fixed, shown against -`console_line_atomicity` as it stands at `1808fb8d` (its binary, the test -runner's `CONSOLE_JOBS` stdin and its harness arm), restored and green. -Owner: orchestrator. diff --git a/issues/fat-unlink-reallocate-leaks-a-cluster-under-load.md b/issues/fat-unlink-reallocate-leaks-a-cluster-under-load.md deleted file mode 100644 index 9de021efe8b..00000000000 --- a/issues/fat-unlink-reallocate-leaks-a-cluster-under-load.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -status: open -kind: defect -severity: [data-loss] -opened: 2026-08-25 ---- - -# `fat_backing_revoked`'s unlink-and-reallocate leaks a cluster under host load - -Seen once, dev host, in a full Fast suite (`cargo test`) run beside seven -`qemu-system-x86_64` guests belonging to two other worktrees on the same -14-core machine. The third-party checker refused the log volume after the -test's own scenario had completed: - -``` -FAIL fat_backing_revoked: the unlink-and-reallocate cycle left the log volume breaking the format: -1 cluster(s) from 134 are marked allocated and no directory entry reaches them -``` - -The harness re-ran it alone in the same session and it was green — `PASS -fat_backing_revoked (4s)`, with the test's own verdict line intact: *the victim -is gone from the volume, the 32768-byte file written into its place holds 0x5c -end to end on the host's own reader, and the checker is silent*. The rest of -the suite was 285 passed, 1 failed, 286 total. - -**This is not the setup failure the build tracker carried against this test's -name** (since closed: the setup asks again on a `WouldBlock` from -`fs::File::create` within a stated patience now, instead of panicking on the -first one). This one runs the whole scenario, produces the right bytes, -and leaves one cluster allocated with no directory entry reaching it — a FAT -the driver wrote and `toyos-fat32-check` refuses. The oracle is -Microsoft's fatgen103 checker and not this tree's own reader, which is what -makes the verdict worth acting on rather than re-running away. - -Load-coincident, and that is a reason to look rather than a reason to dismiss: -the unlink frees a chain and the create allocates into it, so the window where -one write's budget can expire between the two is exactly what a loaded host -widens. `kernel/CLAUDE.md`'s block-layer rule names the shape — a -`BudgetExpired` that discards its pages instead of re-enqueuing splits a FAT -the moment one copy's write outlives the other's budget — and a leaked cluster -is the same family with one FAT rather than two. - -**A second sighting, 2026-08-27, and it left two chains rather than one.** Dev -host, full fast tier twelve wide with a second worktree's suite on the machine, -on `wt/toyos-md1` at `03af5421` — a branch touching no kernel byte and nothing -under `toyos-fat32/`: - -``` -FAIL fat_backing_revoked: the unlink-and-reallocate cycle left the log volume -breaking the format: -1 cluster(s) from 44 are marked allocated and no directory entry reaches them -1 cluster(s) from 137 are marked allocated and no directory entry reaches them -``` - -Alone on the same tree minutes later: green, 5 s. Two lost chains, each one -cluster long, each from a different start — so the observation is one red beside -eleven other guests and still no rate. - -**Two lost chains admit a second mechanism the first sighting did not.** A chain -freed in the FAT and not in the directory, or a directory entry rewritten before -its old chain was released: which of those it is nobody has looked at. - -**This one leaves bytes behind, unlike a timing red**, so a reproduction keeps -its own artifact. The image is the evidence and the checker is the instrument. - -What is owed first is a reproduction that says which side leaks: the free-chain -walk of the unlink, or the allocation the create takes out of it. The suite's -own alone/loaded arms are the instrument, and -`tests/guest/rust-tests/src/bin/fat_backing_revoked.rs` plus -`tests/common/volumes.rs` are where the scenario and the checker call live. - -Found by the comment sweep on `wt/toyos-sw3`, whose diff is comments and -whitespace only and cannot have caused it. - -**Its test is deleted**, as a flaky test is: `3aad43b13` took -`fat_backing_revoked` out, and `git revert 3aad43b13` brings it back.