From c81f79fb679e75ab642d710189a85348c7157652 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 20:10:59 +0200 Subject: [PATCH 1/4] issues: the all-CPU build waits on the frame allocator's lock, for the 2 MiB windows A diagnosis boot a bench split each build-shaped child's CPU by system call, fault, lock spin and shootdown wait, at j1 and one child a CPU. The growth in the children's CPU is 59 % (T14 j8) and 72 % (AMD j16) their four 2 MiB directory windows' SHM_CREATE, and 12 and 14 % their faults; the frame allocator's one lock is 99 % of every contended lock wait, 1.06 and 7.1 CPUs spinning for the whole round, and no shootdown was waited on. The spawn record carries the attribution in place of "unmeasured", and names the frame allocator and the window as the parallel build's owners. The frame allocator's record and the window's record carry their parts and a proposal each: batched draws under one hold; a window as large as what it still carries, or a region's frames drawn at first touch. Co-Authored-By: Claude Opus 5.5 --- ...osts-each-process-a-two-mebibyte-window.md | 33 +++++++++++++++-- ...pmm-state-shares-lines-every-cpu-writes.md | 25 +++++++++++++ ...cess-costs-toyos-up-to-147-times-linuxs.md | 35 +++++++++++++++++-- 3 files changed, 88 insertions(+), 5 deletions(-) diff --git a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md index 83d6310ddfa..375c3d22d04 100644 --- a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md +++ b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md @@ -4,7 +4,7 @@ kind: tooling opened: 2026-09-27 --- -# A served directory costs each process a 2 MiB window, and nobody measured it +# A served directory costs each process a 2 MiB window Every process that names a file under a served directory connects to that directory's server once and lends it a `toyos::fs::WINDOW_BYTES` region @@ -13,7 +13,34 @@ touched, held for the process's life, and mapped into the server too. A desktop process that touches `/home`, `/config`, `/state` and `/apps` holds 8 MiB of windows; fsd maps up to `MAX_SERVED` of them. Whether the window's size is paid for by the reads that fill it, and what the desktop's processes -hold in windows together, has not been measured. +hold in windows together, has not been measured. The size's stated reason, +that 2 MiB is the size shared memory comes in, is gone: a region is whole +4 KiB pages. + +**Its creation is most of what the all-CPU build loses** (2026-11-11, +[evidence](EVIDENCE); the attribution is in +`issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). A build +child makes four `SHM_CREATE`s beside its four directory connections, by +inference from the SDK their windows. Each is 512 frames drawn one at a time +under the frame allocator's one lock and zeroed: 95 µs on the T14 and 133 µs +on the AMD bench with one child at a time, 17 and 19 % of a child's CPU, and +with one child a CPU 59 % (T14) and 72 % (AMD) of the growth in the +children's CPU, most of it spun on that lock +(`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`). + +Proposed, not built, either of: + +- **The window is as large as what it still carries.** Once a served file's + reads and writes go through its paged object (PR #1258 and the work after + it), the window holds paths, small answers and the writes a holder's + admission refuses; it is cut to the largest of those, with a larger write + split. 64 KiB would be 16 frames a connection. +- **A region's frames are drawn at first touch.** Promised whole when it is + made, so it is still refused then and never at a touch, and each frame drawn + and zeroed by the first fault or kernel copy that reaches its page, so a + window costs what its requests touch. It changes every region the kernel + lends but a device's run, which is laid out apart. **Exit**: the resident memory of a desktop boot's processes with and without -their windows, measured, and the window's size justified from it or cut. +their windows, measured, and the window's size justified from it or cut; and +a build child's window cost read again after the change. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index 274b1de62c5..67455f8b23e 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -52,6 +52,31 @@ benches. A fourth, `IN_PASS`, was measured with them and now sits in its CPU's own record (`kernel/src/cpu_local.rs`), a line no other CPU writes. +**The frame allocator's lock is what the all-CPU build waits on** +(2026-11-11, [evidence](EVIDENCE); the attribution is in +`issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). Under the +build's 256 children, one a CPU, it is 99 % of every contended lock wait on +both benches: 418,000 contended acquisitions and 243 ms of spin a round on +the T14 (`j8`), 1.06 CPUs spinning for the whole round, and 605,000 and +2,348 ms on the AMD bench (`j16`), 7.1 of its 16 CPUs. 82 % of the spin is in +the children's `SHM_CREATE`s, which draw each frame of a 2 MiB window under its +own hold (`SharedMemObject::create`), and 13 to 14 % in their faults, which draw +each frame of a populated run under its own hold (`owned_frame`). The next +contended lock waits under a hundredth as long. + +Proposed, not built: **a draw of many frames takes the lock once.** A +promised batch (`Frames::alloc_promised` for `n`, handing each to the caller +as `alloc_held` does) serves a shared-memory region's frames and a fault's +whole run, the way frees already go back, a region's drop through +`pmm::free_all` and a space's teardown through `free_drawn`. That turns 512 +acquisitions a window and up to 16 a fault into one each, keeps promises and +ledgers exactly as they are, and is host-testable in `kernel::frames`. A +per-CPU cache of frames in front of the lock (Linux's per-CPU page lists) +would take single draws off it too, but every promise must stay backed by a +frame free now, so a draw that finds the shared pool empty while another +CPU's cache holds frames has to reach them; that is worth building only if +the batched build still waits on the lock. + **Exit**: that attribution, posted, and read again after the change; and each of the three either written per CPU on a line no other CPU writes, or kept with its refusal reason at its site. diff --git a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md index cc7aa1a1334..7fa00833840 100644 --- a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md +++ b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md @@ -62,7 +62,6 @@ spawner now reads a served image whole, through the pager, into fresh memory (`issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md`), and shared text is stage 5 of `issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`. -The parallel build's loss where the serial build wins is unmeasured. `issues/idle-cpus-enter-the-c-states-cst-names.md` records the T14's `j1` under deep idle. @@ -97,9 +96,41 @@ behind: `posix_spawn_wait.std` and `process.spawn_wait` are unresolved against Linux's 60% spread, and the rest are ahead. The all-CPU build is behind on both, 2.22x and 2.57x, where the one-at-a-time build is ahead on both. +**Where the all-CPU build's time goes, 2026-11-11** ([evidence](EVIDENCE)): one +diagnosis boot a bench at main 34cb4c918 with measurement commits, the build's +256 children at `j1` and at one a CPU, three rounds each, a child's CPU split by +system call, fault, lock spin and shootdown wait. A child does the same work at +every width (53 system calls, 90 faults), and its CPU grows from 2.28 to 5.14 ms +on the T14 (`j8`) and from 2.83 to 15.5 ms on the AMD bench (`j16`). Of that +growth over the 256 children: + +| | T14 `j1` -> `j8` | AMD `j1` -> `j16` | +|---|---|---| +| children's CPU, ms a round | 583 -> 1,316 | 724 -> 3,977 | +| `SHM_CREATE`, of the growth | 432 ms, 59 % (201 ms spin) | 2,342 ms, 72 % (1,934 ms spin) | +| faults, of the growth | 91 ms, 12 % (33 ms spin) | 456 ms, 14 % (324 ms spin) | +| every other system call | 28 ms | 57 ms | +| user time and exit, the rest | 182 ms | 400 ms | +| lock spin across the machine, a CPU's worth | 1.06 of 8 | 7.1 of 16 | +| of the contended lock waits, the frame allocator's one lock | 99 % | 99.7 % | +| shootdown waits, issued or served | none | none | + +The four `SHM_CREATE`s a child makes are, by inference from the SDK, the +2 MiB windows of its four directory connections, each 512 frames drawn one at a +time under the frame allocator's one lock and zeroed: 95 µs a window on the T14 +and 133 µs on the AMD bench even at `j1`, 17 and 19 % of a child's CPU there. At +`j` the draws queue on that lock. The windows are +`issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md`, the +lock `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`. +The next contended lock, the process table's in `process_data`, waits 1.2 ms a +round on the T14 and 3.7 ms on the AMD bench; the kernel heap's 0.7 and 2.7 ms. +The rest's growth is not split further: the lock spin outside the children's +system calls and faults, which bounds the exit path's, is 10 and 77 ms. + **Exit**: these rows within Linux's on both benches in a labelled current-state comparison, or each remaining difference measured to its parts and justified. **Owner**: process creation (`kernel/src/process.rs`, the SDK's spawn and -`sdk/libc/src/spawn.rs`) and, for the parallel build, the scheduler (`kernel/src/sched/`). +`sdk/libc/src/spawn.rs`) and, for the parallel build, the frame allocator +(`kernel/src/mm/pmm.rs`) and the file-call window (`sdk/toyos/src/fs.rs`). From a11cab8e6d07fd4d038b2425ebf37cf1fc4cd7d1 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 20:11:54 +0200 Subject: [PATCH 2/4] issues: the diagnosis records link their evidence Co-Authored-By: Claude Opus 5.5 --- ...served-directory-costs-each-process-a-two-mebibyte-window.md | 2 +- ...cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md | 2 +- issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md index 375c3d22d04..a1709aa01a5 100644 --- a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md +++ b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md @@ -18,7 +18,7 @@ that 2 MiB is the size shared memory comes in, is gone: a region is whole 4 KiB pages. **Its creation is most of what the all-CPU build loses** (2026-11-11, -[evidence](EVIDENCE); the attribution is in +[evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). A build child makes four `SHM_CREATE`s beside its four directory connections, by inference from the SDK their windows. Each is 512 frames drawn one at a time diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index 67455f8b23e..187a91e8b21 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -53,7 +53,7 @@ A fourth, `IN_PASS`, was measured with them and now sits in its CPU's own record (`kernel/src/cpu_local.rs`), a line no other CPU writes. **The frame allocator's lock is what the all-CPU build waits on** -(2026-11-11, [evidence](EVIDENCE); the attribution is in +(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). Under the build's 256 children, one a CPU, it is 99 % of every contended lock wait on both benches: 418,000 contended acquisitions and 243 ms of spin a round on diff --git a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md index 7fa00833840..cdd664557fa 100644 --- a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md +++ b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md @@ -96,7 +96,7 @@ behind: `posix_spawn_wait.std` and `process.spawn_wait` are unresolved against Linux's 60% spread, and the rest are ahead. The all-CPU build is behind on both, 2.22x and 2.57x, where the one-at-a-time build is ahead on both. -**Where the all-CPU build's time goes, 2026-11-11** ([evidence](EVIDENCE)): one +**Where the all-CPU build's time goes, 2026-11-11** ([evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag)): one diagnosis boot a bench at main 34cb4c918 with measurement commits, the build's 256 children at `j1` and at one a CPU, three rounds each, a child's CPU split by system call, fault, lock spin and shootdown wait. A child does the same work at From 9cbcbf628a4ed05dc1b380eeabbbc16c37ad03a5 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 20:43:21 +0200 Subject: [PATCH 3/4] Review r1 of #1265: the orchestrator's decisions as decisions, and the figures as measured The three follow-ups are decided, in order: a draw of many frames takes the allocator's lock once, with per-CPU caches held until it is measured; a shared-memory region's frames are drawn at first touch, promised when it is made, with a device's run drawn whole, which gets a record of its own (a defect, assigned); and the window's size waits for the file-call work. Figures corrected: on the T14 `SHM_CREATE`'s growth is 47 % lock spin (83 % on the AMD bench), not most; 95 us is one window, and 17 and 19 % are a child's four; the allocator lock's own wait is 1.05 CPUs on the T14. The 2 MiB size and "most of what the build loses" are labelled as inferences, the probes' slowing of the round is stated, and the T14's lock spin is a third of its children's growth. Co-Authored-By: Claude Opus 5.5 --- ...osts-each-process-a-two-mebibyte-window.md | 41 ++++++++--------- ...-and-zeroes-every-frame-when-it-is-made.md | 34 ++++++++++++++ ...pmm-state-shares-lines-every-cpu-writes.md | 46 ++++++++++--------- ...cess-costs-toyos-up-to-147-times-linuxs.md | 23 ++++++++-- 4 files changed, 97 insertions(+), 47 deletions(-) create mode 100644 issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md diff --git a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md index a1709aa01a5..b1f69e7929d 100644 --- a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md +++ b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md @@ -17,30 +17,29 @@ hold in windows together, has not been measured. The size's stated reason, that 2 MiB is the size shared memory comes in, is gone: a region is whole 4 KiB pages. -**Its creation is most of what the all-CPU build loses** (2026-11-11, -[evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in +**Creating it is most of what the all-CPU build's children add in CPU** +(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). A build child makes four `SHM_CREATE`s beside its four directory connections, by inference from the SDK their windows. Each is 512 frames drawn one at a time -under the frame allocator's one lock and zeroed: 95 µs on the T14 and 133 µs -on the AMD bench with one child at a time, 17 and 19 % of a child's CPU, and -with one child a CPU 59 % (T14) and 72 % (AMD) of the growth in the -children's CPU, most of it spun on that lock -(`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`). +under the frame allocator's one lock and zeroed: 95 µs a window on the T14 and +133 µs on the AMD bench with one child at a time, where a child's four are 17 +and 19 % of its CPU. With one child a CPU, `SHM_CREATE` is 59 % (T14) and 72 % +(AMD) of the growth in the children's CPU; of that, 47 % (T14) and 83 % (AMD) +is spin on that lock +(`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), +and the T14's other 231 ms is not split. How much of the build's wall-clock +loss against Linux this is has not been measured; that it is most of it is an +inference. -Proposed, not built, either of: - -- **The window is as large as what it still carries.** Once a served file's - reads and writes go through its paged object (PR #1258 and the work after - it), the window holds paths, small answers and the writes a holder's - admission refuses; it is cut to the largest of those, with a larger write - split. 64 KiB would be 16 frames a connection. -- **A region's frames are drawn at first touch.** Promised whole when it is - made, so it is still refused then and never at a touch, and each frame drawn - and zeroed by the first fault or kernel copy that reaches its page, so a - window costs what its requests touch. It changes every region the kernel - lends but a device's run, which is laid out apart. +**Decided** (the orchestrator, 2026-11-11), third of three: the window's size +waits for the file-call work. Once a served file's reads, writes and +attributes go through its paged object (PR #1258 and the work after it), the +window carries paths, small answers and the writes a holder's admission +refuses, and is cut to what that needs. Before that, its creation costs what +its region's first-touch draw makes it +(`issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md`). **Exit**: the resident memory of a desktop boot's processes with and without -their windows, measured, and the window's size justified from it or cut; and -a build child's window cost read again after the change. +their windows, measured, and the window's size justified from it or cut once +the file-call work carries reads and writes on the paged object. diff --git a/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md b/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md new file mode 100644 index 00000000000..c4cd1453b39 --- /dev/null +++ b/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md @@ -0,0 +1,34 @@ +--- +status: assigned +kind: defect +opened: 2026-11-11 +--- + +# A shared-memory region draws and zeroes every frame when it is made + +`SharedMemObject::create` (`kernel/src/object/shm.rs`) draws every frame of a +region and zeroes it before the call returns, so a region costs its whole size +in time and memory however little of it is touched, where a process's private +memory takes a frame at each page's first touch. A served directory's 2 MiB +window is the largest such region on the spawn path: 95 µs a window on the T14 +and 133 µs on the AMD bench with one build child at a time, a child's four 17 +and 19 % of its CPU, and most of what the all-CPU build's children add in CPU +(`issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md`, +[evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag)). + +**Decided** (the orchestrator, 2026-11-11), second of three after the batched +draw (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`): +a region's frames are drawn at its pages' first touch. The whole region is +promised when it is made, so it is refused then and never at a touch; a +device's run stays drawn whole, as a device needs. It is one mechanism with +the kernel's zeroed frame for a touched hole of a MEMORY volume's paged +object, which mapped paged objects bring (7b in +`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`), and +is built on 7b's object mappings (#1259). The module header's "a fault on a +mapping of one never allocates" changes with it. + +**Assigned**: the spawn-path work, after the batched draw lands. + +**Exit**: a region's creation draws no frame and a first touch draws one, +promised whole at creation and refused only there; and a build child's +`SHM_CREATE` cost read again. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index 187a91e8b21..92535fbf173 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -56,27 +56,29 @@ record (`kernel/src/cpu_local.rs`), a line no other CPU writes. (2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). Under the build's 256 children, one a CPU, it is 99 % of every contended lock wait on -both benches: 418,000 contended acquisitions and 243 ms of spin a round on -the T14 (`j8`), 1.06 CPUs spinning for the whole round, and 605,000 and -2,348 ms on the AMD bench (`j16`), 7.1 of its 16 CPUs. 82 % of the spin is in -the children's `SHM_CREATE`s, which draw each frame of a 2 MiB window under its -own hold (`SharedMemObject::create`), and 13 to 14 % in their faults, which draw -each frame of a populated run under its own hold (`owned_frame`). The next -contended lock waits under a hundredth as long. +both benches: 418,000 contended acquisitions and 243 ms of wait a round on the +T14 (`j8`), 1.05 of its 8 CPUs for the whole round, and 605,000 and 2,348 ms on +the AMD bench (`j16`), 7.1 of its 16. 82 % of the spin is in the children's +`SHM_CREATE`s, which draw each frame of what are, by inference from the SDK, +their 2 MiB directory windows under its own hold (`SharedMemObject::create`), +and 13 to 14 % in their faults, which draw each frame of a populated run under +its own hold (`owned_frame`). The next contended lock waits under a hundredth +as long. On the T14 the spin is a third of the children's growth in CPU, so +the lock bounds less of the build there than on the AMD bench, where it is 72 %. -Proposed, not built: **a draw of many frames takes the lock once.** A -promised batch (`Frames::alloc_promised` for `n`, handing each to the caller -as `alloc_held` does) serves a shared-memory region's frames and a fault's -whole run, the way frees already go back, a region's drop through -`pmm::free_all` and a space's teardown through `free_drawn`. That turns 512 -acquisitions a window and up to 16 a fault into one each, keeps promises and -ledgers exactly as they are, and is host-testable in `kernel::frames`. A -per-CPU cache of frames in front of the lock (Linux's per-CPU page lists) -would take single draws off it too, but every promise must stay backed by a -frame free now, so a draw that finds the shared pool empty while another -CPU's cache holds frames has to reach them; that is worth building only if -the batched build still waits on the lock. +**Decided** (the orchestrator, 2026-11-11), first of three: **a draw of many +frames takes the lock once.** A promised batch serves a shared-memory region's +frames and a fault's whole run, the way frees already go back, a region's drop +through `pmm::free_all` and a space's teardown through `free_drawn`: one +acquisition a window or a run instead of one a frame, with promises and ledgers +moving as for as many single draws. Per-CPU frame caches in front of the lock +are held until the batched build is measured; they would take single draws off +it too, but every promise must stay backed by a frame free now, so a draw that +finds the shared pool empty while another CPU's cache holds frames has to reach +them. The second, a region's frames drawn at first touch, and the third, the +window's size, are in their own records (the spawn record names them). -**Exit**: that attribution, posted, and read again after the change; and -each of the three either written per CPU on a line no other CPU writes, or -kept with its refusal reason at its site. +**Exit**: that attribution, posted, and read again after the change; the +all-CPU build's lock waits read again with the batched draw; and each of the +three either written per CPU on a line no other CPU writes, or kept with its +refusal reason at its site. diff --git a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md index cdd664557fa..e2f61a16a6e 100644 --- a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md +++ b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md @@ -99,7 +99,9 @@ one-at-a-time build is ahead on both. **Where the all-CPU build's time goes, 2026-11-11** ([evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag)): one diagnosis boot a bench at main 34cb4c918 with measurement commits, the build's 256 children at `j1` and at one a CPU, three rounds each, a child's CPU split by -system call, fault, lock spin and shootdown wait. A child does the same work at +system call, fault, lock spin and shootdown wait. The probes slow the round +(the AMD bench's `j16` round took 331 ms against 313 to 316 ms without them), +so the shares are of a slightly slowed machine. A child does the same work at every width (53 system calls, 90 faults), and its CPU grows from 2.28 to 5.14 ms on the T14 (`j8`) and from 2.83 to 15.5 ms on the AMD bench (`j16`). Of that growth over the 256 children: @@ -118,8 +120,11 @@ growth over the 256 children: The four `SHM_CREATE`s a child makes are, by inference from the SDK, the 2 MiB windows of its four directory connections, each 512 frames drawn one at a time under the frame allocator's one lock and zeroed: 95 µs a window on the T14 -and 133 µs on the AMD bench even at `j1`, 17 and 19 % of a child's CPU there. At -`j` the draws queue on that lock. The windows are +and 133 µs on the AMD bench even at `j1`, where a child's four are 17 and 19 % of +its CPU. At `j` the draws queue on that lock. Lock spin across the machine +grows by 244 ms on the T14, a third of the children's growth, and by 2,354 ms +on the AMD bench, 72 %; the rest of the growth, user time and exit among it, +is not lock spin. The windows are `issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md`, the lock `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`. The next contended lock, the process table's in `process_data`, waits 1.2 ms a @@ -127,10 +132,20 @@ round on the T14 and 3.7 ms on the AMD bench; the kernel heap's 0.7 and 2.7 ms. The rest's growth is not split further: the lock spin outside the children's system calls and faults, which bounds the exit path's, is 10 and 77 ms. +Decided, in this order (the orchestrator, 2026-11-11): a draw of many frames +takes the allocator's lock once, with per-CPU frame caches held until that is +measured (the frame allocator's record); a shared-memory region's frames are +drawn at its pages' first touch, promised when it is made, and a device's run +stays drawn whole +(`issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md`); +and the window's size waits for the file-call work (the window's record). The +rest's growth is read again after the first two. + **Exit**: these rows within Linux's on both benches in a labelled current-state comparison, or each remaining difference measured to its parts and justified. **Owner**: process creation (`kernel/src/process.rs`, the SDK's spawn and `sdk/libc/src/spawn.rs`) and, for the parallel build, the frame allocator -(`kernel/src/mm/pmm.rs`) and the file-call window (`sdk/toyos/src/fs.rs`). +(`kernel/src/mm/pmm.rs`), the shared-memory object (`kernel/src/object/shm.rs`) +and the file-call window (`sdk/toyos/src/fs.rs`). From 3e79317c886ba52f97e8ee5b949e1065d18ec2fc Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 21:45:34 +0200 Subject: [PATCH 4/4] Review r2 of #1265: the first-touch decision as decided, and its figures as SHM_CREATE's The first-touch record states the orchestrator's decision as made: one draw, `pmm::alloc_promised`, for a region's hole (`SharedMemObject::page()`, into the region's slot) and a MEMORY volume's hole (into the pager's index under its lock), two stores because one is revoked and the other never; and an inbox's rings stay drawn whole beside a device's run, since the kernel writes them in interrupt context, in the decision and its exit. The build plan (on #1259) leaves the decision. Its figures are `SHM_CREATE`'s, and that those are the windows is labelled an inference. Every record links the evidence whose script sorts deterministically. Co-Authored-By: Claude Opus 5.5 --- ...osts-each-process-a-two-mebibyte-window.md | 2 +- ...-and-zeroes-every-frame-when-it-is-made.md | 32 +++++++++++-------- ...pmm-state-shares-lines-every-cpu-writes.md | 2 +- ...cess-costs-toyos-up-to-147-times-linuxs.md | 2 +- 4 files changed, 22 insertions(+), 16 deletions(-) diff --git a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md index b1f69e7929d..65b0b6efb93 100644 --- a/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md +++ b/issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md @@ -18,7 +18,7 @@ that 2 MiB is the size shared memory comes in, is gone: a region is whole 4 KiB pages. **Creating it is most of what the all-CPU build's children add in CPU** -(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in +(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/a22307591cfdefcf9fa1fef2ae5d1de79b926e9e/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). A build child makes four `SHM_CREATE`s beside its four directory connections, by inference from the SDK their windows. Each is 512 frames drawn one at a time diff --git a/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md b/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md index c4cd1453b39..bfc1d1bc104 100644 --- a/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md +++ b/issues/a-shared-memory-region-draws-and-zeroes-every-frame-when-it-is-made.md @@ -9,26 +9,32 @@ opened: 2026-11-11 `SharedMemObject::create` (`kernel/src/object/shm.rs`) draws every frame of a region and zeroes it before the call returns, so a region costs its whole size in time and memory however little of it is touched, where a process's private -memory takes a frame at each page's first touch. A served directory's 2 MiB -window is the largest such region on the spawn path: 95 µs a window on the T14 -and 133 µs on the AMD bench with one build child at a time, a child's four 17 -and 19 % of its CPU, and most of what the all-CPU build's children add in CPU +memory takes a frame at each page's first touch. On the spawn path it is a +build child's `SHM_CREATE`: 95 µs a call on the T14 and 133 µs on the AMD bench +with one child at a time, a child's four 17 and 19 % of its CPU, and most of +what the all-CPU build's children add in CPU; by inference from the SDK, those +four are the 2 MiB windows of its directory connections (`issues/a-served-directory-costs-each-process-a-two-mebibyte-window.md`, -[evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag)). +[evidence](https://github.com/ToyOSOrg/evidence/tree/a22307591cfdefcf9fa1fef2ae5d1de79b926e9e/spawnperf-diag)). **Decided** (the orchestrator, 2026-11-11), second of three after the batched draw (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`): a region's frames are drawn at its pages' first touch. The whole region is -promised when it is made, so it is refused then and never at a touch; a -device's run stays drawn whole, as a device needs. It is one mechanism with -the kernel's zeroed frame for a touched hole of a MEMORY volume's paged -object, which mapped paged objects bring (7b in -`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`), and -is built on 7b's object mappings (#1259). The module header's "a fault on a -mapping of one never allocates" changes with it. +promised when it is made, so it is refused then and never at a touch. Two +kinds stay drawn whole when they are made: a device's run, as a device needs, +and an inbox's rings, which the kernel writes in interrupt context, where no +frame may be drawn. One draw serves both kernel-made zeroed frames for a +touched hole, `pmm::alloc_promised`: a region's page, drawn by +`SharedMemObject::page()` into the region's own slot for that page, and a +touched hole of a MEMORY volume's paged object (mapped paged objects, 7b in +`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`), +drawn into the pager's index under its lock. A region's slot is never revoked +and a pager's page is, so the two keep their own stores. The module header's +"a fault on a mapping of one never allocates" changes with it. **Assigned**: the spawn-path work, after the batched draw lands. **Exit**: a region's creation draws no frame and a first touch draws one, -promised whole at creation and refused only there; and a build child's +promised whole at creation and refused only there, but for a device's run and +an inbox's rings, which are drawn whole when made; and a build child's `SHM_CREATE` cost read again. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index 92535fbf173..95879749e97 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -53,7 +53,7 @@ A fourth, `IN_PASS`, was measured with them and now sits in its CPU's own record (`kernel/src/cpu_local.rs`), a line no other CPU writes. **The frame allocator's lock is what the all-CPU build waits on** -(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag); the attribution is in +(2026-11-11, [evidence](https://github.com/ToyOSOrg/evidence/tree/a22307591cfdefcf9fa1fef2ae5d1de79b926e9e/spawnperf-diag); the attribution is in `issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md`). Under the build's 256 children, one a CPU, it is 99 % of every contended lock wait on both benches: 418,000 contended acquisitions and 243 ms of wait a round on the diff --git a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md index e2f61a16a6e..2edc41bfd02 100644 --- a/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md +++ b/issues/spawning-a-process-costs-toyos-up-to-147-times-linuxs.md @@ -96,7 +96,7 @@ behind: `posix_spawn_wait.std` and `process.spawn_wait` are unresolved against Linux's 60% spread, and the rest are ahead. The all-CPU build is behind on both, 2.22x and 2.57x, where the one-at-a-time build is ahead on both. -**Where the all-CPU build's time goes, 2026-11-11** ([evidence](https://github.com/ToyOSOrg/evidence/tree/36117dda5c9a1a3803745c45785cc8243c5b5d51/spawnperf-diag)): one +**Where the all-CPU build's time goes, 2026-11-11** ([evidence](https://github.com/ToyOSOrg/evidence/tree/a22307591cfdefcf9fa1fef2ae5d1de79b926e9e/spawnperf-diag)): one diagnosis boot a bench at main 34cb4c918 with measurement commits, the build's 256 children at `j1` and at one a CPU, three rounds each, a child's CPU split by system call, fault, lock spin and shootdown wait. The probes slow the round