From 9bdbb86753e74a3d1bc2de113761ee5f4eaefa46 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 11:09:04 +0100 Subject: [PATCH 1/4] issues: a spawn onto a busy CPU can miss it for seconds at boot osbench's wake.onto_busy refused a partner's placement in three of PR #1074's nine T14 benchcase boots, and main 544f9cc3c spent 3.9 s of the same 5 s budget in one of its eight, around netstack's bring-up. Recorded for the scheduler track, with the boot counts. Co-Authored-By: Claude Opus 5.5 --- ...usy-cpu-can-miss-it-for-seconds-at-boot.md | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md diff --git a/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md b/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md new file mode 100644 index 00000000000..f45d0545faf --- /dev/null +++ b/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md @@ -0,0 +1,42 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A spawn onto a busy CPU can miss it for seconds at boot + +`osbench`'s `wake.onto_busy` (`tests/guest/osbench/src/wake.rs`) places +partner threads on one busy CPU through `place::spawn_on`. That spawns +threads until one first runs on the wanted CPU, within a 5 s budget per +partner. Usually the row takes 30–60 ms. Sometimes, early in the T14's +`benchcase` boot (22–28 s, around netstack's DHCP lease and its `ready` +line), one partner's placement runs for seconds: 138,904 threads in 5 s +and none first ran where it was wanted. The row refuses, so `osbench_sched` +exits 1. + +T14 `benchcase` boots, all on 2026-10-28: + +| image | boots | refused | slow but under 5 s | +|---|---|---|---| +| main `544f9cc3c`, and its diagnostic image | 8 | 0 | 1 (session 4 base2: 3.9 s, 23.156–27.025 s, netstack `ready` at 24.99 s) | +| PR #1074's heads and diagnostic images | 9 | 3 | — | + +The slow placement happens on `544f9cc3c`, so it predates #1074. Whether +#1074 makes a refusal likelier is not established (one-sided p ≈ 0.09). The +spawn path the row takes (`vm_reserve`, `vm_protect`, `thread_spawn`, and a +std channel) touches no pipe. The readbacks are in PR #1074's sessions 2–5 +(`tests/toyos.rs`'s `osbench` row). + +What makes spawn placement avoid one CPU for seconds is not known. +`issues/placement-has-no-migration-cost-or-wake-affinity.md` says a spawn +prefers a sleeping CPU on a free core. The wanted CPU holds the parked +partners, and netstack's bring-up is busy at the same time. + +Owner: the scheduler track, +`issues/placement-has-no-migration-cost-or-wake-affinity.md`. + +Exit: the cause is named from a boot log that shows the spawn's choices +while it misses, and either fixed or held by a bounded placement test. Then +`wake.onto_busy` measures in at least 20 consecutive `benchcase` boots of +main. From 4fc1e497e6e787e647ec73a797055a7be6f445cd Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 11:26:02 +0100 Subject: [PATCH 2/4] review r1: the counts name their sessions, the test its value, and the exit a bound Head boots are 3 of 10 with session 1's head-bench; the one-sided Fisher exact p is 0.15; the pipe-independence claim goes, base's slow row being the evidence; the slug says what was observed; the window is 22.7-29.0 s; place's rule and its two readings make the exit's first step specific. Co-Authored-By: Claude Opus 5.5 --- ...ding-parked-threads-for-seconds-at-boot.md | 50 +++++++++++++++++++ ...usy-cpu-can-miss-it-for-seconds-at-boot.md | 42 ---------------- 2 files changed, 50 insertions(+), 42 deletions(-) create mode 100644 issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md delete mode 100644 issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md diff --git a/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md new file mode 100644 index 00000000000..8b75dc3c7a9 --- /dev/null +++ b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md @@ -0,0 +1,50 @@ +--- +status: open +kind: defect +opened: 2026-10-28 +--- + +# A spawn can miss a CPU holding parked threads for seconds at boot + +`osbench`'s `wake.onto_busy` (`tests/guest/osbench/src/wake.rs`) places +partner threads on one CPU, where each partner parks. `place::spawn_on` +spawns threads until one first runs on that CPU, with a 5 s budget per +partner. The rows that measured took 32–63 ms. In some T14 `benchcase` +boots the row instead ran for seconds, inside about 22.7–29.0 s of boot, +the span in which netstack's DHCP lease and `ready` line also land. Three +rows refused, for example "partner_33_was_not_placed: +138904_threads_started_in_5s_and_none_first_ran_where_it_was_wanted", and +`osbench_sched` exited 1. + +T14 `benchcase` boots, all on 2026-10-28: + +| images | boots | refused | slow but measured | +|---|---|---|---| +| main `544f9cc3c`: PR #1074's session 1 base-bench, sessions 3–5 base1/base2, and the diagnostic base image | 8 | 0 | 1: session 4 base2's row ran 23.156–27.025 s (3.87 s), around netstack's `ready` at 24.99 s | +| PR #1074's heads: session 1 head-bench (`1f828f331`), session 2 p1 (`46848c491`), sessions 3 (`5e0cb52b0`), 4 (`cd42d13a4`) and 5 (`0a48bf8b4`) head1/head2, and the diagnostic images at `cd42d13a4` and `0a48bf8b4` | 10 | 3: session 2 p1 (23.963–28.967 s), session 5 head1 (22.705–27.710 s), the `0a48bf8b4` diagnostic image (22.802–27.803 s) | — | + +The slow row happens on `544f9cc3c`, so the behaviour predates #1074. Whether +#1074 raises the refusal rate is not established: 0 of 8 against 3 of 10, +one-sided Fisher exact p = 0.15. The readbacks are PR #1074's sessions 1–5. + +**What could cause it.** `place`'s rule (`kernel/src/sched/driver.rs`) takes +the CPUs that are answering, then the least `load`, then a CPU that is not +sleeping, then busy siblings, from a global rotating start. So: + +- If the wanted CPU stays awake or loaded while another answering CPU + sleeps, for example with netstack's bring-up running there, the rule never + chooses it. The scheduler would be following its specification, and what + is false is `place::spawn_on`'s premise that spawns rotate over the CPUs + (`STAGE`). The work would then belong to the bench, or to + `issues/no-test-can-hold-a-thread-on-a-named-cpu.md`, which is why the + bench stages at all. +- If the wanted CPU stops answering for seconds, that is a kernel defect, + and it is the scheduler's. + +Owner: the scheduler track, `issues/placement-has-no-migration-cost-or-wake-affinity.md`, +for the diagnosis that decides which of the two it is. + +Exit: one of the two alternatives is named from a boot log that records +the spawn's choices while it misses. Then it is fixed, or moved to the +bench's record, and every `wake.onto_busy` row in 20 consecutive main +`benchcase` boots completes within 250 ms. diff --git a/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md b/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md deleted file mode 100644 index f45d0545faf..00000000000 --- a/issues/a-spawn-onto-a-busy-cpu-can-miss-it-for-seconds-at-boot.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-28 ---- - -# A spawn onto a busy CPU can miss it for seconds at boot - -`osbench`'s `wake.onto_busy` (`tests/guest/osbench/src/wake.rs`) places -partner threads on one busy CPU through `place::spawn_on`. That spawns -threads until one first runs on the wanted CPU, within a 5 s budget per -partner. Usually the row takes 30–60 ms. Sometimes, early in the T14's -`benchcase` boot (22–28 s, around netstack's DHCP lease and its `ready` -line), one partner's placement runs for seconds: 138,904 threads in 5 s -and none first ran where it was wanted. The row refuses, so `osbench_sched` -exits 1. - -T14 `benchcase` boots, all on 2026-10-28: - -| image | boots | refused | slow but under 5 s | -|---|---|---|---| -| main `544f9cc3c`, and its diagnostic image | 8 | 0 | 1 (session 4 base2: 3.9 s, 23.156–27.025 s, netstack `ready` at 24.99 s) | -| PR #1074's heads and diagnostic images | 9 | 3 | — | - -The slow placement happens on `544f9cc3c`, so it predates #1074. Whether -#1074 makes a refusal likelier is not established (one-sided p ≈ 0.09). The -spawn path the row takes (`vm_reserve`, `vm_protect`, `thread_spawn`, and a -std channel) touches no pipe. The readbacks are in PR #1074's sessions 2–5 -(`tests/toyos.rs`'s `osbench` row). - -What makes spawn placement avoid one CPU for seconds is not known. -`issues/placement-has-no-migration-cost-or-wake-affinity.md` says a spawn -prefers a sleeping CPU on a free core. The wanted CPU holds the parked -partners, and netstack's bring-up is busy at the same time. - -Owner: the scheduler track, -`issues/placement-has-no-migration-cost-or-wake-affinity.md`. - -Exit: the cause is named from a boot log that shows the spawn's choices -while it misses, and either fixed or held by a bounded placement test. Then -`wake.onto_busy` measures in at least 20 consecutive `benchcase` boots of -main. From 7e9725876a461b22734a46edccf076d7a129382f Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 11:26:24 +0100 Subject: [PATCH 3/4] review r1: the rule is driver.rs's placement Co-Authored-By: Claude Opus 5.5 --- ...can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md index 8b75dc3c7a9..85ed2c54202 100644 --- a/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md +++ b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md @@ -27,7 +27,7 @@ The slow row happens on `544f9cc3c`, so the behaviour predates #1074. Whether #1074 raises the refusal rate is not established: 0 of 8 against 3 of 10, one-sided Fisher exact p = 0.15. The readbacks are PR #1074's sessions 1–5. -**What could cause it.** `place`'s rule (`kernel/src/sched/driver.rs`) takes +**What could cause it.** `placement`'s rule (`kernel/src/sched/driver.rs`) takes the CPUs that are answering, then the least `load`, then a CPU that is not sleeping, then busy siblings, from a global rotating start. So: From 05da59f65bc8f1b5f546e034363a4e0058ab1501 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 28 Oct 2026 11:28:54 +0100 Subject: [PATCH 4/4] review r2: placement picks the start, and CpuHandles::place prefers the least load, then a sleeping CPU, then the fewest busy siblings Co-Authored-By: Claude Opus 5.5 --- ...ss-a-cpu-holding-parked-threads-for-seconds-at-boot.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md index 85ed2c54202..33242f61000 100644 --- a/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md +++ b/issues/a-spawn-can-miss-a-cpu-holding-parked-threads-for-seconds-at-boot.md @@ -27,9 +27,11 @@ The slow row happens on `544f9cc3c`, so the behaviour predates #1074. Whether #1074 raises the refusal rate is not established: 0 of 8 against 3 of 10, one-sided Fisher exact p = 0.15. The readbacks are PR #1074's sessions 1–5. -**What could cause it.** `placement`'s rule (`kernel/src/sched/driver.rs`) takes -the CPUs that are answering, then the least `load`, then a CPU that is not -sleeping, then busy siblings, from a global rotating start. So: +**What could cause it.** `placement` (`kernel/src/sched/driver.rs`) picks a +global rotating start, and `CpuHandles::place` (`kernel/pure/sched/cpu.rs`) +scans the CPUs from it. Among the CPUs still answering, it takes the least +`load`, then prefers a sleeping CPU, then the one with the fewest busy SMT +siblings. So: - If the wanted CPU stays awake or loaded while another answering CPU sleeps, for example with netstack's bring-up running there, the rule never