Idle CPUs enter the deep states their firmware declares

images/system.toml

@@ -115,11 +115,13 @@ receives = ["input"]
 devices = ["isa:0060,0064:1,12"]
 memory-floor-mib = 24
 
+# `wake-latency` holds every idle CPU to waking within half a period while a
+# stream plays, so a CPU in a deep state does not make the mix late.
 [programs.soundserver]
 service = true
 serves = ["soundserver", "volume", "sound-output"]
 devices = ["pci:04:01:00/1af4:1059"]
-syscap = ["rt"]
+syscap = ["rt", "wake-latency"]
 
 # The Intel HD Audio driver: one process per HDA function the kernel does not
 # drive, each walking its own controller's codecs and offering soundserver the

issues/a-counters-read-within-10-ms-of-another-returns-the-older-reading-as-new.md

@@ -0,0 +1,31 @@
+---
+status: assigned
+kind: defect
+opened: 2026-10-28
+---
+
+# A counters read within 10 ms of another returns the older reading as new
+
+`SYS_COUNTERS` reads every CPU in a round: the reader kicks each CPU and
+waits for its answer (`kernel/src/counters.rs`). A read issued within
+`JOIN` (10 ms) of the last round's issue waits on that round and is served
+its values. Nothing in the reply says so. The reader gets a reading taken up
+to 10 ms before it asked, and two reads 5 ms apart can get one reading twice.
+
+#1073's group-start diagnostic read the counters around each group's 5 ms
+`cpu.clock` window. 13 of its 17 windows came back with before equal to
+after on every CPU (`ToyOSOrg/evidence` branch `pr-1073`, `r6/`). osbench's
+shipped instruments now take each window from the readings' own counter
+stamps and refuse a CPU whose readings are not the row's
+(`osbench_rules::window`). That makes the defect visible, not absent: a
+caller that does not compare stamps still reads an old reading as current.
+
+Owner: the idle track's follow-up
+(`issues/idle-cpus-enter-the-c-states-cst-names.md`), in
+`kernel/src/counters.rs` and the `toyos_abi::counters` record.
+
+**Exit**: a read never returns a reading older than its own call as
+current. Either it waits for a round issued after it, or its record says it
+was served an earlier round and when that round was issued; the ABI test
+and a kernel host test hold whichever is chosen. osbench's window rule
+stays as the reader's check.

issues/aarch64-cpus-idle-only-in-wfi.md

@@ -0,0 +1,36 @@
+---
+status: open
+kind: track
+opened: 2026-10-28
+---
+
+# AArch64 CPUs idle only in `wfi`
+
+On x86 the kernel enters the states a processor's `_CST` names
+(`kernel/src/idle.rs`); on AArch64 every idle CPU waits in `wfi`
+(`kernel/src/arch/aarch64/hw.rs`), and `arch::idle::facts` admits nothing
+deeper (`kernel/src/arch/aarch64/idle.rs`).
+
+**The equivalent.** An Arm platform declares its idle states in `_LPI` under
+its processor and processor-container devices (ACPI 6.5 §8.4.3), each entered
+through PSCI `CPU_SUSPEND` with the `power_state` its FFH entry method names
+(Arm DEN0022, DEN0048). A retention state comes back as `wfi` does. A
+power-down state loses the core's context and needs a resume path. The
+selection (`kernel::sched::idle`) and the `WakeLatency` bound are the
+architecture's already; what is missing is the declaration's channel and
+entry:
+
+- AArch64 has no `acpi` claim or AML server to hand a declaration over
+  (`kernel/src/arch/aarch64/acpi_mode.rs`);
+- nothing decodes `_LPI` or processor containers, which claiming LPI support
+  in `\_SB._OSC` asserts (§6.2.11.2, §8.4.3.5);
+- no `CPU_SUSPEND` entry, and no always-on timer check (the GTDT's flag)
+  standing for x86's ARAT.
+
+No fleet machine runs AArch64 on metal, and whether QEMU's `virt` declares an
+`_LPI` is unread.
+
+**Exit**: an AArch64 machine whose firmware declares `_LPI` enters its
+retention states through `CPU_SUSPEND`, chosen by the same selector, with a
+metal or QEMU reading of the state entered and returned from. Power-down
+states, with a resume path, follow once such a machine declares one.

issues/idle-cpus-enter-the-c-states-cst-names.md

@@ -1,15 +1,26 @@
 ---
-status: assigned
+status: open
 kind: track
 opened: 2026-09-29
 ---
 
 # Idle CPUs enter the C-states `_CST` names
 
-The kernel runs no `mwait` and reads no `_CST`; this track is being built on
-branch `deep-idle`. ToyOS's cores never go below C1/C1E and its package
-never leaves PC0, and its idle package draws 3.16 W against Linux's 1.18 to
-1.57 W (#866, https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119513916).
+Built (#1073): acpiserver hands the kernel what each
+processor's `_CST` names that it may enter as it is
+(`system/acpiserver/src/cst.rs`), the kernel checks it against the CPU
+(`lib/userbound/src/idle.rs`) and chooses a state each idle period
+(`kernel/pure/sched/idle.rs`, `kernel/src/idle.rs`), and a `WakeLatency`
+hold bounds the exit (`soundserver` holds half a period while it plays).
+On the T14 that is `MWAIT 0x31` and `0x60` (253 and 1048 µs); on the AMD
+bench port 0x414 (400 µs), its type-3 0x415 withheld
+(`issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md`).
+AArch64 keeps `wfi` (`issues/aarch64-cpus-idle-only-in-wfi.md`). What
+remains is the exit below, read on metal.
+
+Before it, ToyOS's cores never went below C1/C1E and its package never left
+PC0, and its idle package drew 3.16 W against Linux's 1.18 to 1.57 W (#866,
+https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119513916).
 What `hlt` alone costs a wake, firmware's C1E promotion included, is
 `issues/a-cpu-leaving-hlt-runs-its-first-microseconds-slow-under-c1e.md`.
 
@@ -23,6 +34,78 @@ faster than a `hlt` control. Its `work.build_shaped.j1` p50 read 1176 ms
 against the control's 967, with its own 5 runs spread from 817 to 1594 ms,
 the control on a different `main`: unresolved.
 
+**The exit cost of deep states, accepted** (orchestrator, 2026-10-28, under
+the owner's direction of 2026-10-27), as #1073's first boots read it at
+`59dcd1f03`, p50 against main's `ea6414e53` and against Linux 6.8 on the same
+bench (Linux's 50 µs timer slack included):
+
+| row | T14 main | T14 deep idle | T14 Linux | AMD main | AMD deep idle | AMD Linux |
+|---|---|---|---|---|---|---|
+| `time.sleep_late.1ms` | 17.7 µs | 22.9 µs | 71.5-71.8 µs | 32.5 µs | 5.4 µs | 69.7-70.8 µs |
+| `time.sleep_late.10ms` | 17.8 µs | 29.5 µs | 213 µs | 32.6 µs | 32.3 µs | 342-344 µs |
+| `wake.storm.drain.w4` | 37 µs | 127 µs | 121-123 µs | 105 µs | 307 µs | 77-78 µs |
+| `wake.storm.drain.w16` | 77 µs | 144 µs | 370-1041 µs | 333 µs | 329 µs | 113-200 µs |
+
+A sleep long enough for a deep state is woken early by its exit and still
+lands later the deeper the state was (none at 100 µs, C2 at 1 ms, C10 at
+10 ms), which an early wake that fell short would make hundreds of µs:
+consistent with the cost of the caches a deep state powers off, not measured
+as that. A burst of wakes to CPUs in deep states pays their exits.
+
+**Where the exit stands** (#1073's second boots at `f8749dc69`, two boots
+each against its control with the deep states off, all else equal):
+
+- T14 `crunch.single` 1.10 to 1.11 times faster (deflate 60.3 to 54.5 ms,
+  Linux's 54.6 to 54.7). The clock beside both deflate and
+  `malloc_churn.1048576.t1` read 4187 to 4189 MHz in all eight boots with
+  the deep states, against 3788 to 3789 in the four A\* boots (each row's
+  `mhz`, `cpu.clock`'s reading beside it). Across deflate, every core but
+  the thread's was 99.5 to 99.9 % of the row in core C7, which counts at the
+  TSC's rate.
+- T14 `cpu.clock` at 4189 at 8 and 8 group starts of 17, the control at
+  none; Linux reaches 4190 at 13. Boot, identity, energy, system and mem
+  stay at 3789 even when every idle CPU waits deepest (#1073's first
+  always-deepest control), so idle depth does not hold them there;
+  osbench now names what ran beside each group's clock
+  (`<group>.cpu.clock.ran`).
+- AMD: no clock gap and no throughput change; the IPC cost of the first
+  boots is gone with the idle choice folded into the pass's timer. The slow
+  deflate is not the idle states: it came back with every idle CPU in C1,
+  at the same clock
+  (`issues/the-amd-benchs-deflate-runs-6-percent-slow-in-some-boots-at-the-same-clock.md`).
+- Third boots at `f6510defd`: the T14 at 4189 at 8 and 10 group starts.
+  Boot, identity, energy and system run while acpiserver is still loading
+  its tables (12.2 s by the kernel log's clock in the first B3 boot,
+  `r4/` in the evidence), and acpiserver ran for the whole of each of those
+  clock windows: a measured busy core beside the clock's thread. At the
+  starts of net, procs, mem, tail and nvme still at 3789, the roster names
+  nothing beside osbench for more than 0.9 ms of the 5.5 ms window. Mem
+  stays unexplained. Net, procs, tail and nvme reached 4189 under the
+  always-deepest control, so the history holding idle CPUs shallow after
+  the group before is the open hypothesis there; a per-CPU state read
+  inside the window would test it.
+- Group starts at 4189 across the eight T14 boots with the deep states:
+  8 and 8 (r2), 8 and 10 (r3), 10 (B4), 5 (B5), 10 (B6) and 10 (B7), counted
+  as a maximum above 4100 MHz. Only sched, files, ipc and work reached it
+  in all eight; B5 and B6 lost time, which the others had. It is a spread
+  from boot to boot, not a fixed set of groups.
+- Soundserver's worst IRQ lateness, over every window line of the
+  testcases boots: with every idle CPU in C1 (one boot a bench), T14 181 to
+  572 µs and AMD 0 to 2661; with the deep states (two boots a bench), T14 0
+  to 564 µs and AMD 0 to 2661. The deep states move neither, and no window
+  underran.
+
+**#1066's exit is unmet** where this track stands: the T14's `cpu.clock` reaches
+ratio 42 at 5 to 10 of 17 group starts against Linux's 13. Of the groups
+below it, boot, identity, energy and system are held by acpiserver's
+measured start-up work; the rest are the open question above. Three
+diagnostic boots (#1073's `r6/`) read the counters around each group's
+clock, the same with the history switched off, and the scheduler's diary
+across it. The counters read came back as one reading twice in 13 of 17
+windows (`issues/a-counters-read-within-10-ms-of-another-returns-the-older-reading-as-new.md`),
+and switching the history off left the count inside the spread. The
+finding goes into this record with the track's next change.
+
 **Exit**: on the T14, since a KVM guest's idle is its host's: the idle
 minute's package energy, timer lateness no worse than `hlt`'s by more than
 the chosen state's exit latency, and osbench's `cpu.clock` maximum at ratio

issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md

@@ -6,8 +6,10 @@ opened: 2026-10-12
 
 # Idle is `hlt`, so every wake of an idle CPU is an IPI
 
-An idle CPU waits in `sti; hlt` (`kernel/src/arch/x86_64/hw.rs:21`), so the
-only thing that ends its wait is an interrupt: a wake posted to it rings its
+An idle CPU waits in `sti; hlt` (`kernel/src/arch/x86_64/hw.rs`) or in a
+deep state its firmware declared, an `MWAIT` on a line nothing writes or a
+trapped port read (`kernel/src/arch/x86_64/idle.rs`), so the only thing that
+ends its wait is an interrupt: a wake posted to it rings its
 doorbell (`kernel/pure/sched/mailbox.rs:474-490`) and then sends a kick IPI
 (`kernel/src/arch/x86_64/apic.rs:256-260`), whose entry and handler run on the
 woken CPU before its `hlt` returns (`kernel/src/arch/x86_64/idt/kick.rs:3-9`).
@@ -26,8 +28,9 @@ unless firmware disables them, gives the monitor line as 64 bytes, and has
 MWAIT support one C1 sub-state (PPR 55922 rev. 3.06, `CPUID_Fn00000001_ECX`
 bit 3, `CPUID_Fn00000005`). So the stage is backed on Intel by §11.10.4 and
 §11.10.6.4, and is Intel's alone: the AMD bench keeps `hlt` and the kick, and
-whether it takes the doorbell is the orchestrator's to decide once that bench's
-`_CST` is read. A 2020+ Intel CPU whose specification update names an erratum
+whether it takes the doorbell is the orchestrator's to decide; that bench's
+`_CST` reads C1 as MWAIT and its deeper states as port reads, which no store
+ends. A 2020+ Intel CPU whose specification update names an erratum
 by which a store may not wake MWAIT keeps the kick. The T14's lists no MONITOR
 or MWAIT erratum (631123-034); Intel's Core Ultra 200V specification update is
 unread. Whether the T14 and the AMD bench enumerate MONITOR/MWAIT is ToyOS's

issues/no-published-run-compares-toyos-with-linux-on-one-machine.md

@@ -48,6 +48,12 @@ OpenSSL and AWS-LC by the crypto oracle probe, and that table is the
 evidence: `ToyOSOrg/evidence` at `bed0b00efac82546b995f0841da54aea11c45694`,
 `r2/table.md`.
 
+The AMD bench's `crunch.single.deflate` read 4 to 6 % slower in four boots
+of twelve, at the same clock, with and without the deep idle states
+(`issues/the-amd-benchs-deflate-runs-6-percent-slow-in-some-boots-at-the-same-clock.md`).
+An AMD single-thread row compared across boots needs repeats enough to
+see which mode each boot read.
+
 What ToyOS cannot measure is reported `unsupported` by name, and is not a
 comparison until it can:
 

issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md

@@ -0,0 +1,37 @@
+---
+status: open
+kind: defect
+opened: 2026-10-28
+---
+
+# The AMD bench's C3 is withheld for want of a coherence statement
+
+The AMD bench's `_CST` names three states: C1 as FFH `MWAIT 0`, C2 as a
+read of port 0x414 (type 2, 400 µs) and C3 as a read of 0x415 (type 3,
+350 µs). acpiserver hands the kernel C2 and withholds C3
+(`system/acpiserver/src/cst.rs`), so that bench idles no deeper than 0x414.
+
+**Why.** ACPI 6.5 §8.1.4 makes OSPM responsible for the caches' coherency in
+a type-3 state, on a multiprocessor by flushing them before entry. ToyOS
+does not flush: `WBINVD` on Zen 2 writes back and invalidates the whole
+CCX's shared L3 under its running siblings. Neither PPR 55922, 55901,
+55772, OSRR 56255 nor APM Vol. 2 says that an I/O-trapped C-state
+(`CStateBaseAddr`, MSRC001_0073) keeps caches coherent on its own. AMD's
+Family 16h BKDG (52740 §2.5.3.2.3) does say so of the same interface on that
+family: a core in such a state either has its probes serviced or flushes
+its L1 in hardware before CC6. That family's BIOS requirement declares its
+I/O C-state type 2. It is another family's document, so it is not taken as
+Zen 2's.
+
+The same firmware's `_LPI` declares 0x415 with no context to lose (18 and
+350 µs wake for 0x414 and 0x415, 60 and 1000 µs residency). OSPM uses `_LPI`
+only once `\_SB._OSC` has claimed support for processor containers, and
+never mixed with `_CST` (§8.4.3.5). ToyOS claims no such support.
+
+**Exit**: either an AMD document for family 17h saying that its I/O C-states
+keep caches coherent with no OS flush, after which `cst.rs` hands over a
+type-3 SystemIO state on that family and the AMD's metal row reads 0x415
+entered; or ToyOS's `_LPI` support (`issues/aarch64-cpus-idle-only-in-wfi.md`
+owes the same decode and processor containers) used on that bench in
+place of `_CST`. **Mutation**: the type-3 rule removed, so 0x415 is handed
+over and the host test in `cst.rs` reds.

issues/the-amd-benchs-deflate-runs-6-percent-slow-in-some-boots-at-the-same-clock.md

@@ -0,0 +1,33 @@
+---
+status: open
+kind: finding
+opened: 2026-10-28
+---
+
+# The AMD bench's deflate runs 6 % slow in some boots at the same clock
+
+osbench's `crunch.single.deflate` on the AMD bench (Ryzen 7 4800U) has more
+than one value across twelve boots (`ToyOSOrg/evidence` branch `pr-1073`,
+`r3/` to `r8/`):
+
+- In three boots its p50 was 66.6 to 66.8 ms. Two ran the thread on CPU 4
+  and one on CPU 10.
+- In one, B6's, it was 65.7 ms, on CPU 0.
+- In the eight others it was 62.9 to 63.7 ms, on CPUs 0, 2, 6, 10 and 12.
+
+The thread's APERF/MPERF ratio, the one figure of the pair AMD defines
+(APM Vol. 2 §17.3), differed by under 1 % across the boots. The slow boots
+include one with every idle CPU in C1 and the rest with the deep states.
+In the slow A\* boot and the slow B4 boot, no other CPU's MPERF advanced by
+a thousandth of its TSC's count across the row, and MPERF counts only in
+C0 (§17.3).
+So it is not the idle states, the clock, the placement, or a running
+neighbour.
+
+`sha256`, `matmul` and `parse` show no such split. Every AMD single-thread
+timing row may carry this spread. A comparison between two AMD boots,
+ToyOS's or Linux's, can therefore read 6 % that neither system caused.
+
+**Exit**: one boot running deflate several times, at intervals, on one
+pinned CPU, which says whether the mode is per boot or per run, and with the
+package's power and temperature beside it once a counter carries them.

issues/the-scheduler-and-the-clocks-never-talk.md

@@ -19,12 +19,16 @@ Two couplings every production scheduler has and ours lacks:
   wake. A latency-sensitive wake should prefer a shallow-idle core; a
   throughput task can pay to wake a deep one.
 
-Idle-state awareness waits for the states
-`issues/idle-cpus-enter-the-c-states-cst-names.md` is building. The kernel
-chooses no P-state: it declares HWP's bounds
+Idle-state awareness has its first half: a wake forward prefers, among
+equally near free cores, the one whose state wakes quickest
+(`CpuHandles::idle_near`, with the exit each idle CPU publishes from
+`kernel/src/idle.rs`). A latency-sensitive wake preferring a shallow core
+over a nearer deep one, and a throughput task choosing to pay a deep one's
+exit, wait for the explicit latency requests below, whose caller says which
+it is. The kernel chooses no P-state: it declares HWP's bounds
 (`kernel/src/arch/x86_64/control_regs.rs`), and the CPU chooses within them.
-Filed so the placement track (see `placement-has-no-migration-cost-or-wake-affinity.md`)
-leaves room for an idle-state input when it designs wake placement.
+The placement track (`placement-has-no-migration-cost-or-wake-affinity.md`)
+takes the published exit as its idle-state input.
 
 ## Realtime exhaustion (owner decision J2, 2026-10-20; stage 4)
 

issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md

@@ -15,8 +15,9 @@ the kernel its sleep type, which the kernel powers the machine off with and
 has from nowhere else. It initialises the namespace, negotiates `_OSC`,
 serves the runtime GPEs and delivers Notifies (#875). What else
 ToyOS takes from them it takes another way: the loader asks UEFI for the root
-bridges' windows `_CRS` would name (`loader/src/rootbridge.rs`). Nothing
-reads `_CST`, which names a CPU's C-states.
+bridges' windows `_CRS` would name (`loader/src/rootbridge.rs`). The
+processors' `_CST`, which names their C-states, is the idle track's
+(`issues/idle-cpus-enter-the-c-states-cst-names.md`).
 
 **Ruled** (owner, 2026-10-03, on the scout in #681's comment 5966817345): the
 switch to ACPI mode lands as stage 1, together with a userland server that
@@ -485,8 +486,12 @@ ACPI Interface Specification (302223-007) for a processor's `_OSC` on Intel:
 - The server does OSPM's side of the protocol. It:
   - loads every block, static and loaded at run time;
   - initialises the namespace;
-  - negotiates `\_SB._OSC` and each processor's `_OSC` with only what ToyOS
-    honours;
+  - negotiates `\_SB._OSC`, query then commit, and commits each
+    processor's `_OSC` with only what ToyOS honours and no query (the
+    orchestrator's ruling of 2026-10-27, under the owner's direction for
+    deep idle that day: 302223-007 §2.1 makes a processor's bits statements
+    of what OSPM does, and ACPI 6.5 §6.2.11.1.1 only recommends the query;
+    the T14's processor query answers 0x1 whatever is asked);
   - runs every runtime GPE's method;
   - delivers every Notify.
 - On the T14 and the AMD bench, in the state the owner ruled, ACPI 6.5
@@ -503,9 +508,8 @@ Not in finished:
 - concurrent method execution, which §5.5.2 permits and does not require;
 - the PCI host bridges' `_OSC`;
 - sleep states;
-- `_CST`, which no part of this track reads: idle CPUs entering the states
-  it names read it (`issues/idle-cpus-enter-the-c-states-cst-names.md`,
-  #870);
+- `_CST`, which the idle track reads (`system/acpiserver/src/cst.rs`,
+  `issues/idle-cpus-enter-the-c-states-cst-names.md`);
 - `_PDC`, which Intel deprecated in favour of `_OSC` in ACPI 3.0 and which
   ACPI 6.5 does not define.