diff --git a/issues/idle-cpus-enter-the-c-states-cst-names.md b/issues/idle-cpus-enter-the-c-states-cst-names.md index 941b5b223dc..c6455dc50d1 100644 --- a/issues/idle-cpus-enter-the-c-states-cst-names.md +++ b/issues/idle-cpus-enter-the-c-states-cst-names.md @@ -12,11 +12,18 @@ processor's `_CST` names that it may enter as it is (`lib/userbound/src/idle.rs`) and chooses a state each idle period (`kernel/pure/sched/idle.rs`, `kernel/src/idle.rs`), and a `WakeLatency` hold bounds the exit (`soundserver` holds half a period while it plays). -On the T14 that is `MWAIT 0x31` and `0x60` (253 and 1048 µs); on the AMD -bench port 0x414 (400 µs), its type-3 0x415 withheld -(`issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md`). -AArch64 keeps `wfi` (`issues/aarch64-cpus-idle-only-in-wfi.md`). What -remains is the exit below, read on metal. +What is handed over is a rule per vendor and family: Intel's FFH `MWAIT` +states, and trapped port reads off Intel, of type 3 only on AMD from Family +17h and on Hygon. On the T14 that is `MWAIT 0x31` and `0x60` (253 and +1048 µs); on the AMD bench, Family 17h, ports 0x414 and 0x415 (400 and +350 µs). AArch64 keeps `wfi` (`issues/aarch64-cpus-idle-only-in-wfi.md`). +What remains is the exit below, read on metal. + +Not established for the AMD rule: whether any Zen part needs BM_STS read +before a type-3 entry, which ToyOS does not do, or BM_RLD set, which ToyOS +leaves as firmware set it; only PPR 55901 Vol. 5 speaks to them, for Family +19h Model 11h (BM_STS "no longer used except for server CPU", BM_RLD "no +longer needed"). Before it, ToyOS's cores never went below C1/C1E and its package never left PC0, and its idle package drew 3.16 W against Linux's 1.18 to 1.57 W (#866, @@ -42,20 +49,29 @@ build (`f6510defd`) and one session a bench, B7 at `18fe4e150` beside them, and Linux 6.8 on the same bench (its 50 µs timer slack included; #1073's first boots): -| row | T14 C1 | T14 deep | T14 delta | T14 Linux | AMD C1 | AMD deep | AMD delta | AMD Linux | +| row | T14 C1 | T14 deep | T14 delta | T14 Linux | AMD C1 | AMD 0x414 | AMD 0x415 | AMD Linux | |---|---|---|---|---|---|---|---|---| -| `time.sleep_late.1ms` | 17.7, 17.8 | 22.1, 14.6 (B7 17.2) | −3 to +4 | 71.5-71.8 | 32.2, 32.5 | 5.6, 5.6 (B7 5.4) | −27 | 69.7-70.8 | -| `time.sleep_late.10ms` | 17.8, 17.8 | 28.3, 28.3 (B7 27.4) | +10.5 to +10.6 | 213 | 32.3, 32.6 | 32.6, 33.3 (B7 32.8) | 0 to +1 | 342-344 | -| `wake.storm.drain.w4` | 38.5, 37.6 | 126.1, 128.6 (B7 127.9) | +88 to +91 | 121-123 | 306, 107 | 112, 306 (B7 107) | none: 107 or 306 either way | 77-78 | -| `wake.storm.drain.w16` | 74.1, 77.6 | 161.1, 145.5 (B7 130.1) | +68 to +87 (B7 +52) | 370-1041 | 304, 328 | 321, 331 (B7 350) | within the spread | 113-200 | +| `time.sleep_late.1ms` | 17.7, 17.8 | 22.1, 14.6 (B7 17.2) | −3 to +4 | 71.5-71.8 | 32.2, 32.5 | 5.6, 5.6 (B7 5.4) | 5.5 | 69.7-70.8 | +| `time.sleep_late.10ms` | 17.8, 17.8 | 28.3, 28.3 (B7 27.4) | +10.5 to +10.6 | 213 | 32.3, 32.6 | 32.6, 33.3 (B7 32.8) | 52.4 | 342-344 | +| `wake.storm.drain.w4` | 38.5, 37.6 | 126.1, 128.6 (B7 127.9) | +88 to +91 | 121-123 | 306, 107 | 112, 306 (B7 107) | 64.0 | 77-78 | +| `wake.storm.drain.w16` | 74.1, 77.6 | 161.1, 145.5 (B7 130.1) | +68 to +87 (B7 +52) | 370-1041 | 304, 328 | 321, 331 (B7 350) | 80.3 | 113-200 | + +The AMD's 0x415 column is one benchcase boot of #1089 at `31db18c19`, where +every deep period requests 0x415 (its exit and residency are both below +0x414's). It is on a later `main` than the columns beside it, with main's +scheduler placement change (`05da59f65`) between them. Against 0x414, a +1 ms sleep lands as it did, and a 10 ms sleep lands 20 µs later (52.4 +against 32.6 to 33.3), consistent with a deeper state's exit and not measured as that. The storms drain +faster than in either earlier column, but no control on the same base was +read, so that is not attributed to 0x415. A sleep long enough for a deep state is woken early by its exit and still lands later the deeper the state was (none at 100 µs, C2 at 1 ms, C10 at 10 ms), which an early wake that fell short would make hundreds of µs: consistent with the cost of the caches a deep state powers off, not measured as that. A burst of wakes to CPUs in deep states pays their exits. The cost -is the T14's; on the AMD the deep state lands a 1 ms sleep 27 µs sooner and -moves nothing else. +is the T14's. On the AMD, 0x415 lands a 1 ms sleep 27 µs sooner than C1 and +a 10 ms sleep 20 µs later, and its storms are not attributed (above). What it costs, as inference, not measured per workload: @@ -64,7 +80,7 @@ What it costs, as inference, not measured per workload: about 70 to 90 µs later, against a 60 Hz frame's 16.7 ms. - **Audio**: while a stream plays, soundserver holds the exit to half its 2.9 ms period, about 1.45 ms, which both T14 states (253 and 1048 µs) and - the AMD's (400 µs) are inside: so a period's wake may still pay up to + the AMD's (400 and 350 µs) are inside: so a period's wake may still pay up to 1048 µs of exit on the T14, a third of the period. Measured, its worst IRQ lateness is not moved and no window underran (below). - **Normal workloads**: a server or build that wakes many sleeping threads @@ -108,7 +124,8 @@ each against its control with the deep states off, all else equal): testcases boots: with every idle CPU in C1 (one boot a bench), T14 181 to 572 µs and AMD 0 to 2661; with the deep states (two boots a bench), T14 0 to 564 µs and AMD 0 to 2661. The deep states move neither, and no window - underran. + underran. With 0x415 (#1089's first testcases boot, 6 window lines), AMD 0 + to 228 µs, and no underrun. **`cpu.clock` at group starts** is a diagnostic of the turbo grant, not a product target: the T14's reaches ratio 42 at 5 to 10 of 17 group starts diff --git a/issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md b/issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md deleted file mode 100644 index 4344f4119ce..00000000000 --- a/issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-28 ---- - -# The AMD bench's C3 is withheld for want of a coherence statement - -The AMD bench's `_CST` names three states: C1 as FFH `MWAIT 0`, C2 as a -read of port 0x414 (type 2, 400 µs) and C3 as a read of 0x415 (type 3, -350 µs). acpiserver hands the kernel C2 and withholds C3 -(`system/acpiserver/src/cst.rs`), so that bench idles no deeper than 0x414. - -**Why.** ACPI 6.5 §8.1.4 makes OSPM responsible for the caches' coherency in -a type-3 state, on a multiprocessor by flushing them before entry. ToyOS -does not flush: `WBINVD` on Zen 2 writes back and invalidates the whole -CCX's shared L3 under its running siblings. Neither PPR 55922, 55901, -55772, OSRR 56255 nor APM Vol. 2 says that an I/O-trapped C-state -(`CStateBaseAddr`, MSRC001_0073) keeps caches coherent on its own. AMD's -Family 16h BKDG (52740 §2.5.3.2.3) does say so of the same interface on that -family: a core in such a state either has its probes serviced or flushes -its L1 in hardware before CC6. That family's BIOS requirement declares its -I/O C-state type 2. It is another family's document, so it is not taken as -Zen 2's. - -The same firmware's `_LPI` declares 0x415 with no context to lose (18 and -350 µs wake for 0x414 and 0x415, 60 and 1000 µs residency). OSPM uses `_LPI` -only once `\_SB._OSC` has claimed support for processor containers, and -never mixed with `_CST` (§8.4.3.5). ToyOS claims no such support. - -**Exit**: either an AMD document for family 17h saying that its I/O C-states -keep caches coherent with no OS flush, after which `cst.rs` hands over a -type-3 SystemIO state on that family and the AMD's metal row reads 0x415 -entered; or ToyOS's `_LPI` support (`issues/aarch64-cpus-idle-only-in-wfi.md` -owes the same decode and processor containers) used on that bench in -place of `_CST`. **Mutation**: the type-3 rule removed, so 0x415 is handed -over and the host test in `cst.rs` reds. diff --git a/lib/acpiserver-api/src/lib.rs b/lib/acpiserver-api/src/lib.rs index 16950ac8212..17b5c3b9cf7 100644 --- a/lib/acpiserver-api/src/lib.rs +++ b/lib/acpiserver-api/src/lib.rs @@ -14,6 +14,11 @@ pub const QUERIES_COUNTED: &str = "embedded controller queries taken: "; pub const CONTROLLER_SERVED: &str = "acpiserver: embedded controller on GPE "; pub const CONTROLLER_NONE: &str = "acpiserver: embedded controller none: "; +/// The server's census of the processors' idle states, written once their +/// `_CST`s are handed to the kernel: how many declarations it kept, the +/// states each held, and what it withheld. +pub const IDLE_CENSUS: &str = "acpiserver: the processors' idle states: "; + /// The server's census of the batteries and adapters it found, written once /// at the end of its start where a battery is present, after their first /// reading. diff --git a/sdk/abi/src/arch/aarch64.rs b/sdk/abi/src/arch/aarch64.rs index bab478eee86..4fc56f7835c 100644 --- a/sdk/abi/src/arch/aarch64.rs +++ b/sdk/abi/src/arch/aarch64.rs @@ -29,9 +29,9 @@ pub fn counter() -> u64 { now } -/// Whether the processor is Intel's: no Intel processor implements AArch64. -pub fn processor_is_intel() -> bool { - false +/// AArch64 has no CPUID identity. +pub fn processor_id() -> Option { + None } /// AArch64 has no port space, and no grant names a port on it: each of these diff --git a/sdk/abi/src/arch/mod.rs b/sdk/abi/src/arch/mod.rs index 5c1a9f443d9..cb4ac4d86e2 100644 --- a/sdk/abi/src/arch/mod.rs +++ b/sdk/abi/src/arch/mod.rs @@ -1,7 +1,7 @@ //! What a process reads off its own CPU without a syscall, one module per //! architecture: the counter the clock page describes, the thread's id in //! its thread control block, `in` and `out` where there is a port space, and -//! whether the processor is Intel's. +//! the processor's CPUID identity. #[cfg(target_arch = "aarch64")] mod aarch64; @@ -9,6 +9,6 @@ mod aarch64; mod x86_64; #[cfg(target_arch = "aarch64")] -pub use aarch64::{counter, current_tid, ioport, processor_is_intel, TCB_TID}; +pub use aarch64::{counter, current_tid, ioport, processor_id, TCB_TID}; #[cfg(target_arch = "x86_64")] -pub use x86_64::{counter, current_tid, ioport, processor_is_intel, TCB_TID}; +pub use x86_64::{counter, current_tid, ioport, processor_id, TCB_TID}; diff --git a/sdk/abi/src/arch/x86_64.rs b/sdk/abi/src/arch/x86_64.rs index 8d8e1b3d94e..aae795211be 100644 --- a/sdk/abi/src/arch/x86_64.rs +++ b/sdk/abi/src/arch/x86_64.rs @@ -38,27 +38,11 @@ pub fn counter() -> u64 { } } -/// Whether the processor is Intel's, by CPUID leaf 0's vendor string, -/// `GenuineIntel` in EBX, EDX and ECX (Intel SDM Vol. 2A, "CPUID"): the -/// vendor whose interface a processor object's `_OSC` speaks under Intel's -/// UUID (Intel 302223-007 §2.2). -pub fn processor_is_intel() -> bool { - let (ebx, ecx, edx): (u64, u32, u32); - // SAFETY: CPUID is unprivileged, and the kernel enables no CPUID - // faulting; RBX, which LLVM reserves, is put back as it was. - unsafe { - core::arch::asm!( - "mov {rbx}, rbx", - "cpuid", - "xchg {rbx}, rbx", - rbx = out(reg) ebx, - inout("eax") 0u32 => _, - inout("ecx") 0u32 => ecx, - out("edx") edx, - options(nomem, nostack, preserves_flags), - ); - } - [ebx as u32, edx, ecx] == [*b"Genu", *b"ineI", *b"ntel"].map(u32::from_le_bytes) +/// The processor's CPUID identity. +pub fn processor_id() -> Option { + let leaf_0 = core::arch::x86_64::__cpuid(0); + let vendor = core::array::from_fn(|i| [leaf_0.ebx, leaf_0.edx, leaf_0.ecx][i / 4].to_le_bytes()[i % 4]); + Some(crate::ProcessorId { vendor, signature: core::arch::x86_64::__cpuid(1).eax }) } /// `in` and `out` from Ring 3: a port this process holds no grant for, which diff --git a/sdk/abi/src/lib.rs b/sdk/abi/src/lib.rs index 6ec1d15dea8..4596b33cc0d 100644 --- a/sdk/abi/src/lib.rs +++ b/sdk/abi/src/lib.rs @@ -125,4 +125,14 @@ pub mod tcb { const _: () = assert!(AARCH64_BYTES == 16, "AArch64's psABI TCB is two words"); } -pub use arch::{current_tid, ioport, processor_is_intel, TCB_TID}; +pub use arch::{current_tid, ioport, processor_id, TCB_TID}; + +/// An x86 processor's CPUID identity (Intel SDM Vol. 2A, "CPUID"; AMD APM +/// Vol. 3, CPUID Fn0000_0000 and Fn0000_0001_EAX). +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub struct ProcessorId { + /// Leaf 0's vendor string: its EBX, EDX and ECX bytes, in that order. + pub vendor: [u8; 12], + /// Leaf 1's EAX: the family, model and stepping signature. + pub signature: u32, +} diff --git a/system/acpiserver/src/cpu.rs b/system/acpiserver/src/cpu.rs new file mode 100644 index 00000000000..415da738ad8 --- /dev/null +++ b/system/acpiserver/src/cpu.rs @@ -0,0 +1,60 @@ +//! The processor's vendor and family, which the processor objects' rules key +//! on: whose `_OSC` interface they speak (`osc`) and what their idle states +//! ask of OSPM (`cst`). + +use toyos_abi::ProcessorId; + +/// Whose processor it is, by CPUID leaf 0's vendor string. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Vendor { + Intel, + Amd, + Hygon, + Other, +} + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub struct Cpu { + pub vendor: Vendor, + /// The DisplayFamily, 0 where the processor has no CPUID. + pub family: u32, +} + +impl Cpu { + pub fn here() -> Self { + Self::of(toyos_abi::processor_id()) + } + + /// `id`'s vendor, and its family by the DisplayFamily rule (Intel SDM + /// Vol. 2A, "CPUID"; AMD APM Vol. 3, CPUID Fn0000_0001_EAX): the base + /// family, plus the extended family where the base is Fh. + pub fn of(id: Option) -> Self { + let Some(ProcessorId { vendor, signature }) = id else { return Self { vendor: Vendor::Other, family: 0 } }; + let base = signature >> 8 & 0xF; + let family = if base == 0xF { base + (signature >> 20 & 0xFF) } else { base }; + let vendor = match &vendor { + b"GenuineIntel" => Vendor::Intel, + b"AuthenticAMD" => Vendor::Amd, + b"HygonGenuine" => Vendor::Hygon, + _ => Vendor::Other, + }; + Self { vendor, family } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_family_adds_the_extended_family_only_to_a_base_of_f() { + let of = |vendor: &[u8; 12], signature| Cpu::of(Some(ProcessorId { vendor: *vendor, signature })); + assert_eq!(of(b"AuthenticAMD", 0x0086_0F01), Cpu { vendor: Vendor::Amd, family: 0x17 }); + assert_eq!(of(b"AuthenticAMD", 0x0073_0F01), Cpu { vendor: Vendor::Amd, family: 0x16 }); + assert_eq!(of(b"AuthenticAMD", 0x00B4_0F00), Cpu { vendor: Vendor::Amd, family: 0x1A }); + assert_eq!(of(b"HygonGenuine", 0x0090_0F02), Cpu { vendor: Vendor::Hygon, family: 0x18 }); + assert_eq!(of(b"GenuineIntel", 0x0008_06C1), Cpu { vendor: Vendor::Intel, family: 6 }); + assert_eq!(of(b"CentaurHauls", 0x00F0_06F2), Cpu { vendor: Vendor::Other, family: 6 }); + assert_eq!(Cpu::of(None), Cpu { vendor: Vendor::Other, family: 0 }); + } +} diff --git a/system/acpiserver/src/cst.rs b/system/acpiserver/src/cst.rs index f37e314aab0..b4058b83752 100644 --- a/system/acpiserver/src/cst.rs +++ b/system/acpiserver/src/cst.rs @@ -13,21 +13,31 @@ //! - an Intel FFH state (302223-007 Table 4: vendor 1, class 2) is `MWAIT` //! with Arg0 as its hint, withheld where Arg1 bit 1 asks bus-master //! avoidance, which ToyOS does not track; any other FFH encoding has no -//! document in hand; +//! document in hand. On AMD from Family 17h `MWAIT` reaches only C1: CPUID +//! Fn0000_0005_EDX enumerates no deeper sub-state (PPR 55922, 57896), so no +//! AMD FFH state deeper than C1 exists to support; //! - a SystemIO state is a byte read of its register, handed over off Intel, //! where the core traps it as a C-state request (AMD PPR 55922, //! MSRC001_0073); on Intel it is a chipset's P_LVLx read (§8.1.3) no machine //! ToyOS runs needs; -//! - a type-3 SystemIO state is withheld: for one OSPM keeps the caches -//! coherent, on a multiprocessor by flushing them before entry (§8.1.4), -//! which ToyOS does not, and no document in hand says the processor's own -//! hardware makes that unnecessary for it -//! (`issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md`). -//! An Intel FFH state of any type is not: its entry is the vendor's native -//! instruction, whose only OSPM duty Intel encodes is bus-master avoidance -//! (302223-007 Table 4, withheld above), and the processor flushes what a -//! C-state powers off as it enters it (SDM Vol. 3B §20.19.6.4: data "may have -//! been flushed to memory during the process of entering the idle state"); +//! - a type-3 SystemIO state is handed over only on AMD from Family 17h and +//! on Hygon. For one OSPM keeps the caches coherent, on a multiprocessor by +//! flushing them before entry (§8.1.4), which ToyOS does not. AMD's rule for +//! those processors, which its engineers hold and no AMD document in hand +//! states, is that their caches stay coherent through it with no flush and +//! that ARB_DIS is unused. BM_STS, which §8.1.4 makes an optional input to +//! OSPM's C3 policy and no coherence duty, is not read, and BM_RLD, which +//! OSPM "can optionally" set for C3, stays as firmware set it: the rule +//! says nothing of either, and PPR 55901 Vol. 5 says of Family 19h Model +//! 11h that BM_RLD is "no longer needed" and BM_STS "no longer used except +//! for server CPU". AMD's earlier families have no such rule, and their +//! BKDGs declare their I/O C-state type 2 (Family 16h's 52740 +//! §2.5.3.2.3). An Intel FFH state of any type is handed over: its entry +//! is the vendor's native instruction, whose only OSPM duty Intel encodes +//! is bus-master avoidance (302223-007 Table 4, withheld above), and the +//! processor flushes what a C-state powers off as it enters it (SDM Vol. +//! 3B §20.19.6.4: data "may have been flushed to memory during the process +//! of entering the idle state"); //! - a state `_CSD` names for software coordination (Table 8.1, SW_ALL or //! SW_ANY) is withheld: ToyOS coordinates no CPU's state with another's. //! HW_ALL asks nothing of OSPM. @@ -40,7 +50,9 @@ //! **`_LPI` is not read**, though a platform may declare both: OSPM uses it in //! preference to `_CST` only once it has said in `\_SB._OSC` that it supports //! processor containers, and never mixes the two (§8.4.3.5, §6.2.11.2), which -//! ToyOS does not say. +//! ToyOS does not say. ACPI defines an x86 state's context-loss flags for +//! Intel alone (§8.4.3.3.5), so an AMD `_LPI` state's 0 there has no defined +//! meaning. AArch64's `_LPI` is `issues/aarch64-cpus-idle-only-in-wfi.md`. use std::collections::BTreeMap; @@ -48,6 +60,7 @@ use toyos_abi::acpi::{IdleDeclaration, IdleEntry, IdleState, IDLE_STATES}; use toyos_aml::{Kind, Value}; use crate::aml::{kind, Aml}; +use crate::cpu::{Cpu, Vendor}; use crate::host::{Declared, Kernel, Stopping, OWN}; use crate::init::Names; use crate::ledger::Ledger; @@ -108,7 +121,8 @@ impl Census { pub fn line(&self) -> String { let shapes: Vec = self.shapes.iter().map(|(shape, n)| format!("{n} with {shape}")).collect(); format!( - "acpiserver: the processors' idle states: {} kept, {} naming no CPU, {} refused by the kernel; kept: {}; withheld: {}; unread: {}", + "{}{} kept, {} naming no CPU, {} refused by the kernel; kept: {}; withheld: {}; unread: {}", + acpiserver_api::IDLE_CENSUS, self.kept, self.no_cpu, self.refused, @@ -119,9 +133,9 @@ impl Census { } } -/// Read `processor`'s `_CST` and `_CSD` and hand the kernel what of them it -/// may enter: on an Intel processor where `intel`. -pub fn declare(aml: &mut Aml<'_, K>, kernel: &K, processor: &Processor, intel: bool, census: &mut Census) -> Result<(), Stopping> { +/// Read `processor`'s `_CST` and `_CSD` and hand the kernel what of them +/// `cpu` may enter. +pub fn declare(aml: &mut Aml<'_, K>, kernel: &K, processor: &Processor, cpu: Cpu, census: &mut Census) -> Result<(), Stopping> { let read = |aml: &mut Aml<'_, K>, method: &str| aml.interpreter.evaluate(&mut aml.host, &format!("{}.{method}", processor.path), &[]); let states = match read(aml, "_CST").map_err(|why| kind(&why)).and_then(|v| cst(&v)) { Ok(states) => states, @@ -145,7 +159,7 @@ pub fn declare(aml: &mut Aml<'_, K>, kernel: &K, processor: &Processo } else { Vec::new() }; - let (declaration, withheld) = translate(processor.uid, &states, &software, intel); + let (declaration, withheld) = translate(processor.uid, &states, &software, cpu); for why in withheld { census.withheld.see(&why); } @@ -260,11 +274,11 @@ pub fn csd(value: &Value) -> Result, String> { /// What of `states` the kernel may enter as they are, and why each other was /// withheld. -pub fn translate(uid: u32, states: &[CState], software: &[u64], intel: bool) -> (IdleDeclaration, Vec) { +pub fn translate(uid: u32, states: &[CState], software: &[u64], cpu: Cpu) -> (IdleDeclaration, Vec) { let mut declaration = IdleDeclaration { processor_uid: uid, count: 0, states: [IdleState { address: 0, latency_us: 0, residency_us: 0, entry: 0, reserved: [0; 7] }; IDLE_STATES] }; let mut withheld = Vec::new(); for (index, state) in states.iter().enumerate() { - let entry = match enterable(state, software.contains(&(index as u64)), intel) { + let entry = match enterable(state, software.contains(&(index as u64)), cpu) { Ok(Some(entry)) => entry, Ok(None) => continue, Err(why) => { @@ -293,7 +307,7 @@ pub fn translate(uid: u32, states: &[CState], software: &[u64], intel: bool) -> } /// How `state` is entered, `None` for C1, or why ToyOS may not enter it as it is. -fn enterable(state: &CState, software: bool, intel: bool) -> Result, String> { +fn enterable(state: &CState, software: bool, cpu: Cpu) -> Result, String> { let r = state.register; match state.kind { 1 => return Ok(None), @@ -317,13 +331,25 @@ fn enterable(state: &CState, software: bool, intel: bool) -> Result Err(format!("FFH vendor {} class {}, an encoding no document in hand defines", r.bit_width, r.bit_offset)), - SYSTEM_IO if intel => Err("a SystemIO state on Intel, a chipset's P_LVLx read".into()), - SYSTEM_IO if state.kind == 3 => Err("a type-3 SystemIO state, whose caches OSPM keeps coherent (ACPI 6.5 §8.1.4)".into()), + SYSTEM_IO if cpu.vendor == Vendor::Intel => Err("a SystemIO state on Intel, a chipset's P_LVLx read".into()), + SYSTEM_IO if state.kind == 3 && !coherent_through_c3(cpu) => { + Err("a type-3 SystemIO state, whose caches OSPM keeps coherent (ACPI 6.5 §8.1.4), off AMD Family 17h on and Hygon".into()) + } SYSTEM_IO => Ok(Some(IdleEntry::Port)), space => Err(format!("a state in space {space:#x}, which no C-state is entered through")), } } +/// Whether `cpu`'s caches stay coherent through a type-3 SystemIO state with +/// no flush and no ARB_DIS: AMD's rule from Family 17h, and Hygon's. +fn coherent_through_c3(cpu: Cpu) -> bool { + match cpu.vendor { + Vendor::Amd => cpu.family >= 0x17, + Vendor::Hygon => true, + Vendor::Intel | Vendor::Other => false, + } +} + #[cfg(test)] mod tests { use super::*; @@ -336,14 +362,18 @@ mod tests { CState { register, kind, latency } } - /// The T14's `_CST` as its firmware answers ToyOS's processor `_OSC` - /// claim, decoded from its tables (#875's check outside the tree). - fn t14() -> Vec { + const INTEL: Cpu = Cpu { vendor: Vendor::Intel, family: 6 }; + const AMD_17H: Cpu = Cpu { vendor: Vendor::Amd, family: 0x17 }; + + /// An Intel `_CST`: the T14's, as its firmware answers ToyOS's processor + /// `_OSC` claim, decoded from its tables (#875's check outside the tree). + fn intel_ffh() -> Vec { vec![state(reg(FFH, 1, 2, 1, 0x00), 1, 1), state(reg(FFH, 1, 2, 1, 0x31), 2, 253), state(reg(FFH, 1, 2, 1, 0x60), 3, 1048)] } - /// The AMD bench's. - fn amd() -> Vec { + /// An AMD `_CST` of C1 in FFH and two trapped port reads, the second type + /// 3: the AMD bench's (Family 17h). + fn amd_system_io() -> Vec { vec![state(reg(FFH, 2, 2, 0, 0x00), 1, 1), state(reg(SYSTEM_IO, 8, 0, 1, 0x414), 2, 400), state(reg(SYSTEM_IO, 8, 0, 1, 0x415), 3, 350)] } @@ -352,42 +382,59 @@ mod tests { } #[test] - fn the_t14s_c2_and_c3_are_handed_over_as_mwait_and_its_c1_is_hlt() { - let (d, withheld) = translate(3, &t14(), &[], true); + fn intel_ffh_states_of_type_2_and_3_are_handed_over_as_mwait_and_c1_is_hlt() { + let (d, withheld) = translate(3, &intel_ffh(), &[], INTEL); assert_eq!(d.processor_uid, 3); assert_eq!(kept(&d), [(IdleEntry::Mwait as u8, 0x31, 253, 506), (IdleEntry::Mwait as u8, 0x60, 1048, 2096)]); assert!(withheld.is_empty(), "{withheld:?}"); } #[test] - fn the_amds_c2_is_handed_over_and_its_type_3_state_withheld() { - let (d, withheld) = translate(0, &amd(), &[0], false); - assert_eq!(kept(&d), [(IdleEntry::Port as u8, 0x414, 400, 800)]); - assert_eq!(withheld, ["a type-3 SystemIO state, whose caches OSPM keeps coherent (ACPI 6.5 §8.1.4)"]); + fn a_type_3_systemio_state_is_handed_over_from_amd_family_17h_and_on_hygon() { + let both = [(IdleEntry::Port as u8, 0x414, 400, 800), (IdleEntry::Port as u8, 0x415, 350, 700)]; + for family in [0x17, 0x19, 0x1A] { + let (d, withheld) = translate(0, &amd_system_io(), &[], Cpu { vendor: Vendor::Amd, family }); + assert_eq!((kept(&d), withheld), (both.to_vec(), vec![]), "AMD Family {family:#x}"); + } + let (d, withheld) = translate(0, &amd_system_io(), &[], Cpu { vendor: Vendor::Hygon, family: 0x18 }); + assert_eq!((kept(&d), withheld), (both.to_vec(), vec![]), "Hygon"); + } + + #[test] + fn a_type_3_systemio_state_is_withheld_before_amd_family_17h_and_off_amd_and_hygon() { + let c3 = "a type-3 SystemIO state, whose caches OSPM keeps coherent (ACPI 6.5 §8.1.4), off AMD Family 17h on and Hygon"; + for cpu in [Cpu { vendor: Vendor::Amd, family: 0x16 }, Cpu { vendor: Vendor::Amd, family: 0x15 }, Cpu { vendor: Vendor::Other, family: 0x17 }] { + let (d, withheld) = translate(0, &amd_system_io(), &[], cpu); + assert_eq!(kept(&d), [(IdleEntry::Port as u8, 0x414, 400, 800)], "{cpu:?}"); + assert_eq!(withheld, [c3], "{cpu:?}"); + } + let (d, withheld) = translate(0, &amd_system_io(), &[], INTEL); + assert_eq!(d.count, 0); + assert_eq!(withheld, ["a SystemIO state on Intel, a chipset's P_LVLx read"; 2]); } #[test] fn each_state_ospm_owes_work_around_is_withheld_by_name() { - let withheld = |s: CState, software: bool, intel: bool| { - let (d, why) = translate(0, &[s], if software { &[0] } else { &[] }, intel); + let withheld = |s: CState, software: bool, cpu: Cpu| { + let (d, why) = translate(0, &[s], if software { &[0] } else { &[] }, cpu); assert_eq!(d.count, 0); why }; - assert_eq!(withheld(state(reg(FFH, 1, 2, 0b11, 0x31), 2, 253), false, true), ["an Intel FFH state asking bus-master avoidance"]); - assert_eq!(withheld(state(reg(FFH, 2, 2, 0, 0x10), 2, 50), false, false), ["FFH vendor 2 class 2, an encoding no document in hand defines"]); - assert_eq!(withheld(state(reg(FFH, 1, 1, 0, 0x1816), 2, 50), false, true), ["FFH vendor 1 class 1, an encoding no document in hand defines"]); - assert_eq!(withheld(state(reg(SYSTEM_IO, 8, 0, 0, 0x1816), 2, 253), false, true), ["a SystemIO state on Intel, a chipset's P_LVLx read"]); - assert_eq!(withheld(state(reg(SYSTEM_IO, 8, 0, 1, 0x414), 2, 400), true, false), ["coordinated in software by _CSD"]); - assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 0x31), 4, 253), false, true), ["type 4, which Table 8.2 does not define"]); - assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 0x31), 2, 0), false, true), ["a latency of 0"]); - assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 1 << 32), 2, 10), false, true), ["an Intel FFH state whose hint sets reserved bits 63:32"]); - assert_eq!(withheld(state(reg(0, 8, 0, 1, 0x1000), 2, 10), false, false), ["a state in space 0x0, which no C-state is entered through"]); + assert_eq!(withheld(state(reg(FFH, 1, 2, 0b11, 0x31), 2, 253), false, INTEL), ["an Intel FFH state asking bus-master avoidance"]); + assert_eq!(withheld(state(reg(FFH, 2, 2, 0, 0x10), 2, 50), false, AMD_17H), ["FFH vendor 2 class 2, an encoding no document in hand defines"]); + assert_eq!(withheld(state(reg(FFH, 1, 1, 0, 0x1816), 2, 50), false, INTEL), ["FFH vendor 1 class 1, an encoding no document in hand defines"]); + assert_eq!(withheld(state(reg(SYSTEM_IO, 8, 0, 0, 0x1816), 2, 253), false, INTEL), ["a SystemIO state on Intel, a chipset's P_LVLx read"]); + assert_eq!(withheld(state(reg(SYSTEM_IO, 8, 0, 1, 0x415), 3, 350), true, AMD_17H), ["coordinated in software by _CSD"]); + assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 0x31), 4, 253), false, INTEL), ["type 4, which Table 8.2 does not define"]); + assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 0x31), 2, 0), false, INTEL), ["a latency of 0"]); + assert_eq!(withheld(state(reg(FFH, 1, 2, 1, 1 << 32), 2, 10), false, INTEL), ["an Intel FFH state whose hint sets reserved bits 63:32"]); + assert_eq!(withheld(state(reg(0, 8, 0, 1, 0x1000), 2, 10), false, AMD_17H), ["a state in space 0x0, which no C-state is entered through"]); } #[test] fn more_states_than_a_declaration_holds_are_withheld_past_the_last() { let many: Vec = (0..10).map(|i| state(reg(FFH, 1, 2, 1, 0x10 + i), 2, 10 + i)).collect(); - let (d, withheld) = translate(0, &many, &[], true); + let (d, withheld) = translate(0, &many, &[], INTEL); assert_eq!(d.count as usize, IDLE_STATES); assert_eq!(withheld.len(), 2); } diff --git a/system/acpiserver/src/main.rs b/system/acpiserver/src/main.rs index f1cd5bf894a..70c517fb9f3 100644 --- a/system/acpiserver/src/main.rs +++ b/system/acpiserver/src/main.rs @@ -71,6 +71,7 @@ mod aml; mod battery; +mod cpu; mod cst; mod devices; mod ec; @@ -135,9 +136,8 @@ struct Server<'a, K, P> { census: Census, /// What `_OSC` answered. negotiated: osc::Negotiated, - /// The processor is Intel's, whose `_OSC` interface a processor object - /// speaks ([`toyos_abi::processor_is_intel`]). - intel: bool, + /// The processor's vendor and family. + cpu: cpu::Cpu, /// The batteries and adapters the namespace names, which a Notify of an /// information change reads again. sources: (Vec, Vec), @@ -372,7 +372,7 @@ impl<'a, K: Kernel, P: Ports> Server<'a, K, P> { census: Census::default(), negotiated: osc::Negotiated::default(), idle: Vec::new(), - intel: toyos_abi::processor_is_intel(), + cpu: cpu::Cpu::here(), sources: (Vec::new(), Vec::new()), gpe_runs: BTreeMap::new(), delivered: ledger::Ledger::default(), @@ -460,7 +460,7 @@ impl<'a, K: Kernel, P: Ports> Server<'a, K, P> { fn negotiate(&mut self, names: Option) -> Option { let (names, aml) = (names?, self.aml.as_mut()?); let before = aml.interpreter.defined(); - self.negotiated = osc::negotiate_all(aml, &names, self.intel); + self.negotiated = osc::negotiate_all(aml, &names, self.cpu.vendor == cpu::Vendor::Intel); let names = self.settle(before, names)?; let aml = self.aml.as_ref()?; println!( @@ -483,7 +483,7 @@ impl<'a, K: Kernel, P: Ports> Server<'a, K, P> { self.idle = cst::processors(aml, names); let mut census = cst::Census::default(); for processor in &self.idle { - if cst::declare(aml, self.kernel, processor, self.intel, &mut census).is_err() { + if cst::declare(aml, self.kernel, processor, self.cpu, &mut census).is_err() { return; } } @@ -494,7 +494,7 @@ impl<'a, K: Kernel, P: Ports> Server<'a, K, P> { fn redeclare_idle(&mut self, object: &str) { let (Some(aml), Some(processor)) = (&mut self.aml, self.idle.iter().find(|p| p.path == object)) else { return }; let mut census = cst::Census::default(); - if cst::declare(aml, self.kernel, processor, self.intel, &mut census).is_ok() { + if cst::declare(aml, self.kernel, processor, self.cpu, &mut census).is_ok() { println!("{OWN}{object}'s idle states read again: {}", census.line()); } } @@ -1169,11 +1169,13 @@ mod tests { /// calls neither reserved nor NVS, is refused and keeps nothing. #[test] fn osc_commits_only_what_toyos_claims_and_what_its_load_adds_is_initialised() { - for (intel, block_type, loaded) in [(true, 0, true), (false, 0, false), (true, 9, false)] { + let (on_intel, on_amd) = (cpu::Cpu { vendor: cpu::Vendor::Intel, family: 6 }, cpu::Cpu { vendor: cpu::Vendor::Amd, family: 0x17 }); + for (cpu, block_type, loaded) in [(on_intel, 0, true), (on_amd, 0, false), (on_intel, 9, false)] { + let intel = cpu == on_intel; let kernel = osc_machine(block_type); let info = AcpiInfo { rsdp: CRAFTED_RSDP, pm1_event: Block::NONE, gpe0: GPE0, flags: FIXED_POWER_BUTTON, reserved: [0; 3] }; let mut server = Server::new(&kernel, Bytes::default(), info); - server.intel = intel; + server.cpu = cpu; server.aml = load(&kernel, CRAFTED_RSDP).1; let names = server.initialise(); server.arm_controller(names.as_ref()); @@ -1217,7 +1219,7 @@ mod tests { let kernel = crafted(&sealed(b"DSDT", &cat(&[&processor("\\_PR.CPU0", 2), &processor("\\_PR.CPU1", 5)])), &[]); let info = AcpiInfo { rsdp: CRAFTED_RSDP, pm1_event: Block::NONE, gpe0: GPE0, flags: FIXED_POWER_BUTTON, reserved: [0; 3] }; let mut server = Server::new(&kernel, Bytes::default(), info); - server.intel = true; + server.cpu = cpu::Cpu { vendor: cpu::Vendor::Intel, family: 6 }; server.aml = load(&kernel, CRAFTED_RSDP).1; let names = server.initialise().expect("the namespace stands"); server.declare_idle(&names); diff --git a/tests/guest/rust-tests/src/bin/idle_entered.rs b/tests/guest/rust-tests/src/bin/idle_entered.rs new file mode 100644 index 00000000000..d5346dceb92 --- /dev/null +++ b/tests/guest/rust-tests/src/bin/idle_entered.rs @@ -0,0 +1,62 @@ +//! Every CPU's deep idle periods across a span of an idle machine, once +//! `/system/bin/acpiserver` has handed the kernel its processors' idle +//! states, printed for the `idle_entered` metal row to judge +//! (`tests/toyos.rs`). It asserts nothing: whether a CPU chose a declared +//! state is the kernel's count to say. +//! +//! **A period is counted once the CPU is back from it**, after the +//! interrupt that ended it was handled, and the read that ends the span is +//! itself a kick to every CPU: a CPU that waited deep through the whole span +//! is counted only by a later round. So the span ends with a read that wakes +//! every CPU, and the periods are those a fresh round after it counts. + +use std::time::{Duration, Instant}; + +use toyos::endow::{Endowments, SYSCAP_LABEL}; +use toyos::syscap::SysCap; +use toyos_abi::counters::{Counter, RawRecord, Record}; +use toyos_abi::syscall; +use toyos_logstream::program_line; + +#[path = "../served_log.rs"] +mod served_log; + +/// The idle span: a thousand residencies of a 1 ms state. +const IDLE: Duration = Duration::from_secs(1); + +/// How long after boot the wait for the server's census gives up: the +/// runner's bound less a tenth, as `acpi_hold`'s. +const UNTIL_MS: u64 = toyos_tco::JOB_BOUND_MS - toyos_tco::JOB_BOUND_MS / 10; + +fn round(cap: &SysCap) -> Vec { + let mut raw = vec![RawRecord::EMPTY; syscall::cpu_count() as usize]; + let n = cap.counters(&mut raw).expect("the estate's capability reads the counters"); + raw[..n].iter().map(|r| Record::decode(r).expect("a record that decodes")).collect() +} + +fn main() { + let cap: SysCap = Endowments::get().take(SYSCAP_LABEL).expect("test-runner endows a capability"); + let since_boot = toyos_abi::clock::nanos_since_boot() / 1_000_000; + served_log::Log::open().until("the ACPI server's idle states", Duration::from_millis(UNTIL_MS.saturating_sub(since_boot)), |line| { + program_line(line).is_some_and(|said| said.tag == "acpiserver" && said.text.starts_with(acpiserver_api::IDLE_CENSUS)) + }); + let before = round(&cap); + let began = Instant::now(); + std::thread::sleep(IDLE); + round(&cap); + let span = began.elapsed(); + // A round issued after this call began, at least `INTERVAL_NS` after + // the waking round's (`kernel::counters`): every period that round ended is + // in it. + let after = round(&cap); + for (b, a) in before.iter().zip(&after) { + let periods = a.get(Counter::DeepIdles).zip(b.get(Counter::DeepIdles)).map(|(a, b)| a - b); + let Some(periods) = periods else { panic!("cpu{} carries no deep idle count: {b:?} then {a:?}", a.cpu) }; + println!( + "idle_entered: cpu{} {periods} deep idle periods in {} ns{}", + a.cpu, + span.as_nanos(), + if a.stale || b.stale { ", stale" } else { "" }, + ); + } +} diff --git a/tests/toyos.rs b/tests/toyos.rs index 56c71fdc049..0d9aebe0615 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -142,6 +142,10 @@ const RUST_SKIP: &[&str] = &[ // every CPU, which the `counters` metal row judges; on a guest it would be // seconds of four CPUs spinning, read by nothing. "counters_metal", + // Its product is each CPU's deep idle periods once the ACPI server has + // declared the firmware's idle states. A guest's idle is its host's: the + // `idle_entered` metal row judges it. + "idle_entered", // Its product is what a diary record costs, which the `trace_record_cost` // metal row reads off the test kernel its flood needs. "trace_flood", @@ -886,6 +890,17 @@ const METAL: &[(&str, metal::Metal)] = &[ judge: |b| acpi_tables_on_metal(b[0]), }, ), + ( + // The AMD bench's declared idle states as every CPU keeps them, and + // each CPU in one of them across an idle second: + // `idle_entered_on_metal` says what is held. The same boot as + // `acpi_tables_loaded`. + "idle_entered", + metal::Metal { + arms: &[metal::Arm { jobs: &["test_rs_idle_entered"], ..TESTCASES_HELD[0] }], + judge: |b| idle_entered_on_metal(b[0]), + }, + ), ( // The T14's battery and AC adapter as the server read them through // its AML and its embedded controller: `battery_on_metal` says what @@ -6730,9 +6745,9 @@ const T14: Bench = Bench { /// table defines, refused whole as the owner ruled /// (`issues/the-amd-benchs-ssdt8-scopes-a-controller-no-table-defines.md`); /// its `\_S5`'s `SLP_TYPa` as the check outside the tree evaluated it; no -/// processor names an `_OSC`; each `_CST` hands over its port 0x414 and -/// withholds its type-3 0x415 -/// (`issues/the-amd-benchs-c3-is-withheld-for-want-of-a-coherence-statement.md`). Its controller's `_REG(3, 1)` and its +/// processor names an `_OSC`; each `_CST` hands over its port 0x414 and its +/// type-3 0x415, a Family 17h part's (`system/acpiserver/src/cst.rs`). Its +/// controller's `_REG(3, 1)` and its /// adapter's `_PSR` each reach `ALIB`, whose mailbox writes the root /// complex's 0xB8 and 0xBC through ECAM, in 00:00.0's page at 0xF8000000. const AMD: Bench = Bench { @@ -6741,7 +6756,7 @@ const AMD: Bench = Bench { refused: &[(9, "a name that does not resolve")], s5: "with SLP_TYPa=5,", loaded: &[], - idle: " with port 0x414 400 µs; withheld: a type-3 SystemIO state, whose caches OSPM keeps coherent (ACPI 6.5 §8.1.4) x16; unread: none", + idle: " with port 0x414 400 µs + port 0x415 350 µs; withheld: none; unread: none", kept: &[], first_written: &[At::Address(0xF800_0000)], }; @@ -6853,7 +6868,7 @@ fn acpi_tables_on_metal(back: &metal::Readback) -> Result<(), String> { } eprintln!(" [acpi] {}", processors.trim()); } - let idle = log.must_say("acpiserver: the processors' idle states: ")?; + let idle = log.must_say(acpiserver_api::IDLE_CENSUS)?; if !idle.contains(" 0 refused by the kernel;") || !idle.trim_end().ends_with(bench.idle) { return Err(format!("the processors' idle states are not this bench's, or the kernel refused one: {idle}")); } @@ -6879,6 +6894,54 @@ fn acpi_tables_on_metal(back: &metal::Readback) -> Result<(), String> { Ok(()) } +/// The end of the kernel's line for each of the AMD bench's CPUs, its `_CST` +/// kept whole. +const AMD_KEPT_STATES: &str = "waits in C1 or port 0x414 (exit 400 µs, residency 800 µs), port 0x415 (exit 350 µs, residency 700 µs)"; + +/// The AMD bench's idle states kept on every CPU, and each CPU in one across +/// an idle second: once the server's census is in the log, `idle_entered` +/// reads each CPU's deep idle periods across the second, those the read +/// ending it woke the CPU from included. Held: the kernel's line for every CPU +/// the bring-up started names [`AMD_KEPT_STATES`], each CPU's reads fresh, +/// and each CPU chose a declared state at least once in the second. Every such period is a +/// request of port 0x415: the kernel chooses the deepest state whose exit and +/// residency fit and whose history admits it (`kernel::sched::idle::choose`), +/// and 0x415's exit and residency are both below 0x414's. What the core then +/// entered no counter on this part says. The AMD bench alone: the T14 has not +/// run it. +fn idle_entered_on_metal(back: &metal::Readback) -> Result<(), String> { + back.job_passed("test_rs_idle_entered")?; + let kernel = back.kernel(); + if !kernel.text().contains(AMD.row) { + return Err("this row runs on the AMD bench, and this machine's ACPI row is not that bench's".into()); + } + let cpus = back.cpus()?; + for cpu in 0..cpus { + let kept = kernel.must_say(&format!("idle: cpu{cpu} (processor UID "))?; + if !kept.trim_end().ends_with(AMD_KEPT_STATES) { + return Err(format!("cpu{cpu} keeps {kept:?}, where the bench declares {AMD_KEPT_STATES:?}")); + } + } + let log = back.log(); + let entered: Vec<&str> = log + .text() + .lines() + .filter_map(|l| toyos_logstream::program_line(l)?.text.strip_prefix("idle_entered: ").map(str::trim)) + .collect(); + if entered.len() != cpus as usize { + return Err(format!("idle_entered read {} CPUs, and the bring-up started {cpus}: {entered:#?}", entered.len())); + } + for (cpu, said) in entered.iter().enumerate() { + let periods = number_between(said, &format!("cpu{cpu} "), " deep idle periods")?; + if periods == 0 || said.ends_with(", stale") { + return Err(format!("cpu{cpu} chose no declared state across the idle second, or was read stale: {said}")); + } + eprintln!(" [idle] {said}"); + } + eprintln!(" [idle] every CPU {AMD_KEPT_STATES}"); + Ok(()) +} + /// What the T14's tables name, by a census of them read outside the tree: /// one control-method battery, one AC adapter, and one embedded controller, /// attached to the server's AML.