diff --git a/issues/kernel/the-kernel-owns-cpu-performance-state.md b/issues/kernel/the-kernel-owns-cpu-performance-state.md new file mode 100644 index 00000000000..81d3c82caa7 --- /dev/null +++ b/issues/kernel/the-kernel-owns-cpu-performance-state.md @@ -0,0 +1,43 @@ +--- +status: open +kind: track +opened: 2026-09-28 +--- + +# The kernel owns CPU performance state + +The self-hosting bar is measured under a fixed power envelope that must be +read back for a whole build span. The kernel declares the envelope from the +one CPU-state declaration (`kernel/src/arch/x86_64/control_regs.rs`) and +`SYS_COUNTERS` reads it back per CPU, under `Rights::TRACE`. + +**A register reaches a read only once boot has proven it**: the kernel has no +`rdmsr` fault fixup, so a register a read is the first to touch is a kernel +`#GP` a holder of the right can cause. Each register is enumerated by CPUID +or read at boot by the declaration's own check before a read can reach it. + +Declared and read back: the HWP request, its package's and the energy/ +performance bias, on every CPU that has all of them +(`toyos_cpuvuln::hwp`). Still to build: + +- **RAPL**: PL1, PL2 and their windows through `MSR_PKG_POWER_LIMIT` and the + MMIO mirror in the host bridge's MCHBAR, at the bar's values; a limit + firmware locked (bit 63) is refused by name. Neither register is + enumerated by CPUID. *Exit*: the T14 reads both back at the bar's values. +- **Turbo**: `IA32_MISC_ENABLE` bit 38 is firmware's. Its other bits are + model-specific, so declaring the one bit needs the owner's ruling on + writing that register whole. *Exit*: the ruling, and the bit declared or + recorded as firmware's. +- **The sampler**: a program that reads the envelope every 60 s and at a + span's start and end, with package power from `MSR_PKG_ENERGY_STATUS` and + temperature from `IA32_PACKAGE_THERM_STATUS` and `MSR_TEMPERATURE_TARGET`, + the bar's validity conditions. The energy counter is a side channel + (CVE-2020-8694), under `Rights::TRACE` like the rest. *Exit*: one valid + span on the T14. + +Not covered: hybrid Intel and AMD CPPC, +`issues/kernel/the-perf-state-declaration-refuses-hybrid-intel-and-amd-cppc.md`; +AArch64 declares nothing. No QEMU CPU enumerates HWP, so every write and read +of these registers runs only on the T14. + +Owner: the orchestrator, which holds the T14. diff --git a/issues/kernel/the-perf-state-declaration-refuses-hybrid-intel-and-amd-cppc.md b/issues/kernel/the-perf-state-declaration-refuses-hybrid-intel-and-amd-cppc.md new file mode 100644 index 00000000000..92b339856ab --- /dev/null +++ b/issues/kernel/the-perf-state-declaration-refuses-hybrid-intel-and-amd-cppc.md @@ -0,0 +1,31 @@ +--- +status: open +kind: defect +opened: 2026-09-29 +--- + +# The perf-state declaration refuses hybrid Intel and AMD CPPC + +Every Intel client CPU since Alder Lake is hybrid, and AMD CPUs name their +performance controls through CPPC, not HWP. On both the kernel declares no +performance request, firmware's values stand, and `SYS_COUNTERS` carries no +power envelope. + +The refusals are `toyos_cpuvuln::hwp`'s (`toyos-cpuvuln/src/hwp.rs`): +- `HwpRefusal::Hybrid`, for CPUID.(7,0):EDX[15]: the declared minimum is a + ratio (`MSR_PLATFORM_INFO[47:40]`), and a hybrid CPU's HWP scale is not its + ratio scale. +- `HwpRefusal::NoHwp`, for CPUID.6:EAX[7] clear, which is where an AMD CPU + with CPPC is refused; its controls are not read. +- `HwpRefusal::NotIntel`, for HWP on any vendor but Intel, because the + minimum comes from `MSR_PLATFORM_INFO`, which is Intel's. + +`control_regs::hwp_declared` logs the refusal once and declares nothing on +any CPU. + +Owner: the orchestrator. **Exit**: on a hybrid Intel CPU each core type's +request is declared on that core's own HWP scale; on an AMD CPU with CPPC, +`MSR_AMD_CPPC_ENABLE` and `MSR_AMD_CPPC_REQ` are declared from its +`MSR_AMD_CPPC_CAP1` and asserted on every CPU; and on one machine of each +kind the `counters` metal row holds the envelope it reads back to the +declaration. diff --git a/issues/kernel/the-scheduler-and-the-clocks-never-talk.md b/issues/kernel/the-scheduler-and-the-clocks-never-talk.md index c9a9165e011..752d501f70f 100644 --- a/issues/kernel/the-scheduler-and-the-clocks-never-talk.md +++ b/issues/kernel/the-scheduler-and-the-clocks-never-talk.md @@ -21,6 +21,8 @@ Two couplings every production scheduler has and ours lacks: Both wait for an AML interpreter (`issues/kernel/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md`), -because today the kernel neither reads nor sets P-states or C-states on metal. +because the kernel reads no C-state on metal and chooses no P-state: it +declares HWP's bounds (`kernel/src/arch/x86_64/control_regs.rs`), and the CPU +chooses within them. Filed so the placement track (see `placement-is-blind-to-caches-and-topology.md`) leaves room for an idle-state input when it designs wake placement. diff --git a/issues/kernel/toyos-runs-the-t14-under-load-below-the-clock-linux-reaches.md b/issues/kernel/toyos-runs-the-t14-under-load-below-the-clock-linux-reaches.md deleted file mode 100644 index 8a74e88ce38..00000000000 --- a/issues/kernel/toyos-runs-the-t14-under-load-below-the-clock-linux-reaches.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-10-04 ---- - -# ToyOS runs the T14 under load below the clock Linux reaches - -The `counters` metal row, at `ce1786ff0` (pull request #705), spun one thread -per CPU for its `spin` phase and read every CPU's busy clock as 2318 MHz -(TSC 2419 MHz times APERF over MPERF, MPERF at 0.986 to 0.990 of the stamp, -so every CPU was in C0 the whole span). Linux's turbostat on the same machine -under one `yes` per CPU (`tests/t14-linux/turbostat-loaded.txt`) reads -3800 MHz for its first 20 s and 3075 to 3094 MHz once the package settles near -20 W. ToyOS runs a loaded T14 at 61 to 75% of Linux's busy clock, below the -TSC's own rate. - -The CPU enumerates HWP (`hwp`, `hwp_epp` in -`toyos-cpuvuln/fixtures/t14/cpuinfo.txt`), and the kernel writes neither -`IA32_PM_ENABLE` nor `IA32_PERF_CTL`: the clock is whatever the firmware left. -`issues/kernel/the-scheduler-and-the-clocks-never-talk.md` is the track that -would couple the scheduler to frequency; this is the floor beneath it, a -machine that never leaves the firmware's operating point. - -Owner: the orchestrator, which holds the T14. **Exit**: the `counters` row -holds every CPU's busy clock across its spin to -Linux's loaded range in `tests/t14-linux/`, and the reading that held it is in -the closing pull request. diff --git a/kernel/src/arch/aarch64/counters.rs b/kernel/src/arch/aarch64/counters.rs index b808d6c166b..38e9eeb8333 100644 --- a/kernel/src/arch/aarch64/counters.rs +++ b/kernel/src/arch/aarch64/counters.rs @@ -1,10 +1,11 @@ //! No model-specific counter is read on this architecture: the activity -//! monitors' cycle counters (`FEAT_AMUv1`) are unread. +//! monitors' cycle counters (`FEAT_AMUv1`) are unread, and no performance +//! request is declared. use crate::counters::Hardware; pub fn bring_up() {} pub fn read() -> Hardware { - Hardware { smi: None, aperf: None, mperf: None } + Hardware { smi: None, aperf: None, mperf: None, envelope: None } } diff --git a/kernel/src/arch/x86_64/boot.rs b/kernel/src/arch/x86_64/boot.rs index a606ea3f475..b58d961a7f5 100644 --- a/kernel/src/arch/x86_64/boot.rs +++ b/kernel/src/arch/x86_64/boot.rs @@ -119,8 +119,7 @@ pub fn clock(args: &KernelArgs) { /// and the TSC is time since power-on. pub fn report_counter_origin() { const IA32_TSC_ADJUST: u32 = 0x3b; - // Leaf 7 exists when the maximum leaf reaches it. - if super::cpu::cpuid(0, 0).0 < 7 || super::cpu::cpuid(7, 0).1 & (1 << 1) == 0 { + if super::cpu::leaf_7().1 & (1 << 1) == 0 { log!("boot: IA32_TSC_ADJUST not on this CPU"); return; } diff --git a/kernel/src/arch/x86_64/control_regs.rs b/kernel/src/arch/x86_64/control_regs.rs index d1c42c00d0d..81267b30008 100644 --- a/kernel/src/arch/x86_64/control_regs.rs +++ b/kernel/src/arch/x86_64/control_regs.rs @@ -1,11 +1,22 @@ -//! What `CR0`, `CR4` and `IA32_EFER` hold on every CPU in this machine. One -//! declaration, applied by the BSP and every AP and checked on each; nothing -//! else may write any of the three. Each register is written whole: `CR0` -//! and `EFER` are constants, `CR4` is required bits plus whatever optional -//! bits this CPU offers. `EFER.NXE` lets bit 63 of a paging entry mean *not -//! executable* ([`Prot`](crate::mm::policy::Prot)). - -use core::sync::atomic::{AtomicU64, Ordering}; +//! What `CR0`, `CR4`, `IA32_EFER` and the performance request hold on every +//! CPU in this machine. One declaration, applied by the BSP and every AP and +//! checked on each; nothing else may write any of them. Each register is +//! written whole: `CR0` and `EFER` are constants, `CR4` is required bits plus +//! whatever optional bits this CPU offers. `EFER.NXE` lets bit 63 of a paging +//! entry mean *not executable* ([`Prot`](crate::mm::policy::Prot)). +//! +//! The performance request is HWP's: `IA32_HWP_INTERRUPT` where it exists, +//! `IA32_PM_ENABLE`, `IA32_HWP_REQUEST` as `toyos_cpuvuln::hwp_request` derives +//! it from this CPU's own registers, `IA32_HWP_REQUEST_PKG` and +//! `IA32_ENERGY_PERF_BIAS`, on a machine whose CPUs have every one of them, +//! and none on one that does not: refused by name once, firmware's values +//! standing. `IA32_MISC_ENABLE`'s turbo bit is not declared: that register's +//! other bits are model-specific and firmware's, and writing it whole would +//! decide them. + +use core::sync::atomic::{AtomicU64, AtomicU8, Ordering}; + +use toyos_cpuvuln::HwpFacts; use super::cpu; use crate::log; @@ -47,6 +58,33 @@ mod cr4 { pub const SMAP: u64 = 1 << 21; } +/// The performance request's registers, SDM Vol. 4. +mod hwp { + pub const PLATFORM_INFO: u32 = 0xCE; + pub const ENERGY_PERF_BIAS: u32 = 0x1B0; + pub const PM_ENABLE: u32 = 0x770; + pub const HWP_CAPABILITIES: u32 = 0x771; + pub const HWP_REQUEST_PKG: u32 = 0x772; + pub const HWP_INTERRUPT: u32 = 0x773; + pub const HWP_REQUEST: u32 = 0x774; +} + +/// `IA32_PM_ENABLE` on every CPU: HWP on, which only a reset turns off. +const PM_ENABLE: u64 = 1; + +/// `IA32_HWP_INTERRUPT` on every CPU that has it: nothing in this kernel takes +/// an HWP notification. +const HWP_INTERRUPT: u64 = 0; + +/// `IA32_HWP_REQUEST_PKG` on every CPU: the widest range at the request's own +/// preference. Inert, since no CPU's request sets package control, and +/// declared so that firmware does not decide it either. +const HWP_REQUEST_PKG: u64 = 0x01 | 0xff << 8 | toyos_cpuvuln::HWP_EPP << 24; + +/// `IA32_ENERGY_PERF_BIAS` on every CPU: Linux's `ENERGY_PERF_BIAS_NORMAL` +/// (`arch/x86/include/asm/msr-index.h:858`). +const ENERGY_PERF_BIAS: u64 = 6; + /// `CR0` on every CPU: `TS` stays clear because lazy FP switching would leak /// a register file across `#NM`; `AM` stays clear because Ring 3 has no `#AC` path. pub const CR0: u64 = cr0::PE | cr0::MP | cr0::ET | cr0::NE | cr0::WP | cr0::PG; @@ -106,8 +144,45 @@ pub fn init_cr0(cpu_id: u32) { bench::report(cpu_id, before); } -/// Puts this CPU's `CR4` and `EFER` into the declaration and checks all -/// three against it. Must run after [`init_cr0`] and before `arch::syscall::init`, which needs `SCE` set. +/// Whether the machine's declaration carries the performance request: the +/// BSP's verdict, which every AP must reach too. +static HWP: AtomicU8 = AtomicU8::new(HWP_UNDECIDED); +const HWP_UNDECIDED: u8 = 0; +const HWP_DECLARED: u8 = 1; +const HWP_REFUSED: u8 = 2; + +/// Whether this CPU gets a performance request, from CPUID alone, and whether +/// it has `IA32_HWP_INTERRUPT`: the machine's answer, a CPU that disagrees +/// named. +fn hwp_declared(cpu_id: u32) -> Option { + let (cpuid_6_eax, cpuid_6_ecx) = cpu::leaf_6(); + let verdict = toyos_cpuvuln::hwp(&HwpFacts { + vendor: cpu::vendor(), + signature: cpu::cpuid(1, 0).0, + cpuid_6_eax, + cpuid_6_ecx, + cpuid_7_0_edx: cpu::leaf_7().3, + }); + let mine = if verdict.is_ok() { HWP_DECLARED } else { HWP_REFUSED }; + match HWP.compare_exchange(HWP_UNDECIDED, mine, Ordering::Release, Ordering::Acquire) { + Ok(_) => { + if let Err(refusal) = verdict { + log!("control_regs: no performance request is declared: {}", refusal.reason()); + } + } + Err(machine) => assert!( + machine == mine, + "control_regs: cpu{cpu_id} {} a performance request and the BSP {} one", + if mine == HWP_DECLARED { "can hold" } else { "cannot hold" }, + if machine == HWP_DECLARED { "declared" } else { "refused" }, + ), + } + verdict.ok() +} + +/// Puts this CPU's `CR4` and `EFER` into the declaration and checks both +/// against it. Must run after [`init_cr0`] and before `arch::syscall::init`, +/// which needs `SCE` set. pub fn init(cpu_id: u32) { let declared = declaration(cpu_id); if !skipped(cpu_id) { @@ -128,6 +203,47 @@ pub fn init(cpu_id: u32) { self_check(cpu_id, declared); } +/// Puts this CPU's performance request into the declaration and checks it. +/// Must run after [`init`] and after this CPU's IDT is loaded, so a fault in +/// its `wrmsr` or `rdmsr` panics by name. +/// +/// The interrupt is cleared and HWP enabled before any other HWP register is +/// touched, the request's input included: SDM Vol. 3B §17.4.2 (253669-093US), +/// "Additional MSRs associated with HWP may only be accessed after HWP is +/// enabled, with the exception of IA32_HWP_INTERRUPT and MSR_PPERF", and +/// `intel_pstate_hwp_enable`'s order. +pub fn init_performance(cpu_id: u32) { + let Some(hwp) = hwp_declared(cpu_id) else { return }; + // SAFETY: `hwp_declared` admitted this CPU only where CPUID enumerates HWP, + // and `IA32_HWP_INTERRUPT` where it enumerates that too; both values hold + // only defined bits. + unsafe { + if hwp.notify { + cpu::wrmsr(hwp::HWP_INTERRUPT, HWP_INTERRUPT); + } + cpu::wrmsr(hwp::PM_ENABLE, PM_ENABLE); + } + let request = toyos_cpuvuln::hwp_request(cpu::rdmsr(hwp::HWP_CAPABILITIES), cpu::rdmsr(hwp::PLATFORM_INFO)); + // SAFETY: HWP is enabled, so its registers are live; CPUID enumerated EPP, + // the package request and EPB, and every value holds only defined bits. + unsafe { + cpu::wrmsr(hwp::HWP_REQUEST, request); + cpu::wrmsr(hwp::HWP_REQUEST_PKG, HWP_REQUEST_PKG); + cpu::wrmsr(hwp::ENERGY_PERF_BIAS, ENERGY_PERF_BIAS); + } + hwp_check(cpu_id, request, hwp.notify); +} + +/// The power envelope this CPU runs under, where the performance request is +/// declared: read only after this CPU's [`init_performance`], which enabled HWP. +pub fn envelope() -> Option { + (HWP.load(Ordering::Acquire) == HWP_DECLARED).then(|| crate::counters::Envelope { + hwp_request: cpu::rdmsr(hwp::HWP_REQUEST), + hwp_request_pkg: cpu::rdmsr(hwp::HWP_REQUEST_PKG), + energy_perf_bias: cpu::rdmsr(hwp::ENERGY_PERF_BIAS), + }) +} + /// Whether the declaration carries `PCIDE`, and therefore whether `INVPCID` is this machine's flush. pub fn pcid_active() -> bool { DECLARED_CR4.load(Ordering::Acquire) & cr4::PCIDE != 0 @@ -205,11 +321,8 @@ fn supported() -> u64 { [(0, cr4::FSGSBASE), (7, cr4::SMEP), (20, cr4::SMAP)]; const CPUID_7_ECX: [(u32, u64); 1] = [(2, cr4::UMIP)]; - let (max_leaf, _, _, _) = cpu::cpuid(0, 0); let (_, _, ecx1, edx1) = cpu::cpuid(1, 0); - // A leaf above the maximum answers with the highest basic leaf's data - // instead of faulting, so an unguarded read here would misreport bits. - let (_, ebx7, ecx7, _) = if max_leaf >= 7 { cpu::cpuid(7, 0) } else { (0, 0, 0, 0) }; + let (_, ebx7, ecx7, _) = cpu::leaf_7(); let mut have = 0; for (bit, flag) in CPUID_1_EDX { @@ -272,6 +385,42 @@ fn self_check(cpu_id: u32, declared_cr4: u64) { CHECKED.fetch_add(1, Ordering::Relaxed); } +/// [`self_check`] for the performance request, logged first for the same +/// reason; the line carries the request's two inputs, so a reader can +/// recompute it. +fn hwp_check(cpu_id: u32, request: u64, notify: bool) { + let pm_enable = cpu::rdmsr(hwp::PM_ENABLE); + let live = cpu::rdmsr(hwp::HWP_REQUEST); + let pkg = cpu::rdmsr(hwp::HWP_REQUEST_PKG); + let epb = cpu::rdmsr(hwp::ENERGY_PERF_BIAS); + let interrupt = notify.then(|| cpu::rdmsr(hwp::HWP_INTERRUPT)); + log!( + "control_regs: cpu{} pm_enable={} hwp_request={:#010x} hwp_request_pkg={:#010x} epb={} \ + hwp_interrupt={:x?} hwp_capabilities={:#010x} platform_info={:#018x}", + cpu_id, + pm_enable, + live, + pkg, + epb, + interrupt, + cpu::rdmsr(hwp::HWP_CAPABILITIES), + cpu::rdmsr(hwp::PLATFORM_INFO), + ); + for (name, holds, declared) in [ + ("pm_enable", Some(pm_enable), PM_ENABLE), + ("hwp_request", Some(live), request), + ("hwp_request_pkg", Some(pkg), HWP_REQUEST_PKG), + ("epb", Some(epb), ENERGY_PERF_BIAS), + ("hwp_interrupt", interrupt, HWP_INTERRUPT), + ] { + let Some(holds) = holds else { continue }; + assert!( + holds == declared, + "control_regs: cpu{cpu_id} holds {name}={holds:#x}, the declaration is {declared:#x}", + ); + } +} + /// How many CPUs hold the declaration, said once after the last of them has /// been checked. A divergent CPU panics inside [`self_check`], so what this /// line adds is the *count*: a CPU that never reached [`init`] at all is diff --git a/kernel/src/arch/x86_64/counters.rs b/kernel/src/arch/x86_64/counters.rs index efbc1eb891e..4c4b25dae3b 100644 --- a/kernel/src/arch/x86_64/counters.rs +++ b/kernel/src/arch/x86_64/counters.rs @@ -1,9 +1,10 @@ //! Which of this CPU's model-specific counters it reads: decided on the CPU -//! itself at bring-up, by `toyos_cpuvuln::counters` over its own CPUID. +//! itself at bring-up, by `toyos_cpuvuln::counters` over its own CPUID, and +//! the power envelope where `control_regs` declared it. use core::sync::atomic::{AtomicU8, Ordering::Relaxed}; -use toyos_cpuvuln::{CounterFacts, Vendor}; +use toyos_cpuvuln::CounterFacts; use super::{cpu, percpu}; use crate::counters::Hardware; @@ -20,13 +21,10 @@ const APERF_MPERF: u8 = 1 << 1; static ADMITTED: [AtomicU8; MAX_CPUS] = [const { AtomicU8::new(0) }; MAX_CPUS]; pub fn bring_up() { - let (max_leaf, ebx, ecx, edx) = cpu::cpuid(0, 0); - let id: [u8; 12] = core::array::from_fn(|i| [ebx, edx, ecx][i / 4].to_le_bytes()[i % 4]); let (signature, _, cpuid_1_ecx, _) = cpu::cpuid(1, 0); - // As `init_scattered_cpuid_features` reads it: a level past the range's own is no level. - let cpuid_6_ecx = if (6..=0xFFFF).contains(&max_leaf) { cpu::cpuid(6, 0).2 } else { 0 }; + let (_, cpuid_6_ecx) = cpu::leaf_6(); let verdict = toyos_cpuvuln::counters(&CounterFacts { - vendor: Vendor::from_id(&id), + vendor: cpu::vendor(), signature, cpuid_1_ecx, cpuid_6_ecx, @@ -44,5 +42,6 @@ pub fn read() -> Hardware { smi: msr(SMI, MSR_SMI_COUNT).map(|count| count & 0xFFFF_FFFF), aperf: msr(APERF_MPERF, IA32_APERF), mperf: msr(APERF_MPERF, IA32_MPERF), + envelope: super::control_regs::envelope(), } } diff --git a/kernel/src/arch/x86_64/cpu.rs b/kernel/src/arch/x86_64/cpu.rs index 6e67f58bbf2..0f514f69b12 100644 --- a/kernel/src/arch/x86_64/cpu.rs +++ b/kernel/src/arch/x86_64/cpu.rs @@ -127,6 +127,31 @@ pub fn df_witness(site: &str) { ); } +/// CPUID.0's vendor string, as `toyos_cpuvuln` names it. +pub fn vendor() -> toyos_cpuvuln::Vendor { + let (_, ebx, ecx, edx) = cpuid(0, 0); + toyos_cpuvuln::Vendor::from_id(&core::array::from_fn(|i| [ebx, edx, ecx][i / 4].to_le_bytes()[i % 4])) +} + +/// CPUID.6's EAX and ECX, the thermal and power leaf; zero where CPUID.0 +/// does not reach it, as `init_scattered_cpuid_features` reads it: a level +/// past the range's own is no level. +pub fn leaf_6() -> (u32, u32) { + if (6..=0xFFFF).contains(&cpuid(0, 0).0) { + let (eax, _, ecx, _) = cpuid(6, 0); + (eax, ecx) + } else { + (0, 0) + } +} + +/// CPUID.(7,0)'s four registers, the structured extended features; zero +/// where CPUID.0 does not reach leaf 7, since a leaf above the maximum answers +/// with the highest basic leaf's data instead of faulting. +pub fn leaf_7() -> (u32, u32, u32, u32) { + if cpuid(0, 0).0 >= 7 { cpuid(7, 0) } else { (0, 0, 0, 0) } +} + /// CPUID with both index registers; `rbx` is saved by hand since Rust reserves it as an operand. pub fn cpuid(leaf: u32, subleaf: u32) -> (u32, u32, u32, u32) { let eax: u32; diff --git a/kernel/src/arch/x86_64/percpu.rs b/kernel/src/arch/x86_64/percpu.rs index 07cc73c15a7..5bdfdf2bea3 100644 --- a/kernel/src/arch/x86_64/percpu.rs +++ b/kernel/src/arch/x86_64/percpu.rs @@ -518,6 +518,7 @@ pub fn init_bsp(lapic_id: u32) { // handlers report on no channel of this kernel's — so a fault in `fpu` // below would stop the machine with the panel holding the record before it. super::idt::init(); + super::control_regs::init_performance(0); super::fpu::init(0); // Between `fpu::init` and this function's own line: the facts `fpu::init` @@ -550,6 +551,7 @@ pub fn init_ap(percpu_ptr: *mut PerCpu) { // SAFETY: `load_gdt`'s once-per-CPU contract; this is this AP's call. unsafe { percpu.load_gdt(); } super::control_regs::init(percpu.cpu_id); + super::control_regs::init_performance(percpu.cpu_id); super::fpu::init(percpu.cpu_id); super::fpu::log_state(percpu.cpu_id); } diff --git a/kernel/src/counters.rs b/kernel/src/counters.rs index b0a43211113..7d7cd8c953b 100644 --- a/kernel/src/counters.rs +++ b/kernel/src/counters.rs @@ -12,7 +12,8 @@ //! [`JOIN`]. //! //! A model-specific counter is read only where the CPU's own verdict at -//! [`bring_up`] admitted it, and that bring-up's sample reads it once: a wrong +//! [`bring_up`] admitted it, and the power envelope only where the CPU-state +//! declaration declared it; that bring-up's sample reads each once, so a wrong //! verdict is `#GP` at boot, never inside a read. use kernel::sched::task::WaitClass; @@ -48,6 +49,15 @@ pub struct Hardware { pub smi: Option, pub aperf: Option, pub mperf: Option, + pub envelope: Option, +} + +/// The power envelope's registers, where the CPU's performance request is +/// declared. +pub struct Envelope { + pub hwp_request: u64, + pub hwp_request_pkg: u64, + pub energy_perf_bias: u64, } /// A block's words: the hardware id, a bit per counter it holds, and a word @@ -68,8 +78,19 @@ pub fn bring_up() { let me = percpu::cpu_id() as usize; let words = sample(me); BLOCKS[me].publish(words); - let [smi, aperf, mperf] = [Counter::Smi, Counter::Aperf, Counter::Mperf].map(|c| held(&words, c)); - log!("counters: cpu{me} reads smi={smi} aperf={aperf} mperf={mperf}"); + let [smi, aperf, mperf, request, pkg, epb] = [ + Counter::Smi, + Counter::Aperf, + Counter::Mperf, + Counter::HwpRequest, + Counter::HwpRequestPkg, + Counter::EnergyPerfBias, + ] + .map(|c| held(&words, c)); + log!( + "counters: cpu{me} reads smi={smi} aperf={aperf} mperf={mperf} hwp_request={request} \ + hwp_request_pkg={pkg} energy_perf_bias={epb}" + ); } fn held(words: &[u64; WORDS], counter: Counter) -> bool { @@ -89,6 +110,9 @@ fn sample(me: usize) -> [u64; WORDS] { Counter::Aperf => hardware.aperf, Counter::Mperf => hardware.mperf, Counter::Kicks => crate::irq_census::deliveries(me as u32, Source::Kick), + Counter::HwpRequest => hardware.envelope.as_ref().map(|e| e.hwp_request), + Counter::HwpRequestPkg => hardware.envelope.as_ref().map(|e| e.hwp_request_pkg), + Counter::EnergyPerfBias => hardware.envelope.as_ref().map(|e| e.energy_perf_bias), }; if let Some(value) = value { words[1] |= 1 << counter as usize; diff --git a/kernel/src/syscall/machine.rs b/kernel/src/syscall/machine.rs index df469aa4150..da3a809c58d 100644 --- a/kernel/src/syscall/machine.rs +++ b/kernel/src/syscall/machine.rs @@ -277,7 +277,8 @@ pub(super) fn sys_sched_info() -> toyos_abi::syscall::SchedInfo { /// Every online CPU's counters into `out`, or the CPU count when `out` is /// empty: requires a `SysCap` carrying `Rights::COUNTERS`, and answers the -/// counters that time programs only where it carries `Rights::TRACE` too. +/// counters that time programs and the power envelope only where it carries +/// `Rights::TRACE` too. pub(super) fn sys_counters(syscap: RawHandle, out: &mut UserBytesMut) -> u64 { let rights = match demand_syscap(syscap, Rights::COUNTERS) { Ok(rights) => rights, diff --git a/tests/t14-linux/SOURCE b/tests/t14-linux/SOURCE index d59763d1e66..20e69eb8c43 100644 --- a/tests/t14-linux/SOURCE +++ b/tests/t14-linux/SOURCE @@ -12,3 +12,13 @@ beside them. sleep 2; turbostat --quiet --interval 10 --num_iterations 6 --show Core,CPU,Avg_MHz,Busy%,Bzy_MHz,TSC_MHz,IRQ,SMI,CPU%c1,CPU%c6,CPU%c7,CoreTmp,PkgTmp,Pkg%pc2,Pkg%pc6,Pk%pc10,PkgWatt,CorWatt; wait` + + hwp-request.txt IA32_HWP_REQUEST (0x774), the same on all eight CPUs, + read as root through /dev/cpu/N/msr under the same + Ubuntu with intel_pstate `active`, the `powersave` + governor and EPP `balance_performance`; recorded in + the message of commit 868378b87 (#568, branch + `wt/toyos-llvmbar`): "IA32_HWP_REQUEST 0x80002a04 on + all eight CPUs (min 4, max 42, desired 0, EPP 128)". + `counters_on_metal` holds every CPU's boot + `hwp_request` to it. diff --git a/tests/t14-linux/hwp-request.txt b/tests/t14-linux/hwp-request.txt new file mode 100644 index 00000000000..3e9991a79bb --- /dev/null +++ b/tests/t14-linux/hwp-request.txt @@ -0,0 +1 @@ +0x80002a04 diff --git a/tests/toyos.rs b/tests/toyos.rs index 7fac3b67117..32269893a20 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -3264,17 +3264,21 @@ const SMI_SPAN_NS: u64 = 4_444_000_000; /// /// Held: a record per CPU the bring-up started, naming the local APIC id the /// bring-up gave that CPU and carrying the counters its `counters: cpuN -/// reads` line names, and none stale. From `idle0` to `spin`, at least +/// reads` line names, and none stale; every CPU's performance request +/// declared at boot, `pm_enable=1`, the request Linux makes on this machine +/// (`tests/t14-linux/hwp-request.txt`), and its power envelope in every read +/// the one its `control_regs:` line holds. From `idle0` to `spin`, at least /// [`SMI_SPAN_NS`] apart, every CPU's SMI count rose alike and by two or more: /// the firmware's legacy mode, the positive control ACPI stage 1's flatness /// is read against, and the row that stage changes. Across the spin every /// CPU's MPERF ran nine tenths of its stamp or more [e], a CPU in C0 the whole /// span: MPERF counts at the TSC's rate there (SDM Vol. 3B, "Hardware -/// Coordination Feedback"). +/// Coordination Feedback"); and every CPU's busy frequency reached the lowest +/// Linux's turbostat read under one `yes` per CPU on the same machine over the +/// same span of load (`tests/t14-linux/turbostat-loaded.txt`). /// -/// Read and not held, beside Linux's turbostat on the same machine -/// (`tests/t14-linux/`): each CPU's idle busy fraction, its busy -/// frequency under the spin, and what one round cost its reader. +/// Read and not held, beside Linux's turbostat: each CPU's idle busy +/// fraction, and what one round cost its reader. fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { type Read<'a> = BTreeMap>; back.job_passed("test_rs_counters_metal")?; @@ -3307,8 +3311,10 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { Ok((read, clock.get(name).ok_or_else(|| format!("counters_metal printed no {name} clock"))?.0)) }; let (idle0, at0) = phase("idle0")?; - let (idle1, _) = phase("idle1")?; + let (idle1, at1) = phase("idle1")?; let (spin, at2) = phase("spin")?; + let linux_request = u64::from_str_radix(include_str!("t14-linux/hwp-request.txt").trim().trim_start_matches("0x"), 16) + .map_err(|e| format!("t14-linux/hwp-request.txt: {e}"))?; let bsp = kernel.must_say("percpu: BSP cpu_id=0 lapic_id=")?; let mut roster = vec![bsp.rsplit("lapic_id=").next().unwrap_or_default().trim().to_string()]; for cpu in 1..cpus { @@ -3320,7 +3326,7 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { } for (cpu, counters) in read { let reads = kernel.must_say(&format!("counters: cpu{cpu} reads "))?; - for counter in ["smi", "aperf", "mperf"] { + for counter in ["smi", "aperf", "mperf", "hwp_request", "hwp_request_pkg", "energy_perf_bias"] { if !reads.contains(&format!("{counter}={}", counters.contains_key(counter))) { return Err(format!("{name}: cpu{cpu} carries {counters:?} and its bring-up said {reads:?}")); } @@ -3328,6 +3334,21 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { if counters.get("stale") != Some(&0) || counters.get("hardware_id").map(u64::to_string) != Some(roster[*cpu].clone()) { return Err(format!("{name}: cpu{cpu} is stale or not lapic {}: {counters:?}", roster[*cpu])); } + let declared = kernel.must_say(&format!("control_regs: cpu{cpu} pm_enable=1 "))?; + if !declared.contains(&format!(" hwp_request={linux_request:#010x} ")) { + return Err(format!("cpu{cpu} declared {declared:?}, and Linux requests {linux_request:#010x} on this machine")); + } + for (counter, field, radix) in + [("hwp_request", "hwp_request=0x", 16), ("hwp_request_pkg", "hwp_request_pkg=0x", 16), ("energy_perf_bias", "epb=", 10)] + { + let held = declared + .split(' ') + .find_map(|w| u64::from_str_radix(w.strip_prefix(field)?, radix).ok()) + .ok_or_else(|| format!("cpu{cpu}'s declaration carries no {field}: {declared:?}"))?; + if counters.get(counter) != Some(&held) { + return Err(format!("{name}: cpu{cpu} reads {counter} {:?} and boot declared {held}: {declared:?}", counters.get(counter))); + } + } } } for cpu in 0..cpus { @@ -3353,15 +3374,27 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { if busy.iter().any(|&b| b < 0.9) { return Err(format!("MPERF ran {busy:?} of each stamp across the spin, some cpu under 0.9")); } - let linux = |file: &str, column: &str| -> Result<(f64, f64), String> { + // Each interval's machine-wide row, in order. + let linux = |file: &str, column: &str| -> Result, String> { let mut rows = file.lines().map(|l| l.split('\t').collect::>()); let header = rows.next().ok_or("an empty turbostat reading")?; let at = header.iter().position(|c| *c == column).ok_or_else(|| format!("turbostat read no {column}"))?; - let values: Vec = rows.filter(|r| r[0] == "-").filter_map(|r| r.get(at)?.parse().ok()).collect(); - Ok(values.iter().fold((f64::MAX, f64::MIN), |(lo, hi), &v| (lo.min(v), hi.max(v)))) + Ok(rows.filter(|r| r[0] == "-").filter_map(|r| r.get(at)?.parse().ok()).collect()) }; - let idle = linux(include_str!("t14-linux/turbostat-idle.txt"), "Busy%")?; - let loaded = linux(include_str!("t14-linux/turbostat-loaded.txt"), "Bzy_MHz")?; + let range = |values: &[f64]| values.iter().fold((f64::MAX, f64::MIN), |(lo, hi), &v| (lo.min(v), hi.max(v))); + let idle = range(&linux(include_str!("t14-linux/turbostat-idle.txt"), "Busy%")?); + let loaded_rows = linux(include_str!("t14-linux/turbostat-loaded.txt"), "Bzy_MHz")?; + let loaded = range(&loaded_rows); + // The floor is Linux's clock over the spin's own span of load, not after + // the package settles: the intervals that open within it, the first 2 s + // into Linux's load and each 10 s long (`tests/t14-linux/SOURCE`). + let spin_ns = at2 - at1; + let opened = loaded_rows.iter().enumerate().take_while(|&(k, _)| 2_000_000_000 + k as u64 * 10_000_000_000 < spin_ns); + let floor = opened.map(|(_, &mhz)| mhz).fold(f64::MAX, f64::min); + if floor == f64::MAX { + return Err(format!("the spin lasted {spin_ns} ns, and no Linux interval opens within it")); + } + let spinning: Vec = (0..cpus).map(|cpu| tsc_mhz * ratio(idle1, spin, cpu, "aperf", "mperf")).collect(); eprintln!( " [counters] {cpus} cpus, SMI +{} each over {} ms; TSC {tsc_mhz:.0} MHz", smis[0], @@ -3369,17 +3402,20 @@ fn counters_on_metal(back: &metal::Readback) -> Result<(), String> { ); for (cpu, busy) in busy.iter().enumerate() { eprintln!( - " [counters] cpu{cpu}: idle busy {:.2}% (Linux {:.2}-{:.2}%), spinning {:.0} MHz (Linux {:.0}-{:.0}), \ - busy {busy:.3}", + " [counters] cpu{cpu}: idle busy {:.2}% (Linux {:.2}-{:.2}%), spinning {:.0} MHz (Linux {floor:.0} over \ + this span, {:.0}-{:.0} loaded), busy {busy:.3}", ratio(idle0, idle1, cpu, "mperf", "stamp") * 100.0, idle.0, idle.1, - tsc_mhz * ratio(idle1, spin, cpu, "aperf", "mperf"), + spinning[cpu], loaded.0, loaded.1, ); } eprintln!(" [counters] the spin's read, a whole round, took its reader {} ns", clock["spin"].1); + if spinning.iter().any(|&mhz| mhz < floor) { + return Err(format!("spinning {spin_ns} ns at {spinning:.0?} MHz, some cpu below the {floor:.0} Linux held over that span")); + } Ok(()) } diff --git a/toyos-abi/src/counters.rs b/toyos-abi/src/counters.rs index fd7109da628..69643bd7ff6 100644 --- a/toyos-abi/src/counters.rs +++ b/toyos-abi/src/counters.rs @@ -9,7 +9,9 @@ //! absent and never zero; a record with no stamp is a CPU whose answer could //! not be read whole, and says nothing about it but its index. //! -//! Every value counts up from boot, so the difference of two reads is exact. +//! Every counter counts up from boot, so the difference of two reads is exact; +//! the power envelope's registers ([`Counter::HwpRequest`] and those after it) +//! count nothing and are what the CPU held when it read them. //! //! [`SYS_COUNTERS`]: crate::syscall::SYS_COUNTERS @@ -31,12 +33,27 @@ pub enum Counter { Mperf, /// Kicks the CPU took: the interrupt another CPU sends it to wake it. Kicks, + /// The CPU's performance request (`IA32_HWP_REQUEST`), where the kernel + /// declares one. + HwpRequest, + /// Its package's request (`IA32_HWP_REQUEST_PKG`), beside it. + HwpRequestPkg, + /// Its energy/performance bias (`IA32_ENERGY_PERF_BIAS`), beside it. + EnergyPerfBias, } impl Counter { - pub const COUNT: usize = 5; - pub const ALL: [Counter; Self::COUNT] = - [Self::Stamp, Self::Smi, Self::Aperf, Self::Mperf, Self::Kicks]; + pub const COUNT: usize = 8; + pub const ALL: [Counter; Self::COUNT] = [ + Self::Stamp, + Self::Smi, + Self::Aperf, + Self::Mperf, + Self::Kicks, + Self::HwpRequest, + Self::HwpRequestPkg, + Self::EnergyPerfBias, + ]; pub const fn name(self) -> &'static str { match self { @@ -45,6 +62,9 @@ impl Counter { Self::Aperf => "aperf", Self::Mperf => "mperf", Self::Kicks => "kicks", + Self::HwpRequest => "hwp_request", + Self::HwpRequestPkg => "hwp_request_pkg", + Self::EnergyPerfBias => "energy_perf_bias", } } @@ -52,7 +72,12 @@ impl Counter { pub const fn needs(self) -> Rights { match self { Self::Stamp | Self::Smi => Rights::COUNTERS, - Self::Aperf | Self::Mperf | Self::Kicks => Rights::COUNTERS.union(Rights::TRACE), + Self::Aperf + | Self::Mperf + | Self::Kicks + | Self::HwpRequest + | Self::HwpRequestPkg + | Self::EnergyPerfBias => Rights::COUNTERS.union(Rights::TRACE), } } } @@ -154,8 +179,8 @@ mod tests { #[test] fn every_shape_of_record_reads_back_as_written() { for r in [ - record(false, [Some(1), Some(u64::from(u32::MAX)), Some(3), Some(u64::MAX), Some(0)]), - record(true, [Some(9), None, None, None, Some(4)]), + record(false, [Some(1), Some(u64::from(u32::MAX)), Some(3), Some(u64::MAX), Some(0), Some(0x8000_2a04), Some(6), Some(0)]), + record(true, [Some(9), None, None, None, Some(4), None, Some(0x8000_ff01), None]), record(true, [None; Counter::COUNT]), ] { assert_eq!(Record::decode(&r.encode()), Ok(r)); @@ -165,7 +190,7 @@ mod tests { /// Absent is not zero: a zero that is there reads back there. #[test] fn a_present_zero_is_not_an_absent_counter() { - let r = record(false, [Some(5), Some(0), None, None, None]); + let r = record(false, [Some(5), Some(0), None, None, None, None, None, None]); assert_eq!(Record::decode(&r.encode()).unwrap().get(Counter::Smi), Some(0)); assert_eq!(Record::decode(&r.encode()).unwrap().get(Counter::Aperf), None); } @@ -182,20 +207,20 @@ mod tests { #[test] fn a_word_under_an_absent_counter_is_refused() { - let mut raw = record(false, [Some(1), None, None, None, None]).encode(); + let mut raw = record(false, [Some(1), None, None, None, None, None, None, None]).encode(); raw.0[16 + 8 * Counter::Mperf as usize] = 1; assert_eq!(Record::decode(&raw), Err(Undecodable::Absent(Counter::Mperf))); } - /// The counters that time programs are the trace right's, and the two that - /// do not are readable on `COUNTERS` alone. + /// The counters that time programs and the power envelope are the trace + /// right's, and the two that are neither are readable on `COUNTERS` alone. #[test] - fn what_times_a_program_needs_the_trace_right() { + fn what_times_a_program_or_is_power_needs_the_trace_right() { for counter in Counter::ALL { assert!(counter.needs().contains(Rights::COUNTERS), "{counter:?}"); assert_eq!( counter.needs().contains(Rights::TRACE), - matches!(counter, Counter::Aperf | Counter::Mperf | Counter::Kicks), + !matches!(counter, Counter::Stamp | Counter::Smi), "{counter:?}" ); } diff --git a/toyos-abi/src/handle.rs b/toyos-abi/src/handle.rs index 5c503c112a7..220b683b98b 100644 --- a/toyos-abi/src/handle.rs +++ b/toyos-abi/src/handle.rs @@ -151,8 +151,8 @@ impl Rights { /// /// Beside [`COUNTERS`](Self::COUNTERS), [`SYS_COUNTERS`] answers each /// CPU's `APERF` and `MPERF` — its frequency and busy fraction, the - /// Hertzbleed power channel — and the kicks it took, the wake-ups that time - /// keystrokes. Admin tools hold it, `inspect` and `test-runner`; no + /// Hertzbleed power channel — the kicks it took, the wake-ups that time + /// keystrokes, and the power envelope it runs under. Admin tools hold it, `inspect` and `test-runner`; no /// ordinary program and no toybox applet does while a manifest row grants /// a whole binary. /// diff --git a/toyos-abi/src/syscall.rs b/toyos-abi/src/syscall.rs index d63e214f3c3..162b831292f 100644 --- a/toyos-abi/src/syscall.rs +++ b/toyos-abi/src/syscall.rs @@ -260,8 +260,8 @@ pub const SYS_DEVICE_DMA_MAP: u64 = 122; pub const SYS_DEVICE_DMA_UNMAP: u64 = 123; /// Every online CPU's counters, as [`crate::counters`] records. Gated by -/// [`Rights::COUNTERS`] on a `SysCap`, and the counters that time programs by -/// [`Rights::TRACE`] beside it. See [`counters`]. +/// [`Rights::COUNTERS`] on a `SysCap`, and the counters that time programs and +/// the power envelope by [`Rights::TRACE`] beside it. See [`counters`]. /// /// [`Rights::COUNTERS`]: crate::handle::Rights::COUNTERS /// [`Rights::TRACE`]: crate::handle::Rights::TRACE diff --git a/toyos-cpuvuln/Cargo.toml b/toyos-cpuvuln/Cargo.toml index 2bee37a101e..6ce1bb2b9f3 100644 --- a/toyos-cpuvuln/Cargo.toml +++ b/toyos-cpuvuln/Cargo.toml @@ -3,7 +3,7 @@ [package] name = "toyos-cpuvuln" -description = "Linux's verdicts from a CPU's CPUID and MSR facts, pure: each speculative-execution vulnerability and its mitigation choice, and which model-specific counters the CPU has." +description = "Linux's verdicts from a CPU's CPUID and MSR facts, pure: each speculative-execution vulnerability and its mitigation choice, which model-specific counters the CPU has, and the HWP request intel_pstate asks of it." version = "0.1.0" edition = "2021" license = "MIT OR Apache-2.0" diff --git a/toyos-cpuvuln/src/hwp.rs b/toyos-cpuvuln/src/hwp.rs new file mode 100644 index 00000000000..636f0763913 --- /dev/null +++ b/toyos-cpuvuln/src/hwp.rs @@ -0,0 +1,196 @@ +//! Whether a CPU's performance request is declared through HWP, and the +//! request: the one `intel_pstate` asks of it under the `powersave` governor +//! at its defaults, read at upstream Linux `v6.8`'s +//! `drivers/cpufreq/intel_pstate.c` and `arch/x86/include/asm/msr-index.h`, +//! not at this crate's pinned tag. The minimum is `MSR_PLATFORM_INFO[47:40]` +//! (`core_get_min_pstate`, `:1859-1864`), the maximum the CPU's highest level, +//! turbo included (`__intel_pstate_get_hwp_cap`, `:940-948`), and the +//! preference `HWP_EPP_BALANCE_PERFORMANCE` (`msr-index.h:500`), the +//! `balance_performance` it leaves on every CPU `intel_epp_balance_perf` +//! (`:3410-3419`) does not list. +//! +//! **The refusals are ToyOS's, not Linux's**: `intel_pstate` drives hybrid +//! CPUs on a scale of their own and leaves other vendors to other drivers, +//! and neither is modelled here. + +use crate::{Ident, Vendor}; + +/// What one CPU says about itself, read on that CPU. +#[derive(Clone, Copy, Debug)] +pub struct HwpFacts { + pub vendor: Vendor, + /// CPUID.1:EAX. + pub signature: u32, + /// 0 unless CPUID.0:EAX reaches 6. + pub cpuid_6_eax: u32, + /// 0 unless CPUID.0:EAX reaches 6. + pub cpuid_6_ecx: u32, + /// 0 unless CPUID.0:EAX reaches 7. + pub cpuid_7_0_edx: u32, +} + +/// A CPU whose request is declared. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Hwp { + /// `IA32_HWP_INTERRUPT` exists (CPUID.6:EAX[8]); it is `#GP` where not. + pub notify: bool, +} + +/// Why a CPU's request is not declared, and its performance state stays what +/// firmware left. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum HwpRefusal { + NoHwp, + NotIntel, + NotFamily6, + NoEpp, + NoPackageRequest, + NoEnergyPerfBias, + Hybrid, +} + +impl HwpRefusal { + pub const fn reason(self) -> &'static str { + match self { + Self::NoHwp => "no HWP (CPUID.6:EAX[7])", + Self::NotIntel => "HWP on a CPU that is not Intel's, and the minimum is MSR_PLATFORM_INFO's, which is Intel's", + Self::NotFamily6 => "an Intel CPU outside family 6, which no CPUID bit enumerates MSR_PLATFORM_INFO on", + Self::NoEpp => "HWP without an energy/performance preference (CPUID.6:EAX[10])", + Self::NoPackageRequest => "HWP without a package request (CPUID.6:EAX[11])", + Self::NoEnergyPerfBias => "no energy/performance bias (CPUID.6:ECX[3])", + Self::Hybrid => "a hybrid CPU (CPUID.(7,0):EDX[15]), whose HWP scale is not its ratio scale", + } + } +} + +// `arch/x86/include/asm/cpufeatures.h`'s bits at their CPUID positions. +const CPUID_6_EAX_HWP: u32 = 1 << 7; +const CPUID_6_EAX_HWP_NOTIFY: u32 = 1 << 8; +const CPUID_6_EAX_HWP_EPP: u32 = 1 << 10; +const CPUID_6_EAX_HWP_PKG_REQ: u32 = 1 << 11; +const CPUID_6_ECX_EPB: u32 = 1 << 3; +const CPUID_7_0_EDX_HYBRID_CPU: u32 = 1 << 15; + +/// **All or nothing**: a CPU lacking one register the declaration writes gets +/// none of them, so no reader takes firmware's value of one for the +/// declaration's. +pub fn hwp(facts: &HwpFacts) -> Result { + let eax = facts.cpuid_6_eax; + let refusal = if eax & CPUID_6_EAX_HWP == 0 { + Some(HwpRefusal::NoHwp) + } else if facts.vendor != Vendor::Intel { + Some(HwpRefusal::NotIntel) + } else if Ident::new(facts.vendor, facts.signature).family != 6 { + Some(HwpRefusal::NotFamily6) + } else if eax & CPUID_6_EAX_HWP_EPP == 0 { + Some(HwpRefusal::NoEpp) + } else if eax & CPUID_6_EAX_HWP_PKG_REQ == 0 { + Some(HwpRefusal::NoPackageRequest) + } else if facts.cpuid_6_ecx & CPUID_6_ECX_EPB == 0 { + Some(HwpRefusal::NoEnergyPerfBias) + } else if facts.cpuid_7_0_edx & CPUID_7_0_EDX_HYBRID_CPU != 0 { + Some(HwpRefusal::Hybrid) + } else { + None + }; + match refusal { + Some(refusal) => Err(refusal), + None => Ok(Hwp { notify: eax & CPUID_6_EAX_HWP_NOTIFY != 0 }), + } +} + +/// `HWP_EPP_BALANCE_PERFORMANCE`. +pub const HWP_EPP: u64 = 0x80; + +/// `IA32_HWP_REQUEST` for a CPU whose `IA32_HWP_CAPABILITIES` reads +/// `capabilities`, in a package whose `MSR_PLATFORM_INFO` reads +/// `platform_info`: no desired level, no activity window, no package control, +/// so the CPU chooses between the two levels itself (SDM Vol. 3B, "Managing +/// HWP"; `msr-index.h:495-504`). +pub const fn hwp_request(capabilities: u64, platform_info: u64) -> u64 { + let min = (platform_info >> 40) & 0xff; + let max = capabilities & 0xff; + min | max << 8 | HWP_EPP << 24 +} + +#[cfg(test)] +mod tests { + extern crate std; + + use std::vec::Vec; + + use super::*; + + /// The T14's CPU 0, off `fixtures/t14/cpuid.txt`. + fn t14() -> HwpFacts { + let leaves: Vec> = include_str!("../fixtures/t14/cpuid.txt") + .lines() + .map(|l| l.split(' ').map(|w| u32::from_str_radix(w, 16).expect(w)).collect()) + .collect(); + let leaf = |n: u32| leaves.iter().find(|l| l[..2] == [n, 0]).expect("a leaf the capture read"); + let id: Vec = [leaf(0)[3], leaf(0)[5], leaf(0)[4]].iter().flat_map(|r| r.to_le_bytes()).collect(); + assert!(leaf(0)[2] >= 7, "the T14 reaches leaf 7"); + HwpFacts { + vendor: Vendor::from_id(id[..].try_into().unwrap()), + signature: leaf(1)[2], + cpuid_6_eax: leaf(6)[2], + cpuid_6_ecx: leaf(6)[4], + cpuid_7_0_edx: leaf(7)[5], + } + } + + /// `cpuinfo.txt` lists `hwp hwp_notify hwp_act_window hwp_epp + /// hwp_pkg_req` and `epb`, and no `hybrid_cpu`. + #[test] + fn the_t14_is_declared_with_its_notification() { + let flags = include_str!("../fixtures/t14/cpuinfo.txt") + .lines() + .find_map(|l| l.strip_prefix("flags\t\t: ")) + .expect("a flags line"); + for flag in ["hwp", "hwp_notify", "hwp_epp", "hwp_pkg_req", "epb"] { + assert!(flags.split(' ').any(|f| f == flag), "{flag}"); + } + assert!(!flags.split(' ').any(|f| f == "hybrid_cpu")); + assert_eq!(hwp(&t14()), Ok(Hwp { notify: true })); + assert_eq!(hwp(&HwpFacts { cpuid_6_eax: t14().cpuid_6_eax & !CPUID_6_EAX_HWP_NOTIFY, ..t14() }), Ok(Hwp { notify: false })); + } + + /// QEMU's `qemu64`, the TCG model: `AuthenticAMD`, and no HWP in leaf 6. + #[test] + fn the_tcg_model_is_refused() { + let tcg = HwpFacts { vendor: Vendor::Amd, signature: 0x0006_0fb1, cpuid_6_eax: 1 << 2, cpuid_6_ecx: 0, cpuid_7_0_edx: 0 }; + assert_eq!(hwp(&tcg), Err(HwpRefusal::NoHwp)); + } + + #[test] + fn a_missing_register_refuses_the_whole_request_by_name() { + let without = |eax: u32, ecx: u32| hwp(&HwpFacts { cpuid_6_eax: t14().cpuid_6_eax & !eax, cpuid_6_ecx: t14().cpuid_6_ecx & !ecx, ..t14() }); + assert_eq!(without(CPUID_6_EAX_HWP, 0), Err(HwpRefusal::NoHwp)); + assert_eq!(without(CPUID_6_EAX_HWP_EPP, 0), Err(HwpRefusal::NoEpp)); + assert_eq!(without(CPUID_6_EAX_HWP_PKG_REQ, 0), Err(HwpRefusal::NoPackageRequest)); + assert_eq!(without(0, CPUID_6_ECX_EPB), Err(HwpRefusal::NoEnergyPerfBias)); + assert_eq!(hwp(&HwpFacts { cpuid_7_0_edx: CPUID_7_0_EDX_HYBRID_CPU, ..t14() }), Err(HwpRefusal::Hybrid)); + assert_eq!(hwp(&HwpFacts { vendor: Vendor::Amd, ..t14() }), Err(HwpRefusal::NotIntel)); + } + + /// The family is `x86_family`'s: 0xF plus the extended field, never the + /// extended field beside family 6. + #[test] + fn a_cpu_outside_family_6_is_refused_by_name() { + let of = |signature: u32| hwp(&HwpFacts { signature, ..t14() }); + assert_eq!(of(0x0000_0f41), Err(HwpRefusal::NotFamily6)); + assert_eq!(of(0x0030_0f00), Err(HwpRefusal::NotFamily6)); + assert_eq!(of(0x0ff0_0600), Ok(Hwp { notify: true })); + } + + /// Each field where `msr-index.h` puts it, from the bits its source names + /// and no other. + #[test] + fn the_request_is_the_minimum_ratio_the_highest_level_and_the_preference() { + assert_eq!(hwp_request(0, 0), 0x8000_0000); + assert_eq!(hwp_request(0xffff_ff2a, 0), 0x8000_2a00); + assert_eq!(hwp_request(0, 0x04 << 40), 0x8000_0004); + assert_eq!(hwp_request(0, !(0xff << 40)), 0x8000_0000); + assert_eq!(hwp_request(u64::MAX, u64::MAX), 0x8000_ffff); + } +} diff --git a/toyos-cpuvuln/src/lib.rs b/toyos-cpuvuln/src/lib.rs index 181e3bd1125..20c9e7270d8 100644 --- a/toyos-cpuvuln/src/lib.rs +++ b/toyos-cpuvuln/src/lib.rs @@ -16,7 +16,8 @@ //! [`Unmodelled`], never approximated. //! //! [`counters`] is the same tag's other verdict on a CPU: which of its -//! model-specific counters exist. +//! model-specific counters exist. [`hwp`] is the third: whether its +//! performance request is declared, and [`hwp_request`] the request. //! //! Pure: no I/O, no allocation, no `unsafe`. The caller reads the facts. @@ -26,9 +27,11 @@ use core::fmt; mod counters; +mod hwp; mod table; pub use counters::{counters, CounterFacts, Counters}; +pub use hwp::{hwp, hwp_request, Hwp, HwpFacts, HwpRefusal, HWP_EPP}; #[cfg(test)] mod tests; diff --git a/toyos-inspect/src/kernel.rs b/toyos-inspect/src/kernel.rs index ca5ff1513ff..47b8de70d8b 100644 --- a/toyos-inspect/src/kernel.rs +++ b/toyos-inspect/src/kernel.rs @@ -2,7 +2,7 @@ //! //! A root no port answers for, as `dev` is: the reader asks the kernel on a //! `SysCap` carrying `Rights::COUNTERS`, and `Rights::TRACE` for the counters -//! that time programs, and renders its records here. +//! that time programs and the power envelope, and renders its records here. //! //! ```text //! kernel.cpu..stale whether the CPU missed this read's bound @@ -65,7 +65,7 @@ mod tests { /// The T14's shape: every counter on every CPU. fn whole(cpu: u32) -> Record { - record(cpu, false, [Some(10), Some(4817), Some(100), Some(200), Some(3)]) + record(cpu, false, [Some(10), Some(4817), Some(100), Some(200), Some(3), Some(0x8000_2a04), Some(0x8000_ff01), Some(6)]) } #[test] @@ -84,7 +84,7 @@ mod tests { /// `TRACE`: only what the record carries, and no zero in place of the rest. #[test] fn an_absent_counter_has_no_path() { - let got = render(&[record(0, false, [Some(10), None, None, None, None])]).unwrap(); + let got = render(&[record(0, false, [Some(10), None, None, None, None, None, None, None])]).unwrap(); let paths: Vec<&str> = got.keys().map(String::as_str).collect(); assert_eq!(paths, vec!["kernel.cpu.0.hardware_id", "kernel.cpu.0.stale", "kernel.cpu.0.stamp"]); } diff --git a/toyos/src/syscap.rs b/toyos/src/syscap.rs index 9e20f70f690..50e2824dc15 100644 --- a/toyos/src/syscap.rs +++ b/toyos/src/syscap.rs @@ -134,8 +134,9 @@ impl SysCap { /// how many. An empty `buf` asks how many CPUs there are — see /// [`syscall::counters`]. /// - /// Needs [`Rights::COUNTERS`]; the counters that time programs come back - /// only where the capability carries [`Rights::TRACE`] too. + /// Needs [`Rights::COUNTERS`]; the counters that time programs and the + /// power envelope come back only where the capability carries + /// [`Rights::TRACE`] too. pub fn counters( &self, buf: &mut [toyos_abi::counters::RawRecord],