From 952bd2a57cebd5647d107fdbe88260cb40477d93 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:33:14 +0200 Subject: [PATCH 1/8] A pass skips the timer write and the ask swaps it does not owe Every pass wrote its timer and swapped all sixteen ask words, whether or not anything had changed. On the T14 the probes put the timer's clock read, conversion, write and trace at about 110 ns a pass and the empty drain, which is the ask swaps, at about 75; a yield with nothing to yield to probed 620 ns. - A running CPU leaves a one-shot that is still to fire and due no later than it needs. An early fire is a spurious pass, never a missed deadline, so the timer is written once a request, after a fire, or when something is owed sooner. `Machine::timer_pending` says whether the last arm's interrupt is still to be taken: both architectures keep a per-CPU flag that the arm sets and every timer entry and stop clears, and the simulator answers from its modelled one-shot. A CPU going idle still arms exactly, so nothing wakes it early. - An ask word read empty is not swapped. An ask landing after the read rings the doorbell after it, and the pass that ring owes takes it. - A task's mask and its reach are one value with one constructor, so the queue cannot count a reach its mask has moved from. The TSC-deadline record loses "re-armed every pass" from its name: what a deadline would save on a running CPU is now per arm. Co-Authored-By: Claude Opus 5.5 --- ...-is-a-lapic-count-never-a-tsc-deadline.md} | 15 ++- ...s-what-modern-hardware-offers-for-speed.md | 4 +- kernel/pure/sched/affinity.rs | 27 ++++- kernel/pure/sched/cpu.rs | 111 ++++++++++++++++-- kernel/pure/sched/hw.rs | 8 ++ kernel/pure/sched/task.rs | 20 ++-- kernel/pure/sched/timer.rs | 3 +- kernel/sim/src/hw_impl.rs | 6 + kernel/src/arch/aarch64/irqchip.rs | 11 ++ kernel/src/arch/aarch64/percpu.rs | 14 ++- kernel/src/arch/x86_64/apic.rs | 11 ++ kernel/src/arch/x86_64/percpu.rs | 17 ++- kernel/src/hw.rs | 4 + 13 files changed, 217 insertions(+), 34 deletions(-) rename issues/{the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md => the-timer-is-a-lapic-count-never-a-tsc-deadline.md} (85%) diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md similarity index 85% rename from issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md rename to issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md index 66ca8365fa1..cbf0f60aa29 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md @@ -4,10 +4,10 @@ kind: track opened: 2026-10-12 --- -# The timer is a LAPIC count re-armed every pass, never a TSC deadline +# The timer is a LAPIC count, never a TSC deadline -Every scheduler pass ends by handing its timer an absolute deadline -(`kernel/pure/sched/cpu.rs:1912-1918`), and the kernel turns it back into a +A scheduler pass ends by handing its timer an absolute deadline +(`SchedPass::apply_timer_no_later_than`), and the kernel turns it back into a span from now (`kernel/src/hw.rs:47-51`) that `apic::arm_one_shot` converts to LAPIC ticks by the boot's calibration against the TSC-derived clock (`kernel/src/arch/x86_64/apic.rs`) and writes into `TIMER_INIT`, one APIC @@ -28,6 +28,15 @@ CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, `CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration and its spin. +**A running CPU re-arms once a request, since #PRNUM.** A pass on a CPU +that keeps running leaves a one-shot that is still to fire and due no later +than it needs, so the timer is written once a request, after a fire, or when +something is owed sooner; a CPU going idle still arms exactly at every sleep +and stops at every idle entry. What a deadline would save is now per arm and +not per pass, which leaves it the idle path's arm and stop and the boot's +calibration spin, on the T14 alone. The figures below are the old code's, +which wrote at every pass. + **Its speed reading decides it.** The SDM defines the mode and recommends it over the count nowhere, and AMD documents none, so nothing published covers the choice. Intel's specification update for the T14's processor lists no diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 475aaa6f052..1517fc20d41 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -46,7 +46,7 @@ and contention). - A TSC deadline: before #1061 cut the arm to one register write, the re-arm was at most 11.7 to 12.1 % of a pass's pick and switch on the T14, and at the AMD bench's TSC floor; it is to be read again - (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). + (`issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md`). - Deep idle: on the T14 `hlt` holds a one-thread load to the turbo ratio for three or more active cores, 38 of `MSR_TURBO_RATIO_LIMIT`'s `0x2626262626262a2a` against 42 for one or two (#866's dump, @@ -88,7 +88,7 @@ yet measured. - `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md` - `issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md` - `issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md` -- `issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md` +- `issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md` - `issues/no-gate-decodes-kernel-elfs-instructions.md` - `issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md` - `issues/the-direct-map-uses-1-gib-leaves-where-the-cpu-has-them.md` diff --git a/kernel/pure/sched/affinity.rs b/kernel/pure/sched/affinity.rs index a3244504560..703c6927d65 100644 --- a/kernel/pure/sched/affinity.rs +++ b/kernel/pure/sched/affinity.rs @@ -106,15 +106,34 @@ impl CpuMask { pub fn words(&self) -> &[u64; MASK_WORDS] { &self.0 } +} - pub fn reach(&self) -> Reach { - if *self == Self::ANY { +/// A task's mask as its owner CPU loaded it, with the [`Reach`] a run queue +/// counts it by: one value, so the two cannot disagree. +#[derive(Clone, Copy, Debug)] +pub struct LoadedMask { + mask: CpuMask, + reach: Reach, +} + +impl LoadedMask { + pub fn new(mask: CpuMask) -> Self { + let reach = if mask == CpuMask::ANY { Reach::Any - } else if self.single().is_some() { + } else if mask.single().is_some() { Reach::One } else { Reach::Several - } + }; + Self { mask, reach } + } + + pub fn mask(&self) -> CpuMask { + self.mask + } + + pub fn reach(&self) -> Reach { + self.reach } } diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 1c44f3af940..9a9d87bd129 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -2186,18 +2186,30 @@ impl SchedPass<'_, '_, H, P, Disposed> { Some(_) => Some(self.cpu.settle_decision(self.now)), None => None, }; - let plan = TimerPlan::compute(decision, deadline).no_later_than(extra); - match plan { - TimerPlan::Arm(at) => self.env.hw.set_timer(at), - TimerPlan::Stop => self.env.hw.stop_timer(), + match TimerPlan::compute(decision, deadline).no_later_than(extra) { + // A one-shot still to fire and due no later is left: an early fire + // is a spurious pass, so a running CPU writes its timer once a + // request rather than once a pass. An idle CPU arms exactly, so + // nothing wakes it before it is owed. + TimerPlan::Arm(at) + if self.cpu.running.is_some() + && self.cpu.armed.is_some_and(|armed| self.now < armed && armed <= at) + && self.env.hw.timer_pending() => {} + TimerPlan::Arm(at) => { + self.env.hw.set_timer(at); + self.cpu.armed = Some(at); + } + TimerPlan::Stop => { + self.env.hw.stop_timer(); + self.cpu.armed = None; + } } - self.cpu.armed = plan.armed(); // Invariant T is a statement about a CPU *outside* a pass, and the // arming that makes it true is the last thing a pass does — so the // check cannot move earlier. #[cfg(feature = "sched-check")] crate::sched::invariants::check_cpu(self.cpu); - TimerApplied::new(plan.armed()) + TimerApplied::new(self.cpu.armed) } fn switch_to_current(&mut self) -> Action<::Payload> { @@ -2551,9 +2563,14 @@ impl CpuHandle { } /// Every ask since the last take, emptied: this CPU's own pass, after its - /// `begin_pass`. + /// `begin_pass`. A word read empty is not swapped: an ask landing after + /// the read rings the doorbell after it, and the pass that ring owes + /// takes it. pub fn take_asks(&self) -> [u64; MASK_WORDS] { - core::array::from_fn(|word| self.asks[word].swap(0, Ordering::Relaxed)) + core::array::from_fn(|word| match self.asks[word].load(Ordering::Relaxed) { + 0 => 0, + _ => self.asks[word].swap(0, Ordering::Relaxed), + }) } pub fn publish_weight(&self, weight: u64) { @@ -2937,6 +2954,9 @@ mod tests { timers: Vec>, /// The revisit an idle plan asks, where a test sets one. revisit: Option, + /// The last arm's one-shot, not yet fired: a test fires it by + /// clearing this. + pending: bool, } #[derive(Default)] @@ -2962,10 +2982,17 @@ mod tests { Nanos::ZERO } fn set_timer(&self, deadline: Nanos) { - self.state().timers.push(Some(deadline)); + let mut state = self.state(); + state.timers.push(Some(deadline)); + state.pending = true; } fn stop_timer(&self) { - self.state().timers.push(None); + let mut state = self.state(); + state.timers.push(None); + state.pending = false; + } + fn timer_pending(&self) -> bool { + self.state().pending } fn idle_plan(&self, deadline: Option, now: Nanos) -> crate::sched::idle::Plan { let mut plan = crate::sched::idle::Plan::c1(deadline.map(|d| d.0), now.0); @@ -4922,6 +4949,70 @@ mod tests { assert_eq!(w.hw.state().timers, [None], "a plan with no wake leaves the plan main makes: stopped"); } + /// **A running CPU writes its timer once a request, not once a pass.** A + /// pass that finds its one-shot pending and due no later than it needs + /// leaves it, so a thread that keeps entering the kernel re-arms nothing; + /// once it fired, or once something is owed sooner, the pass writes. + #[test] + fn a_running_cpu_leaves_a_pending_one_shot_due_no_later() { + let mut w = World::new(1); + w.spawn(C0); + w.run_a_pass_at(C0, NOW); + let armed = w.cpus[0].armed().expect("a running thread's request is armed"); + assert_eq!(w.hw.state().timers, [Some(armed)]); + + w.run_a_pass_at(C0, NOW.after(1_000)); + assert_eq!(w.hw.state().timers, [Some(armed)], "the request's end has not moved: nothing written"); + assert_eq!(w.cpus[0].armed(), Some(armed)); + + w.hw.state().pending = false; + w.run_a_pass_at(C0, NOW.after(2_000)); + assert_eq!(w.hw.state().timers.len(), 2, "a fired one-shot is armed again"); + + // A sleeper due before the request's end is owed sooner. + let (_, sleeper) = w.spawn(C0); + let (cpus, env) = w.split(); + let pass = SchedPass::begin(&mut cpus[0], env, NOW.after(3_000)); + let _ = pass.dispose_yield().finish(); + let due = NOW.after(50_000); + let (cpus, env) = w.split(); + let sched = &mut cpus[0]; + let current = CurrentTask::new(sched.running().expect("one of them runs").shared(), C0); + let ticket = crate::sched::park::prepare(¤t, crate::sched::park::Cancel::Answers, WaitClass::Other) + .expect("nothing notified this task"); + let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") }; + let pass = SchedPass::begin(sched, env, NOW.after(4_000)); + let _ = pass.dispose_block(committed, Some(due)).finish(); + assert!(w.cpus[0].running().is_some(), "the other thread runs"); + assert_eq!(w.cpus[0].armed(), Some(due), "a deadline before the armed one-shot is written"); + assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due)); + let _ = sleeper; + w.abandon(); + } + + /// **An idle CPU arms exactly**: a pending one-shot due before the + /// deadline it parks with is not left to wake it early. + #[test] + fn a_cpu_going_idle_writes_its_deadline_over_a_pending_earlier_one_shot() { + let mut w = World::new(1); + w.spawn(C0); + w.run_a_pass_at(C0, NOW); + let armed = w.cpus[0].armed().expect("armed for the running thread"); + let due = armed.after(1_000_000); + assert!(w.hw.state().pending); + let (cpus, env) = w.split(); + let sched = &mut cpus[0]; + let current = CurrentTask::new(sched.running().expect("it runs").shared(), C0); + let ticket = crate::sched::park::prepare(¤t, crate::sched::park::Cancel::Answers, WaitClass::Other) + .expect("nothing notified this task"); + let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") }; + let pass = SchedPass::begin(sched, env, NOW.after(1_000)); + let _ = pass.dispose_block(committed, Some(due)).finish(); + assert_eq!(w.cpus[0].armed(), Some(due)); + assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due), "the idle CPU's deadline is written"); + w.abandon(); + } + /// **Locality first, then a free physical core, then rotation**: a forward /// or push from cpu0 takes a core whose threads all sleep in its own LLC, /// shares a busy core there before crossing to the other LLC's free ones, diff --git a/kernel/pure/sched/hw.rs b/kernel/pure/sched/hw.rs index 50a1b0bf280..22f4d42f830 100644 --- a/kernel/pure/sched/hw.rs +++ b/kernel/pure/sched/hw.rs @@ -98,6 +98,14 @@ pub trait Machine: Kicker + 'static { fn stop_timer(&self); + /// Whether the one-shot the last [`Self::set_timer`] armed is still to + /// be taken: its interrupt not yet handled, so a pass is still owed by + /// it. A machine that cannot tell answers `false`, and the core then + /// writes every arm. + fn timer_pending(&self) -> bool { + false + } + /// The idle state this CPU waits in once the pass that asks ends, chosen /// with the scheduler's timer due at `deadline` (`crate::sched::idle`). /// The pass folds [`idle::Plan::wake_at`] into its one timer plan, and the diff --git a/kernel/pure/sched/task.rs b/kernel/pure/sched/task.rs index 861a9e98433..6632408ed14 100644 --- a/kernel/pure/sched/task.rs +++ b/kernel/pure/sched/task.rs @@ -14,7 +14,7 @@ use crate::sched::hw::{CpuId, Nanos}; use crate::sched::job::Job; use crate::sched::mailbox::MailboxNode; use crate::sched::msg::Msg; -use crate::sched::affinity::{affinity_fence, CpuMask, Reach, SharedMask}; +use crate::sched::affinity::{affinity_fence, CpuMask, LoadedMask, Reach, SharedMask}; use crate::sched::sync::{Arc, AtomicBool, AtomicU64, Ordering}; use crate::sched::park::CommittedTicket; use crate::sched::queue::{next_request, SLICE_NS}; @@ -737,10 +737,9 @@ struct TaskInner { ctx: X::Ctx, rt: RtState, acct: TaskAccounting, - /// The owner CPU's copy of [`TaskShared::mask`], its reach and the - /// version it was loaded at. Changed only by [`Task::refresh_affinity`]. - affinity: CpuMask, - reach: Reach, + /// The owner CPU's copy of [`TaskShared::mask`] and the version it was + /// loaded at. Changed only by [`Task::refresh_affinity`]. + affinity: LoadedMask, affinity_version: u64, /// What it asks of a queue at a time, in ns: [`next_request`] of its /// bursts, each the CPU time from a wake to the next park. @@ -797,11 +796,11 @@ impl Task { } pub fn affinity(&self) -> CpuMask { - self.0.affinity + self.0.affinity.mask() } pub fn reach(&self) -> Reach { - self.0.reach + self.0.affinity.reach() } pub fn request_ns(&self) -> u64 { @@ -814,8 +813,8 @@ impl Task { if self.0.shared.mask().version() == self.0.affinity_version { return false; } - (self.0.affinity, self.0.affinity_version) = self.0.shared.mask().load(); - self.0.reach = self.0.affinity.reach(); + let (mask, version) = self.0.shared.mask().load(); + (self.0.affinity, self.0.affinity_version) = (LoadedMask::new(mask), version); true } @@ -980,8 +979,7 @@ impl TaskBuilder { ctx: self.ctx, rt: self.rt, acct: TaskAccounting::default(), - affinity: self.mask, - reach: self.mask.reach(), + affinity: LoadedMask::new(self.mask), affinity_version: 0, request: SLICE_NS, woke_cpu_ns: 0, diff --git a/kernel/pure/sched/timer.rs b/kernel/pure/sched/timer.rs index 6d65e6e48e7..c861aa1bfd9 100644 --- a/kernel/pure/sched/timer.rs +++ b/kernel/pure/sched/timer.rs @@ -69,7 +69,8 @@ impl TimerPlan { } } -/// Proof that a [`TimerPlan`] reached the hardware. [`crate::sched::cpu::SleepToken`] +/// Proof that the hardware holds a [`TimerPlan`], or a one-shot still to fire +/// that is due no later. [`crate::sched::cpu::SleepToken`] /// cannot be built without one, so "halted with a deadline pending and the /// timer unarmed" is unrepresentable rather than asserted. pub struct TimerApplied { diff --git a/kernel/sim/src/hw_impl.rs b/kernel/sim/src/hw_impl.rs index 22b851904cd..6c5defe54a8 100644 --- a/kernel/sim/src/hw_impl.rs +++ b/kernel/sim/src/hw_impl.rs @@ -179,6 +179,12 @@ impl Machine for SimHw { }); } + /// A fire disarms the modelled one-shot (`Step::FireTimer`), so one still + /// armed is one whose pass is still owed. + fn timer_pending(&self) -> bool { + self.with(|s| s.armed[Self::cpu(s)].is_some()) + } + fn halt(&self) { self.with(|s| { let cpu = Self::cpu(s); diff --git a/kernel/src/arch/aarch64/irqchip.rs b/kernel/src/arch/aarch64/irqchip.rs index 8d7d428d026..b8fde4ebf68 100644 --- a/kernel/src/arch/aarch64/irqchip.rs +++ b/kernel/src/arch/aarch64/irqchip.rs @@ -496,6 +496,7 @@ fn stop_timer_hardware() { /// This CPU's timer, armed to fire `nanos` from now, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { arm_ticks(crate::clock::counter_ticks(nanos)); + percpu::set_timer_pending(true); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } @@ -534,8 +535,16 @@ pub fn rearmed_a_quantum(armed: u64) -> bool { comparator() >= armed + crate::clock::counter_ticks(kernel::sched::task::QUANTUM_NS) } +/// Whether the one-shot [`arm_one_shot`] last armed is still to fire: both +/// of the timer's entry paths clear it, and [`arm_within`] leaves it, since +/// what it arms is never later than what it found. +pub fn timer_pending() -> bool { + percpu::timer_pending() +} + /// Stop the timer: no interrupt until it is armed again. pub fn stop_timer() { + percpu::set_timer_pending(false); percpu::set_armed_ticks(0); stop_timer_hardware(); crate::trace::trace(crate::trace::Kind::TimerStop, 0); @@ -544,6 +553,7 @@ pub fn stop_timer() { /// A timer interrupt taken from EL0: armed again for what it was last armed /// for, or stopped if it was stopped — the one thing that deasserts it. pub(super) fn rearm() { + percpu::set_timer_pending(false); match percpu::armed_ticks() { 0 => stop_timer_hardware(), ticks => { @@ -555,6 +565,7 @@ pub(super) fn rearm() { /// A timer interrupt taken at EL1: one quantum on, or stopped if it was /// stopped, leaving what the scheduler armed for its next pass to re-arm. pub(super) fn rearm_in_kernel() { + percpu::set_timer_pending(false); match percpu::armed_ticks() { 0 => stop_timer_hardware(), _ => compare_at(cpu::counter() + crate::clock::counter_ticks(kernel::sched::task::QUANTUM_NS)), diff --git a/kernel/src/arch/aarch64/percpu.rs b/kernel/src/arch/aarch64/percpu.rs index 899782ea6d1..65eebf2ed89 100644 --- a/kernel/src/arch/aarch64/percpu.rs +++ b/kernel/src/arch/aarch64/percpu.rs @@ -8,7 +8,7 @@ //! an exception on this CPU cannot split (Arm ARM K.a, B2.17.5: taking an //! exception clears the monitor). -use core::sync::atomic::{AtomicU32, AtomicU64, AtomicU8, Ordering::Relaxed}; +use core::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicU8, Ordering::Relaxed}; use alloc::boxed::Box; @@ -41,6 +41,9 @@ pub struct PerCpu { /// Counter ticks the timer was last armed for: what a timer interrupt /// taken at EL0 re-arms it with, and zero when it is stopped. armed_ticks: AtomicU64, + /// Whether the scheduler's last one-shot is armed and its interrupt not + /// yet taken. + timer_pending: AtomicBool, /// The kernel stack a switch last installed, for the stack witness: on /// AArch64 an entry from EL0 lands on `SP_EL1` as the last `ERET` left it, /// and nothing here points it anywhere. @@ -86,6 +89,7 @@ pub fn alloc(cpu_id: u32) -> &'static PerCpu { kernel_timer_fires: AtomicU32::new(0), last_seen_kernel_timer_fires: AtomicU32::new(0), armed_ticks: AtomicU64::new(0), + timer_pending: AtomicBool::new(false), kernel_stack: AtomicU64::new(0), // The same size as a task's kernel stack: a `deferred` [`kobject!`] // object may own an `immediate` one, whose destructor then runs here. @@ -338,3 +342,11 @@ pub(super) fn armed_ticks() -> u64 { pub(super) fn set_armed_ticks(ticks: u64) { this().armed_ticks.store(ticks, Relaxed); } + +pub(super) fn timer_pending() -> bool { + this().timer_pending.load(Relaxed) +} + +pub(super) fn set_timer_pending(pending: bool) { + this().timer_pending.store(pending, Relaxed); +} diff --git a/kernel/src/arch/x86_64/apic.rs b/kernel/src/arch/x86_64/apic.rs index 44a40be68d9..083a55b016b 100644 --- a/kernel/src/arch/x86_64/apic.rs +++ b/kernel/src/arch/x86_64/apic.rs @@ -380,6 +380,7 @@ impl OneShot { /// Arm a one-shot timer to fire after `nanos` nanoseconds, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { OneShot::after(nanos).arm(); + percpu::set_timer_pending(true); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } @@ -408,19 +409,29 @@ pub fn rearmed_a_quantum(_armed: u64) -> bool { /// The Ring 3 fire's re-arm, before its Rust half runs: what this CPU last /// armed, again, so the timer survives a handler that panics first. pub(crate) extern "sysv64" fn rearm_last() { + percpu::set_timer_pending(false); Reg::TIMER_INIT.write(percpu::last_armed_ticks()); } /// The Ring 0 fire's local-APIC work, called from its stub with interrupts /// closed: the EOI, then one quantum on, or a stopped timer left stopped. pub(crate) extern "sysv64" fn ring0_fire() { + percpu::set_timer_pending(false); eoi(); let stopped = percpu::last_armed_ticks() == 0; Reg::TIMER_INIT.write(if stopped { 0 } else { TIMER_TICKS.load(Ordering::Relaxed) }); } +/// Whether the one-shot [`arm_one_shot`] last armed is still to fire: both of +/// the timer's entry paths clear it, and [`arm_within`] leaves it, since what +/// it arms is never later than what it found. +pub fn timer_pending() -> bool { + percpu::timer_pending() +} + /// Stop the timer. No more interrupts until re-armed. pub fn stop_timer() { + percpu::set_timer_pending(false); percpu::set_last_armed_ticks(0); Reg::TIMER_INIT.write(0); crate::trace::trace(crate::trace::Kind::TimerStop, 0); diff --git a/kernel/src/arch/x86_64/percpu.rs b/kernel/src/arch/x86_64/percpu.rs index ff1b12e8900..5af68fd4157 100644 --- a/kernel/src/arch/x86_64/percpu.rs +++ b/kernel/src/arch/x86_64/percpu.rs @@ -109,7 +109,9 @@ pub struct PerCpu { pub ring0_timer_fires: AtomicU32, pub last_seen_ring0_fires: u32, fault_state: u8, - _pad_after_fault_state: [u8; 3], + /// Non-zero while the scheduler's last one-shot is armed and its interrupt not yet taken. + timer_pending: u8, + _pad_after_timer_pending: [u8; 2], /// Ticks the Ring 3 timer branch re-arms with, and zero when stopped, which the Ring 0 branch leaves stopped. pub last_armed_ticks: AtomicU32, /// This CPU's [`log::Shard`]; never null on a live CPU ([`alloc_percpu`] fills it first). @@ -209,6 +211,7 @@ pub(crate) const OFF_PREEMPT_COUNT: u32 = offset_of!(PerCpu, preempt_count) as u pub(crate) const OFF_NEED_RESCHED: u32 = offset_of!(PerCpu, need_resched) as u32; /// Read by `preempt::enable`'s slow path to decline rescheduling a faulting CPU. pub(crate) const OFF_FAULT_STATE: u32 = offset_of!(PerCpu, fault_state) as u32; +const OFF_TIMER_PENDING: u32 = offset_of!(PerCpu, timer_pending) as u32; /// Set/cleared only by `arch::idt::nmi`'s naked entry, via this constant. pub(crate) const OFF_NMI_ACTIVE: u32 = offset_of!(PerCpu, nmi_active) as u32; /// The AP's bring-up token, read by `ap_entry` to answer for its own attempt. @@ -371,7 +374,8 @@ fn alloc_percpu(cpu_id: u32) -> *mut PerCpu { ring0_timer_fires: AtomicU32::new(0), last_seen_ring0_fires: 0, fault_state: CpuFaultState::Normal as u8, - _pad_after_fault_state: [0; 3], + timer_pending: 0, + _pad_after_timer_pending: [0; 2], last_armed_ticks: AtomicU32::new(0), log_shard: log::shard_for(cpu_id) as *const log::Shard as u64, nmi_active: 0, @@ -684,6 +688,15 @@ pub fn last_armed_ticks() -> u32 { gs::read_u32::() } +/// Whether the scheduler's one-shot is armed and its interrupt not yet taken; `arch::apic` is the only writer. +pub fn set_timer_pending(pending: bool) { + gs::write_u8::(u8::from(pending)); +} + +pub fn timer_pending() -> bool { + gs::read_u8::() != 0 +} + /// The last byte of this CPU's idle guard page — the first byte an overflow reaches. #[cfg(feature = "test-actuators")] pub fn idle_guard_byte() -> u64 { diff --git a/kernel/src/hw.rs b/kernel/src/hw.rs index 80d0cc040db..03faf65fb13 100644 --- a/kernel/src/hw.rs +++ b/kernel/src/hw.rs @@ -62,6 +62,10 @@ impl Machine for KernelHw { irqchip::stop_timer(); } + fn timer_pending(&self) -> bool { + irqchip::timer_pending() + } + fn idle_plan(&self, deadline: Option, now: Nanos) -> kernel::sched::idle::Plan { crate::idle::plan(deadline, now) } From dee81903eb781a2feb02e805e4e3426f4facfc15 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:34:29 +0200 Subject: [PATCH 2/8] A pass takes an ask once Reading an ask word before swapping it made a take that leaves the word set representable, and no test held that a taken ask is gone. Co-Authored-By: Claude Opus 5.5 --- kernel/pure/sched/cpu.rs | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 9a9d87bd129..a819a309ccb 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -4990,6 +4990,20 @@ mod tests { w.abandon(); } + /// A pass takes an ask once: the word it found set is left empty, and the + /// next take finds nothing. + #[test] + fn a_taken_ask_is_gone() { + let w = World::new(2); + let victim = w.handles.get(C0); + let _ = victim.ask(C1); + victim.doorbell().begin_pass(); + assert_eq!(victim.take_asks()[0], 1 << C1.0, "the ask is taken"); + assert!(!victim.asked_by(C1)); + assert_eq!(victim.take_asks(), [0; MASK_WORDS], "and taken once"); + w.abandon(); + } + /// **An idle CPU arms exactly**: a pending one-shot due before the /// deadline it parks with is not left to wake it early. #[test] From 9e64f2b04fa4c6e7bdc6c8fce6c3f646a5b67066 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:38:05 +0200 Subject: [PATCH 3/8] issues: the TSC-deadline record names the change that re-arms once a request Co-Authored-By: Claude Opus 5.5 --- issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md index cbf0f60aa29..9ebab2f5e5e 100644 --- a/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md @@ -28,7 +28,7 @@ CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, `CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration and its spin. -**A running CPU re-arms once a request, since #PRNUM.** A pass on a CPU +**A running CPU re-arms once a request, since #1263.** A pass on a CPU that keeps running leaves a one-shot that is still to fire and due no later than it needs, so the timer is written once a request, after a fire, or when something is owed sooner; a CPU going idle still arms exactly at every sleep From 24582905f9f14b65f0dfe01ec739a3449dfd5fb5 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 19:50:12 +0200 Subject: [PATCH 4/8] The timer test names no binding it does not use Co-Authored-By: Claude Opus 5.5 --- kernel/pure/sched/cpu.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index a819a309ccb..81d6decd486 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -4970,7 +4970,7 @@ mod tests { assert_eq!(w.hw.state().timers.len(), 2, "a fired one-shot is armed again"); // A sleeper due before the request's end is owed sooner. - let (_, sleeper) = w.spawn(C0); + w.spawn(C0); let (cpus, env) = w.split(); let pass = SchedPass::begin(&mut cpus[0], env, NOW.after(3_000)); let _ = pass.dispose_yield().finish(); @@ -4986,7 +4986,6 @@ mod tests { assert!(w.cpus[0].running().is_some(), "the other thread runs"); assert_eq!(w.cpus[0].armed(), Some(due), "a deadline before the armed one-shot is written"); assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due)); - let _ = sleeper; w.abandon(); } From 481810424f544abe4426962116430782635e63cd Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 20:22:12 +0200 Subject: [PATCH 5/8] The corpus's old-protocol traces are found again under the timer's steps A running CPU that leaves an earlier one-shot pending fires it where it used to be rewritten, so the simulator offers timer steps where it did not, and two committed decision traces no longer drove their old protocols into the failure they were recorded for. They are found again (`kernel-sim find`, a search for I8, and `shrink`): `old_steal_port_i8` from seed 10, `old_migrate_kept_the_corpse_i14` from seed 70, its notes kept. Both fail as committed again. Co-Authored-By: Claude Opus 5.5 --- kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace | 4 ++-- kernel/sim/corpus/old_steal_port_i8.trace | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace index 3b7038762d7..13d6b09640a 100644 --- a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace +++ b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace @@ -10,5 +10,5 @@ # note: job-group scheduler's request-length decisions, and again when a steal # note: probe became a bit in its victim's word instead of a message. # note: it fails I14's "killed and then migrated" half. -4 3 3 0 0 0 1 1 2 0 2 0 2 0 0 0 0 1 2 1 1 2 1 0 1 0 0 1 1 0 1 1 -0 0 0 0 0 0 0 1 1 1 0 2 0 0 1 1 0 1 0 1 1 2 +0 0 1 1 1 0 2 0 0 1 0 0 2 0 1 2 1 3 0 0 2 1 2 1 0 0 1 0 0 0 0 1 +0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 1 2 1 diff --git a/kernel/sim/corpus/old_steal_port_i8.trace b/kernel/sim/corpus/old_steal_port_i8.trace index 575c24e7f06..d1f72220807 100644 --- a/kernel/sim/corpus/old_steal_port_i8.trace +++ b/kernel/sim/corpus/old_steal_port_i8.trace @@ -1,3 +1,4 @@ # scenario: old_steal_port # expect: fail -1 1 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 1 +2 0 2 1 0 1 0 0 0 0 0 0 0 0 0 0 2 1 0 1 0 0 1 1 0 1 1 1 0 0 1 1 +1 0 0 0 0 2 2 0 0 1 1 1 0 1 1 0 0 1 2 0 1 0 1 1 1 1 1 1 1 From 9633182d07f5529444af7bea6f6e79ca92bc7cb4 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 21:49:52 +0200 Subject: [PATCH 6/8] The timer's pending flag goes up before its write A pass from a system call runs with interrupts open, so a fire taken between the hardware write and the flag's store cleared a flag that the store then raised again, over a timer the fire had re-armed a quantum on: every later pass that needed nothing sooner skipped its write, and the running request's end waited for the quantum, up to 10 ms late. The flag is now raised first, so a fire between the two clears it and the next pass writes again. AArch64's comparator write is a `nomem` asm, so a compiler fence holds the store before it. The setter is `pub(super)` on both architectures. Co-Authored-By: Claude Opus 5.5 --- kernel/src/arch/aarch64/irqchip.rs | 6 +++++- kernel/src/arch/x86_64/apic.rs | 4 +++- kernel/src/arch/x86_64/percpu.rs | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/kernel/src/arch/aarch64/irqchip.rs b/kernel/src/arch/aarch64/irqchip.rs index b8fde4ebf68..7bc98d0121b 100644 --- a/kernel/src/arch/aarch64/irqchip.rs +++ b/kernel/src/arch/aarch64/irqchip.rs @@ -495,8 +495,12 @@ fn stop_timer_hardware() { /// This CPU's timer, armed to fire `nanos` from now, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { - arm_ticks(crate::clock::counter_ticks(nanos)); + // Before the write, and held there against the `nomem` comparator write: a + // fire taken between the two then clears it, and the next pass writes again + // rather than trusting a quantum's re-arm. percpu::set_timer_pending(true); + core::sync::atomic::compiler_fence(core::sync::atomic::Ordering::SeqCst); + arm_ticks(crate::clock::counter_ticks(nanos)); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } diff --git a/kernel/src/arch/x86_64/apic.rs b/kernel/src/arch/x86_64/apic.rs index 083a55b016b..2da39b07ae9 100644 --- a/kernel/src/arch/x86_64/apic.rs +++ b/kernel/src/arch/x86_64/apic.rs @@ -379,8 +379,10 @@ impl OneShot { /// Arm a one-shot timer to fire after `nanos` nanoseconds, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { - OneShot::after(nanos).arm(); + // Before the write: a fire taken between the two then clears it, and the + // next pass writes again rather than trusting a quantum's re-arm. percpu::set_timer_pending(true); + OneShot::after(nanos).arm(); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } diff --git a/kernel/src/arch/x86_64/percpu.rs b/kernel/src/arch/x86_64/percpu.rs index 5af68fd4157..b8d8759df45 100644 --- a/kernel/src/arch/x86_64/percpu.rs +++ b/kernel/src/arch/x86_64/percpu.rs @@ -689,7 +689,7 @@ pub fn last_armed_ticks() -> u32 { } /// Whether the scheduler's one-shot is armed and its interrupt not yet taken; `arch::apic` is the only writer. -pub fn set_timer_pending(pending: bool) { +pub(super) fn set_timer_pending(pending: bool) { gs::write_u8::(u8::from(pending)); } From 2350163385a52da564f1a19b0123ce4ca53f9953 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 21:49:52 +0200 Subject: [PATCH 7/8] A corpus trace names the invariant it was committed for `# expect: fail I14` and its kind say which invariant a failing trace must still break; the corpus test checks it, so a replay that fails only some other way is no longer read as the case it was kept for. The three old-protocol traces name theirs. `old_steal_port_i8` is found again at this head (seed 5, shrunk); the other two still break theirs, I1 and I14's killed-then-migrated task. Co-Authored-By: Claude Opus 5.5 --- kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace | 2 +- kernel/sim/corpus/old_steal_port_i1.trace | 2 +- kernel/sim/corpus/old_steal_port_i8.trace | 6 +++--- kernel/sim/src/shrink.rs | 9 ++++++++- kernel/sim/tests/scenarios.rs | 8 ++++++++ 5 files changed, 21 insertions(+), 6 deletions(-) diff --git a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace index 13d6b09640a..cf883b8c7c9 100644 --- a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace +++ b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace @@ -1,5 +1,5 @@ # scenario: old_migrate_kept_the_corpse -# expect: fail +# expect: fail I14 # note: the T14's panic, minimized — retire_task: task not released after 1s: # note: InTransit(CpuId(1)). soundserver boost-wakes a client thread whose process # note: has just been torn down, on the CPU soundserver itself is running, with a diff --git a/kernel/sim/corpus/old_steal_port_i1.trace b/kernel/sim/corpus/old_steal_port_i1.trace index f2b7e8da255..0c05a85734f 100644 --- a/kernel/sim/corpus/old_steal_port_i1.trace +++ b/kernel/sim/corpus/old_steal_port_i1.trace @@ -1,3 +1,3 @@ # scenario: old_steal_port -# expect: fail +# expect: fail I1 0 1 1 2 diff --git a/kernel/sim/corpus/old_steal_port_i8.trace b/kernel/sim/corpus/old_steal_port_i8.trace index d1f72220807..0e63c8b110a 100644 --- a/kernel/sim/corpus/old_steal_port_i8.trace +++ b/kernel/sim/corpus/old_steal_port_i8.trace @@ -1,4 +1,4 @@ # scenario: old_steal_port -# expect: fail -2 0 2 1 0 1 0 0 0 0 0 0 0 0 0 0 2 1 0 1 0 0 1 1 0 1 1 1 0 0 1 1 -1 0 0 0 0 2 2 0 0 1 1 1 0 1 1 0 0 1 2 0 1 0 1 1 1 1 1 1 1 +# expect: fail I8 +3 0 0 1 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 +0 0 1 1 1 0 0 2 0 0 1 1 1 0 0 1 1 0 1 diff --git a/kernel/sim/src/shrink.rs b/kernel/sim/src/shrink.rs index 64c6def5327..569e662e2e3 100644 --- a/kernel/sim/src/shrink.rs +++ b/kernel/sim/src/shrink.rs @@ -100,18 +100,24 @@ pub fn encode(scenario: &str, expect_failure: bool, decisions: &[usize]) -> Stri pub struct CorpusEntry { pub scenario: String, pub expect_failure: bool, + /// The invariant a failing trace was committed for, as `# expect: fail I14` + /// names it: a replay that fails only some other way has lost its case. + pub expect_violation: Option, pub decisions: Vec, } pub fn decode(text: &str) -> CorpusEntry { let mut scenario = String::new(); let mut expect_failure = false; + let mut expect_violation = None; let mut decisions = Vec::new(); for line in text.lines() { if let Some(rest) = line.strip_prefix("# scenario:") { scenario = rest.trim().to_string(); } else if let Some(rest) = line.strip_prefix("# expect:") { - expect_failure = rest.trim() == "fail"; + let mut words = rest.split_whitespace(); + expect_failure = words.next() == Some("fail"); + expect_violation = words.next().map(str::to_string); } else if !line.starts_with('#') { decisions.extend(line.split_whitespace().map(|token| { token @@ -124,6 +130,7 @@ pub fn decode(text: &str) -> CorpusEntry { CorpusEntry { scenario, expect_failure, + expect_violation, decisions, } } diff --git a/kernel/sim/tests/scenarios.rs b/kernel/sim/tests/scenarios.rs index bc33ec5322b..5e239b233fb 100644 --- a/kernel/sim/tests/scenarios.rs +++ b/kernel/sim/tests/scenarios.rs @@ -592,6 +592,14 @@ fn corpus_traces_still_do_what_they_were_committed_for() { }, outcome.report(), ); + if let Some(invariant) = &entry.expect_violation { + assert!( + outcome.violations.iter().any(|v| v.starts_with(&format!("{invariant}:"))), + "{}: expected {invariant}, got {}", + path.display(), + outcome.report(), + ); + } checked += 1; } assert!( From 49095e6e6daff9a86843307162d67a69fe11c0f7 Mon Sep 17 00:00:00 2001 From: japabu Date: Wed, 11 Nov 2026 22:05:51 +0200 Subject: [PATCH 8/8] issues: what #1263 owes on metal Co-Authored-By: Claude Opus 5.5 --- issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md index 9ebab2f5e5e..c8a6147badb 100644 --- a/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md @@ -81,6 +81,8 @@ lower and the two spreads do not overlap; where they overlap, the count, the path every CPU has, is kept. The split is reported. The twin is a measurement arm, deleted once it has answered. +**Owed on metal:** one A/B of main from before and after #1263 (a pass skips the timer write and the ask swaps it does not owe), on the T14 and the AMD bench, of osbench's `system`, `sched`, `ipc` and `time` groups, with a Linux reference timed through the TSC and its `.slack1ns` rows. Main must read `time.yield`, `pingpong.*` and `ipc.reqrep.*` faster by more than 3% on both benches, `time.sleep_late.*`, `time.timer_1ms.late` and `wake.periodic.*` within 3%, and no scheduler, IPC, wake, yield, spawn or timer row slower. The speed-up is not yet measured on metal. + **Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it cited for bit 24; the twin's reading, posted, and the twin deleted. Where the deadline's lateness measures better, a deadline is armed on every CPU that