A pass skips the timer write and the ask swaps it does not owe

issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md → issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md

@@ -4,10 +4,10 @@ kind: track
 opened: 2026-10-12
 ---
 
-# The timer is a LAPIC count re-armed every pass, never a TSC deadline
+# The timer is a LAPIC count, never a TSC deadline
 
-Every scheduler pass ends by handing its timer an absolute deadline
-(`kernel/pure/sched/cpu.rs:1912-1918`), and the kernel turns it back into a
+A scheduler pass ends by handing its timer an absolute deadline
+(`SchedPass::apply_timer_no_later_than`), and the kernel turns it back into a
 span from now (`kernel/src/hw.rs:47-51`) that `apic::arm_one_shot` converts to
 LAPIC ticks by the boot's calibration against the TSC-derived clock
 (`kernel/src/arch/x86_64/apic.rs`) and writes into `TIMER_INIT`, one APIC
@@ -28,6 +28,15 @@ CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3,
 `CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration
 and its spin.
 
+**A running CPU re-arms once a request, since #1263.** A pass on a CPU
+that keeps running leaves a one-shot that is still to fire and due no later
+than it needs, so the timer is written once a request, after a fire, or when
+something is owed sooner; a CPU going idle still arms exactly at every sleep
+and stops at every idle entry. What a deadline would save is now per arm and
+not per pass, which leaves it the idle path's arm and stop and the boot's
+calibration spin, on the T14 alone. The figures below are the old code's,
+which wrote at every pass.
+
 **Its speed reading decides it.** The SDM defines the mode and recommends it
 over the count nowhere, and AMD documents none, so nothing published covers
 the choice. Intel's specification update for the T14's processor lists no
@@ -72,6 +81,8 @@ lower and the two spreads do not overlap; where they overlap, the count, the
 path every CPU has, is kept. The split is reported. The twin is a measurement
 arm, deleted once it has answered.
 
+**Owed on metal:** one A/B of main from before and after #1263 (a pass skips the timer write and the ask swaps it does not owe), on the T14 and the AMD bench, of osbench's `system`, `sched`, `ipc` and `time` groups, with a Linux reference timed through the TSC and its `.slack1ns` rows. Main must read `time.yield`, `pingpong.*` and `ipc.reqrep.*` faster by more than 3% on both benches, `time.sleep_late.*`, `time.timer_1ms.late` and `wake.periodic.*` within 3%, and no scheduler, IPC, wake, yield, spawn or timer row slower. The speed-up is not yet measured on metal.
+
 **Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it
 cited for bit 24; the twin's reading, posted, and the twin deleted. Where the
 deadline's lateness measures better, a deadline is armed on every CPU that

issues/toyos-uses-what-modern-hardware-offers-for-speed.md

@@ -46,7 +46,7 @@ and contention).
 - A TSC deadline: before #1061 cut the arm to one register write, the
   re-arm was at most 11.7 to 12.1 % of a pass's pick and switch on the T14,
   and at the AMD bench's TSC floor; it is to be read again
-  (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`).
+  (`issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md`).
 - Deep idle: on the T14 `hlt` holds a one-thread load to the turbo ratio
   for three or more active cores, 38 of `MSR_TURBO_RATIO_LIMIT`'s
   `0x2626262626262a2a` against 42 for one or two (#866's dump,
@@ -88,7 +88,7 @@ yet measured.
 - `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`
 - `issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md`
 - `issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`
-- `issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`
+- `issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md`
 - `issues/no-gate-decodes-kernel-elfs-instructions.md`
 - `issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md`
 - `issues/the-direct-map-uses-1-gib-leaves-where-the-cpu-has-them.md`

kernel/pure/sched/affinity.rs

@@ -106,15 +106,34 @@ impl CpuMask {
     pub fn words(&self) -> &[u64; MASK_WORDS] {
         &self.0
     }
+}
 
-    pub fn reach(&self) -> Reach {
-        if *self == Self::ANY {
+/// A task's mask as its owner CPU loaded it, with the [`Reach`] a run queue
+/// counts it by: one value, so the two cannot disagree.
+#[derive(Clone, Copy, Debug)]
+pub struct LoadedMask {
+    mask: CpuMask,
+    reach: Reach,
+}
+
+impl LoadedMask {
+    pub fn new(mask: CpuMask) -> Self {
+        let reach = if mask == CpuMask::ANY {
             Reach::Any
-        } else if self.single().is_some() {
+        } else if mask.single().is_some() {
             Reach::One
         } else {
             Reach::Several
-        }
+        };
+        Self { mask, reach }
+    }
+
+    pub fn mask(&self) -> CpuMask {
+        self.mask
+    }
+
+    pub fn reach(&self) -> Reach {
+        self.reach
     }
 }
 

kernel/pure/sched/cpu.rs

@@ -2186,18 +2186,30 @@ impl<H: Hw, P: PreemptGuard> SchedPass<'_, '_, H, P, Disposed> {
             Some(_) => Some(self.cpu.settle_decision(self.now)),
             None => None,
         };
-        let plan = TimerPlan::compute(decision, deadline).no_later_than(extra);
-        match plan {
-            TimerPlan::Arm(at) => self.env.hw.set_timer(at),
-            TimerPlan::Stop => self.env.hw.stop_timer(),
+        match TimerPlan::compute(decision, deadline).no_later_than(extra) {
+            // A one-shot still to fire and due no later is left: an early fire
+            // is a spurious pass, so a running CPU writes its timer once a
+            // request rather than once a pass. An idle CPU arms exactly, so
+            // nothing wakes it before it is owed.
+            TimerPlan::Arm(at)
+                if self.cpu.running.is_some()
+                    && self.cpu.armed.is_some_and(|armed| self.now < armed && armed <= at)
+                    && self.env.hw.timer_pending() => {}
+            TimerPlan::Arm(at) => {
+                self.env.hw.set_timer(at);
+                self.cpu.armed = Some(at);
+            }
+            TimerPlan::Stop => {
+                self.env.hw.stop_timer();
+                self.cpu.armed = None;
+            }
         }
-        self.cpu.armed = plan.armed();
         // Invariant T is a statement about a CPU *outside* a pass, and the
         // arming that makes it true is the last thing a pass does — so the
         // check cannot move earlier.
         #[cfg(feature = "sched-check")]
         crate::sched::invariants::check_cpu(self.cpu);
-        TimerApplied::new(plan.armed())
+        TimerApplied::new(self.cpu.armed)
     }
 
     fn switch_to_current(&mut self) -> Action<<H as Hw>::Payload> {
@@ -2551,9 +2563,14 @@ impl<M: SchedMsg> CpuHandle<M> {
     }
 
     /// Every ask since the last take, emptied: this CPU's own pass, after its
-    /// `begin_pass`.
+    /// `begin_pass`. A word read empty is not swapped: an ask landing after
+    /// the read rings the doorbell after it, and the pass that ring owes
+    /// takes it.
     pub fn take_asks(&self) -> [u64; MASK_WORDS] {
-        core::array::from_fn(|word| self.asks[word].swap(0, Ordering::Relaxed))
+        core::array::from_fn(|word| match self.asks[word].load(Ordering::Relaxed) {
+            0 => 0,
+            _ => self.asks[word].swap(0, Ordering::Relaxed),
+        })
     }
 
     pub fn publish_weight(&self, weight: u64) {
@@ -2937,6 +2954,9 @@ mod tests {
         timers: Vec<Option<Nanos>>,
         /// The revisit an idle plan asks, where a test sets one.
         revisit: Option<u64>,
+        /// The last arm's one-shot, not yet fired: a test fires it by
+        /// clearing this.
+        pending: bool,
     }
 
     #[derive(Default)]
@@ -2962,10 +2982,17 @@ mod tests {
             Nanos::ZERO
         }
         fn set_timer(&self, deadline: Nanos) {
-            self.state().timers.push(Some(deadline));
+            let mut state = self.state();
+            state.timers.push(Some(deadline));
+            state.pending = true;
         }
         fn stop_timer(&self) {
-            self.state().timers.push(None);
+            let mut state = self.state();
+            state.timers.push(None);
+            state.pending = false;
+        }
+        fn timer_pending(&self) -> bool {
+            self.state().pending
         }
         fn idle_plan(&self, deadline: Option<Nanos>, now: Nanos) -> crate::sched::idle::Plan {
             let mut plan = crate::sched::idle::Plan::c1(deadline.map(|d| d.0), now.0);
@@ -4922,6 +4949,83 @@ mod tests {
         assert_eq!(w.hw.state().timers, [None], "a plan with no wake leaves the plan main makes: stopped");
     }
 
+    /// **A running CPU writes its timer once a request, not once a pass.** A
+    /// pass that finds its one-shot pending and due no later than it needs
+    /// leaves it, so a thread that keeps entering the kernel re-arms nothing;
+    /// once it fired, or once something is owed sooner, the pass writes.
+    #[test]
+    fn a_running_cpu_leaves_a_pending_one_shot_due_no_later() {
+        let mut w = World::new(1);
+        w.spawn(C0);
+        w.run_a_pass_at(C0, NOW);
+        let armed = w.cpus[0].armed().expect("a running thread's request is armed");
+        assert_eq!(w.hw.state().timers, [Some(armed)]);
+
+        w.run_a_pass_at(C0, NOW.after(1_000));
+        assert_eq!(w.hw.state().timers, [Some(armed)], "the request's end has not moved: nothing written");
+        assert_eq!(w.cpus[0].armed(), Some(armed));
+
+        w.hw.state().pending = false;
+        w.run_a_pass_at(C0, NOW.after(2_000));
+        assert_eq!(w.hw.state().timers.len(), 2, "a fired one-shot is armed again");
+
+        // A sleeper due before the request's end is owed sooner.
+        w.spawn(C0);
+        let (cpus, env) = w.split();
+        let pass = SchedPass::begin(&mut cpus[0], env, NOW.after(3_000));
+        let _ = pass.dispose_yield().finish();
+        let due = NOW.after(50_000);
+        let (cpus, env) = w.split();
+        let sched = &mut cpus[0];
+        let current = CurrentTask::new(sched.running().expect("one of them runs").shared(), C0);
+        let ticket = crate::sched::park::prepare(&current, crate::sched::park::Cancel::Answers, WaitClass::Other)
+            .expect("nothing notified this task");
+        let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") };
+        let pass = SchedPass::begin(sched, env, NOW.after(4_000));
+        let _ = pass.dispose_block(committed, Some(due)).finish();
+        assert!(w.cpus[0].running().is_some(), "the other thread runs");
+        assert_eq!(w.cpus[0].armed(), Some(due), "a deadline before the armed one-shot is written");
+        assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due));
+        w.abandon();
+    }
+
+    /// A pass takes an ask once: the word it found set is left empty, and the
+    /// next take finds nothing.
+    #[test]
+    fn a_taken_ask_is_gone() {
+        let w = World::new(2);
+        let victim = w.handles.get(C0);
+        let _ = victim.ask(C1);
+        victim.doorbell().begin_pass();
+        assert_eq!(victim.take_asks()[0], 1 << C1.0, "the ask is taken");
+        assert!(!victim.asked_by(C1));
+        assert_eq!(victim.take_asks(), [0; MASK_WORDS], "and taken once");
+        w.abandon();
+    }
+
+    /// **An idle CPU arms exactly**: a pending one-shot due before the
+    /// deadline it parks with is not left to wake it early.
+    #[test]
+    fn a_cpu_going_idle_writes_its_deadline_over_a_pending_earlier_one_shot() {
+        let mut w = World::new(1);
+        w.spawn(C0);
+        w.run_a_pass_at(C0, NOW);
+        let armed = w.cpus[0].armed().expect("armed for the running thread");
+        let due = armed.after(1_000_000);
+        assert!(w.hw.state().pending);
+        let (cpus, env) = w.split();
+        let sched = &mut cpus[0];
+        let current = CurrentTask::new(sched.running().expect("it runs").shared(), C0);
+        let ticket = crate::sched::park::prepare(&current, crate::sched::park::Cancel::Answers, WaitClass::Other)
+            .expect("nothing notified this task");
+        let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") };
+        let pass = SchedPass::begin(sched, env, NOW.after(1_000));
+        let _ = pass.dispose_block(committed, Some(due)).finish();
+        assert_eq!(w.cpus[0].armed(), Some(due));
+        assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due), "the idle CPU's deadline is written");
+        w.abandon();
+    }
+
     /// **Locality first, then a free physical core, then rotation**: a forward
     /// or push from cpu0 takes a core whose threads all sleep in its own LLC,
     /// shares a busy core there before crossing to the other LLC's free ones,

kernel/pure/sched/hw.rs

@@ -98,6 +98,14 @@ pub trait Machine: Kicker + 'static {
 
     fn stop_timer(&self);
 
+    /// Whether the one-shot the last [`Self::set_timer`] armed is still to
+    /// be taken: its interrupt not yet handled, so a pass is still owed by
+    /// it. A machine that cannot tell answers `false`, and the core then
+    /// writes every arm.
+    fn timer_pending(&self) -> bool {
+        false
+    }
+
     /// The idle state this CPU waits in once the pass that asks ends, chosen
     /// with the scheduler's timer due at `deadline` (`crate::sched::idle`).
     /// The pass folds [`idle::Plan::wake_at`] into its one timer plan, and the

kernel/pure/sched/task.rs

@@ -14,7 +14,7 @@ use crate::sched::hw::{CpuId, Nanos};
 use crate::sched::job::Job;
 use crate::sched::mailbox::MailboxNode;
 use crate::sched::msg::Msg;
-use crate::sched::affinity::{affinity_fence, CpuMask, Reach, SharedMask};
+use crate::sched::affinity::{affinity_fence, CpuMask, LoadedMask, Reach, SharedMask};
 use crate::sched::sync::{Arc, AtomicBool, AtomicU64, Ordering};
 use crate::sched::park::CommittedTicket;
 use crate::sched::queue::{next_request, SLICE_NS};
@@ -737,10 +737,9 @@ struct TaskInner<X: SchedPayload> {
     ctx: X::Ctx,
     rt: RtState,
     acct: TaskAccounting,
-    /// The owner CPU's copy of [`TaskShared::mask`], its reach and the
-    /// version it was loaded at. Changed only by [`Task::refresh_affinity`].
-    affinity: CpuMask,
-    reach: Reach,
+    /// The owner CPU's copy of [`TaskShared::mask`] and the version it was
+    /// loaded at. Changed only by [`Task::refresh_affinity`].
+    affinity: LoadedMask,
     affinity_version: u64,
     /// What it asks of a queue at a time, in ns: [`next_request`] of its
     /// bursts, each the CPU time from a wake to the next park.
@@ -797,11 +796,11 @@ impl<X: SchedPayload> Task<X> {
     }
 
     pub fn affinity(&self) -> CpuMask {
-        self.0.affinity
+        self.0.affinity.mask()
     }
 
     pub fn reach(&self) -> Reach {
-        self.0.reach
+        self.0.affinity.reach()
     }
 
     pub fn request_ns(&self) -> u64 {
@@ -814,8 +813,8 @@ impl<X: SchedPayload> Task<X> {
         if self.0.shared.mask().version() == self.0.affinity_version {
             return false;
         }
-        (self.0.affinity, self.0.affinity_version) = self.0.shared.mask().load();
-        self.0.reach = self.0.affinity.reach();
+        let (mask, version) = self.0.shared.mask().load();
+        (self.0.affinity, self.0.affinity_version) = (LoadedMask::new(mask), version);
         true
     }
 
@@ -980,8 +979,7 @@ impl<X: SchedPayload> TaskBuilder<X> {
             ctx: self.ctx,
             rt: self.rt,
             acct: TaskAccounting::default(),
-            affinity: self.mask,
-            reach: self.mask.reach(),
+            affinity: LoadedMask::new(self.mask),
             affinity_version: 0,
             request: SLICE_NS,
             woke_cpu_ns: 0,

kernel/pure/sched/timer.rs

@@ -69,7 +69,8 @@ impl TimerPlan {
     }
 }
 
-/// Proof that a [`TimerPlan`] reached the hardware. [`crate::sched::cpu::SleepToken`]
+/// Proof that the hardware holds a [`TimerPlan`], or a one-shot still to fire
+/// that is due no later. [`crate::sched::cpu::SleepToken`]
 /// cannot be built without one, so "halted with a deadline pending and the
 /// timer unarmed" is unrepresentable rather than asserted.
 pub struct TimerApplied {

kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace

@@ -1,5 +1,5 @@
 # scenario: old_migrate_kept_the_corpse
-# expect: fail
+# expect: fail I14
 # note: the T14's panic, minimized — retire_task: task not released after 1s:
 # note: InTransit(CpuId(1)). soundserver boost-wakes a client thread whose process
 # note: has just been torn down, on the CPU soundserver itself is running, with a
@@ -10,5 +10,5 @@
 # note: job-group scheduler's request-length decisions, and again when a steal
 # note: probe became a bit in its victim's word instead of a message.
 # note: it fails I14's "killed and then migrated" half.
-4 3 3 0 0 0 1 1 2 0 2 0 2 0 0 0 0 1 2 1 1 2 1 0 1 0 0 1 1 0 1 1
-0 0 0 0 0 0 0 1 1 1 0 2 0 0 1 1 0 1 0 1 1 2 
+0 0 1 1 1 0 2 0 0 1 0 0 2 0 1 2 1 3 0 0 2 1 2 1 0 0 1 0 0 0 0 1
+0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 1 2 1

kernel/sim/corpus/old_steal_port_i1.trace

@@ -1,3 +1,3 @@
 # scenario: old_steal_port
-# expect: fail
+# expect: fail I1
 0 1 1 2 

kernel/sim/corpus/old_steal_port_i8.trace

@@ -1,3 +1,4 @@
 # scenario: old_steal_port
-# expect: fail
-1 1 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 1 
+# expect: fail I8
+3 0 0 1 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1
+0 0 1 1 1 0 0 2 0 0 1 1 1 0 0 1 1 0 1 

kernel/sim/src/hw_impl.rs

@@ -179,6 +179,12 @@ impl Machine for SimHw {
         });
     }
 
+    /// A fire disarms the modelled one-shot (`Step::FireTimer`), so one still
+    /// armed is one whose pass is still owed.
+    fn timer_pending(&self) -> bool {
+        self.with(|s| s.armed[Self::cpu(s)].is_some())
+    }
+
     fn halt(&self) {
         self.with(|s| {
             let cpu = Self::cpu(s);