diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md similarity index 77% rename from issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md rename to issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md index 66ca8365fa1..c8a6147badb 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md @@ -4,10 +4,10 @@ kind: track opened: 2026-10-12 --- -# The timer is a LAPIC count re-armed every pass, never a TSC deadline +# The timer is a LAPIC count, never a TSC deadline -Every scheduler pass ends by handing its timer an absolute deadline -(`kernel/pure/sched/cpu.rs:1912-1918`), and the kernel turns it back into a +A scheduler pass ends by handing its timer an absolute deadline +(`SchedPass::apply_timer_no_later_than`), and the kernel turns it back into a span from now (`kernel/src/hw.rs:47-51`) that `apic::arm_one_shot` converts to LAPIC ticks by the boot's calibration against the TSC-derived clock (`kernel/src/arch/x86_64/apic.rs`) and writes into `TIMER_INIT`, one APIC @@ -28,6 +28,15 @@ CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, `CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration and its spin. +**A running CPU re-arms once a request, since #1263.** A pass on a CPU +that keeps running leaves a one-shot that is still to fire and due no later +than it needs, so the timer is written once a request, after a fire, or when +something is owed sooner; a CPU going idle still arms exactly at every sleep +and stops at every idle entry. What a deadline would save is now per arm and +not per pass, which leaves it the idle path's arm and stop and the boot's +calibration spin, on the T14 alone. The figures below are the old code's, +which wrote at every pass. + **Its speed reading decides it.** The SDM defines the mode and recommends it over the count nowhere, and AMD documents none, so nothing published covers the choice. Intel's specification update for the T14's processor lists no @@ -72,6 +81,8 @@ lower and the two spreads do not overlap; where they overlap, the count, the path every CPU has, is kept. The split is reported. The twin is a measurement arm, deleted once it has answered. +**Owed on metal:** one A/B of main from before and after #1263 (a pass skips the timer write and the ask swaps it does not owe), on the T14 and the AMD bench, of osbench's `system`, `sched`, `ipc` and `time` groups, with a Linux reference timed through the TSC and its `.slack1ns` rows. Main must read `time.yield`, `pingpong.*` and `ipc.reqrep.*` faster by more than 3% on both benches, `time.sleep_late.*`, `time.timer_1ms.late` and `wake.periodic.*` within 3%, and no scheduler, IPC, wake, yield, spawn or timer row slower. The speed-up is not yet measured on metal. + **Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it cited for bit 24; the twin's reading, posted, and the twin deleted. Where the deadline's lateness measures better, a deadline is armed on every CPU that diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 475aaa6f052..1517fc20d41 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -46,7 +46,7 @@ and contention). - A TSC deadline: before #1061 cut the arm to one register write, the re-arm was at most 11.7 to 12.1 % of a pass's pick and switch on the T14, and at the AMD bench's TSC floor; it is to be read again - (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). + (`issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md`). - Deep idle: on the T14 `hlt` holds a one-thread load to the turbo ratio for three or more active cores, 38 of `MSR_TURBO_RATIO_LIMIT`'s `0x2626262626262a2a` against 42 for one or two (#866's dump, @@ -88,7 +88,7 @@ yet measured. - `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md` - `issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md` - `issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md` -- `issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md` +- `issues/the-timer-is-a-lapic-count-never-a-tsc-deadline.md` - `issues/no-gate-decodes-kernel-elfs-instructions.md` - `issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md` - `issues/the-direct-map-uses-1-gib-leaves-where-the-cpu-has-them.md` diff --git a/kernel/pure/sched/affinity.rs b/kernel/pure/sched/affinity.rs index a3244504560..703c6927d65 100644 --- a/kernel/pure/sched/affinity.rs +++ b/kernel/pure/sched/affinity.rs @@ -106,15 +106,34 @@ impl CpuMask { pub fn words(&self) -> &[u64; MASK_WORDS] { &self.0 } +} - pub fn reach(&self) -> Reach { - if *self == Self::ANY { +/// A task's mask as its owner CPU loaded it, with the [`Reach`] a run queue +/// counts it by: one value, so the two cannot disagree. +#[derive(Clone, Copy, Debug)] +pub struct LoadedMask { + mask: CpuMask, + reach: Reach, +} + +impl LoadedMask { + pub fn new(mask: CpuMask) -> Self { + let reach = if mask == CpuMask::ANY { Reach::Any - } else if self.single().is_some() { + } else if mask.single().is_some() { Reach::One } else { Reach::Several - } + }; + Self { mask, reach } + } + + pub fn mask(&self) -> CpuMask { + self.mask + } + + pub fn reach(&self) -> Reach { + self.reach } } diff --git a/kernel/pure/sched/cpu.rs b/kernel/pure/sched/cpu.rs index 1c44f3af940..81d6decd486 100644 --- a/kernel/pure/sched/cpu.rs +++ b/kernel/pure/sched/cpu.rs @@ -2186,18 +2186,30 @@ impl SchedPass<'_, '_, H, P, Disposed> { Some(_) => Some(self.cpu.settle_decision(self.now)), None => None, }; - let plan = TimerPlan::compute(decision, deadline).no_later_than(extra); - match plan { - TimerPlan::Arm(at) => self.env.hw.set_timer(at), - TimerPlan::Stop => self.env.hw.stop_timer(), + match TimerPlan::compute(decision, deadline).no_later_than(extra) { + // A one-shot still to fire and due no later is left: an early fire + // is a spurious pass, so a running CPU writes its timer once a + // request rather than once a pass. An idle CPU arms exactly, so + // nothing wakes it before it is owed. + TimerPlan::Arm(at) + if self.cpu.running.is_some() + && self.cpu.armed.is_some_and(|armed| self.now < armed && armed <= at) + && self.env.hw.timer_pending() => {} + TimerPlan::Arm(at) => { + self.env.hw.set_timer(at); + self.cpu.armed = Some(at); + } + TimerPlan::Stop => { + self.env.hw.stop_timer(); + self.cpu.armed = None; + } } - self.cpu.armed = plan.armed(); // Invariant T is a statement about a CPU *outside* a pass, and the // arming that makes it true is the last thing a pass does — so the // check cannot move earlier. #[cfg(feature = "sched-check")] crate::sched::invariants::check_cpu(self.cpu); - TimerApplied::new(plan.armed()) + TimerApplied::new(self.cpu.armed) } fn switch_to_current(&mut self) -> Action<::Payload> { @@ -2551,9 +2563,14 @@ impl CpuHandle { } /// Every ask since the last take, emptied: this CPU's own pass, after its - /// `begin_pass`. + /// `begin_pass`. A word read empty is not swapped: an ask landing after + /// the read rings the doorbell after it, and the pass that ring owes + /// takes it. pub fn take_asks(&self) -> [u64; MASK_WORDS] { - core::array::from_fn(|word| self.asks[word].swap(0, Ordering::Relaxed)) + core::array::from_fn(|word| match self.asks[word].load(Ordering::Relaxed) { + 0 => 0, + _ => self.asks[word].swap(0, Ordering::Relaxed), + }) } pub fn publish_weight(&self, weight: u64) { @@ -2937,6 +2954,9 @@ mod tests { timers: Vec>, /// The revisit an idle plan asks, where a test sets one. revisit: Option, + /// The last arm's one-shot, not yet fired: a test fires it by + /// clearing this. + pending: bool, } #[derive(Default)] @@ -2962,10 +2982,17 @@ mod tests { Nanos::ZERO } fn set_timer(&self, deadline: Nanos) { - self.state().timers.push(Some(deadline)); + let mut state = self.state(); + state.timers.push(Some(deadline)); + state.pending = true; } fn stop_timer(&self) { - self.state().timers.push(None); + let mut state = self.state(); + state.timers.push(None); + state.pending = false; + } + fn timer_pending(&self) -> bool { + self.state().pending } fn idle_plan(&self, deadline: Option, now: Nanos) -> crate::sched::idle::Plan { let mut plan = crate::sched::idle::Plan::c1(deadline.map(|d| d.0), now.0); @@ -4922,6 +4949,83 @@ mod tests { assert_eq!(w.hw.state().timers, [None], "a plan with no wake leaves the plan main makes: stopped"); } + /// **A running CPU writes its timer once a request, not once a pass.** A + /// pass that finds its one-shot pending and due no later than it needs + /// leaves it, so a thread that keeps entering the kernel re-arms nothing; + /// once it fired, or once something is owed sooner, the pass writes. + #[test] + fn a_running_cpu_leaves_a_pending_one_shot_due_no_later() { + let mut w = World::new(1); + w.spawn(C0); + w.run_a_pass_at(C0, NOW); + let armed = w.cpus[0].armed().expect("a running thread's request is armed"); + assert_eq!(w.hw.state().timers, [Some(armed)]); + + w.run_a_pass_at(C0, NOW.after(1_000)); + assert_eq!(w.hw.state().timers, [Some(armed)], "the request's end has not moved: nothing written"); + assert_eq!(w.cpus[0].armed(), Some(armed)); + + w.hw.state().pending = false; + w.run_a_pass_at(C0, NOW.after(2_000)); + assert_eq!(w.hw.state().timers.len(), 2, "a fired one-shot is armed again"); + + // A sleeper due before the request's end is owed sooner. + w.spawn(C0); + let (cpus, env) = w.split(); + let pass = SchedPass::begin(&mut cpus[0], env, NOW.after(3_000)); + let _ = pass.dispose_yield().finish(); + let due = NOW.after(50_000); + let (cpus, env) = w.split(); + let sched = &mut cpus[0]; + let current = CurrentTask::new(sched.running().expect("one of them runs").shared(), C0); + let ticket = crate::sched::park::prepare(¤t, crate::sched::park::Cancel::Answers, WaitClass::Other) + .expect("nothing notified this task"); + let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") }; + let pass = SchedPass::begin(sched, env, NOW.after(4_000)); + let _ = pass.dispose_block(committed, Some(due)).finish(); + assert!(w.cpus[0].running().is_some(), "the other thread runs"); + assert_eq!(w.cpus[0].armed(), Some(due), "a deadline before the armed one-shot is written"); + assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due)); + w.abandon(); + } + + /// A pass takes an ask once: the word it found set is left empty, and the + /// next take finds nothing. + #[test] + fn a_taken_ask_is_gone() { + let w = World::new(2); + let victim = w.handles.get(C0); + let _ = victim.ask(C1); + victim.doorbell().begin_pass(); + assert_eq!(victim.take_asks()[0], 1 << C1.0, "the ask is taken"); + assert!(!victim.asked_by(C1)); + assert_eq!(victim.take_asks(), [0; MASK_WORDS], "and taken once"); + w.abandon(); + } + + /// **An idle CPU arms exactly**: a pending one-shot due before the + /// deadline it parks with is not left to wake it early. + #[test] + fn a_cpu_going_idle_writes_its_deadline_over_a_pending_earlier_one_shot() { + let mut w = World::new(1); + w.spawn(C0); + w.run_a_pass_at(C0, NOW); + let armed = w.cpus[0].armed().expect("armed for the running thread"); + let due = armed.after(1_000_000); + assert!(w.hw.state().pending); + let (cpus, env) = w.split(); + let sched = &mut cpus[0]; + let current = CurrentTask::new(sched.running().expect("it runs").shared(), C0); + let ticket = crate::sched::park::prepare(¤t, crate::sched::park::Cancel::Answers, WaitClass::Other) + .expect("nothing notified this task"); + let crate::sched::park::Commit::Parked(committed) = ticket.commit() else { panic!("an uncontended park") }; + let pass = SchedPass::begin(sched, env, NOW.after(1_000)); + let _ = pass.dispose_block(committed, Some(due)).finish(); + assert_eq!(w.cpus[0].armed(), Some(due)); + assert_eq!(*w.hw.state().timers.last().unwrap(), Some(due), "the idle CPU's deadline is written"); + w.abandon(); + } + /// **Locality first, then a free physical core, then rotation**: a forward /// or push from cpu0 takes a core whose threads all sleep in its own LLC, /// shares a busy core there before crossing to the other LLC's free ones, diff --git a/kernel/pure/sched/hw.rs b/kernel/pure/sched/hw.rs index 50a1b0bf280..22f4d42f830 100644 --- a/kernel/pure/sched/hw.rs +++ b/kernel/pure/sched/hw.rs @@ -98,6 +98,14 @@ pub trait Machine: Kicker + 'static { fn stop_timer(&self); + /// Whether the one-shot the last [`Self::set_timer`] armed is still to + /// be taken: its interrupt not yet handled, so a pass is still owed by + /// it. A machine that cannot tell answers `false`, and the core then + /// writes every arm. + fn timer_pending(&self) -> bool { + false + } + /// The idle state this CPU waits in once the pass that asks ends, chosen /// with the scheduler's timer due at `deadline` (`crate::sched::idle`). /// The pass folds [`idle::Plan::wake_at`] into its one timer plan, and the diff --git a/kernel/pure/sched/task.rs b/kernel/pure/sched/task.rs index 861a9e98433..6632408ed14 100644 --- a/kernel/pure/sched/task.rs +++ b/kernel/pure/sched/task.rs @@ -14,7 +14,7 @@ use crate::sched::hw::{CpuId, Nanos}; use crate::sched::job::Job; use crate::sched::mailbox::MailboxNode; use crate::sched::msg::Msg; -use crate::sched::affinity::{affinity_fence, CpuMask, Reach, SharedMask}; +use crate::sched::affinity::{affinity_fence, CpuMask, LoadedMask, Reach, SharedMask}; use crate::sched::sync::{Arc, AtomicBool, AtomicU64, Ordering}; use crate::sched::park::CommittedTicket; use crate::sched::queue::{next_request, SLICE_NS}; @@ -737,10 +737,9 @@ struct TaskInner { ctx: X::Ctx, rt: RtState, acct: TaskAccounting, - /// The owner CPU's copy of [`TaskShared::mask`], its reach and the - /// version it was loaded at. Changed only by [`Task::refresh_affinity`]. - affinity: CpuMask, - reach: Reach, + /// The owner CPU's copy of [`TaskShared::mask`] and the version it was + /// loaded at. Changed only by [`Task::refresh_affinity`]. + affinity: LoadedMask, affinity_version: u64, /// What it asks of a queue at a time, in ns: [`next_request`] of its /// bursts, each the CPU time from a wake to the next park. @@ -797,11 +796,11 @@ impl Task { } pub fn affinity(&self) -> CpuMask { - self.0.affinity + self.0.affinity.mask() } pub fn reach(&self) -> Reach { - self.0.reach + self.0.affinity.reach() } pub fn request_ns(&self) -> u64 { @@ -814,8 +813,8 @@ impl Task { if self.0.shared.mask().version() == self.0.affinity_version { return false; } - (self.0.affinity, self.0.affinity_version) = self.0.shared.mask().load(); - self.0.reach = self.0.affinity.reach(); + let (mask, version) = self.0.shared.mask().load(); + (self.0.affinity, self.0.affinity_version) = (LoadedMask::new(mask), version); true } @@ -980,8 +979,7 @@ impl TaskBuilder { ctx: self.ctx, rt: self.rt, acct: TaskAccounting::default(), - affinity: self.mask, - reach: self.mask.reach(), + affinity: LoadedMask::new(self.mask), affinity_version: 0, request: SLICE_NS, woke_cpu_ns: 0, diff --git a/kernel/pure/sched/timer.rs b/kernel/pure/sched/timer.rs index 6d65e6e48e7..c861aa1bfd9 100644 --- a/kernel/pure/sched/timer.rs +++ b/kernel/pure/sched/timer.rs @@ -69,7 +69,8 @@ impl TimerPlan { } } -/// Proof that a [`TimerPlan`] reached the hardware. [`crate::sched::cpu::SleepToken`] +/// Proof that the hardware holds a [`TimerPlan`], or a one-shot still to fire +/// that is due no later. [`crate::sched::cpu::SleepToken`] /// cannot be built without one, so "halted with a deadline pending and the /// timer unarmed" is unrepresentable rather than asserted. pub struct TimerApplied { diff --git a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace index 3b7038762d7..cf883b8c7c9 100644 --- a/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace +++ b/kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace @@ -1,5 +1,5 @@ # scenario: old_migrate_kept_the_corpse -# expect: fail +# expect: fail I14 # note: the T14's panic, minimized — retire_task: task not released after 1s: # note: InTransit(CpuId(1)). soundserver boost-wakes a client thread whose process # note: has just been torn down, on the CPU soundserver itself is running, with a @@ -10,5 +10,5 @@ # note: job-group scheduler's request-length decisions, and again when a steal # note: probe became a bit in its victim's word instead of a message. # note: it fails I14's "killed and then migrated" half. -4 3 3 0 0 0 1 1 2 0 2 0 2 0 0 0 0 1 2 1 1 2 1 0 1 0 0 1 1 0 1 1 -0 0 0 0 0 0 0 1 1 1 0 2 0 0 1 1 0 1 0 1 1 2 +0 0 1 1 1 0 2 0 0 1 0 0 2 0 1 2 1 3 0 0 2 1 2 1 0 0 1 0 0 0 0 1 +0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 0 1 1 2 1 diff --git a/kernel/sim/corpus/old_steal_port_i1.trace b/kernel/sim/corpus/old_steal_port_i1.trace index f2b7e8da255..0c05a85734f 100644 --- a/kernel/sim/corpus/old_steal_port_i1.trace +++ b/kernel/sim/corpus/old_steal_port_i1.trace @@ -1,3 +1,3 @@ # scenario: old_steal_port -# expect: fail +# expect: fail I1 0 1 1 2 diff --git a/kernel/sim/corpus/old_steal_port_i8.trace b/kernel/sim/corpus/old_steal_port_i8.trace index 575c24e7f06..0e63c8b110a 100644 --- a/kernel/sim/corpus/old_steal_port_i8.trace +++ b/kernel/sim/corpus/old_steal_port_i8.trace @@ -1,3 +1,4 @@ # scenario: old_steal_port -# expect: fail -1 1 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 1 +# expect: fail I8 +3 0 0 1 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 1 0 0 0 0 0 0 1 0 0 1 +0 0 1 1 1 0 0 2 0 0 1 1 1 0 0 1 1 0 1 diff --git a/kernel/sim/src/hw_impl.rs b/kernel/sim/src/hw_impl.rs index 22b851904cd..6c5defe54a8 100644 --- a/kernel/sim/src/hw_impl.rs +++ b/kernel/sim/src/hw_impl.rs @@ -179,6 +179,12 @@ impl Machine for SimHw { }); } + /// A fire disarms the modelled one-shot (`Step::FireTimer`), so one still + /// armed is one whose pass is still owed. + fn timer_pending(&self) -> bool { + self.with(|s| s.armed[Self::cpu(s)].is_some()) + } + fn halt(&self) { self.with(|s| { let cpu = Self::cpu(s); diff --git a/kernel/sim/src/shrink.rs b/kernel/sim/src/shrink.rs index 64c6def5327..569e662e2e3 100644 --- a/kernel/sim/src/shrink.rs +++ b/kernel/sim/src/shrink.rs @@ -100,18 +100,24 @@ pub fn encode(scenario: &str, expect_failure: bool, decisions: &[usize]) -> Stri pub struct CorpusEntry { pub scenario: String, pub expect_failure: bool, + /// The invariant a failing trace was committed for, as `# expect: fail I14` + /// names it: a replay that fails only some other way has lost its case. + pub expect_violation: Option, pub decisions: Vec, } pub fn decode(text: &str) -> CorpusEntry { let mut scenario = String::new(); let mut expect_failure = false; + let mut expect_violation = None; let mut decisions = Vec::new(); for line in text.lines() { if let Some(rest) = line.strip_prefix("# scenario:") { scenario = rest.trim().to_string(); } else if let Some(rest) = line.strip_prefix("# expect:") { - expect_failure = rest.trim() == "fail"; + let mut words = rest.split_whitespace(); + expect_failure = words.next() == Some("fail"); + expect_violation = words.next().map(str::to_string); } else if !line.starts_with('#') { decisions.extend(line.split_whitespace().map(|token| { token @@ -124,6 +130,7 @@ pub fn decode(text: &str) -> CorpusEntry { CorpusEntry { scenario, expect_failure, + expect_violation, decisions, } } diff --git a/kernel/sim/tests/scenarios.rs b/kernel/sim/tests/scenarios.rs index bc33ec5322b..5e239b233fb 100644 --- a/kernel/sim/tests/scenarios.rs +++ b/kernel/sim/tests/scenarios.rs @@ -592,6 +592,14 @@ fn corpus_traces_still_do_what_they_were_committed_for() { }, outcome.report(), ); + if let Some(invariant) = &entry.expect_violation { + assert!( + outcome.violations.iter().any(|v| v.starts_with(&format!("{invariant}:"))), + "{}: expected {invariant}, got {}", + path.display(), + outcome.report(), + ); + } checked += 1; } assert!( diff --git a/kernel/src/arch/aarch64/irqchip.rs b/kernel/src/arch/aarch64/irqchip.rs index 8d7d428d026..7bc98d0121b 100644 --- a/kernel/src/arch/aarch64/irqchip.rs +++ b/kernel/src/arch/aarch64/irqchip.rs @@ -495,6 +495,11 @@ fn stop_timer_hardware() { /// This CPU's timer, armed to fire `nanos` from now, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { + // Before the write, and held there against the `nomem` comparator write: a + // fire taken between the two then clears it, and the next pass writes again + // rather than trusting a quantum's re-arm. + percpu::set_timer_pending(true); + core::sync::atomic::compiler_fence(core::sync::atomic::Ordering::SeqCst); arm_ticks(crate::clock::counter_ticks(nanos)); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } @@ -534,8 +539,16 @@ pub fn rearmed_a_quantum(armed: u64) -> bool { comparator() >= armed + crate::clock::counter_ticks(kernel::sched::task::QUANTUM_NS) } +/// Whether the one-shot [`arm_one_shot`] last armed is still to fire: both +/// of the timer's entry paths clear it, and [`arm_within`] leaves it, since +/// what it arms is never later than what it found. +pub fn timer_pending() -> bool { + percpu::timer_pending() +} + /// Stop the timer: no interrupt until it is armed again. pub fn stop_timer() { + percpu::set_timer_pending(false); percpu::set_armed_ticks(0); stop_timer_hardware(); crate::trace::trace(crate::trace::Kind::TimerStop, 0); @@ -544,6 +557,7 @@ pub fn stop_timer() { /// A timer interrupt taken from EL0: armed again for what it was last armed /// for, or stopped if it was stopped — the one thing that deasserts it. pub(super) fn rearm() { + percpu::set_timer_pending(false); match percpu::armed_ticks() { 0 => stop_timer_hardware(), ticks => { @@ -555,6 +569,7 @@ pub(super) fn rearm() { /// A timer interrupt taken at EL1: one quantum on, or stopped if it was /// stopped, leaving what the scheduler armed for its next pass to re-arm. pub(super) fn rearm_in_kernel() { + percpu::set_timer_pending(false); match percpu::armed_ticks() { 0 => stop_timer_hardware(), _ => compare_at(cpu::counter() + crate::clock::counter_ticks(kernel::sched::task::QUANTUM_NS)), diff --git a/kernel/src/arch/aarch64/percpu.rs b/kernel/src/arch/aarch64/percpu.rs index 899782ea6d1..65eebf2ed89 100644 --- a/kernel/src/arch/aarch64/percpu.rs +++ b/kernel/src/arch/aarch64/percpu.rs @@ -8,7 +8,7 @@ //! an exception on this CPU cannot split (Arm ARM K.a, B2.17.5: taking an //! exception clears the monitor). -use core::sync::atomic::{AtomicU32, AtomicU64, AtomicU8, Ordering::Relaxed}; +use core::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicU8, Ordering::Relaxed}; use alloc::boxed::Box; @@ -41,6 +41,9 @@ pub struct PerCpu { /// Counter ticks the timer was last armed for: what a timer interrupt /// taken at EL0 re-arms it with, and zero when it is stopped. armed_ticks: AtomicU64, + /// Whether the scheduler's last one-shot is armed and its interrupt not + /// yet taken. + timer_pending: AtomicBool, /// The kernel stack a switch last installed, for the stack witness: on /// AArch64 an entry from EL0 lands on `SP_EL1` as the last `ERET` left it, /// and nothing here points it anywhere. @@ -86,6 +89,7 @@ pub fn alloc(cpu_id: u32) -> &'static PerCpu { kernel_timer_fires: AtomicU32::new(0), last_seen_kernel_timer_fires: AtomicU32::new(0), armed_ticks: AtomicU64::new(0), + timer_pending: AtomicBool::new(false), kernel_stack: AtomicU64::new(0), // The same size as a task's kernel stack: a `deferred` [`kobject!`] // object may own an `immediate` one, whose destructor then runs here. @@ -338,3 +342,11 @@ pub(super) fn armed_ticks() -> u64 { pub(super) fn set_armed_ticks(ticks: u64) { this().armed_ticks.store(ticks, Relaxed); } + +pub(super) fn timer_pending() -> bool { + this().timer_pending.load(Relaxed) +} + +pub(super) fn set_timer_pending(pending: bool) { + this().timer_pending.store(pending, Relaxed); +} diff --git a/kernel/src/arch/x86_64/apic.rs b/kernel/src/arch/x86_64/apic.rs index 44a40be68d9..2da39b07ae9 100644 --- a/kernel/src/arch/x86_64/apic.rs +++ b/kernel/src/arch/x86_64/apic.rs @@ -379,6 +379,9 @@ impl OneShot { /// Arm a one-shot timer to fire after `nanos` nanoseconds, or after [`MIN_ONE_SHOT`] if that is longer. pub fn arm_one_shot(nanos: u64) { + // Before the write: a fire taken between the two then clears it, and the + // next pass writes again rather than trusting a quantum's re-arm. + percpu::set_timer_pending(true); OneShot::after(nanos).arm(); crate::trace::trace(crate::trace::Kind::TimerArm, nanos as u32); } @@ -408,19 +411,29 @@ pub fn rearmed_a_quantum(_armed: u64) -> bool { /// The Ring 3 fire's re-arm, before its Rust half runs: what this CPU last /// armed, again, so the timer survives a handler that panics first. pub(crate) extern "sysv64" fn rearm_last() { + percpu::set_timer_pending(false); Reg::TIMER_INIT.write(percpu::last_armed_ticks()); } /// The Ring 0 fire's local-APIC work, called from its stub with interrupts /// closed: the EOI, then one quantum on, or a stopped timer left stopped. pub(crate) extern "sysv64" fn ring0_fire() { + percpu::set_timer_pending(false); eoi(); let stopped = percpu::last_armed_ticks() == 0; Reg::TIMER_INIT.write(if stopped { 0 } else { TIMER_TICKS.load(Ordering::Relaxed) }); } +/// Whether the one-shot [`arm_one_shot`] last armed is still to fire: both of +/// the timer's entry paths clear it, and [`arm_within`] leaves it, since what +/// it arms is never later than what it found. +pub fn timer_pending() -> bool { + percpu::timer_pending() +} + /// Stop the timer. No more interrupts until re-armed. pub fn stop_timer() { + percpu::set_timer_pending(false); percpu::set_last_armed_ticks(0); Reg::TIMER_INIT.write(0); crate::trace::trace(crate::trace::Kind::TimerStop, 0); diff --git a/kernel/src/arch/x86_64/percpu.rs b/kernel/src/arch/x86_64/percpu.rs index ff1b12e8900..b8d8759df45 100644 --- a/kernel/src/arch/x86_64/percpu.rs +++ b/kernel/src/arch/x86_64/percpu.rs @@ -109,7 +109,9 @@ pub struct PerCpu { pub ring0_timer_fires: AtomicU32, pub last_seen_ring0_fires: u32, fault_state: u8, - _pad_after_fault_state: [u8; 3], + /// Non-zero while the scheduler's last one-shot is armed and its interrupt not yet taken. + timer_pending: u8, + _pad_after_timer_pending: [u8; 2], /// Ticks the Ring 3 timer branch re-arms with, and zero when stopped, which the Ring 0 branch leaves stopped. pub last_armed_ticks: AtomicU32, /// This CPU's [`log::Shard`]; never null on a live CPU ([`alloc_percpu`] fills it first). @@ -209,6 +211,7 @@ pub(crate) const OFF_PREEMPT_COUNT: u32 = offset_of!(PerCpu, preempt_count) as u pub(crate) const OFF_NEED_RESCHED: u32 = offset_of!(PerCpu, need_resched) as u32; /// Read by `preempt::enable`'s slow path to decline rescheduling a faulting CPU. pub(crate) const OFF_FAULT_STATE: u32 = offset_of!(PerCpu, fault_state) as u32; +const OFF_TIMER_PENDING: u32 = offset_of!(PerCpu, timer_pending) as u32; /// Set/cleared only by `arch::idt::nmi`'s naked entry, via this constant. pub(crate) const OFF_NMI_ACTIVE: u32 = offset_of!(PerCpu, nmi_active) as u32; /// The AP's bring-up token, read by `ap_entry` to answer for its own attempt. @@ -371,7 +374,8 @@ fn alloc_percpu(cpu_id: u32) -> *mut PerCpu { ring0_timer_fires: AtomicU32::new(0), last_seen_ring0_fires: 0, fault_state: CpuFaultState::Normal as u8, - _pad_after_fault_state: [0; 3], + timer_pending: 0, + _pad_after_timer_pending: [0; 2], last_armed_ticks: AtomicU32::new(0), log_shard: log::shard_for(cpu_id) as *const log::Shard as u64, nmi_active: 0, @@ -684,6 +688,15 @@ pub fn last_armed_ticks() -> u32 { gs::read_u32::() } +/// Whether the scheduler's one-shot is armed and its interrupt not yet taken; `arch::apic` is the only writer. +pub(super) fn set_timer_pending(pending: bool) { + gs::write_u8::(u8::from(pending)); +} + +pub fn timer_pending() -> bool { + gs::read_u8::() != 0 +} + /// The last byte of this CPU's idle guard page — the first byte an overflow reaches. #[cfg(feature = "test-actuators")] pub fn idle_guard_byte() -> u64 { diff --git a/kernel/src/hw.rs b/kernel/src/hw.rs index 80d0cc040db..03faf65fb13 100644 --- a/kernel/src/hw.rs +++ b/kernel/src/hw.rs @@ -62,6 +62,10 @@ impl Machine for KernelHw { irqchip::stop_timer(); } + fn timer_pending(&self) -> bool { + irqchip::timer_pending() + } + fn idle_plan(&self, deadline: Option, now: Nanos) -> kernel::sched::idle::Plan { crate::idle::plan(deadline, now) }