Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions build/ci.rs
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,21 @@ pub(crate) const CONTROLS: &[Control] = &[
Fails("an_acknowledged_flush_postdates_the_page_table_write"),
Fails("one_serve_answers_two_concurrent_shootdowns"),
]),
red(KERNEL_LOOM, "shootdown-take-relaxed", Some("tlb_shootdown"), &[
Fails("an_answered_ask_flushed_its_range_after_the_write"),
]),
red(KERNEL_LOOM, "shootdown-answer-relaxed", Some("tlb_shootdown"), &[
Fails("an_answered_ask_flushed_its_range_after_the_write"),
]),
red(KERNEL_LOOM, "shootdown-ask-range-late", Some("tlb_shootdown"), &[
Fails("an_answered_ask_flushed_its_range_after_the_write"),
]),
red(KERNEL_LOOM, "shootdown-ask-stale-answers", Some("tlb_shootdown"), &[
Fails("a_mailbox_reused_waits_for_its_new_answer"),
]),
red(KERNEL_LOOM, "shootdown-ask-many-as-one", Some("tlb_shootdown"), &[
Fails("two_asks_of_one_cpu_are_each_answered_for_their_own_page"),
]),
red(KERNEL_LOOM, "tlb-tags-enter-relaxed", Some("tlb_tags"), &[
Fails("a_change_reaches_a_runner_or_the_entering_cpu_flushes"),
]),
Expand Down
39 changes: 32 additions & 7 deletions issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,24 @@ that space (`kernel/src/arch/x86_64/tlb.rs`, `user_change`, over
`kernel/src/tlb_tags.rs`). A CPU that switched away keeps the space's entries
under PCID, since every CR3 load sets NOFLUSH, so it is not reached; it
flushes every user translation before it next enters a space that changed
since its own last such flush. The initiator flushes its own entries by page
for up to 64 pages, and a change no other CPU runs sends no IPI. AArch64
broadcasts its invalidations in hardware and needs none of this. Every other
shootdown, a kernel-half change and a PCID reclaim among them, still reaches
every CPU, and a reached CPU still answers with a flush of every user
translation, not of the range.
since its own last such flush. A change no other CPU runs sends no IPI.
Each CPU reached is asked the change's range, from the initiator's own
mailbox, and answers that initiator alone: it flushes the range by page, up
to 64 pages, or every user translation when the range is longer or it was
asked by more than one initiator at once. So a CPU no change asks owes no
flush, and the initiator owes itself none: it sends its IPIs first and then
flushes its own entries by page while they fly. Only a flush of every user
translation counts as the CPU's last such flush. AArch64 broadcasts its
invalidations in hardware and needs none of this. Every other shootdown, a
kernel-half change and a PCID reclaim among them, still reaches every CPU,
which answers with a flush of every user translation.

What holds it: `kernel/loom/tests/tlb_tags.rs` checks the pairing a CPU
entering a space and a change reading who runs it rest on, and reds with the
pairing's fence removed; `tlb_shootdown_waits` shows on the test kernel that
pairing's fence removed; `kernel/loom/tests/tlb_shootdown.rs` checks that an
answered ask flushed its range after the write, alone or with a second
initiator's, and that a mailbox reused waits for its new answer, each red
under its mutation; `tlb_shootdown_waits` shows on the test kernel that
a change waits for a CPU running a thread of the space and for none when no
other CPU runs one, each half red under its mutation.

Expand All @@ -40,5 +48,22 @@ four boots against five Linux runs) the T14 read `mem.munmap_part.4k` at
(3,341 against 2,279 ns), each resolved, and `mem.mprotect.4k` ahead (816
against 1,019 ns); the AMD bench is ahead on all three.

- **Owed on metal** for the asks above (a reached CPU flushing only the
change's range, the initiator owing itself nothing):
- The T14's `shared-debug` boot (the actuator kernel), where main must pass
`test_rs_tlb_shootdown_waits`: a change still waits for every CPU running
the space.
- An A/B on both benches of `osbench_system` and `osbench_mem`, main from
before and after. Main after must read `mem.munmap_touched_remote.4k`
faster, and `mem.munmap_part.4k`, `mem.mprotect.4k` and the multi-thread
`mem.malloc_churn.*` rows no slower. Until then the gain is unmeasured.
- With it, each bench's cycles per flush, one page by `INVPCID` (the T14)
or `INVLPG` (the AMD bench, which has no PCID) against one flush of every
user translation. From those, the number of pages at which flushing one
by one costs as much as a full flush; the reached CPU's limit, 64, is
lowered if that number is well below it, or the reason not to is
recorded. Branches `memperf-sd-ab-base` and `memperf-sd-ab-head` carry a
measurement-only probe that logs both; they were staged and never booted.

**Exit**: that case, on each proving machine with PCID, red with the flush on
entry removed; and the T14's munmap and `vm_protect` figures against Linux.
15 changes: 15 additions & 0 deletions kernel/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,21 @@ reap-raise-relaxed = []
shootdown-serve-relaxed = []
# `src/shootdown.rs`'s `served` reads `Relaxed`, and `counters_round` reds.
shootdown-served-relaxed = []
# `src/shootdown.rs`'s `Asks::serve` takes its asks `Relaxed`, and
# `tlb_shootdown` reds.
shootdown-take-relaxed = []
# `src/shootdown.rs`'s `Asks::serve` answers `Relaxed`, and `tlb_shootdown`
# reds.
shootdown-answer-relaxed = []
# `src/shootdown.rs`'s `Asks::ask` writes its range after asking, and
# `tlb_shootdown` reds.
shootdown-ask-range-late = []
# `src/shootdown.rs`'s `Asks::ask` leaves the last ask's answers standing, and
# `tlb_shootdown` reds.
shootdown-ask-stale-answers = []
# `src/shootdown.rs`'s `Asks::serve` answers two asks with one's range, and
# `tlb_shootdown` reds.
shootdown-ask-many-as-one = []
# `src/tlb_tags.rs`'s `enter` loses its fence, and `tlb_tags` reds.
tlb-tags-enter-relaxed = []
# `src/log/shard.rs`'s publishing store goes `Relaxed`, and `log_record` reds.
Expand Down
23 changes: 23 additions & 0 deletions kernel/loom/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,29 @@ shootdown-serve-relaxed = []
# Never on by default and never reachable from a kernel build, which declares
# the same name only so `cfg` checking knows it.
shootdown-served-relaxed = []
# The negative controls for a user change's ask (`shootdown.rs`'s `Asks`),
# each of which `tlb_shootdown.rs` must red:
#
# cargo test --manifest-path kernel/loom/Cargo.toml --features <one> \
# --test tlb_shootdown
#
# `shootdown-take-relaxed`: the target takes its asks `Relaxed`, so its flush
# no longer follows the asking CPU's page-table write.
# `shootdown-answer-relaxed`: the target answers `Relaxed`, so an asking CPU
# can read its answer before the flush it stands for.
# `shootdown-ask-range-late`: the asking CPU writes its range after asking, so
# a target can flush a range it was not asked.
# `shootdown-ask-stale-answers`: an ask leaves the last ask's answers standing,
# so a reused mailbox is answered before its targets have flushed.
# `shootdown-ask-many-as-one`: an answer to two asks at once flushes one range.
#
# Never on by default and never reachable from a kernel build, which declares
# the same names only so `cfg` checking knows them.
shootdown-take-relaxed = []
shootdown-answer-relaxed = []
shootdown-ask-range-late = []
shootdown-ask-stale-answers = []
shootdown-ask-many-as-one = []
# The negative control for the CPU roster's publication. It makes
# `smp_roster.rs`'s `commit` publish the count with a `Relaxed` store, so a
# reader can see a count over a slot the commit has not filled, and
Expand Down
218 changes: 171 additions & 47 deletions kernel/loom/tests/tlb_shootdown.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@

use std::sync::atomic::{AtomicBool, Ordering::SeqCst};

use kernel_loom::shootdown::{Reach, Shootdown};
use kernel_loom::shootdown::{Asks, CpuSet, Flush, Reach, Shootdown};
use loom::sync::Arc;
use loom::sync::atomic::{AtomicU64, Ordering::Relaxed};

Expand Down Expand Up @@ -126,7 +126,8 @@ impl Machine {
}
}

/// One serve of vector 0xFE's body on cpu 1.
/// One serve on cpu 1: vector 0xFE's body serves when owed, and serving
/// regardless explores every schedule that one does and more.
fn serve(&self) {
self.shootdown.serve(1, |reach| self.flush(reach));
}
Expand Down Expand Up @@ -221,49 +222,6 @@ fn one_serve_answers_two_concurrent_shootdowns() {
});
}

/// A CPU that is waiting for an acknowledgement has not stopped giving them.
///
/// **Not an interleaving question, so loom explores nothing here** — the
/// schedule below is written out because it is the one two CPUs took on
/// 2026-08-07, and it is the whole defect. `loom::model` is still what wraps it:
/// this crate compiles `shootdown.rs` against loom's atomics, which panic
/// outside a model, and the property under test is the protocol's shape rather
/// than any edge between its atomics.
///
/// `IF` is clear for the whole of a syscall (`arch::syscall`'s `MSR_FMASK`), so
/// neither CPU below can take the other's IPI. Each one's own wait is the only
/// thing left that can answer the other, and until this test existed the wait
/// asked without answering: both spun until `ACK_TIMEOUT_NS` killed them, as a
/// double kernel panic and as seven wide-phase test failures sharing one line.
#[test]
fn an_initiator_answers_while_it_waits() {
model(|| {
let s = two_cpus();

// cpu 0 unmaps and answers for itself; cpu 1 has not issued yet, so the
// generation cpu 0 publishes is its own.
let g0 = s.issue(Reach::Narrow);
s.serve(0, |_| {});

// cpu 1 unmaps while cpu 0 is between its own `serve` and its wait.
let g1 = s.issue(Reach::Narrow);

// cpu 0's wait is not satisfied — cpu 1 has not flushed — and that turn
// is nevertheless where cpu 1's answer comes from.
assert!(!s.wait_turn(0, 1, g0, |_| {}));
assert!(
s.served(0, g1),
"cpu 0 is waiting for cpu 1 and has stopped answering it, which is \
a wait neither CPU can leave",
);

// And cpu 1's own wait then has an answer to find.
s.serve(1, |_| {});
assert!(s.wait_turn(1, 0, g1, |_| {}));
assert!(s.wait_turn(0, 1, g0, |_| {}));
});
}

/// A serve that an interrupt nests inside another publishes the later
/// generation, and the outer serve, finishing after it, must not take it back.
///
Expand Down Expand Up @@ -397,8 +355,9 @@ fn a_serve_nested_in_an_unfinished_wide_one_answers_wide() {
/// kernel-half one, which it answers wide once: a user-half change keeps the
/// kernel's global translations.
///
/// **Not an interleaving question**, as `an_initiator_answers_while_it_waits`
/// is not; reds when a serve answers wide with no wide generation owed.
/// **Not an interleaving question**: `loom::model` wraps it because this
/// crate's atomics are loom's. Reds when a serve answers wide with no wide
/// generation owed.
#[test]
fn only_a_kernel_half_shootdown_asks_a_wide_answer() {
model(|| {
Expand Down Expand Up @@ -429,3 +388,168 @@ fn two_cpus() -> Shootdown {
shootdown.size(2);
shootdown
}

/// A user change's range asked of the CPUs running its space ([`Asks`]): cpu 1
/// is the target and holds a translation of each of two pages, page 0 and
/// page 1, as one cell each. A range answer re-walks the pages it names, an
/// answer to more than one ask at once every page.
struct Asked {
asks: Asks,
pte: [AtomicU64; 2],
tlb: [AtomicU64; 2],
}

impl Asked {
fn new() -> Arc<Self> {
let asks = Asks::new();
asks.size(3);
Arc::new(Self {
asks,
pte: [AtomicU64::new(MAPPED), AtomicU64::new(MAPPED)],
tlb: [AtomicU64::new(MAPPED), AtomicU64::new(MAPPED)],
})
}

fn rewalk(&self, page: usize) {
self.tlb[page].store(self.pte[page].load(Relaxed), Relaxed);
}

/// cpu 1's answers, `SERVES` of them.
fn take_interrupts(&self) {
for _ in 0..SERVES {
loom::thread::yield_now();
self.asks.serve(1, |what| match what {
Flush::Range { start, end, .. } => (start..end.min(2)).for_each(|page| self.rewalk(page as usize)),
Flush::User => (0..2).for_each(|page| self.rewalk(page)),
});
}
}

/// `me` unmaps `page` and asks cpu 1 to flush it; what cpu 1 holds of the
/// page once it has answered, `None` when the bounded poll ran out.
fn unmap_and_ask(&self, me: usize, page: usize) -> Option<u64> {
self.pte[page].store(UNMAPPED, Relaxed);
let mut targets = CpuSet::default();
targets[0] = 1 << 1;
self.asks.ask(me, 0, page as u64, page as u64 + 1, &targets);
for _ in 0..POLLS {
if self.asks.answered(me, 1) {
return Some(self.tlb[page].load(Relaxed));
}
loom::thread::yield_now();
}
None
}
}

/// An answered ask means cpu 1 has re-walked the page asked after the unmap.
///
/// Reds when the target takes its asks `Relaxed`
/// (`shootdown-take-relaxed`), so its flush no longer follows the write; when
/// it answers `Relaxed` (`shootdown-answer-relaxed`), so the asking CPU can
/// read its answer before the flush; and when the asking CPU writes its range
/// after asking (`shootdown-ask-range-late`), so the target can flush a range
/// it was never asked.
#[test]
fn an_answered_ask_flushed_its_range_after_the_write() {
acknowledged_model(|| {
let m = Asked::new();
let target = {
let m = m.clone();
loom::thread::spawn(move || m.take_interrupts())
};
let answer = m.unmap_and_ask(0, 0);
if let Some(reachable) = answer {
assert_eq!(reachable, UNMAPPED, "cpu 1 answered the ask while still holding the page asked");
}
target.join().unwrap();
answer.is_some()
});
}

/// Two CPUs ask cpu 1 at once, each for its own page: an answer to one alone
/// flushes its range, an answer to both every page, and each asking CPU finds
/// its own page gone once answered, whichever way its ask was taken.
///
/// Reds when an answer to two asks at once flushes one of their ranges
/// (`shootdown-ask-many-as-one`).
#[test]
fn two_asks_of_one_cpu_are_each_answered_for_their_own_page() {
acknowledged_model(|| {
let m = Asked::new();
let target = {
let m = m.clone();
loom::thread::spawn(move || m.take_interrupts())
};
let second = {
let m = m.clone();
loom::thread::spawn(move || m.unmap_and_ask(2, 1))
};
let first = m.unmap_and_ask(0, 0);
if let Some(reachable) = first {
assert_eq!(reachable, UNMAPPED, "cpu 0 was answered while cpu 1 still held page 0");
}
let second = second.join().unwrap();
if let Some(reachable) = second {
assert_eq!(reachable, UNMAPPED, "cpu 2 was answered while cpu 1 still held page 1");
}
target.join().unwrap();
first.is_some() || second.is_some()
});
}

/// A CPU asks again only once every CPU it asked has answered, and its next
/// ask waits for answers of its own: its earlier answers do not stand for it.
///
/// Reds when an ask leaves the previous ask's answers standing
/// (`shootdown-ask-stale-answers`): the mailbox reused would be answered at
/// once, while cpu 1 still holds the second page.
#[test]
fn a_mailbox_reused_waits_for_its_new_answer() {
acknowledged_model(|| {
let m = Asked::new();
let target = {
let m = m.clone();
loom::thread::spawn(move || {
m.take_interrupts();
m.take_interrupts();
})
};
let mut answered = false;
if m.unmap_and_ask(0, 0).is_some() {
let again = m.unmap_and_ask(0, 1);
if let Some(reachable) = again {
assert_eq!(reachable, UNMAPPED, "cpu 0's second ask was answered while cpu 1 still held page 1");
}
answered = again.is_some();
}
target.join().unwrap();
answered
});
}

/// A serve answers each CPU asking it once, with that CPU's own range: two
/// CPUs that ask each other each find their answer once the other has
/// served, and a second serve finds nothing left to answer. **Not an
/// interleaving question.** The order in which a waiting CPU answers before
/// it reads its own answer is `arch::tlb::wait_for`'s, which loom does not
/// compile; its doc states it and inspection holds it.
#[test]
fn each_asker_is_answered_once_with_its_own_range() {
model(|| {
let asks = Asks::new();
asks.size(2);
let (mut to_one, mut to_zero) = (CpuSet::default(), CpuSet::default());
to_one[0] = 1 << 1;
to_zero[0] = 1;
asks.ask(0, 0, 0, 1, &to_one);
asks.ask(1, 0, 1, 2, &to_zero);
// cpu 0's serve answers cpu 1 with cpu 1's range.
assert!(asks.serve(0, |what| assert_eq!(what, Flush::Range { tag: 0, start: 1, end: 2 })));
assert!(!asks.answered(0, 1));
assert!(asks.answered(1, 0), "cpu 0 served and did not answer cpu 1");
assert!(asks.serve(1, |_| {}));
assert!(asks.answered(0, 1));
assert!(!asks.serve(0, |_| {}), "an ask was answered twice");
});
}
Loading
Loading