Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 47 additions & 19 deletions tests/guest/osbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,11 @@ and the C library beneath it are each system's own.
- **At least four runs of each system, interleaved**: Linux, ToyOS, Linux,
ToyOS. A ToyOS run is one boot and one readback directory. A Linux run is
`osbench all </dev/null` over ssh, each group a process of its own as the
ToyOS job list runs them. A run's value is its p50, compared by the exact
two-sided Mann-Whitney test at 0.05. With four runs each the only p under
ToyOS job list runs them. It starts `nvme` through `sudo -n` where that
asks no password, since the raw namespace takes root, and exits 3 before
any group on a boot whose clocksource the vDSO cannot read ("The clock",
below). A run's value is its p50, compared by the exact two-sided
Mann-Whitney test at 0.05. With four runs each the only p under
0.05 is 2/70, so a resolved row is one whose every run of one system beat
every run of the other. Fewer runs of either system than four cannot reach
that, except against more of the other: three against five can (2/56).
Expand All @@ -48,11 +51,11 @@ and the C library beneath it are each system's own.
with `--metal-host` for a bench other than the T14.
- **Compare** every run of the bench in one call:
`cargo run -p osbench-compare -- <linux runs> <toyos kernel.logs>`. It
refuses runs of one system that disagree on their setup (`SETUP`) and
states each system's, and what each run's boot chose (`PER_RUN`, the
clocksource) run by run. Each row's verdict says whether the runs resolve a
difference. Each p is its row's alone: with no difference anywhere, about
2.9% of rows resolve by chance, a few in a table this size.
refuses runs of one system that disagree on their setup (`SETUP`, the
clocksource among it) and states each system's. Each row's verdict says
whether the runs resolve a difference. Each p is its row's alone: with no
difference anywhere, about 2.9% of rows resolve by chance, a few in a table
this size.
- **The clocks are told beside each verdict.** No program reads the
package's limit state or temperature on ToyOS, so the clock is what both
systems report. A clock reading is the difference between a short add
Expand All @@ -71,12 +74,11 @@ and the C library beneath it are each system's own.

## Linux as installed, and Linux under ToyOS's requests

**The baseline is Linux as the bench boots it**: its own governor, its own
clocksource, its own checks. What it does on that machine is ordinary OS
behaviour and part of what is compared. A comparison names its
configuration in its setup lines, and `osbench-compare` refuses to mix two.
What a boot chooses for itself, the clocksource, belongs to Linux as booted:
each run states its own, and runs that differ are compared together.
**The baseline is Linux as the bench boots it**: its own governor and its
own checks, on a boot that kept the TSC as its clocksource. What it does on
that machine is ordinary OS behaviour and part of what is compared. A
comparison names its configuration in its setup lines, and `osbench-compare`
refuses to mix two.

**A matched request is a controlled experiment, labelled as one.** It asks
the CPU's frequency control for what ToyOS's kernel asks of the same CPU, so
Expand Down Expand Up @@ -126,8 +128,15 @@ carried by every per-operation timing of that run. In 30 boots' kernel logs
In a 13th boot the calibration read 1799.565 MHz and the watchdog let it
stand, so that Linux kept a TSC clock some 1,630 ppm fast for the whole
boot. Its `clocksource=tsc` reads the same as a right one's. The 12 others
kept the TSC on a right calibration. Each run's clocksource is part of the
Linux it measured, so the compare states it per run.
kept the TSC on a right calibration.

An HPET boot is no run. The scoreboard of 2026-11-11 took three of the AMD
bench's five Linux runs on the HPET: their per-operation rows read slower by
the timestamps' cost and their all-CPU rows a third lower, so the median of
five was an HPET run's, and `fair.shared.total` read ToyOS above it and under
both TSC runs. So `osbench` on Linux exits 3 before any group where the
clocksource is not the one its vDSO reads, and the bench is booted again
until it is; `osbench-compare` refuses runs whose clocksources differ.

At runtime neither fault shows: across 16 CPUs and idle gaps up to 1 s, no
CPU's TSC read behind another's and no step exceeded the 1.2 µs of a read.
Expand Down Expand Up @@ -239,11 +248,30 @@ rest of the suite, on the benchcase image. Of the three only `persist`
writes the bench's NVMe; `files` is the memory filesystem's rows, and `nvme`
is read-only and unsupported on ToyOS.

## Waits are judged slack-free

An ordinary Linux thread's sleep is padded by its timer slack, 50 µs by
default, and ToyOS pads none. Linux runs each wait row a second time with
the slack at 1 ns (`<row>.slack1ns`), and that is the row judged: the
compare sets ToyOS's row of the unsuffixed name against it, and gives the
unsuffixed row no verdict.

## A futex round trip across cores is one that slept

A side of `pingpong.futex.*` that finds its word already changed never
sleeps. Across two cores a whole round can pass that way: three of the T14's
four boots of 2026-11-11 read the row at 250 to 257 ns and the fourth at
4.6 µs. So `pingpong.futex.cross_core` keeps the rounds in which each side's
wake found the other queued in the kernel; the others are
`pingpong.futex.cross_core.unparked`, and a run in which no round parked
reports the row `unmeasured`. On one CPU no round passes without both sides
leaving it, so that row keeps every round. Both say how many such wakes a
kept round had, of two (`parks_per_round`).

## What is not a comparison

The table also holds rows only one system can measure. They show the
system's figure beside the other's reason: `unsupported` and `unmeasured`
rows, Linux's `.slack1ns` variants and `fair.session`, and ToyOS's
`tail.diag` and `fs.*.direct.*`, the file server's client without std's
layer above it, which Linux has no counterpart to. They are records, not
verdicts.
rows, Linux's `fair.session`, and ToyOS's `tail.diag` and `fs.*.direct.*`,
the file server's client without std's layer above it, which Linux has no
counterpart to. They are records, not verdicts.
108 changes: 68 additions & 40 deletions tests/guest/osbench/compare/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,11 @@
//!
//! A system's runs on one machine are one configuration ([`SETUP`]): runs that
//! disagree on it are refused, since their median would mix two, and each
//! machine's table states what each system's runs were taken under. What a
//! system chooses afresh at each boot ([`PER_RUN`]) is part of that system as
//! booted: it is stated run by run and never refused.
//! machine's table states what each system's runs were taken under.
//!
//! Linux pads an ordinary thread's sleep by its timer slack and ToyOS pads
//! none, so a row Linux also ran slack-free ([`SLACK_FREE`]) is judged there,
//! against ToyOS's row of the unsuffixed name, and not under that name.

use std::collections::BTreeMap;
use std::fmt::Write as _;
Expand Down Expand Up @@ -261,27 +263,33 @@ pub enum Better {
}

/// Which way `unit` is better, for the units whose direction is known: a
/// time and a rate. A clock, a power or a share is no better higher or lower.
/// time, a rate and a distance from an entitlement. A clock, a power or a
/// share is no better higher or lower.
pub fn better(unit: &str) -> Option<Better> {
match unit {
"ns" => Some(Better::Lower),
"MB/s" => Some(Better::Higher),
"ns" | "ps" | "ns/MiB" | "permille_off" => Some(Better::Lower),
"MB/s" | "kops/s" | "chunks/s" => Some(Better::Higher),
_ => None,
}
}

/// The suffix of a row Linux ran with its thread's timer slack at 1 ns.
pub const SLACK_FREE: &str = ".slack1ns";

/// The condition every run of a machine must agree on for a benchmark's
/// runs to be compared, and how many of its leading words must agree: the
/// peer's link kind and driver, and the internet link's kind, not what
/// varies by the run.
const PATHS: [(&str, &str, usize); 2] = [("lan.", "lan.peer_link", 2), ("internet.", "internet.link", 1)];

/// The conditions every run of one system on one machine must share: the
/// kernel, the CPU's frequency request, what serves each filesystem the `fs`
/// and `persist` rows use, and the panel, its light, the radios and the links
/// the `energy` rows draw beside.
pub const SETUP: [&str; 17] = [
/// kernel, the clock every timestamp is read from, the CPU's frequency
/// request, what serves each filesystem the `fs` and `persist` rows use, and
/// the panel, its light, the radios and the links the `energy` rows draw
/// beside.
pub const SETUP: [&str; 18] = [
"kernel",
"clocksource",
"cpufreq_driver",
"governor",
"epp",
Expand All @@ -300,25 +308,6 @@ pub const SETUP: [&str; 17] = [
"energy.links",
];

/// The conditions a system chooses at each boot, stated per run: the clock
/// every timestamp is read from, which Linux's watchdog can move off the TSC
/// in one boot and not the next.
pub const PER_RUN: [&str; 1] = ["clocksource"];

/// `os`'s [`PER_RUN`] conditions, each as `key=<run 1's>,<run 2's>...` in the
/// runs' order, `absent` for a run that names none.
fn per_run(runs: &[&Run], os: Os) -> Vec<String> {
let runs: Vec<&&Run> = runs.iter().filter(|r| r.os == os).collect();
PER_RUN
.iter()
.filter(|key| runs.iter().any(|r| r.conds.contains_key(**key)))
.map(|key| {
let said: Vec<&str> = runs.iter().map(|r| r.conds.get(*key).map_or("absent", String::as_str)).collect();
format!("{key}={}", said.join(","))
})
.collect()
}

/// `os`'s setup on these runs, as `key=value` in [`SETUP`]'s order, or the
/// first key its runs disagree on.
fn setup(runs: &[&Run], os: Os) -> Result<Vec<String>, String> {
Expand Down Expand Up @@ -370,10 +359,6 @@ pub fn compare(runs: &[Run]) -> Result<String, String> {
if !said.is_empty() {
writeln!(out, "{} setup: {}.\n", os.name(), said.join(", ")).unwrap();
}
let each = per_run(&runs, os);
if !each.is_empty() {
writeln!(out, "{} per run: {}.\n", os.name(), each.join(", ")).unwrap();
}
}
writeln!(out, "| bench | unit | Linux p50s | Linux median | Linux range | ToyOS p50s | ToyOS median | ToyOS range | ToyOS/Linux | p | verdict |").unwrap();
writeln!(out, "|---|---|---|---|---|---|---|---|---|---|---|").unwrap();
Expand Down Expand Up @@ -403,8 +388,15 @@ fn compare_one(runs: &[&Run], name: &str) -> String {
let mut settled: BTreeMap<Os, Vec<u64>> = BTreeMap::new();
let mut media: BTreeMap<Os, std::collections::BTreeSet<Option<String>>> = BTreeMap::new();
let mut not: BTreeMap<Os, Vec<String>> = BTreeMap::new();
// ToyOS's side of a slack-free row is its row of the unsuffixed name.
let unpadded = name.strip_suffix(SLACK_FREE).filter(|_| !runs.iter().any(|r| r.os == Os::Toyos && r.benches.contains_key(name)));
let padded = runs.iter().any(|r| r.os == Os::Linux && r.benches.contains_key(&format!("{name}{SLACK_FREE}")));
for run in runs {
match run.benches.get(name) {
let outcome = match unpadded {
Some(unpadded) if run.os == Os::Toyos => run.benches.get(unpadded),
_ => run.benches.get(name),
};
match outcome {
Some(Outcome::Measured { unit: u, p50, clock_fell, settled: mhz, unsettled, medium }) => {
if unit.get_or_insert(u.as_str()) != u {
return "| | | | | | | | | units differ between runs".into();
Expand Down Expand Up @@ -442,6 +434,9 @@ fn compare_one(runs: &[&Run], name: &str) -> String {
};
("-".into(), "-".into(), format!("the media differ: Linux {}, ToyOS {}", named(Os::Linux), named(Os::Toyos)))
}
(Some(_), Some(_)) if padded => {
("-".into(), "-".into(), format!("Linux's timer slack pads this wait: judged as {name}{SLACK_FREE}"))
}
(Some(l), Some(t)) => {
let ratio = median(t) / median(l);
let exact = mann_whitney(t, l);
Expand Down Expand Up @@ -479,6 +474,9 @@ fn compare_one(runs: &[&Run], name: &str) -> String {
note(format!("{said} in {n} of {of} {} run(s)", os.name()));
}
}
if let Some(unpadded) = unpadded.filter(|_| values.contains_key(&Os::Toyos)) {
note(format!("ToyOS's is its {unpadded}"));
}
for (os, reasons) in &not {
let mut distinct = reasons.clone();
distinct.sort();
Expand Down Expand Up @@ -629,7 +627,10 @@ mod tests {
}

#[test]
fn a_throughputs_higher_value_is_the_better_one() {
fn a_rates_higher_value_is_the_better_one_and_a_times_lower() {
assert_eq!(["ns", "ps", "ns/MiB", "permille_off"].map(better), [Some(Better::Lower); 4]);
assert_eq!(["MB/s", "kops/s", "chunks/s"].map(better), [Some(Better::Higher); 3]);
assert_eq!(["MHz", "mW", "permille", "us"].map(better), [None; 4]);
let mbps = |os: &str, v: u64| run(os, &[&format!("bench=lan.wired.tcp.upload unit=MB/s n=5 min=1 p50={v} max=9 mean=5")]);
let mut runs: Vec<Run> = [40, 41, 42, 43].iter().map(|v| mbps("linux", *v)).collect();
runs.extend([50, 51, 52, 53].iter().map(|v| mbps("toyos", *v)));
Expand Down Expand Up @@ -715,7 +716,7 @@ mod tests {
#[test]
fn a_systems_runs_on_two_setups_are_refused_and_one_setup_is_stated() {
let under = |os: &str, governor: &str| {
run(os, &[&format!("cond governor={governor}"), "cond clocksource=tsc", "bench=time.now unit=ps n=15 min=1 p50=14 max=1 mean=1"])
run(os, &[&format!("cond governor={governor}"), "bench=time.now unit=ps n=15 min=1 p50=14 max=1 mean=1"])
};
let mixed = [under("linux", "performance*16"), under("linux", "schedutil*16"), under("toyos", "absent")];
let why = compare(&mixed).unwrap_err();
Expand All @@ -728,14 +729,41 @@ mod tests {
}

#[test]
fn a_clocksource_each_boot_chose_is_stated_per_run_and_never_refused() {
fn runs_timed_by_two_clocksources_are_refused() {
let booted = |clocksource: &str| {
run("linux", &[&format!("cond clocksource={clocksource}"), "bench=time.now unit=ps n=15 min=1 p50=14 max=1 mean=1"])
};
let runs = [booted("hpet"), booted("tsc"), booted("hpet"), run("toyos", &["bench=time.now unit=ps n=15 min=1 p50=9 max=1 mean=1"])];
let mixed = [booted("hpet"), booted("tsc"), booted("hpet")];
assert!(compare(&mixed).unwrap_err().contains("the Linux runs disagree on clocksource: hpet / tsc"));
let one = [booted("tsc"), booted("tsc")];
assert!(compare(&one).unwrap().contains("Linux setup: clocksource=tsc."));
}

#[test]
fn a_wait_linux_ran_slack_free_is_judged_there_against_toyos_unpadded_row() {
let linux = |padded: u64, slack_free: u64| {
run(
"linux",
&[
&format!("bench=time.timer_1ms.late unit=ns n=300 min=1 p50={padded} max=9 mean=5"),
&format!("bench=time.timer_1ms.late.slack1ns unit=ns n=300 min=1 p50={slack_free} max=9 mean=5"),
],
)
};
let toyos = |v: u64| run("toyos", &[&format!("bench=time.timer_1ms.late unit=ns n=300 min=1 p50={v} max=9 mean=5")]);
let mut runs: Vec<Run> = [(53_000, 3_000), (54_000, 3_100), (55_000, 3_200), (56_000, 3_300)].iter().map(|&(p, s)| linux(p, s)).collect();
runs.extend([9_000, 9_100, 9_200, 9_300].iter().map(|v| toyos(*v)));
let table = compare(&runs).unwrap();
assert!(table.contains("Linux per run: clocksource=hpet,tsc,hpet."), "{table}");
assert!(!table.contains("ToyOS per run"), "{table}");
let row = |name: &str| table.lines().find(|l| l.starts_with(&format!("| {name} |"))).unwrap().to_string();
// Against the padded row ToyOS would read six times better.
let padded = row("time.timer_1ms.late");
assert!(padded.ends_with("| - | - | Linux's timer slack pads this wait: judged as time.timer_1ms.late.slack1ns |"), "{padded}");
let judged = row("time.timer_1ms.late.slack1ns");
assert!(judged.contains("| 9000, 9100, 9200, 9300 |"), "{judged}");
assert!(judged.ends_with("| 0.029 | ToyOS worse; ToyOS's is its time.timer_1ms.late |"), "{judged}");
// A row Linux ran only padded is judged as it is.
let alone = [run("linux", &["bench=time.yield unit=ps n=15 min=1 p50=10 max=1 mean=1"]), run("toyos", &["bench=time.yield unit=ps n=15 min=1 p50=30 max=1 mean=1"])];
assert!(!compare(&alone).unwrap().contains("slack"));
}

#[test]
Expand Down
2 changes: 1 addition & 1 deletion tests/guest/osbench/rules/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "osbench-rules"
description = "osbench's pure rules, host-tested: when a munmap pair's x2APIC readings admit its figure, where the CPU crunch stops for heat, the package energy counter's span, unit and power, when a machine is quiet, whether two counters readings describe their row, and when a CPU's clock has settled."
description = "osbench's pure rules, host-tested: when a munmap pair's x2APIC readings admit its figure, where the CPU crunch stops for heat, the package energy counter's span, unit and power, when a machine is quiet, whether two counters readings describe their row, when a CPU's clock has settled, each contending job's fair share of the machine, and which rounds of a futex exchange slept."
version = "0.1.0"
edition = "2021"
license = "MIT OR Apache-2.0"
Expand Down
Loading