Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
status: open
kind: tooling
opened: 2026-11-05
---

# A red run keeps no serial log for a boot whose UART is QEMU's stdio

A suite that is not green copies every `uart-*.log` under its scratch to
`target/red-run-serial/` (`tests/common/lane.rs`'s `keep_serial`), which
`guest-rows.yml` uploads as `serial`. A boot writes `uart-<seq>.log` only
where its UART is on the socket the harness listens on, a profile with the
console's port (`toyos_build::vmconsole::UartSocket`). Where the 16550 is
QEMU's stdio, the console exists only in the harness's memory, so a red boot
of such a profile leaves no file.

Guest-rows 42010705757 reddened `acpi_press_is_one_sci`
(`Profile::HeadlessNoConsolePort`) with a STALL, and its `serial` artifact
held one file: lane-2's `uart-4.log`, a passing console-port boot of
`acpi_holder_is_refused_a_held_functions_configuration`. The stalled boot's
console survived only as the failure message's tail.

**Exit**: every boot of a red run leaves its whole console under
`target/red-run-serial/`, whatever carries it.
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
status: open
kind: defect
opened: 2026-11-05
---

# `acpi_press_is_one_sci` boots to a console that carries no line past the UART's hand-over

`acpi_press_is_one_sci` (`tests/images/acpicase.toml` with the probe's
`press` arm, `Profile::HeadlessNoConsolePort`, the shipping kernel) stalls
on every run since the UART became its driver's (#1175): `STALLED: waiting
for the probe to arm the power button — it went quiet`. The console carries
the boot through the kernel's `boot: power-on to loader … kernel to Boot:
complete` and then nothing for the 17 s the harness waits: no supervisor
line, no `spawn:`, nothing of the userland the UART's driver carries. The
harness presses the button only on the probe's armed line, so the boot never
ends.

Runs, all `guest-rows.yml`, x86-64 KVM:
- 42010705757 and 42016111039 (`--nocapture`), #1187's head 0383550b0;
- 42025888487 (`--nocapture`), main at b27dd3e2.

Three of three, on main too, so it is not #1187's. The nightly of 11-03 (41618038084),
before #1175, passed the row on its kvm lane; the nightly has not run since.

**What the same runs establish beside it.** The other four acpicase rows
boot the same image on the same profile, and their consoles carry the
driver's lines after the hand-over and end in the stop: but each of those
boots ends within about half a second of `Boot: complete`. In 42010705757 the
console-port boot of `acpi_holder_is_refused_a_held_functions_configuration`
(its `uart-4.log`) carried the driver's lines to 2.139 s, stopped mid-line,
and carried nothing more until the kernel took the UART back for the stop at
2.544 s, whose replay then started past records the UART never showed. So a
boot that does not end soon is the one that shows it: whether the UART's
driver stops transmitting, `logkeeper` stops feeding it, or userland itself
stops is not read. None of the three stalls left a serial file
(`issues/a-red-run-keeps-no-serial-for-a-boot-whose-uart-is-qemus-stdio.md`),
and nothing read the guest's CPUs: the next step is the row with QMP's `info
registers -a` and the trace diary, or a boot of acpicase held open by a job
that sleeps.

Owner: the console UART's driver (`system/uart`) and its feed
(`system/logkeeper/src/carrier.rs`); held by the orchestrator.

**Exit**: the cause named and fixed, and `acpi_press_is_one_sci` green.
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,10 @@ opened: 2026-10-07
`smi_cmd::write` (`kernel/src/arch/x86_64/smi_cmd.rs`) makes every write to
`SMI_CMD` on the boot processor: a caller on another CPU kicks it and spins
until it has written. The T14's `acpi_server_death` row reads that crossing
for the enable, because its job claims from a thread it found off the boot
processor and the judge refuses `asked from cpu0` there. #1164 registered
the row again; its job now claims from the boot processor after 64 tries
that all landed there, and says so, which the judge then refuses. Three
things it leaves to the scheduler or does not read:
for the enable: its job claims from a thread held to the last CPU
(`tests/guest/rust-tests/src/off_the_boot_processor.rs`), and the judge
refuses `asked from cpu0` there. Two things it leaves to the scheduler or
does not read:

- **The disable asked from another CPU.** `acpi_mode::release` runs in the
task that drops the claim's last handle. On the `acpicase` boot that is the
Expand All @@ -28,14 +27,6 @@ things it leaves to the scheduler or does not read:
asker may hold, where no kick reaches it. No row arranges that contention,
and the kernel's line does not say whether a write was answered from the
kick or from a spin.
- **The enable's arrangement has a window.** The job's claiming thread reads
its x2APIC id and then enters the kernel
(`acpi_release`'s `claim_off_the_boot_processor`). The kernel moves no running thread, so the
asker is the CPU the thread read, unless a preemption between that read and
the kernel's lock in `smi_cmd::write` queues the thread behind another and
an idle boot processor takes it. Pinning the claiming thread off the boot
processor (`toyos_abi::syscall::thread_set_affinity`) closes the window,
and the job does not pin it yet.

The disable, on the recorded `acpicase` boots. Ten before `289906d5b`: the
`legacy mode again` line was logged on cpu0 in nine and on cpu1 in one, an
Expand All @@ -49,30 +40,14 @@ the job's: of the three boots whose line names the asker, one did not cross.
Whether any write on any boot was answered from a lock's spin is not known:
no line says which path answered.

The window, on those two boots, the only ones of the job that places its
claim: neither hit it. Each enable reads `asked from cpu1` beside the job's `acpi_release: claimed from
the CPU of x2APIC id 2`. A boot that does hit it reds the row as

`FAIL acpi_server_death: the job's claim was asked from the boot processor, so no CPU asked it for this write:`

over an enable line reading `on cpu0, asked from cpu0`, while the job's own
line names a non-zero x2APIC id. That pair is this window and no defect of
the kernel's write: it is answered here, by closing the window or by the
row's judge giving up the refusal, and never by a second boot.

What could arrange the disable's asker, unbuilt and unmeasured: a holder that
reads itself off the boot processor and ends holding the claim, so its own
teardown is the release; or a thread of the job's kept running on the boot
processor from before the server's spawn to after its kill, so that no
placement puts the server there, which rests on where a wake puts a parked
thread and that has not been read against it. Either inherits the window
above.
What could arrange the disable's asker, unbuilt and unmeasured: the server
spawned from the job's held thread, which starts with that thread's mask
(`toyos_abi::syscall::thread_set_affinity`), so its threads, and the
teardown its kill ends in, run on that CPU alone.

Owned by `issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md`.

**Exit**: a T14 row's judge refuses a boot whose `legacy mode again` line
reads `asked from cpu0`, on a job that arranges the asker and does not leave
it to the scheduler; a T14 row reads a kernel line saying a write to
`SMI_CMD` was answered from a lock's spin; and the thread whose asker a judge
refuses is held to its CPU from before it reads where it is until the kernel
has taken its request.
it to the scheduler; and a T14 row reads a kernel line saying a write to
`SMI_CMD` was answered from a lock's spin.
45 changes: 45 additions & 0 deletions issues/sixty-four-spawns-in-a-row-landed-on-the-spawners-cpu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
status: open
kind: finding
opened: 2026-11-05
---

# Sixty-four thread spawns in a row landed on the spawner's own CPU, where placement prefers the other

`acpi_mediated`, before #1187 held its asker to the last CPU, spawned a
thread up to 64 times from its main thread on cpu0 of a two-CPU q35 guest,
joining each, until one read itself off cpu0. In four runs all 64 ran on
cpu0 (`acpi: 64 threads in a row started on the boot processor`): nightly
39566825102 (tcg), 40847028162 and 41618038084 (kvm), and guest-rows
42008942191 on main at 15c26bfbc (kvm), where the 64 took about 15 ms.

**The rule it contradicts.** A spawn is placed by `CpuHandles::place`
(`kernel/pure/sched/cpu.rs`) from a start that rotates per spawn
(`placement`, `kernel/src/sched/driver.rs`), by the least of `(load,
!sleeping_unrung, busy_siblings, homed, away, idle_exit)` among answering
CPUs. With equal published load an idle cpu1 wins on `sleeping_unrung`, and
otherwise on `homed`, since the spawner's CPU is home to the process's main
thread; a remaining tie alternates with the start. Sixty-four landings on
cpu0 in a row are systematic.

**Candidates, none established:**
- cpu1's published load stayed above cpu0's for the whole burst;
- cpu1 read as not `answering`: a doorbell edge standing past
`STALE_PASS_NS`, so the scan left it out. A wake signal left unserviced
that long is a kernel defect, and this record becomes a `defect` with
severity `hang`;
- each thread went to cpu1 and was handed to cpu0 before its first
instruction: the spawner blocks in `join` at once, so cpu0 turns idle and
asks to steal while cpu1 is still waking from its halt
(`answer_steal_requests`).

No row arranges this any more: the probes pin. The cheapest reproduction is a
guest binary that spawns and joins in a loop from cpu0 and counts where each
thread's first `current_cpu()` lands, with the scheduler's trace diary read
after it.

Owner: `kernel::sched`'s placement, held by the orchestrator.

**Exit**: the cause named from a reproduction, and fixed if it is a defect;
or the rule shown to put these spawns on cpu0, and its documentation made to
say so.
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@ QEMU, while `guest.yml`'s container still installs 11.1.1 from its snapshot
archive, so its instrument step reds until the container moves to an image
carrying 11.1.3.

`nightly.yml`'s `portability-linux` job runs in
`public.ecr.aws/docker/library/debian:sid`, a moving tag pulled from the
public internet on every run, so a pull refused for rate is a red: six
nightlies from 2026-10-24 to 10-31 failed that job at `Error response from
daemon: toomanyrequests: Data limit exceeded` before any step ran
(e.g. run 41187139929, job 124496246843). Resumption owes it an image pinned by
digest from a mirror the build controls.

**Exit**: the nightly resumes with its guest lanes on the guest runner rather
than GitHub's machines. Only a `workflow_dispatch`-only workflow may name that
runner, so resumption owes a design for a scheduled job under that guard. Held
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -437,17 +437,6 @@ neither bench's dump does AML write a port the kernel refuses.
kernel's own commands were not; the time a handler takes, and the SMI
count either side of one, are read on the T14 alone, and there only for
the enable and the disable.
- **`acpi_mediated_access` needs one of three calls to be asked off the boot
processor**, each from a thread that read itself there first. The job
does not pin that thread, though a mask now could, so
a thread preempted between that read and the kernel's lock may be taken by
the boot processor, the window
`issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md`
describes. A boot where all three fall in it reds as `no firmware call was
asked from another CPU`, over three kernel lines reading `asked from cpu0`
beside the probe's own line naming three non-zero x2APIC ids: that is the
window and no defect of the write, and it is answered by the pin, never by
a second run. None has been read.
- **`acpi_mediated_access`'s storm is refused only if nine calls fit in one
second of the guest's clock.** The probe asks one after another and needs
the ninth refused `CommandRate`. A guest whose host gives it less than
Expand Down
5 changes: 3 additions & 2 deletions tests/guest/rust-tests/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ ring = "0.17"
toyos-crypto = { path = "../../../lib/crypto", features = ["sign"] }
toyos-parity = { path = "../../../lib/parity" }
ml-kem = { version = "0.3", features = ["hazmat"] }
# The ecosystem's way to pin a thread, for `affinity_pins`: its ToyOS arm is
# what holds that existing Rust pins as it does on Linux.
# The ecosystem's way to pin a thread, for `affinity_pins`, whose ToyOS arm is
# what holds that existing Rust pins as it does on Linux, and for the ACPI
# probes' requests asked off the boot processor.
core_affinity = "0.8"
getrandom = "0.2"

Expand Down
49 changes: 9 additions & 40 deletions tests/guest/rust-tests/src/bin/acpi_mediated.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ use toyos_abi::RawHandle;

#[path = "../arch/cpu.rs"]
mod cpu;
#[path = "../off_the_boot_processor.rs"]
mod off_the_boot_processor;

const SELF_PATH: &str = "/system/bin/test_rs_acpi_mediated";
const CLAIM_LABEL: &str = "acpi-claim";
Expand Down Expand Up @@ -362,19 +364,15 @@ fn ports(holder: &Holder, info: &AcpiInfo) {
}

/// Bytes this guest's FADT gives no meaning, which its chipset model keeps
/// and does nothing on: the calls asked from a thread off the boot processor
/// where there is one, and the two the storm alternates.
/// and does nothing on: the calls asked off the boot processor, and the two
/// the storm alternates.
const CROSSED: [u64; 3] = [0x51, 0x52, 0x53];
const STORMED: [u64; 2] = [0x54, 0x55];

/// Calls the storm asks at most: what firmware's AML asks of a handler that
/// never answers, one a millisecond for ten seconds.
const STORM: usize = 10_000;

/// Threads started in turn before one must have found itself off the boot
/// processor.
const STARTS: usize = 64;

/// What the boot processor counts of the kernel's writes to `SMI_CMD`, in a
/// round every CPU answered: every answer is taken after its read was called.
fn firmware_calls(cap: &SysCap) -> u64 {
Expand All @@ -393,44 +391,15 @@ fn firmware_calls(cap: &SysCap) -> u64 {
}
}

/// One byte for `SMI_CMD`, asked by a thread that read itself off the boot
/// processor as its first act where the machine has another CPU — this one if
/// it is, or one started for it: the x2APIC id it read, 0 on a machine of one.
///
/// This thread is asked first because a thread started from a CPU that is
/// busy goes to an idle one where there is one, and on a machine of two the
/// idle one is the boot processor whenever this thread is not on it.
/// One byte for `SMI_CMD`, asked from the last CPU, which is not the boot
/// processor where the machine has another: the x2APIC id it asked from.
fn call_off_the_boot_processor(holder: &Holder, smi_cmd: u64, value: u64) -> u32 {
let claim = holder.handle();
let ask = || {
off_the_boot_processor::run(|| {
let mut access = Access::write(Space::SystemIo, smi_cmd, Width::Byte, value);
assert_eq!(syscall::acpi_access(claim, &mut access), Ok(Ok(value)), "acpi: firmware call {value:#04x}");
};
if syscall::cpu_count() == 1 {
ask();
return 0;
}
let here = cpu::x2apic_id();
if here != 0 {
ask();
return here;
}
for _ in 0..STARTS {
let on = std::thread::scope(|threads| {
let asker = threads.spawn(|| {
let on = cpu::x2apic_id();
(on != 0).then(|| {
ask();
on
})
});
asker.join().expect("acpi: the calling thread")
});
if let Some(on) = on {
return on;
}
}
panic!("acpi: {STARTS} threads in a row started on the boot processor");
cpu::x2apic_id()
})
}

fn firmware(holder: &Holder, info: &AcpiInfo, cap: &SysCap) {
Expand Down
40 changes: 7 additions & 33 deletions tests/guest/rust-tests/src/bin/acpi_release.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
//! The ACPI server's death, on a boot that does not start it: this job claims
//! the fixed hardware from a thread off the boot processor, or, after 64 tries
//! that all landed on it, from the boot processor, which it says. The claim
//! the fixed hardware from a thread held off the boot processor. The claim
//! puts the machine in ACPI mode by a write the asking CPU asked for (the T14
//! judge requires one off the boot processor; a guest's row does not). It hands the claim to
//! `/system/bin/acpiserver`, and kills it once it has declared the root
Expand All @@ -13,7 +12,6 @@
use std::io::{BufRead, BufReader};
use std::os::toyos::process::CommandExt;
use std::process::{Command, Stdio};
use std::thread;

use toyos::endow::{Endowments, DEV_PREFIX, SYSCAP_LABEL};
use toyos::syscap::SysCap;
Expand All @@ -22,6 +20,8 @@ use toyos_abi::syscall::DeviceType;

#[path = "../arch/cpu.rs"]
mod cpu;
#[path = "../off_the_boot_processor.rs"]
mod off_the_boot_processor;

const SERVER: &str = "/system/bin/acpiserver";

Expand All @@ -35,38 +35,12 @@ const UNDECLARED: [&str; 3] = [
"acpiserver: the kernel already holds an answer",
];

/// Threads started in turn before one must have found itself off the boot
/// processor: a spawn goes to the least loaded CPU from a start that rotates.
const STARTS: usize = 64;

/// The claim, asked by a thread that read itself off the boot processor as
/// its first act: x2APIC id 0, which the judge holds against the kernel's own
/// `asked from`. The kernel moves no running thread, so the asker is that CPU
/// unless a preemption between the read and the kernel's lock queued the
/// thread behind another and an idle boot processor took it. Past [`STARTS`]
/// it claims from wherever it runs.
fn claim_off_the_boot_processor(cap: &SysCap) -> Device {
for _ in 0..STARTS {
let claimed = thread::scope(|threads| {
let asker = threads.spawn(|| {
let on = cpu::x2apic_id();
(on != 0).then(|| (on, cap.claim::<Device>(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim")))
});
asker.join().expect("acpi_release: the claiming thread")
});
if let Some((on, claim)) = claimed {
println!("acpi_release: claimed from the CPU of x2APIC id {on}");
return claim;
}
}
// Which CPU asked is the judge's to hold, off the kernel's own line.
println!("acpi_release: {STARTS} threads in a row started on the boot processor, so it claims from there");
cap.claim::<Device>(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim")
}

fn main() {
let cap: SysCap = Endowments::get().take(SYSCAP_LABEL).expect("test-runner endows a device-minting capability");
let claim = claim_off_the_boot_processor(&cap);
let (on, claim) = off_the_boot_processor::run(|| {
(cpu::x2apic_id(), cap.claim::<Device>(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim"))
});
println!("acpi_release: claimed from the CPU of x2APIC id {on}");
let mut server = Command::new(SERVER)
.endow(&format!("{DEV_PREFIX}{}", DeviceType::Acpi.class_name()), claim.into())
.stdout(Stdio::piped())
Expand Down
18 changes: 18 additions & 0 deletions tests/guest/rust-tests/src/off_the_boot_processor.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
//! A request asked from a CPU other than the boot processor.

/// Run `ask` on a thread held to the machine's last CPU, which is the boot
/// processor only on a machine of one: what `ask` answered. The hold moves
/// the thread before `ask` starts and keeps it there, so every CPU `ask`
/// reads, and every one the kernel names for its requests, is that one.
pub fn run<R: Send>(ask: impl FnOnce() -> R + Send) -> R {
let last = toyos_abi::syscall::cpu_count() as usize - 1;
std::thread::scope(|threads| {
threads
.spawn(|| {
assert!(core_affinity::set_for_current(core_affinity::CoreId { id: last }), "a thread held to CPU {last}");
ask()
})
.join()
.expect("the thread held off the boot processor")
})
}
Loading