diff --git a/issues/a-red-run-keeps-no-serial-for-a-boot-whose-uart-is-qemus-stdio.md b/issues/a-red-run-keeps-no-serial-for-a-boot-whose-uart-is-qemus-stdio.md new file mode 100644 index 00000000000..4ef28b8d680 --- /dev/null +++ b/issues/a-red-run-keeps-no-serial-for-a-boot-whose-uart-is-qemus-stdio.md @@ -0,0 +1,24 @@ +--- +status: open +kind: tooling +opened: 2026-11-05 +--- + +# A red run keeps no serial log for a boot whose UART is QEMU's stdio + +A suite that is not green copies every `uart-*.log` under its scratch to +`target/red-run-serial/` (`tests/common/lane.rs`'s `keep_serial`), which +`guest-rows.yml` uploads as `serial`. A boot writes `uart-.log` only +where its UART is on the socket the harness listens on, a profile with the +console's port (`toyos_build::vmconsole::UartSocket`). Where the 16550 is +QEMU's stdio, the console exists only in the harness's memory, so a red boot +of such a profile leaves no file. + +Guest-rows 42010705757 reddened `acpi_press_is_one_sci` +(`Profile::HeadlessNoConsolePort`) with a STALL, and its `serial` artifact +held one file: lane-2's `uart-4.log`, a passing console-port boot of +`acpi_holder_is_refused_a_held_functions_configuration`. The stalled boot's +console survived only as the failure message's tail. + +**Exit**: every boot of a red run leaves its whole console under +`target/red-run-serial/`, whatever carries it. diff --git a/issues/acpi-press-is-one-sci-boots-to-a-console-that-carries-no-line-past-the-hand-over.md b/issues/acpi-press-is-one-sci-boots-to-a-console-that-carries-no-line-past-the-hand-over.md new file mode 100644 index 00000000000..7b9b63b13b1 --- /dev/null +++ b/issues/acpi-press-is-one-sci-boots-to-a-console-that-carries-no-line-past-the-hand-over.md @@ -0,0 +1,45 @@ +--- +status: open +kind: defect +opened: 2026-11-05 +--- + +# `acpi_press_is_one_sci` boots to a console that carries no line past the UART's hand-over + +`acpi_press_is_one_sci` (`tests/images/acpicase.toml` with the probe's +`press` arm, `Profile::HeadlessNoConsolePort`, the shipping kernel) stalls +on every run since the UART became its driver's (#1175): `STALLED: waiting +for the probe to arm the power button — it went quiet`. The console carries +the boot through the kernel's `boot: power-on to loader … kernel to Boot: +complete` and then nothing for the 17 s the harness waits: no supervisor +line, no `spawn:`, nothing of the userland the UART's driver carries. The +harness presses the button only on the probe's armed line, so the boot never +ends. + +Runs, all `guest-rows.yml`, x86-64 KVM: +- 42010705757 and 42016111039 (`--nocapture`), #1187's head 0383550b0; +- 42025888487 (`--nocapture`), main at b27dd3e2. + +Three of three, on main too, so it is not #1187's. The nightly of 11-03 (41618038084), +before #1175, passed the row on its kvm lane; the nightly has not run since. + +**What the same runs establish beside it.** The other four acpicase rows +boot the same image on the same profile, and their consoles carry the +driver's lines after the hand-over and end in the stop: but each of those +boots ends within about half a second of `Boot: complete`. In 42010705757 the +console-port boot of `acpi_holder_is_refused_a_held_functions_configuration` +(its `uart-4.log`) carried the driver's lines to 2.139 s, stopped mid-line, +and carried nothing more until the kernel took the UART back for the stop at +2.544 s, whose replay then started past records the UART never showed. So a +boot that does not end soon is the one that shows it: whether the UART's +driver stops transmitting, `logkeeper` stops feeding it, or userland itself +stops is not read. None of the three stalls left a serial file +(`issues/a-red-run-keeps-no-serial-for-a-boot-whose-uart-is-qemus-stdio.md`), +and nothing read the guest's CPUs: the next step is the row with QMP's `info +registers -a` and the trace diary, or a boot of acpicase held open by a job +that sleeps. + +Owner: the console UART's driver (`system/uart`) and its feed +(`system/logkeeper/src/carrier.rs`); held by the orchestrator. + +**Exit**: the cause named and fixed, and `acpi_press_is_one_sci` green. diff --git a/issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md b/issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md index 60a09807de0..1c6814de4a0 100644 --- a/issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md +++ b/issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md @@ -9,11 +9,10 @@ opened: 2026-10-07 `smi_cmd::write` (`kernel/src/arch/x86_64/smi_cmd.rs`) makes every write to `SMI_CMD` on the boot processor: a caller on another CPU kicks it and spins until it has written. The T14's `acpi_server_death` row reads that crossing -for the enable, because its job claims from a thread it found off the boot -processor and the judge refuses `asked from cpu0` there. #1164 registered -the row again; its job now claims from the boot processor after 64 tries -that all landed there, and says so, which the judge then refuses. Three -things it leaves to the scheduler or does not read: +for the enable: its job claims from a thread held to the last CPU +(`tests/guest/rust-tests/src/off_the_boot_processor.rs`), and the judge +refuses `asked from cpu0` there. Two things it leaves to the scheduler or +does not read: - **The disable asked from another CPU.** `acpi_mode::release` runs in the task that drops the claim's last handle. On the `acpicase` boot that is the @@ -28,14 +27,6 @@ things it leaves to the scheduler or does not read: asker may hold, where no kick reaches it. No row arranges that contention, and the kernel's line does not say whether a write was answered from the kick or from a spin. -- **The enable's arrangement has a window.** The job's claiming thread reads - its x2APIC id and then enters the kernel - (`acpi_release`'s `claim_off_the_boot_processor`). The kernel moves no running thread, so the - asker is the CPU the thread read, unless a preemption between that read and - the kernel's lock in `smi_cmd::write` queues the thread behind another and - an idle boot processor takes it. Pinning the claiming thread off the boot - processor (`toyos_abi::syscall::thread_set_affinity`) closes the window, - and the job does not pin it yet. The disable, on the recorded `acpicase` boots. Ten before `289906d5b`: the `legacy mode again` line was logged on cpu0 in nine and on cpu1 in one, an @@ -49,30 +40,14 @@ the job's: of the three boots whose line names the asker, one did not cross. Whether any write on any boot was answered from a lock's spin is not known: no line says which path answered. -The window, on those two boots, the only ones of the job that places its -claim: neither hit it. Each enable reads `asked from cpu1` beside the job's `acpi_release: claimed from -the CPU of x2APIC id 2`. A boot that does hit it reds the row as - -`FAIL acpi_server_death: the job's claim was asked from the boot processor, so no CPU asked it for this write:` - -over an enable line reading `on cpu0, asked from cpu0`, while the job's own -line names a non-zero x2APIC id. That pair is this window and no defect of -the kernel's write: it is answered here, by closing the window or by the -row's judge giving up the refusal, and never by a second boot. - -What could arrange the disable's asker, unbuilt and unmeasured: a holder that -reads itself off the boot processor and ends holding the claim, so its own -teardown is the release; or a thread of the job's kept running on the boot -processor from before the server's spawn to after its kill, so that no -placement puts the server there, which rests on where a wake puts a parked -thread and that has not been read against it. Either inherits the window -above. +What could arrange the disable's asker, unbuilt and unmeasured: the server +spawned from the job's held thread, which starts with that thread's mask +(`toyos_abi::syscall::thread_set_affinity`), so its threads, and the +teardown its kill ends in, run on that CPU alone. Owned by `issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md`. **Exit**: a T14 row's judge refuses a boot whose `legacy mode again` line reads `asked from cpu0`, on a job that arranges the asker and does not leave -it to the scheduler; a T14 row reads a kernel line saying a write to -`SMI_CMD` was answered from a lock's spin; and the thread whose asker a judge -refuses is held to its CPU from before it reads where it is until the kernel -has taken its request. +it to the scheduler; and a T14 row reads a kernel line saying a write to +`SMI_CMD` was answered from a lock's spin. diff --git a/issues/sixty-four-spawns-in-a-row-landed-on-the-spawners-cpu.md b/issues/sixty-four-spawns-in-a-row-landed-on-the-spawners-cpu.md new file mode 100644 index 00000000000..c6b5455aa42 --- /dev/null +++ b/issues/sixty-four-spawns-in-a-row-landed-on-the-spawners-cpu.md @@ -0,0 +1,45 @@ +--- +status: open +kind: finding +opened: 2026-11-05 +--- + +# Sixty-four thread spawns in a row landed on the spawner's own CPU, where placement prefers the other + +`acpi_mediated`, before #1187 held its asker to the last CPU, spawned a +thread up to 64 times from its main thread on cpu0 of a two-CPU q35 guest, +joining each, until one read itself off cpu0. In four runs all 64 ran on +cpu0 (`acpi: 64 threads in a row started on the boot processor`): nightly +39566825102 (tcg), 40847028162 and 41618038084 (kvm), and guest-rows +42008942191 on main at 15c26bfbc (kvm), where the 64 took about 15 ms. + +**The rule it contradicts.** A spawn is placed by `CpuHandles::place` +(`kernel/pure/sched/cpu.rs`) from a start that rotates per spawn +(`placement`, `kernel/src/sched/driver.rs`), by the least of `(load, +!sleeping_unrung, busy_siblings, homed, away, idle_exit)` among answering +CPUs. With equal published load an idle cpu1 wins on `sleeping_unrung`, and +otherwise on `homed`, since the spawner's CPU is home to the process's main +thread; a remaining tie alternates with the start. Sixty-four landings on +cpu0 in a row are systematic. + +**Candidates, none established:** +- cpu1's published load stayed above cpu0's for the whole burst; +- cpu1 read as not `answering`: a doorbell edge standing past + `STALE_PASS_NS`, so the scan left it out. A wake signal left unserviced + that long is a kernel defect, and this record becomes a `defect` with + severity `hang`; +- each thread went to cpu1 and was handed to cpu0 before its first + instruction: the spawner blocks in `join` at once, so cpu0 turns idle and + asks to steal while cpu1 is still waking from its halt + (`answer_steal_requests`). + +No row arranges this any more: the probes pin. The cheapest reproduction is a +guest binary that spawns and joins in a loop from cpu0 and counts where each +thread's first `current_cpu()` lands, with the scheduler's trace diary read +after it. + +Owner: `kernel::sched`'s placement, held by the orchestrator. + +**Exit**: the cause named from a reproduction, and fixed if it is a defect; +or the rule shown to put these spawns on cpu0, and its documentation made to +say so. diff --git a/issues/the-nightly-is-frozen-so-nothing-runs-the-guest-matrix.md b/issues/the-nightly-is-frozen-so-nothing-runs-the-guest-matrix.md index 0cfcb2546b4..430386e8c29 100644 --- a/issues/the-nightly-is-frozen-so-nothing-runs-the-guest-matrix.md +++ b/issues/the-nightly-is-frozen-so-nothing-runs-the-guest-matrix.md @@ -20,6 +20,14 @@ QEMU, while `guest.yml`'s container still installs 11.1.1 from its snapshot archive, so its instrument step reds until the container moves to an image carrying 11.1.3. +`nightly.yml`'s `portability-linux` job runs in +`public.ecr.aws/docker/library/debian:sid`, a moving tag pulled from the +public internet on every run, so a pull refused for rate is a red: six +nightlies from 2026-10-24 to 10-31 failed that job at `Error response from +daemon: toomanyrequests: Data limit exceeded` before any step ran +(e.g. run 41187139929, job 124496246843). Resumption owes it an image pinned by +digest from a mirror the build controls. + **Exit**: the nightly resumes with its guest lanes on the guest runner rather than GitHub's machines. Only a `workflow_dispatch`-only workflow may name that runner, so resumption owes a design for a scheduled job under that guard. Held diff --git a/issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md b/issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md index 4ab9ae8cce8..3e000da711f 100644 --- a/issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md +++ b/issues/toyos-runs-the-machine-in-acpi-mode-and-interprets-its-aml.md @@ -437,17 +437,6 @@ neither bench's dump does AML write a port the kernel refuses. kernel's own commands were not; the time a handler takes, and the SMI count either side of one, are read on the T14 alone, and there only for the enable and the disable. -- **`acpi_mediated_access` needs one of three calls to be asked off the boot - processor**, each from a thread that read itself there first. The job - does not pin that thread, though a mask now could, so - a thread preempted between that read and the kernel's lock may be taken by - the boot processor, the window - `issues/no-t14-row-arranges-an-acpi-disable-asked-off-the-boot-processor.md` - describes. A boot where all three fall in it reds as `no firmware call was - asked from another CPU`, over three kernel lines reading `asked from cpu0` - beside the probe's own line naming three non-zero x2APIC ids: that is the - window and no defect of the write, and it is answered by the pin, never by - a second run. None has been read. - **`acpi_mediated_access`'s storm is refused only if nine calls fit in one second of the guest's clock.** The probe asks one after another and needs the ninth refused `CommandRate`. A guest whose host gives it less than diff --git a/tests/guest/rust-tests/Cargo.toml b/tests/guest/rust-tests/Cargo.toml index c99b9dccc11..ee37ce88278 100644 --- a/tests/guest/rust-tests/Cargo.toml +++ b/tests/guest/rust-tests/Cargo.toml @@ -43,8 +43,9 @@ ring = "0.17" toyos-crypto = { path = "../../../lib/crypto", features = ["sign"] } toyos-parity = { path = "../../../lib/parity" } ml-kem = { version = "0.3", features = ["hazmat"] } -# The ecosystem's way to pin a thread, for `affinity_pins`: its ToyOS arm is -# what holds that existing Rust pins as it does on Linux. +# The ecosystem's way to pin a thread, for `affinity_pins`, whose ToyOS arm is +# what holds that existing Rust pins as it does on Linux, and for the ACPI +# probes' requests asked off the boot processor. core_affinity = "0.8" getrandom = "0.2" diff --git a/tests/guest/rust-tests/src/bin/acpi_mediated.rs b/tests/guest/rust-tests/src/bin/acpi_mediated.rs index 225a314c11f..7126a44fc07 100644 --- a/tests/guest/rust-tests/src/bin/acpi_mediated.rs +++ b/tests/guest/rust-tests/src/bin/acpi_mediated.rs @@ -69,6 +69,8 @@ use toyos_abi::RawHandle; #[path = "../arch/cpu.rs"] mod cpu; +#[path = "../off_the_boot_processor.rs"] +mod off_the_boot_processor; const SELF_PATH: &str = "/system/bin/test_rs_acpi_mediated"; const CLAIM_LABEL: &str = "acpi-claim"; @@ -362,8 +364,8 @@ fn ports(holder: &Holder, info: &AcpiInfo) { } /// Bytes this guest's FADT gives no meaning, which its chipset model keeps -/// and does nothing on: the calls asked from a thread off the boot processor -/// where there is one, and the two the storm alternates. +/// and does nothing on: the calls asked off the boot processor, and the two +/// the storm alternates. const CROSSED: [u64; 3] = [0x51, 0x52, 0x53]; const STORMED: [u64; 2] = [0x54, 0x55]; @@ -371,10 +373,6 @@ const STORMED: [u64; 2] = [0x54, 0x55]; /// never answers, one a millisecond for ten seconds. const STORM: usize = 10_000; -/// Threads started in turn before one must have found itself off the boot -/// processor. -const STARTS: usize = 64; - /// What the boot processor counts of the kernel's writes to `SMI_CMD`, in a /// round every CPU answered: every answer is taken after its read was called. fn firmware_calls(cap: &SysCap) -> u64 { @@ -393,44 +391,15 @@ fn firmware_calls(cap: &SysCap) -> u64 { } } -/// One byte for `SMI_CMD`, asked by a thread that read itself off the boot -/// processor as its first act where the machine has another CPU — this one if -/// it is, or one started for it: the x2APIC id it read, 0 on a machine of one. -/// -/// This thread is asked first because a thread started from a CPU that is -/// busy goes to an idle one where there is one, and on a machine of two the -/// idle one is the boot processor whenever this thread is not on it. +/// One byte for `SMI_CMD`, asked from the last CPU, which is not the boot +/// processor where the machine has another: the x2APIC id it asked from. fn call_off_the_boot_processor(holder: &Holder, smi_cmd: u64, value: u64) -> u32 { let claim = holder.handle(); - let ask = || { + off_the_boot_processor::run(|| { let mut access = Access::write(Space::SystemIo, smi_cmd, Width::Byte, value); assert_eq!(syscall::acpi_access(claim, &mut access), Ok(Ok(value)), "acpi: firmware call {value:#04x}"); - }; - if syscall::cpu_count() == 1 { - ask(); - return 0; - } - let here = cpu::x2apic_id(); - if here != 0 { - ask(); - return here; - } - for _ in 0..STARTS { - let on = std::thread::scope(|threads| { - let asker = threads.spawn(|| { - let on = cpu::x2apic_id(); - (on != 0).then(|| { - ask(); - on - }) - }); - asker.join().expect("acpi: the calling thread") - }); - if let Some(on) = on { - return on; - } - } - panic!("acpi: {STARTS} threads in a row started on the boot processor"); + cpu::x2apic_id() + }) } fn firmware(holder: &Holder, info: &AcpiInfo, cap: &SysCap) { diff --git a/tests/guest/rust-tests/src/bin/acpi_release.rs b/tests/guest/rust-tests/src/bin/acpi_release.rs index 263a313f69c..a823e6fc94a 100644 --- a/tests/guest/rust-tests/src/bin/acpi_release.rs +++ b/tests/guest/rust-tests/src/bin/acpi_release.rs @@ -1,6 +1,5 @@ //! The ACPI server's death, on a boot that does not start it: this job claims -//! the fixed hardware from a thread off the boot processor, or, after 64 tries -//! that all landed on it, from the boot processor, which it says. The claim +//! the fixed hardware from a thread held off the boot processor. The claim //! puts the machine in ACPI mode by a write the asking CPU asked for (the T14 //! judge requires one off the boot processor; a guest's row does not). It hands the claim to //! `/system/bin/acpiserver`, and kills it once it has declared the root @@ -13,7 +12,6 @@ use std::io::{BufRead, BufReader}; use std::os::toyos::process::CommandExt; use std::process::{Command, Stdio}; -use std::thread; use toyos::endow::{Endowments, DEV_PREFIX, SYSCAP_LABEL}; use toyos::syscap::SysCap; @@ -22,6 +20,8 @@ use toyos_abi::syscall::DeviceType; #[path = "../arch/cpu.rs"] mod cpu; +#[path = "../off_the_boot_processor.rs"] +mod off_the_boot_processor; const SERVER: &str = "/system/bin/acpiserver"; @@ -35,38 +35,12 @@ const UNDECLARED: [&str; 3] = [ "acpiserver: the kernel already holds an answer", ]; -/// Threads started in turn before one must have found itself off the boot -/// processor: a spawn goes to the least loaded CPU from a start that rotates. -const STARTS: usize = 64; - -/// The claim, asked by a thread that read itself off the boot processor as -/// its first act: x2APIC id 0, which the judge holds against the kernel's own -/// `asked from`. The kernel moves no running thread, so the asker is that CPU -/// unless a preemption between the read and the kernel's lock queued the -/// thread behind another and an idle boot processor took it. Past [`STARTS`] -/// it claims from wherever it runs. -fn claim_off_the_boot_processor(cap: &SysCap) -> Device { - for _ in 0..STARTS { - let claimed = thread::scope(|threads| { - let asker = threads.spawn(|| { - let on = cpu::x2apic_id(); - (on != 0).then(|| (on, cap.claim::(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim"))) - }); - asker.join().expect("acpi_release: the claiming thread") - }); - if let Some((on, claim)) = claimed { - println!("acpi_release: claimed from the CPU of x2APIC id {on}"); - return claim; - } - } - // Which CPU asked is the judge's to hold, off the kernel's own line. - println!("acpi_release: {STARTS} threads in a row started on the boot processor, so it claims from there"); - cap.claim::(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim") -} - fn main() { let cap: SysCap = Endowments::get().take(SYSCAP_LABEL).expect("test-runner endows a device-minting capability"); - let claim = claim_off_the_boot_processor(&cap); + let (on, claim) = off_the_boot_processor::run(|| { + (cpu::x2apic_id(), cap.claim::(DeviceType::Acpi).expect("acpi_release: the fixed hardware's claim")) + }); + println!("acpi_release: claimed from the CPU of x2APIC id {on}"); let mut server = Command::new(SERVER) .endow(&format!("{DEV_PREFIX}{}", DeviceType::Acpi.class_name()), claim.into()) .stdout(Stdio::piped()) diff --git a/tests/guest/rust-tests/src/off_the_boot_processor.rs b/tests/guest/rust-tests/src/off_the_boot_processor.rs new file mode 100644 index 00000000000..6bf00fb2e63 --- /dev/null +++ b/tests/guest/rust-tests/src/off_the_boot_processor.rs @@ -0,0 +1,18 @@ +//! A request asked from a CPU other than the boot processor. + +/// Run `ask` on a thread held to the machine's last CPU, which is the boot +/// processor only on a machine of one: what `ask` answered. The hold moves +/// the thread before `ask` starts and keeps it there, so every CPU `ask` +/// reads, and every one the kernel names for its requests, is that one. +pub fn run(ask: impl FnOnce() -> R + Send) -> R { + let last = toyos_abi::syscall::cpu_count() as usize - 1; + std::thread::scope(|threads| { + threads + .spawn(|| { + assert!(core_affinity::set_for_current(core_affinity::CoreId { id: last }), "a thread held to CPU {last}"); + ask() + }) + .join() + .expect("the thread held off the boot processor") + }) +} diff --git a/tests/toyos.rs b/tests/toyos.rs index 41e6e9c5553..1f17350f37e 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -2934,8 +2934,8 @@ const ACPI_MEDIATED_SAID: [&str; 10] = [ /// this kernel makes on a guest, whose firmware hands it over in ACPI mode: /// the kernel says the first call of each byte, five of them, each written /// on the boot processor by that CPU's own reading beside the `out`, and -/// where there is a second CPU one at least asked from it, the probe having -/// asked three from a thread that read itself there. +/// the three the probe asked from a thread held to the last CPU asked from +/// that CPU. fn acpi_mediated_access(cpus: u32) -> Result<(), String> { const JOB: &str = "test_rs_acpi_mediated"; const HELD_INTO_THE_STOP: &str = "acpi: holding the Global Lock, and asking for the power-off with it"; @@ -2989,15 +2989,17 @@ fn acpi_mediated_access(cpus: u32) -> Result<(), String> { if calls.len() != 5 { return Err(format!("the kernel said {} first firmware calls where the probe made calls of five bytes: {calls:#?}", calls.len())); } - let mut crossed = 0; + let held = u64::from(cpus - 1); for line in &calls { smi_cmd_writer(line)?; - crossed += usize::from(number_between(line, ", asked from cpu", "; the write held cpu")? != 0); + let byte = field_between(line, "acpi: firmware call ", " written to SMI_CMD ")?; + let asker = number_between(line, ", asked from cpu", "; the write held cpu")?; + // The probe's `CROSSED`. + if ["0x51", "0x52", "0x53"].contains(&byte) && asker != held { + return Err(format!("firmware call {byte} was asked from cpu{asker}, where the probe held its asker to cpu{held}: {line}")); + } eprintln!(" [acpi] {}", line.trim()); } - if cpus > 1 && crossed == 0 { - return Err(format!("no firmware call was asked from another CPU than the boot processor, so none crossed to it: {calls:#?}")); - } // The shutdown refused by the kernel's own name for it, before anything // was stopped: the probe ran on and said every line above. eprintln!(" [acpi] {}", said.must_say(NO_S5)?.trim());