diff --git a/docs/src/architecture/diagnostics.md b/docs/src/architecture/diagnostics.md index 2d6351a8ac5..f058cddf46e 100644 --- a/docs/src/architecture/diagnostics.md +++ b/docs/src/architecture/diagnostics.md @@ -20,9 +20,14 @@ exports them. - The kernel's logging responsibility is its own record ring and its console. Userland output never enters the kernel ring. Reading the machine's log needs a right on the system capability. -- The log keeper owns the log files. It holds each program to an allowance, - keeps every boot's beginning, and syncs on an alert or an interval. Severity - is an ordered ladder. +- The log keeper owns the log files. It holds each program to an allowance + and keeps every boot's beginning. Severity is an ordered ladder. +- The file is written in batches: at once for an error, an alert or the + machine's stop, and otherwise within a second. A machine that dies without + stopping loses at most that second of its file while the volume keeps pace; + the console is not batched. + No file call holds up the console or a reader, and a stop waits on the file + within a bound. - A slow sink drops and counts; it never blocks a producer and fails alone. - The kernel waits on the log keeper neither at a panic nor at a stop. diff --git a/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md b/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md deleted file mode 100644 index 1fd3804b6bf..00000000000 --- a/issues/logkeeper-hears-a-reader-leave-only-at-its-next-write.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-10-08 ---- - -# logkeeper hears a reader leave only at its next write - -A log reader's thread (`system/logkeeper/src/serve.rs`, `feed`) waits on -`shared.grew` once it has caught up, and learns its reader is gone only when a -write to the sink fails. A reader that leaves while the log is quiet keeps its -thread and its place in `MAX_READERS` until the next record lands. - -Read from the code, not measured. - -**Exit condition**: a reader's end is what wakes its thread, or the header of -`serve.rs` says why the next record is soon enough. - -**Owner**: whoever holds `issues/redesign-the-log-subsystem.md`. diff --git a/issues/redesign-the-log-subsystem.md b/issues/redesign-the-log-subsystem.md index e480730e633..cb3b00acee5 100644 --- a/issues/redesign-the-log-subsystem.md +++ b/issues/redesign-the-log-subsystem.md @@ -109,3 +109,19 @@ sources, and the review's target was one `gates` directory under `kernel/src/` so that what test machinery exists is auditable in one listing. `input_merge_test` was the tell that pure logic was trapped in the kernel: the merge is `toyos-input`'s now, host-tested with synthetic multi-source streams. + +**The file written in batches** (#1232) rests on host tests and guest rows. +The benches went down before its T14 half ran; on the AMD bench its +`spawn_cost` A/B (one boot per arm) and `machine_reboot` passed. + +- **Owed on metal:** `machine_reboot` on the T14, at main. Main must pass the + boot verdict with the stop mark and the supervisor's stop line in the file, + `machine_reboot`'s reset-register line, and `test_rs_ps2server_hold` exit 0 + (a served-log reader is handed its line through the reader's wake), with no + "had not answered the stop's flush", "stop mark was asked … after its time", + "answers but slowly" or "refusing reader" line. +- **Owed on metal:** `spawn_cost`'s A/B on the T14, main before #1232 against + main after it, back to back. Main after must show "the machine busy … a + spawn" lower than before, by about one write and sync per logkeeper round the + loop spans (on the AMD bench, 430 µs against 360 µs a spawn), min and median + no worse than the two runs' spread, and both boots passing with the stop mark. diff --git a/issues/the-console-waits-on-logkeepers-file-writes.md b/issues/the-console-waits-on-logkeepers-file-writes.md deleted file mode 100644 index 71f20a5c4e5..00000000000 --- a/issues/the-console-waits-on-logkeepers-file-writes.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-11-05 -severity: [hang] ---- - -# The console waits on logkeeper's file writes - -`logkeeper`'s loop is the only thing that reads the rings and grows the -replay the console's carriers and readers are fed from, and it makes every -call on the LOG file server itself, each one blocking (`system/logkeeper/src`): - -- a round's write and sync (`Log::to_volume`: `Volume::write`, `Volume::sync`); -- the catch-up of a file behind the replay (`Log::caught_up`, from a round and - from `Log::adopt`: `Volume::catch_up`'s reopen, writes, sync and rotation); -- a rotation's sync, retention sweep and create (`Log::rotate_if_full`); -- the stop's catch-up and sync (`Volume::finish`), whose time bounds how many - calls it starts and not how long one takes; -- the stop mark and its removal (`store::mark_stopped`, `unmark_stopped`). - -`LOG_WRITE_BUDGET` judges a call only once it returns. So a LOG server that -takes a call and never answers, alive but wedged, leaves every console channel -silent for as long as it does, with no failure bound, and the supervisor -restarts only a server that ended. The stop's wait on the volume's opener is -the one wait here with a bound (`STOP_CATCH_UP`). - -Read from the code, not reproduced. The volume's open waited the same way -before it moved to a thread of its own, which is how the console stood silent -for the supervisor's 30 s declaration bound on `acpi_press_is_one_sci`'s boot. - -**Exit**: no console channel waits on a `/log` call; a guest boot whose LOG -server stops answering mid-boot still carries its later lines. The shape: the -file is a consumer of the replay with its own offset, written on its own -thread as the readers and carriers are, and the stop asks it to reach the -replay's end and sync within a bound. The per-file offset and the stop's -bounded catch-up are in the tree (`Volume::through`, `catch_up`, `finish`). -What is left is the move: the loop still owns the `Volume` and makes the calls -above, so the volume goes to a thread of its own that reads the replay, and -the stop's bound becomes a wait on that thread, which the loop can give up on. - -Owner: `system/logkeeper`; unheld. diff --git a/lib/logstream/src/lib.rs b/lib/logstream/src/lib.rs index d5198e92440..b3b9020bde5 100644 --- a/lib/logstream/src/lib.rs +++ b/lib/logstream/src/lib.rs @@ -122,10 +122,11 @@ pub const REGISTER: u32 = 1; pub const SERVED: u32 = 2; /// The supervisor → `logkeeper`: the machine is about to be stopped. `logkeeper` reads every ring -/// and the kernel's records, writes them, makes the volume durable — or says, -/// in the file and on the console, what of the boot it could not — and -/// answers [`FLUSHED`]; the supervisor asks for the stop once it has, or once its bound -/// is spent. +/// and the kernel's records, writes them, makes the volume durable — or says +/// what of the boot it could not, in the file where the volume answers and on +/// the console, and gives up on a volume that does not answer within its own +/// bound — and answers [`FLUSHED`]; the supervisor asks for the stop once it +/// has, or once its bound is spent. pub const FLUSH: u32 = 5; pub const FLUSHED: u32 = 6; diff --git a/sdk/abi/src/log.rs b/sdk/abi/src/log.rs index dc2075b8cb0..d14fd807fa9 100644 --- a/sdk/abi/src/log.rs +++ b/sdk/abi/src/log.rs @@ -63,9 +63,7 @@ pub const RECORD_BYTES: usize = 1024; /// **Four because four have writers and readers.** The kernel writes `Info` /// (`log!`) and `Alert` (`alert!`); a program's stdout is `Info` and its stderr /// `Error`, and its own lines choose. Every rendered line names one above -/// `Info` by its [`word`](Self::word), a screen colours the line by it, and -/// `/system/bin/logkeeper` makes the volume durable at `Alert` rather than on -/// its interval. +/// `Info` by its [`word`](Self::word), and a screen colours the line by it. #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] #[repr(u8)] pub enum Severity { diff --git a/system/logkeeper/src/file.rs b/system/logkeeper/src/file.rs new file mode 100644 index 00000000000..cce0c544709 --- /dev/null +++ b/system/logkeeper/src/file.rs @@ -0,0 +1,551 @@ +//! This boot's file on `/log`: a consumer of the replay with an offset of its +//! own, on a thread of its own, as the readers and the console's carriers are. +//! +//! **The loop makes no call on `/log`.** It appends a round to the replay and +//! says so ([`File::grew`]). This thread opens the volume, writes, syncs, +//! rotates and asks a refusing volume again, so a `/log` server that takes a +//! call and never answers holds this thread and nothing else. The stop's +//! flush and mark are asks the loop waits on within a bound and can give up on. +//! +//! **The file is written in batches**: once [`BATCH_BYTES`] are owed, once +//! the oldest owed line has waited [`BATCH_AGE`], and at once for a round +//! holding a line at `Error` or above (a program's panic report is its +//! stderr, a fault the kernel reports an `Alert`) and at the stop's flush. +//! Each batch is written and synced whole. +//! +//! **What a crash costs the file** is the batch not yet written: on a volume +//! that answers, at most [`BATCH_AGE`] of lines below `Error`, which the +//! console was handed as they were read. From the stop's flush until the stop +//! is refused this thread makes no call but the stop mark, so the mark is the +//! volume's last write from this program. + +use std::sync::atomic::{AtomicU8, Ordering}; +use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender, SyncSender}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use toyos::say; +use toyos_logstream::{Severity, LOG_OPENED}; + +use crate::inspect::{Published, State}; +use crate::policy::{Retry, LOG_WRITE_BUDGET}; +use crate::serve::Hub; +use crate::store::{Volume, DIR, MAX_LOG_BYTES, MAX_LOG_PARTS}; + +/// How long a line below `Error` may wait for the file. Policy, against two +/// costs: each write and sync costs the volume's server, its driver and this +/// program a fixed amount whatever its size, and a spawn says two lines, so a +/// sync per round is that cost per spawn; a machine that dies without +/// stopping loses this much of its file. A second keeps the file a second +/// behind the console at worst and makes one write of everything a busy +/// second says. +pub const BATCH_AGE: Duration = Duration::from_secs(1); + +/// How much owed text is written whatever its age. Policy: a flood reaches +/// the file in writes of about this size rather than in one large one a +/// [`BATCH_AGE`], and at this size the fixed cost of a write is small beside +/// what it carries. +pub const BATCH_BYTES: u64 = 64 * 1024; + +/// Whether a batch is written at `now`: `owed` bytes, the oldest owed since +/// `since`, of which a line at `Error` or above makes the batch `urgent`. +fn batch_due(owed: u64, since: Option, urgent: bool, now: Instant) -> bool { + owed > 0 && (urgent || owed >= BATCH_BYTES || since.is_some_and(|since| now >= since + BATCH_AGE)) +} + +/// Whether a round's line at `severity` makes its batch written at once. +pub fn urgent(severity: Severity) -> bool { + severity >= Severity::Error +} + +/// The replay's growth the file's thread has not taken yet, one atomic shared +/// by the two: [`TOLD`] while an [`Ask::Grew`] is in the channel, so any +/// number of rounds is one ask and a file whose `/log` call never returns has +/// no more than that queued; [`URGENT`] for a line among them at `Error` or +/// above. +const TOLD: u8 = 1; +const URGENT: u8 = 2; + +enum Ask { + /// The replay grew: what by is in the shared growth. + Grew, + /// The stop: reach the replay's end and sync, stopping the catch-up at + /// `by`, and answer. + Finish { by: Instant, answer: SyncSender<()> }, + /// Write the stop mark, unless `by` has passed, and answer. + Mark { by: Instant, answer: SyncSender<()> }, + /// The stop was refused: take back any mark it asked for, and write what + /// the replay was given since at once. + Resume, +} + +/// The loop's handle on the file's thread. +pub struct File { + asks: Sender, + growth: Arc, +} + +impl File { + /// Start the file's thread, which opens this boot's file under `stem` + /// (an undated boot's where `None`) and says so with `dated`. + pub fn start(stem: Option, dated: String, hub: Arc, published: Arc) -> Self { + let (asks, asked) = mpsc::channel(); + let growth = Arc::new(AtomicU8::new(0)); + let theirs = Arc::clone(&growth); + std::thread::Builder::new() + .name("log-file".into()) + .spawn(move || { + published.publish(None, State::Opening); + let volume = open(stem, &dated); + let retry = volume.as_ref().is_some_and(|v| !v.is_open()).then(|| Retry::after(Instant::now())); + let writer = Writer { + volume, + hub, + published, + growth: theirs, + retry, + degraded: false, + owed_since: None, + urgent: false, + finished: false, + stopped: false, + marked: false, + }; + writer.run(asked); + }) + .expect("logkeeper: the file's thread could not be started"); + Self { asks, growth } + } + + /// The replay grew by a round, holding a line written at once where + /// `urgent`. + pub fn grew(&self, urgent: bool) { + let bits = if urgent { TOLD | URGENT } else { TOLD }; + if self.growth.fetch_or(bits, Ordering::AcqRel) & TOLD == 0 { + self.ask(Ask::Grew); + } + } + + /// The stop's flush, its catch-up ended at `by`: whether the file + /// answered by `until`. + pub fn finish(&self, by: Instant, until: Instant) -> bool { + let (answer, answered) = mpsc::sync_channel(1); + self.ask(Ask::Finish { by, answer }); + answered.recv_timeout(until.saturating_duration_since(Instant::now())).is_ok() + } + + /// The stop mark, written by `by`: whether the file answered by then. + /// Whether the mark is on the volume is the file thread's record. + pub fn mark(&self, by: Instant) -> bool { + let (answer, answered) = mpsc::sync_channel(1); + self.ask(Ask::Mark { by, answer }); + answered.recv_timeout(by.saturating_duration_since(Instant::now())).is_ok() + } + + /// The stop was refused. + pub fn resume(&self) { + self.ask(Ask::Resume); + } + + fn ask(&self, ask: Ask) { + self.asks.send(ask).expect("logkeeper: the file's thread ended"); + } +} + +/// This boot's file, opened, and this program's half of the startup report: +/// the kernel says whether it has a console, this whether it has a volume +/// and what its name was decided by. +fn open(stem: Option, dated: &str) -> Option { + let volume = Volume::open(DIR, stem, MAX_LOG_BYTES, |line| say!("{line}")); + match &volume { + Some(v) if v.is_open() => say!("{LOG_OPENED}{} ({dated})", v.path()), + Some(v) => say!( + "logkeeper: {} is asked again - this boot's lines are kept until {DIR} takes them ({dated})", + v.path() + ), + // `Volume::open` has said why. + None => say!("logkeeper: this boot's kernel log is on the console only ({dated})"), + } + volume +} + +struct Writer { + volume: Option, + hub: Arc, + published: Arc, + growth: Arc, + /// When a part the volume refused is next opened again. + retry: Option, + /// Whether the volume answers, slower than [`LOG_WRITE_BUDGET`] a batch. + degraded: bool, + /// When the replay first grew past the file's last batch. + owed_since: Option, + urgent: bool, + /// The stop's flush reached the file, which the mark is written over. + finished: bool, + /// From the stop's flush until it is refused: no call on `/log` but the mark. + stopped: bool, + /// The stop mark may be on the volume: this stop asked for it, whatever + /// the volume answered, and whether or not the loop still waited. + marked: bool, +} + +impl Writer { + fn run(mut self, asked: Receiver) { + loop { + self.publish(); + let ask = match self.next_due() { + Some(at) => asked.recv_timeout(at.saturating_duration_since(Instant::now())), + None => asked.recv().map_err(|_| RecvTimeoutError::Disconnected), + }; + match ask { + Ok(Ask::Grew) => self.took_growth(), + Ok(Ask::Finish { by, answer }) => { + self.finish(by); + // A loop that gave up on the answer has said so. + let _ = answer.send(()); + } + Ok(Ask::Mark { by, answer }) => { + self.mark(by); + let _ = answer.send(()); + } + Ok(Ask::Resume) => self.resume(), + Err(RecvTimeoutError::Timeout) => {} + // The loop never lets go of its handle while the program runs. + Err(RecvTimeoutError::Disconnected) => return, + } + if self.due(Instant::now()) { + self.write(); + } + } + } + + /// The replay's growth since the last [`Ask::Grew`], taken whole: a round + /// after this one asks again. + fn took_growth(&mut self) { + let urgent = self.growth.swap(0, Ordering::AcqRel) & URGENT != 0; + self.owe(urgent); + } + + /// The file is owed what the replay holds past it, at once where + /// `urgent`: the batch's age counts from now. A growth a write has taken + /// already, with the round before it, is owed nothing, so no wait starts + /// that no batch would end. + fn owe(&mut self, urgent: bool) { + if self.volume.as_ref().is_none_or(|v| self.hub.end() == v.through()) { + return; + } + self.urgent |= urgent; + self.owed_since.get_or_insert_with(Instant::now); + } + + fn publish(&self) { + let state = match &self.volume { + None => State::ConsoleOnly, + Some(v) if !v.is_open() || self.degraded => State::Degraded, + Some(_) => State::Writing, + }; + self.published.publish(self.volume.as_ref(), state); + } + + /// When a batch next falls due with no ask: the oldest owed line's age, + /// or a refused part's retry. `None` while nothing can be written. + fn next_due(&self) -> Option { + let v = self.volume.as_ref().filter(|_| !self.stopped)?; + if !v.is_open() { + return self.retry.map(|retry| retry.at()); + } + self.owed_since.map(|since| since + BATCH_AGE) + } + + fn due(&self, now: Instant) -> bool { + let Some(v) = self.volume.as_ref().filter(|_| !self.stopped) else { return false }; + if !v.is_open() { + return self.retry.is_some_and(|retry| retry.due(now)); + } + batch_due(self.hub.end() - v.through(), self.owed_since, self.urgent, now) + } + + /// Give the file what the replay owes it, a part's worth, and make it + /// durable once it has all of it; a part refused is opened again here. + fn write(&mut self) { + let began = Instant::now(); + let Some(v) = self.volume.as_mut() else { return }; + let was_open = v.is_open(); + let hub = &self.hub; + let caught = v.catch_up( + |at, max| hub.owed(at, max), + |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), + MAX_LOG_BYTES, + |line| say!("{line}"), + ); + match caught { + Ok(caught) => { + if !was_open { + self.retry = None; + say!("logkeeper: {DIR} answers again - {} holds this boot's lines", v.path()); + } + if !v.is_open() { + // A continuation's create was refused. + self.retry = Some(Retry::after(Instant::now())); + } else if caught { + (self.owed_since, self.urgent) = (None, false); + } + let took = began.elapsed(); + if took > LOG_WRITE_BUDGET && !self.degraded { + self.degraded = true; + toyos::warn!( + "logkeeper: {DIR} answers but slowly (the write took {took:?}) - degraded, nothing lost, \ + {} is still this boot's log", + v.path() + ); + } else if took <= LOG_WRITE_BUDGET && self.degraded { + self.degraded = false; + say!("logkeeper: {DIR} answers at pace again - {}", v.path()); + } + } + Err(e) if v.part() >= MAX_LOG_PARTS => { + toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); + self.volume = None; + } + Err(e) => { + v.lost(); + match self.retry.as_mut() { + Some(retry) if !was_open => retry.again(Instant::now()), + _ => { + toyos::error!( + "logkeeper: {DIR} has not answered ({e}) - {} is asked again, and this boot's lines \ + are kept until it answers", + v.path() + ); + self.retry = Some(Retry::after(Instant::now())); + } + } + } + } + } + + /// The stop: the file given the replay to its end, catching up until + /// `by`, and made durable with what it ends short by said in it. + fn finish(&mut self, by: Instant) { + self.stopped = true; + self.finished = false; + (self.owed_since, self.urgent) = (None, false); + if Instant::now() >= by { + toyos::error!("logkeeper: {DIR} took the stop's flush after its time - the file is as its last batch left it"); + return; + } + let Some(v) = self.volume.as_mut() else { return }; + let hub = &self.hub; + let finished = v.finish( + |at, max| hub.owed(at, max), + |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), + hub.end(), + MAX_LOG_BYTES, + || Instant::now() < by, + |short| { + let text = format!("logkeeper: this file ends {short} bytes short of the boot: the stop came before {DIR} took them"); + hub.own_line(Severity::Error, &text) + }, + |line| say!("{line}"), + ); + let short = hub.end() - v.through(); + match finished { + Ok(_) if short == 0 => {} + Ok(_) => toyos::error!("logkeeper: {} ends {short} bytes short of the boot at the stop", v.path()), + Err(e) => toyos::error!( + "logkeeper: {DIR} refused the stop's flush ({e}) - {} ends {short} bytes short of the boot", + v.path() + ), + } + self.finished = v.is_open(); + } + + /// The stop mark, over a file the stop's flush made durable, and before + /// `by`. Withheld, and no call made, unless the flush reached the file: + /// `/log`'s server may never have answered, and a part the volume refused + /// is not this boot's log, so the mark would call a stop clean whose log + /// is not on the volume. Once asked for, the mark is taken to be there, + /// whatever the volume answered, until a refused stop takes it back. + fn mark(&mut self, by: Instant) { + let Some(v) = self.volume.as_ref().filter(|_| self.finished) else { + toyos::warn!("logkeeper: {DIR} does not hold this boot's log, so the stop is not marked - the next boot will not count this one as clean"); + return; + }; + if Instant::now() >= by { + toyos::error!("logkeeper: the stop mark was asked of {DIR} after its time - the next boot will not count this one as clean"); + return; + } + self.marked = true; + if let Err(e) = v.mark_stopped() { + toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean"); + } + } + + /// The stop was refused: any mark it asked for is taken back, and what + /// the replay was given since is written at once. + fn resume(&mut self) { + if std::mem::take(&mut self.marked) { + let v = self.volume.as_ref().expect("a stop's volume stays until the stop is refused"); + if let Err(e) = v.unmark_stopped() { + toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); + } + } + (self.stopped, self.finished) = (false, false); + self.owe(true); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// **A batch is written at its size, at its age, or at once for a line + /// that matters**, and never for nothing. + #[test] + fn a_batch_falls_due_at_its_size_its_age_or_an_urgent_line() { + let since = Instant::now(); + let young = (since + BATCH_AGE).checked_sub(Duration::from_nanos(1)).expect("a nanosecond after `since`"); + assert!(!batch_due(BATCH_BYTES - 1, Some(since), false, young)); + assert!(batch_due(BATCH_BYTES, Some(since), false, young), "a full batch waits for no age"); + assert!(batch_due(1, Some(since), false, since + BATCH_AGE), "an old line waits no longer"); + assert!(batch_due(1, Some(since), true, since), "an urgent line waits for nothing"); + assert!(!batch_due(0, Some(since), true, since + BATCH_AGE), "nothing owed is no write"); + assert!(urgent(Severity::Error) && urgent(Severity::Alert) && !urgent(Severity::Warn)); + } + + fn writer(dir: &str) -> Writer { + let volume = Volume::open(dir, Some("2026-11-11-120000".into()), MAX_LOG_BYTES, |_| {}); + Writer { + volume, + hub: Arc::new(Hub::start(1 << 20, None)), + published: Arc::new(Published::new()), + growth: Arc::new(AtomicU8::new(0)), + retry: None, + degraded: false, + owed_since: None, + urgent: false, + finished: false, + stopped: false, + marked: false, + } + } + + fn grew(w: &mut Writer, text: &[u8], urgent: bool, at: Instant) { + w.hub.append(text, None); + w.owed_since.get_or_insert(at); + w.urgent |= urgent; + } + + fn on_disk(w: &Writer) -> Vec { + std::fs::read(w.volume.as_ref().unwrap().path()).unwrap() + } + + /// **The replay reaches the file a batch at a time**, all of it at the + /// batch's age or at once for a line that matters. + #[test] + fn the_file_takes_the_replay_a_batch_at_a_time() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let t = Instant::now(); + grew(&mut w, b"one\n", false, t); + grew(&mut w, b"two\n", false, t + BATCH_AGE / 2); + assert!(!w.due(t + BATCH_AGE / 2)); + assert!(w.due(t + BATCH_AGE), "the first line's age is the batch's"); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\n"); + assert!(w.owed_since.is_none() && w.next_due().is_none(), "a file with everything waits on nothing"); + grew(&mut w, b"panicked\n", true, t + BATCH_AGE); + assert!(w.due(t + BATCH_AGE)); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\npanicked\n"); + } + + /// **Any number of rounds is one ask in the channel** until the file's + /// thread takes it, with an urgent line among them kept: a file whose + /// `/log` call never returns has no growing queue behind it. + #[test] + fn rounds_the_file_has_not_taken_are_one_ask() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let (asks, asked) = mpsc::channel(); + let file = File { asks, growth: Arc::clone(&w.growth) }; + w.hub.append(b"line\n", None); + for round in 0..100 { + file.grew(round == 50); + } + assert_eq!(asked.try_iter().count(), 1); + w.took_growth(); + assert!(w.urgent && w.owed_since.is_some(), "the urgent round among them is kept"); + file.grew(false); + assert_eq!(asked.try_iter().count(), 1, "a round after the take asks again"); + } + + /// **From the stop's flush until it is refused the file takes no call**, + /// so the stop mark is the last write; and the mark is never asked of a + /// file the flush did not reach. A refused stop's lines are written at once. + #[test] + fn nothing_reaches_the_file_between_the_stops_flush_and_its_refusal() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let t = Instant::now(); + w.mark(t + BATCH_AGE); + assert!(!w.marked, "no flush, no mark"); + grew(&mut w, b"before\n", false, t); + w.finish(Instant::now() + BATCH_AGE); + assert!(w.finished); + assert_eq!(on_disk(&w), b"before\n"); + grew(&mut w, b"after\n", true, t); + w.volume.as_mut().unwrap().lost(); + w.retry = Some(Retry::after(t)); + assert!(!w.due(t + Duration::from_secs(3600)) && w.next_due().is_none(), "a stopped file is written nothing"); + w.resume(); + let retry = w.retry.unwrap().at(); + assert_eq!(w.next_due(), Some(retry), "a refused part is asked again at its retry"); + assert!(w.due(retry)); + w.write(); + assert_eq!(on_disk(&w), b"before\nafter\n"); + } + + /// **A growth a write has taken already starts no wait**: a round + /// appended while a write catches up is written by it, and the ask that + /// round sent, taken after, owes nothing, so the thread parks rather than + /// waking at once until the next line. A refused stop that held nothing + /// owes nothing either. + #[test] + fn a_growth_already_written_starts_no_wait() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let (asks, _asked) = mpsc::channel(); + let file = File { asks, growth: Arc::clone(&w.growth) }; + w.hub.append(b"one\n", None); + file.grew(false); + w.took_growth(); + w.hub.append(b"two\n", None); + file.grew(true); + w.write(); + assert_eq!(on_disk(&w), b"one\ntwo\n"); + w.took_growth(); + assert!(w.next_due().is_none() && !w.urgent, "a file with everything waits on nothing"); + w.finish(Instant::now() + BATCH_AGE); + w.resume(); + assert!(w.next_due().is_none(), "a refused stop that held nothing owes nothing"); + } + + /// **A refused stop takes back the mark it asked for, whatever the loop + /// heard of it**: the file's thread keeps the record, so a mark that lands + /// after the loop stopped waiting is not left for a later death to read as + /// a clean stop. + #[test] + fn a_refused_stop_takes_back_the_mark_it_asked_for() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-file"); + let mut w = writer(tmp.path().to_str().unwrap()); + let mark = tmp.join(toyos_logstream::STOPPED); + grew(&mut w, b"before\n", false, Instant::now()); + w.finish(Instant::now() + BATCH_AGE); + w.mark(Instant::now() + BATCH_AGE); + assert!(mark.exists(), "the mark goes over the flushed file"); + w.resume(); + assert!(!mark.exists(), "a refused stop leaves no mark"); + } +} diff --git a/system/logkeeper/src/inspect.rs b/system/logkeeper/src/inspect.rs index 3f185a41fae..f907cc862c3 100644 --- a/system/logkeeper/src/inspect.rs +++ b/system/logkeeper/src/inspect.rs @@ -1,9 +1,9 @@ //! The `log` port: `inspect`'s answer about where this boot's log is going, //! and a reader on this machine handed the log itself ([`READ`]). //! -//! **On a thread of its own, so no reader can reach the loop that writes the -//! file.** The loop publishes what it knows once a round — a few relaxed -//! stores, and a lock only when the file it writes changes — and this thread +//! **On a thread of its own, so no reader can reach the loop or the file.** +//! They publish what they know — a few relaxed stores, and a lock only when +//! the file changes — and this thread //! owns the acceptor, reads each reader's request without ever blocking on it, //! answers in one non-blocking write and closes. A reader that connects and //! says nothing costs a slot and [`HANDSHAKE_TIMEOUT`], never the log. @@ -40,11 +40,11 @@ const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(2); /// The poll token of the acceptor, clear of every handle's own value. const ACCEPT: u64 = u64::MAX; -/// Where the file side of the loop stands. +/// Where the file stands. #[derive(Clone, Copy, PartialEq, Eq, Debug)] #[repr(u8)] pub enum State { - /// Every round written, flushed and published. + /// Every batch written and synced as it falls due. Writing = 0, /// The volume answers slower than a log is worth, or refused the part /// and is asked again: the replay keeps what the file is owed. @@ -67,11 +67,11 @@ impl State { } } -/// The loop's side of the answer. One writer, the loop; one reader, this -/// module's thread. +/// The answer's facts: the file's, published by the file's thread, and the +/// records lost, by the loop; one reader, this module's thread. pub struct Published { state: AtomicU8, - /// The part the loop last published, or 0 for none: the loop's own record + /// The part last published, or 0 for none: the file thread's own record /// of when `file` changes, and read by nothing else. part: AtomicU32, /// The part being written and its path, one pair under one lock, so no @@ -92,24 +92,27 @@ impl Published { } } - /// What the loop knows at the top of a round. `lost` is the records the - /// kernel overwrote before this reader reached them. - pub fn publish(&self, volume: Option<&Volume>, state: State, lost: u64) { + /// Where the file stands, from the file's thread. + pub fn publish(&self, volume: Option<&Volume>, state: State) { self.state.store(state as u8, Ordering::Relaxed); - self.lost.store(lost, Ordering::Relaxed); let part = volume.map_or(0, Volume::part); self.bytes.store(volume.map_or(0, Volume::bytes), Ordering::Relaxed); - // The lock is taken when the file changes and at no other round. + // The lock is taken when the file changes and at no other batch. if self.part.swap(part, Ordering::Relaxed) != part { *self.file.lock().expect("the inspect thread does not panic holding this") = volume.map(|v| (v.part(), v.path())); } } + /// The records the kernel overwrote before the loop reached them. + pub fn publish_lost(&self, lost: u64) { + self.lost.store(lost, Ordering::Relaxed); + } + fn snapshot(&self) -> Vec { let mut snap = Snapshot::new(toyos_inspect::LOG); snap.put("volume.state", State::word(self.state.load(Ordering::Relaxed))); - let file = self.file.lock().expect("the loop does not panic holding this").clone(); + let file = self.file.lock().expect("the file's thread does not panic holding this").clone(); if let Some((part, path)) = file { snap.put("volume.path", path); snap.put("volume.part", part); diff --git a/system/logkeeper/src/main.rs b/system/logkeeper/src/main.rs index 72197c9a77f..b0c6769c2bd 100644 --- a/system/logkeeper/src/main.rs +++ b/system/logkeeper/src/main.rs @@ -40,19 +40,17 @@ //! //! # Three sinks, and only one of them is the sink of record //! -//! The file is. [`serve`]'s readers are the second: the same lines, in the same -//! order, to whoever asks, from the boot's first line however late they ask. -//! The console's carriers are the third ([`carrier`]): the console UART's -//! driver, and a VM's console channel, which the UART goes quiet for while it -//! carries the log. Neither of the two can slow the file: each is a thread -//! with an offset into the replay. The replay follows the file while the file -//! has all of it, and leads it while the part is refused or catching up. +//! The file is ([`file`]). [`serve`]'s readers are the second: the same +//! lines, in the same order, to whoever asks, from the boot's first line +//! however late they ask. The console's carriers are the third ([`carrier`]): +//! the console UART's driver, and a VM's console channel, which the UART goes +//! quiet for while it carries the log. Each of the three is a thread with an +//! offset into the replay the loop appends each round to, so none can slow +//! the loop or another. //! -//! **The volume is opened on a thread of its own**: /log's file server may -//! start late. Only the stop's flush waits for it, within [`STOP_CATCH_UP`]. -//! Until it opens, the readers and the console have each round as it is read; -//! once it opens, the file takes the boot so far from the replay, a part's -//! worth a round. +//! **The file is opened and written on its thread**: /log's file server may +//! start late, and may take a call and never answer. Only the stop waits for +//! the file, within [`STOP_FILE`]. //! //! # Its whole authority //! @@ -69,31 +67,31 @@ //! //! # Durability, which is a contract and not a hope //! -//! **A round's lines are written and the volume made durable before the next -//! round**, and again when the supervisor asks before the machine stops +//! **The file is written and made durable in batches** ([`file`]): within +//! [`file::BATCH_AGE`] of a line, at once for a line at `Error` or above, and +//! when the supervisor asks before the machine stops //! ([`toyos_logstream::FLUSH`]); once the stop has synced its files it asks -//! for the stop mark ([`toyos_logstream::MARK`]). The kernel waits on none of it, so -//! what it says after the flush, at a panic or in the stop, reaches only its -//! console and panel. +//! for the stop mark ([`toyos_logstream::MARK`]). The kernel waits on none of +//! it, so what it says after the flush, at a panic or in the stop, reaches +//! only its console and panel. //! **A program's line after that flush reaches the UART and is held back from -//! the file**, to a bound ([`STOPPING_BYTES`]): the stop syncs no file still open, -//! so the file ends where the flush made it durable, and a reader of the -//! served log is handed what the file holds and no more. A stop the kernel -//! refused is the supervisor's [`toyos_logstream::RESUME`], and what was held is written -//! then. +//! the replay**, to a bound ([`STOPPING_BYTES`]): the stop syncs no file still +//! open, so the file ends where the flush made it durable and the mark is its +//! last write. A stop the kernel refused is the supervisor's +//! [`toyos_logstream::RESUME`], and what was held is written then. //! //! **A volume that refuses, or whose server ends, is asked again** //! ([`policy::Retry`], and at the stop's flush whatever the wait), and its //! file given back what it missed ([`store::Volume`]). mod carrier; +mod file; mod inspect; mod origin; mod policy; mod serve; mod store; -use std::sync::mpsc::{self, RecvTimeoutError, TryRecvError}; use std::sync::Arc; use std::time::{Duration, Instant}; @@ -106,7 +104,7 @@ use toyos::say; use toyos::syscap::SysCap; use toyos::wake::Bell; use toyos_logstream::{ - ProgramLine, Registration, Severity, Tag, CARRIER, FLUSH, FLUSHED, LOG_OPENED, MARK, MARKED, MAX_TAG, ORIGINS, REGISTER, RESUME, + ProgramLine, Registration, Tag, CARRIER, FLUSH, FLUSHED, MARK, MARKED, MAX_TAG, ORIGINS, REGISTER, RESUME, SERVICE, WIRE, }; use toyos_wallclock::Civil; @@ -114,8 +112,8 @@ use toyos_wallclock::Civil; use carrier::Carriers; use serve::Positions; use origin::{Origin, Said}; -use policy::{Retry, Step, LOG_WRITE_BUDGET}; -use store::{Volume, DIR, MAX_LOG_BYTES, MAX_LOG_FILES, MAX_LOG_PARTS}; +use file::File; +use store::{DIR, MAX_LOG_BYTES, MAX_LOG_FILES}; /// Records asked of `SYS_LOG_READ` at once: large enough that an ordinary /// boot's burst is a handful of syscalls rather than one per line. @@ -133,8 +131,8 @@ const MAX_ORIGINS: usize = Poller::MAX_HANDLES as usize - ORIGIN_BASE as usize; /// How soon the rings are read again after a round that read any, and the /// longest the cadence backs off to while none has anything: the latency of a -/// program's line to the file and the console, against wakes on an idle -/// machine. +/// program's line to the replay, and so to the console and the readers, +/// against wakes on an idle machine. The file's is its batch's. const POLL_QUICK: Duration = Duration::from_millis(10); const POLL_SLOW: Duration = Duration::from_millis(250); @@ -157,31 +155,11 @@ fn main() { // this program nothing new. let (stem, boot_secs, dated) = boot_stamp(); - // Opened off the loop: a request on /log waits for its file server, which - // the supervisor starts only with storage. - let (opened, opening) = mpsc::sync_channel(1); - std::thread::Builder::new() - .name("log-volume".into()) - .spawn(move || { - let volume = Volume::open(DIR, stem, MAX_LOG_BYTES, |line| say!("{line}")); - match &volume { - // This program's half of the startup report, in one line: the - // kernel says whether it has a console, this program whether it - // has a volume and what the name it chose was decided by. - Some(v) if v.is_open() => say!("{LOG_OPENED}{} ({dated})", v.path()), - Some(v) => say!( - "logkeeper: {} is asked again - this boot's lines are kept until {DIR} takes them ({dated})", - v.path() - ), - // `Volume::open` has said why. - None => say!("logkeeper: this boot's kernel log is on the console only ({dated})"), - } - opened.send(volume).expect("logkeeper: the loop holds the volume's receiver for good"); - }) - .expect("logkeeper: the volume's thread could not be started"); - let hub = Arc::new(serve::Hub::start(REPLAY_BYTES, boot_secs)); let published = Arc::new(inspect::Published::new()); + // A request on /log waits for its file server, which the supervisor + // starts only with storage: the file is opened and written off the loop. + let file = File::start(stem, dated, Arc::clone(&hub), Arc::clone(&published)); if let Some(acceptor) = endow::acceptor(SERVICE) { inspect::serve(acceptor, Arc::clone(&published), Arc::clone(&hub)); } @@ -210,10 +188,7 @@ fn main() { records: vec![Record::EMPTY; BATCH], lost: 0, waiting: Vec::new(), - volume: None, - opening: Some(opening), - retry: None, - degraded: false, + file, boot_secs, hub, stopping: None, @@ -236,13 +211,7 @@ struct Log { /// Lines stamped after the round they were read in began, which the next /// round merges. waiting: Vec, - volume: Option, - /// The volume's opener, until it answers. - opening: Option>>, - /// When a part the volume refused is next opened again. - retry: Option, - /// Whether the volume answers, slower than `LOG_WRITE_BUDGET` a round. - degraded: bool, + file: File, boot_secs: Option, hub: Arc, /// The supervisor's flush was answered and the machine stops: the file's text held @@ -283,12 +252,10 @@ impl Handover { } } -/// The file's text held back after a flush, the lines past the bound, and -/// whether the flush wrote the stop mark. +/// The file's text held back after a flush, and the lines past the bound. struct Stopping { text: String, unwritten: u64, - marked: bool, } /// How long the stop's flush may spend giving the file what it is owed: half @@ -297,7 +264,12 @@ struct Stopping { /// it does not reach is said, in the file and on the console. const STOP_CATCH_UP: Duration = Duration::from_millis(toyos_quiesce::FLUSH_MS / 2); -const OPENER_ENDED: &str = "logkeeper: the volume's thread ended without answering"; +/// How long the loop waits for the file to answer the stop's flush or its +/// mark, each of which the supervisor bounds by [`toyos_quiesce::FLUSH_MS`]: +/// past [`STOP_CATCH_UP`] by the final sync's time, and short of the bound by +/// the answer's. A file that has not answered by then is given up on and +/// said, and the stop goes on. +const STOP_FILE: Duration = Duration::from_millis(toyos_quiesce::FLUSH_MS * 3 / 4); /// Text a stop may hold back from the file: a refused stop writes it, and /// what is past it is counted and said then. @@ -323,15 +295,7 @@ impl Log { // Programs whose end a watch reported, until a round has swept them. let mut ended: Vec = Vec::new(); loop { - self.adopt(); - let state = match (&self.volume, self.degraded) { - (None, _) if self.opening.is_some() => inspect::State::Opening, - (None, _) => inspect::State::ConsoleOnly, - (Some(v), _) if !v.is_open() => inspect::State::Degraded, - (Some(_), true) => inspect::State::Degraded, - (Some(_), false) => inspect::State::Writing, - }; - published.publish(self.volume.as_ref(), state, self.tail.lost()); + published.publish_lost(self.tail.lost()); // **Armed before anything is read**, in the shape every reader of // an edge needs: what arrives after a read and before the park has @@ -525,7 +489,7 @@ impl Log { lines.into_iter().partition(|line| all || line.at_ns <= cut); let through = written_through(self.tail.positions(), &later); self.waiting = later; - self.write(now, all, Arc::clone(&through)); + self.write(now, Arc::clone(&through)); self.start_wire(&through); oldest } @@ -565,30 +529,33 @@ impl Log { out } - /// The lines, in stamp order, into the file, the readers and the console's - /// carriers, which are the kernel's log through the read positions - /// `through`; while the machine stops, held back from the file, and each - /// program's line to the UART's carrier. - fn write(&mut self, mut lines: Vec, flushing: bool, through: Positions) { + /// The lines, in stamp order, into the replay the file, the readers and + /// the console's carriers take them from, which are the kernel's log + /// through the read positions `through`; while the machine stops, held + /// back from the replay, and each program's line to the UART's carrier. + fn write(&mut self, mut lines: Vec, through: Positions) { if lines.is_empty() { return; } lines.sort_by_key(|line| line.at_ns); - let mut file = String::new(); + let mut rendered = String::new(); // What the UART is owed of these lines, while the kernel's own on it // are yet to be read past. let mut wire = String::new(); let mut programs = String::new(); + let mut urgent = false; for line in &lines { match &line.kind { Kind::Kernel(record) => { + urgent |= record.severity().is_some_and(file::urgent); let text = format!("{}\n", record.dated(&stamp(self.boot_secs, record.at_ns()))); if !self.handover.as_ref().is_some_and(|handover| on_the_uart(record, &handover.spoken)) { wire.push_str(&text); } - file.push_str(&text); + rendered.push_str(&text); } Kind::Program { tag, owner, said } => { + urgent |= file::urgent(said.severity); let tag = Tag::new(tag).expect("an origin's name is a tag"); let pid = (said.pid != *owner).then_some(said.pid); let at = stamp(self.boot_secs, Some(said.at_ns)); @@ -602,7 +569,7 @@ impl Log { text: &said.text, }; let line = format!("{line}\n"); - file.push_str(&line); + rendered.push_str(&line); wire.push_str(&line); if self.stopping.is_some() { programs.push_str(&line); @@ -614,18 +581,15 @@ impl Log { // The kernel's records from the flush on are the stop's to put on // the UART; a program's reach it only through its carrier. self.hub.append_stopping(programs.as_bytes()); - if held.text.len() + file.len() > STOPPING_BYTES { + if held.text.len() + rendered.len() > STOPPING_BYTES { held.unwritten += lines.len() as u64; } else { - held.text.push_str(&file); + held.text.push_str(&rendered); } return; } - // The file first: while it has all of the replay, a reader is served - // only what /log already holds, whenever the machine stops between the - // two. A part refused or catching up takes this text from the replay. - self.to_volume(file.as_bytes(), !flushing); - self.hub.append(file.as_bytes(), Some(through)); + self.hub.append(rendered.as_bytes(), Some(through)); + self.file.grew(urgent); if let Some(handover) = &mut self.handover { handover.owed.push_str(&wire); } @@ -648,243 +612,41 @@ impl Log { self.hub.set_wire_from(self.hub.end(), handover.owed.into_bytes()); } - /// Write a round to the volume, and make it durable unless it is one of a - /// flush's rounds, which `flushed` makes durable together: a stop's flush - /// is bounded by the supervisor, and a sync per round of it is what outruns that. - fn to_volume(&mut self, text: &[u8], sync: bool) { - if !self.caught_up() { - return; - } - let Some(v) = self.volume.as_mut() else { return }; - let began = Instant::now(); - let mut refused = v.write(text).err().map(|e| (Step::Append, e.to_string())); - if refused.is_none() && sync { - refused = self.sync().err(); - } - // A volume that answered, and took longer than a log is worth doing it. - if refused.is_none() && began.elapsed() > LOG_WRITE_BUDGET { - refused = Some((Step::TooSlow, format!("it took {:?}", began.elapsed()))); - } - self.answered(refused); - } - - /// The volume, once its opener answers outside a stop. Its file stands - /// at the replay's start and is given the boot so far, a part's worth - /// here and a part's worth each round after. - fn adopt(&mut self) { - if self.stopping.is_some() { - return; - } - let Some(opening) = &self.opening else { return }; - match opening.try_recv() { - Ok(volume) => { - self.took(volume); - self.caught_up(); - } - Err(TryRecvError::Empty) => {} - Err(TryRecvError::Disconnected) => panic!("{OPENER_ENDED}"), - } - } - - /// The opener's answer. - fn took(&mut self, volume: Option) { - self.opening = None; - self.volume = volume; - } - - /// Whether the file has every byte the replay has, so a round's text - /// goes after them. A part the volume refused is opened again once the - /// retry is due, and the file is given what it is owed, a part's worth a - /// round. - fn caught_up(&mut self) -> bool { - let now = Instant::now(); - let Some(v) = self.volume.as_mut() else { return false }; - let was_open = v.is_open(); - if was_open && v.through() == self.hub.end() { - return true; - } - if !was_open && !self.retry.get_or_insert(Retry::after(now)).due(now) { - return false; - } - let hub = &self.hub; - let caught = v.catch_up( - |at, max| hub.owed(at, max), - |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), - MAX_LOG_BYTES, - |line| say!("{line}"), - ); - match caught { - Ok(caught) => { - if !was_open { - self.retry = None; - say!("logkeeper: {DIR} answers again - {} holds this boot's lines", v.path()); - } - if !v.is_open() { - // A continuation's create was refused. - self.retry = Some(Retry::after(now)); - } - caught && v.is_open() - } - Err(e) if v.part() >= MAX_LOG_PARTS => { - toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); - self.volume = None; - false - } - Err(e) => { - v.lost(); - match self.retry.as_mut() { - Some(retry) if !was_open => retry.again(now), - _ => { - toyos::error!("logkeeper: {DIR} has not answered ({e}) - {} is asked again", v.path()); - self.retry = Some(Retry::after(now)); - } - } - false - } - } - } - - /// Make the volume durable now. - fn sync(&mut self) -> Result<(), (Step, String)> { - let Some(v) = self.volume.as_mut() else { return Ok(()) }; - v.sync().map_err(|e| (Step::Flush, e.to_string())) - } - - /// What a round's write came to: rotation after a clean one, and the - /// give-up policy after a refused one. - fn answered(&mut self, refused: Option<(Step, String)>) { - let Some(path) = self.volume.as_ref().map(Volume::path) else { return }; - let Some((step, why)) = refused else { - if self.degraded { - self.degraded = false; - say!("logkeeper: {DIR} answers at pace again - {path}"); - } - self.rotate_if_full(); - return; - }; - match step { - // Say so once, keep the lines in the replay, and ask again. - Step::Append | Step::Flush => { - toyos::error!( - "logkeeper: {DIR} has not answered ({}: {why}) - {path} is asked again, and this \ - boot's lines are kept until it answers", - step.as_str() - ); - self.lost(); - } - // Every call answered, slowly: the round is durable. - Step::TooSlow => { - if !self.degraded { - self.degraded = true; - toyos::warn!( - "logkeeper: {DIR} answers but slowly ({}: {why}) - degraded, nothing lost, \ - {path} is still this boot's log", - step.as_str() - ); - } - self.rotate_if_full(); - } - } - } - - fn rotate_if_full(&mut self) { - let Some(v) = self.volume.as_mut() else { return }; - if !v.full() { - return; - } - match v.rotate(|line| say!("{line}")) { - Ok(()) if v.is_open() => {} - Ok(()) => self.lost(), - Err(e) if v.part() >= MAX_LOG_PARTS => { - toyos::error!("logkeeper: {DIR} would not take a continuation ({e}) - {}", v.path()); - self.volume = None; - } - Err(e) => { - toyos::error!("logkeeper: {DIR} has not answered ({e}) - {} is asked again", v.path()); - self.lost(); - } - } - } - - /// The volume refused the part: it is asked again once the retry is due. - fn lost(&mut self) { - if let Some(v) = self.volume.as_mut() { - v.lost(); - } - self.retry = Some(Retry::after(Instant::now())); - } - - /// The supervisor asked for the log whole before the machine stops: every line is - /// written by now, so the volume is made durable, and the supervisor is told. + /// The supervisor asked for the log whole before the machine stops: every + /// line is in the replay by now, so the file is given the rest and made + /// durable, within [`STOP_FILE`], and the supervisor is told. fn flushed(&mut self) { - let deadline = Instant::now() + STOP_CATCH_UP; - // An opener that has answered since the loop last asked, or answers - // in the stop's own time: the file takes the boot in `finish` below, - // inside the deadline. - if let Some(opening) = &self.opening { - match opening.recv_timeout(deadline.saturating_duration_since(Instant::now())) { - Ok(volume) => self.took(volume), - Err(RecvTimeoutError::Timeout) => { - toyos::error!("logkeeper: {DIR} had not opened by the stop - this boot's log is on the console only") - } - Err(RecvTimeoutError::Disconnected) => panic!("{OPENER_ENDED}"), - } - } - if let Some(v) = self.volume.as_mut() { - let hub = &self.hub; - let finished = v.finish( - |at, max| hub.owed(at, max), - |lost| hub.own_line(Severity::Warn, &format!("logkeeper: {lost} bytes of this boot were let go before {DIR} took them")), - hub.end(), - MAX_LOG_BYTES, - || Instant::now() < deadline, - |short| { - let text = format!("logkeeper: this file ends {short} bytes short of the boot: the stop came before {DIR} took them"); - hub.own_line(Severity::Error, &text) - }, - |line| say!("{line}"), + let began = Instant::now(); + if !self.file.finish(began + STOP_CATCH_UP, began + STOP_FILE) { + toyos::error!( + "logkeeper: {DIR} had not answered the stop's flush in {} ms - this boot's end may be on the console only", + STOP_FILE.as_millis() ); - let short = hub.end() - v.through(); - match finished { - Ok(_) if short == 0 => {} - Ok(_) => toyos::error!("logkeeper: {} ends {short} bytes short of the boot at the stop", v.path()), - Err(e) => toyos::error!( - "logkeeper: {DIR} refused the stop's flush ({e}) - {} ends {short} bytes short of the boot", - v.path() - ), - } } // The console's carriers take the log up to here, and the UART what // the machine says from here as it stops. if let Some(carriers) = &self.carriers { carriers.stop(self.hub.end()); } - self.stopping = Some(Stopping { text: String::new(), unwritten: 0, marked: false }); + self.stopping = Some(Stopping { text: String::new(), unwritten: 0 }); if let Err(e) = self.from_supervisor.signal(FLUSHED) { panic!("logkeeper: the supervisor could not be told the flush is done: {e:?}"); } } /// The supervisor's stop has synced its files and is about to stop its - /// drivers: write the stop mark over a durable log, and say so. The mark - /// is withheld, and no call made, unless the part is open: `/log`'s - /// server may never have answered, and a part the volume refused is not - /// this boot's log, so the mark would call a stop clean whose log is not - /// on the volume. + /// drivers: the file writes the stop mark over a durable log, within + /// [`STOP_FILE`], and the supervisor is told. fn mark(&mut self) { - let Some(stopping) = self.stopping.as_mut() else { + if self.stopping.is_none() { panic!("logkeeper: the supervisor asked for the stop mark, and no stop was flushed for"); - }; - let open = self.volume.as_ref().is_some_and(Volume::is_open); - if !open { - toyos::warn!("logkeeper: {DIR} does not hold this boot's log, so the stop is not marked - the next boot will not count this one as clean"); } - stopping.marked = open - && store::mark_stopped() - .inspect_err(|e| { - toyos::error!("logkeeper: {DIR} would not take the stop mark ({e}) - the next boot will not count this one as clean") - }) - .is_ok(); + if !self.file.mark(Instant::now() + STOP_FILE) { + toyos::error!( + "logkeeper: {DIR} had not answered the stop mark in {} ms - the stop goes on without knowing whether it is written", + STOP_FILE.as_millis() + ); + } if let Err(e) = self.from_supervisor.signal(MARKED) { panic!("logkeeper: the supervisor could not be told the stop is marked: {e:?}"); } @@ -897,11 +659,6 @@ impl Log { panic!("logkeeper: the supervisor said the machine runs on, and no stop was flushed for"); }; toyos::warn!("logkeeper: the stop was refused, so {DIR} takes this boot's lines again"); - if held.marked - && let Err(e) = store::unmark_stopped() - { - toyos::error!("logkeeper: {DIR} would not give back the stop mark ({e}) - a later death of this boot will read as clean"); - } if held.unwritten > 0 { toyos::warn!( "logkeeper: {} line(s) said while the machine stopped were past {STOPPING_BYTES} bytes \ @@ -909,8 +666,8 @@ impl Log { held.unwritten ); } - self.to_volume(held.text.as_bytes(), true); self.hub.resume(held.text.as_bytes()); + self.file.resume(); if let Some(carriers) = &self.carriers { carriers.resume(); } diff --git a/system/logkeeper/src/policy.rs b/system/logkeeper/src/policy.rs index 4f495f08ac5..4a31ee723d2 100644 --- a/system/logkeeper/src/policy.rs +++ b/system/logkeeper/src/policy.rs @@ -1,6 +1,6 @@ use std::time::{Duration, Instant}; -/// The round time past which a volume that answered every call is announced +/// The batch time past which a volume that answered every call is announced /// slow. Nothing in this program ends a volume for slowness: only the device's /// own refusal does. /// @@ -34,6 +34,10 @@ impl Retry { now >= self.at } + pub fn at(&self) -> Instant { + self.at + } + /// Asked again at `now` and refused: twice the wait, to [`RETRY_MAX`]. pub fn again(&mut self, now: Instant) { self.wait = (self.wait * 2).min(RETRY_MAX); @@ -41,29 +45,6 @@ impl Retry { } } -/// Which call refused. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum Step { - /// One of the batch's lines did not reach the file. - Append, - /// The bytes are in the file and did not reach the device. - Flush, - /// Everything answered, and the round took longer than - /// [`LOG_WRITE_BUDGET`]. - TooSlow, -} - -impl Step { - /// The word this program's console line uses for it. - pub fn as_str(self) -> &'static str { - match self { - Self::Append => "the append", - Self::Flush => "the sync", - Self::TooSlow => "the write", - } - } -} - #[cfg(test)] mod tests { use super::*; @@ -83,12 +64,4 @@ mod tests { } assert_eq!(waits, [1, 2, 4, 8, 16, 32, 64, 64, 64]); } - - /// The console words, which the table in `main` is written against. - #[test] - fn every_step_names_itself() { - assert_eq!(Step::Append.as_str(), "the append"); - assert_eq!(Step::Flush.as_str(), "the sync"); - assert_eq!(Step::TooSlow.as_str(), "the write"); - } } diff --git a/system/logkeeper/src/serve.rs b/system/logkeeper/src/serve.rs index 2be0559ecf4..3ebbca107f9 100644 --- a/system/logkeeper/src/serve.rs +++ b/system/logkeeper/src/serve.rs @@ -3,17 +3,17 @@ //! //! **Every reader gets the boot from its first line**, however late it //! connects: the main loop appends each round's lines to one -//! [`toyos_logstream::Replay`], after the file has them while the file has -//! all of the replay; a part not open yet, refused or catching up takes them -//! from the replay later. A reader is a thread with an offset into it. So the -//! same text reaches `/log` and every reader, in the same order, and nothing -//! a reader does can reach the file. +//! [`toyos_logstream::Replay`], which the file (`crate::file`) takes its +//! batches from. A reader is a thread with an offset into it. So the same +//! text reaches `/log` and every reader, in the same order, and nothing a +//! reader does can reach the file. //! //! A reader that takes none for [`STALLED`] while bytes are owed to it is let //! go, and that is a line in the log. //! -//! **A reader that is caught up waits on the replay growing**, and nothing -//! else wakes it: there is no poll and no timer. +//! **A reader that is caught up waits on the replay growing or on its reader +//! leaving**, whichever comes first: a reader that leaves while the log is +//! quiet gives back its thread and its slot then. There is no timer. //! //! The console's carriers read the same replay (`crate::carrier`): the hub //! keeps the kernel's read positions each round is written through @@ -28,7 +28,8 @@ use std::sync::{Arc, Condvar, Mutex}; use std::time::Duration; use toyos::ipc::Connection; -use toyos::poller::{Poller, WRITABLE}; +use toyos::poller::{Poller, OTHER_END_GONE, READABLE, WRITABLE}; +use toyos::wake::{self, Bell, Waker}; use toyos::{AsHandle, Pipe}; use toyos_abi::syscall::SyscallError; use toyos::say; @@ -79,6 +80,9 @@ struct Shared { replay: Mutex, grew: Condvar, readers: AtomicUsize, + /// A wake per reader, rung as the replay grows: a reader waits on its + /// own end beside it, which a condition variable cannot. + reader_wakes: Mutex>, /// The wall clock the boot started at, for a line a reader is owed. boot_secs: Option, } @@ -95,6 +99,7 @@ impl Hub { }), grew: Condvar::new(), readers: AtomicUsize::new(0), + reader_wakes: Mutex::new(Vec::new()), boot_secs, }); Self { shared } @@ -104,11 +109,12 @@ impl Hub { /// of a pipe of its own, which the log is written into. pub fn read(&self, conn: &Connection) { let Some(pipe) = self.hand_over(conn) else { return }; - admit(&self.shared, format!("reader {}", conn.as_handle().raw().0), PipeSink::new(pipe)); + admit(&self.shared, format!("reader {}", conn.as_handle().raw().0), pipe); } - /// The lines the file has just taken, for every reader, and the kernel's - /// read positions they are written through where they are a round's. + /// Lines for every reader and the console's carriers, and for the file, + /// which takes them from here in its next batch; with the kernel's read + /// positions they are written through where they are a round's. pub fn append(&self, lines: &[u8], through: Option) { let mut held = self.lock(); held.replay.append(lines); @@ -121,6 +127,7 @@ impl Hub { } drop(held); self.shared.grew.notify_all(); + self.shared.reader_wakes.lock().expect("logkeeper: a reader's wake is poisoned").retain(|w| w.ring().is_ok()); } /// The positions of the last round wholly before `at`: what a channel @@ -264,16 +271,27 @@ enum Left { } /// A reader thread, where its count allows one. -fn admit(shared: &Arc, who: String, sink: impl Write + Send + 'static) { +fn admit(shared: &Arc, who: String, pipe: Pipe) { let count = &shared.readers; if count.fetch_add(1, Ordering::SeqCst) >= MAX_READERS { count.fetch_sub(1, Ordering::SeqCst); say!("logkeeper: refusing {who}: {MAX_READERS} readers on this machine are already served"); return; } + let (waker, bell) = match wake::pair() { + Ok(ends) => ends, + Err(e) => { + count.fetch_sub(1, Ordering::SeqCst); + say!("logkeeper: no wake for {who}: {e:?}"); + return; + } + }; + // Before the thread reads the replay, so no growth after its first look + // goes unrung. + shared.reader_wakes.lock().expect("logkeeper: a reader's wake is poisoned").push(waker); let theirs = Arc::clone(shared); let spawned = std::thread::Builder::new().name("log-reader".into()).spawn(move || { - let (sent, left) = feed(&theirs, sink); + let (sent, left) = feed(&theirs, PipeSink::new(pipe, bell)); theirs.readers.fetch_sub(1, Ordering::SeqCst); match left { Left::Stalled => say!( @@ -291,11 +309,16 @@ fn admit(shared: &Arc, who: String, sink: impl Write + Send + 'static) { /// Write the boot to `sink` from its first byte, then each round as it lands, /// until the reader goes or stalls. Answers the bytes it sent and why it ended. -fn feed(shared: &Shared, mut sink: impl Write) -> (u64, Left) { +fn feed(shared: &Shared, mut sink: PipeSink) -> (u64, Left) { let mut at = 0u64; let mut sent = 0u64; loop { - let chunk = next(shared, &mut at); + let Some(chunk) = next(shared, &mut at) else { + if !sink.await_more() { + return (sent, Left::Gone); + } + continue; + }; match sink.write_all(&chunk) { Ok(()) => sent += chunk.len() as u64, Err(e) if e.kind() == std::io::ErrorKind::TimedOut => return (sent, Left::Stalled), @@ -304,23 +327,20 @@ fn feed(shared: &Shared, mut sink: impl Write) -> (u64, Left) { } } -/// The bytes after `at`, at most [`CHUNK`] of them, once there are any; `at` -/// moves past them. A reader the replay no longer holds the bytes of is -/// handed the line that says so instead. -fn next(shared: &Shared, at: &mut u64) -> Vec { - let mut held = shared.replay.lock().expect("logkeeper: the replay is poisoned"); - loop { - match held.replay.next(*at, CHUNK) { - Next::Bytes(bytes) => { - *at += bytes.len() as u64; - return bytes.to_vec(); - } - Next::Evicted { lost, at: resume } => { - *at = resume; - return evicted(shared.boot_secs, lost); - } - Next::CaughtUp => held = shared.grew.wait(held).expect("logkeeper: the replay is poisoned"), +/// The bytes after `at`, at most [`CHUNK`] of them, or `None` where there are +/// none yet; `at` moves past them. A reader the replay no longer holds the +/// bytes of is handed the line that says so instead. +fn next(shared: &Shared, at: &mut u64) -> Option> { + match shared.replay.lock().expect("logkeeper: the replay is poisoned").replay.next(*at, CHUNK) { + Next::Bytes(bytes) => { + *at += bytes.len() as u64; + Some(bytes.to_vec()) + } + Next::Evicted { lost, at: resume } => { + *at = resume; + Some(evicted(shared.boot_secs, lost)) } + Next::CaughtUp => None, } } @@ -339,33 +359,84 @@ fn own_line(boot_secs: Option, severity: Severity, text: &str) -> Vec { format!("{line}\n").into_bytes() } -/// A pipe as a byte sink: one `write` takes what fits, and a full pipe is -/// waited on for [`STALLED`] and no longer. +/// The tokens of a reader's watches: room in its pipe, the replay grown, and +/// its reader gone. +const ROOM: u64 = 0; +const MORE: u64 = 1; +const GONE: u64 = 2; + +/// A reader's pipe as a byte sink: one `write` takes what fits, and a full +/// pipe is waited on for [`STALLED`] and no longer. Its watches are one-shot, +/// so each is armed again only once it has answered. struct PipeSink { pipe: Pipe, + /// Rung as the replay grows. + more: Bell, poller: Poller, + armed: [bool; 3], + gone: bool, } impl PipeSink { - fn new(pipe: Pipe) -> Self { - Self { pipe, poller: Poller::new(1) } + fn new(pipe: Pipe, more: Bell) -> Self { + Self { pipe, more, poller: Poller::new(3), armed: [false; 3], gone: false } + } + + fn arm(&mut self, token: u64) { + if !std::mem::replace(&mut self.armed[token as usize], true) { + match token { + ROOM => self.poller.watch(&self.pipe, WRITABLE, ROOM), + MORE => self.poller.watch(&self.more, READABLE, MORE), + _ => self.poller.watch(&self.pipe, OTHER_END_GONE, GONE), + } + } + } + + /// Wait at most `timeout_nanos` for any armed watch; the tokens answered. + fn wait(&mut self, timeout_nanos: u64) -> Vec { + self.arm(GONE); + let mut answered = Vec::new(); + self.poller.wait(1, timeout_nanos, |token| answered.push(token)); + for &token in &answered { + self.armed[token as usize] = false; + self.gone |= token == GONE; + } + answered + } + + /// Wait until the replay may have grown: `false` once the reader is gone. + fn await_more(&mut self) -> bool { + loop { + if self.gone { + return false; + } + if self.more.take() { + return true; + } + self.arm(MORE); + self.wait(u64::MAX); + } } } impl Write for PipeSink { fn write(&mut self, buf: &[u8]) -> std::io::Result { + let until = std::time::Instant::now() + STALLED; loop { match self.pipe.write_nonblock(buf) { Ok(n) => return Ok(n), Err(SyscallError::WouldBlock) => {} Err(e) => return Err(std::io::Error::other(format!("{e:?}"))), } - self.poller.watch(&self.pipe, WRITABLE, 0); - let mut room = false; - self.poller.wait(1, STALLED.as_nanos() as u64, |_| room = true); - if !room { + let left = until.saturating_duration_since(std::time::Instant::now()); + if self.gone { + return Err(std::io::ErrorKind::BrokenPipe.into()); + } + if left.is_zero() { return Err(std::io::ErrorKind::TimedOut.into()); } + self.arm(ROOM); + self.wait(left.as_nanos() as u64); } } diff --git a/system/logkeeper/src/store.rs b/system/logkeeper/src/store.rs index 38bd7b0911d..f62017cfd35 100644 --- a/system/logkeeper/src/store.rs +++ b/system/logkeeper/src/store.rs @@ -54,24 +54,6 @@ pub const MAX_LOG_PARTS: u32 = 9999; /// what `/system/bin/console` reads off USB before it paints anything. pub const MAX_LOG_BYTES: u64 = 1024 * 1024; -/// Say on the volume that this boot stopped cleanly -/// ([`toyos_logstream::STOPPED`]), durably. -pub fn mark_stopped() -> std::io::Result<()> { - File::create(stopped_path())?.sync_all() -} - -/// Take back [`mark_stopped`]: the stop it was written for was refused. -pub fn unmark_stopped() -> std::io::Result<()> { - match fs::remove_file(stopped_path()) { - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), - done => done, - } -} - -fn stopped_path() -> String { - format!("{DIR}/{}", toyos_logstream::STOPPED) -} - /// Where one file of this boot's sequence is: `toyos_wallclock::Part` is its /// name. pub fn path(dir: &str, stem: &str, part: u32) -> String { @@ -264,6 +246,25 @@ impl Volume { path(&self.dir, &self.stem, self.part) } + /// Say on this volume that the boot stopped cleanly + /// ([`toyos_logstream::STOPPED`]), durably. + pub fn mark_stopped(&self) -> std::io::Result<()> { + File::create(self.stopped_path())?.sync_all() + } + + /// Take back [`Self::mark_stopped`]: the stop it was written for was + /// refused. A mark that is not there is taken back already. + pub fn unmark_stopped(&self) -> std::io::Result<()> { + match fs::remove_file(self.stopped_path()) { + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + done => done, + } + } + + fn stopped_path(&self) -> String { + format!("{}/{}", self.dir, toyos_logstream::STOPPED) + } + /// Whether the part is open: false once the volume refused it, until /// [`Self::catch_up`] opens it again. pub fn is_open(&self) -> bool { @@ -329,7 +330,8 @@ impl Volume { /// `budget` bytes of the boot this call, through part-sized writes and /// their continuations, each part synced as it fills; `note(lost)` is the /// line in place of a gap. Whether the file has every byte the replay - /// has; an `Err` is the volume refusing again. + /// has, which a file that does is synced to, whatever the budget left; + /// an `Err` is the volume refusing again. pub fn catch_up( &mut self, mut owed: impl FnMut(u64, usize) -> Owed, @@ -348,11 +350,13 @@ impl Volume { return Ok(false); } } - if written >= budget { - return Ok(false); - } - let room = (self.rotate_at - self.size).min(budget - written); + let room = (self.rotate_at - self.size).min(budget.saturating_sub(written)); match owed(self.through, room as usize) { + Owed::Nothing => { + self.sync()?; + return Ok(true); + } + _ if written >= budget => return Ok(false), Owed::Text(text) => { written += text.len() as u64; self.write(&text)?; @@ -361,10 +365,6 @@ impl Volume { self.put(¬e(lost))?; self.through = at; } - Owed::Nothing => { - self.sync()?; - return Ok(true); - } } } } @@ -460,9 +460,8 @@ mod tests { format!("[{lost} bytes let go]\n").into_bytes() } - /// One of the loop's rounds: the file given what it is owed, then the - /// round's text written and synced where it is caught up, then the - /// replay's. + /// A batch: the file given what it is owed, then the text written and + /// synced where it is caught up, then the replay's. fn round(v: &mut Volume, hub: &Hub, text: &[u8]) { let caught = (v.is_open() && v.through() == hub.end()) || v.catch_up(|at, max| hub.owed(at, max), note, MAX_LOG_BYTES, |_| {}).unwrap_or(false); @@ -543,6 +542,23 @@ mod tests { } } + /// **A catch-up whose budget runs out on the replay's last line is caught + /// up and synced**: nothing is left owed, so no later batch would come + /// for that tail, and the file thread would have no due write to wait for. + #[test] + fn a_budget_spent_on_the_replays_last_line_is_caught_up_and_synced() { + let tmp = toyos_tmpdir::TempDir::new("logkeeper-store"); + let dir = tmp.path().to_str().unwrap(); + let hub = Hub::start(1 << 16, None); + let mut v = Volume::open(dir, Some(STEM.into()), 1 << 16, |_| {}).unwrap(); + for n in 0..4 { + hub.append(&line(n), None); + } + let caught = v.catch_up(|at, max| hub.owed(at, max), note, hub.end(), |_| {}).unwrap(); + assert!(caught && v.through() == hub.end(), "the file has every byte"); + assert_eq!((v.durable, v.durable_through), (v.size, v.through), "and they are synced"); + } + /// A stop whose reopen is refused after an unsynced write: the file is /// what its last sync left, and the bytes it lacks are counted from /// there, the unsynced write's included. diff --git a/tests/guest/rust-tests/src/bin/spawn_cost.rs b/tests/guest/rust-tests/src/bin/spawn_cost.rs index 7a1a81789e6..f733a0661ca 100644 --- a/tests/guest/rust-tests/src/bin/spawn_cost.rs +++ b/tests/guest/rust-tests/src/bin/spawn_cost.rs @@ -5,6 +5,11 @@ //! runs on, `tests/images/testcases.toml`, and carries the most `RELATIVE` entries of what //! that ROOT does. //! +//! **And what it costs the machine**: the CPU every CPU was busy for over the +//! repetitions, a spawn's share, which counts what the spawn costs the +//! processes it reaches off its own path, the log among them. Work a spawn +//! leaves for later than the last repetition is not in it. +//! //! **No threshold, and the host reads the numbers instead**, as //! `syscall_cost`'s: a duration measured under TCG is no duration, and one //! measured on metal is for a same-session A/B against another build of this @@ -14,6 +19,7 @@ use std::process::{Command, Stdio}; use toyos_abi::clock::nanos_since_boot as clock_nanos; +use toyos_abi::syscall::{SysinfoHeader, SYSINFO_HEADER_SIZE}; const PROGRAM: &str = "/system/bin/toybox"; const REPS: usize = 50; @@ -32,9 +38,24 @@ fn spawn_ns() -> u64 { end - start } +/// The CPU time every CPU has been busy for since the boot. +fn busy_ns() -> u64 { + let mut header = [0u8; SYSINFO_HEADER_SIZE]; + let n = toyos::system::sysinfo(&mut header); + assert!(n >= SYSINFO_HEADER_SIZE, "sysinfo returned {n} bytes"); + SysinfoHeader::decode(&header).total_cpu_ns +} + fn main() { spawn_ns(); + let busy = busy_ns(); let mut ns: Vec = (0..REPS).map(|_| spawn_ns()).collect(); + let machine = (busy_ns() - busy) / REPS as u64; ns.sort_unstable(); - println!("spawn_cost: {PROGRAM} min {} us median {} us over {REPS}", ns[0] / 1000, ns[REPS / 2] / 1000); + println!( + "spawn_cost: {PROGRAM} min {} us median {} us over {REPS}, the machine busy {} us a spawn", + ns[0] / 1000, + ns[REPS / 2] / 1000, + machine / 1000 + ); } diff --git a/tests/guest/rust-tests/src/served_log.rs b/tests/guest/rust-tests/src/served_log.rs index 716be5f9885..5ac95f23b07 100644 --- a/tests/guest/rust-tests/src/served_log.rs +++ b/tests/guest/rust-tests/src/served_log.rs @@ -1,6 +1,5 @@ //! This boot's log as logkeeper serves it on test-runner's `log` port, from -//! its first line. A reader is handed each round only after it is on the -//! stick. +//! its first line, each round as logkeeper reads it. use std::time::{Duration, Instant}; diff --git a/tests/toyos.rs b/tests/toyos.rs index f930b853955..2ef2ad96309 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -1072,9 +1072,9 @@ const METAL: &[(&str, metal::Metal)] = &[ }, ), ( - // How long a spawn of `toybox` takes to its exit, printed by the job and - // read off the log for an A/B against another build: that it ran, and - // what it said, are the verdict. + // How long a spawn of `toybox` takes to its exit, and the machine's + // busy CPU a spawn, printed by the job and read off the log for an A/B + // against another build: that it ran, and what it said, are the verdict. "spawn_cost", metal::Metal { arms: TESTCASES,