Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,15 @@ stays drawn whole
and the window's size waits for the file-call work (the window's record). The
rest's growth is read again after the first two.

- **Owed on metal:** #1250, a spawn that reserves its stack once and writes a
copied page once. An A/B of main before and after it on the T14 and the AMD
bench, of `spawn_cost` and `osbench procs` (`proc.spawn_wait.*`,
`proc.posix_spawn_wait.*`, `process.spawn_wait`). Main after must show
`proc.spawn_wait.tiny` and `proc.posix_spawn_wait.tiny` p50 lower by about the
stack stage it removed (7.0 µs on the T14 and 7.4 µs on the AMD bench, hot,
in the spawn diagnosis's boot 2), and no `proc` row slower beyond the before
run's spread.

**Exit**: these rows within Linux's on both benches in a labelled
current-state comparison, or each remaining difference measured to its parts
and justified.
Expand Down
60 changes: 19 additions & 41 deletions kernel/pure/frames.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
//! a free frame, or was split, which keeps 4 KiB traffic in few superframes.
//!
//! **A pinned frame is never reissued.** [`Pins`] counts, per frame and
//! without the allocator's lock, the user copies reading or writing it through
//! the direct map. A frame freed while pinned is deferred: the allocator does
//! without the allocator's lock, the kernel's copies reading or writing a
//! paged object's frame through the direct map; an address space's frame is
//! never pinned. A frame freed while pinned is deferred: the allocator does
//! not take it back until its last pin goes, and [`Pins::unpin`] says when
//! that is.
//!
Expand Down Expand Up @@ -119,15 +120,6 @@ enum Which {
Partial,
}

/// What [`Frames::discard`] did with a frame.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Discarded {
/// Freed, and its page promised a frame again.
Freed,
/// Pinned by a copy, so kept for its page.
Kept,
}

/// One account's memory: what it holds, and the floor held back for it.
/// Written only under the allocator's lock, by the [`Frames`] methods that
/// take it, and read by anyone.
Expand Down Expand Up @@ -733,22 +725,14 @@ impl<'m> Frames<'m> {
}

/// Free the frame at `phys` of a page that stays committed, promising
/// the page a frame again — or, where a user copy pins the frame, keep it
/// for the page: the caller zeroes it in place and leaves it in the
/// page's leaf, not present, for the next touch to map again.
///
/// Sound only where no pin can be taken meanwhile: a copy pins a frame
/// under the lock of the space that maps it, which the caller holds, so a
/// frame read unpinned here stays unpinned. Then every promise is backed
/// by a frame that is free now, never by one deferred to a later unpin.
pub fn discard(&mut self, phys: u64, pins: &Pins) -> Discarded {
if pins.pinned(phys) {
return Discarded::Kept;
}
/// the page a frame again, which the frame backs at once: nothing pins an
/// address space's frame, so it is free now and never deferred to a later
/// unpin.
pub fn discard(&mut self, phys: u64, pins: &Pins) {
assert!(!pins.pinned(phys), "frames: {phys:#x} discarded pinned, and nothing pins an address space's frame");
self.give_back(phys, pins);
self.repromise(1);
self.refill();
Discarded::Freed
}

/// Promise `frames` again, which a commitment still holds and whose
Expand Down Expand Up @@ -1378,7 +1362,7 @@ mod tests {

// A frame a commitment still holds, given back, is promised to it
// again; one given back by none is anyone's.
assert_eq!(m.frames.discard(drawn[0], &m.pins), Discarded::Freed);
m.frames.discard(drawn[0], &m.pins);
assert_eq!(m.frames.promised(), 1);
assert_eq!(held(&mut m.frames, &m.l), None, "a frame promised again went to another");
m.frames.free_drawn(&m.l, [drawn[1]], &m.pins);
Expand Down Expand Up @@ -1478,23 +1462,17 @@ mod tests {
assert_eq!((m.frames.charges(), m.frames.held_back()), ((0, 0), (1, 0)));
}

/// A frame a copy pins is kept by a discard rather than freed: freed, it
/// would come back only at its last unpin, and the promise made for it
/// meanwhile would be backed by no free frame, so a touch of the page, or
/// any other promised page, could find none.
/// A discarded frame is promised to its page again, backed by itself: a
/// pinned one would come back only at its last unpin, leaving the promise
/// backed by no free frame, and nothing pins an address space's frame.
#[test]
fn a_discard_keeps_a_pinned_frame_and_promises_only_what_came_back() {
#[should_panic(expected = "discarded pinned")]
fn a_pinned_frame_discarded_is_a_panic() {
let mut m = whole(1);
let all = PER_SUPER as u64;
assert!(m.frames.promise(&m.l, all));
let drawn: Vec<u64> = (0..all).map(|_| m.frames.alloc_promised()).collect();
assert!(m.pins.pin(drawn[0]));
assert_eq!(m.frames.discard(drawn[0], &m.pins), Discarded::Kept, "a pinned frame was freed");
assert_eq!(m.frames.discard(drawn[1], &m.pins), Discarded::Freed);
assert_eq!((m.frames.promised(), m.frames.free_frames()), (1, 1));
// The touch of the page whose frame went finds its frame.
assert_eq!(m.frames.alloc_promised(), drawn[1]);
assert_eq!(m.pins.unpin(drawn[0]), Unpinned::Kept, "a kept frame was deferred");
assert!(m.frames.promise(&m.l, 1));
let frame = m.frames.alloc_promised();
assert!(m.pins.pin(frame));
m.frames.discard(frame, &m.pins);
}

#[test]
Expand Down Expand Up @@ -2131,7 +2109,7 @@ mod tests {
3 if !drawn.is_empty() => {
// A discard keeps the page committed: its frame is promised again.
let f = drawn.swap_remove(next() as usize % drawn.len());
assert_eq!(m.frames.discard(f, &m.pins), Discarded::Freed);
m.frames.discard(f, &m.pins);
promised += 1;
}
4 if !drawn.is_empty() => {
Expand Down
53 changes: 53 additions & 0 deletions kernel/pure/vm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,21 @@ pub fn around(start: u64, end: u64, page: u64, populated: impl Fn(u64) -> bool)
(run.max(start), (run + AROUND).min(end))
}

/// Make `frame`, which holds what its last owner left, the page `at` bytes
/// into a range that copies `copied` bytes of a `size`-byte source from
/// `offset`: as many of the source's bytes as both go, which `read` writes
/// from the source offset it is given, and zeros after them. A page that
/// takes none, such as one wholly past the source's end, reads nothing.
pub fn copied_page(frame: &mut [u8], at: u64, offset: u64, copied: u64, size: u64, read: impl FnOnce(u64, &mut [u8])) {
let from = offset.saturating_add(at);
let n = copied.saturating_sub(at).min(frame.len() as u64).min(size.saturating_sub(from)) as usize;
let (bytes, rest) = frame.split_at_mut(n);
if n > 0 {
read(from, bytes);
}
rest.fill(0);
}

/// What a page may be used for. No variant is both writable and executable.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Prot {
Expand Down Expand Up @@ -390,6 +405,7 @@ impl<O: Object, S: Object> Inventory<O, S> {
#[cfg(test)]
mod tests {
use super::*;
use std::format;
use std::vec;
use std::vec::Vec;

Expand Down Expand Up @@ -432,6 +448,43 @@ mod tests {
}
}

/// The page `at` of a range copying `copied` bytes of a `size`-byte
/// source from `offset`, from a frame that held other bytes: what it
/// asked the source for, and the page.
fn copy_page(at: u64, offset: u64, copied: u64, size: u64) -> (Option<(u64, usize)>, Vec<u8>) {
let mut frame = vec![0xa5u8; PAGE as usize];
let mut asked = None;
copied_page(&mut frame, at, offset, copied, size, |from, bytes| {
asked = Some((from, bytes.len()));
bytes.fill(7);
});
(asked, frame)
}

/// A copied page holds the source's bytes as far as the range's copy and
/// the source both go, and zeros after them, whatever the frame held; a
/// page that takes none asks the source for nothing.
#[test]
fn a_copied_page_keeps_nothing_its_frame_held() {
for (at, offset, copied, size, n) in [
(PAGE, 0, 3 * PAGE, 4 * PAGE, PAGE),
(0, PAGE, 100, 2 * PAGE, 100),
(PAGE, 0, PAGE + 50, PAGE + 50, 50),
(PAGE, 0, 100, 4 * PAGE, 0),
(0, 0, 0, PAGE, 0),
// A range longer than its object: a page wholly past its end,
// where a read would be past the object, which no read may be.
(2 * PAGE, 0, 100, PAGE, 0),
] {
let (asked, frame) = copy_page(at, offset, copied, size);
let n = n as usize;
let what = format!("page {at:#x} of {copied:#x} bytes copied from {offset:#x} of {size:#x}");
assert_eq!(asked, (n > 0).then_some((offset + at, n)), "{what}: what it read");
assert!(frame[..n].iter().all(|&b| b == 7), "{what}: its {n} bytes");
assert!(frame[n..].iter().all(|&b| b == 0), "{what}: zeros past {n} bytes");
}
}

/// An object by number.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
struct Obj(u32);
Expand Down
31 changes: 0 additions & 31 deletions kernel/src/arch/aarch64/paging.rs
Original file line number Diff line number Diff line change
Expand Up @@ -716,37 +716,6 @@ impl UserTables {
published();
}

/// [`take_owned`](Self::take_owned), where `decide` says of each frame of
/// a table's batch, with the address its leaf maps, whether the leaf
/// keeps it: a kept frame's leaf stays, every other is emptied.
pub fn discard_owned(&mut self, start: u64, end: u64, mut decide: impl FnMut(&[u64], &[u64], &mut [bool])) {
self.each_table(start, end, |base, table| {
let mut frames = [0u64; 512];
let mut vas = [0u64; 512];
let mut slot = [0usize; 512];
let mut n = 0;
for i in slots(start, end, base, 12) {
if let Leaf::Held(phys) | Leaf::Mapped(Mapping { phys, owned: true, .. }) = Leaf::decode(table.0[i]) {
frames[n] = phys;
vas[n] = base + ((i as u64) << 12);
slot[n] = i;
n += 1;
}
}
if n == 0 {
return;
}
let mut kept = [false; 512];
decide(&frames[..n], &vas[..n], &mut kept[..n]);
for k in 0..n {
if !kept[k] {
table.set(slot[k], Leaf::Empty.encode());
}
}
});
published();
}

/// Empty every leaf of `[start, end)` holding a frame the space owns, one
/// table of leaves at a time, handing each table's frames to `taken`: at
/// most 512 at once, in a buffer of its own, so nothing is allocated.
Expand Down
30 changes: 0 additions & 30 deletions kernel/src/arch/x86_64/paging.rs
Original file line number Diff line number Diff line change
Expand Up @@ -966,36 +966,6 @@ impl UserTables {
});
}

/// [`take_owned`](Self::take_owned), where `decide` says of each frame of
/// a table's batch, with the address its leaf maps, whether the leaf
/// keeps it: a kept frame's leaf stays, every other is emptied.
pub fn discard_owned(&mut self, start: u64, end: u64, mut decide: impl FnMut(&[u64], &[u64], &mut [bool])) {
self.each_table(start, end, |base, table| {
let mut frames = [0u64; 512];
let mut vas = [0u64; 512];
let mut slot = [0usize; 512];
let mut n = 0;
for i in slots(start, end, base, 12) {
if let Leaf::Held(phys) | Leaf::Mapped(Mapping { phys, owned: true, .. }) = Leaf::decode(table.0[i]) {
frames[n] = phys;
vas[n] = base + ((i as u64) << 12);
slot[n] = i;
n += 1;
}
}
if n == 0 {
return;
}
let mut kept = [false; 512];
decide(&frames[..n], &vas[..n], &mut kept[..n]);
for k in 0..n {
if !kept[k] {
table.store(slot[k], Leaf::Empty.encode());
}
}
});
}

/// Empty every leaf of `[start, end)` holding a frame the space owns, one
/// table of leaves at a time, handing each table's frames to `taken`: at
/// most 512 at once, in a buffer of its own, so nothing is allocated.
Expand Down
5 changes: 1 addition & 4 deletions kernel/src/loader/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -186,16 +186,13 @@ pub fn spawn<C: Commit>(
return Err(SyscallError::ResourceExhausted.into());
};
let (stack_base, sp) = {
use crate::mm::space::At;
use kernel::vm::Prot;
let mut space = child_pt.lock();
crate::clock::map_page(&mut space);
lay(&mut space, &maps)?;
// Never executable, and a reserved guard below it, so running off the
// stack's bottom faults instead of writing what lies there.
let stack = space.reserve(STACK_GUARD + USER_STACK_SIZE as u64, At::ANYWHERE, Prot::None)?;
let stack_base = stack + STACK_GUARD;
space.reserve(USER_STACK_SIZE as u64, At::Fixed(stack_base), Prot::ReadWrite)?;
let stack_base = space.reserve_above_guard(STACK_GUARD, USER_STACK_SIZE as u64, Prot::ReadWrite)?;
(stack_base, stack_base.raw() + USER_STACK_SIZE as u64)
};

Expand Down
30 changes: 18 additions & 12 deletions kernel/src/mm/pmm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ use core::sync::atomic::{AtomicPtr, AtomicU32, AtomicU64, AtomicUsize, Ordering}

use kernel::frames::{self, Frames, Pins, PoolSplit, Super, Unpinned, FRAME, SUPER};
use kernel::sched::job::Job;
pub use kernel::frames::Discarded;

use super::{DirectMap, PAGE_2M};
use crate::sync::Lock;
Expand Down Expand Up @@ -134,8 +133,8 @@ pub fn release_floor(job: &Job) {
/// under one hold of the allocator's lock.
///
/// # Safety
/// Each is a frame [`Frame::into_raw`] gave up, that no leaf on any CPU maps
/// any more and nothing else frees.
/// Each is a frame drawn for a committed page of `job`'s, which no leaf on
/// any CPU maps any more and nothing else frees.
pub unsafe fn free_drawn(job: &Job, frames: &[u64]) {
let pins = pins();
with(|f| f.free_drawn(job.memory(), frames.iter().copied(), &pins));
Expand Down Expand Up @@ -262,8 +261,9 @@ fn kept_supers(frames: &Frames) -> u64 {
}

/// Promise `frames` to a commitment of `job`'s, which draws each with
/// [`alloc_promised`] when it is first touched; `false`, and nothing
/// promised, where that would take what is held back.
/// [`alloc_promised`] or [`alloc_promised_unzeroed`] when it is first
/// touched; `false`, and nothing promised, where that would take what is
/// held back.
pub fn promise(job: &Job, frames: u64) -> bool {
frames == 0 || with(|f| f.promise(job.memory(), frames))
}
Expand All @@ -288,15 +288,14 @@ pub fn unpromise(job: &Job, frames: u64) {
}
}

/// [`Frames::discard`] of `phys`, a committed page's frame the caller holds
/// the lock of the mapping space for.
/// Give back `phys`, the frame of a page that stays committed, and promise
/// the page a frame again ([`Frames::discard`]).
///
/// # Safety
/// `phys` is an owned leaf's frame, which no leaf on any CPU maps any more,
/// unless this answers [`Discarded::Kept`].
pub unsafe fn discard(phys: u64) -> Discarded {
/// `phys` is an owned leaf's frame, which no leaf on any CPU maps any more.
pub unsafe fn discard(phys: u64) {
let pins = pins();
with(|f| f.discard(phys, &pins))
with(|f| f.discard(phys, &pins));
}

/// A zeroed frame a promise held, which the promise guarantees is free.
Expand All @@ -307,6 +306,13 @@ pub fn alloc_promised() -> Frame {
frame
}

/// [`alloc_promised`]'s frame holding what its last owner left, by its
/// address: its caller writes every byte of it before anything maps it, and
/// gives it back through [`free_drawn`] or [`discard`].
pub(super) fn alloc_promised_unzeroed() -> u64 {
with(Frames::alloc_promised)
}

/// A zeroed frame for a page table of `job`'s, refused rather than take what
/// is held back: a process's tables are not promised. It goes back through
/// [`free_held`].
Expand Down Expand Up @@ -445,7 +451,7 @@ pub fn alloc_frame() -> Option<Frame> {
Some(frame)
}

/// Pin every managed frame `[phys, phys + len)` touches so none is reissued while a user-copy window covers it; `false`, and nothing pinned, on a pin-count overflow.
/// Pin every managed frame `[phys, phys + len)` touches so none is reissued while a copy reads or writes it; `false`, and nothing pinned, on a pin-count overflow.
pub fn pin_range(phys: u64, len: usize) -> bool {
let pins = pins();
let frames = frame_span(phys, len);
Expand Down
Loading