Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions build/sysroot.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@
//! the primary's fork repository at the commit this tree pins and never moved ([`fork_checkout`]).
//! `library/std` names `toyos-abi` and `toyos` by version, and every bootstrap
//! run is handed this worktree's by one `[patch]` ([`std_patch`]); each file of
//! its ToyOS backend, `sdk/std` and the C allocator `sdk/libc/src/malloc.rs`
//! it includes, it names by a `#[path]` up out of the fork. So
//! each checkout's std compiles against its own worktree's ABI and backend with
//! nothing rewritten, and the fork names no path to either crate, so moving one
//! breaks no fork commit pinned before the move. The build is bootstrap's
//! its ToyOS backend, `sdk/std` and the allocator files of `sdk/libc/src` it
//! includes (`malloc.rs`, `mapped.rs`), it names by a `#[path]` up out of the
//! fork. So each checkout's std compiles against its own worktree's ABI and
//! backend with nothing rewritten, and the fork names no path to either crate,
//! so moving one breaks no fork commit pinned before the move. The build is bootstrap's
//! stage-0 local rebuild: the compiler the checkout names (`build/compiler.rs`)
//! compiles the checkout's `library/` for the guest targets into
//! `<checkout>/build/toyos-std/`.
Expand Down
33 changes: 33 additions & 0 deletions issues/a-refused-release-of-a-large-block-leaks-it.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
status: open
kind: defect
opened: 2026-10-27
---

# A refused release of a large block leaks it

std's and libc's allocators free a block of 256 KiB or more with `vm_release`
(`sdk/libc/src/mapped.rs`), and nothing can be done with a refusal. The block
stays mapped, committed and unreachable, so its frames and its promise are lost
until the process ends.

The release is refused in two ways:
- At `MAX_RANGES`.
- When the space's range charge cannot grow (`AddressSpace::room`, then
`Charge::resize`, then `pmm::charge`). `pmm::charge` refuses exactly when
memory is low.

Freeing a block from the middle of one inventory entry cuts it, because
adjacent large blocks and heap segments that continue each other join. `room`
then counts the two cuts. When that count crosses a `RANGE_GRAIN` above what is
already charged, the release needs a new charge.

dlmalloc keeps a segment whose release is refused and reuses it. A large block
has no such fallback.

Owner: the memory track. Exit: no refusal of a large block's release loses its
memory. One way is for a release, which leaves a space at most one range
larger, to need no new charge, because every change that adds ranges charges
for two more than it leaves. A release at `MAX_RANGES` must also give the
memory back, for example by freeing the frames (`vm_discard`) and returning
the promise.
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,10 @@ live split of a user leaf.
4. **Demand-zero anonymous `mmap`.** Built, over 4 KiB pages only: a fault
populates one page, unmapping a range no thread touched sends no IPI, and
`munmap` and a `FIXED` `mmap` take any run of pages, as POSIX does.
`vm_reserve`, `vm_protect`, `vm_discard`, `vm_decommit` and `vm_release`
are the calls, behind libc's `mmap`, `mprotect`, `madvise` and `munmap`
and std's allocator, which trims, and its thread stacks, each reserved
`vm_reserve`, `vm_protect`, `vm_discard`, `vm_decommit`, `vm_remap` and
`vm_release` are the calls, behind libc's `mmap`, `mprotect`, `madvise` and
`munmap`, std's and libc's allocators, which trim and grow a large block
by `vm_remap` in place or by moving its pages, and std's thread stacks, each reserved
at the size asked with a guard page below. A commitment is promised its
frames when it is made, and every other allocation is refused rather than
take a promised frame (`kernel::frames`), so a commitment past what memory
Expand Down
14 changes: 9 additions & 5 deletions issues/user-memory-hot-paths-are-slower-than-2-mib-pages.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@ each), what is still slower than the base:
| `mem.first_touch.4k` ns/page | 887; 508 with runs | 11 (2 MiB pages; Linux 927) |
| `mem.map_touch_unmap.4k` ns/page | 909; 532 with runs | 207 |
| `mem.anon.total` ns/MiB | 228,677; 132,877 with runs | 51,612 (Linux 258,177) |
| `mem.realloc_grow.1m_256m` ms | 116; 67 with runs | 30 (Linux 131 ns, by `mremap`) |
| `fs.tmp.seq_read.1048576` MB/s | 6,060 | 9,425 |
| `fs.tmp.seq_write.1048576` MB/s | 4,297 | 5,889 |
| `fs.tmp.readdir.1000` ms | 1.20 | 0.74 |
Expand All @@ -79,11 +78,16 @@ each), what is still slower than the base:

The copies (`fs.tmp`, `ipc`, `pipe`) point at the user-copy path above.
"With runs" is `a75abd3f6` against `b48c7dd9d` on the T14, one run each;
`6d826e313` gave 500, 521, 132,434 and 67 ms for the first four rows.
`6d826e313` gave 500, 521 and 132,434 for the first three rows.

`mem.realloc_grow` still copies: std's dlmalloc 0.2.14 never maps a block on
its own, so its `remap` hook is never called, and moving pages instead needs
large blocks mapped alone, which `mem.malloc_churn.1048576` would pay for.
`mem.realloc_grow` no longer copies. std and libc map a block of 256 KiB or
more on its own (`sdk/libc/src/mapped.rs`) and grow it by `vm_remap`, which
moves pages. On the T14 at 204243e7f, `1m_256m` took 45.9 us, its last
doubling 10.0 us. Linux, with osbench's block made to escape, took 86.5 and
20.0 us. Its earlier 131 ns came from a build that never allocated. The owner
accepted on 2026-10-27 that `mem.malloc_churn.1048576` loses heap reuse for
this (T14: 29,505 to 194 kops/s; Linux 215) until ToyOS's own allocator
brings it back.

Owner: the memory track. Exit: each path above measured on the T14 at a
change and at its base, and against Linux; a regression is fixed or its cost
Expand Down
91 changes: 91 additions & 0 deletions kernel/src/mm/space.rs
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,97 @@ impl AddressSpace {
Ok(())
}

/// Resize `[start, start + old)`, committed private memory of one
/// protection that is not executable, to `new` bytes, whole pages both
/// (`toyos_abi::syscall::vm_remap`): a shrink releases the tail, a growth
/// takes the free pages after it, or else, if `may_move`, its leaves move
/// to a new place and nothing is copied. Promised and all or nothing, as
/// [`reserve`](Self::reserve); answers where the range is now.
pub fn remap(&mut self, start: UserAddr, old: u64, new: u64, may_move: bool) -> Result<UserAddr, SyscallError> {
let (start, end) = self.held(start, old)?;
let (_, entry) = self.inventory.find(start).expect("space: a page `held` answered for");
let prot = entry.prot;
// Neighbours alike join, so one entry is the whole of one kind and protection.
if !matches!(entry.kind, Kind::Private { committed: true }) || prot == Prot::ReadExec || entry.end < end {
return Err(SyscallError::InvalidArgument);
}
if new == 0 || !new.is_multiple_of(PAGE_SIZE) || new > self.inventory.top() - self.inventory.floor() {
return Err(SyscallError::InvalidArgument);
}
if new <= old {
if new < old {
self.room(start + new, end, 0)?;
self.release(UserAddr::new(start + new), old - new);
}
return Ok(UserAddr::new(start));
}
let in_place = start.checked_add(new).is_some_and(|grown| self.inventory.holds(end, grown) && self.inventory.is_free(end, grown));
let to = if in_place {
self.room(end, start + new, 1)?;
start
} else if may_move {
self.room(start, end, 1)?;
self.inventory.place(new, PAGE_SIZE).ok_or(SyscallError::ResourceExhausted)?
} else {
return Err(SyscallError::ResourceExhausted);
};
let added = (new - old) / PAGE_SIZE;
if !pmm::promise(&self.job, added) {
return Err(SyscallError::ResourceExhausted);
}
let built = if in_place { self.build(end, start + new) } else { self.build(to, to + new) };
built.inspect_err(|_| pmm::unpromise(&self.job, added))?;
let kind = Kind::Private { committed: true };
if in_place {
self.inventory.insert(end, Entry { end: start + new, prot, kind });
} else {
let present = self.move_leaves(start, end, to);
self.inventory.remove(start, end);
self.inventory.insert(to, Entry { end: to + new, prot, kind });
let inventory = &self.inventory;
let unlinked = self.tables.unlink(start, end, |s, e| inventory.backed(s, e));
if present || !unlinked.is_empty() {
self.shoot_down(start, end, &unlinked);
}
self.tables.free(unlinked);
self.usage.taken.fetch_add(1, Relaxed);
}
self.usage.made.fetch_add(1, Relaxed);
self.settle();
Ok(UserAddr::new(to))
}

/// Move every leaf of `[from, end)` to the same place in the range from
/// `to`, whose tables are built and whose leaves are empty, leaving the
/// old ones empty: their frames stay this space's. Answers whether one
/// was present, which some CPU may still cache.
fn move_leaves(&mut self, from: u64, end: u64, to: u64) -> bool {
// A held `LeafTable` borrows all the tables, so leaves are taken out
// of the source a batch at a time and then written at the destination.
const BATCH: usize = 64;
let mut present = false;
let mut va = from;
while va < end {
let mut batch = [(0, Leaf::Empty); BATCH];
let mut n = 0;
let mut leaves = self.tables.leaf_table(va).unwrap_or_else(|| panic!("space: no tables under {va:#x}"));
while va < end && leaves.covers(va) && n < BATCH {
let leaf = leaves.get(va);
if leaf != Leaf::Empty {
leaves.set(va, Leaf::Empty);
batch[n] = (va, leaf);
n += 1;
}
va += PAGE_SIZE;
}
for &(at, leaf) in &batch[..n] {
present |= matches!(leaf, Leaf::Mapped(_));
self.tables.set(at - from + to, leaf);
}
}
present
}

/// `[start, start + len)` as pages every one of which a range holds.
fn held(&self, start: UserAddr, len: u64) -> Result<(u64, u64), SyscallError> {
let end = pages(start.raw(), len).ok_or(SyscallError::InvalidArgument)?;
Expand Down
3 changes: 2 additions & 1 deletion kernel/src/syscall/dispatch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ use super::proc::{
sys_account_info, sys_job_create, sys_job_floor, sys_job_own, sys_process_quit, sys_process_wait, sys_rt_enter, sys_spawn, sys_thread_exit, sys_thread_join, sys_thread_spawn,
};
use super::vm::{
shared_image, sys_dlopen, sys_dlsym, sys_query_modules, sys_tls_alloc_block, sys_vm_decommit, sys_vm_discard,
shared_image, sys_dlopen, sys_dlsym, sys_query_modules, sys_tls_alloc_block, sys_vm_decommit, sys_vm_discard, sys_vm_remap,
sys_vm_protect, sys_vm_release, sys_vm_reserve,
};

Expand Down Expand Up @@ -364,6 +364,7 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) ->
SYS_VM_PROTECT => sys_vm_protect(a1, a2, a3),
SYS_VM_DISCARD => sys_vm_discard(a1, a2),
SYS_VM_DECOMMIT => sys_vm_decommit(a1, a2),
SYS_VM_REMAP => sys_vm_remap(a1, a2, a3, a4),
SYS_PREAD => {
let Some(mut buf) = ctx.user_bytes_mut(UserAddr::new(a2), a3) else { return bad_addr };
with_object(RawHandle(a1 as u32), Rights::READ, |o| ops::pread(o, a4, &mut buf))
Expand Down
19 changes: 17 additions & 2 deletions kernel/src/syscall/vm.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
//! The caller's own address space: `vm_*` reserve, commit, protect, discard
//! and take away private memory, `dlopen` maps a library image, `SYS_TLS_ALLOC_BLOCK`
//! The caller's own address space: `vm_*` reserve, commit, protect, discard,
//! resize and take away private memory, `dlopen` maps a library image, `SYS_TLS_ALLOC_BLOCK`
//! maps a thread's per-module TLS block, `SYS_QUERY_MODULES` reports what is
//! loaded. Exhausting address space is an error return, never an `.expect`.
//!
Expand Down Expand Up @@ -89,6 +89,21 @@ pub(super) fn sys_vm_decommit(addr: u64, len: u64) -> u64 {
answer(range(addr, len).and_then(|(start, len)| process::current_address_space().lock().decommit(start, len)))
}

/// Resize committed private memory in place, or move it when `may_move` is 1.
pub(super) fn sys_vm_remap(addr: u64, old: u64, new: u64, may_move: u64) -> u64 {
let remapped = (|| {
let may_move = match may_move {
0 => false,
1 => true,
_ => return Err(SyscallError::InvalidArgument),
};
let (start, old) = range(addr, old)?;
let new = whole_pages(new).ok_or(SyscallError::InvalidArgument)?;
process::current_address_space().lock().remap(start, old, new, may_move)
})();
remapped.map_or_else(SyscallError::to_u64, UserAddr::raw)
}

/// The first `len` bytes of the shared memory object `handle` names, as a
/// program image; `Err` is the syscall's answer.
pub(super) fn shared_image(handle: u64, len: u64) -> Result<crate::file_backing::SharedImage, u64> {
Expand Down
24 changes: 22 additions & 2 deletions sdk/abi/src/syscall.rs
Original file line number Diff line number Diff line change
Expand Up @@ -333,20 +333,22 @@ pub const SYS_SHM_SUB: u64 = 143;
pub const SYS_JOB_FLOOR: u64 = 144;
/// What a job's account holds. See [`account_info`].
pub const SYS_ACCOUNT_INFO: u64 = 145;
/// Resize a run of committed private memory, in place or moved. See [`vm_remap`].
pub const SYS_VM_REMAP: u64 = 146;

/// Bins in the per-process syscall profile — one for every number this ABI
/// issues, and one at the end for every number it does not.
///
/// **The profile's parts sum to its total, and that is the whole requirement.**
/// A bin array narrower than the ABI reaches drops calls out of the line while
/// the total goes on counting them.
pub const SYSCALL_PROFILE_BINS: usize = 147;
pub const SYSCALL_PROFILE_BINS: usize = 148;

/// Where a number this ABI does not issue is counted. Merging is a degradation
/// a reader can see in the line; dropping is one nobody can.
pub const SYSCALL_PROFILE_OTHER: usize = SYSCALL_PROFILE_BINS - 1;

const _: () = assert!(SYS_ACCOUNT_INFO < SYSCALL_PROFILE_OTHER as u64);
const _: () = assert!(SYS_VM_REMAP < SYSCALL_PROFILE_OTHER as u64);

pub const WNOHANG: u64 = 1;
/// Every bit `SYS_PROCESS_WAIT`'s flag word defines; a wait carrying another
Expand Down Expand Up @@ -2487,6 +2489,24 @@ pub unsafe fn vm_decommit(addr: *mut u8, len: usize) -> Result<(), SyscallError>
check_unit(syscall(SYS_VM_DECOMMIT, addr as u64, len as u64, 0, 0))
}

/// Resize `[addr, addr + old_len)`, committed private memory of one
/// protection that is not executable, to `new_len` bytes, both rounded up to
/// whole pages: its pages keep their contents and a grown tail reads as
/// zeros. It grows in place where the pages after it are free; otherwise, if
/// `may_move`, its pages move to a new place, unmapped at the old, with
/// nothing copied. A shrink gives back the tail in place.
/// Answers where the range is now. Refused, it is left as it was:
/// `BadAddress` for a page no range holds, `InvalidArgument` for any other
/// memory or a zero length, and `ResourceExhausted` where it has no room.
///
/// # Safety
/// Nothing reaches the old pages after this answers a new place, nor past
/// `new_len` after a shrink.
pub unsafe fn vm_remap(addr: *mut u8, old_len: usize, new_len: usize, may_move: bool) -> Result<*mut u8, SyscallError> {
check(syscall(SYS_VM_REMAP, addr as u64, old_len as u64, new_len as u64, may_move as u64))
.map(|at| core::ptr::with_exposed_provenance_mut(at as usize))
}

/// Non-blocking read. Returns bytes read, or `Err(WouldBlock)` if no data available.
pub fn read_nonblock(handle: RawHandle, buf: &mut [u8]) -> Result<usize, SyscallError> {
check(syscall(SYS_READ_NONBLOCK, handle.0 as u64, buf.as_mut_ptr() as u64, buf.len() as u64, 0)).map(|n| n as usize)
Expand Down
20 changes: 19 additions & 1 deletion sdk/libc/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ mod uname;
mod utf8;
mod wchar;

#[cfg(not(feature = "std-runtime"))]
mod mapped;

// C runtime: the entry `arch::_start` calls, panic handler, and global allocator.
// Only for pure C programs (no Rust std). When linked into a Rust program
// with std, std provides these.
Expand Down Expand Up @@ -193,15 +196,30 @@ mod runtime {
#[global_allocator]
pub(crate) static ALLOCATOR: LibcAllocator = LibcAllocator;

use super::mapped;

unsafe impl core::alloc::GlobalAlloc for LibcAllocator {
unsafe fn alloc(&self, layout: core::alloc::Layout) -> *mut u8 {
if mapped::holds(layout) {
return mapped::alloc(layout);
}
unsafe { DLMALLOC.lock().malloc(layout.size(), layout.align()) }
}
unsafe fn alloc_zeroed(&self, layout: core::alloc::Layout) -> *mut u8 {
if mapped::holds(layout) {
return mapped::alloc(layout);
}
unsafe { DLMALLOC.lock().calloc(layout.size(), layout.align()) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: core::alloc::Layout) {
if mapped::holds(layout) {
return unsafe { mapped::dealloc(ptr, layout) };
}
unsafe { DLMALLOC.lock().free(ptr, layout.size(), layout.align()) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: core::alloc::Layout, new_size: usize) -> *mut u8 {
unsafe { DLMALLOC.lock().realloc(ptr, layout.size(), layout.align(), new_size) }
let small = || unsafe { DLMALLOC.lock().realloc(ptr, layout.size(), layout.align(), new_size) };
unsafe { mapped::realloc(self, ptr, layout, new_size, small) }
}
}
}
Loading