diff --git a/issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md b/issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md new file mode 100644 index 00000000000..79a78861672 --- /dev/null +++ b/issues/a-large-spawn-on-the-amd-bench-is-a-seventh-slower-than-mains.md @@ -0,0 +1,38 @@ +--- +status: open +kind: defect +opened: 2026-11-05 +--- + +# A large spawn on the AMD bench is a seventh slower than main's + +`osbench`'s `proc.spawn_wait.large` spawns a 32 MiB program from a served +file. The spawner reads the image through `SYS_PAGED_READ` into fresh shared +memory. #1150 copies it a frame at a time through the reader's own mapping +(`arch::uaccess`), after faulting each chunk of 16 frames in +(`PagerObject::read`, `UserBytesMut::fault_in`). Main copied it through the +kernel's direct map, reaching and pinning each page as it went. + +On the AMD bench (metal round 8; evidence `ToyOSOrg/evidence`, branch +`pr-1150`, r15): +- **The row:** 12.79 and 12.75 ms against main's 11.00 and 11.53, worse in + both pairs. +- **The copy:** 760 ns per page against 685 for main's whole reach, pin and + copy; with 8192 pages a spawn, that is 0.6 ms. +- **The populate:** `fault_in` adds 48 ns per page in a pass of its own, + 0.4 ms a spawn. +- **Unaccounted:** about 0.5 ms of the 1.5 ms. + +Before #1150's accessed-and-dirty change to x86-64 user leaves the copy cost +1871 ns per page and the row was 96% worse. On the T14 the row is now 4% +better than main's. + +**Hypothesis, unmeasured:** the remaining copy cost is translating each +destination page through the 4 KiB user mapping, where main's direct map uses +huge pages. The separate populate pass touches each page's leaf a second time. +No performance counter has been read. + +**Exit:** the row within main's run-to-run range on the AMD bench, or the +per-page difference measured by counter (TLB misses and walk cycles, against +main) and recorded here as the cost of copying through the user mapping that +#1150's design accepts. diff --git a/issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md b/issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md index fbe4c32682c..d476f8338ca 100644 --- a/issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md +++ b/issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md @@ -4,55 +4,33 @@ kind: defect opened: 2026-08-28 --- -# A pipe's lock spans a 2 MiB user copy and the frames it takes for it - -Each pipe owns its state behind its own `sync::Lock` (`kernel/src/pipe.rs`), -which disables preemption before it spins (`kernel/src/sync.rs`). The bulk copy -runs inside that lock, through the frame store pipes and connections share -(`kernel/src/frame_bytes.rs`): - -* `FrameBytes::read` makes one walk of the caller's window, - `buf.write_pieces`, and copies each pinned piece out of the frames it - covers with `bytes.peek`, under it; -* `FrameBytes::write` makes one walk, `buf.read_pieces`, and copies each piece - into its frames with `bytes.place`, in the same acquisition. - -Each 16 pages of either copy take the caller's space lock and populate any -untouched page, a frame from the allocator and a 4 KiB zeroing. - -A pipe's copy is bounded only by the pipe: `PIPE_CAPACITY` is 2 MiB, the -largest single copy under the lock, and nothing above caps it -(`SYS_READ`/`SYS_WRITE` pass the caller's length through). - -A connection direction (`kernel/src/message.rs`) copies one whole message under -its queue's lock, so its copy is bounded at `MAX_MESSAGE_LEN`, 16 KiB, and its -frames are taken the same way. Both its send and its receive run that copy -inside `process::with_process_data`, nesting the queue's lock in the process -lock: the receive installs the message's handles in the caller's table and -copies its bytes in one hold, so a refusal takes nothing. - -A write that grows the pipe also takes its frames under the same lock, -nesting the frame allocator's lock: up to 513 frames, unzeroed, under one -hold, and the heap the list of them grows into. A read that empties frames -past the spares kept gives them back under the same nesting. - -What queues behind it is that one pipe's traffic: its other end's reads and -writes, `has_data`/`has_space` from its blocking wrappers and polls, and its -handles' clones and closes. The scheduler's own bar for an uninterruptible -stretch is `MAX_PASS_NS = 200_000` ns (`kernel/pure/sched/cpu.rs`); a 2 MiB copy -is past it at any plausible bandwidth. The hold time is not measured, and that -is owed first. The length is the caller's. - -## Fix direction - -The lock is also the serialization `kernel::pipe::Bytes` relies on, so the -copy cannot simply be lifted out: a claimed-range token per end, or a copy -bounded per acquisition, has to carry the data-race argument. The frames can -be taken before the lock and given back after it, as the read end's last -holder already gives back what the pipe held. - -**Exit**: no pipe or queue lock is held across more than a bounded copy or any -allocation, the connection's nesting in the process lock included, the hold -time is read before and after, and `audio_tone`'s worst wake does not move. +# A pipe's or queue's lock hold is unmeasured, and a queue still grows under it + +A pipe (`kernel/src/pipe.rs`) and a connection direction +(`kernel/src/message.rs`) each keep their state behind a `sync::Lock`, which +disables preemption before it spins (`kernel/src/sync.rs`). No user copy and no +frame allocation runs under either any more: a read or a write claims its bytes +under the lock, copies them and takes or gives back frames with it let go, and +ends its claim under it (`kernel/src/frame_bytes.rs`, `kernel::pipe::Bytes`). +A hold is a claim's arithmetic, one chunk's runs (at most 17), a list moved +into a longer one (at most 513 words), or a batch of 32 spare frames listed. + +What is left: + +* **The hold time has not been read**, before or after. Before, a pipe's lock + spanned up to a 2 MiB copy and 513 frames from the allocator; the scheduler's + bar for an uninterruptible stretch is `MAX_PASS_NS = 200_000` ns + (`kernel/pure/sched/cpu.rs`). The bound above is by inspection. QEMU's clock + cannot read it; a bench can. +* **A connection's queue grows its two lists under the lock.** + `kernel::message::Queue` keeps its heads and its handles in `VecDeque`s that + start empty and double as `push` fills them, a heap allocation under the + queue's lock, up to `MAX_QUEUED_MESSAGES` and `MAX_QUEUED_HANDLES` entries. + Their bound is charged when the connection is made, so reserving it then is + the direct fix, at that memory per direction whether or not it is used. +* **`audio_tone`'s worst wake** has not been read against the change. + +**Exit**: the hold time read on a bench before and after, the queue's lists +grown outside its lock or reserved, and `audio_tone`'s worst wake unmoved. This is stage S0-a of `issues/anonymous-memory-swaps-to-an-encrypted-partition.md`. diff --git a/issues/a-user-copy-demand-pages-under-whatever-its-caller-holds.md b/issues/a-user-copy-demand-pages-under-whatever-its-caller-holds.md deleted file mode 100644 index df3ea953c38..00000000000 --- a/issues/a-user-copy-demand-pages-under-whatever-its-caller-holds.md +++ /dev/null @@ -1,29 +0,0 @@ ---- -status: open -kind: finding -opened: 2026-10-04 ---- - -# A user copy demand-pages under whatever its caller holds - -`user_ptr::translate_user` (`kernel/src/user_ptr.rs`) calls -`process::handle_page_fault` directly when a user address is not mapped, and -every user copy faults its windows in through it (`fault_in`). -`handle_page_fault` takes `PROCESS_TABLE`, the current process's -`process_data` and its address-space lock, all three a `sync::Lock`. A syscall -that copies user memory while holding any of the three re-enters that ticket -lock on the first unmapped window: it spins, and the kernel panics -"DEADLOCK at …" (`Lock::lock`, `kernel/src/sync.rs`). Nothing in the tree -refuses that order: no lock-order check covers these locks. - -The global VFS lock (`vfs::lock()`) is a fourth nothing orders. `ops::open` -takes it under `process_data`, and `sys_chdir` took `process_data` under it -until its guard was made to end before the store; that no holder of the VFS -guard takes `process_data` is held by reading alone. A checked lock level -ranks it with the three. - -**Exit:** the order is refused by something that fails (a type or a -checked lock level), or every caller is shown to hold none of the three and -the invariant is one line in `user_ptr.rs`'s header. - -This is stage S0-a of `issues/anonymous-memory-swaps-to-an-encrypted-partition.md`. diff --git a/issues/anonymous-memory-swaps-to-an-encrypted-partition.md b/issues/anonymous-memory-swaps-to-an-encrypted-partition.md index c7103f92d2a..b31249dd009 100644 --- a/issues/anonymous-memory-swaps-to-an-encrypted-partition.md +++ b/issues/anonymous-memory-swaps-to-an-encrypted-partition.md @@ -319,7 +319,7 @@ Each check goes red under the mutation named after "red". | Stage | After | Builds | Exit | |---|---|---|---| -| S0-a (`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`'s owner) | — | Every user copy faults or swaps in its chunk before any kernel lock, and pins after a re-check, with no pin across a wait. A lost anonymous page ends the caller, and a lost file-mapping page answers `EIO`. The futex re-translation never waits | Closes `issues/a-user-copy-demand-pages-under-whatever-its-caller-holds.md` and `issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md`. **H:** a chunk model in which an anonymous loss ends the caller with no partial result and a file-mapping loss answers `EIO`, red if either takes the other's outcome. **Q:** the pipe and futex sites, red when a copy runs under the pipe lock | +| S0-a (`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`'s owner) | — | Every user copy faults or swaps in its chunk before any kernel lock, and pins after a re-check, with no pin across a wait. A lost anonymous page ends the caller, and a lost file-mapping page answers `EIO`. The futex re-translation never waits | Closes `issues/a-pipes-lock-spans-a-user-copy-and-its-first-allocation.md`. One half is built: no user copy runs under a spin lock, and one that would is refused by name (`user_ptr`). By the orchestrator's ruling of 2026-11-03, the rest of this row is built as part of the process-memory track's 7b; the kernel has had no sleeping lock for the gate to extend to since #1152. **H:** a chunk model in which an anonymous loss ends the caller with no partial result and a file-mapping loss answers `EIO`, red if either takes the other's outcome. **Q:** the pipe and futex sites, red when a copy runs under the pipe lock | | S0-b (`issues/storage-is-layers-and-a-role-is-a-filesystem.md`'s owner) | — | The block-session client and driver side allocate nothing per I/O: tables sized by `DEPTH`, reads opened straight from the arena. A reopen refused `Held` by a closing session waits for its retirement | **H:** `DEPTH` traffic with an allocator that fails after warm-up, red on any per-I/O allocation; a reopen during a dead session's close answered only after its last completion, red if the hold is released early | | P1 (`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`'s owner) | prerequisites | Both rules and the frame rule in `kernel/pure/frames.rs`; residency by charged job; `CHARGE`; `vm_wire`, the per-job allowance and additive charges; `resident` and `restart-hold`; the main stack sized by `PT_GNU_STACK` (8 MiB otherwise); resident by kind; unmovable frames in their own superframes; the ledger figures; `S = 0` | **H:** both rules at every `W` and `A` increase, red when a table build skips the backing rule; a resident fault never waits, red when a swappable fault may take a held frame; `SHM_FOR_JOB` to a resident job gives `W` and to an ordinary one `A`; wiring refused past the allowance; a helper shares its job's allowance; an additive wire, red if the charged job pays; two wirers where the first exits, red if any release can be refused or leaves a rule exceeded; superframe invariants under wire and lend; a hold kept whole. **Q:** a `CHARGE`-only handle creates an object `SHM_FOR_JOB` and is refused as a spawn's job and at `SYS_JOB_FLOOR`, red if either refusal accepts it; `mmap_stress`; each resident row's committed and populated totals reported | | P2 (`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`'s owner) | P1 | Reverse records and the RAM-object reverse index; `mem_scan`; `mem_reclaim` for clean pages and never-writable copied pages; `mem_read`/`mem_write` and `vm_wire_ticket` on the ticket table; watermarks; the kept fallback; superframe evacuation; the `reclaimer` role; service v0 (plain CLOCK) | **H:** revalidation, floor protection, revoke order, an evacuation case; `Pending` without waiting, every non-resident page requested at once, red if a call waits or requests fewer; an unmap cancels a parked ticket and removes the wirer's charge without waiting, red if the unmap waits or the charge stays; `Lost` removes the wirer's charge, red if it stays; a blocking wire of another job refused, red if accepted; a dropped copied page refilled byte-exact, red if a once-`ReadWrite` or resident range is dropped; a version not purged under a copy range, red if the range is not counted; the fallback with a stuck service; a remap between scan and reclaim. **Q:** a page-cache-heavy run with the controls usable | diff --git a/issues/every-descriptor-call-takes-one-process-wide-lock-twice.md b/issues/every-descriptor-call-takes-one-process-wide-lock-twice.md new file mode 100644 index 00000000000..ab71522a6c3 --- /dev/null +++ b/issues/every-descriptor-call-takes-one-process-wide-lock-twice.md @@ -0,0 +1,27 @@ +--- +status: open +kind: defect +opened: 2026-11-04 +--- + +# Every descriptor call takes one process-wide lock twice + +A `read` or `write` through a descriptor (`sdk/std/sys/pal/mod.rs`'s `on_fd`) +holds its object for the call through `toyos::fd::hold`. That takes the +process's one descriptor-table lock (`sdk/toyos/src/fd.rs`, `TABLE`) to acquire +the slot and again to let it go. Two threads of one process doing I/O on +different descriptors meet on that lock twice per call. + +The lock was a plain futex lock, so any collision slept in `FUTEX_WAIT` and +woke through `FUTEX_WAKE`. In the S0-a metal A/B (#1150, round 2), the pipe +benchmark's writer and reader threads (`osbench` `pipe.throughput.*`) made +about 125-150 futex waits a run on main. With the kernel no longer +serializing their pipe calls under its process lock, they made 128,000-153,000 +on the T14. That was much of the small-pipe-write regression measured there. + +`toyos::sync::lock_word` now spins briefly before sleeping, which turns a +collision into a short wait. Spinning bounds the cost of a collision; it does +not stop two threads meeting on a lock neither needs. + +**Exit**: a descriptor's hold for one call takes no lock that a call on another +descriptor takes. diff --git a/issues/osbench-s-malloc-churn-on-the-t14-moves-a-fifth-with-the-allocator-s-code-placement.md b/issues/osbench-s-malloc-churn-on-the-t14-moves-a-fifth-with-the-allocator-s-code-placement.md new file mode 100644 index 00000000000..259ad9f9edf --- /dev/null +++ b/issues/osbench-s-malloc-churn-on-the-t14-moves-a-fifth-with-the-allocator-s-code-placement.md @@ -0,0 +1,35 @@ +--- +status: open +kind: tooling +opened: 2026-11-05 +--- + +# osbench's malloc churn on the T14 moves a fifth with the allocator's code placement + +`mem.malloc_churn.16.t1` and `.4096.t1` lost 18-21% on the T14 when #1150 +grew `toyos::sync::lock_word`, which std's allocator never calls: its lock is +std's futex `Mutex`, which goes straight to `toyos_abi::syscall` +(`sdk/std/sys/pal/futex.rs`). +- **The shift:** the growth moved std's allocator code 0x720 bytes, which puts + dlmalloc's `malloc` and `free`, `__rdl_alloc`, `__rdl_dealloc`, std's + `dealloc` and `Mutex::lock_contended` 32 bytes off their earlier position + modulo 64. +- **The proof:** 32 inert bytes beside `lock_word` put them back. That arm, + which keeps the grown lock, reads base's values: 36514 and 36658 kops/s + against base's 36586, and 28800 for the grown lock without the bytes + (evidence `ToyOSOrg/evidence`, branch `pr-1150`, r13 for the symbol tables + and r14 for the boots). + +The AMD bench does not show it. So on this core a hot loop's throughput +depends on where an unrelated change leaves the allocator, and a benchmark +comparison across any change that resizes earlier code can report a fifth +of noise as a regression or a gain. + +The mechanism inside the core is not established: which front-end structure +(the decoded-µop cache's 32-byte windows, or a branch boundary) the +placement crosses. No performance counter has been read. + +**Exit:** the churn rows hold within run-to-run under a placement change: the +hot allocator functions aligned so that a shift of earlier code moves them by +whole aligned units, or osbench's comparison reports placement as its own +arm. Either is measured on the T14 with a padding arm like this one. diff --git a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md index ed5e8e71bb0..ef870a3f044 100644 --- a/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md +++ b/issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md @@ -108,4 +108,19 @@ are split in two. **7a**, delivered by the storage track's file-server work: a pager's objects, read through `SYS_PAGED_READ` and never mapped (`kernel/src/object/paged.rs`). **7b** stays this track's: mappings of them, faults that park, their page tables and reverse records, holders' kernel -writes and their admission, after stage 4's seal. +writes and their admission. **7b comes before stage 4**, by the +orchestrator's ruling of 2026-11-01 on the owner's delegation: file mappings +are on the critical path for file parity with Linux and for the storage +track's name cache, and stage 4's seal layers onto mappings rather than the +reverse. By the orchestrator's ruling of 2026-11-03, 7b needs two things: +the swap track's S0-a half that is met, every user copy outside every spin +lock (a fault that parks on a pager can hold none, and `user_ptr` refuses a +copy made under one); and, after the storage consolidation (#1152) lands, the +same gate extended to sleeping locks, so a user copy under any kernel lock is +refused. #1152 deleted the kernel's sleeping lock (`kernel/src/sleeplock.rs`), +so the spin-lock gate is that gate and both are met. The rest of S0-a's row +(a chunk faulted in before any lock and pinned after a re-check, the lost-page +outcomes, its H model) is built as part of 7b. One outcome 7b must place: a pipe read or message receive of at most +`frame_bytes::BOUNCE` bytes lets its bytes go, and a receive installs its +handles, before copying out to the caller. A lost page met in that copy must +not lose them. Faulting the destination in before the take is one way. diff --git a/issues/toyos-runs-on-arm64.md b/issues/toyos-runs-on-arm64.md index f1250d7cdb3..ce00e01f6e9 100644 --- a/issues/toyos-runs-on-arm64.md +++ b/issues/toyos-runs-on-arm64.md @@ -444,3 +444,14 @@ Linux guest. The timing is one measurement, taken by hand and not automated (`issues/process-memory-in-4-kib-pages-still-copies-each-programs-text.md`): 2 MiB L2 blocks map identically on both arches, so no change is forced by this track. + +## Unmeasured until an AArch64 bench + +- **User-copy throughput.** Since #1150, every user copy on AArch64 goes + through the unprivileged `LDTR`/`STTR` routines of `arch::uaccess`: 8 bytes + an access, where it went through the direct map's `memcpy` before. Their + correctness is checked under the guest runner (`virt_user_copy_faults`); + their cost against the old copy is not measured anywhere, as no AArch64 bench + exists and a guest run is no verdict on it. **Exit**: an AArch64 bench's + pipe and IPC rows against the commit before #1150, interleaved, with no row + worse. diff --git a/issues/user-pointer-checks-have-no-spectre-v1-fence.md b/issues/user-pointer-checks-have-no-spectre-v1-fence.md index ade879c63ad..bdfd0b3ba61 100644 --- a/issues/user-pointer-checks-have-no-spectre-v1-fence.md +++ b/issues/user-pointer-checks-have-no-spectre-v1-fence.md @@ -7,10 +7,15 @@ opened: 2026-09-29 # User-pointer checks have no Spectre v1 fence -`user_ptr`'s range checks carry no `lfence` and no index mask, so a -mispredicted check can load through a user pointer. Linux at -`Ubuntu-6.8.0-142.142` reports `usercopy/swapgs barriers and __user pointer -sanitization`. +**Built** (#1150): every user copy goes through `arch::uaccess`, whose +routines replace a user address at or past the half's top with 0 without a +branch (`cmovae` on x86-64; `csel` and `csdb` on AArch64), so a mispredicted +range check in `user_ptr` loads no kernel byte through a user pointer. Linux +at `Ubuntu-6.8.0-142.142` reports `usercopy/swapgs barriers and __user +pointer sanitization` for its own user-address masking. + +**Left:** the measurement the exit asks for. No gadget has been run against +these routines on any proving machine. **Exit**: on every proving machine a gadget through each `user_ptr` site recovers a planted byte no more often than chance, 1 in 256, by a bound the diff --git a/kernel/pure/message.rs b/kernel/pure/message.rs index 66db4be8686..9ec96a78965 100644 --- a/kernel/pure/message.rs +++ b/kernel/pure/message.rs @@ -87,20 +87,30 @@ impl Queue { self.messages.push_back(Head { len, handles: count }); } + /// The front message, where a reader holding `buf` bytes and `cap` + /// handles can take it whole: what a receive claims before it copies the + /// message's bytes out. + pub fn front(&self, buf: usize, cap: usize) -> Option> { + let head = self.head()?; + Some(match head.len > buf || head.handles > cap { + true => Err(Untaken::TooSmall), + false => Ok(head), + }) + } + /// Take the front message into a reader holding `buf` bytes and `cap` /// handles, whose table `room` says can take its handles, which come - /// back in order. Its bytes are the caller's to copy out before this - /// hold ends. + /// back in order. Its bytes are the caller's to let go of. pub fn pop( &mut self, buf: usize, cap: usize, room: impl FnOnce(usize) -> bool, ) -> Option + '_), Untaken>> { - let head = self.head()?; - if head.len > buf || head.handles > cap { - return Some(Err(Untaken::TooSmall)); - } + let head = match self.front(buf, cap)? { + Ok(head) => head, + Err(untaken) => return Some(Err(untaken)), + }; if head.handles > 0 && !room(head.handles) { return Some(Err(Untaken::NoRoom)); } @@ -183,6 +193,21 @@ mod tests { assert_eq!(d.recv(4, 2, true), Some(Ok((alloc::vec![0, 1, 2, 3], alloc::vec![1, 2])))); } + /// A receive claims the front without taking it: `front` refuses what the + /// reader cannot hold as `pop` would, and leaves the queue as it was. + #[test] + fn the_front_is_answered_and_left_queued() { + let mut d = Direction::new(64); + assert!(d.send(4, &[1, 2])); + assert!(d.send(1, &[])); + assert_eq!(d.queue.front(3, 8), Some(Err(Untaken::TooSmall))); + assert_eq!(d.queue.front(4, 1), Some(Err(Untaken::TooSmall))); + assert_eq!(d.queue.front(4, 2), Some(Ok(Head { len: 4, handles: 2 }))); + assert_eq!(d.queue.len(), 2, "an answered front left the queue"); + assert_eq!(d.recv(4, 2, true), Some(Ok((alloc::vec![0, 1, 2, 3], alloc::vec![1, 2])))); + assert_eq!(d.queue.front(1, 0), Some(Ok(Head { len: 1, handles: 0 }))); + } + /// The table's room is asked only for a message that carries handles. #[test] fn a_message_without_handles_asks_no_room() { diff --git a/kernel/pure/paged.rs b/kernel/pure/paged.rs index f60832144ba..a43cf83ed01 100644 --- a/kernel/pure/paged.rs +++ b/kernel/pure/paged.rs @@ -571,6 +571,21 @@ impl Pages { o.size = o.size.max(end); } + /// The end of a [`write_pin`](Self::write_pin) whose bytes did not all + /// come: released as [`write_unpin`](Self::write_unpin) releases it, the + /// size left as it was. + pub fn write_abandon(&mut self, o: &Object, page: u64, frame: F) + where + F: PartialEq, + { + let Some(i) = self.find(o.id, page) else { return }; + let s = self.slot_mut(i); + if s.frame != Some(frame) { + return; + } + s.state.write_pins = s.state.write_pins.checked_sub(1).expect("a write pin released twice"); + } + fn range_ok(o: &Object, first: u64, count: u64) -> bool { (1..=RANGE_PAGES).contains(&count) && first.checked_add(count).is_some() && !o.gone } @@ -851,6 +866,19 @@ mod tests { assert_eq!(p.write_pin(&mut o, 1), Err(Refused::InvalidArgument), "an absent page is supplied, not pinned"); } + /// A write pin whose copy faulted is released with the size as it was. + #[test] + fn an_abandoned_write_pin_grows_nothing_and_holds_nothing() { + let mut p = Pages::new(8, 4).unwrap(); + let mut o = p.object(6, 100); + p.supply(&mut o, 0, 7, false, 100).unwrap(); + assert_eq!(p.write_pin(&mut o, 0), Ok(7)); + p.write_abandon(&o, 0, 7); + assert_eq!(o.size(), 100, "an abandoned write grows nothing"); + let slot = p.find(o.id, 0).expect("the page stays"); + assert_eq!(p.slot(slot).state.write_pins, 0, "an abandoned write holds no pin"); + } + #[test] fn an_unpin_after_a_shrink_removed_the_page_grows_nothing() { let mut p = Pages::new(8, 4).unwrap(); diff --git a/kernel/pure/pipe.rs b/kernel/pure/pipe.rs index 7b85fe6bd96..1d28668f309 100644 --- a/kernel/pure/pipe.rs +++ b/kernel/pure/pipe.rs @@ -136,17 +136,50 @@ pub fn takes(space: usize, len: usize) -> Option { /// largest, never more than [`MOST_FRAMES`], and a stream of writes of one /// size takes no frame from the allocator whether its reader keeps up or /// lags. +/// +/// **A copy runs with no lock held, inside a claim.** A write claims the room +/// past the last byte held and a read the oldest bytes held, one of each at +/// a time; the kernel's lock is held only to claim, to hand out a claim's +/// frames and to end it. Nothing but the claim's holder reaches its bytes: a +/// claimed write's are past every reader's, its frames are kept as used, and +/// only the read's own [`Self::consume`] lets claimed bytes go. A claim's +/// frames keep their contents across another's calls, but not their place in +/// the list, which a read's consume rotates; so a copier takes each frame +/// it needs under the lock, as [`Self::place`] and [`Self::peek`] hand them +/// out, and uses it once the lock is let go. +/// +/// A copy that finds its end's claim held waits for [`Self::turn`] to answer +/// `true`, and each claim's end answers whether to wake such copies +/// ([`Ended`]). pub struct Bytes { frames: VecDeque, head: usize, len: usize, /// The frames the last write spanned: the working set its next one wants. last: usize, + /// The bytes past the last held a write has claimed. + writing: Option, + /// The oldest bytes held a read has claimed. + reading: Option, + /// By [`End`]: a copy found that end's claim held since it was last ended. + waiting: [bool; 2], +} + +/// What a claim's end owes the copies it may have kept waiting, done once the +/// lock is let go. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +#[must_use = "a copy that found this claim held sleeps until its end wakes it"] +pub struct Ended { + /// A copy of its kind found it held: wake each to look again. + pub turn: bool, + /// Room it claimed comes back unused, which a wait for room or a writable + /// poll found taken while it was held: wake every waiter. + pub room: bool, } impl Bytes { pub const fn new() -> Self { - Self { frames: VecDeque::new(), head: 0, len: 0, last: 0 } + Self { frames: VecDeque::new(), head: 0, len: 0, last: 0, writing: None, reading: None, waiting: [false; 2] } } pub fn len(&self) -> usize { @@ -157,8 +190,9 @@ impl Bytes { self.len == 0 } + /// Room for a write to claim: none of it claimed. pub fn space(&self) -> usize { - PIPE_CAPACITY - self.len + PIPE_CAPACITY - self.len - self.writing.unwrap_or(0) } /// The frames it holds, spares among them. @@ -171,39 +205,83 @@ impl Bytes { self.frames.capacity() } - /// Grow its list to `frames`, at most [`MOST_FRAMES`]. - pub fn list(&mut self, frames: usize) { - assert!(frames <= MOST_FRAMES, "pipe: a list of {frames} frames"); - self.frames.reserve_exact(frames.saturating_sub(self.frames.len())); + /// Move the frames into `list`, empty and with room for [`Self::wants`] + /// more, so that adding them grows nothing; answers the old list, empty, + /// for the caller to drop. + pub fn relist(&mut self, mut list: VecDeque) -> VecDeque { + assert!(list.is_empty(), "pipe: a new list that holds frames"); + assert!(list.capacity() >= self.frames.len() + self.wants(), "pipe: a new list with no room for its claim"); + list.extend(self.frames.drain(..)); + core::mem::replace(&mut self.frames, list) } - /// The frames holding bytes. + /// The frames holding bytes or claimed for them. fn used(&self) -> usize { - (self.head + self.len).div_ceil(FRAME) + (self.head + self.len + self.writing.unwrap_or(0)).div_ceil(FRAME) + } + + /// Whether a copy at `end` may claim now. While another holds `end`'s + /// claim, `false`, and that claim's end answers [`Ended::turn`]: a waiter + /// asks under every hold of the lock it re-checks in, so a claim made + /// between a wake and its re-check still wakes it as it ends. + pub fn turn(&mut self, end: End) -> bool { + let held = match end { + End::Read => self.reading.is_some(), + End::Write => self.writing.is_some(), + }; + if held { + self.waiting[end as usize] = true; + } + !held + } + + /// Whether `end`'s claim is held, asking nothing of its end: for a copy + /// that will not wait for it. + pub fn claim_held(&self, end: End) -> bool { + match end { + End::Read => self.reading.is_some(), + End::Write => self.writing.is_some(), + } + } + + fn ended(&mut self, end: End, room: bool) -> Ended { + Ended { turn: core::mem::take(&mut self.waiting[end as usize]), room } } - /// How many frames more a write of `n` bytes, at most [`Self::space`], - /// needs than it holds. - pub fn wants(&self, n: usize) -> usize { + /// Claim `n` bytes of room, at most [`Self::space`], for one write, its + /// [`Self::turn`] answered. + pub fn claim_write(&mut self, n: usize) { + assert!(self.writing.is_none(), "pipe: a write claimed beside another"); assert!(n <= self.space(), "pipe: a {n}-byte write with room for {}", self.space()); - (self.head + self.len + n).div_ceil(FRAME).saturating_sub(self.frames.len()) + self.writing = Some(n); + } + + /// Whether a write holds its claim. + pub fn writing(&self) -> bool { + self.writing.is_some() } - /// One frame more for the next write. + /// How many frames more the claimed write needs than it holds. + pub fn wants(&self) -> usize { + self.used().saturating_sub(self.frames.len()) + } + + /// One frame more for the claimed write, into the room its list has. pub fn add(&mut self, frame: F) { assert!(self.frames.len() < MOST_FRAMES, "pipe: a frame past the most it holds"); + assert!(self.frames.len() < self.frames.capacity(), "pipe: a frame past its list"); self.frames.push_back(frame); } - /// Place the `len` bytes `at` past the last byte held, inside the room a - /// write that [`Self::wants`] no more frames for may fill: `copy` is - /// handed each run of them as its frame, where they go in it, and where - /// they start among the `len`. They are held once [`Self::commit`] counts - /// them. + /// Place the `len` claimed bytes `at` into the claim, once it + /// [`Self::wants`] no more frames: `copy` is handed each run of them as + /// its frame, where they go in it, and where they start among the `len`. + /// They are held once [`Self::commit`] counts them. #[inline(always)] pub fn place(&mut self, at: usize, len: usize, mut copy: impl FnMut(&mut F, Range, usize)) { - let end = self.head + self.len + at + len; - assert!(end <= self.frames.len() * FRAME, "pipe: bytes placed past the frames it was given"); + let claim = self.writing.expect("pipe: bytes placed with no write claimed"); + assert!(at + len <= claim, "pipe: {len} bytes placed at {at} of a {claim}-byte claim"); + assert_eq!(self.wants(), 0, "pipe: bytes placed past the frames it was given"); let (mut done, mut pos) = (0, self.head + self.len + at); while done < len { let (frame, off) = (pos / FRAME, pos % FRAME); @@ -213,18 +291,45 @@ impl Bytes { } } - /// Count `n` more bytes held: what [`Self::place`] put at the end. - pub fn commit(&mut self, n: usize) { - assert_eq!(self.wants(n), 0, "pipe: bytes held past the frames it was given"); + /// Count the claimed write's bytes held, ending its claim. + pub fn commit(&mut self) -> Ended { + let n = self.writing.expect("pipe: a commit with no write claimed"); + assert_eq!(self.wants(), 0, "pipe: bytes held past the frames it was given"); self.last = (self.head + self.len + n).div_ceil(FRAME) - (self.head + self.len) / FRAME; - self.len += n; + (self.len, self.writing) = (self.len + n, None); + self.ended(End::Write, false) + } + + /// End the claimed write holding none of it. + pub fn abandon_write(&mut self) -> Ended { + let n = self.writing.take().expect("pipe: an abandon with no write claimed"); + self.ended(End::Write, n > 0) + } + + /// Claim the `n` oldest bytes, at most [`Self::len`], for one read, its + /// [`Self::turn`] answered. + pub fn claim_read(&mut self, n: usize) { + assert!(self.reading.is_none(), "pipe: a read claimed beside another"); + assert!(n <= self.len, "pipe: {n} bytes claimed of {} held", self.len); + self.reading = Some(n); + } + + /// Whether a read holds its claim. + pub fn reading(&self) -> bool { + self.reading.is_some() } - /// The `len` held bytes `at` past the oldest, handed to `copy` as + /// Whether either claim is held. + pub fn claimed(&self) -> bool { + self.writing.is_some() || self.reading.is_some() + } + + /// The `len` claimed bytes `at` past the oldest, handed to `copy` as /// [`Self::place`] hands them out; [`Self::consume`] lets them go. #[inline(always)] pub fn peek(&self, at: usize, len: usize, mut copy: impl FnMut(&F, Range, usize)) { - assert!(at + len <= self.len, "pipe: {len} bytes at {at} of {} held", self.len); + let claim = self.reading.expect("pipe: bytes peeked with no read claimed"); + assert!(at + len <= claim, "pipe: {len} bytes peeked at {at} of a {claim}-byte claim"); let (mut done, mut pos) = (0, self.head + at); while done < len { let (frame, off) = (pos / FRAME, pos % FRAME); @@ -234,26 +339,57 @@ impl Bytes { } } - /// Let the `n` oldest bytes go; a frame it empties becomes a spare. - pub fn consume(&mut self, n: usize) { - assert!(n <= self.len, "pipe: {n} bytes let go of {} held", self.len); + /// Let the claimed bytes go, ending the read's claim; a frame it empties + /// becomes a spare. The room it frees is its caller's to announce. + pub fn consume(&mut self) -> Ended { + let n = self.reading.take().expect("pipe: a consume with no read claimed"); let emptied = (self.head + n) / FRAME; self.frames.rotate_left(emptied); (self.head, self.len) = ((self.head + n) % FRAME, self.len - n); - if self.len == 0 { + if self.len == 0 && self.writing.is_none() { self.head = 0; } + self.ended(End::Read, false) + } + + /// A write of `n` bytes, at most [`Self::space`], whole in one hold of + /// the lock, in its [`Self::turn`]: claimed, handed to `copy` as + /// [`Self::place`] hands them out, and committed. `false`, claiming + /// nothing, where it would want a frame, which a claim takes with the + /// lock let go. No copy waits on it: one asks only while a claim is held. + pub fn put(&mut self, n: usize, copy: impl FnMut(&mut F, Range, usize)) -> bool { + self.claim_write(n); + if self.wants() > 0 { + self.writing = None; + return false; + } + self.place(0, n, copy); + assert_eq!(self.commit(), Ended { turn: false, room: false }, "pipe: a copy waited on a put"); + true } - /// The spares past those it keeps, for the caller to give back. - pub fn surplus(&mut self) -> Drain<'_, F> { - let used = self.used(); + /// A read of the `n` oldest bytes, at most [`Self::len`], whole in one + /// hold of the lock, in its [`Self::turn`]: claimed, handed to `copy` as + /// [`Self::peek`] hands them out, and let go. + pub fn take(&mut self, n: usize, copy: impl FnMut(&F, Range, usize)) { + self.claim_read(n); + self.peek(0, n, copy); + assert_eq!(self.consume(), Ended { turn: false, room: false }, "pipe: a copy waited on a take"); + } + + /// At most `most` of the spares past those it keeps, for the caller to + /// give back; fewer than `most` once none is left past them. + pub fn surplus(&mut self, most: usize) -> Drain<'_, F> { + // A claim may want frames its writer is still taking: every one held is used. + let used = self.used().min(self.frames.len()); let keep = used + (self.frames.len() - used).min(used.max(self.last).max(1)); - self.frames.drain(keep..) + self.frames.drain(keep.max(self.frames.len().saturating_sub(most))..) } - /// Every frame, for the caller to give back, and with them every byte. + /// Every frame, for the caller to give back, and with them every byte; + /// never while a claim is held, whose copy may still reach its frames. pub fn clear(&mut self) -> Drain<'_, F> { + assert!(!self.claimed(), "pipe: bytes cleared under a claim"); (self.head, self.len, self.last) = (0, 0, 0); self.frames.drain(..) } @@ -635,29 +771,42 @@ mod tests { Self { bytes: Bytes::new(), taken: 0, returned: Vec::new() } } - /// Write `src` as `kernel::pipe` does: the frames it wants, then - /// the copy placed and committed. - fn write(&mut self, src: &[u8]) { - for _ in 0..self.bytes.wants(src.len()) { + /// Claim `n` bytes of room and supply the frames the claim wants, as + /// `kernel::frame_bytes` does: a list grown first, then the frames. + fn claim(&mut self, n: usize) { + self.bytes.claim_write(n); + let wants = self.bytes.wants(); + if self.bytes.frames() + wants > self.bytes.listed() { + let old = self.bytes.relist(VecDeque::with_capacity((self.bytes.frames() + wants).next_power_of_two())); + assert!(old.is_empty()); + } + for _ in 0..wants { self.bytes.add(Frame { id: self.taken, bytes: vec![0xEE; FRAME] }); self.taken += 1; } + } + + /// Write `src` as `kernel::pipe` does: claimed, then the copy placed + /// and committed. + fn write(&mut self, src: &[u8]) { + self.claim(src.len()); self.bytes.place(0, src.len(), |frame, within, at| { frame.bytes[within.clone()].copy_from_slice(&src[at..at + within.len()]); }); - self.bytes.commit(src.len()); + let _ = self.bytes.commit(); } - /// Read up to `want` bytes as `kernel::pipe` does: peeked, consumed, - /// and the surplus given back. + /// Read up to `want` bytes as `kernel::pipe` does: claimed, peeked, + /// consumed, and the surplus given back. fn read(&mut self, want: usize) -> Vec { let n = want.min(self.bytes.len()); let mut out = vec![0; n]; + self.bytes.claim_read(n); self.bytes.peek(0, n, |frame, within, at| { out[at..at + within.len()].copy_from_slice(&frame.bytes[within]); }); - self.bytes.consume(n); - self.returned.extend(self.bytes.surplus().map(|frame| frame.id)); + let _ = self.bytes.consume(); + self.returned.extend(self.bytes.surplus(usize::MAX).map(|frame| frame.id)); out } @@ -726,10 +875,7 @@ mod tests { let (mut written, mut read) = (0, 0); for _ in 0..300 { let n = random(3 * FRAME + 1).min(pipe.bytes.space()); - for _ in 0..pipe.bytes.wants(n) { - pipe.bytes.add(Frame { id: pipe.taken, bytes: vec![0xEE; FRAME] }); - pipe.taken += 1; - } + pipe.claim(n); let src = pattern(written, n); let mut cuts: Vec = (0..random(6)).map(|_| random(n + 1)).chain([0, n]).collect(); cuts.sort_unstable(); @@ -740,11 +886,12 @@ mod tests { frame.bytes[within.clone()].copy_from_slice(&src[at + from..at + from + within.len()]); }); } - pipe.bytes.commit(n); + let _ = pipe.bytes.commit(); written += n; let m = random(pipe.bytes.len() + 1); let mut out = vec![0; m]; + pipe.bytes.claim_read(m); let mut at = 0; while at < m { let len = (1 + random(2 * FRAME)).min(m - at); @@ -753,8 +900,8 @@ mod tests { }); at += len; } - pipe.bytes.consume(m); - pipe.returned.extend(pipe.bytes.surplus().map(|frame| frame.id)); + let _ = pipe.bytes.consume(); + pipe.returned.extend(pipe.bytes.surplus(usize::MAX).map(|frame| frame.id)); assert_eq!(out, pattern(read, m), "bytes {read}.. came back wrong"); read += m; pipe.accounted(); @@ -810,4 +957,283 @@ mod tests { assert_eq!(pipe.bytes.clear().count(), 1); assert_eq!((pipe.bytes.len(), pipe.bytes.frames()), (0, 0)); } + + /// **The claims carry the data-race argument.** A write placed a run at a + /// time with reads claimed, peeked and consumed between its runs, as + /// another CPU's read runs while a write copies with no lock held: every + /// frame a run is handed is the one an earlier run of the claim was + /// handed for those bytes, no read is handed a claimed byte, and none of + /// the claim's frames is given back as surplus. + #[test] + fn a_write_claim_keeps_its_frames_while_reads_drain_beside_it() { + let mut seed = 0x2545_f491_u64; + let mut random = move |below: usize| { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + (seed % below as u64) as usize + }; + let mut pipe = Held::new(); + let (mut written, mut read) = (0, 0); + for _ in 0..300 { + let n = random(4 * FRAME + 1).min(pipe.bytes.space()); + pipe.claim(n); + let src = pattern(written, n); + // Where each claimed byte's run was first handed out, by frame id and offset. + let mut handed: Vec> = vec![None; n]; + let mut at = 0; + while at < n { + let len = (1 + random(FRAME)).min(n - at); + pipe.bytes.place(at, len, |frame, within, from| { + frame.bytes[within.clone()].copy_from_slice(&src[at + from..at + from + within.len()]); + for (i, off) in within.enumerate() { + handed[at + from + i] = Some((frame.id, off)); + } + }); + at += len; + let m = random(pipe.bytes.len() + 1); + let mut out = vec![0; m]; + pipe.bytes.claim_read(m); + pipe.bytes.peek(0, m, |frame, within, from| { + out[from..from + within.len()].copy_from_slice(&frame.bytes[within]); + }); + let _ = pipe.bytes.consume(); + pipe.returned.extend(pipe.bytes.surplus(usize::MAX).map(|frame| frame.id)); + assert_eq!(out, pattern(read, m), "a read beside a claim came back wrong"); + read += m; + pipe.accounted(); + for (byte, place) in handed.iter().enumerate().take(at) { + let (id, _) = place.expect("placed"); + assert!(!pipe.returned.contains(&id), "claimed byte {byte}'s frame {id} was given back"); + } + } + // Placed again over what was placed: each byte lands where it did. + pipe.bytes.place(0, n, |frame, within, from| { + for (i, off) in within.enumerate() { + assert_eq!(handed[from + i], Some((frame.id, off)), "claimed byte {} moved", from + i); + } + }); + let _ = pipe.bytes.commit(); + written += n; + } + let rest = pipe.read(PIPE_CAPACITY); + assert_eq!(rest, pattern(read, written - read)); + } + + /// One claim of each kind at a time: a second waits for its turn and + /// claims nothing, and a claim's bytes are not another's. + #[test] + fn a_second_claim_waits_until_the_first_ends() { + let mut pipe = Held::new(); + pipe.claim(10); + assert!(!pipe.bytes.turn(End::Write), "a second write's turn beside the first"); + assert!(pipe.bytes.turn(End::Read), "a write's claim held up a read"); + assert_eq!(pipe.bytes.len(), 0, "a claimed write was held before its commit"); + assert_eq!(pipe.bytes.space(), PIPE_CAPACITY - 10, "the claimed room was offered again"); + pipe.bytes.place(0, 10, |frame, within, at| { + frame.bytes[within.clone()].copy_from_slice(&pattern(at, within.len())); + }); + assert_eq!(pipe.bytes.commit(), Ended { turn: true, room: false }); + pipe.bytes.claim_read(4); + assert!(!pipe.bytes.turn(End::Read), "a second read's turn beside the first"); + assert_eq!(pipe.bytes.consume(), Ended { turn: true, room: false }); + assert!(!pipe.bytes.claimed()); + assert_eq!(pipe.read(10), pattern(4, 6)); + } + + /// A claim ended with nobody waiting for it wakes nobody. + #[test] + fn an_unwaited_claim_wakes_nobody() { + let mut pipe = Held::new(); + pipe.claim(10); + assert!(pipe.bytes.turn(End::Read)); + pipe.bytes.place(0, 10, |frame, within, _| frame.bytes[within].fill(1)); + assert_eq!(pipe.bytes.commit(), Ended { turn: false, room: false }); + pipe.bytes.claim_read(10); + assert_eq!(pipe.bytes.consume(), Ended { turn: false, room: false }); + } + + /// A waiter woken by one claim's end that finds another's claim made + /// before its re-check is woken by that one's end too: the re-check + /// counts as waiting again. + #[test] + fn a_claim_made_before_a_woken_waiter_looks_again_still_wakes_it() { + for end in [End::Read, End::Write] { + let mut pipe = Held::new(); + pipe.write(&pattern(0, 100)); + let claim = |pipe: &mut Held| match end { + End::Read => pipe.bytes.claim_read(10), + End::Write => pipe.claim(10), + }; + let end_claim = |pipe: &mut Held| match end { + End::Read => pipe.bytes.consume(), + End::Write => { + pipe.bytes.place(0, 10, |frame, within, _| frame.bytes[within].fill(1)); + pipe.bytes.commit() + } + }; + // A holds the claim; B finds it held. + claim(&mut pipe); + assert!(!pipe.bytes.turn(end)); + // A ends it and wakes B; C claims before B looks again. + assert!(end_claim(&mut pipe).turn, "{end:?}: A's end woke nobody"); + assert!(pipe.bytes.turn(end)); + claim(&mut pipe); + // B looks again and parks. + assert!(!pipe.bytes.turn(end)); + assert!(end_claim(&mut pipe).turn, "{end:?}: C's end left B parked"); + } + } + + /// Writes and reads made whole in one hold, beside a large write's claim + /// in flight and a read's: a stream comes back exact, `put` takes no + /// frame (a claim supplies them), and the claim keeps its place. + #[test] + fn whole_transfers_in_one_hold_keep_the_stream_and_the_claims() { + let mut pipe = Held::new(); + let (mut written, mut read) = (0, 0); + for round in 0..200 { + let n = [1, 64, 300, FRAME, FRAME + 7][round % 5]; + let src = pattern(written, n); + if !pipe.bytes.put(n, |frame, within, at| frame.bytes[within.clone()].copy_from_slice(&src[at..at + within.len()])) { + assert!(!pipe.bytes.writing() && pipe.bytes.turn(End::Write), "a refused put claimed"); + pipe.write(&src); + } + written += n; + if round % 3 == 0 { + // A large write claims its room and places part of it; small + // reads drain what was there before it. + let big = 3 * FRAME; + pipe.claim(big); + let src = pattern(written, big); + pipe.bytes.place(0, FRAME, |frame, within, at| frame.bytes[within.clone()].copy_from_slice(&src[at..at + within.len()])); + assert!(!pipe.bytes.turn(End::Write), "a put's turn beside a write's claim"); + while !pipe.bytes.is_empty() { + let m = pipe.bytes.len().min(100); + let mut out = vec![0; m]; + pipe.bytes.take(m, |frame, within, at| out[at..at + within.len()].copy_from_slice(&frame.bytes[within])); + assert_eq!(out, pattern(read, m), "round {round}: a take beside a claim"); + read += m; + pipe.returned.extend(pipe.bytes.surplus(usize::MAX).map(|frame| frame.id)); + } + pipe.bytes.place(FRAME, big - FRAME, |frame, within, at| { + frame.bytes[within.clone()].copy_from_slice(&src[FRAME + at..FRAME + at + within.len()]); + }); + let _ = pipe.bytes.commit(); + written += big; + } + let m = pipe.bytes.len().min(500); + let mut out = vec![0; m]; + pipe.bytes.take(m, |frame, within, at| out[at..at + within.len()].copy_from_slice(&frame.bytes[within])); + assert_eq!(out, pattern(read, m), "round {round}"); + read += m; + pipe.returned.extend(pipe.bytes.surplus(usize::MAX).map(|frame| frame.id)); + assert!(pipe.bytes.frames() <= MOST_FRAMES); + } + assert_eq!(pipe.read(PIPE_CAPACITY), pattern(read, written - read)); + pipe.accounted(); + } + + /// An abandoned write gives back the room it claimed, which a wait for + /// room found taken: its end says so, and an empty one does not. + #[test] + fn an_abandoned_write_gives_its_room_back() { + let mut pipe = Held::new(); + pipe.claim(FRAME); + assert_eq!(pipe.bytes.abandon_write(), Ended { turn: false, room: true }); + pipe.bytes.claim_write(0); + assert_eq!(pipe.bytes.abandon_write(), Ended { turn: false, room: false }); + } + + /// An abandoned write holds nothing. + #[test] + fn an_abandoned_claim_changes_nothing() { + let mut pipe = Held::new(); + pipe.write(&pattern(0, 8)); + pipe.claim(FRAME); + let _ = pipe.bytes.abandon_write(); + assert_eq!((pipe.bytes.len(), pipe.bytes.space()), (8, PIPE_CAPACITY - 8)); + assert_eq!(pipe.read(8), pattern(0, 8)); + } + + /// A read that empties the pipe while a write holds its claim does not + /// move the claim's bytes back to the start of the frame they began in. + #[test] + fn a_drained_pipe_beside_a_claim_keeps_the_claims_place() { + let mut pipe = Held::new(); + pipe.write(&pattern(0, 100)); + pipe.claim(50); + pipe.bytes.place(0, 20, |frame, within, at| { + frame.bytes[within.clone()].copy_from_slice(&pattern(100 + at, within.len())); + }); + assert_eq!(pipe.read(100), pattern(0, 100)); + pipe.bytes.place(20, 30, |frame, within, at| { + frame.bytes[within.clone()].copy_from_slice(&pattern(120 + at, within.len())); + }); + let _ = pipe.bytes.commit(); + assert_eq!(pipe.read(50), pattern(100, 50)); + } + + #[test] + #[should_panic(expected = "cleared under a claim")] + fn bytes_are_never_cleared_under_a_claim() { + let mut pipe = Held::new(); + pipe.claim(1); + let _ = pipe.bytes.clear(); + } + + /// The surplus given back a batch at a time is the surplus given back + /// at once. + #[test] + fn the_surplus_comes_back_in_batches() { + let (mut whole, mut batched) = (Held::new(), Held::new()); + for pipe in [&mut whole, &mut batched] { + pipe.write(&vec![1; PIPE_CAPACITY - 1]); + pipe.write(&[1]); + pipe.bytes.claim_read(PIPE_CAPACITY); + let _ = pipe.bytes.consume(); + } + let all: Vec = whole.bytes.surplus(usize::MAX).map(|f| f.id).collect(); + let mut got: Vec = Vec::new(); + loop { + let batch: Vec = batched.bytes.surplus(7).map(|f| f.id).collect(); + got.extend(&batch); + if batch.len() < 7 { + break; + } + } + got.sort_unstable(); + let mut all = all; + all.sort_unstable(); + assert_eq!(got, all); + assert_eq!(whole.bytes.frames(), batched.bytes.frames()); + } + + /// A read that ends while a write's claim still wants its frames, as one + /// does while the writer takes them with no lock held: the surplus keeps + /// every frame held, and the claim is supplied and lands after it. + #[test] + fn a_read_beside_an_unsupplied_claim_gives_back_nothing() { + let mut pipe = Held::new(); + pipe.write(&pattern(0, 100)); + pipe.bytes.claim_write(3 * FRAME); + assert!(pipe.bytes.wants() > 0, "the claim's frames are not here yet"); + assert_eq!(pipe.read(100), pattern(0, 100)); + assert_eq!(pipe.returned, Vec::::new(), "a frame was given back from under a claim"); + let wants = pipe.bytes.wants(); + if pipe.bytes.frames() + wants > pipe.bytes.listed() { + let _ = pipe.bytes.relist(VecDeque::with_capacity((pipe.bytes.frames() + wants).next_power_of_two())); + } + for _ in 0..wants { + pipe.bytes.add(Frame { id: pipe.taken, bytes: vec![0xEE; FRAME] }); + pipe.taken += 1; + } + let src = pattern(100, 3 * FRAME); + pipe.bytes.place(0, src.len(), |frame, within, at| { + frame.bytes[within.clone()].copy_from_slice(&src[at..at + within.len()]); + }); + let _ = pipe.bytes.commit(); + assert_eq!(pipe.read(3 * FRAME), src); + pipe.accounted(); + } } diff --git a/kernel/src/arch/aarch64/mod.rs b/kernel/src/arch/aarch64/mod.rs index 9e0934ea37d..9c6b28b80cc 100644 --- a/kernel/src/arch/aarch64/mod.rs +++ b/kernel/src/arch/aarch64/mod.rs @@ -48,6 +48,7 @@ pub mod smp; pub mod switch; pub mod tlb; pub mod trap; +pub mod uaccess; pub mod watchdog; pub use irqchip::its as message_unit; diff --git a/kernel/src/arch/aarch64/trap.rs b/kernel/src/arch/aarch64/trap.rs index 3637895faec..04bd2f647e1 100644 --- a/kernel/src/arch/aarch64/trap.rs +++ b/kernel/src/arch/aarch64/trap.rs @@ -6,7 +6,8 @@ //! it, the top of the running thread's kernel stack — and [`dispatch`] routes //! it: an interrupt to [`irq`], an `SVC` to the syscall dispatcher, a //! translation fault to the demand pager, anything else from EL0 to the end -//! of its process, and anything else from EL1 to a panic. Every return to EL0 +//! of its process, a user copy's abort to the demand pager or the copy's +//! fixup (`super::uaccess`), and anything else from EL1 to a panic. Every return to EL0 //! runs `scheduler::exit_to_user` last. The entry saves no FP/SIMD register: //! the kernel never touches one, and a switch saves the thread's //! (`super::switch`). @@ -129,10 +130,48 @@ extern "C" fn dispatch(frame: &mut Frame, entry: u64) { } crate::scheduler::exit_to_user(); } + EL1_SYNC if user_copy_abort(frame) => {} _ => exception(frame, entry), } } +/// `ESR_EL1.EC` of a data abort taken from EL1 itself. +const EC_DABT_SAME: u64 = 0x25; + +/// An EL1 data abort inside a user copy (`super::uaccess`) on a user address: +/// served as the process's own fault would be, after which the access is +/// taken again, or else resumed at the copy's fixup, which answers the bytes +/// not copied. The copy holds no lock at its syscall's depth (`user_ptr`), so +/// the service runs at that depth too. +/// `false` for any other EL1 exception. +fn user_copy_abort(frame: &mut Frame) -> bool { + let Some(fixup) = super::uaccess::fixup(frame.elr) else { return false }; + // `FnV`: a syndrome whose `FAR_EL1` is not valid names no address. + let far_valid = frame.esr & (1 << 10) == 0; + if frame.esr >> 26 != EC_DABT_SAME || !far_valid || !toyos_userbound::is_user_addr(frame.far) { + return false; + } + if is_translation_fault(frame.esr) { + let prev = percpu::swap_fault_state(CpuFaultState::PageFault); + if prev != CpuFaultState::Normal { + // Put back: the panic classifies a recursive fault by what it finds. + percpu::set_fault_state(prev); + return false; + } + cpu::enable_interrupts(); + // `WnR`, bit 6: the access was a write. + let access = if frame.esr & (1 << 6) != 0 { kernel::vm::Access::Write } else { kernel::vm::Access::Read }; + let served = crate::process::handle_page_fault(frame.far, access); + cpu::disable_interrupts(); + percpu::set_fault_state(CpuFaultState::Normal); + if served { + return true; + } + } + frame.elr = fixup; + true +} + /// The report of an exception this kernel does not return from: its own, at /// EL1, or an FIQ or SError from anywhere. The panic handler's branch puts it /// on the console and the panel. diff --git a/kernel/src/arch/aarch64/uaccess.rs b/kernel/src/arch/aarch64/uaccess.rs new file mode 100644 index 00000000000..c5b9e2a6bcf --- /dev/null +++ b/kernel/src/arch/aarch64/uaccess.rs @@ -0,0 +1,204 @@ +//! Copies between kernel memory and the running process's user half, through +//! that process's own mapping (`crate::user_ptr`'s contract). +//! +//! Each user access is an unprivileged `LDTR`/`STTR`, checked at EL0's +//! permissions whatever `PSTATE.PAN` says (Arm ARM K.a, C6.2: "Load/Store +//! Register (unprivileged)"), so a copy needs no PAN toggle and reaches only +//! what the process could. The bulk moves 32 bytes a step, four 8-byte user +//! accesses beside two paired kernel-side ones, then 8 bytes, then single +//! bytes. The count left is lowered only once a step's accesses are done, so +//! a fault anywhere in a routine leaves `x2` counting every byte not yet +//! written: a fault the abort path serves resumes at the faulting access, and +//! one it cannot serve resumes at the routine's fixup, which answers `x2`. +//! [`fixup`] is the exception table: only these routines' bodies may take an +//! EL1 abort on a user address and live. +//! +//! The user address is checked against the half's top by the caller; each +//! routine also replaces an address at or past it with 0 by `csel` and a +//! `csdb`, so a mispredicted check reads no kernel byte even speculatively +//! (`issues/user-pointer-checks-have-no-spectre-v1-fence.md`). +//! A copy through a user mapping of device memory makes only unprivileged +//! accesses the process could make through that same mapping itself; an +//! unaligned one faults, and the copy ends there. + +use toyos_userbound::USER_TOP; + +core::arch::global_asm!( + ".pushsection .text.toyos_uaccess, \"ax\"", + // toyos_copy_from_user(x0 = kernel dst, x1 = user src, x2 = len) -> x0 = bytes not copied + ".balign 16", + ".global toyos_copy_from_user", + "toyos_copy_from_user:", + "mov x9, #{top_hi}", + "lsl x9, x9, #32", + "cmp x1, x9", + "csel x1, xzr, x1, hs", + "hint #20", // csdb + ".global toyos_copy_from_user_begin", + "toyos_copy_from_user_begin:", + "1:", + "cmp x2, #32", + "b.lo 2f", + "ldtr x3, [x1]", + "ldtr x4, [x1, #8]", + "ldtr x5, [x1, #16]", + "ldtr x6, [x1, #24]", + "stp x3, x4, [x0]", + "stp x5, x6, [x0, #16]", + "add x0, x0, #32", + "add x1, x1, #32", + "sub x2, x2, #32", + "b 1b", + "2:", + "cmp x2, #8", + "b.lo 3f", + "ldtr x3, [x1]", + "str x3, [x0], #8", + "add x1, x1, #8", + "sub x2, x2, #8", + "b 2b", + "3:", + "cbz x2, 4f", + "ldtrb w3, [x1]", + "strb w3, [x0], #1", + "add x1, x1, #1", + "sub x2, x2, #1", + "b 3b", + ".global toyos_copy_from_user_end", + "toyos_copy_from_user_end:", + "4:", + "mov x0, x2", + "ret", + // toyos_copy_to_user(x0 = user dst, x1 = kernel src, x2 = len) -> x0 = bytes not copied + ".balign 16", + ".global toyos_copy_to_user", + "toyos_copy_to_user:", + "mov x9, #{top_hi}", + "lsl x9, x9, #32", + "cmp x0, x9", + "csel x0, xzr, x0, hs", + "hint #20", // csdb + ".global toyos_copy_to_user_begin", + "toyos_copy_to_user_begin:", + "1:", + "cmp x2, #32", + "b.lo 2f", + "ldp x3, x4, [x1]", + "ldp x5, x6, [x1, #16]", + "sttr x3, [x0]", + "sttr x4, [x0, #8]", + "sttr x5, [x0, #16]", + "sttr x6, [x0, #24]", + "add x0, x0, #32", + "add x1, x1, #32", + "sub x2, x2, #32", + "b 1b", + "2:", + "cmp x2, #8", + "b.lo 3f", + "ldr x3, [x1], #8", + "sttr x3, [x0]", + "add x0, x0, #8", + "sub x2, x2, #8", + "b 2b", + "3:", + "cbz x2, 4f", + "ldrb w3, [x1], #1", + "sttrb w3, [x0]", + "add x0, x0, #1", + "sub x2, x2, #1", + "b 3b", + ".global toyos_copy_to_user_end", + "toyos_copy_to_user_end:", + "4:", + "mov x0, x2", + "ret", + // toyos_clear_user(x0 = user dst, x1 = len) -> x0 = bytes not cleared + ".balign 16", + ".global toyos_clear_user", + "toyos_clear_user:", + "mov x9, #{top_hi}", + "lsl x9, x9, #32", + "cmp x0, x9", + "csel x0, xzr, x0, hs", + "hint #20", // csdb + "mov x2, x1", + ".global toyos_clear_user_begin", + "toyos_clear_user_begin:", + "1:", + "cmp x2, #8", + "b.lo 3f", + "sttr xzr, [x0]", + "add x0, x0, #8", + "sub x2, x2, #8", + "b 1b", + "3:", + "cbz x2, 4f", + "sttrb wzr, [x0]", + "add x0, x0, #1", + "sub x2, x2, #1", + "b 3b", + ".global toyos_clear_user_end", + "toyos_clear_user_end:", + "4:", + "mov x0, x2", + "ret", + ".popsection", + top_hi = const USER_TOP >> 32, +); + +// The top is a whole multiple of 2^32, so one `mov` and a shift build it. +const _: () = assert!(USER_TOP & 0xFFFF_FFFF == 0 && USER_TOP >> 32 <= 0xFFFF); + +unsafe extern "C" { + fn toyos_copy_from_user(dst: *mut u8, user: u64, len: usize) -> usize; + fn toyos_copy_to_user(user: u64, src: *const u8, len: usize) -> usize; + fn toyos_clear_user(user: u64, len: usize) -> usize; + static toyos_copy_from_user_begin: u8; + static toyos_copy_from_user_end: u8; + static toyos_copy_to_user_begin: u8; + static toyos_copy_to_user_end: u8; + static toyos_clear_user_begin: u8; + static toyos_clear_user_end: u8; +} + +/// Copy `len` bytes from user address `user` to `dst`; answers the bytes not +/// copied. +/// +/// # Safety +/// `dst` is writable kernel memory for `len` bytes; `[user, user + len)` lies +/// below [`USER_TOP`]; no lock is held (`crate::user_ptr`). +pub unsafe fn copy_from_user(dst: *mut u8, user: u64, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_copy_from_user(dst, user, len) } +} + +/// Copy `len` bytes from `src` to user address `user`; answers the bytes not +/// copied. +/// +/// # Safety +/// As [`copy_from_user`], with `src` readable. +pub unsafe fn copy_to_user(user: u64, src: *const u8, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_copy_to_user(user, src, len) } +} + +/// Zero `len` bytes at user address `user`; answers the bytes not zeroed. +/// +/// # Safety +/// As [`copy_from_user`]. +pub unsafe fn clear_user(user: u64, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_clear_user(user, len) } +} + +/// Where an EL1 abort at `pc` resumes when it cannot be served: `Some` only +/// inside one of the routines above, at its fixup. +pub fn fixup(pc: u64) -> Option { + let bodies = [ + (&raw const toyos_copy_from_user_begin as u64, &raw const toyos_copy_from_user_end as u64), + (&raw const toyos_copy_to_user_begin as u64, &raw const toyos_copy_to_user_end as u64), + (&raw const toyos_clear_user_begin as u64, &raw const toyos_clear_user_end as u64), + ]; + bodies.iter().find(|&&(begin, end)| (begin..end).contains(&pc)).map(|&(_, end)| end) +} diff --git a/kernel/src/arch/x86_64/cpu.rs b/kernel/src/arch/x86_64/cpu.rs index 3c3de77c434..bff46f94a9a 100644 --- a/kernel/src/arch/x86_64/cpu.rs +++ b/kernel/src/arch/x86_64/cpu.rs @@ -394,9 +394,8 @@ pub fn enable_interrupts() { /// /// **A measurement and not a guard.** The answer is stale the instant any `sti` /// or `cli` runs, so nothing may branch on it to decide whether to mask; what it -/// is for is a site that has to *say* which state it inherited, which is a fact -/// about its caller rather than about the instruction after it. -#[cfg(any(feature = "boot-actuators", feature = "mask-windows"))] +/// is for is a site that has to *say* or assert which state it inherited, +/// which is a fact about its caller rather than about the instruction after it. pub fn interrupts_enabled() -> bool { let rflags: u64; // SAFETY: balanced push/pop leaves rsp unchanged; the pair uses the stack, so no nomem. diff --git a/kernel/src/arch/x86_64/idt/exceptions.rs b/kernel/src/arch/x86_64/idt/exceptions.rs index 56b73fee86b..ff31fb7df2c 100644 --- a/kernel/src/arch/x86_64/idt/exceptions.rs +++ b/kernel/src/arch/x86_64/idt/exceptions.rs @@ -455,7 +455,7 @@ pub(super) fn machine_check_handler(frame: &TrapFrame) -> ! { } /// Returns if the fault was resolved (page mapped in); diverges if fatal. -pub(super) fn page_fault_handler(frame: &TrapFrame) { +pub(super) fn page_fault_handler(frame: &mut TrapFrame) { let prev = percpu::swap_fault_state(percpu::CpuFaultState::PageFault); if prev != percpu::CpuFaultState::Normal { // Restores the prior fault state: `fatal_exception` classifies a @@ -468,6 +468,13 @@ pub(super) fn page_fault_handler(frame: &TrapFrame) { } let fault_addr = cpu::read_cr2(); + let copy_fixup = if Ring::of_cs(frame.cs).is_user() { None } else { crate::arch::uaccess::fixup(frame.rip) }; + if let Some(fixup) = copy_fixup.filter(|_| toyos_userbound::is_user_addr(fault_addr)) { + user_copy_fault(frame, fault_addr, fixup); + percpu::set_fault_state(percpu::CpuFaultState::Normal); + return; + } + if frame.error_code & PF_PRESENT != 0 && !Ring::of_cs(frame.cs).is_user() && mm::is_kernel_addr(fault_addr) { @@ -506,6 +513,28 @@ pub(super) fn page_fault_handler(frame: &TrapFrame) { } +/// A user copy's fault on `addr` (`arch::uaccess`): served as the process's +/// own fault would be, after which the copy resumes where it stopped, or else +/// resumed at `fixup`, which answers the bytes not copied. The copy was made +/// holding no lock at its syscall's depth (`user_ptr`), so the service runs at +/// that depth too. +fn user_copy_fault(frame: &mut TrapFrame, addr: u64, fixup: u64) { + // Every copy instruction runs between its routine's `stac` and `clac`: a + // fault there with `AC` clear is a kernel bug, never a bad user address. + const AC: u64 = 1 << 18; + assert!(frame.rflags & AC != 0, "uaccess: a copy faulted at {:#x} with RFLAGS.AC clear", frame.rip); + let access = if frame.error_code & PF_WRITE != 0 { kernel::vm::Access::Write } else { kernel::vm::Access::Read }; + if frame.error_code & PF_PRESENT == 0 { + percpu::preempt_count_down(); + let served = process::handle_page_fault(addr, access); + percpu::preempt_count_up(); + if served { + return; + } + } + frame.rip = fixup; +} + /// Fatal exception handler for #UD and #GP. Never returns. pub(super) fn exception_handler(frame: &TrapFrame) -> ! { let cr2 = if frame.vector == 0x0E { cpu::read_cr2() } else { 0 }; diff --git a/kernel/src/arch/x86_64/idt/mod.rs b/kernel/src/arch/x86_64/idt/mod.rs index 908d42e3ab5..efeb42f6244 100644 --- a/kernel/src/arch/x86_64/idt/mod.rs +++ b/kernel/src/arch/x86_64/idt/mod.rs @@ -344,9 +344,14 @@ extern "sysv64" fn trap_dispatch(frame: *mut TrapFrame) { // wrote nothing; the registers are what this entry can say without risking // that, and the report overwrites them if the machine gets that far. Every // vector seals but a Ring 3 page fault, which every first touch of a page - // takes: it is served or ends its process through the crash path, which - // reports, and a fault inside its handler is a Ring 0 fault and seals. - let user_page_fault = Vector::from_raw(frame.vector) == Vector::PageFault && toyos_userbound::Ring::of_cs(frame.cs).is_user(); + // takes, and a user copy's (`arch::uaccess`): it is served or ends its + // process through the crash path, which reports, or the copy answers what + // it did not copy; any other fault inside its handler is a Ring 0 fault and + // seals. + let page_fault = Vector::from_raw(frame.vector) == Vector::PageFault; + // A copy's fault on its kernel side is a kernel fault, and seals. + let copy_fault = super::uaccess::fixup(frame.rip).is_some() && toyos_userbound::is_user_addr(cpu::read_cr2()); + let user_page_fault = page_fault && (toyos_userbound::Ring::of_cs(frame.cs).is_user() || copy_fault); if !user_page_fault { crate::blackbox::record_fault(&fault_of(frame)); } @@ -359,8 +364,9 @@ extern "sysv64" fn trap_dispatch(frame: *mut TrapFrame) { cpu::enable_interrupts(); exceptions::page_fault_handler(frame); cpu::disable_interrupts(); - // Only a Ring 3 fault comes back: `common_entry` lowers the count - // next, and `exit_to_user` opens interrupts. + // Only a Ring 3 fault or a user copy's comes back: `common_entry` + // lowers the count next, and `exit_to_user` or the `iretq` opens + // interrupts. #[cfg(feature = "mask-windows")] crate::windows::preempt_lowering(); } @@ -378,7 +384,7 @@ extern "sysv64" fn trap_dispatch(frame: *mut TrapFrame) { /// place of the fault's report. #[cfg(feature = "mask-windows")] fn windows_entered(frame: &TrapFrame) { - if !toyos_userbound::Ring::of_cs(frame.cs).is_user() { + if !toyos_userbound::Ring::of_cs(frame.cs).is_user() && super::uaccess::fixup(frame.rip).is_none() { crate::windows::stop_here(); } crate::windows::irqs_masked(); diff --git a/kernel/src/arch/x86_64/mod.rs b/kernel/src/arch/x86_64/mod.rs index 74e8ea8159c..3692de91b0b 100644 --- a/kernel/src/arch/x86_64/mod.rs +++ b/kernel/src/arch/x86_64/mod.rs @@ -33,6 +33,7 @@ pub mod copy; pub mod counters; pub mod console_uart; pub mod cpu; +pub mod uaccess; pub mod entropy; pub mod entry; pub mod fpu; diff --git a/kernel/src/arch/x86_64/paging.rs b/kernel/src/arch/x86_64/paging.rs index 10c4027627f..fd111aba2e1 100644 --- a/kernel/src/arch/x86_64/paging.rs +++ b/kernel/src/arch/x86_64/paging.rs @@ -29,7 +29,7 @@ const PAGE_WRITE: u64 = 1 << 1; const PAGE_USER: u64 = 1 << 2; const PAGE_WRITE_THROUGH: u64 = 1 << 3; const PAGE_CACHE_DISABLE: u64 = 1 << 4; -/// Set by hardware on access or write; masked out when comparing entries. +/// Masked out when comparing entries; nothing reads either. const PAGE_ACCESSED: u64 = 1 << 5; const PAGE_DIRTY: u64 = 1 << 6; const PAGE_SIZE_BIT: u64 = 1 << 7; @@ -52,11 +52,15 @@ const TABLE_FLAGS: u64 = PAGE_PRESENT | PAGE_WRITE | PAGE_USER; impl Prot { /// The permission bits a user leaf carries; address and cache policy stay the caller's. + /// + /// Accessed, and dirty where writable, from the start (SDM Vol. 3A §4.8; + /// AMD APM Vol. 2 §5.4.2): otherwise the walker writes them on a fresh + /// page's first access. A clean-page tracker would have to stop setting `DIRTY`. fn leaf_bits(self) -> u64 { - let common = PAGE_PRESENT | PAGE_USER; + let common = PAGE_PRESENT | PAGE_USER | PAGE_ACCESSED; match self { Self::Read => common | PAGE_NX, - Self::ReadWrite => common | PAGE_WRITE | PAGE_NX, + Self::ReadWrite => common | PAGE_WRITE | PAGE_DIRTY | PAGE_NX, Self::ReadExec => common, } } diff --git a/kernel/src/arch/x86_64/uaccess.rs b/kernel/src/arch/x86_64/uaccess.rs new file mode 100644 index 00000000000..7368311373d --- /dev/null +++ b/kernel/src/arch/x86_64/uaccess.rs @@ -0,0 +1,130 @@ +//! Copies between kernel memory and the running process's user half, through +//! that process's own mapping (`crate::user_ptr`'s contract). +//! +//! Each routine opens SMAP's door with `stac` around one `rep movsb` or +//! `rep stosb` and closes it with `clac`. "A repeating string operation can be +//! suspended by an exception or interrupt. When this happens, the state of the +//! registers is preserved to allow the string operation to be resumed" (Intel +//! SDM Vol. 2B, REP/REPE/REPZ/REPNE/REPNZ). So a fault the page-fault path +//! serves resumes the copy where it stopped, and one it cannot serve resumes +//! at the instruction after it, the fixup, which closes the door and answers +//! what `RCX` still counts. [`fixup`] is the exception +//! table: only these three instructions may take a Ring 0 fault on a user +//! address and live. +//! +//! The user address is checked against the half's top by the caller; each +//! routine also replaces an address at or past it with 0 by `cmovae`, without +//! a branch, so a mispredicted check reads no kernel byte even speculatively +//! (`issues/user-pointer-checks-have-no-spectre-v1-fence.md`). +//! A copy through a user mapping of device memory makes only the accesses the +//! process could make through that same mapping with the same instruction. + +use toyos_userbound::USER_TOP; + +core::arch::global_asm!( + ".pushsection .text.toyos_uaccess, \"ax\"", + // toyos_copy_from_user(rdi = kernel dst, rsi = user src, rdx = len) -> rax = bytes not copied + ".balign 16", + ".global toyos_copy_from_user", + "toyos_copy_from_user:", + "xor ecx, ecx", + "mov rax, {top}", + "cmp rsi, rax", + "cmovae rsi, rcx", + "mov rcx, rdx", + "stac", + ".global toyos_copy_from_user_access", + "toyos_copy_from_user_access:", + "rep movsb", + "clac", + "mov rax, rcx", + "ret", + // toyos_copy_to_user(rdi = user dst, rsi = kernel src, rdx = len) -> rax = bytes not copied + ".balign 16", + ".global toyos_copy_to_user", + "toyos_copy_to_user:", + "xor ecx, ecx", + "mov rax, {top}", + "cmp rdi, rax", + "cmovae rdi, rcx", + "mov rcx, rdx", + "stac", + ".global toyos_copy_to_user_access", + "toyos_copy_to_user_access:", + "rep movsb", + "clac", + "mov rax, rcx", + "ret", + // toyos_clear_user(rdi = user dst, rsi = len) -> rax = bytes not cleared + ".balign 16", + ".global toyos_clear_user", + "toyos_clear_user:", + "xor ecx, ecx", + "mov rax, {top}", + "cmp rdi, rax", + "cmovae rdi, rcx", + "mov rcx, rsi", + "xor eax, eax", + "stac", + ".global toyos_clear_user_access", + "toyos_clear_user_access:", + "rep stosb", + "clac", + "mov rax, rcx", + "ret", + ".popsection", + top = const USER_TOP, +); + +unsafe extern "C" { + fn toyos_copy_from_user(dst: *mut u8, user: u64, len: usize) -> usize; + fn toyos_copy_to_user(user: u64, src: *const u8, len: usize) -> usize; + fn toyos_clear_user(user: u64, len: usize) -> usize; + static toyos_copy_from_user_access: u8; + static toyos_copy_to_user_access: u8; + static toyos_clear_user_access: u8; +} + +/// Bytes a string instruction takes: its `rep` prefix and its opcode. +const REP_STRING_LEN: u64 = 2; + +/// Copy `len` bytes from user address `user` to `dst`; answers the bytes not +/// copied. +/// +/// # Safety +/// `dst` is writable kernel memory for `len` bytes; `[user, user + len)` lies +/// below [`USER_TOP`]; no lock is held (`crate::user_ptr`). +pub unsafe fn copy_from_user(dst: *mut u8, user: u64, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_copy_from_user(dst, user, len) } +} + +/// Copy `len` bytes from `src` to user address `user`; answers the bytes not +/// copied. +/// +/// # Safety +/// As [`copy_from_user`], with `src` readable. +pub unsafe fn copy_to_user(user: u64, src: *const u8, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_copy_to_user(user, src, len) } +} + +/// Zero `len` bytes at user address `user`; answers the bytes not zeroed. +/// +/// # Safety +/// As [`copy_from_user`]. +pub unsafe fn clear_user(user: u64, len: usize) -> usize { + // SAFETY: the caller's contract; the routine touches nothing else. + unsafe { toyos_clear_user(user, len) } +} + +/// Where a Ring 0 fault at `rip` resumes when it cannot be served: `Some` only +/// for the string instruction of one of the routines above. +pub fn fixup(rip: u64) -> Option { + let accesses = [ + &raw const toyos_copy_from_user_access as u64, + &raw const toyos_copy_to_user_access as u64, + &raw const toyos_clear_user_access as u64, + ]; + accesses.contains(&rip).then_some(rip + REP_STRING_LEN) +} diff --git a/kernel/src/counters.rs b/kernel/src/counters.rs index 96db13a1003..d8b91e3296c 100644 --- a/kernel/src/counters.rs +++ b/kernel/src/counters.rs @@ -19,7 +19,6 @@ //! declaration declared it; that bring-up's sample reads each once, so a wrong //! verdict is `#GP` at boot, never inside a read. -use alloc::vec::Vec; use kernel::sched::task::WaitClass; use toyos_abi::counters::{Counter, Record, RECORD_BYTES}; @@ -253,10 +252,9 @@ pub fn read(rights: Rights, out: &mut UserBytesMut) -> Result = (0..cpus).flat_map(|cpu| record(cpu, generation, rights).encode().0).collect(); - out.write_at(0, &records); + for cpu in 0..cpus { + out.write_at(cpu * RECORD_BYTES, &record(cpu, generation, rights).encode().0)?; + } Ok(cpus) } diff --git a/kernel/src/frame_bytes.rs b/kernel/src/frame_bytes.rs index 2aebed946bb..4077f82a8a8 100644 --- a/kernel/src/frame_bytes.rs +++ b/kernel/src/frame_bytes.rs @@ -3,22 +3,40 @@ //! pipe's stream and a connection direction's messages. //! //! The frames are kernel-only and written before they are read, so they are -//! never zeroed. Every call is made under its object's lock, which is the -//! only thing that reaches them. - -use core::ops::Range; +//! never zeroed. **The object's lock is never held across a user copy or the +//! frame allocator**: under it a write or a read claims its bytes and is handed +//! its first chunk's frames ([`FrameBytes::first_fill`], +//! [`FrameBytes::first_drain`]), and ends its claim; [`supply`], [`fill`], +//! [`drain`] and [`give_back`] take frames, copy and give frames back between +//! its holds. The claims are the data-race argument (`kernel::pipe::Bytes`). +//! +//! One claim of each kind at a time: another copy waits for its [`turn`], on +//! its end's watch, and each claim's end is answered by [`wake`]. +//! +//! A small transfer is not claimed across a copy: it is copied between its +//! caller and a [`Bounce`] on the caller's stack with no lock held, and +//! between the bounce and the frames in one hold ([`FrameBytes::put`], +//! [`FrameBytes::take`]). One hold, where a claim takes two or more. +use alloc::collections::VecDeque; +use core::mem::MaybeUninit; use alloc::sync::Arc; +use alloc::vec::Vec; -use kernel::pipe::{Bytes, MOST_FRAMES}; +use kernel::pipe::{Bytes, End, Ended, FRAME, MOST_FRAMES}; use kernel::sched::job::Job; +use kernel::sched::task::WaitClass; use crate::mm::charge::{self, Charge, Exhausted}; use crate::mm::{pmm, DirectMap}; -use crate::user_ptr::{UserBytes, UserBytesMut}; +use crate::scheduler::Parkable; +use crate::sync::{Lock, LockGuard}; +use crate::time::Deadline; +use crate::user_ptr::{Faulted, UserBytes, UserBytesMut}; +use crate::watch::{self, Cancelled, Watch}; /// The frames and the heap their list takes, charged to one job; the frames -/// go back when it drops. +/// go back when it drops, which no claim may be held across. pub struct FrameBytes { bytes: Bytes, list: Charge, @@ -33,7 +51,7 @@ impl FrameBytes { self.bytes.len() } - /// Room for more bytes, up to the capacity. + /// Room for a write to claim, up to the capacity. pub fn space(&self) -> usize { self.bytes.space() } @@ -47,130 +65,437 @@ impl FrameBytes { self.list.job() } - /// Append the first `n` bytes of `buf`, taking the frames they need, all - /// of them or none: a refusal appends nothing. `n` is at most - /// [`Self::space`]. - pub fn write(&mut self, buf: &UserBytes, n: usize) -> Result<(), Exhausted> { - self.take_frames(n)?; - let bytes = &mut self.bytes; - // One walk of the user window, each pinned piece copied into the - // frames it covers. - buf.read_pieces(0, n, |from, at, piece| { - let mut gather = Gather::NONE; - bytes.place(at, piece, |frame, within, off| { - // SAFETY: `from` is valid for `piece` bytes until this walk's - // callback returns, and `off + within.len()` is at most - // `piece`; the frame is this object's alone, reached only - // under its lock, and no process maps it. - unsafe { gather.add(from.add(off), frame_bytes(frame, within.clone()), within.len()) } - }); - // SAFETY: as above, before the callback returns. - unsafe { gather.flush() }; + /// Whether `end`'s claim is held, for a copy that will not wait for it. + pub fn claim_held(&self, end: End) -> bool { + self.bytes.claim_held(end) + } + + /// Whether a copy at `end` may claim in this hold; `false` asks the + /// claim held to wake it as it ends (`kernel::pipe::Bytes::turn`). + pub fn turn(&mut self, end: End) -> bool { + self.bytes.turn(end) + } + + /// Claim `n` bytes of room, at most [`Self::space`], for one write, in + /// its [`turn`]. + pub fn claim_write(&mut self, n: usize) { + self.bytes.claim_write(n); + } + + /// Hold the claimed write's bytes, ending its claim. + pub fn commit(&mut self) -> Ended { + self.bytes.commit() + } + + pub fn abandon_write(&mut self) -> Ended { + self.bytes.abandon_write() + } + + /// Write `src`, at most [`Self::space`], whole in this hold, in its + /// [`turn`]: `false`, claiming nothing, where it would want frames, which + /// a claim takes ([`supply`]). It owes no wake. + pub fn put(&mut self, src: &[u8]) -> bool { + self.bytes.put(src.len(), |frame, within, at| { + // SAFETY: `within` is inside the frame, which the direct map covers + // and only this hold's write reaches; `src` holds `at + within.len()`. + unsafe { copy_to_frame(*frame, within, &src[at..]) } + }) + } + + /// Copy `src` into the claimed write, whose frames are all supplied, and + /// commit it. + pub fn place_all(&mut self, src: &[u8]) -> Ended { + self.bytes.place(0, src.len(), |frame, within, at| { + // SAFETY: as in `put`, under the claim this hold ends. + unsafe { copy_to_frame(*frame, within, &src[at..]) } }); - bytes.commit(n); - Ok(()) + self.bytes.commit() } - /// Move the first `n` bytes held into the start of `buf`, giving back the - /// spares the read leaves past those kept. `n` is at most [`Self::len`]. - pub fn read(&mut self, buf: &mut UserBytesMut, n: usize) { - let bytes = &mut self.bytes; - buf.write_pieces(0, n, |to, at, piece| { - let mut gather = Gather::NONE; - bytes.peek(at, piece, |frame, within, off| { - // SAFETY: `to` is valid for `piece` bytes until this walk's - // callback returns, and `off + within.len()` is at most - // `piece`; the frame is as in `write`. - unsafe { gather.add(frame_bytes(frame, within.clone()), to.add(off), within.len()) } - }); - // SAFETY: as above, before the callback returns. - unsafe { gather.flush() }; + /// Read the `n` oldest bytes, at most [`Self::len`] and `N`, into `into` + /// whole in this hold, in its [`turn`]; answers the spares it leaves, for + /// [`give_back`]. It owes no wake. + pub fn take(&mut self, n: usize, into: &mut Bounce) -> Spares { + assert!(n <= N, "frame_bytes: a {n}-byte take past a {N}-byte bounce"); + let to = into.bytes.as_mut_ptr().cast::(); + self.bytes.take(n, |frame, within, at| { + // SAFETY: `within` is inside the frame, which the direct map + // covers; `at + within.len()` is at most `n`, inside `into`. + unsafe { + let from = DirectMap::from_phys(*frame + within.start as u64).as_mut_ptr::(); + core::ptr::copy_nonoverlapping(from, to.add(at), within.len()); + } }); - bytes.consume(n); - // SAFETY: a spare holds no byte, and nothing but this list names it. - unsafe { pmm::free_held(self.list.job(), self.bytes.surplus()) }; + into.len = n; + Spares::take(self) } - fn take_frames(&mut self, n: usize) -> Result<(), Exhausted> { - let wants = self.bytes.wants(n); - if wants == 0 { - return Ok(()); - } - let frames = self.bytes.frames() + wants; - if frames > self.bytes.listed() { - let list = frames.next_power_of_two().min(MOST_FRAMES); - self.list.resize(charge::array::(list))?; - self.bytes.list(list); + /// Claim the `n` oldest bytes, at most [`Self::len`], for one read, in + /// its [`turn`]. + pub fn claim_read(&mut self, n: usize) { + self.bytes.claim_read(n); + } + + /// Let the claimed read's bytes go, ending its claim; answers the first + /// of the spares it leaves past those kept, for [`give_back`]. + pub fn consume(&mut self) -> (Spares, Ended) { + let ended = self.bytes.consume(); + (Spares::take(self), ended) + } + + /// Whether a copy may still reach these frames: while it does, they are + /// not given back. + pub fn claimed(&self) -> bool { + self.bytes.claimed() + } + + /// The claimed write's first `n` bytes, at most a chunk, as runs to copy + /// into, or the frames to take first ([`supply`]). + pub fn first_fill(&mut self, n: usize) -> Fill { + match self.bytes.wants() { + 0 => Fill::Runs(self.placed(0, n)), + wants => { + let frames = self.bytes.frames() + wants; + let list = (frames > self.bytes.listed()).then(|| frames.next_power_of_two().min(MOST_FRAMES)); + Fill::Supply(Needs { wants, list, job: Arc::clone(self.job()) }) + } } - let bytes = &mut self.bytes; - pmm::alloc_held(self.list.job(), wants, |frame| bytes.add(frame)).then_some(()).ok_or(Exhausted) + } + + /// The claimed read's first `n` bytes from `at`, at most a chunk, as runs + /// to copy out of. + pub fn first_drain(&self, at: usize, n: usize) -> Runs { + self.peeked(at, n) + } + + fn placed(&mut self, at: usize, len: usize) -> Runs { + let mut runs = Runs::new(at, len.min(CHUNK)); + self.bytes.place(at, runs.len, |frame, within, _| runs.add(*frame, within)); + runs + } + + fn peeked(&self, at: usize, len: usize) -> Runs { + let mut runs = Runs::new(at, len.min(CHUNK)); + self.bytes.peek(at, runs.len, |frame, within, _| runs.add(*frame, within)); + runs } } impl Drop for FrameBytes { fn drop(&mut self) { - // SAFETY: nothing but this list names its frames. + // SAFETY: nothing but this list names its frames, and no claim's copy + // reaches them: `clear` refuses under one. unsafe { pmm::free_held(self.list.job(), self.bytes.clear()) }; } } -/// One copy gathered from runs that continue each other in both source and -/// destination, so a user page's piece over two physically adjacent frames -/// is one copy, not two. -struct Gather { - from: *const u8, - to: *mut u8, +/// The most bytes a pipe's read or write bounces through its caller's stack +/// rather than claiming them across its copy. +pub const BOUNCE: usize = 4096; + +/// Up to `N` bytes of one transfer, on its caller's stack: filled from the +/// caller or the frames, and read back only as far as it was filled. +pub struct Bounce { + bytes: [MaybeUninit; N], len: usize, } -impl Gather { - const NONE: Self = Self { from: core::ptr::null(), to: core::ptr::null_mut(), len: 0 }; +impl Bounce { + pub const fn new() -> Self { + Self { bytes: [MaybeUninit::uninit(); N], len: 0 } + } - /// Add `len` bytes from `from` to `to`, copying what was gathered first - /// unless these continue it. - /// - /// # Safety - /// `from` is readable and `to` writable for `len` bytes, and stay so - /// until the next [`Self::flush`]; the two never overlap. - #[inline(always)] - unsafe fn add(&mut self, from: *const u8, to: *mut u8, len: usize) { - if self.len > 0 && self.from.wrapping_add(self.len) == from && self.to.wrapping_add(self.len) == to { - self.len += len; - return; + /// Copy `buf`'s bytes, at most `N`, in with no lock held. Filled in + /// place: a bounce is never moved, which would copy all of it. + pub fn read(&mut self, buf: &UserBytes) -> Result<&Self, Faulted> { + assert!(buf.len() <= N, "frame_bytes: a {}-byte bounce", buf.len()); + // SAFETY: the buffer is writable for `N` bytes and `buf.len()` is at most that. + unsafe { buf.read_raw(0, self.bytes.as_mut_ptr().cast(), buf.len())? }; + self.len = buf.len(); + Ok(self) + } + + pub fn bytes(&self) -> &[u8] { + // SAFETY: the first `len` bytes were written by `read` or `FrameBytes::take`. + unsafe { core::slice::from_raw_parts(self.bytes.as_ptr().cast(), self.len) } + } +} + +/// Copy `src`'s first `within.len()` bytes into `frame` at `within`. +/// +/// # Safety +/// `within` lies inside the frame, which nothing else reaches meanwhile. +unsafe fn copy_to_frame(frame: u64, within: core::ops::Range, src: &[u8]) { + let src = &src[..within.len()]; + // SAFETY: the caller's contract; the direct map covers every frame. + unsafe { + let to = DirectMap::from_phys(frame + within.start as u64).as_mut_ptr::(); + core::ptr::copy_nonoverlapping(src.as_ptr(), to, src.len()); + } +} + +/// The token a copy waits under for its end's claim to end: a claim's end +/// wakes it and no poll, and any post on its watch wakes it to look again. +const TURN: u64 = 1; + +/// `lock`, held once no copy holds `end`'s claim on the bytes `of` names, +/// parking on `watch`, `end`'s, while one does. +pub fn turn<'l, S>( + p: &Parkable, + lock: &'l Lock, + of: impl Fn(&mut S) -> &mut FrameBytes, + end: End, + watch: &Watch, + class: WaitClass, +) -> Result, Cancelled> { + loop { + let mut state = lock.lock(); + if of(&mut state).turn(end) { + return Ok(state); + } + drop(state); + // Each look that finds the claim held asks to be woken again, so a + // claim made between a wake and its look wakes this wait as it ends. + watch::wait_until(p, watch, TURN, class, Deadline::never(), || of(&mut lock.lock()).turn(end))?; + } +} + +/// Do what a claim's end owes, with its object's lock let go, on its end's +/// `watch`. +pub fn wake(watch: &Watch, ended: Ended) { + if ended.room { + watch.post(); + } else if ended.turn { + watch.post_n(TURN, usize::MAX); + } +} + +/// What a claimed write's first chunk is: its runs, or the frames it needs. +#[expect(clippy::large_enum_variant, reason = "one lives on its writer's stack for a claim; boxed, every write would allocate")] +pub enum Fill { + Runs(Runs), + Supply(Needs), +} + +/// The frames a claimed write wants and the longer list they need, if any. +pub struct Needs { + wants: usize, + list: Option, + job: Arc, +} + +/// Take what `needs` names with `lock` let go, then hand them to the claimed +/// write and answer its first `n` bytes' runs, as [`FrameBytes::first_fill`] +/// would; refused, the claim holds none of them and is the caller's to +/// abandon. Only the claim's holder adds frames, and a read meanwhile only +/// lowers what it wants. +pub fn supply(lock: &Lock, of: impl Fn(&mut S) -> &mut FrameBytes, needs: Needs, n: usize) -> Result { + let Needs { wants, list, job } = needs; + let list = match list { + Some(len) => Some((Charge::new(&job, charge::array::(len))?, VecDeque::with_capacity(len))), + None => None, + }; + let mut frames = Vec::new(); + frames.try_reserve_exact(wants).map_err(|_| Exhausted)?; + if !pmm::alloc_held(&job, wants, |frame| frames.push(frame)) { + return Err(Exhausted); + } + let (runs, old) = { + let mut state = lock.lock(); + let held = of(&mut state); + // A read meanwhile may have made the longer list unnecessary: the + // unused one goes back as the old one does, with the lock let go. + let old = list.map(|(charge, list)| { + if held.bytes.frames() + held.bytes.wants() > held.bytes.listed() { + (core::mem::replace(&mut held.list, charge), held.bytes.relist(list)) + } else { + (charge, list) + } + }); + for frame in frames.drain(..held.bytes.wants()) { + held.bytes.add(frame); + } + (held.placed(0, n), old) + }; + // SAFETY: taken above and named by nothing since. + unsafe { pmm::free_held(&job, frames.drain(..)) }; + drop(old); + Ok(runs) +} + +/// Bytes of a claim one hold of the lock hands out frames for. +const CHUNK: usize = 16 * FRAME; + +/// One chunk of a claim's bytes, `len` of them from `at`, as runs inside its +/// frames: each run's direct-map address and length, in order and continuing +/// each other in the claim. Its frames are the claim's, which only its holder +/// reaches. +#[derive(Clone, Copy)] +pub struct Runs { + at: usize, + len: usize, + runs: [(u64, usize); CHUNK / FRAME + 1], + count: usize, +} + +impl Runs { + fn new(at: usize, len: usize) -> Self { + Self { at, len, runs: [(0, 0); CHUNK / FRAME + 1], count: 0 } + } + + /// Whether claim byte `pos` is one of these. + fn holds(&self, pos: usize) -> bool { + (self.at..self.at + self.len).contains(&pos) + } + + fn add(&mut self, frame: u64, within: core::ops::Range) { + if within.end > FRAME { + past_a_frame(within); } - // SAFETY: what was gathered meets this function's contract. - unsafe { self.flush() }; - *self = Self { from, to, len }; + self.runs[self.count] = (frame + within.start as u64, within.len()); + self.count += 1; } - /// Copy what was gathered. - /// - /// # Safety - /// As [`Self::add`] for every run gathered. + /// Hand `copy` each run's part of the chunk's `[at, at + len)`: its + /// direct-map pointer, where it starts among the `len`, and its length. #[inline(always)] - unsafe fn flush(&mut self) { - if self.len > 0 { - // SAFETY: the runs gathered are contiguous on both sides and - // each met `add`'s contract. - unsafe { core::ptr::copy_nonoverlapping(self.from, self.to, self.len) }; + fn each(&self, at: usize, len: usize, mut copy: impl FnMut(*mut u8, usize, usize)) { + let mut start = 0; + for &(phys, run) in &self.runs[..self.count] { + let (lo, hi) = (start.max(at), (start + run).min(at + len)); + if lo < hi { + // SAFETY: inside the run's frame, which the direct map covers. + let to = unsafe { DirectMap::from_phys(phys).as_mut_ptr::().add(lo - start) }; + copy(to, lo - at, hi - lo); + } + start += run; + if start >= at + len { + break; + } + } + } +} + +/// Copy the first `n` bytes of `buf` into the claimed write, `n` at most its +/// claim, from its `first` chunk's runs and then a chunk at a time: each +/// chunk's frames handed out under `lock`, its copy made with the lock let go. +/// A user byte its process cannot reach stops it. +pub fn fill(lock: &Lock, of: impl Fn(&mut S) -> &mut FrameBytes, buf: &UserBytes, n: usize, first: Runs) -> Result<(), Faulted> { + let mut runs = first; + let mut done = 0; + while done < n { + if !runs.holds(done) { + runs = of(&mut lock.lock()).placed(done, n - done); + } + let (base, len) = (done - runs.at, (runs.at + runs.len - done).min(n - done)); + // SAFETY: each run is the claim's frames, which no process maps and + // only this claim's holder reaches, and `at + n` stays inside `buf`. + Span::copy(&runs, base, len, done, |frame, at, n| unsafe { buf.read_raw(at, frame, n) })?; + done += len; + } + Ok(()) +} + +/// Copy the `n` claimed bytes `at` past the claimed read's first into the +/// start of `buf`, as [`fill`] copies, from `runs` where they hold them. +pub fn drain(lock: &Lock, of: impl Fn(&mut S) -> &mut FrameBytes, at: usize, buf: &mut UserBytesMut, n: usize, runs: Runs) -> Result<(), Faulted> { + let mut runs = runs; + let mut done = 0; + while done < n { + let pos = at + done; + if !runs.holds(pos) { + runs = of(&mut lock.lock()).peeked(pos, n - done); + } + let (base, len) = (pos - runs.at, (runs.at + runs.len - pos).min(n - done)); + // SAFETY: as in `fill`. + Span::copy(&runs, base, len, done, |frame, at, n| unsafe { buf.write_raw(at, frame, n) })?; + done += len; + } + Ok(()) +} + +/// Runs of a chunk gathered while each continues the last in the direct map, +/// so a run over two physically adjacent frames is one user copy, not two. +struct Span { + frame: *mut u8, + at: usize, + len: usize, +} + +impl Span { + /// Hand `copy` the chunk's `[base, base + len)` as gathered spans: a + /// frame pointer, the user offset `user` plus where the span starts among + /// the `len`, and the span's length. + fn copy(runs: &Runs, base: usize, len: usize, user: usize, mut copy: impl FnMut(*mut u8, usize, usize) -> Result<(), Faulted>) -> Result<(), Faulted> { + let mut span = Span { frame: core::ptr::null_mut(), at: 0, len: 0 }; + let mut stopped = Ok(()); + runs.each(base, len, |frame, off, run| { + if stopped.is_err() { + return; + } + if span.len > 0 && span.frame.wrapping_add(span.len) == frame { + span.len += run; + return; + } + if span.len > 0 { + stopped = copy(span.frame, span.at, span.len); + } + span = Span { frame, at: user + off, len: run }; + }); + stopped?; + if span.len > 0 { + copy(span.frame, span.at, span.len)?; + } + Ok(()) + } +} + +/// Frames past those a pipe keeps, listed under its lock to give back with +/// it let go: at most a batch. +pub struct Spares { + frames: [u64; Spares::BATCH], + count: usize, + job: Option>, +} + +impl Spares { + const BATCH: usize = 32; + + pub const NONE: Self = Self { frames: [0; Self::BATCH], count: 0, job: None }; + + fn take(held: &mut FrameBytes) -> Self { + let mut spares = Self::NONE; + for (slot, frame) in spares.frames.iter_mut().zip(held.bytes.surplus(Self::BATCH)) { + (*slot, spares.count) = (frame, spares.count + 1); + } + if spares.count > 0 { + spares.job = Some(Arc::clone(held.job())); + } + spares + } +} + +impl Drop for Spares { + fn drop(&mut self) { + if let Some(job) = &self.job { + // SAFETY: a spare holds no byte, and the list that named it no longer does. + unsafe { pmm::free_held(job, self.frames[..self.count].iter().copied()) }; } - *self = Self::NONE; } } -/// `within` of a frame an object holds, which no process maps and only that -/// object's lock reaches. -#[inline(always)] -fn frame_bytes(frame: &u64, within: Range) -> *mut u8 { - if within.end > kernel::pipe::FRAME { - past_a_frame(within); +/// Give back `first`, the spares a read's consume listed, and any left past +/// them a batch per hold of `lock`, each with it let go. +pub fn give_back(lock: &Lock, of: impl Fn(&mut S) -> &mut FrameBytes, first: Spares) { + let mut spares = first; + while spares.count == Spares::BATCH { + drop(spares); + spares = Spares::take(of(&mut lock.lock())); } - // SAFETY: inside the frame, which the direct map covers. - unsafe { DirectMap::from_phys(*frame).as_mut_ptr::().add(within.start) } } #[cold] #[inline(never)] -fn past_a_frame(within: Range) -> ! { +fn past_a_frame(within: core::ops::Range) -> ! { panic!("frame_bytes: bytes {within:?} past a frame") } diff --git a/kernel/src/loader/start.rs b/kernel/src/loader/start.rs index 885c259c694..8f64cb98a38 100644 --- a/kernel/src/loader/start.rs +++ b/kernel/src/loader/start.rs @@ -270,7 +270,7 @@ pub fn build_child_handles(endow: &UserBytes, labels: &[u8], pager: bool) -> Res return Err(SyscallError::InvalidArgument.into()); } let mut raw = alloc::vec![0u8; endow.len()]; - endow.read_at(0, &mut raw); + endow.read_at(0, &mut raw).map_err(SyscallError::from)?; let job = child_job(&raw, labels)?; Ok(PendingHandles { table: HandleTable::new(&job), endow: raw, labels: labels.to_vec(), job, pager }) } diff --git a/kernel/src/log/user.rs b/kernel/src/log/user.rs index fb7dd7059a7..6dc869a18dc 100644 --- a/kernel/src/log/user.rs +++ b/kernel/src/log/user.rs @@ -52,7 +52,10 @@ impl RecordSink for UserRecords<'_, '_> { return false; } let bytes = toyos_abi::usersafe::bytes(record); - self.out.write_at(self.written * bytes.len(), bytes); + // A record the reader cannot take ends the reading as a full window does. + if self.out.write_at(self.written * bytes.len(), bytes).is_err() { + return false; + } self.written += 1; true } diff --git a/kernel/src/message.rs b/kernel/src/message.rs index 206d0769e97..3764e6439b0 100644 --- a/kernel/src/message.rs +++ b/kernel/src/message.rs @@ -5,21 +5,36 @@ //! of it queued, and taken whole or left queued (`kernel::message`). Its //! bytes are kept in frames counted to the job that made the connection //! (`crate::frame_bytes`), its handles beside them in the same order. +//! +//! **No user copy runs under the queue's lock.** A message, at most +//! [`MAX_MESSAGE_LEN`] bytes, bounces through its caller's stack +//! ([`MessageBounce`]): a send copies it in with no lock held and queues it +//! whole in one hold ([`QueueRef::send_now`]), and a receive takes it whole +//! and installs its handles in one hold ([`QueueRef::recv_now`]), copying it +//! out after. A refusal takes nothing and queues nothing. A send whose bytes +//! want frames claims its room and takes them with the lock let go +//! ([`QueueRef::claim_send`]); one such claim at a time, and a send that meets +//! it waits for its turn (`frame_bytes::turn`). +use alloc::collections::VecDeque; use alloc::sync::Arc; use alloc::vec::Vec; use kernel::message::{Head, Queue, Untaken}; use kernel::pipe::{End, Holders, Released}; +use kernel::sched::task::WaitClass; use toyos_abi::handle::RawHandle; -use toyos_abi::syscall::{MAX_QUEUED_HANDLES, MAX_QUEUED_MESSAGES, MAX_TRANSFER_HANDLES}; +use toyos_abi::syscall::{MAX_MESSAGE_LEN, MAX_QUEUED_HANDLES, MAX_QUEUED_MESSAGES, MAX_TRANSFER_HANDLES}; -use crate::frame_bytes::FrameBytes; +use crate::frame_bytes::{self, Bounce, Fill, FrameBytes, Spares}; use crate::mm::charge::{self, Charge, Exhausted}; use crate::object::handle::{HandleEntry, HandleTable}; -use crate::sync::Lock; -use crate::user_ptr::{UserBytes, UserBytesMut}; -use crate::watch::Watch; +use crate::scheduler::Parkable; +use crate::sync::{Lock, LockGuard}; +use crate::watch::{Cancelled, Watch}; + +/// A whole message's bytes, on its sender's or receiver's stack. +pub type MessageBounce = Bounce; /// A reference to a queue that holds neither end, as [`crate::pipe::PipeRef`] /// is a pipe's. @@ -94,6 +109,34 @@ struct State { rt_boost_pending: bool, } +impl State { + /// Every message nobody can take any more, to give back with the lock + /// let go; kept while a claim's copy may still reach its bytes, whose end + /// asks again. + fn untaken(&mut self) -> Option<(FrameBytes, VecDeque)> { + let owed = !self.holders.held(End::Read) && !self.bytes.claimed() && (self.bytes.frames() > 0 || !self.queue.is_empty()); + owed.then(|| { + let job = Arc::clone(self.bytes.job()); + (core::mem::replace(&mut self.bytes, FrameBytes::new(&job)), self.queue.clear()) + }) + } + + /// What a claim's end, already made, leaves to do once the lock is let + /// go: `spares` given back, what nobody can take, and its wake. + fn ended(&mut self, ended: kernel::pipe::Ended, spares: Spares) -> Ended { + Ended { spares, untaken: self.untaken(), ended, boost: false } + } +} + +/// [`State::ended`]'s answer. +struct Ended { + spares: Spares, + untaken: Option<(FrameBytes, VecDeque)>, + ended: kernel::pipe::Ended, + /// The real-time window a sender lent, taken with the message. + boost: bool, +} + /// Why a message was not sent; nothing of it was. pub enum SendRefused { /// No room for the whole message now: wait for [`QueueRef::fits`]. @@ -102,19 +145,86 @@ pub enum SendRefused { Gone, /// Its bytes' frames could not be taken; no wait makes them appear. NoMemory, + /// A kill ended the wait for another send's claim. + Cancelled, +} + +/// Why [`QueueRef::send_now`] queued nothing. +pub enum Unsent { + Refused(SendRefused), + /// It cannot be made whole in this hold: another send's claim is held, + /// or its bytes want frames. A claim sends it. + Claim, +} + +/// A message taken whole by [`QueueRef::recv_now`], its bytes in the +/// caller's bounce: what is left of the hold is done as it drops, with every +/// lock let go. Spares past its batch stay listed for the next take. +pub struct Received { + pub head: Head, + pub installed: [RawHandle; MAX_TRANSFER_HANDLES], + _spares: Spares, + boost: bool, +} + +impl Drop for Received { + fn drop(&mut self) { + if self.boost { + crate::scheduler::boost_current_rt_inherited(); + } + } } -/// What a receive answered. -pub enum Taken { - /// A whole message, its bytes copied out and its handles installed at - /// these numbers, the first `Head::handles` of them. - Message(Head, [RawHandle; MAX_TRANSFER_HANDLES]), +/// Why a receive claimed no message. +pub enum Unclaimed { /// Nothing queued, and a sender is left to send. Empty, /// Nothing queued, and no sender is left. Gone, /// The front message stays queued. Untaken(Untaken), + /// The front message stays queued until the table grows by this many + /// slots, which it may not under the queue's lock. + Room(usize), +} + +/// A send's room, claimed, with its frames taken: its bytes placed and the +/// message queued whole by [`Self::commit`], or dropped first, nothing of it. +pub struct SendClaim<'q> { + queue: &'q QueueRef, + src: &'q [u8], + /// Set by the commit, which ends the claim. + ended: Option, +} + +impl SendClaim<'_> { + /// Queue the message with `batch`, its handles; a refusal hands the batch + /// back and queues nothing. + pub fn commit(&mut self, mut batch: Vec) -> Result<(), (Vec, SendRefused)> { + let mut state = self.queue.0.state.lock(); + if !state.holders.held(End::Read) { + return Err((batch, SendRefused::Gone)); + } + // It fits: the claim was made where it did, and only this claim adds. + let ended = state.bytes.place_all(self.src); + state.queue.push(self.src.len(), batch.drain(..)); + self.ended = Some(state.ended(ended, Spares::NONE)); + drop(state); + // The empty batch's storage goes back with the lock let go. + drop(batch); + Ok(()) + } +} + +impl Drop for SendClaim<'_> { + fn drop(&mut self) { + let ended = self.ended.take().unwrap_or_else(|| { + let mut state = self.queue.0.state.lock(); + let ended = state.bytes.abandon_write(); + state.ended(ended, Spares::NONE) + }); + self.queue.finish(End::Write, ended); + } } /// Make a queue, answering its two ends. @@ -144,54 +254,105 @@ impl QueueRef { self.0.state.lock().holders.hold(end); } - /// Queue `bytes` and `batch` as one message, or nothing of it. A refusal - /// hands the batch back: dropped here, it would be capabilities destroyed - /// silently. - pub fn send(&self, bytes: &UserBytes, batch: Vec) -> Result<(), (Vec, SendRefused)> { + /// Queue `src`, already copied in, and `batch`, its handles, whole in + /// this hold, called under the sender's process lock as its handles move; + /// a refusal hands the batch back and queues nothing. No copy waits on it. + pub fn send_now(&self, src: &[u8], mut batch: Vec) -> Result<(), (Vec, Unsent)> { let mut state = self.0.state.lock(); + if !state.bytes.turn(End::Write) { + return Err((batch, Unsent::Claim)); + } if !state.holders.held(End::Read) { - return Err((batch, SendRefused::Gone)); + return Err((batch, Unsent::Refused(SendRefused::Gone))); } - if !state.queue.fits(state.bytes.space(), bytes.len(), batch.len()) { - return Err((batch, SendRefused::NoRoom)); + if !state.queue.fits(state.bytes.space(), src.len(), batch.len()) { + return Err((batch, Unsent::Refused(SendRefused::NoRoom))); } - if state.bytes.write(bytes, bytes.len()).is_err() { - return Err((batch, SendRefused::NoMemory)); + if !state.bytes.put(src) { + return Err((batch, Unsent::Claim)); } - state.queue.push(bytes.len(), batch.into_iter()); + state.queue.push(src.len(), batch.drain(..)); + drop(state); + // The empty batch's storage goes back with the queue's lock let go. + drop(batch); Ok(()) } - /// Take the front message into `head` and then `body`, its first bytes - /// filling `head`, and at most `cap` handles, each installed in `table` - /// under this queue's lock. - pub fn recv(&self, head: &mut UserBytesMut, body: &mut UserBytesMut, cap: usize, table: &mut HandleTable) -> Taken { - let (taken, boost) = { - let mut guard = self.0.state.lock(); - let State { bytes, queue, holders, rt_boost_pending } = &mut *guard; - // Bound, not the block's tail: a tail match's temporaries outlive `guard`. - let taken = match queue.pop(head.len() + body.len(), cap, |width| table.make_room(width)) { - Some(Ok((message, handles))) => { - let mut installed = [RawHandle(0); MAX_TRANSFER_HANDLES]; - for (slot, entry) in installed.iter_mut().zip(handles) { - *slot = table.install(entry).expect("room was made first"); - } - let first = message.len.min(head.len()); - bytes.read(head, first); - bytes.read(body, message.len - first); - (Taken::Message(message, installed), core::mem::take(rt_boost_pending)) - } - Some(Err(untaken)) => (Taken::Untaken(untaken), false), - None if holders.held(End::Write) => (Taken::Empty, false), - None => (Taken::Gone, false), - }; - taken + /// Take the front message whole into `into` and install its handles in + /// `table`, in this hold, where a reader holding `buf` bytes and `cap` + /// handles can take it, called under the receiver's process lock. A table + /// that must grow first answers [`Unclaimed::Room`], taking nothing. + pub fn recv_now(&self, table: &mut HandleTable, buf: usize, cap: usize, into: &mut MessageBounce) -> Result { + let mut state = self.0.state.lock(); + let head = match state.queue.front(buf, cap) { + Some(Ok(head)) => head, + Some(Err(untaken)) => return Err(Unclaimed::Untaken(untaken)), + None if state.holders.held(End::Write) => return Err(Unclaimed::Empty), + None => return Err(Unclaimed::Gone), + }; + if !table.has_room(head.handles) { + return Err(Unclaimed::Room(head.handles)); + } + let mut installed = [RawHandle(0); MAX_TRANSFER_HANDLES]; + let Some(Ok((message, handles))) = state.queue.pop(head.len, head.handles, |_| true) else { + unreachable!("message: a front its reader holds left the queue or was refused"); + }; + assert_eq!(message, head, "message: the front moved under its lock"); + for (slot, entry) in installed.iter_mut().zip(handles) { + *slot = table.install(entry).expect("the table had room"); + } + let spares = state.bytes.take(head.len, into); + let boost = core::mem::take(&mut state.rt_boost_pending); + Ok(Received { head, installed, _spares: spares, boost }) + } + + /// Claim room for `src`, a message's bytes, and its `handles`, and take + /// the frames its bytes want with the lock let go: for a send + /// [`Self::send_now`] could not make whole. Waits only for another send's + /// claim to end, never for room. + pub fn claim_send<'q>(&'q self, p: &Parkable, src: &'q [u8], handles: usize) -> Result, SendRefused> { + let len = src.len(); + let first = { + let mut state = self.turn(p, End::Write).map_err(|_| SendRefused::Cancelled)?; + if !state.holders.held(End::Read) { + return Err(SendRefused::Gone); + } + if !state.queue.fits(state.bytes.space(), len, handles) { + return Err(SendRefused::NoRoom); + } + state.bytes.claim_write(len); + state.bytes.first_fill(len) }; + let claim = SendClaim { queue: self, src, ended: None }; + if let Fill::Supply(needs) = first { + frame_bytes::supply(&self.0.state, |s| &mut s.bytes, needs, len).map_err(|Exhausted| SendRefused::NoMemory)?; + } + Ok(claim) + } + + /// The queue's lock, held in `end`'s turn to claim. + fn turn(&self, p: &Parkable, end: End) -> Result, Cancelled> { + frame_bytes::turn(p, &self.0.state, |s| &mut s.bytes, end, self.watch(end), WaitClass::Ipc) + } + + fn watch(&self, end: End) -> &Watch { + match end { + End::Read => &self.0.read_watch, + End::Write => &self.0.write_watch, + } + } + + /// What a claim's end left, done with the lock let go: frames go back to + /// the allocator, a handle's release can run another object's hook, and + /// the scheduler takes its own CPU-queue lock for a lent window. + fn finish(&self, end: End, ended: Ended) { + let Ended { spares, untaken, ended, boost } = ended; + frame_bytes::give_back(&self.0.state, |s| &mut s.bytes, spares); + drop(untaken); if boost { - // Outside the queue's lock: the scheduler takes its own CPU-queue lock. crate::scheduler::boost_current_rt_inherited(); } - taken + frame_bytes::wake(self.watch(end), ended); } /// A message to take, or no sender left: what a receive or a readable @@ -236,11 +397,7 @@ impl QueueRef { let (released, untaken) = { let mut state = self.0.state.lock(); let released = state.holders.release(end); - let untaken = (!state.holders.held(End::Read) && (state.bytes.frames() > 0 || !state.queue.is_empty())).then(|| { - let job = Arc::clone(state.bytes.job()); - (core::mem::replace(&mut state.bytes, FrameBytes::new(&job)), state.queue.clear()) - }); - (released, untaken) + (released, state.untaken()) }; // Outside the lock: frames go back to the allocator, and a handle's // release can run another object's hook. diff --git a/kernel/src/mm/space.rs b/kernel/src/mm/space.rs index d7792a78452..20f1321cf3d 100644 --- a/kernel/src/mm/space.rs +++ b/kernel/src/mm/space.rs @@ -817,15 +817,6 @@ impl AddressSpace { Some(phys + va % PAGE_SIZE) } - /// Whether every page of `[start, end)` is in a range that allows - /// `access` and is memory the kernel can copy through the direct map: - /// what a copy the kernel makes for a thread asks first. A device's - /// registers or firmware's framebuffer, an aperture, are no buffer. - pub fn allows(&self, start: u64, end: u64, access: Access) -> bool { - self.inventory - .covered_where(start & !(PAGE_SIZE - 1), end, |entry| entry.prot.allows(access) && entry.backed() && copyable(entry)) - } - /// The frame and protection a present leaf maps `va` with, populating /// nothing: for a check made where no fault may be taken. pub fn get(&self, va: u64) -> Option<(u64, policy::Prot)> { diff --git a/kernel/src/object/device.rs b/kernel/src/object/device.rs index 72be419d11e..d94778a00aa 100644 --- a/kernel/src/object/device.rs +++ b/kernel/src/object/device.rs @@ -136,24 +136,16 @@ impl DeviceClaim { self.info_read.load(Ordering::Relaxed) } - /// Write this claim's description into `buf`, minting handles on first call only. - pub fn describe( - &self, - table: &mut HandleTable, - buf: &mut crate::user_ptr::UserBytesMut, - ) -> u64 { + /// This claim's description, for its reader to copy out, minting handles + /// on first call only. + pub fn describe(&self, table: &mut HandleTable) -> Result, SyscallError> { let mut described = self.described.lock(); if described.bytes.is_none() { - match described.info.mint(table) { - Ok(minted) => described.bytes = Some(minted), - Err(e) => return e.to_u64(), - } + described.bytes = Some(described.info.mint(table)?); } - let bytes = described.bytes.as_deref().expect("just minted"); - let count = buf.len().min(bytes.len()); - buf.write_at(0, &bytes[..count]); + let bytes = described.bytes.clone().expect("just minted"); self.info_read.store(true, Ordering::Relaxed); - count as u64 + Ok(bytes) } } diff --git a/kernel/src/object/handle.rs b/kernel/src/object/handle.rs index 989f804fc4f..87fdf241561 100644 --- a/kernel/src/object/handle.rs +++ b/kernel/src/object/handle.rs @@ -198,6 +198,12 @@ impl HandleTable { self.grow_for(self.slots.len() + n.saturating_sub(self.free.len())) } + /// Whether `n` more `install`s all succeed without growing the table: an + /// install under another object's lock, where nothing may allocate. + pub fn has_room(&self, n: usize) -> bool { + self.slots.len() + n.saturating_sub(self.free.len()) <= self.slots.capacity().min(MAX_HANDLES) + } + /// Storage for `len` slots, doubling: `false` where it cannot be charged. fn grow_for(&mut self, len: usize) -> bool { if len <= self.slots.capacity() { diff --git a/kernel/src/object/ops.rs b/kernel/src/object/ops.rs index d78170a943d..6b0f3369dc5 100644 --- a/kernel/src/object/ops.rs +++ b/kernel/src/object/ops.rs @@ -97,17 +97,6 @@ pub fn close_all(table: &mut HandleTable) { drop(core::mem::replace(table, HandleTable::new(&job))); } -/// The pipe a read of `object` took from, held past the handle table's lock -/// to wake its writers. -pub fn read_pipe(object: &KObjectRef) -> Option { - pipe_read(object).cloned() -} - -/// The pipe a write to `object` filled, held the same way to wake its readers. -pub fn written_pipe(object: &KObjectRef) -> Option { - pipe_write(object).cloned() -} - /// The pipe a blocking read of `object` parks on. pub fn pipe_read(object: &KObjectRef) -> Option<&PipeRef> { match object { @@ -277,31 +266,53 @@ fn close_ends_polls(object: &KObjectRef) -> bool { } } -/// Read a device claim; takes `table` because describing a device installs handles into it. -pub fn read_device( - claim: &DeviceClaim, - table: &mut HandleTable, - buf: &mut UserBytesMut, -) -> Option { +/// What a device read answers, to copy out once no lock is held. +pub enum DeviceRead { + /// The claim's description, which the reader takes as much of as fits. + Description(alloc::boxed::Box<[u8]>), + Record(toyos_abi::pci::DeviceIrqRecord), + /// The answer, with nothing to copy. + Word(u64), +} + +impl DeviceRead { + /// Copy what it answers into `buf`, answering the syscall's word. + pub fn deliver(self, buf: &mut UserBytesMut) -> u64 { + match self { + Self::Description(bytes) => { + let count = buf.len().min(bytes.len()); + buf.write_at(0, &bytes[..count]).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| count as u64) + } + Self::Record(record) => buf + .write_at(0, record_bytes(&record)) + .map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| toyos_abi::pci::DeviceIrqRecord::SIZE as u64), + Self::Word(word) => word, + } + } +} + +/// Read a device claim into a reader with `room` bytes; takes `table` +/// because describing a device installs handles into it. +pub fn read_device(claim: &DeviceClaim, table: &mut HandleTable, room: usize) -> Option { + let refused = |e: SyscallError| Some(DeviceRead::Word(e.to_u64())); + let described = |table: &mut HandleTable| Some(claim.describe(table).map_or_else(|e| DeviceRead::Word(e.to_u64()), DeviceRead::Description)); match claim.class() { - device_registry::DeviceType::Framebuffer => Some(claim.describe(table, buf)), + device_registry::DeviceType::Framebuffer => described(table), // The description first and interrupts after: a driver reads what it // is driving once, and everything it reads afterwards is what its // device has been doing. device_registry::DeviceType::PciFunction => { if !claim.info_read() { - return Some(claim.describe(table, buf)); + return described(table); + } + if room < toyos_abi::pci::DeviceIrqRecord::SIZE { + return refused(SyscallError::InvalidArgument); } - if buf.len() < toyos_abi::pci::DeviceIrqRecord::SIZE { - return Some(SyscallError::InvalidArgument.to_u64()); + match claim.pci(crate::pcidev::take_record) { + Some(Ok(record)) => record.map(DeviceRead::Record), + Some(Err(e)) => refused(e), + None => refused(SyscallError::Gone), } - let record = match claim.pci(crate::pcidev::take_record) { - Some(Ok(record)) => record?, - Some(Err(refused)) => return Some(refused.to_u64()), - None => return Some(SyscallError::Gone.to_u64()), - }; - buf.write_at(0, record_bytes(&record)); - Some(toyos_abi::pci::DeviceIrqRecord::SIZE as u64) } // The PCI shape, but the description is what binds the ports to the // reader, and nothing after it answers any other process. @@ -309,11 +320,11 @@ pub fn read_device( let pid = crate::process::current_process(); if !claim.info_read() { if claim.isa(|row| crate::isa::bind(row, pid)).is_none() { - return Some(SyscallError::Gone.to_u64()); + return refused(SyscallError::Gone); } - return Some(claim.describe(table, buf)); + return described(table); } - let room = buf.len() >= toyos_abi::pci::DeviceIrqRecord::SIZE; + let room = room >= toyos_abi::pci::DeviceIrqRecord::SIZE; // The holder's check and the take are one borrow of the row. let taken = claim.isa(|row| { if !crate::isa::held_by(row, pid) { @@ -324,13 +335,11 @@ pub fn read_device( Ok(crate::isa::take_record(row)) } }); - let record = match taken { - Some(Ok(record)) => record?, - Some(Err(refused)) => return Some(refused.to_u64()), - None => return Some(SyscallError::Gone.to_u64()), - }; - buf.write_at(0, record_bytes(&record)); - Some(toyos_abi::pci::DeviceIrqRecord::SIZE as u64) + match taken { + Some(Ok(record)) => record.map(DeviceRead::Record), + Some(Err(e)) => refused(e), + None => refused(SyscallError::Gone), + } } } } @@ -355,7 +364,13 @@ fn record_bytes(record: &toyos_abi::pci::DeviceIrqRecord) -> &[u8] { /// Read whatever a handle names except a device claim; [`read_device`] needs the table mutably. pub fn try_read(object: &KObjectRef, buf: &mut UserBytesMut) -> Option { match object { - KObjectRef::PipeRead(r) => r.pipe().try_read(buf).map(|n| n as u64), + // A kill that ends the wait for another read's claim answers what + // a cancelled wait does; no answer reaches its Ring 3. + KObjectRef::PipeRead(r) => match r.pipe().read(&crate::scheduler::Parkable::at_entry(), buf) { + Ok(read) => read.map(|n| n as u64), + Err(pipe::ReadStopped::Cancelled) => Some(SyscallError::Gone.to_u64()), + Err(pipe::ReadStopped::Faulted) => Some(SyscallError::BadAddress.to_u64()), + }, KObjectRef::Device(_) => unreachable!("a device claim is read by `read_device`"), KObjectRef::Null(_) => Some(0), // An empty read listens and takes nothing: the reason would have nowhere to go. @@ -364,8 +379,7 @@ pub fn try_read(object: &KObjectRef, buf: &mut UserBytesMut) -> Option { Some(0) } KObjectRef::Notice(n) => n.read().map(|reason| { - buf.write_at(0, &[reason.to_byte()]); - 1 + buf.write_at(0, &[reason.to_byte()]).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 1) }), // A connection carries whole messages, which `SYS_MESSAGE_RECV` takes. KObjectRef::Connection(_) | KObjectRef::PipeWrite(_) | KObjectRef::Acceptor(_) | KObjectRef::Inbox(_) @@ -377,10 +391,20 @@ pub fn try_read(object: &KObjectRef, buf: &mut UserBytesMut) -> Option { fn write_pipe(object: &KObjectRef, buf: &UserBytes) -> Option { let pipe = pipe_write(object).expect("a pipe's write end writes its pipe"); - match pipe.write(buf) { + let Ok(wrote) = pipe.write(&crate::scheduler::Parkable::at_entry(), buf) else { + // As a read's cancelled wait for its turn answers. + return Some(SyscallError::Gone.to_u64()); + }; + pipe_written(wrote) +} + +/// What a pipe write answers its caller: `None` to wait for room. +pub fn pipe_written(wrote: pipe::PipeWrite) -> Option { + match wrote { // The reader is gone, not absent: `NotFound` means the name is not there and nothing else may say so. pipe::PipeWrite::BrokenPipe => Some(SyscallError::Gone.to_u64()), pipe::PipeWrite::NoMemory => Some(SyscallError::ResourceExhausted.to_u64()), + pipe::PipeWrite::Faulted => Some(SyscallError::BadAddress.to_u64()), pipe::PipeWrite::Wrote(n) => Some(n as u64), pipe::PipeWrite::NoRoom => None, } @@ -545,10 +569,10 @@ fn write_device(claim: &DeviceClaim, buf: &UserBytes) -> u64 { return SyscallError::PermissionDenied.to_u64(); } let mut word = [0u8; 4]; - let acknowledges = buf.len() == word.len() && { - buf.read_at(0, &mut word); - u32::from_ne_bytes(word) == ack - }; + if buf.len() == word.len() && buf.read_at(0, &mut word).is_err() { + return SyscallError::BadAddress.to_u64(); + } + let acknowledges = buf.len() == word.len() && u32::from_ne_bytes(word) == ack; let pid = crate::process::current_process(); // The holder's check and the unmask are one borrow of the row. let acked = claim.isa(|row| { diff --git a/kernel/src/object/paged.rs b/kernel/src/object/paged.rs index fb5693a90e3..c173bc75300 100644 --- a/kernel/src/object/paged.rs +++ b/kernel/src/object/paged.rs @@ -425,7 +425,10 @@ impl PagerObject { }; // SAFETY: a `PagedNotice` is `UserSafe`: every byte of it is a // field, and the batch outlives the copy. - unsafe { out.write_raw(filled * size, batch.as_ptr().cast::(), n * size) }; + if unsafe { out.write_raw(filled * size, batch.as_ptr().cast::(), n * size) }.is_err() { + // The notices taken are lost with the window that could not hold them. + return if filled > 0 { filled as u64 } else { SyscallError::BadAddress.to_u64() }; + } filled += n; if n < want { break; @@ -484,14 +487,24 @@ impl PagerObject { }; match step { Step::Copy(count) => { + // A chunk of frames is copied a frame at a time: populated + // first, fresh memory takes one hold rather than a fault + // for each page. A one-frame read, the common small one, + // would pay that hold on every call to save at most two faults. + if count > 1 { + out.fault_in(done, pieces[..count].iter().map(|piece| piece.len).sum()); + } + let mut faulted = false; for piece in &pieces[..count] { - match piece.phys { - Some(phys) => { - // SAFETY: `phys` is pinned by `pin_chunk`, - // and `within + len` lies inside its frame. - unsafe { out.write_raw(done, DirectMap::from_phys(phys).as_ptr::().add(piece.within), piece.len) }; - } + let copied = match piece.phys { + // SAFETY: `phys` is pinned by `pin_chunk`, and + // `within + len` lies inside its frame. + Some(phys) => unsafe { out.write_raw(done, DirectMap::from_phys(phys).as_ptr::().add(piece.within), piece.len) }, None => out.fill_zero(done, piece.len), + }; + if copied.is_err() { + faulted = true; + break; } done += piece.len; } @@ -510,6 +523,11 @@ impl PagerObject { if written { return SyscallError::WouldBlock.to_u64(); } + // After the checks: bytes copied before the fault count only + // as every copied chunk's do. + if faulted { + return if done > 0 { done as u64 } else { SyscallError::BadAddress.to_u64() }; + } } Step::Written => return SyscallError::WouldBlock.to_u64(), Step::Done => return done as u64, @@ -686,6 +704,8 @@ impl PagerObject { } let mut frames = [0u64; CHUNK]; let mut ends = [0u64; CHUNK]; + // The frames filled from `src`, in order: a fault stops the fill. + let mut filled = 0; for k in 0..reserved { let page_at = (first + k as u64) * PAGE; let from = done + k * PAGE as usize; @@ -703,17 +723,20 @@ impl PagerObject { } None => pmm::alloc_promised().into_raw(), }; - // SAFETY: a frame just drawn, which nothing else reaches yet. - unsafe { src.read_raw(from, DirectMap::from_phys(phys).as_mut_ptr::(), n) }; frames[k] = phys; ends[k] = page_at + (src.len() - from).min(PAGE as usize) as u64; + // SAFETY: a frame just drawn, which nothing else reaches yet. + if filled == k && unsafe { src.read_raw(from, DirectMap::from_phys(phys).as_mut_ptr::(), n) }.is_ok() { + filled += 1; + } } let mut refused = None; let mut stopped = false; { let mut state = self.state.lock(); for k in 0..reserved { - if stopped { + // A frame the owner's buffer could not fill goes back as a refused one does. + if stopped || k >= filled { let gone = state.gone; state.held.give_back(self.core.job(), gone, frames[k], false); continue; @@ -738,15 +761,18 @@ impl PagerObject { Err(why) => { let gone = state.gone; state.held.give_back(self.core.job(), gone, frames[k], false); - refused.get_or_insert(why); + refused.get_or_insert(word(why)); stopped = true; } } } } self.changed.post(); + if filled < reserved && !stopped { + refused.get_or_insert(SyscallError::BadAddress); + } if let Some(why) = refused { - return refusal(installed, word(why)); + return refusal(installed, why); } if stopped { return installed; @@ -775,6 +801,7 @@ impl PagerObject { let mut pinned = [(0u64, 0u64, 0usize, 0usize); CHUNK]; let mut count = 0; let mut stopped = None; + let mut faulted = false; let slot = { let mut state = self.state.lock(); let slot = match state.live(id) { @@ -803,26 +830,39 @@ impl PagerObject { slot }; let mut from = done; + // The pages whose bytes all came: a fault in the owner's buffer stops there. + let mut copied = 0; for &(_, phys, within, n) in &pinned[..count] { // SAFETY: a frame of the object, pinned and write-pinned above. - unsafe { src.read_raw(from, DirectMap::from_phys(phys).as_mut_ptr::().add(within), n) }; + if unsafe { src.read_raw(from, DirectMap::from_phys(phys).as_mut_ptr::().add(within), n) }.is_err() { + faulted = true; + break; + } from += n; + copied += 1; } { let mut state = self.state.lock(); - for &(page, phys, within, n) in &pinned[..count] { + for (k, &(page, phys, within, n)) in pinned[..count].iter().enumerate() { // A release meanwhile took the object: nothing left to unpin. if state.live(id) != Ok(slot) { break; } let (pages, object) = state.split(slot); - pages.write_unpin(object, page, phys, page * PAGE + (within + n) as u64); + if k < copied { + pages.write_unpin(object, page, phys, page * PAGE + (within + n) as u64); + } else { + pages.write_abandon(object, page, phys); + } } } for &(_, phys, _, _) in &pinned[..count] { pmm::unpin_range(phys, PAGE as usize); } done = from; + if faulted { + return refusal(done as u64, SyscallError::BadAddress); + } match stopped { None => {} Some(Refused::InvalidArgument) => return refusal(done as u64, SyscallError::NotFound), @@ -885,8 +925,7 @@ impl PagerObject { for (chunk, word) in bytes.as_chunks_mut::<8>().0.iter_mut().zip(bits) { chunk.copy_from_slice(&word.to_le_bytes()); } - out.write_at(0, &bytes); - 0 + out.write_at(0, &bytes).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 0) } Err(why) => word(why).to_u64(), } diff --git a/kernel/src/pipe.rs b/kernel/src/pipe.rs index 7b1882ccd8d..7c407f625f8 100644 --- a/kernel/src/pipe.rs +++ b/kernel/src/pipe.rs @@ -1,13 +1,24 @@ -use crate::frame_bytes::FrameBytes; +//! A pipe: its two ends' holders and the bytes between them, behind its own +//! lock, which no copy and no frame allocation runs under +//! (`crate::frame_bytes`). A small read or write bounces through its stack in +//! one hold; a larger one claims its bytes. A second of either kind waits for +//! its turn (`frame_bytes::turn`). + +use crate::frame_bytes::{self, Bounce, FrameBytes, BOUNCE}; + +/// A small read's or write's bytes, on its caller's stack. +pub type PipeBounce = Bounce; use crate::mm::charge::{self, Charge, Exhausted}; use kernel::pipe::{End, Holders, Released}; +use kernel::sched::task::WaitClass; use alloc::sync::Arc; -use crate::watch::Watch; +use crate::scheduler::Parkable; use crate::sync::Lock; use crate::user_ptr::{UserBytes, UserBytesMut}; +use crate::watch::{Cancelled, Watch}; /// A reference to a pipe that holds neither end: what an end's object, a /// waiter and a poll keep to reach it. The pipe is its own: nothing else @@ -103,6 +114,17 @@ struct State { rt_boost_pending: bool, } +impl State { + /// What no reader is left to read, to give back with the lock let go; + /// kept while a claim's copy may still reach it, whose end asks again. + fn unread(&mut self) -> Option { + (!self.holders.held(End::Read) && !self.held.claimed() && self.held.frames() > 0).then(|| { + let job = Arc::clone(self.held.job()); + core::mem::replace(&mut self.held, FrameBytes::new(&job)) + }) + } +} + impl Pipe { /// The heap one takes: itself and its four watches. fn heap() -> usize { @@ -127,6 +149,22 @@ pub fn create() -> Result<(PipeReader, PipeWriter), Exhausted> { Ok((PipeReader::acquire(&pipe), PipeWriter::acquire(&pipe))) } +/// Bytes [`PipeRef::take_now`] took: the frames they left spare go back, +/// and a lent real-time window is taken, as it drops with every lock let go. +pub struct Taken { + pub n: usize, + _spares: frame_bytes::Spares, + boost: bool, +} + +impl Drop for Taken { + fn drop(&mut self) { + if self.boost { + crate::scheduler::boost_current_rt_inherited(); + } + } +} + pub enum PipeWrite { Wrote(usize), /// Not the room the write needs now: wait for `kernel::pipe::room`. @@ -135,6 +173,22 @@ pub enum PipeWrite { /// The memory its bytes need could not be taken; no wait for room makes /// it appear, so a caller must not park on it. NoMemory, + /// A byte of the writer's buffer it cannot reach: nothing was written. + Faulted, +} + +/// Why a read answered no count. +pub enum ReadStopped { + /// A kill ended the wait for another read's claim. + Cancelled, + /// A byte of the reader's buffer it cannot reach: the bytes it took are lost. + Faulted, +} + +impl From for ReadStopped { + fn from(_: Cancelled) -> Self { + Self::Cancelled + } } impl PipeRef { @@ -142,39 +196,177 @@ impl PipeRef { self.0.state.lock().holders.hold(end); } - pub fn try_read(&self, buf: &mut UserBytesMut) -> Option { - let (result, boost) = { - let mut pipe = self.0.state.lock(); + /// Read what the pipe holds into `buf`, as much as fits: `None` while it + /// holds nothing and a writer is left. Waits only for another read's + /// claim to end, never for bytes. + pub fn read(&self, p: &Parkable, buf: &mut UserBytesMut) -> Result, ReadStopped> { + let mut bounce = PipeBounce::new(); + let (n, runs, boost) = { + let mut pipe = self.turn(p, End::Read)?; let available = pipe.held.len(); - if available > 0 { - let n = buf.len().min(available); - pipe.held.read(buf, n); - (Some(n), core::mem::take(&mut pipe.rt_boost_pending)) - } else if !pipe.holders.held(End::Write) && available == 0 { - (Some(0), false) - } else { - (None, false) + if available == 0 { + return Ok((!pipe.holders.held(End::Write)).then_some(0)); + } + let n = buf.len().min(available); + if n == 0 { + return Ok(Some(0)); + } + let boost = core::mem::take(&mut pipe.rt_boost_pending); + if n <= BOUNCE { + let spares = pipe.held.take(n, &mut bounce); + drop(pipe); + frame_bytes::give_back(&self.0.state, |s| &mut s.held, spares); + if boost { + crate::scheduler::boost_current_rt_inherited(); + } + buf.write_at(0, bounce.bytes()).map_err(|_| ReadStopped::Faulted)?; + return Ok(Some(n)); } + pipe.held.claim_read(n); + (n, pipe.held.first_drain(0, n), boost) }; if boost { // Outside the pipe's lock: the scheduler takes its own CPU-queue lock. crate::scheduler::boost_current_rt_inherited(); } - result + let copied = frame_bytes::drain(&self.0.state, |s| &mut s.held, 0, buf, n, runs); + let (spares, ended, unread) = { + let mut pipe = self.0.state.lock(); + let (spares, ended) = pipe.held.consume(); + (spares, ended, pipe.unread()) + }; + frame_bytes::give_back(&self.0.state, |s| &mut s.held, spares); + drop(unread); + frame_bytes::wake(&self.0.read_watch, ended); + copied.map_err(|_| ReadStopped::Faulted)?; + Ok(Some(n)) } - /// Write `buf` as a pipe write lands (`kernel::pipe::takes`), under the - /// one lock hold. - pub fn write(&self, buf: &UserBytes) -> PipeWrite { + /// Write `buf` as a pipe write lands (`kernel::pipe::takes`). Waits only + /// for another write's claim to end, never for room. + pub fn write(&self, p: &Parkable, buf: &UserBytes) -> Result { + if buf.len() <= BOUNCE { + let mut bounce = PipeBounce::new(); + return match bounce.read(buf) { + Ok(src) => self.write_bounced(p, src), + Err(_) => Ok(PipeWrite::Faulted), + }; + } + let (n, first) = { + let mut pipe = self.turn(p, End::Write)?; + if !pipe.holders.held(End::Read) { + return Ok(PipeWrite::BrokenPipe); + } + match kernel::pipe::takes(pipe.held.space(), buf.len()) { + None => return Ok(PipeWrite::NoRoom), + Some(0) => return Ok(PipeWrite::Wrote(0)), + Some(n) => { + pipe.held.claim_write(n); + (n, pipe.held.first_fill(n)) + } + } + }; + let supplied = match first { + frame_bytes::Fill::Runs(runs) => Ok(runs), + frame_bytes::Fill::Supply(needs) => frame_bytes::supply(&self.0.state, |s| &mut s.held, needs, n), + }; + let wrote = match supplied { + Ok(runs) => match frame_bytes::fill(&self.0.state, |s| &mut s.held, buf, n, runs) { + Ok(()) => PipeWrite::Wrote(n), + Err(_) => PipeWrite::Faulted, + }, + Err(Exhausted) => PipeWrite::NoMemory, + }; + let (ended, unread) = { + let mut pipe = self.0.state.lock(); + let ended = match wrote { + PipeWrite::Wrote(_) => pipe.held.commit(), + _ => pipe.held.abandon_write(), + }; + (ended, pipe.unread()) + }; + drop(unread); + frame_bytes::wake(&self.0.write_watch, ended); + Ok(wrote) + } + + /// [`Self::write`] of `src`, already copied in: whole in one hold, unless + /// it needs frames, which it takes with the lock let go. + fn write_bounced(&self, p: &Parkable, src: &PipeBounce) -> Result { + let (n, needs) = { + let mut pipe = self.turn(p, End::Write)?; + if !pipe.holders.held(End::Read) { + return Ok(PipeWrite::BrokenPipe); + } + let n = match kernel::pipe::takes(pipe.held.space(), src.bytes().len()) { + None => return Ok(PipeWrite::NoRoom), + Some(0) => return Ok(PipeWrite::Wrote(0)), + Some(n) => n, + }; + if pipe.held.put(&src.bytes()[..n]) { + return Ok(PipeWrite::Wrote(n)); + } + pipe.held.claim_write(n); + match pipe.held.first_fill(n) { + frame_bytes::Fill::Supply(needs) => (n, needs), + frame_bytes::Fill::Runs(_) => unreachable!("a put refused wants frames"), + } + }; + let supplied = frame_bytes::supply(&self.0.state, |s| &mut s.held, needs, n); + let (ended, unread) = { + let mut pipe = self.0.state.lock(); + let ended = match supplied { + Ok(_) => pipe.held.place_all(&src.bytes()[..n]), + Err(Exhausted) => pipe.held.abandon_write(), + }; + (ended, pipe.unread()) + }; + drop(unread); + frame_bytes::wake(&self.0.write_watch, ended); + Ok(match supplied { + Ok(_) => PipeWrite::Wrote(n), + Err(Exhausted) => PipeWrite::NoMemory, + }) + } + + /// Write `src`, already copied in, whole in one hold of the pipe's lock, + /// for a caller holding its process's lock: `None` where that cannot answer + /// it without a wait or frames taken, which [`Self::write`] does. + pub fn put_now(&self, src: &[u8]) -> Option { let mut pipe = self.0.state.lock(); + if pipe.held.claim_held(End::Write) { + return None; + } if !pipe.holders.held(End::Read) { - return PipeWrite::BrokenPipe; + return Some(PipeWrite::BrokenPipe); + } + match kernel::pipe::takes(pipe.held.space(), src.len())? { + 0 => Some(PipeWrite::Wrote(0)), + n => pipe.held.put(&src[..n]).then_some(PipeWrite::Wrote(n)), } - let Some(n) = kernel::pipe::takes(pipe.held.space(), buf.len()) else { return PipeWrite::NoRoom }; - if pipe.held.write(buf, n).is_err() { - return PipeWrite::NoMemory; + } + + /// Take at most `max` bytes, up to [`BOUNCE`], into `into` whole in one + /// hold of the pipe's lock, for a caller holding its process's lock: + /// `None` where the pipe holds nothing or more than a bounce would take, + /// or another read holds its claim, which [`Self::read`] answers. + pub fn take_now(&self, max: usize, into: &mut PipeBounce) -> Option { + let mut pipe = self.0.state.lock(); + let n = max.min(pipe.held.len()); + if n == 0 || n > BOUNCE || pipe.held.claim_held(End::Read) { + return None; } - PipeWrite::Wrote(n) + let spares = pipe.held.take(n, into); + Some(Taken { n, _spares: spares, boost: core::mem::take(&mut pipe.rt_boost_pending) }) + } + + /// The pipe's lock, held in `end`'s turn to claim. + fn turn(&self, p: &Parkable, end: End) -> Result, Cancelled> { + let watch = match end { + End::Read => &self.0.read_watch, + End::Write => &self.0.write_watch, + }; + frame_bytes::turn(p, &self.0.state, |s| &mut s.held, end, watch, WaitClass::Pipe) } pub fn has_data(&self) -> bool { @@ -200,16 +392,13 @@ impl PipeRef { /// One reference to `end` lets go; its end's last posts the other end's /// watch and its leaving watch, and the read end's last gives back what - /// the pipe holds, which nobody can read any more. + /// the pipe holds, which nobody can read any more, or leaves it to the + /// claim in flight to give back as it ends. fn release(&self, end: End) { let (released, unread) = { let mut pipe = self.0.state.lock(); let released = pipe.holders.release(end); - let unread = (!pipe.holders.held(End::Read) && pipe.held.frames() > 0).then(|| { - let job = Arc::clone(pipe.held.job()); - core::mem::replace(&mut pipe.held, FrameBytes::new(&job)) - }); - (released, unread) + (released, pipe.unread()) }; // Outside the lock: its frames go back to the allocator. drop(unread); diff --git a/kernel/src/process.rs b/kernel/src/process.rs index 74141146821..bb5fd5902c8 100644 --- a/kernel/src/process.rs +++ b/kernel/src/process.rs @@ -1110,6 +1110,17 @@ pub fn handle_page_fault(fault_addr: u64, access: kernel::vm::Access) -> bool { served } +/// Populate the running thread's pages over `[start, end)` for `access`, in +/// one hold of its space's lock, stopping at the first its ranges refuse. +pub fn fault_in(start: u64, end: u64, access: kernel::vm::Access) { + let Some(space) = scheduler::current_address_space() else { return }; + let mut space = space.lock(); + let mut page = start & !(crate::mm::PAGE_SIZE - 1); + while page < end && space.fault(page, access) { + page += crate::mm::PAGE_SIZE; + } +} + /// Dump memory around `fault_addr` and the instruction pointer for the /// current process; called from the exception handler on user-mode crashes. pub fn dump_crash_diagnostics(fault_addr: u64, rip: u64) { diff --git a/kernel/src/random.rs b/kernel/src/random.rs index 9246c289c1e..0548c876aae 100644 --- a/kernel/src/random.rs +++ b/kernel/src/random.rs @@ -108,15 +108,20 @@ pub fn word() -> u64 { } /// Fill a caller's window with one draw. -pub fn fill_user(out: &mut UserBytesMut) { +pub fn fill_user(out: &mut UserBytesMut) -> Result<(), crate::user_ptr::Faulted> { let mut stream = stream(); let mut chunk = [0u8; 256]; let mut at = 0; while at < out.len() { let n = (out.len() - at).min(chunk.len()); stream.fill(&mut chunk[..n]); - out.write_at(at, &chunk[..n]); + let wrote = out.write_at(at, &chunk[..n]); + if wrote.is_err() { + wipe(&mut chunk); + return wrote; + } at += n; } wipe(&mut chunk); + Ok(()) } diff --git a/kernel/src/scheduler.rs b/kernel/src/scheduler.rs index fb2faae9a34..42437e6d6f2 100644 --- a/kernel/src/scheduler.rs +++ b/kernel/src/scheduler.rs @@ -36,8 +36,10 @@ fn assert_baseline(baseline: u32) { } /// Depth an unnested trap handler runs at: one level, raised by the entry asm -/// and lowered on the way out. -const BASELINE_TRAP: u32 = 1; +/// and lowered on the way out (x86_64's `syscall_entry` and `common_entry`, +/// AArch64's `trap::syscall` and `trap::user_abort`). A user copy runs at it +/// and no deeper (`user_ptr`). +pub(crate) const BASELINE_TRAP: u32 = 1; /// Depth the deferred-preempt poll runs at: zero, since all three entry paths /// are past the trap entry level. diff --git a/kernel/src/syscall/device.rs b/kernel/src/syscall/device.rs index 2750b622e35..d320b577681 100644 --- a/kernel/src/syscall/device.rs +++ b/kernel/src/syscall/device.rs @@ -260,6 +260,12 @@ pub(super) fn sys_device_firmware_window(ctx: &SyscallContext, handle: RawHandle let Some(mut out) = ctx.user_bytes_mut(out, len) else { return SyscallError::BadAddress.to_u64(); }; + // Written first, before anything is opened, so a window the caller cannot + // write as the call begins leaves no grant it was never told of. One a + // sibling unmaps meanwhile can; the claim's release takes it back. + if out.fill_zero(0, len as usize).is_err() { + return SyscallError::BadAddress.to_u64(); + } let mut grant = toyos_abi::pci::WindowGrant { shm: toyos_abi::HANDLE_INVALID, refused: 0, offset: 0, len: 0 }; match on_function(&claim, |function| crate::pcidev::firmware_window(function, window)) { Err(e) => return e.to_u64(), @@ -276,8 +282,7 @@ pub(super) fn sys_device_firmware_window(ctx: &SyscallContext, handle: RawHandle // layout assertion beside its declaration proves every byte of it is an // initialised field. let bytes = unsafe { core::slice::from_raw_parts(&grant as *const _ as *const u8, len as usize) }; - out.write_at(0, bytes); - 0 + out.write_at(0, bytes).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 0) } /// Memory the claimed function may reach and nothing else may. @@ -291,12 +296,17 @@ pub(super) fn sys_device_dma_alloc( Ok(claim) => claim, Err(e) => return e.refuse(), }; - // The output window is taken before the allocation: a bad address must - // not leave a mapped grant the caller was never told the address of. + // The output window is taken, and written, before the allocation, so a + // bad address as the call begins leaves no mapped grant the caller was + // never told the address of. One a sibling unmaps meanwhile can; the + // claim's release takes it back. let len = core::mem::size_of::() as u64; let Some(mut window) = ctx.user_bytes_mut(out, len) else { return SyscallError::BadAddress.to_u64(); }; + if window.fill_zero(0, len as usize).is_err() { + return SyscallError::BadAddress.to_u64(); + } let granted = on_function(&claim, |function| crate::pcidev::dma_alloc(function, bytes)); let (memory, device_addr, granted) = match granted { Ok(grant) => grant, @@ -316,16 +326,9 @@ pub(super) fn sys_device_dma_alloc( return e.to_u64(); } }; - window.write_at( - 0, - grant_bytes(&toyos_abi::pci::DmaGrant { - shm, - _pad: 0, - device_addr, - bytes: granted, - }), - ); - 0 + window + .write_at(0, grant_bytes(&toyos_abi::pci::DmaGrant { shm, _pad: 0, device_addr, bytes: granted })) + .map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 0) } /// A region the caller holds, put into the claimed function's domain. @@ -359,6 +362,10 @@ pub(super) fn sys_device_dma_map( let Some(mut window) = ctx.user_bytes_mut(out, len) else { return SyscallError::BadAddress.to_u64(); }; + // Written first, before anything is mapped, for `sys_device_dma_alloc`'s reason. + if window.fill_zero(0, len as usize).is_err() { + return SyscallError::BadAddress.to_u64(); + } let mapped = on_function(&claim, |function| crate::pcidev::dma_map(function, &memory)); let (device_addr, bytes) = match mapped { Ok(mapped) => mapped, @@ -374,8 +381,7 @@ pub(super) fn sys_device_dma_map( core::mem::size_of::(), ) }; - window.write_at(0, raw); - 0 + window.write_at(0, raw).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 0) } /// Take back what [`sys_device_dma_map`] put at `device_addr`. diff --git a/kernel/src/syscall/dispatch.rs b/kernel/src/syscall/dispatch.rs index 4b3f418ca95..1158529b994 100644 --- a/kernel/src/syscall/dispatch.rs +++ b/kernel/src/syscall/dispatch.rs @@ -27,7 +27,7 @@ use super::device::{ sys_device_reg_read, }; use super::event::{sys_event_create, sys_event_signal, sys_event_take, sys_handle_wait}; -use super::handles::{sys_close, sys_handle_dup, with_object, with_object_ref}; +use super::handles::{object, sys_close, sys_handle_dup, with_object}; use super::io::{ sys_null, sys_random, sys_read, sys_read_nonblock, sys_write, sys_write_nonblock, }; @@ -94,8 +94,8 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> } SYS_CLOSE => sys_close(RawHandle(a1 as u32)), SYS_FSTAT => { - let stat = match with_object_ref(RawHandle(a1 as u32), Rights::NONE, ops::fstat) { - Ok(stat) => stat, + let stat = match object(RawHandle(a1 as u32), Rights::NONE) { + Ok(object) => ops::fstat(&object), Err(e) => return e.refuse(), }; match ctx.copy_out(UserAddr::new(a2), &stat) { @@ -322,8 +322,7 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> } else { let Some(mut buf) = ctx.user_bytes_mut(UserAddr::new(a1), a2) else { return bad_addr }; let copy_len = env.len().min(buf.len()); - buf.write_at(0, &env[..copy_len]); - copy_len as u64 + buf.write_at(0, &env[..copy_len]).map_or(bad_addr, |()| copy_len as u64) } } // What a claimed PCI function's driver is built out of. Each is @@ -407,7 +406,9 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> } let Some(bytes) = ctx.user_bytes(UserAddr::new(a2), len) else { return bad_addr }; let mut badge = [0u8; MAX_BADGE]; - bytes.read_at(0, &mut badge[..len as usize]); + if bytes.read_at(0, &mut badge[..len as usize]).is_err() { + return bad_addr; + } sys_port_mint(RawHandle(a1 as u32), &badge[..len as usize]) } SYS_PORT_BADGE => { @@ -572,7 +573,9 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> return bad_addr; }; let mut raw = [0u8; CPU_MASK_WORDS * 8]; - bytes.read_at(0, &mut raw[..a3 as usize * 8]); + if bytes.read_at(0, &mut raw[..a3 as usize * 8]).is_err() { + return bad_addr; + } let words: [u64; CPU_MASK_WORDS] = core::array::from_fn(|i| u64::from_le_bytes(raw[i * 8..][..8].try_into().unwrap())); sys_thread_set_affinity(a1, &words[..a3 as usize]) @@ -586,10 +589,8 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> }; match sys_thread_get_affinity(a1) { Ok(words) => { - for (i, word) in words.iter().enumerate() { - out.write_at(i * 8, &word.to_le_bytes()); - } - 0 + let wrote = words.iter().enumerate().try_for_each(|(i, word)| out.write_at(i * 8, &word.to_le_bytes())); + wrote.map_or(bad_addr, |()| 0) } Err(e) => e.to_u64(), } @@ -606,7 +607,9 @@ pub(crate) fn syscall_dispatch(num: u64, a1: u64, a2: u64, a3: u64, a4: u64) -> return bad_addr; }; let mut name = [0u8; process::THREAD_NAME_LEN]; - bytes.read_at(0, &mut name[..len]); + if bytes.read_at(0, &mut name[..len]).is_err() { + return bad_addr; + } process::set_current_thread_name(&name[..len]); 0 }, diff --git a/kernel/src/syscall/handles.rs b/kernel/src/syscall/handles.rs index 43df5922909..8a78ebe74e8 100644 --- a/kernel/src/syscall/handles.rs +++ b/kernel/src/syscall/handles.rs @@ -8,15 +8,6 @@ use crate::process; use toyos_abi::handle::{RawHandle, Rights}; use toyos_abi::syscall::*; -/// Runs `f` on the object `h` names, holding the table's guard. -pub(super) fn with_object_ref( - h: RawHandle, - need: Rights, - f: impl FnOnce(&KObjectRef) -> R, -) -> Result { - process::with_process_data(|data| data.handles.get_ref(h, need).map(f)) -} - /// Requires that `syscap` resolves to a `SysCap` holding `need`, and answers /// every right it holds, read under the same guard. pub(super) fn demand_syscap(syscap: RawHandle, need: Rights) -> Result { @@ -26,10 +17,17 @@ pub(super) fn demand_syscap(syscap: RawHandle, need: Rights) -> Result Result { + process::with_process_data(|data| data.handles.get_ref(h, need).cloned()) +} + +/// `f` over the object `h` names, outside the process's lock, for the calls +/// whose answer is already a raw syscall word. pub(super) fn with_object(h: RawHandle, need: Rights, f: impl FnOnce(&KObjectRef) -> u64) -> u64 { - match with_object_ref(h, need, f) { - Ok(v) => v, + match object(h, need) { + Ok(object) => f(&object), Err(e) => e.refuse(), } } diff --git a/kernel/src/syscall/io.rs b/kernel/src/syscall/io.rs index 6d4ba13dd89..e7d3773066b 100644 --- a/kernel/src/syscall/io.rs +++ b/kernel/src/syscall/io.rs @@ -1,12 +1,14 @@ //! Blocking `read`/`write` register on the object's watch before re-checking the -//! condition — closing the check-then-block window — and never park while -//! holding a `with_process_data` guard. +//! condition — closing the check-then-block window. Each works on its object +//! cloned out of the process's lock, which no copy and no park runs under. //! //! Buffers arrive already bounded: `super::dispatch` converts the caller's //! pointer and length into a [`UserBytes`]/[`UserBytesMut`] window before //! this module runs. use crate::watch; +use crate::frame_bytes::BOUNCE; +use crate::pipe::PipeBounce; use crate::object::null::NullObject; use crate::object::{ops, KObjectRef}; use crate::time::Deadline; @@ -18,16 +20,13 @@ use toyos_abi::syscall::*; use kernel::sched::task::WaitClass; use super::cancelled; -use super::handles::with_object_ref; +use super::handles::object; /// What `sys_write` does when the object took nothing. enum WriteBlock { /// The pipe, and the room the write waits for. Pipe(pipe::PipeRef, usize), Refused(u64), - /// Carried out of the process's lock: `HandleError::refuse` may take the - /// process down and cannot run under a guard. - BadHandle(crate::object::HandleError), } /// What `sys_read` parks on when the handle has nothing to give. @@ -37,32 +36,68 @@ enum ReadBlock { Notice(alloc::sync::Arc), /// Nothing to wait for: the answer is this word. Refused(u64), - /// Carried out of the process's lock: `HandleError::refuse` may take the - /// process down and cannot run under a guard. - BadHandle(crate::object::HandleError), +} + +/// A write of at most `BOUNCE` bytes to a pipe, made whole in the hold of the +/// process's lock that looks its handle up, its bytes copied in before it, as +/// a write was before any copy left the locks: `None` for anything else, which +/// the general path answers (copying a small write to another object again). +fn pipe_write_now(h: RawHandle, buf: &UserBytes) -> Option { + if buf.len() > BOUNCE { + return None; + } + let mut bounce = PipeBounce::new(); + // A buffer that faults is the general path's to answer, after its handle. + let src = bounce.read(buf).ok()?.bytes(); + let (word, pipe) = process::with_process_data(|data| { + let KObjectRef::PipeWrite(w) = data.handles.get_ref(h, Rights::WRITE).ok()? else { return None }; + let word = ops::pipe_written(w.pipe().put_now(src)?)?; + Some((word, w.pipe().clone())) + })?; + process::wake_pipe_readers(&pipe); + Some(word) +} + +/// A read of a pipe holding at most `BOUNCE` bytes for it, taken in the hold +/// of the process's lock that looks its handle up and copied out after it: +/// `None` for anything else, which the general path answers. +fn pipe_read_now(h: RawHandle, buf: &mut UserBytesMut) -> Option { + let mut bounce = PipeBounce::new(); + let (taken, pipe) = process::with_process_data(|data| { + let KObjectRef::PipeRead(r) = data.handles.get_ref(h, Rights::READ).ok()? else { return None }; + Some((r.pipe().take_now(buf.len(), &mut bounce)?, r.pipe().clone())) + })?; + let n = taken.n; + drop(taken); + // A reader whose buffer faults loses what it took, as a claimed read does. + let copied = buf.write_at(0, bounce.bytes()); + process::wake_pipe_writers(&pipe); + Some(copied.map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| n as u64)) } pub(super) fn sys_write(h: RawHandle, buf: &UserBytes) -> u64 { + if let Some(word) = pipe_write_now(h, buf) { + return word; + } loop { - let action = process::with_process_data(|data| { - let object = match data.handles.get_ref(h, Rights::WRITE) { - Ok(object) => object, - Err(e) => return Err(WriteBlock::BadHandle(e)), - }; - match ops::try_write(object, buf) { - Some(n) => Ok((n, ops::written_pipe(object))), - None => Err(match ops::pipe_write(object) { - Some(pipe) => WriteBlock::Pipe(pipe.clone(), kernel::pipe::room(buf.len())), - None => WriteBlock::Refused(SyscallError::NotFound.to_u64()), - }), - } - }); - match action { - Ok((n, pipe)) => { - if let Some(pipe) = pipe { process::wake_pipe_readers(&pipe); } + // Looked up again after each wait: a write whose handle a sibling + // closed meanwhile is refused, not landed past its reader's EOF. + let object = match object(h, Rights::WRITE) { + Ok(object) => object, + Err(e) => return e.refuse(), + }; + let blocked = match ops::try_write(&object, buf) { + Some(n) => { + if let Some(pipe) = ops::pipe_write(&object) { process::wake_pipe_readers(pipe); } return n; } - Err(WriteBlock::Pipe(pipe, room)) => { + None => match ops::pipe_write(&object) { + Some(pipe) => WriteBlock::Pipe(pipe.clone(), kernel::pipe::room(buf.len())), + None => WriteBlock::Refused(SyscallError::NotFound.to_u64()), + }, + }; + match blocked { + WriteBlock::Pipe(pipe, room) => { let parkable = crate::scheduler::Parkable::at_entry(); if watch::wait_until(&parkable, pipe.write_watch(), 0, WaitClass::Pipe, Deadline::never(), || pipe.has_space(room)) .is_err() @@ -70,15 +105,14 @@ pub(super) fn sys_write(h: RawHandle, buf: &UserBytes) -> u64 { return cancelled(); } } - Err(WriteBlock::Refused(word)) => return word, - Err(WriteBlock::BadHandle(e)) => return e.refuse(), + WriteBlock::Refused(word) => return word, } } } fn read_block(object: &KObjectRef) -> ReadBlock { match object { - KObjectRef::Device(_) => unreachable!("a device claim blocks via `read_block_device`"), + KObjectRef::Device(_) => unreachable!("a device claim never blocks"), KObjectRef::Notice(n) => ReadBlock::Notice(n.clone()), _ => match ops::pipe_read(object).map(|pipe| ReadBlock::Pipe(pipe.clone())) { Some(block) => block, @@ -87,38 +121,40 @@ fn read_block(object: &KObjectRef) -> ReadBlock { } } +/// What a read's handle names, out of the process's lock: a device claim is +/// read under it, since describing one installs handles, and its answer copied +/// out after. +enum Readable { + Object(KObjectRef), + Device(Option), +} + +fn readable(h: RawHandle, room: usize) -> Result { + process::with_process_data(|data| { + let object = data.handles.get_ref(h, Rights::READ)?; + let KObjectRef::Device(claim) = object else { return Ok(Readable::Object(object.clone())) }; + let claim = alloc::sync::Arc::clone(claim); + Ok(Readable::Device(ops::read_device(&claim, &mut data.handles, room))) + }) +} + pub(super) fn sys_read(h: RawHandle, buf: &mut UserBytesMut) -> u64 { + if let Some(n) = pipe_read_now(h, buf) { + return n; + } + let object = match readable(h, buf.len()) { + Ok(Readable::Object(object)) => object, + // No device class blocks: an empty read answers `NotFound`. + Ok(Readable::Device(read)) => return read.map_or(SyscallError::NotFound.to_u64(), |read| read.deliver(buf)), + Err(e) => return e.refuse(), + }; loop { - let action = process::with_process_data(|data| { - let object = match data.handles.get_ref(h, Rights::READ) { - Ok(object) => object, - Err(e) => return Err(ReadBlock::BadHandle(e)), - }; - // Resolved twice rather than cached: this path runs once per - // device per boot, but cloning the `Arc` would cost an atomic - // RMW on the hottest syscall in the kernel. - if matches!(object, KObjectRef::Device(_)) { - let claim = data - .handles - .get::(h, Rights::READ) - .expect("a Device resolved a moment ago under this same hold"); - // No device class blocks: an empty read answers `NotFound`. - return match ops::read_device(&claim, &mut data.handles, buf) { - Some(n) => Ok((n, None)), - None => Err(ReadBlock::Refused(SyscallError::NotFound.to_u64())), - }; - } - match ops::try_read(object, buf) { - Some(n) => Ok((n, ops::read_pipe(object))), - None => Err(read_block(object)), - } - }); - match action { - Ok((n, pipe)) => { - if let Some(pipe) = pipe { process::wake_pipe_writers(&pipe); } - return n; - } - Err(ReadBlock::Pipe(pipe)) => { + if let Some(n) = ops::try_read(&object, buf) { + if let Some(pipe) = ops::pipe_read(&object) { process::wake_pipe_writers(pipe); } + return n; + } + match read_block(&object) { + ReadBlock::Pipe(pipe) => { let parkable = crate::scheduler::Parkable::at_entry(); if watch::wait_until(&parkable, pipe.read_watch(), 0, WaitClass::Pipe, Deadline::never(), || pipe.has_data()) .is_err() @@ -126,7 +162,7 @@ pub(super) fn sys_read(h: RawHandle, buf: &mut UserBytesMut) -> u64 { return cancelled(); } } - Err(ReadBlock::Notice(notice)) => { + ReadBlock::Notice(notice) => { let parkable = crate::scheduler::Parkable::at_entry(); if watch::wait_until( &parkable, @@ -141,49 +177,43 @@ pub(super) fn sys_read(h: RawHandle, buf: &mut UserBytesMut) -> u64 { return cancelled(); } } - Err(ReadBlock::Refused(word)) => return word, - Err(ReadBlock::BadHandle(e)) => return e.refuse(), + ReadBlock::Refused(word) => return word, } } } pub(super) fn sys_read_nonblock(h: RawHandle, buf: &mut UserBytesMut) -> u64 { - let result = process::with_process_data(|data| { - let object = match data.handles.get_ref(h, Rights::READ) { - Ok(object) => object, - Err(e) => return Err(e), - }; - // Same two-step as `sys_read`, for the same reason. - if matches!(object, KObjectRef::Device(_)) { - let claim = data - .handles - .get::(h, Rights::READ) - .expect("a Device resolved a moment ago under this same hold"); - return Ok(ops::read_device(&claim, &mut data.handles, buf).map(|n| (n, None))); - } - Ok(ops::try_read(object, buf).map(|n| (n, ops::read_pipe(object)))) - }); - match result { - Ok(Some((n, wake))) => { - if let Some(pipe) = wake { process::wake_pipe_writers(&pipe); } + if let Some(n) = pipe_read_now(h, buf) { + return n; + } + let object = match readable(h, buf.len()) { + Ok(Readable::Object(object)) => object, + Ok(Readable::Device(read)) => return read.map_or(SyscallError::WouldBlock.to_u64(), |read| read.deliver(buf)), + Err(e) => return e.refuse(), + }; + match ops::try_read(&object, buf) { + Some(n) => { + if let Some(pipe) = ops::pipe_read(&object) { process::wake_pipe_writers(pipe); } n } - Ok(None) => SyscallError::WouldBlock.to_u64(), - Err(e) => e.refuse(), + None => SyscallError::WouldBlock.to_u64(), } } pub(super) fn sys_write_nonblock(h: RawHandle, buf: &UserBytes) -> u64 { - let result = with_object_ref(h, Rights::WRITE, |object| { - ops::try_write(object, buf).map(|n| (n, ops::written_pipe(object))) - }); - match result { - Ok(Some((n, wake))) => { - if let Some(pipe) = wake { process::wake_pipe_readers(&pipe); } + if let Some(word) = pipe_write_now(h, buf) { + return word; + } + let object = match object(h, Rights::WRITE) { + Ok(object) => object, + Err(e) => return e.refuse(), + }; + match ops::try_write(&object, buf) { + Some(n) => { + if let Some(pipe) = ops::pipe_write(&object) { process::wake_pipe_readers(pipe); } n } - Ok(None) => SyscallError::WouldBlock.to_u64(), - Err(e) => e.refuse(), + None => SyscallError::WouldBlock.to_u64(), } } @@ -205,6 +235,8 @@ pub(super) fn sys_null(want: u64) -> u64 { } pub(super) fn sys_random(out: &mut UserBytesMut) -> u64 { - crate::random::fill_user(out); - 0 + match crate::random::fill_user(out) { + Ok(()) => 0, + Err(faulted) => SyscallError::from(faulted).to_u64(), + } } diff --git a/kernel/src/syscall/ipc.rs b/kernel/src/syscall/ipc.rs index ba025a1ab10..0183dec59d8 100644 --- a/kernel/src/syscall/ipc.rs +++ b/kernel/src/syscall/ipc.rs @@ -11,7 +11,8 @@ use alloc::vec::Vec; use crate::watch; use crate::object::{ops, port, KObjectRef}; -use crate::message::{SendRefused, Taken}; +use crate::message::{MessageBounce, QueueRef, SendRefused, Unclaimed, Unsent}; +use crate::object::handle::HandleEntry; use crate::object::service::ConnectionEnd; use crate::time::Deadline; use crate::user_ptr::SyscallContext; @@ -122,7 +123,9 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - }; for i in 0..args.keep_n as usize { let mut raw = [0u8; 8]; - keep.read_at(i * 8, &mut raw); + if keep.read_at(i * 8, &mut raw).is_err() { + return SyscallError::BadAddress.to_u64(); + } let off = u32::from_ne_bytes([raw[0], raw[1], raw[2], raw[3]]); let len = u32::from_ne_bytes([raw[4], raw[5], raw[6], raw[7]]); let Some(name) = name_at(off, len) else { @@ -144,7 +147,9 @@ pub(super) fn sys_namespace_build(ctx: &SyscallContext, args: &NamespaceBuild) - }; for i in 0..args.add_n as usize { let mut raw = [0u8; 16]; - add.read_at(i * 16, &mut raw); + if add.read_at(i * 16, &mut raw).is_err() { + return SyscallError::BadAddress.to_u64(); + } let off = u32::from_ne_bytes([raw[0], raw[1], raw[2], raw[3]]); let len = u32::from_ne_bytes([raw[4], raw[5], raw[6], raw[7]]); let handle = RawHandle(u32::from_ne_bytes([raw[8], raw[9], raw[10], raw[11]])); @@ -207,8 +212,10 @@ pub(super) fn sys_namespace_names(ns_h: RawHandle, out: &mut crate::user_ptr::Us if out.len() >= required { let mut at = 0; for (name, _) in namespace.entries() { - out.write_at(at, &(name.len() as u16).to_le_bytes()); - out.write_at(at + 2, name.as_bytes()); + let wrote = out.write_at(at, &(name.len() as u16).to_le_bytes()).and_then(|()| out.write_at(at + 2, name.as_bytes())); + if let Err(faulted) = wrote { + return SyscallError::from(faulted).to_u64(); + } at += 2 + name.len(); } } @@ -342,8 +349,7 @@ pub(super) fn sys_port_badge( return SyscallError::PermissionDenied.to_u64(); } let Some(badge) = stamp.badge else { return SyscallError::NotFound.to_u64() }; - out.write_at(0, &badge); - badge.len() as u64 + out.write_at(0, &badge).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| badge.len() as u64) } /// Make a shared region and hand back the one handle to it, carrying MAP, WRITE, DUP and TRANSFER — naming it is what authorizes mapping it. @@ -457,111 +463,197 @@ fn send(conn_h: RawHandle, out: &Outgoing<'_>, nonblock: bool) -> Result<(), u64 let mut wanted = [RawHandle(0); MAX_TRANSFER_HANDLES]; for (i, slot) in wanted.iter_mut().enumerate().take(out.count) { let mut raw = [0u8; HANDLE_LEN]; - out.handles.read_at(i * HANDLE_LEN, &mut raw); + out.handles.read_at(i * HANDLE_LEN, &mut raw).map_err(|faulted| SyscallError::from(faulted).to_u64())?; *slot = RawHandle(u32::from_ne_bytes(raw)); } let wanted = &wanted[..out.count]; // `TRANSFER` on the connection only when it carries handles: bytes alone // are a write, which `WRITE` already allows. let need = if out.count == 0 { Rights::WRITE } else { Rights::WRITE.union(Rights::TRANSFER) }; + // Every handle is checked before anything is queued or moved, and again + // under the hold that moves them. + let checked = |data: &process::ProcessData| { + let conn = data.handles.get::(conn_h, need)?; + for (i, h) in wanted.iter().enumerate() { + if *h == conn_h || wanted[..i].contains(h) { + return Err(SyscallError::InvalidArgument.into()); + } + let rights = data.handles.rights_of(*h)?; + if !rights.contains(Rights::TRANSFER) { + return Err(SyscallError::PermissionDenied.into()); + } + } + Ok::<_, crate::object::Refusal>(conn.tx().clone()) + }; + // The message's bytes, copied in once before any lock. + let mut bounce = MessageBounce::new(); + let src = bounce.read(&out.bytes).map_err(|faulted| SyscallError::from(faulted).to_u64())?.bytes(); + let parkable = crate::scheduler::Parkable::at_entry(); loop { - let sent = process::with_process_data(|data| { - let conn = data.handles.get::(conn_h, need)?; - // Every handle is checked before anything is queued or moved. - for (i, h) in wanted.iter().enumerate() { - if *h == conn_h || wanted[..i].contains(h) { - return Err(SyscallError::InvalidArgument.into()); - } - let rights = data.handles.rights_of(*h)?; - if !rights.contains(Rights::TRANSFER) { - return Err(SyscallError::PermissionDenied.into()); - } - } - // A device claim that moves takes back every mapping made through - // it: what it handed out is taken off it here, before the send - // shows it to a receiver that may hand out its own, and revoked - // once this lock is let go. - let claims: Vec<_> = wanted - .iter() - .filter_map(|h| data.handles.get::(*h, Rights::TRANSFER).ok()) - .filter_map(|claim| claim.take_mappings().map(|handed| (claim, handed))) - .collect(); - // A refusal restores every handle at its own number, so it is invisible. - let tx = conn.tx().clone(); - let moved = data.handles.transfer(wanted, |batch| tx.send(&out.bytes, batch)); - let handed: Vec<_> = match moved { - Ok(()) => claims.into_iter().map(|(_, handed)| handed).collect(), - Err(_) => { - for (claim, handed) in claims { - claim.restore_mappings(handed); - } - Vec::new() - } - }; + let moved = process::with_process_data(|data| { + let tx = checked(data)?; + let (moved, handed) = move_handles(data, wanted, |batch| tx.send_now(src, batch)); Ok::<_, crate::object::Refusal>((tx, moved, handed)) }); - let tx = match sent { + let (tx, refused) = match moved { Ok((tx, Ok(()), handed)) => { - for handed in handed { - handed.revoke(); - } - crate::scheduler::post_lending(tx.read_watch(), || tx.set_rt_boost_pending()); + sent(&tx, handed); return Ok(()); } - Ok((tx, Err(SendRefused::NoRoom), _)) if !nonblock => tx, - Ok((_, Err(SendRefused::NoRoom), _)) => return Err(SyscallError::WouldBlock.to_u64()), - Ok((_, Err(SendRefused::Gone), _)) => return Err(SyscallError::Gone.to_u64()), - Ok((_, Err(SendRefused::NoMemory), _)) => return Err(SyscallError::ResourceExhausted.to_u64()), + Ok((tx, Err(Unsent::Refused(refused)), _)) => (tx, Some(refused)), + // Its bytes want frames, or another send's claim is taking them. + Ok((tx, Err(Unsent::Claim), _)) => { + let refused = match tx.claim_send(&parkable, src, out.count) { + Ok(mut claim) => { + let moved = process::with_process_data(|data| { + checked(data)?; + Ok::<_, crate::object::Refusal>(move_handles(data, wanted, |batch| claim.commit(batch))) + }); + drop(claim); + match moved { + Ok((Ok(()), handed)) => { + sent(&tx, handed); + None + } + Ok((Err(refused), _)) => Some(refused), + Err(e) => return Err(e.refuse()), + } + } + Err(refused) => Some(refused), + }; + (tx, refused) + } Err(e) => return Err(e.refuse()), }; + let Some(refused) = refused else { return Ok(()) }; + match refused { + SendRefused::NoRoom if !nonblock => {} + SendRefused::NoRoom => return Err(SyscallError::WouldBlock.to_u64()), + SendRefused::Gone => return Err(SyscallError::Gone.to_u64()), + SendRefused::NoMemory => return Err(SyscallError::ResourceExhausted.to_u64()), + SendRefused::Cancelled => return Err(cancelled()), + } // Outside every lock: room appears as the receiver takes messages, and each receive posts this watch. let (len, count) = (out.bytes.len(), out.count); - if watch::wait_until(&crate::scheduler::Parkable::at_entry(), tx.write_watch(), 0, WaitClass::Ipc, Deadline::never(), || { - tx.fits(len, count) - }) - .is_err() - { + if watch::wait_until(&parkable, tx.write_watch(), 0, WaitClass::Ipc, Deadline::never(), || tx.fits(len, count)).is_err() { return Err(cancelled()); } } } +/// Move `wanted`'s handles, checked under this same hold, through `sink`, +/// which queues them with their message or refuses and hands them back. A +/// device claim that moves takes back every mapping made through it: what it +/// handed out is taken off it here, before the send shows it to a receiver +/// that may hand out its own, and is answered to revoke once the lock is let +/// go. A refusal restores every handle at its own number, and every mapping, +/// so it is invisible. +fn move_handles( + data: &mut process::ProcessData, + wanted: &[RawHandle], + sink: impl FnOnce(Vec) -> Result<(), (Vec, E)>, +) -> (Result<(), E>, Vec) { + let claims: Vec<_> = wanted + .iter() + .filter_map(|h| data.handles.get::(*h, Rights::TRANSFER).ok()) + .filter_map(|claim| claim.take_mappings().map(|handed| (claim, handed))) + .collect(); + let moved = data.handles.transfer(wanted, sink); + let handed = match moved { + Ok(()) => claims.into_iter().map(|(_, handed)| handed).collect(), + Err(_) => { + for (claim, handed) in claims { + claim.restore_mappings(handed); + } + Vec::new() + } + }; + (moved, handed) +} + +/// A message is queued: with every lock let go, revoke what its device +/// claims handed out and wake the receiver. +fn sent(tx: &QueueRef, handed: Vec) { + for handed in handed { + handed.revoke(); + } + crate::scheduler::post_lending(tx.read_watch(), || tx.set_rt_boost_pending()); +} + fn recv(conn_h: RawHandle, into: &mut Incoming<'_>, nonblock: bool) -> u64 { + let parkable = crate::scheduler::Parkable::at_entry(); + let (buf, cap) = (into.head.len() + into.bytes.len(), into.cap); + let mut bounce = MessageBounce::new(); loop { - let taken = process::with_process_data(|data| { - let conn = data.handles.get::(conn_h, Rights::READ)?; - let rx = conn.rx().clone(); - let taken = rx.recv(&mut into.head, &mut into.bytes, into.cap, &mut data.handles); - Ok::<_, crate::object::Refusal>((rx, taken)) - }); - let rx = match taken { - Ok((rx, Taken::Message(head, installed))) => { - for (i, h) in installed[..head.handles].iter().enumerate() { - into.handles.write_at(i * HANDLE_LEN, &h.0.to_ne_bytes()); + // Taken, and its handles installed, in the hold that looks the + // connection up; its bytes are copied out after. + let looked = process::with_process_data(|data| { + let rx = data.handles.get::(conn_h, Rights::READ)?.rx().clone(); + let taken = loop { + match rx.recv_now(&mut data.handles, buf, cap, &mut bounce) { + // Grown with the queue's lock let go, then asked again. + Err(Unclaimed::Room(n)) if data.handles.make_room(n) => {} + Err(Unclaimed::Room(_)) => break Err(Unclaimed::Untaken(kernel::message::Untaken::NoRoom)), + taken => break taken, } - // Room for the sender. - rx.write_watch().post(); - return toyos_abi::syscall::Received::encode(head.len, head.handles); - } - Ok((_, Taken::Gone)) => return SyscallError::Gone.to_u64(), - Ok((_, Taken::Untaken(kernel::message::Untaken::TooSmall))) => return SyscallError::InvalidArgument.to_u64(), - Ok((_, Taken::Untaken(kernel::message::Untaken::NoRoom))) => return SyscallError::ResourceExhausted.to_u64(), - Ok((_, Taken::Empty)) if nonblock => return SyscallError::WouldBlock.to_u64(), - Ok((rx, Taken::Empty)) => rx, + }; + Ok::<_, crate::object::HandleError>((rx, taken)) + }); + let rx = match looked { // Refused outside this hold: ending the caller takes the same non-reentrant lock. Err(e) => return e.refuse(), + Ok((rx, Ok(received))) => { + let bytes = bounce.bytes(); + let first = bytes.len().min(into.head.len()); + let copied = into.head.write_at(0, &bytes[..first]).and_then(|()| into.bytes.write_at(0, &bytes[first..])); + let (head, installed) = (received.head, received.installed); + drop(received); + return match copied { + Ok(()) => delivered(&rx, into, head, &installed), + Err(faulted) => undelivered(&rx, &installed[..head.handles], faulted), + }; + } + Ok((_, Err(Unclaimed::Gone))) => return SyscallError::Gone.to_u64(), + Ok((_, Err(Unclaimed::Untaken(kernel::message::Untaken::TooSmall)))) => return SyscallError::InvalidArgument.to_u64(), + Ok((_, Err(Unclaimed::Untaken(kernel::message::Untaken::NoRoom)))) => return SyscallError::ResourceExhausted.to_u64(), + Ok((_, Err(Unclaimed::Room(_)))) => unreachable!("a table that would not grow answers NoRoom"), + Ok((_, Err(Unclaimed::Empty))) if nonblock => return SyscallError::WouldBlock.to_u64(), + Ok((rx, Err(Unclaimed::Empty))) => rx, }; - if watch::wait_until(&crate::scheduler::Parkable::at_entry(), rx.read_watch(), 0, WaitClass::Ipc, Deadline::never(), || { - rx.readable() - }) - .is_err() - { + if watch::wait_until(&parkable, rx.read_watch(), 0, WaitClass::Ipc, Deadline::never(), || rx.readable()).is_err() { return cancelled(); } } } +/// A message is taken, its bytes in the reader's buffers: answer its +/// handles' numbers and wake the sender, for its room. +fn delivered(rx: &QueueRef, into: &mut Incoming<'_>, head: kernel::message::Head, installed: &[RawHandle]) -> u64 { + let installed = &installed[..head.handles]; + let told = installed.iter().enumerate().try_for_each(|(i, h)| into.handles.write_at(i * HANDLE_LEN, &h.0.to_ne_bytes())); + if let Err(faulted) = told { + return undelivered(rx, installed, faulted); + } + rx.write_watch().post(); + toyos_abi::syscall::Received::encode(head.len, head.handles) +} + +/// A message taken whose reader's buffers faulted: its bytes are lost, and +/// its handles, installed at numbers the reader was never told, are closed. +fn undelivered(rx: &QueueRef, installed: &[RawHandle], faulted: crate::user_ptr::Faulted) -> u64 { + process::with_process_data(|data| { + for &h in installed { + // A sibling may have closed the number since, or reused it for a + // handle of its own, which this then closes: the race is the + // misbehaving process's own, and stays inside it. + let _ = ops::close(&mut data.handles, h); + } + }); + rx.write_watch().post(); + SyscallError::from(faulted).to_u64() +} + /// Make an inbox and tell the caller where it is; the inbox owns its page and only this mapping may name it. `out` is the window the dispatch took for the answer. pub(super) fn sys_inbox_setup(depth: u32, out: &mut crate::user_ptr::UserBytesMut) -> u64 { let (inbox, vaddr) = match crate::inbox::create(depth) { @@ -578,7 +670,11 @@ pub(super) fn sys_inbox_setup(depth: u32, out: &mut crate::user_ptr::UserBytesMu Ok(h) => h, Err(e) => return e.to_u64(), }; - out.write_at(0, toyos_abi::usersafe::bytes(&InboxSetup { handle, _pad: 0, vaddr })); + // A caller that cannot be told its handle does not keep it. + if let Err(faulted) = out.write_at(0, toyos_abi::usersafe::bytes(&InboxSetup { handle, _pad: 0, vaddr })) { + let _ = process::with_process_data(|data| ops::close(&mut data.handles, handle)); + return SyscallError::from(faulted).to_u64(); + } 0 } diff --git a/kernel/src/syscall/machine.rs b/kernel/src/syscall/machine.rs index bb78dd40dd2..072e874260d 100644 --- a/kernel/src/syscall/machine.rs +++ b/kernel/src/syscall/machine.rs @@ -50,7 +50,9 @@ pub(super) fn sys_log_last_words(syscap: RawHandle, text: &UserBytes) -> u64 { } let mut copy = [0u8; toyos_abi::log::MAX_LAST_WORDS]; let copy = &mut copy[..text.len()]; - text.read_at(0, copy); + if let Err(faulted) = text.read_at(0, copy) { + return SyscallError::from(faulted).to_u64(); + } let Some(words) = toyos_abi::log::last_words(copy) else { return SyscallError::InvalidArgument.to_u64(); }; @@ -102,7 +104,7 @@ fn read_on_cursor( let put = |head: CursorHead, positions: &[u64]| -> Result<(), SyscallError> { let mut window = ctx.user_bytes_mut(at, (HEAD + 8 * positions.len()) as u64).ok_or(SyscallError::BadAddress)?; for (i, word) in head.to_words().iter().chain(positions).enumerate() { - window.write_at(8 * i, &word.to_le_bytes()); + window.write_at(8 * i, &word.to_le_bytes())?; } Ok(()) }; @@ -282,7 +284,17 @@ pub(super) fn sys_sysinfo(syscap: RawHandle, out: &mut UserBytesMut) -> u64 { return e.refuse(); } } + match sysinfo_laid(max_entries) { + Ok(laid) => out.write_at(0, &laid).map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| laid.len() as u64), + Err(e) => e.to_u64(), + } +} +/// [`sys_sysinfo`]'s answer, its header and at most `max_entries` entries, +/// laid out under the process table's lock for the caller to copy out after. +fn sysinfo_laid(max_entries: usize) -> Result, SyscallError> { + const HEADER_SIZE: usize = toyos_abi::syscall::SYSINFO_HEADER_SIZE; + const ENTRY_SIZE: usize = toyos_abi::syscall::SYSINFO_ENTRY_SIZE; let (total_mem, used_mem) = crate::mm::pmm::stats(); let cpu_count = crate::smp::cpu_count(); let uptime = crate::clock::nanos_since_boot(); @@ -294,7 +306,7 @@ pub(super) fn sys_sysinfo(syscap: RawHandle, out: &mut UserBytesMut) -> u64 { let entry_count = live_threads(table) as u32; if entry_count as usize > sysinfo_thread_bound() { - return SyscallError::ResourceExhausted.to_u64(); + return Err(SyscallError::ResourceExhausted); } let mut header = [0u8; HEADER_SIZE]; @@ -305,19 +317,20 @@ pub(super) fn sys_sysinfo(syscap: RawHandle, out: &mut UserBytesMut) -> u64 { header[24..32].copy_from_slice(&uptime.to_le_bytes()); header[32..40].copy_from_slice(&total_cpu_ns.to_le_bytes()); header[40..48].copy_from_slice(&total_available_ns.to_le_bytes()); - out.write_at(0, &header); + let mut laid = Vec::new(); + laid.extend_from_slice(&header); - // Header-only callers pay for no allocation and no sort of the roster below. + // Header-only callers pay for no sort of the roster below. if max_entries == 0 { - return HEADER_SIZE as u64; + return Ok(laid); } + laid.reserve_exact(max_entries.min(entry_count as usize) * ENTRY_SIZE); let mut entries: Vec<(process::Tid, &process::ProcessEntry, &process::ThreadEntry)> = Vec::with_capacity(entry_count as usize); entries.extend(table.iter().flat_map(|(_, proc)| proc.threads().iter().map(move |(&tid, thread)| (tid, proc, thread)))); entries.sort_by_key(|(tid, proc, _)| (proc.pid(), *tid)); - let mut pos = HEADER_SIZE; for (i, &(tid, proc, thread)) in entries.iter().enumerate() { if i >= max_entries { break; @@ -346,12 +359,9 @@ pub(super) fn sys_sysinfo(syscap: RawHandle, out: &mut UserBytesMut) -> u64 { entry[16..24].copy_from_slice(&memory.to_le_bytes()); entry[24..32].copy_from_slice(&cpu_ns.to_le_bytes()); entry[32..60].copy_from_slice(name); - out.write_at(pos, &entry); - - pos += ENTRY_SIZE; + laid.extend_from_slice(&entry); } - - pos as u64 + Ok(laid) } /// Every online CPU's counters into `out`, or the CPU count when `out` is @@ -375,8 +385,8 @@ pub(super) fn sys_boot_sources(syscap: RawHandle, out: &mut UserBytesMut) -> u64 if let Err(e) = demand_syscap(syscap, Rights::INVENTORY) { return e.refuse(); } - out.write_at(0, toyos_abi::usersafe::bytes(&crate::loaded::sources())); - 0 + out.write_at(0, toyos_abi::usersafe::bytes(&crate::loaded::sources())) + .map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| 0) } /// The most records one `SYS_DEVICE_INVENTORY` buffer may be declared to hold, @@ -428,9 +438,12 @@ pub(super) fn sys_device_set_read(set: RawHandle, out: &mut UserBytesMut) -> u64 functions: u32::try_from(functions.len()).expect("pcidev: fewer functions than a u32 counts"), declaration, }; - out.write_at(0, &header.encode().0); - for (at, pci) in functions.into_iter().enumerate() { - out.write_at((at + 1) * RECORD_BYTES, &devset::function(pci).0); + let wrote = out.write_at(0, &header.encode().0).and_then(|()| { + functions.into_iter().enumerate().try_for_each(|(at, pci)| out.write_at((at + 1) * RECORD_BYTES, &devset::function(pci).0)) + }); + // A reader that was not told this generation has not seen it. + if let Err(faulted) = wrote { + return SyscallError::from(faulted).to_u64(); } set.saw(generation); count as u64 @@ -453,8 +466,6 @@ pub(super) fn sys_device_inventory(syscap: RawHandle, out: &mut UserBytesMut) -> if out.len() < records.len() * RECORD_BYTES { return SyscallError::ResourceExhausted.to_u64(); } - for (i, record) in records.iter().enumerate() { - out.write_at(i * RECORD_BYTES, &record.encode().0); - } - records.len() as u64 + let wrote = records.iter().enumerate().try_for_each(|(i, record)| out.write_at(i * RECORD_BYTES, &record.encode().0)); + wrote.map_or_else(|faulted| SyscallError::from(faulted).to_u64(), |()| records.len() as u64) } diff --git a/kernel/src/syscall/proc.rs b/kernel/src/syscall/proc.rs index 9fe384dcd5b..b1d1820a483 100644 --- a/kernel/src/syscall/proc.rs +++ b/kernel/src/syscall/proc.rs @@ -216,8 +216,10 @@ pub(super) fn sys_endowments(out: &mut crate::user_ptr::UserBytesMut) -> u64 { let mut buf = alloc::vec![0u8; needed]; data.endowments.encode(&mut buf); drop(data); - out.write_at(0, &buf); - needed as u64 + match out.write_at(0, &buf) { + Ok(()) => needed as u64, + Err(faulted) => SyscallError::from(faulted).to_u64(), + } } /// Spawn a thread; refuses a `stack_base` above `stack_ptr` (no stack to clamp to) diff --git a/kernel/src/syscall/vm.rs b/kernel/src/syscall/vm.rs index 3c166aed0b4..9bf8aa174ae 100644 --- a/kernel/src/syscall/vm.rs +++ b/kernel/src/syscall/vm.rs @@ -367,7 +367,8 @@ pub(super) fn sys_query_modules(out: &mut UserBytesMut) -> u64 { use toyos_abi::syscall::ModuleInfo; let info_size = core::mem::size_of::(); - process::with_process_data(|data| { + // Laid out under the process's lock and copied out after it. + let laid = process::with_process_data(|data| { let module_count = 1 + data.elf.loaded_libs.len(); let exe_path_bytes = data.exe_path.as_bytes(); @@ -376,8 +377,10 @@ pub(super) fn sys_query_modules(out: &mut UserBytesMut) -> u64 { let required = module_count * info_size + total_path_bytes; if out.len() < required { - return required as u64; + return Err(required as u64); } + let mut laid = alloc::vec![0u8; required]; + let mut put = |at: usize, bytes: &[u8]| laid[at..at + bytes.len()].copy_from_slice(bytes); // Record array ends at the first module's `path_offset`; paths are // packed after it in module order. @@ -395,8 +398,8 @@ pub(super) fn sys_query_modules(out: &mut UserBytesMut) -> u64 { path_offset, path_len: exe_path_bytes.len() as u32, }; - out.write_at(0, toyos_abi::usersafe::bytes(&exe_info)); - out.write_at(path_offset as usize, exe_path_bytes); + put(0, toyos_abi::usersafe::bytes(&exe_info)); + put(path_offset as usize, exe_path_bytes); path_offset += exe_path_bytes.len() as u32; for (i, lib) in data.elf.loaded_libs.iter().enumerate() { @@ -415,11 +418,17 @@ pub(super) fn sys_query_modules(out: &mut UserBytesMut) -> u64 { path_offset, path_len: lib_path_bytes.len() as u32, }; - out.write_at((1 + i) * info_size, toyos_abi::usersafe::bytes(&lib_info)); - out.write_at(path_offset as usize, lib_path_bytes); + put((1 + i) * info_size, toyos_abi::usersafe::bytes(&lib_info)); + put(path_offset as usize, lib_path_bytes); path_offset += lib_path_bytes.len() as u32; } - - required as u64 - }) + Ok(laid) + }); + match laid { + Ok(laid) => match out.write_at(0, &laid) { + Ok(()) => laid.len() as u64, + Err(faulted) => SyscallError::from(faulted).to_u64(), + }, + Err(required) => required, + } } diff --git a/kernel/src/user_ptr.rs b/kernel/src/user_ptr.rs index a34eb7a9112..e129405bf9c 100644 --- a/kernel/src/user_ptr.rs +++ b/kernel/src/user_ptr.rs @@ -1,17 +1,22 @@ -//! Safe user memory access through the kernel direct map. A copy into user -//! memory lands only where a ring 3 store from the process itself could. +//! User memory, copied in and out through the running process's own mapping +//! (`arch::uaccess`): a copy reaches only what a load or store the process +//! made itself could reach, and a byte it cannot reach ends the copy as +//! [`Faulted`], which a syscall answers as `BadAddress`. Values are copied +//! out, never referenced. //! -//! SMAP stays enabled; access goes through the direct map, never stac/clac. -//! Values are copied out, never referenced. +//! **A window is a range of the user half**, checked for its end, overflow +//! included, when it is made, and by nothing else: whether its pages are +//! mapped is found by the copy. A page the copy finds absent is served as the +//! process's own fault would be (`process::handle_page_fault`), and the copy +//! resumes; one nothing serves, or one the process may not touch so, ends it. +//! A window over device memory the process mapped is copied through, making +//! only the accesses the process could make through that mapping itself +//! (`arch::uaccess`). //! -//! **A window is a range of the caller's own space**, checked against the -//! space's ranges when it is made. Each copy reaches its pages then, a chunk at -//! a time: under the space's lock the chunk's pages are reached — populated -//! where empty — and their frames pinned, and the copy runs with the lock given -//! back and preemption off. So a frame a sibling's `munmap` frees meanwhile is -//! not reissued until the copy is done with it, no pin outlives its chunk, and -//! no CPU holds more than one chunk's pins. A page the process took away since -//! the window was made reads as zeros and drops what is written to it. +//! **A copy holds no kernel lock**: its fault takes its space's lock, and in +//! 7b will wait on a pager. Every spin lock raises the preemption count, so +//! each copy refuses to run above the depth its syscall's entry runs at +//! ([`gate`]), whether or not a page faults. //! //! Every single-word user read is `read_volatile`, including the futex word //! and the crash dump's walk, both outside this module. @@ -25,7 +30,7 @@ use kernel::vm::Access as Asked; use toyos_abi::syscall::SyscallError; use toyos_abi::UserSafe; -use crate::mm::PAGE_SIZE; +use crate::arch::uaccess; use crate::UserAddr; /// Longest string, in bytes, the kernel accepts from userspace; one bound for every string syscall, since each copies or tokenizes rather than streaming. @@ -33,9 +38,6 @@ pub const MAX_USER_STR: u64 = 64 * 1024; pub(crate) use toyos_userbound::Access; -/// Pages one chunk of a copy reaches and pins at once. -const CHUNK_PAGES: usize = 16; - fn asked(access: Access) -> Asked { match access { Access::Read => Asked::Read, @@ -46,11 +48,49 @@ fn asked(access: Access) -> Asked { /// Translate a user virtual address to its direct-map address, populating its /// page if needed; `pub(crate)` because the futex word outlives its syscall. pub(crate) fn translate_user(addr: UserAddr, access: Access) -> Option { - let space = crate::process::current_address_space(); + let space = unlocked_space(); let phys = space.lock().reach(addr.raw(), asked(access))?; Some(crate::mm::DirectMap::from_phys(phys)) } +/// The current space, for a reach made holding no lock. +fn unlocked_space() -> crate::mm::space::PageTables { + gate(); + crate::process::current_address_space() +} + +/// Refuses a user copy made holding a lock or with interrupts masked, whose +/// fault could not be served: a kernel bug, never a user's doing. +fn gate() { + let depth = crate::preempt::count(); + if depth > crate::scheduler::BASELINE_TRAP { + under_a_lock(depth); + } + assert!(crate::arch::cpu::interrupts_enabled(), "user_ptr: a user copy with interrupts masked"); +} + +/// A copy met a user byte its process could not reach. +#[derive(Debug)] +pub struct Faulted; + +impl From for SyscallError { + fn from(_: Faulted) -> Self { + SyscallError::BadAddress + } +} + +/// `Ok` where a copy left no byte behind. +fn whole(left: usize) -> Result<(), Faulted> { + if left == 0 { Ok(()) } else { Err(Faulted) } +} + +/// The backtrace names the copy's caller. +#[cold] +#[inline(never)] +fn under_a_lock(depth: u32) -> ! { + panic!("user_ptr: a user copy at preemption depth {depth}, past its syscall's: a lock is held, and its fault-in may wait"); +} + /// Context for a single syscall invocation; the lifetime `'a` keeps validated references from escaping it. pub struct SyscallContext<'a> { _scope: PhantomData<&'a mut ()>, @@ -65,12 +105,12 @@ impl<'a> SyscallContext<'a> { /// A bulk buffer the kernel reads out of and never borrows. pub fn user_bytes(&self, ptr: UserAddr, len: u64) -> Option> { - Some(UserBytes(View::window(ptr, len, Access::Read)?, PhantomData)) + Some(UserBytes(View::window(ptr, len)?, PhantomData)) } /// A bulk buffer the kernel writes into and never borrows. pub fn user_bytes_mut(&self, ptr: UserAddr, len: u64) -> Option> { - Some(UserBytesMut(View::window(ptr, len, Access::Write)?, PhantomData)) + Some(UserBytesMut(View::window(ptr, len)?, PhantomData)) } /// Copy a user string of at most [`MAX_USER_STR`] bytes into kernel memory; an over-long or non-UTF-8 string is `InvalidArgument`, not `BadAddress`. @@ -84,16 +124,16 @@ impl<'a> SyscallContext<'a> { /// Read a typed value out of user memory, copied rather than borrowed. pub fn copy_in(&self, ptr: UserAddr) -> Result { - let view = object::(ptr, Access::Read)?; + let view = object::(ptr)?; let mut value = toyos_abi::usersafe::zeroed::(); - view.read_into(0, toyos_abi::usersafe::bytes_mut(&mut value)); + view.read_into(0, toyos_abi::usersafe::bytes_mut(&mut value))?; Ok(value) } /// Write a typed value into user memory. pub fn copy_out(&self, ptr: UserAddr, value: &T) -> Result<(), SyscallError> { - let view = object::(ptr, Access::Write)?; - view.write_from(0, toyos_abi::usersafe::bytes(value)); + let view = object::(ptr)?; + view.write_from(0, toyos_abi::usersafe::bytes(value))?; Ok(()) } @@ -101,103 +141,72 @@ impl<'a> SyscallContext<'a> { pub fn user_vec(&self, ptr: UserAddr, len: u64) -> Result, SyscallError> { let bytes = self.user_bytes(ptr, len).ok_or(SyscallError::BadAddress)?; let mut out = vec![0u8; bytes.len()]; - bytes.read_at(0, &mut out); + bytes.read_at(0, &mut out)?; Ok(out) } } /// A typed value's window: aligned for `T` and inside the user half. -fn object(ptr: UserAddr, access: Access) -> Result { +fn object(ptr: UserAddr) -> Result { let (size, align) = (core::mem::size_of::() as u64, core::mem::align_of::() as u64); if !toyos_userbound::is_user_object(ptr.raw(), size, align) { return Err(SyscallError::BadAddress); } - View::window(ptr, size, access).ok_or(SyscallError::BadAddress) + View::window(ptr, size).ok_or(SyscallError::BadAddress) } -/// `len` bytes of the caller's space from `start`, which its ranges allowed -/// `access` to when the view was made. +/// `len` bytes of the user half from `start`. #[derive(Clone, Copy)] struct View { start: u64, len: usize, - access: Access, } -/// What a page a sibling took away reads as. -static ZEROS: [u8; PAGE_SIZE as usize] = [0; PAGE_SIZE as usize]; - impl View { - fn window(ptr: UserAddr, len: u64, access: Access) -> Option { - if !toyos_userbound::in_user_half(ptr.raw(), len) { - return None; - } - let view = Self { start: ptr.raw(), len: usize::try_from(len).ok()?, access }; - if len == 0 { - return Some(view); - } - let space = crate::process::current_address_space(); - let allowed = space.lock().allows(ptr.raw(), ptr.raw() + len, asked(access)); - allowed.then_some(view) + fn window(ptr: UserAddr, len: u64) -> Option { + toyos_userbound::in_user_half(ptr.raw(), len).then_some(Self { start: ptr.raw(), len: usize::try_from(len).ok()? }) } - /// Hands `copy` each piece of bytes `[off, off + len)` of this view as a - /// direct-map pointer, the piece's offset in that range and its length; - /// panics if out of bounds — the kernel's own arithmetic, never user input. - fn pieces(&self, what: &str, off: usize, len: usize, mut copy: impl FnMut(*mut u8, usize, usize)) { + /// The user address of bytes `[off, off + len)` of this view; panics if + /// out of bounds — the kernel's own arithmetic, never user input. + fn at(&self, what: &str, off: usize, len: usize) -> u64 { assert!(off.checked_add(len).is_some_and(|end| end <= self.len), "{what} {off}+{len} past a {}-byte window", self.len); - let space = crate::process::current_address_space(); - let mut done = 0; - while done < len { - let mut frames = [None::; CHUNK_PAGES]; - let mut pieces = [(0u64, 0usize, 0usize); CHUNK_PAGES]; - let mut count = 0; - { - let mut guard = space.lock(); - while count < CHUNK_PAGES && done < len { - let at = self.start + (off + done) as u64; - let n = ((PAGE_SIZE - at % PAGE_SIZE) as usize).min(len - done); - let phys = guard - .reach(at, asked(self.access)) - .filter(|&phys| crate::mm::pmm::pin_range(phys & !(PAGE_SIZE - 1), PAGE_SIZE as usize)); - frames[count] = phys; - pieces[count] = (phys.unwrap_or(0), done, n); - count += 1; - done += n; - } - } - crate::sched::driver::preempt_off(|_| { - for (frame, &(phys, at, n)) in frames[..count].iter().zip(&pieces[..count]) { - match (frame, self.access) { - (Some(_), _) => copy(crate::mm::DirectMap::from_phys(phys).as_mut_ptr(), at, n), - (None, Access::Read) => copy(ZEROS.as_ptr().cast_mut(), at, n), - (None, Access::Write) => {} - } - } - }); - for phys in frames[..count].iter().flatten() { - crate::mm::pmm::unpin_range(phys & !(PAGE_SIZE - 1), PAGE_SIZE as usize); - } - } + gate(); + self.start + off as u64 + } + + fn read_into(&self, off: usize, dst: &mut [u8]) -> Result<(), Faulted> { + // SAFETY: `dst` is an owned `&mut` of that length, and `at` keeps the + // user range inside this window, inside the user half. + unsafe { self.read_raw(off, dst.as_mut_ptr(), dst.len()) } + } + + fn write_from(&self, off: usize, src: &[u8]) -> Result<(), Faulted> { + // SAFETY: as `read_into`, with `src` readable for its length. + unsafe { self.write_raw(off, src.as_ptr(), src.len()) } + } + + /// # Safety + /// `dst` is writable kernel memory for `len` bytes. + unsafe fn read_raw(&self, off: usize, dst: *mut u8, len: usize) -> Result<(), Faulted> { + let user = self.at("UserBytes::read", off, len); + // SAFETY: the caller's `dst`; `at` keeps the user range in the half, + // and `gate` holds no lock across the copy. + whole(unsafe { uaccess::copy_from_user(dst, user, len) }) } - fn read_into(&self, off: usize, dst: &mut [u8]) { - let into = dst.as_mut_ptr(); - self.pieces("UserBytes::read_at", off, dst.len(), |run, at, n| { - // SAFETY: `pieces` bounds `[at, at + n)` inside `dst` and `run..+n` - // inside one pinned page or the zero page; `dst` is an owned - // `&mut`, so the ranges cannot overlap. - unsafe { core::ptr::copy_nonoverlapping(run, into.add(at), n) } - }); + /// # Safety + /// `src` is readable kernel memory for `len` bytes. + unsafe fn write_raw(&self, off: usize, src: *const u8, len: usize) -> Result<(), Faulted> { + let user = self.at("UserBytesMut::write", off, len); + // SAFETY: as `read_raw`. + whole(unsafe { uaccess::copy_to_user(user, src, len) }) } - fn write_from(&self, off: usize, src: &[u8]) { - let from = src.as_ptr(); - self.pieces("UserBytesMut::write_at", off, src.len(), |run, at, n| { - // SAFETY: `pieces` bounds `[at, at + n)` inside `src` and `run..+n` - // inside one pinned page its range let the process write. - unsafe { core::ptr::copy_nonoverlapping(from.add(at), run, n) } - }); + fn fill_zero(&self, off: usize, len: usize) -> Result<(), Faulted> { + let user = self.at("UserBytesMut::fill_zero", off, len); + // SAFETY: as `read_raw`, with no kernel memory touched. + whole(unsafe { uaccess::clear_user(user, len) }) } } @@ -210,19 +219,8 @@ impl UserBytes<'_> { } /// Copy `dst.len()` bytes out of the window at `off`; panics if out of bounds — the kernel's own arithmetic, never user input. - pub fn read_at(&self, off: usize, dst: &mut [u8]) { - self.0.read_into(off, dst); - } - - /// Hand `copy` each piece of the window's `[off, off + len)`, as - /// [`read_at`](Self::read_at) reaches them, one chunk of pages pinned at - /// a time: a pointer to its bytes, its offset in that range and its - /// length. The pointer is valid for that length until `copy` returns, - /// with preemption off, and is never made a reference; a page the - /// process took away reads as zeros. Panics if out of bounds, as - /// [`read_at`](Self::read_at) does. - pub fn read_pieces(&self, off: usize, len: usize, mut copy: impl FnMut(*const u8, usize, usize)) { - self.0.pieces("UserBytes::read_pieces", off, len, |run, at, n| copy(run.cast_const(), at, n)); + pub fn read_at(&self, off: usize, dst: &mut [u8]) -> Result<(), Faulted> { + self.0.read_into(off, dst) } /// Copy `len` bytes out of the window at `off` to `dst`, never through a @@ -230,12 +228,9 @@ impl UserBytes<'_> { /// /// # Safety /// `dst` is writable for `len` bytes, kernel memory no window names. - pub unsafe fn read_raw(&self, off: usize, dst: *mut u8, len: usize) { - self.0.pieces("UserBytes::read_raw", off, len, |run, at, n| { - // SAFETY: `pieces` bounds `[at, at + n)` inside `len` and `run..+n` - // inside one pinned page or the zero page; `dst` is the caller's. - unsafe { core::ptr::copy_nonoverlapping(run, dst.add(at), n) } - }); + pub unsafe fn read_raw(&self, off: usize, dst: *mut u8, len: usize) -> Result<(), Faulted> { + // SAFETY: the caller's contract. + unsafe { self.0.read_raw(off, dst, len) } } } @@ -252,15 +247,8 @@ impl UserBytesMut<'_> { } /// Copy `src` into the window at `off`; panics if out of bounds, for the same reason as [`UserBytes::read_at`]. - pub fn write_at(&mut self, off: usize, src: &[u8]) { - self.0.write_from(off, src); - } - - /// [`UserBytes::read_pieces`] for a copy into the window: each pointer - /// may be written for its length until `copy` returns, and a page the - /// process took away drops what is written to it. - pub fn write_pieces(&mut self, off: usize, len: usize, copy: impl FnMut(*mut u8, usize, usize)) { - self.0.pieces("UserBytesMut::write_pieces", off, len, copy); + pub fn write_at(&mut self, off: usize, src: &[u8]) -> Result<(), Faulted> { + self.0.write_from(off, src) } /// Copy `len` bytes from `src` into the window at `off`, never through a @@ -268,19 +256,21 @@ impl UserBytesMut<'_> { /// /// # Safety /// `src` is readable for `len` bytes, kernel memory no window names. - pub unsafe fn write_raw(&mut self, off: usize, src: *const u8, len: usize) { - self.0.pieces("UserBytesMut::write_raw", off, len, |run, at, n| { - // SAFETY: `pieces` bounds `[at, at + n)` inside `len` and `run..+n` - // inside one pinned page the process may write; `src` is the caller's. - unsafe { core::ptr::copy_nonoverlapping(src.add(at), run, n) } - }); + pub unsafe fn write_raw(&mut self, off: usize, src: *const u8, len: usize) -> Result<(), Faulted> { + // SAFETY: the caller's contract. + unsafe { self.0.write_raw(off, src, len) } } /// Zero `len` bytes of the window at `off`. - pub fn fill_zero(&mut self, off: usize, len: usize) { - self.0.pieces("UserBytesMut::fill_zero", off, len, |run, _, n| { - // SAFETY: as `write_at`, with a constant zero byte instead of a slice. - unsafe { core::ptr::write_bytes(run, 0, n) } - }); + pub fn fill_zero(&mut self, off: usize, len: usize) -> Result<(), Faulted> { + self.0.fill_zero(off, len) + } + + /// Populate the window's pages under `[off, off + len)` for writing, in + /// one hold of the space's lock, up to the first its process may not + /// write; the copy faults on any left out. `len` bounds the hold. + pub fn fault_in(&mut self, off: usize, len: usize) { + let start = self.0.at("UserBytesMut::fault_in", off, len); + crate::process::fault_in(start, start + len as u64, kernel::vm::Access::Write); } } diff --git a/lib/userbound/src/span.rs b/lib/userbound/src/span.rs index 7dcd8d0b3e7..42c9a224fef 100644 --- a/lib/userbound/src/span.rs +++ b/lib/userbound/src/span.rs @@ -86,8 +86,7 @@ pub fn rebase_base(vm_base: u64, vaddr_min: u64, span: u64) -> Option { } /// Whether the kernel may read or write a `size`-byte value of alignment -/// `align` at `ptr`: aligned, and wholly in the user half. A value across a -/// page boundary is copied in pieces, one per page. +/// `align` at `ptr`: aligned, and wholly in the user half. pub fn is_user_object(ptr: u64, size: u64, align: u64) -> bool { size != 0 && align.is_power_of_two() && ptr.is_multiple_of(align) && in_user_half(ptr, size) } diff --git a/sdk/toyos/src/sync.rs b/sdk/toyos/src/sync.rs index da95b4dd490..9c0f533888a 100644 --- a/sdk/toyos/src/sync.rs +++ b/sdk/toyos/src/sync.rs @@ -10,6 +10,7 @@ use core::cell::UnsafeCell; use core::sync::atomic::{AtomicU32, Ordering}; +#[cfg(not(test))] use toyos_abi::syscall; /// `T` behind a futex lock. @@ -71,6 +72,13 @@ fn wake_all(word: &AtomicU32) { unsafe { syscall::futex_wake(word.as_ptr(), u32::MAX) }; } +/// Wake one sleeper on `word`. +#[cfg(not(test))] +fn wake_one(word: &AtomicU32) { + // SAFETY: `word` is a live, aligned u32. + unsafe { syscall::futex_wake(word.as_ptr(), 1) }; +} + // A host test has no futex: a waiter yields and spins. #[cfg(test)] fn wait(_: &AtomicU32, _: u32) { @@ -80,6 +88,9 @@ fn wait(_: &AtomicU32, _: u32) { #[cfg(test)] fn wake_all(_: &AtomicU32) {} +#[cfg(test)] +fn wake_one(_: &AtomicU32) {} + /// `T` held in turn: each [`Turnstile::lock`] takes a ticket, and the value /// goes to the tickets in the order they were taken. A plain lock lets the /// thread that let go take it straight back ahead of the one it woke, so @@ -145,22 +156,39 @@ impl Drop for Turn<'_, T> { } } +/// How many times a contender looks again before it sleeps. A futex wait and +/// the wake that ends it cost a kernel round trip each way, microseconds, +/// while a holder of these locks lets go within a few hundred nanoseconds: +/// spinning about as long as a block would cost before blocking is the +/// competitive choice (Karlin, Li, Manasse and Owicki, "Empirical Studies of +/// Competitive Spinning for a Shared-Memory Multiprocessor", SOSP 1991). A +/// `spin_loop` hint is tens to about 140 cycles depending on the CPU, so 100 +/// of them stay within that cost. +const SPINS: u32 = 100; + /// Takes the lock `state` is the word of, sleeping while another holds it. pub fn lock_word(state: &AtomicU32) { if state.compare_exchange(0, 1, Ordering::Acquire, Ordering::Relaxed).is_ok() { return; } + // Held with no sleeper: its holder is running and lets go soon. A word + // that says sleepers are waiting goes straight to sleep behind them. + for _ in 0..SPINS { + match state.load(Ordering::Relaxed) { + 0 if state.compare_exchange(0, 1, Ordering::Acquire, Ordering::Relaxed).is_ok() => return, + 2 => break, + _ => core::hint::spin_loop(), + } + } while state.swap(2, Ordering::Acquire) != 0 { - // SAFETY: `state` is a live, aligned u32. - unsafe { syscall::futex_wait(state.as_ptr(), 2, None) }; + wait(state, 2); } } /// Lets go of the lock `state` is the word of, waking one sleeper if any. pub fn unlock_word(state: &AtomicU32) { if state.swap(0, Ordering::Release) == 2 { - // SAFETY: `state` is a live, aligned u32. - unsafe { syscall::futex_wake(state.as_ptr(), 1) }; + wake_one(state); } } @@ -174,6 +202,33 @@ mod tests { /// A thread that lets the value go and asks for it again waits behind /// the threads that asked first, in the order they asked. + /// Threads contending for one lock, each holding it only briefly, never + /// hold it together: every increment made under it lands. + #[test] + fn contenders_for_a_lock_each_hold_it_alone() { + const THREADS: usize = 4; + const EACH: usize = 20_000; + let lock: Arc> = Arc::new(Lock::new(0)); + let threads: Vec<_> = (0..THREADS) + .map(|_| { + let lock = Arc::clone(&lock); + std::thread::spawn(move || { + for _ in 0..EACH { + let mut held = lock.lock(); + // A read and a separate write, which two holders at once would lose. + let seen = *held; + core::hint::spin_loop(); + *held = seen + 1; + } + }) + }) + .collect(); + for thread in threads { + thread.join().unwrap(); + } + assert_eq!(*lock.lock(), THREADS * EACH); + } + #[test] fn the_value_goes_to_the_tickets_in_the_order_they_were_taken() { let stile: Arc>> = Arc::new(Turnstile::new(Vec::new())); diff --git a/tests/guest/rust-tests/src/bin/user_copy_faults.rs b/tests/guest/rust-tests/src/bin/user_copy_faults.rs new file mode 100644 index 00000000000..538747f65f0 --- /dev/null +++ b/tests/guest/rust-tests/src/bin/user_copy_faults.rs @@ -0,0 +1,120 @@ +//! A user copy goes through the process's own mapping (`kernel/src/user_ptr.rs`, +//! `arch::uaccess`), so it meets the pages a program's own loads and stores +//! would: a page never touched faults inside the copy, is served, and the copy +//! resumes where it stopped; a page released, or one the program may not read, +//! ends the copy as `BadAddress`, and the kernel goes on. +//! +//! Each buffer starts in a page the program touched and runs into one it did +//! not, so the first fault comes mid-copy rather than at its first byte. + +use toyos_abi::syscall::{self, NamespaceBuild, SyscallError, VmAt, VmProt, PAGE_SIZE}; + +const PAGE: usize = PAGE_SIZE; +/// Bytes of each buffer on either side of the page boundary it straddles. +const SIDE: usize = 300; + +fn pattern(len: usize, seed: u8) -> Vec { + (0..len).map(|i| (i as u8).wrapping_mul(31).wrapping_add(seed)).collect() +} + +/// `pages` fresh pages, none of them touched yet. +fn fresh(pages: usize) -> *mut u8 { + // SAFETY: a fresh anonymous mapping aliases nothing. + unsafe { syscall::vm_reserve(pages * PAGE, VmProt::ReadWrite, VmAt::ANYWHERE) }.expect("vm_reserve") +} + +/// A pipe write whose buffer runs into an untouched page: the kernel's read of +/// it faults there, is served, and resumes. Then a read into a buffer that runs +/// into an untouched page, which faults the same way on the kernel's write. +fn a_fault_served_mid_copy_resumes(len: usize, seed: u8) { + let (reader, writer) = syscall::pipe().map(|ends| (ends.read, ends.write)).expect("pipe"); + let src = fresh(len.div_ceil(PAGE) + 1); + let expected = pattern(len, seed); + // SAFETY: inside the fresh mapping; only the first page's last `SIDE` + // bytes and none after are touched here. + let start = unsafe { src.add(PAGE - SIDE) }; + // SAFETY: `start..start + SIDE` lies in the first page, which this writes and so populates. + unsafe { core::ptr::copy_nonoverlapping(expected.as_ptr(), start, SIDE) }; + // SAFETY: the rest of the buffer, in pages never touched: written here would + // populate them, so the program's bytes past `SIDE` are written by the kernel's + // fault service as zeros. The buffer is the mapping's own, `len` long. + let buf = unsafe { core::slice::from_raw_parts(start, len) }; + let mut want = expected[..SIDE].to_vec(); + want.resize(len, 0); + let mut wrote = 0; + while wrote < len { + wrote += syscall::write(writer, &buf[wrote..]).unwrap_or_else(|e| panic!("write of {len} from a buffer half untouched: {e:?}")); + } + let dst = fresh(len.div_ceil(PAGE) + 1); + // SAFETY: inside the second mapping; its first page is touched once so + // the read's fault comes past its first `SIDE` bytes. + let into = unsafe { + dst.add(PAGE - SIDE).write_volatile(0); + core::slice::from_raw_parts_mut(dst.add(PAGE - SIDE), len) + }; + let mut read = 0; + while read < len { + read += syscall::read(reader, &mut into[read..]).unwrap_or_else(|e| panic!("read of {len} into a buffer half untouched: {e:?}")); + } + assert!(into == want.as_slice(), "{len} bytes through a pipe, both buffers faulting mid-copy, came back changed"); + println!("a fault served mid-copy resumes: {len} bytes"); + // SAFETY: the pipe's ends are this function's alone. + unsafe { + syscall::close(reader); + syscall::close(writer); + } +} + +/// `SYS_NAMESPACE_BUILD` reads its names blob whole before anything else, so a +/// blob that runs off its mapping, or into a page the program may not read, +/// is refused `BadAddress` mid-copy, and a kernel address before any copy. +fn a_byte_nothing_serves_ends_the_copy() { + let blob = fresh(2); + // SAFETY: inside the mapping: the first page is touched and the second given back. + unsafe { + blob.add(PAGE - SIDE).write_volatile(b'a'); + syscall::vm_release(blob.add(PAGE), PAGE).expect("release the second page"); + } + let build = |names_ptr: u64, names_len: u64| { + let args = NamespaceBuild { + base: toyos_abi::handle::HANDLE_INVALID, + flags: 0, + keep_ptr: 0, + keep_n: 0, + add_ptr: 0, + add_n: 0, + names_ptr, + names_len, + }; + // SAFETY: the pointers name memory the kernel must refuse; nothing here reads it. + unsafe { syscall::namespace_build(&args) } + }; + let across = build(blob as u64 + (PAGE - SIDE) as u64, 2 * SIDE as u64); + assert_eq!(across, Err(SyscallError::BadAddress), "a names blob running into a released page"); + + let guarded = fresh(2); + // SAFETY: inside the mapping: the first page is touched, the second made unreadable. + unsafe { + guarded.add(PAGE - SIDE).write_volatile(b'a'); + syscall::vm_protect(guarded.add(PAGE), PAGE, VmProt::None).expect("protect the second page"); + } + let unreadable = build(guarded as u64 + (PAGE - SIDE) as u64, 2 * SIDE as u64); + assert_eq!(unreadable, Err(SyscallError::BadAddress), "a names blob running into an unreadable page"); + + let kernel = build(0xFFFF_8000_0000_0000, 16); + assert_eq!(kernel, Err(SyscallError::BadAddress), "a names blob at a kernel address"); + + let past = build(0x0000_7FFF_FFFF_FF00, 0x200); + assert_eq!(past, Err(SyscallError::BadAddress), "a names blob running past the user half"); + println!("a byte nothing serves ends the copy as BadAddress"); +} + +fn main() { + a_fault_served_mid_copy_resumes(2 * SIDE, 7); + // Past a bounce: the pipe's claimed path copies a chunk at a time. + a_fault_served_mid_copy_resumes(3 * PAGE + 2 * SIDE, 11); + a_byte_nothing_serves_ends_the_copy(); + // The kernel still answers after every refusal. + a_fault_served_mid_copy_resumes(2 * SIDE, 13); + println!("user copies: all checks passed"); +} diff --git a/tests/images/virtjobcase.toml b/tests/images/virtjobcase.toml index b54f57d5730..406c2b619c8 100644 --- a/tests/images/virtjobcase.toml +++ b/tests/images/virtjobcase.toml @@ -22,7 +22,7 @@ restart = true receives = ["power"] # Four minutes from boot rather than one: a job here may run on an emulated # CPU, and a job past the bound reboots the guest with the job named. -args = ["--bound-ms=240000", "preempt", "fp_isolation", "first_entry", "unmap_touch", "debug_refused", "test_rs_ring0_timer_in_syscall", "test_rs_random_draws", "test_rs_wall_clock_now", "test_rs_abuse_readonly_copyout"] +args = ["--bound-ms=240000", "preempt", "fp_isolation", "first_entry", "unmap_touch", "debug_refused", "test_rs_ring0_timer_in_syscall", "test_rs_random_draws", "test_rs_wall_clock_now", "test_rs_user_copy_faults", "test_rs_abuse_readonly_copyout"] [programs.kernelprobe] diff --git a/tests/toyos.rs b/tests/toyos.rs index 1f17350f37e..a3142fb1e92 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -369,6 +369,9 @@ const DRIVEN_AND_SHARED: &[&str] = &[ // Its shared run is the T14's verdict, on its served root; // `a_spawn_from_an_image_frees_it_when_the_child_ends` runs it under QEMU. "spawn_returns_the_census", + // Its shared run is each bench's verdict on its own CPU's fault path; + // `a_user_copy_faults_in_mid_copy_or_ends_as_bad_address` runs it under QEMU. + "user_copy_faults", // Its shared run is ring's known answers in a guest; the crypto boot runs it // first on each bench, and `crypto_instrument_prints_its_table` before the // instrument under QEMU. @@ -422,6 +425,9 @@ const SCREEN_TESTS: &[(&str, qemu::Profile)] = &[ ("virt_unmap_touch", qemu::Profile::Virt), ("virt_debug_refused", qemu::Profile::Virt), ("virt_readonly_copyout", qemu::Profile::Virt), + // `a_user_copy_faults_in_mid_copy_or_ends_as_bad_address` on AArch64: + // the unprivileged copies and their EL1 fixup (`arch::uaccess`). + ("virt_user_copy_faults", qemu::Profile::Virt), ("virt_ring0_timer_in_syscall", qemu::Profile::Virt), // Emulated, with `virt_el1_smp`, `virt_off_names_the_cpus_left_on` and // `virt_reboot`: under HVF a woken thread can go undispatched for seconds @@ -650,6 +656,13 @@ const MACHINE_TESTS: &[&str] = &[ // holds. One boot of QEMU's x86-64 machine on the actuator kernel, for the // census. "a_spawn_from_an_image_frees_it_when_the_child_ends", + // A user copy through the process's own mapping (`arch::uaccess`): a page + // first touched inside the copy is served and the copy resumes, both + // directions and past a bounce; a released, unreadable or kernel address + // ends it `BadAddress`. The kernel's fault fixup, which no host test + // reaches. One boot of QEMU's x86-64 machine; the shared boots run it on + // each bench. + "a_user_copy_faults_in_mid_copy_or_ends_as_bad_address", // The boot's log written through usbd onto the stick it booted from, and // still there once the machine has stopped: the path every metal boot's // `/log` takes, which no other QEMU row reads, because their console is @@ -2834,6 +2847,11 @@ const VIRT_RANDOM: &str = "random_draws"; /// The same for its job `test_rs_wall_clock_now`. const VIRT_WALL_CLOCK: &str = "wall_clock_now"; +/// The AArch64 guest's user copies through its own mapping (`arch::uaccess`): +/// the same binary and claims as +/// `a_user_copy_faults_in_mid_copy_or_ends_as_bad_address` on x86-64. +const VIRT_USER_COPY: &str = "user_copy_faults"; + /// The instant `tests/images/virtjobcase.toml`'s RTC starts at, and the same in Unix /// seconds: years from any host's own, so a reading on it came from the RTC. const VIRT_RTC_BASE: &str = "2033-03-07T09:14:25"; @@ -2853,6 +2871,7 @@ const VIRT_JOBS: &[(&str, &str)] = &[ ("test_rs_ring0_timer_in_syscall", "the timer interrupted the syscall's body and re-armed a quantum"), ("test_rs_random_draws", RANDOM_DRAWS_SAID), ("test_rs_wall_clock_now", WALL_CLOCK_SAID), + ("test_rs_user_copy_faults", "user copies: all checks passed"), ("test_rs_abuse_readonly_copyout", "a syscall writes only where its caller could store"), ]; @@ -2877,7 +2896,7 @@ fn boot_virt_jobs(profile: qemu::Profile) -> QemuInstance { smp: 1, kernel_features: toyos_build::build::TEST_KERNEL, ready_marker: "control registers: SCTLR_EL1=", - extra_root_files: [VIRT_COPYOUT, VIRT_RING0_TIMER, VIRT_RANDOM, VIRT_WALL_CLOCK] + extra_root_files: [VIRT_COPYOUT, VIRT_RING0_TIMER, VIRT_RANDOM, VIRT_WALL_CLOCK, VIRT_USER_COPY] .map(|name| suite_bin(profile.arch(), name)) .to_vec(), rtc_base: Some(VIRT_RTC_BASE), @@ -3293,6 +3312,33 @@ fn a_spawn_from_an_image_frees_it_when_the_child_ends(test_config: &Path) -> Res Ok(()) } +/// `user_copy_faults` alone: its copies that fault mid-copy and are served, +/// and its refusals, each said by name. +fn a_user_copy_faults_in_mid_copy_or_ends_as_bad_address(test_config: &Path) -> Result<(), String> { + const JOB: &str = "user_copy_faults"; + let bin = qemu::build_toyos_bin(qemu::SUITE_ARCH, &compile::repo_root().join("tests/guest/rust-tests"), JOB); + let mut qemu = QemuInstance::boot_with_options(test_config, &[], &[(JOB.to_string(), bin)], BootOptions::default()); + serial::Serial::boot(&qemu).must_be_clean()?; + let result = qemu.run_test("test_rs_user_copy_faults", Duration::from_secs(60)); + if let Some(why) = &result.error { + return Err(format!("{why}\nthe job said:\n{}", result.stdout)); + } + if result.exit_code != Some(0) { + return Err(format!("the job ended {:?}:\n{}", result.exit_code, result.stdout)); + } + for claim in [ + "a fault served mid-copy resumes: 600 bytes", + "a fault served mid-copy resumes: 12888 bytes", + "a byte nothing serves ends the copy as BadAddress", + "user copies: all checks passed", + ] { + if !result.stdout.contains(claim) { + return Err(format!("the job never said {claim:?}:\n{}", result.stdout)); + } + } + Ok(()) +} + /// What a boot whose marked slot's supervisor is bent says: the loader refused /// that slot by the section, and the kernel was booted from the other. fn bent_supervisor_fell_back(loader: &serial::Serial, kernel: &serial::Serial) -> Result<(), String> { @@ -4702,6 +4748,7 @@ fn run_screen_test(name: &str, profile: qemu::Profile, test_config: &Path) -> Re "virt_unmap_touch" => virt_job(profile, "unmap_touch"), "virt_debug_refused" => virt_job(profile, "debug_refused"), "virt_readonly_copyout" => virt_job(profile, &format!("test_rs_{VIRT_COPYOUT}")), + "virt_user_copy_faults" => virt_job(profile, &format!("test_rs_{VIRT_USER_COPY}")), "virt_ring0_timer_in_syscall" => virt_job(profile, &format!("test_rs_{VIRT_RING0_TIMER}")), "virt_jobs_at_el2" => virt_jobs_at_el2(profile), "virt_random_differs" => virt_random_differs(profile), @@ -5028,6 +5075,7 @@ fn run_machine_test(name: &str, test_config: &Path) -> Result<(), String> { "the_uart_carries_the_boot_once_across_the_hand_over" => the_uart_carries_the_boot_once_across_the_hand_over(), "a_supervisor_that_fails_early_says_why_on_the_uart" => a_supervisor_that_fails_early_says_why_on_the_uart(), "a_spawn_from_an_image_frees_it_when_the_child_ends" => a_spawn_from_an_image_frees_it_when_the_child_ends(test_config), + "a_user_copy_faults_in_mid_copy_or_ends_as_bad_address" => a_user_copy_faults_in_mid_copy_or_ends_as_bad_address(test_config), "a_bent_supervisor_is_refused_and_the_other_slot_boots" => { [qemu::Profile::Headless, qemu::Profile::Virt].into_iter().try_for_each(|profile| { let options = BootOptions { profile, bent: Some(toyos_update::slots::SUPERVISOR_FILE), ..Default::default() };