x86-64 user copies go by memcpy's rule, and a failed copy is a failure

issues/a-freshly-written-8-mib-file-reads-back-slower-than-a-16-mib-one-on-the-amd-bench.md

@@ -0,0 +1,27 @@
+---
+status: open
+kind: finding
+opened: 2026-11-11
+---
+
+# A freshly written 8 MiB file reads back slower than a 16 MiB one on the AMD bench
+
+`persist.seq_read_cached`'s pattern writes 8 MiB in 1 MiB calls, syncs, and
+reads the file back whole in 1 MiB calls. On the AMD bench, ToyOS at
+`422915e2c` with stage timers in a scratch kernel, its paged reads copied
+8.4 and 8.7 µs per chunk of 16 pages, against 6.1 and 6.2 µs for 1 MiB reads
+of a 16 MiB file written without a sync and read once through first. Both
+files were on the memory file system, both buffers at a page's start, so
+neither copy is the store-forwarding stall
+(`issues/a-userland-forward-copy-a-short-way-above-its-source-modulo-4-kib-stalls-on-amd.md`).
+The row's samples ran from 6.3 to 10.8 GB/s within a boot, p50 7.8 and 7.3
+GB/s, against Linux's 9.0 and 9.3. On the T14 the order is the other way:
+4.3 and 4.4 µs per chunk for the 8 MiB file, 5.5 for the 16 MiB one
+([evidence](https://github.com/ToyOSOrg/evidence/tree/cb8da5fc2168bfe0395b6bce364212b4117a170d/diagnosis)).
+
+Cause not found: the two differ in the sync, the file's size against the
+caches and the frames each file was written into.
+
+Exit: the cause measured, for example the same 8 MiB file read with and
+without the sync and at the 16 MiB file's size, and fixed or recorded where
+it lives.

issues/a-pipe-gives-back-the-frames-its-reader-empties-and-takes-them-again.md

@@ -0,0 +1,27 @@
+---
+status: open
+kind: finding
+opened: 2026-11-11
+---
+
+# A pipe gives back the frames its reader empties and takes them again
+
+A pipe's bytes live in frames it takes as writes need them and gives back as
+reads empty them (`kernel/src/pipe.rs`, `kernel/src/frame_bytes.rs`). Under a
+steady stream the same frames leave the pipe and come back: a small write that
+finds no room in a frame the pipe holds cannot be put in one hold of the
+pipe's lock under its process's (`PipeRef::put_now`), and takes the general
+path, which lets the lock go to take frames.
+
+On the T14, ToyOS at `422915e2c` with stage timers in a scratch kernel,
+`pipe.throughput.4096` with osbench's own buffers, two passes of 100 ms
+([evidence](https://github.com/ToyOSOrg/evidence/tree/cb8da5fc2168bfe0395b6bce364212b4117a170d/diagnosis)):
+- 35,552 of 102,720 and 38,050 of 105,216 writes (35% and 36%) took the
+  general path, at 609 and 616 ns each, against 447 and 475 ns for a write put
+  in one hold;
+- a read on the general path spent 412 and 418 ns after its copy, consuming
+  the frames it emptied, giving them back and waking the writer.
+
+Exit: the cost measured against a pipe that keeps the frames its reader
+empties for its next writes, on both benches, and that change made or the
+cost recorded at the pipe's module.

issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md

@@ -7,7 +7,8 @@ opened: 2026-09-29
 # Kernel forward copies and fills are one `rep movsb` or `rep stosb` on every CPU
 
 The kernel's `memcpy` is its own (`kernel/src/arch/x86_64/copy.rs`): one
-`rep movsb` on every CPU, but for a copy it sends highest first. It replaced
+`rep movsb` on every CPU, but for a copy it sends highest first, and its
+user copies go by the same rule (`arch::x86_64::uaccess`). It replaced
 `compiler_builtins`' `rep movsb; rep movsq; rep movsb`, so every forward copy
 changed with it, on Zen 2 without ERMS too: a kernel tmpfs read of 4 or 8 KiB
 took 0.92-0.96 times as long on both benches, at every distance outside the
@@ -40,3 +41,19 @@ with no `rep movsq` or `rep stosq`; the T14's pipe figures; and Zen 2's cost
 recorded, as a family-0x17 figure against the build without `ermsb` or as a
 `tooling` issue that owes it. **Mutation**: the build without `ermsb` reds the
 gate. **Oracle**: Linux's pipe figures.
+
+- **Owed on metal:** #1244, which sends user copies down by this rule, is not
+  measured on metal; its speed-up is a prediction. Owed: one A/B of main from
+  before and after it on both benches, four boots a side, each running
+  `osbench pipeab` (the measurement-only commit on `pipeperf-ab-base`): the
+  pipe rows by reader placement (`ab.pipe.natural.*`, `ab.pipe.<size>.r<r>.w<w>`)
+  and the positioned cached reads at 0, 16 and 64 bytes into a page
+  (`ab.fs.memory.pread.*`). A row changes where all four paired boots move one
+  way and their median ratio is beyond 1 ± 3%. Main after must show, on the
+  AMD bench, `ab.pipe.natural.4096`, `.16384` and `.65536` (the reader 16
+  bytes into a page) up from about 3.0 GB/s to about 7, 15 and 20 GB/s, the
+  rows with the reader at 0; each read 16 bytes into a page at most 1.25 times
+  the same read at 0, against 1.9 to 4 times before; and on the T14 none of
+  those rows changed. And main's `testcases` and `shared-debug` boots pass on
+  both benches, `user_copy_faults` among them, which on the AMD bench copies
+  down by its own rule.

kernel/src/actuator.rs

@@ -43,6 +43,11 @@ actuators! {
     /// x86-64 loads its IDT later, and panics by name instead.
     test_early_fault = "test-early-fault";
 
+    /// Send every copy of at least 64 bytes whose ends differ modulo 4 KiB
+    /// highest first, whatever the CPU (`arch::x86_64::copy`). Judged by
+    /// `a_user_copy_highest_first_faults_or_ends_as_bad_address`.
+    copies_go_down = "copies-go-down";
+
     /// Have the first syscall null SS, force a switch, and report whether it reloaded — the
     /// AMD `SYSRET` SS-attributes workaround's only guest-observable proof.
     sysret_ss_probe = "sysret-ss-probe";

kernel/src/arch/aarch64/uaccess.rs

@@ -23,6 +23,8 @@
 
 use toyos_userbound::USER_TOP;
 
+use crate::user_ptr::Faulted;
+
 core::arch::global_asm!(
     ".pushsection .text.toyos_uaccess, \"ax\"",
     // toyos_copy_from_user(x0 = kernel dst, x1 = user src, x2 = len) -> x0 = bytes not copied
@@ -162,34 +164,38 @@ unsafe extern "C" {
     static toyos_clear_user_end: u8;
 }
 
-/// Copy `len` bytes from user address `user` to `dst`; answers the bytes not
-/// copied.
+/// A routine's answer: 0 where it copied every byte.
+fn whole(left: usize) -> Result<(), Faulted> {
+    if left == 0 { Ok(()) } else { Err(Faulted) }
+}
+
+/// Copy `len` bytes from user address `user` to `dst`. On `Err` some or none
+/// of them were copied.
 ///
 /// # Safety
 /// `dst` is writable kernel memory for `len` bytes; `[user, user + len)` lies
 /// below [`USER_TOP`]; no lock is held (`crate::user_ptr`).
-pub unsafe fn copy_from_user(dst: *mut u8, user: u64, len: usize) -> usize {
+pub unsafe fn copy_from_user(dst: *mut u8, user: u64, len: usize) -> Result<(), Faulted> {
     // SAFETY: the caller's contract; the routine touches nothing else.
-    unsafe { toyos_copy_from_user(dst, user, len) }
+    whole(unsafe { toyos_copy_from_user(dst, user, len) })
 }
 
-/// Copy `len` bytes from `src` to user address `user`; answers the bytes not
-/// copied.
+/// Copy `len` bytes from `src` to user address `user`, as [`copy_from_user`].
 ///
 /// # Safety
 /// As [`copy_from_user`], with `src` readable.
-pub unsafe fn copy_to_user(user: u64, src: *const u8, len: usize) -> usize {
+pub unsafe fn copy_to_user(user: u64, src: *const u8, len: usize) -> Result<(), Faulted> {
     // SAFETY: the caller's contract; the routine touches nothing else.
-    unsafe { toyos_copy_to_user(user, src, len) }
+    whole(unsafe { toyos_copy_to_user(user, src, len) })
 }
 
-/// Zero `len` bytes at user address `user`; answers the bytes not zeroed.
+/// Zero `len` bytes at user address `user`, as [`copy_from_user`].
 ///
 /// # Safety
 /// As [`copy_from_user`].
-pub unsafe fn clear_user(user: u64, len: usize) -> usize {
+pub unsafe fn clear_user(user: u64, len: usize) -> Result<(), Faulted> {
     // SAFETY: the caller's contract; the routine touches nothing else.
-    unsafe { toyos_clear_user(user, len) }
+    whole(unsafe { toyos_clear_user(user, len) })
 }
 
 /// Where an EL1 abort at `pc` resumes when it cannot be served: `Some` only

kernel/src/arch/x86_64/boot.rs

@@ -67,6 +67,7 @@ pub fn after_console(args: &KernelArgs, _maps: &[MemoryMapEntry]) {
     microcode::load(0);
 
     let rule = match cpu::vendor() {
+        _ if crate::actuator::copies_go_down() => copy::Rule { window: 4096 },
         kernel::cpu::Vendor::Amd => copy::Rule::AMD,
         kernel::cpu::Vendor::Intel | kernel::cpu::Vendor::Other(_) => copy::Rule::NEVER,
     };

kernel/src/arch/x86_64/copy.rs

@@ -1,4 +1,5 @@
-//! The kernel's `memcpy`, in place of `compiler_builtins`' weak one: one
+//! The kernel's `memcpy`, in place of `compiler_builtins`' weak one, and the
+//! rule and the descending copy its user copies share (`super::uaccess`): one
 //! `rep movsb`, or a descending copy where a forward one would stall.
 //!
 //! A load matching an older in-flight store on address bits 11:0 alone waits
@@ -54,7 +55,7 @@ pub fn set_rule(rule: Rule) {
 }
 
 #[cfg(target_os = "none")]
-fn rule() -> Rule {
+pub(super) fn rule() -> Rule {
     Rule { window: WINDOW.load(Ordering::Relaxed) }
 }
 
@@ -101,55 +102,71 @@ pub unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) {
     }
 }
 
-/// Copy `n` bytes, at least 8, between disjoint ranges, highest first: the
-/// top 8 bytes in one store, then down 32 and 8 bytes a step from the
+/// The descending copy's instructions, for `rdi` the destination, `rsi` the
+/// source and `rdx` the length, at least 8: it keeps those three, uses no
+/// stack and leaves `rax`, `rcx`, `r8`, `r9` and `r10` changed. The top 8
+/// bytes go in one store, then 32 and 8 bytes a step down from the
 /// destination's last 8-byte boundary, so no other store splits a line, and
 /// the bottom 8 bytes last. Stores past the boundary overlap one another,
 /// with the same bytes.
+macro_rules! copy_down_asm {
+    () => {
+        concat!(
+            "lea rcx, [rdi + rdx]\n",
+            "and rcx, -8\n",
+            "sub rcx, rdi\n",
+            "mov rax, [rsi + rdx - 8]\n",
+            "mov [rdi + rdx - 8], rax\n",
+            "cmp rcx, 32\n",
+            "jb 3f\n",
+            "2:\n",
+            "sub rcx, 32\n",
+            "mov rax, [rsi + rcx + 24]\n",
+            "mov r8, [rsi + rcx + 16]\n",
+            "mov r9, [rsi + rcx + 8]\n",
+            "mov r10, [rsi + rcx]\n",
+            "mov [rdi + rcx + 24], rax\n",
+            "mov [rdi + rcx + 16], r8\n",
+            "mov [rdi + rcx + 8], r9\n",
+            "mov [rdi + rcx], r10\n",
+            "cmp rcx, 32\n",
+            "jae 2b\n",
+            "3:\n",
+            "cmp rcx, 8\n",
+            "jb 5f\n",
+            "4:\n",
+            "sub rcx, 8\n",
+            "mov rax, [rsi + rcx]\n",
+            "mov [rdi + rcx], rax\n",
+            "cmp rcx, 8\n",
+            "jae 4b\n",
+            "5:\n",
+            "mov rax, [rsi]\n",
+            "mov [rdi], rax\n",
+        )
+    };
+}
+#[cfg(target_os = "none")]
+pub(super) use copy_down_asm;
+
+/// Copy `n` bytes, at least 8, between disjoint ranges, highest first
+/// ([`copy_down_asm`]).
 ///
 /// # Safety
 /// As [`copy_by`], with `n` at least 8.
 pub unsafe fn copy_down(dest: *mut u8, src: *const u8, n: usize) {
-    let aligned_end = ((dest as usize).wrapping_add(n) & !7).wrapping_sub(dest as usize);
     // SAFETY: every access is inside `[0, n)` of its range, as the caller promises.
     unsafe {
         core::arch::asm!(
-            "mov {a}, [{s} + {n} - 8]",
-            "mov [{d} + {n} - 8], {a}",
-            "cmp {p}, 32",
-            "jb 3f",
-            "2:",
-            "sub {p}, 32",
-            "mov {a}, [{s} + {p} + 24]",
-            "mov {b}, [{s} + {p} + 16]",
-            "mov {c}, [{s} + {p} + 8]",
-            "mov {e}, [{s} + {p}]",
-            "mov [{d} + {p} + 24], {a}",
-            "mov [{d} + {p} + 16], {b}",
-            "mov [{d} + {p} + 8], {c}",
-            "mov [{d} + {p}], {e}",
-            "cmp {p}, 32",
-            "jae 2b",
-            "3:",
-            "cmp {p}, 8",
-            "jb 5f",
-            "4:",
-            "sub {p}, 8",
-            "mov {a}, [{s} + {p}]",
-            "mov [{d} + {p}], {a}",
-            "cmp {p}, 8",
-            "jae 4b",
-            "5:",
-            "mov {a}, [{s}]",
-            "mov [{d}], {a}",
-            s = in(reg) src,
-            d = in(reg) dest,
-            n = in(reg) n,
-            p = inout(reg) aligned_end => _,
-            a = out(reg) _,
-            b = out(reg) _,
-            c = out(reg) _,
-            e = out(reg) _,
+            copy_down_asm!(),
+            in("rdi") dest,
+            in("rsi") src,
+            in("rdx") n,
+            out("rax") _,
+            out("rcx") _,
+            out("r8") _,
+            out("r9") _,
+            out("r10") _,
             options(nostack),
         );
     }

kernel/src/arch/x86_64/idt/exceptions.rs

@@ -515,7 +515,7 @@ pub(super) fn page_fault_handler(frame: &mut TrapFrame) {
 
 /// A user copy's fault on `addr` (`arch::uaccess`): served as the process's
 /// own fault would be, after which the copy resumes where it stopped, or else
-/// resumed at `fixup`, which answers the bytes not copied. The copy was made
+/// resumed at `fixup`, which answers that the copy failed. The copy was made
 /// holding no lock at its syscall's depth (`user_ptr`), so the service runs at
 /// that depth too.
 fn user_copy_fault(frame: &mut TrapFrame, addr: u64, fixup: u64) {