From 43ac9d1de1a0002de1dfd1daccef2ce99279fe25 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 11:08:31 +0100 Subject: [PATCH 01/13] libc: a forward copy a short way above its source modulo 4 KiB copies down memcpy was one rep movsb at every distance. With the destination 16 bytes above the source modulo 4096, an 8 KiB copy took 2595 ns on the AMD bench against 90 ns at distance 0, and the T14 lost up to twelvefold from 16 to 256 bytes (osbench copyat, measurement only). Two heap Vecs dlmalloc places one after the other sit at exactly 16: copyat's heap row read 2525 ns on the AMD bench, the rate #1121's IPC copy split saw. AMD's Family 17h judges store-to-load forwarding eligibility by address bits 11:0 alone (Software Optimization Guide for AMD EPYC 7002, 56305 rev. 3.02, section 2.12), so each load of a forward copy at such a distance matches an older store it does not depend on. Copied highest first, the store a load could match is 4 KiB minus the distance behind it. copy_forward now takes an SSE2 loop from the top for 64 bytes or more at a distance under 512 between disjoint ranges, and rep movsb otherwise. The overlap guard keeps memmove's forward branch lowest first. Host tests: a forward copy at every distance below 1 KiB and four line offsets, and an overlapping one a page apart; the latter reds without the guard. Co-Authored-By: Claude Opus 5.5 --- ...ay-above-its-source-modulo-4-kib-stalls.md | 29 +++++++++++ ...ns-at-as-little-as-six-tenths-of-linuxs.md | 3 +- sdk/libc/arch/src/lib.rs | 45 ++++++++++++++++- sdk/libc/src/arch/x86_64.rs | 49 ++++++++++++++++++- 4 files changed, 123 insertions(+), 3 deletions(-) create mode 100644 issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md diff --git a/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md b/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md new file mode 100644 index 00000000000..45c79df1809 --- /dev/null +++ b/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md @@ -0,0 +1,29 @@ +--- +status: open +kind: defect +opened: 2026-11-01 +--- + +# A kernel copy a short way above its source modulo 4 KiB stalls + +The kernel's `memcpy` is `compiler_builtins`' one `rep movsb` +(`issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md`). +`osbench copyat` (measurement only, branch `amd-memcpy-ab`) timed `rep movsb` +over 8 KiB with the destination `d` bytes above the source modulo 4096, p50: + +| bench | d = 0 | d = 16 | d = 64 | d = 256 | d >= 1024 | +|---|---|---|---|---|---| +| AMD (Ryzen 7 4800U) | 80 ns | 2515 ns | 80 ns | 80 ns | 80-100 ns | +| T14 (i5-1135G7) | 58 ns | 751 ns | 696 ns | 191 ns | 53-92 ns | + +AMD's Family 17h judges store-to-load forwarding by address bits 11:0 alone +(56305 rev. 3.02, §2.12), so each load matches an unrelated older store. The +kernel copies between its own buffers and the direct map's view of user +frames at whatever distance the two happen to have: a user copy, a pipe's +gather, a message's ring. Userland's `copy_forward` copies such a pair highest +first (`sdk/libc/src/arch/x86_64.rs`); the kernel's does not. + +**Exit**: a kernel copy into a user buffer 16 bytes above its source modulo +4 KiB, measured on both benches, within twice the same copy at distance 0. + +**Owner**: the kernel's copy (the sysroot's `compiler_builtins` `mem`). diff --git a/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md b/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md index d4c2897dd73..b042cd446e1 100644 --- a/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md +++ b/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md @@ -8,7 +8,8 @@ opened: 2026-10-28 `osbench mem`'s `mem.memcpy.` copies a buffer of that size with each system's own `memcpy`. On ToyOS that is `sdk/libc`'s, whose `copy_forward` is -one `rep movsb` at every length (`sdk/libc/src/arch/x86_64.rs`). On Linux it +one `rep movsb` at every length between buffers page-aligned alike, as these +are (`sdk/libc/src/arch/x86_64.rs`). On Linux it is the static musl build's. From the 2026-10-28 scoreboard (evidence branch `scoreboard-2026-10-28`): diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 9450715b9ee..c35ae26b3c1 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -1,7 +1,8 @@ //! libc's modules that read and set nothing but what they are handed, on the //! host, differentially. The architecture module: every copy and fill it has, //! over every length to 300 and every source and destination offset to 20, -//! against `copy_within` and `fill`, with the two buffers overlapping both ways; +//! against `copy_within` and `fill`, with the two buffers overlapping both ways, +//! and a forward copy at every page distance below 1 KiB; //! and its square roots against `f64::sqrt` and `f32::sqrt`. Each host //! architecture checks its own module. The UTF-8 reader against //! `core::str::from_utf8`, the number reader against the host C library's and @@ -219,6 +220,48 @@ mod tests { } } + /// A forward copy at every distance of its destination above its source + /// modulo 4 KiB below 1 KiB, and some above, from sources at four line + /// offsets: the distances some architecture copies another way. + #[test] + fn a_forward_copy_agrees_at_every_page_distance() { + let distances = (0..1024).chain([2048, 4032, 4095]); + let src = pattern(4096 + 64); + for d in distances { + for at in [0, 1, 17, 63] { + for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4096] { + let mut got = vec![0xAAu8; 2 * 4096 + d + 64]; + let to = 4096 + d + at; + // SAFETY: `n` bytes at `at` of `src` and at `to` of `got`, two buffers. + unsafe { arch::copy_forward(got.as_mut_ptr().add(to), src.as_ptr().add(at), n) }; + let mut want = vec![0xAAu8; 2 * 4096 + d + 64]; + want[to..to + n].copy_from_slice(&src[at..at + n]); + assert_eq!(got, want, "n {n} at {at} distance {d}"); + } + } + } + } + + /// A forward copy whose destination sits below an overlapping source by + /// just under 4 KiB, so modulo 4 KiB it is a short way above it: it must + /// still copy lowest first. + #[test] + fn an_overlapping_forward_copy_a_page_apart_agrees() { + for gap in [3584, 3600, 4000, 4080, 4095] { + for over in [1, 64, 500] { + let n = gap + over; + let base = pattern(n + gap); + let mut want = base.clone(); + want.copy_within(gap..gap + n, 0); + let mut got = base.clone(); + let p = got.as_mut_ptr(); + // SAFETY: both ranges are inside `got`, the destination below the source. + unsafe { arch::copy_forward(p, p.add(gap), n) }; + assert_eq!(got, want, "gap {gap} n {n}"); + } + } + } + #[test] fn every_fill_agrees_with_fill() { for n in 0..LENGTHS { diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index e676c8dd482..826a68d7167 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -20,8 +20,23 @@ unsafe extern "C" fn _start() -> ! { ); } -/// Copy `n` bytes from `src` to `dest`, lowest address first. +/// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for +/// any `dest` not above an overlapping `src`. +/// +/// A destination less than [`SHORT_ALIAS`] above its source modulo 4 KiB makes +/// each load of a forward copy match an older, unrelated store on address bits +/// 11:0, which the load/store unit takes as a possible dependence: AMD's Family +/// 17h judges store-to-load forwarding by those bits alone (Software +/// Optimization Guide for AMD EPYC 7002, 56305 rev. 3.02, §2.12), and the +/// T14's Tiger Lake measures the same stall. Copied highest first, every store such a load +/// could match is 4 KiB minus the distance behind it and long retired. pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { + let distance = (dest as usize).wrapping_sub(src as usize) % 4096; + let disjoint = (dest as usize).abs_diff(src as usize) >= n; + if n >= 64 && distance != 0 && distance < SHORT_ALIAS && disjoint { + unsafe { copy_down_64(dest, src, n) }; + return; + } unsafe { core::arch::asm!( "rep movsb", @@ -33,6 +48,38 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { } } +/// The distances below which [`copy_forward`] copies highest first: as far as +/// either bench was measured slower forward than down (the T14 to 256 bytes). +const SHORT_ALIAS: usize = 512; + +/// Copy `n` bytes, at least 64, between disjoint ranges, 64 bytes a step by +/// SSE2 from the top; the lowest 64 are loaded first and stored last, so the +/// last step may overlap them. +unsafe fn copy_down_64(dest: *mut u8, src: *const u8, n: usize) { + unsafe { + core::arch::asm!( + "movups xmm4, [{s}]", "movups xmm5, [{s} + 16]", "movups xmm6, [{s} + 32]", "movups xmm7, [{s} + 48]", + "2:", + "cmp {p}, 64", + "jbe 3f", + "sub {p}, 64", + "movups xmm0, [{s} + {p}]", "movups xmm1, [{s} + {p} + 16]", + "movups xmm2, [{s} + {p} + 32]", "movups xmm3, [{s} + {p} + 48]", + "movups [{d} + {p}], xmm0", "movups [{d} + {p} + 16], xmm1", + "movups [{d} + {p} + 32], xmm2", "movups [{d} + {p} + 48], xmm3", + "jmp 2b", + "3:", + "movups [{d}], xmm4", "movups [{d} + 16], xmm5", "movups [{d} + 32], xmm6", "movups [{d} + 48], xmm7", + s = in(reg) src, + d = in(reg) dest, + p = inout(reg) n => _, + out("xmm0") _, out("xmm1") _, out("xmm2") _, out("xmm3") _, + out("xmm4") _, out("xmm5") _, out("xmm6") _, out("xmm7") _, + options(nostack), + ); + } +} + /// Copy `n` bytes from `src` to `dest`, highest address first: the order a /// `dest` overlapping `src` from above needs. pub(crate) unsafe fn copy_backward(dest: *mut u8, src: *const u8, n: usize) { From 58e360a769567fdaf706b519331fd8ee1e7f7166 Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 12:25:54 +0100 Subject: [PATCH 02/13] review r1: the descending copy is AVX-wide where the CPU has it, and its test exact The base-against-fix A/B (four boots per bench) showed the SSE2 loop at 136-151 ns per 8 KiB on the T14 and 140 on the AMD against rep movsb's 62 and 90 at distance 0: twice the T14's good copy, and on the AMD slower than the rep movsb it replaced at distances 32 to 384, where that bench does not stall. Both benches retire about one store a cycle there, so the loop is now 32 bytes a store where CPUID and XCR0 say AVX runs, asked once, with SSE2 left for a CPU without it. The A/B also read mem.memcpy.64 16% slower on the T14 (18690 to 15767 MB/s): every copy of 64 bytes or more paid the route's distance and overlap arithmetic. A copy now pays a length and one distance compare; the overlap check and the AVX choice are out of line. The page-distance test took its two ends from separate allocations, so the distance it named was off by the allocator's gap (the head-store mutation first failed at "distance 109"). Both ends now come from one buffer from a 64-byte boundary, and the mutation's first failure is "n 64 at 0 distance 1". A second test calls both loops directly; the AVX one only where the host has AVX, which the arm64 host's Rosetta does not, so the guest's copyat rows check it under QEMU (whose CPU declares +avx). Comments: Intel's documented 4K aliasing in place of a bench reading, SHORT_ALIAS's comment cut to what it is. The kernel issue cites the evidence repository instead of a branch that will be deleted. Co-Authored-By: Claude Opus 5.5 --- ...ay-above-its-source-modulo-4-kib-stalls.md | 3 +- sdk/libc/arch/src/lib.rs | 53 ++++++--- sdk/libc/src/arch/x86_64.rs | 102 +++++++++++++++--- 3 files changed, 132 insertions(+), 26 deletions(-) diff --git a/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md b/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md index 45c79df1809..c8d1148f40c 100644 --- a/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md +++ b/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md @@ -8,7 +8,8 @@ opened: 2026-11-01 The kernel's `memcpy` is `compiler_builtins`' one `rep movsb` (`issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md`). -`osbench copyat` (measurement only, branch `amd-memcpy-ab`) timed `rep movsb` +`osbench copyat` (measurement only; evidence +`ToyOSOrg/evidence` branch `pr-1133`, `amd-memcpy/round-2-copyat/`) timed `rep movsb` over 8 KiB with the destination `d` bytes above the source modulo 4096, p50: | bench | d = 0 | d = 16 | d = 64 | d = 256 | d >= 1024 | diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index c35ae26b3c1..962ca578d74 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -220,23 +220,50 @@ mod tests { } } - /// A forward copy at every distance of its destination above its source - /// modulo 4 KiB below 1 KiB, and some above, from sources at four line - /// offsets: the distances some architecture copies another way. + /// A copy with its destination exactly `distance` above its source modulo + /// 4 KiB, the source `at` bytes past a 64-byte boundary, both ends in one + /// buffer: the whole buffer against `copy_within`. + fn agrees_at(what: &str, copy: unsafe fn(*mut u8, *const u8, usize), distance: usize, at: usize, n: usize) { + let mut got = pattern(4 * 4096 + 256); + let base = got.as_ptr().align_offset(64); + let (from, to) = (base + at, base + at + 8192 + distance); + let mut want = got.clone(); + want.copy_within(from..from + n, to); + let p = got.as_mut_ptr(); + // SAFETY: both ranges are inside `got`, 8 KiB or more apart, `n` at most 4096. + unsafe { copy(p.add(to), p.add(from), n) }; + assert_eq!(got, want, "{what}: n {n} at {at} distance {distance}"); + } + + /// A forward copy at every distance below 1 KiB, and some above, from + /// sources at four line offsets: the distances some architecture copies + /// another way. #[test] fn a_forward_copy_agrees_at_every_page_distance() { - let distances = (0..1024).chain([2048, 4032, 4095]); - let src = pattern(4096 + 64); - for d in distances { + for distance in (0..1024).chain([2048, 4032, 4095]) { for at in [0, 1, 17, 63] { for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4096] { - let mut got = vec![0xAAu8; 2 * 4096 + d + 64]; - let to = 4096 + d + at; - // SAFETY: `n` bytes at `at` of `src` and at `to` of `got`, two buffers. - unsafe { arch::copy_forward(got.as_mut_ptr().add(to), src.as_ptr().add(at), n) }; - let mut want = vec![0xAAu8; 2 * 4096 + d + 64]; - want[to..to + n].copy_from_slice(&src[at..at + n]); - assert_eq!(got, want, "n {n} at {at} distance {d}"); + agrees_at("copy_forward", arch::copy_forward, distance, at, n); + } + } + } + } + + /// x86-64's two descending loops, each called directly so the one this + /// host's `copy_forward` does not take is checked too; AVX's only where + /// the host has it. + #[cfg(target_arch = "x86_64")] + #[test] + fn both_descending_loops_agree() { + let mut loops: Vec<(&str, unsafe fn(*mut u8, *const u8, usize))> = vec![("sse2", arch::copy_down_sse2)]; + if arch::has_avx() { + // SAFETY: `has_avx` answered for this CPU. + loops.push(("avx", |d, s, n| unsafe { arch::copy_down_avx(d, s, n) })); + } + for (name, copy) in loops { + for n in (64..300).chain([511, 512, 513, 4095, 4096]) { + for (distance, at) in [(1, 0), (16, 16), (63, 1), (511, 63)] { + agrees_at(name, copy, distance, at, n); } } } diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 826a68d7167..b824886de50 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -25,18 +25,40 @@ unsafe extern "C" fn _start() -> ! { /// /// A destination less than [`SHORT_ALIAS`] above its source modulo 4 KiB makes /// each load of a forward copy match an older, unrelated store on address bits -/// 11:0, which the load/store unit takes as a possible dependence: AMD's Family -/// 17h judges store-to-load forwarding by those bits alone (Software -/// Optimization Guide for AMD EPYC 7002, 56305 rev. 3.02, §2.12), and the -/// T14's Tiger Lake measures the same stall. Copied highest first, every store such a load -/// could match is 4 KiB minus the distance behind it and long retired. +/// 11:0, which the load/store unit takes as a possible dependence: AMD's +/// Family 17h judges store-to-load forwarding by those bits alone (Software +/// Optimization Guide for AMD EPYC 7002, 56305 rev. 3.02, §2.12), as Intel's +/// 4K aliasing does (Intel 64 and IA-32 Architectures Optimization Reference +/// Manual). Copied highest first, every store such a load could match is +/// 4 KiB minus the distance behind it and long retired. pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { let distance = (dest as usize).wrapping_sub(src as usize) % 4096; - let disjoint = (dest as usize).abs_diff(src as usize) >= n; - if n >= 64 && distance != 0 && distance < SHORT_ALIAS && disjoint { - unsafe { copy_down_64(dest, src, n) }; - return; + // `distance` in 1..SHORT_ALIAS, as one compare. + if n >= 64 && distance.wrapping_sub(1) < SHORT_ALIAS - 1 { + // SAFETY: as this function's caller promises. + return unsafe { copy_short_alias(dest, src, n) }; } + unsafe { movsb(dest, src, n) }; +} + +/// [`copy_forward`] at a short distance: down where the ranges are disjoint, +/// out of line so the common copy carries none of it. +#[inline(never)] +unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) { + // SAFETY: `n` is at least 64; down only between disjoint ranges. + unsafe { + if (dest as usize).abs_diff(src as usize) < n { + movsb(dest, src, n); + } else if has_avx() { + copy_down_avx(dest, src, n); + } else { + copy_down_sse2(dest, src, n); + } + } +} + +#[inline(always)] +unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { unsafe { core::arch::asm!( "rep movsb", @@ -48,14 +70,36 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { } } -/// The distances below which [`copy_forward`] copies highest first: as far as -/// either bench was measured slower forward than down (the T14 to 256 bytes). +/// The distance under which [`copy_forward`] copies highest first. const SHORT_ALIAS: usize = 512; +/// Whether this CPU runs AVX and its system saves the YMM state: CPUID.1:ECX +/// AVX (bit 28) and OSXSAVE (bit 27), and XCR0's SSE and AVX bits (1 and 2). +/// Asked once; a racing first call asks twice and stores the same answer. +pub(crate) fn has_avx() -> bool { + use core::sync::atomic::{AtomicU8, Ordering}; + static AVX: AtomicU8 = AtomicU8::new(0); + match AVX.load(Ordering::Relaxed) { + 1 => false, + 2 => true, + _ => { + let ecx = core::arch::x86_64::__cpuid(1).ecx; + let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { + let (lo, _hi): (u32, u32); + // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. + unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; + lo & 0b110 == 0b110 + }; + AVX.store(if avx { 2 } else { 1 }, Ordering::Relaxed); + avx + } + } +} + /// Copy `n` bytes, at least 64, between disjoint ranges, 64 bytes a step by /// SSE2 from the top; the lowest 64 are loaded first and stored last, so the /// last step may overlap them. -unsafe fn copy_down_64(dest: *mut u8, src: *const u8, n: usize) { +pub(crate) unsafe fn copy_down_sse2(dest: *mut u8, src: *const u8, n: usize) { unsafe { core::arch::asm!( "movups xmm4, [{s}]", "movups xmm5, [{s} + 16]", "movups xmm6, [{s} + 32]", "movups xmm7, [{s} + 48]", @@ -80,6 +124,40 @@ unsafe fn copy_down_64(dest: *mut u8, src: *const u8, n: usize) { } } +/// [`copy_down_sse2`] 32 bytes a load and store, half the stores: a CPU +/// retiring one store a cycle copies at twice the rate. `vzeroupper` ends it, +/// so SSE code after it pays no transition (56305 rev. 3.02, §2.11.6). +/// +/// # Safety +/// As [`copy_down_sse2`], on a CPU [`has_avx`] answers for. +#[target_feature(enable = "avx")] +pub(crate) unsafe fn copy_down_avx(dest: *mut u8, src: *const u8, n: usize) { + unsafe { + core::arch::asm!( + "vmovdqu ymm2, [{s}]", "vmovdqu ymm3, [{s} + 32]", + "2:", + "cmp {p}, 64", + "jbe 3f", + "sub {p}, 64", + "vmovdqu ymm0, [{s} + {p}]", "vmovdqu ymm1, [{s} + {p} + 32]", + "vmovdqu [{d} + {p}], ymm0", "vmovdqu [{d} + {p} + 32], ymm1", + "jmp 2b", + "3:", + "vmovdqu [{d}], ymm2", "vmovdqu [{d} + 32], ymm3", + "vzeroupper", + s = in(reg) src, + d = in(reg) dest, + p = inout(reg) n => _, + // `vzeroupper` clears every YMM register's upper half. + out("ymm0") _, out("ymm1") _, out("ymm2") _, out("ymm3") _, + out("ymm4") _, out("ymm5") _, out("ymm6") _, out("ymm7") _, + out("ymm8") _, out("ymm9") _, out("ymm10") _, out("ymm11") _, + out("ymm12") _, out("ymm13") _, out("ymm14") _, out("ymm15") _, + options(nostack), + ); + } +} + /// Copy `n` bytes from `src` to `dest`, highest address first: the order a /// `dest` overlapping `src` from above needs. pub(crate) unsafe fn copy_backward(dest: *mut u8, src: *const u8, n: usize) { From 835ce9dec56b608d7058f66b3e58ac706e3c90eb Mon Sep 17 00:00:00 2001 From: japabu Date: Sun, 1 Nov 2026 15:09:18 +0100 Subject: [PATCH 03/13] review r1, round 4: the descending copy stores aligned, and leaves long copies to ERMS The round-4 pair (evidence round-4-ab): - The AVX loop read 80 ns per 8 KiB on the AMD where its destination stores fell on 32-byte boundaries and 140 where they straddled them; the T14 read 71-78 against 109-124. A store split across two lines cost as much as the stall. Each loop now stores its top 64 bytes unaligned once and every other store aligned to its width. - Past the caches the T14's copies at a short distance were not stalled by rep movsb (1 MiB at d16: 37.7 us against 40.9 at d0), and the loop was slower there (64 MiB at d256: 10.1 ms against 7.2), while the AMD, which has no ERMS, stalled at 64 MiB (21.8 ms against 10.9) and the loop beat it at every size. A copy over 1 MiB keeps rep movsb where the CPU has ERMS. Host tests: the loops at every destination offset in a 32-byte store. A mutation misplacing a tail store reds three tests; the guard and head mutations still red theirs. Co-Authored-By: Claude Opus 5.5 --- sdk/libc/arch/src/lib.rs | 11 +++-- sdk/libc/src/arch/x86_64.rs | 90 ++++++++++++++++++++++++++----------- 2 files changed, 72 insertions(+), 29 deletions(-) diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 962ca578d74..5ff1eff4178 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -250,8 +250,8 @@ mod tests { } /// x86-64's two descending loops, each called directly so the one this - /// host's `copy_forward` does not take is checked too; AVX's only where - /// the host has it. + /// host's `copy_forward` does not take is checked too, at every + /// destination offset in a store; AVX's only where the host has it. #[cfg(target_arch = "x86_64")] #[test] fn both_descending_loops_agree() { @@ -262,8 +262,11 @@ mod tests { } for (name, copy) in loops { for n in (64..300).chain([511, 512, 513, 4095, 4096]) { - for (distance, at) in [(1, 0), (16, 16), (63, 1), (511, 63)] { - agrees_at(name, copy, distance, at, n); + // Every destination offset in a 32-byte store, from two source offsets. + for distance in 1..=32 { + for at in [0, 17] { + agrees_at(name, copy, distance, at, n); + } } } } diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index b824886de50..97e27a850c5 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -47,7 +47,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) { // SAFETY: `n` is at least 64; down only between disjoint ranges. unsafe { - if (dest as usize).abs_diff(src as usize) < n { + if (n > ROUTED_MOST && has_erms()) || (dest as usize).abs_diff(src as usize) < n { movsb(dest, src, n); } else if has_avx() { copy_down_avx(dest, src, n); @@ -73,52 +73,89 @@ unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { /// The distance under which [`copy_forward`] copies highest first. const SHORT_ALIAS: usize = 512; +/// The longest copy [`copy_forward`] copies highest first where `rep movsb` +/// is a fast string at every length (ERMS): a longer one is bound by memory +/// bandwidth, where its stores, which need not read a line before writing it +/// whole (Intel 64 and IA-32 Architectures Optimization Reference Manual, +/// "Enhanced REP MOVSB and STOSB"), outrun the loop's. +const ROUTED_MOST: usize = 1 << 20; + /// Whether this CPU runs AVX and its system saves the YMM state: CPUID.1:ECX /// AVX (bit 28) and OSXSAVE (bit 27), and XCR0's SSE and AVX bits (1 and 2). -/// Asked once; a racing first call asks twice and stores the same answer. pub(crate) fn has_avx() -> bool { + features() & AVX != 0 +} + +/// Whether `rep movsb` is a fast string at any length: ERMS, CPUID.(7,0):EBX +/// bit 9. +fn has_erms() -> bool { + features() & ERMS != 0 +} + +const AVX: u8 = 1 << 0; +const ERMS: u8 = 1 << 1; +const ASKED: u8 = 1 << 7; + +/// The CPU's answers [`has_avx`] and [`has_erms`] read, asked once; a racing +/// first call asks twice and stores the same answer. +fn features() -> u8 { use core::sync::atomic::{AtomicU8, Ordering}; - static AVX: AtomicU8 = AtomicU8::new(0); - match AVX.load(Ordering::Relaxed) { - 1 => false, - 2 => true, - _ => { - let ecx = core::arch::x86_64::__cpuid(1).ecx; - let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { - let (lo, _hi): (u32, u32); - // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. - unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; - lo & 0b110 == 0b110 - }; - AVX.store(if avx { 2 } else { 1 }, Ordering::Relaxed); - avx - } + static FEATURES: AtomicU8 = AtomicU8::new(0); + let known = FEATURES.load(Ordering::Relaxed); + if known & ASKED != 0 { + return known; } + let ecx = core::arch::x86_64::__cpuid(1).ecx; + let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { + let (lo, _hi): (u32, u32); + // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. + unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; + lo & 0b110 == 0b110 + }; + let erms = core::arch::x86_64::__cpuid_count(7, 0).ebx & (1 << 9) != 0; + let answer = ASKED | if avx { AVX } else { 0 } | if erms { ERMS } else { 0 }; + FEATURES.store(answer, Ordering::Relaxed); + answer +} + +/// Where a descending loop of `width`-byte stores starts: the offset of the +/// last `width` boundary of the destination at or below its end. The bytes +/// above it go in one unaligned 64-byte store; every store below it is +/// aligned, since a store split across two lines costs both benches' cores +/// as much as the stall this loop avoids. +fn aligned_end(dest: *mut u8, n: usize, width: usize) -> usize { + ((dest as usize + n) & !(width - 1)) - dest as usize } -/// Copy `n` bytes, at least 64, between disjoint ranges, 64 bytes a step by -/// SSE2 from the top; the lowest 64 are loaded first and stored last, so the -/// last step may overlap them. +/// Copy `n` bytes, at least 64, between disjoint ranges by SSE2 from the +/// top, 64 bytes a step from [`aligned_end`]: the first and last 64 bytes are +/// loaded first and stored last, so the steps may overlap them. pub(crate) unsafe fn copy_down_sse2(dest: *mut u8, src: *const u8, n: usize) { unsafe { core::arch::asm!( "movups xmm4, [{s}]", "movups xmm5, [{s} + 16]", "movups xmm6, [{s} + 32]", "movups xmm7, [{s} + 48]", + "movups xmm8, [{s} + {n} - 64]", "movups xmm9, [{s} + {n} - 48]", + "movups xmm10, [{s} + {n} - 32]", "movups xmm11, [{s} + {n} - 16]", "2:", "cmp {p}, 64", "jbe 3f", "sub {p}, 64", "movups xmm0, [{s} + {p}]", "movups xmm1, [{s} + {p} + 16]", "movups xmm2, [{s} + {p} + 32]", "movups xmm3, [{s} + {p} + 48]", - "movups [{d} + {p}], xmm0", "movups [{d} + {p} + 16], xmm1", - "movups [{d} + {p} + 32], xmm2", "movups [{d} + {p} + 48], xmm3", + "movaps [{d} + {p}], xmm0", "movaps [{d} + {p} + 16], xmm1", + "movaps [{d} + {p} + 32], xmm2", "movaps [{d} + {p} + 48], xmm3", "jmp 2b", "3:", + "movups [{d} + {n} - 64], xmm8", "movups [{d} + {n} - 48], xmm9", + "movups [{d} + {n} - 32], xmm10", "movups [{d} + {n} - 16], xmm11", "movups [{d}], xmm4", "movups [{d} + 16], xmm5", "movups [{d} + 32], xmm6", "movups [{d} + 48], xmm7", s = in(reg) src, d = in(reg) dest, - p = inout(reg) n => _, + n = in(reg) n, + p = inout(reg) aligned_end(dest, n, 16) => _, out("xmm0") _, out("xmm1") _, out("xmm2") _, out("xmm3") _, out("xmm4") _, out("xmm5") _, out("xmm6") _, out("xmm7") _, + out("xmm8") _, out("xmm9") _, out("xmm10") _, out("xmm11") _, options(nostack), ); } @@ -135,19 +172,22 @@ pub(crate) unsafe fn copy_down_avx(dest: *mut u8, src: *const u8, n: usize) { unsafe { core::arch::asm!( "vmovdqu ymm2, [{s}]", "vmovdqu ymm3, [{s} + 32]", + "vmovdqu ymm4, [{s} + {n} - 64]", "vmovdqu ymm5, [{s} + {n} - 32]", "2:", "cmp {p}, 64", "jbe 3f", "sub {p}, 64", "vmovdqu ymm0, [{s} + {p}]", "vmovdqu ymm1, [{s} + {p} + 32]", - "vmovdqu [{d} + {p}], ymm0", "vmovdqu [{d} + {p} + 32], ymm1", + "vmovdqa [{d} + {p}], ymm0", "vmovdqa [{d} + {p} + 32], ymm1", "jmp 2b", "3:", + "vmovdqu [{d} + {n} - 64], ymm4", "vmovdqu [{d} + {n} - 32], ymm5", "vmovdqu [{d}], ymm2", "vmovdqu [{d} + 32], ymm3", "vzeroupper", s = in(reg) src, d = in(reg) dest, - p = inout(reg) n => _, + n = in(reg) n, + p = inout(reg) aligned_end(dest, n, 32) => _, // `vzeroupper` clears every YMM register's upper half. out("ymm0") _, out("ymm1") _, out("ymm2") _, out("ymm3") _, out("ymm4") _, out("ymm5") _, out("ymm6") _, out("ymm7") _, From 492f44c1f5416063b3cae255037c039b02e90a4d Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 2 Nov 2026 01:37:08 +0100 Subject: [PATCH 04/13] round 5: the 1 MiB cap holds on every CPU, and the kernel issue goes to #1139 Round 5 (four boots per bench, evidence round-5-ab): - The AMD's routed copies read 80-86 ns per 8 KiB against 2615 before. - At 4 MiB with d256, a distance its rep movsb does not stall at, the AMD loop read 597 us against rep movsb's 385. With the cap on every CPU, a copy over 1 MiB is base's rep movsb everywhere, measured in base3, and the ERMS reading goes. - What stays is the AMD's stall above 1 MiB, as before (64 MiB at d16: 2.0x of d0), and the T14's routed copies at 1.1-1.4x of rep movsb's best. The kernel-copy issue's premise was false (the old kernel copy was rep movsb; rep movsq; rep movsb and did not stall at those distances), and #1139 meets its exit; it never reached main, so it goes from this branch. Co-Authored-By: Claude Opus 5.5 --- ...ay-above-its-source-modulo-4-kib-stalls.md | 30 ---------- sdk/libc/src/arch/x86_64.rs | 58 +++++++------------ 2 files changed, 21 insertions(+), 67 deletions(-) delete mode 100644 issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md diff --git a/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md b/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md deleted file mode 100644 index c8d1148f40c..00000000000 --- a/issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -status: open -kind: defect -opened: 2026-11-01 ---- - -# A kernel copy a short way above its source modulo 4 KiB stalls - -The kernel's `memcpy` is `compiler_builtins`' one `rep movsb` -(`issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md`). -`osbench copyat` (measurement only; evidence -`ToyOSOrg/evidence` branch `pr-1133`, `amd-memcpy/round-2-copyat/`) timed `rep movsb` -over 8 KiB with the destination `d` bytes above the source modulo 4096, p50: - -| bench | d = 0 | d = 16 | d = 64 | d = 256 | d >= 1024 | -|---|---|---|---|---|---| -| AMD (Ryzen 7 4800U) | 80 ns | 2515 ns | 80 ns | 80 ns | 80-100 ns | -| T14 (i5-1135G7) | 58 ns | 751 ns | 696 ns | 191 ns | 53-92 ns | - -AMD's Family 17h judges store-to-load forwarding by address bits 11:0 alone -(56305 rev. 3.02, §2.12), so each load matches an unrelated older store. The -kernel copies between its own buffers and the direct map's view of user -frames at whatever distance the two happen to have: a user copy, a pipe's -gather, a message's ring. Userland's `copy_forward` copies such a pair highest -first (`sdk/libc/src/arch/x86_64.rs`); the kernel's does not. - -**Exit**: a kernel copy into a user buffer 16 bytes above its source modulo -4 KiB, measured on both benches, within twice the same copy at distance 0. - -**Owner**: the kernel's copy (the sysroot's `compiler_builtins` `mem`). diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 97e27a850c5..3fd40bd2a79 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -47,7 +47,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) { // SAFETY: `n` is at least 64; down only between disjoint ranges. unsafe { - if (n > ROUTED_MOST && has_erms()) || (dest as usize).abs_diff(src as usize) < n { + if n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { movsb(dest, src, n); } else if has_avx() { copy_down_avx(dest, src, n); @@ -73,49 +73,33 @@ unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { /// The distance under which [`copy_forward`] copies highest first. const SHORT_ALIAS: usize = 512; -/// The longest copy [`copy_forward`] copies highest first where `rep movsb` -/// is a fast string at every length (ERMS): a longer one is bound by memory -/// bandwidth, where its stores, which need not read a line before writing it -/// whole (Intel 64 and IA-32 Architectures Optimization Reference Manual, -/// "Enhanced REP MOVSB and STOSB"), outrun the loop's. +/// The longest copy [`copy_forward`] copies highest first. A longer one runs +/// from beyond the caches, where the loop's stores read each line before +/// writing it and measured slower than `rep movsb` at distances that do not +/// stall. const ROUTED_MOST: usize = 1 << 20; /// Whether this CPU runs AVX and its system saves the YMM state: CPUID.1:ECX /// AVX (bit 28) and OSXSAVE (bit 27), and XCR0's SSE and AVX bits (1 and 2). +/// Asked once; a racing first call asks twice and stores the same answer. pub(crate) fn has_avx() -> bool { - features() & AVX != 0 -} - -/// Whether `rep movsb` is a fast string at any length: ERMS, CPUID.(7,0):EBX -/// bit 9. -fn has_erms() -> bool { - features() & ERMS != 0 -} - -const AVX: u8 = 1 << 0; -const ERMS: u8 = 1 << 1; -const ASKED: u8 = 1 << 7; - -/// The CPU's answers [`has_avx`] and [`has_erms`] read, asked once; a racing -/// first call asks twice and stores the same answer. -fn features() -> u8 { use core::sync::atomic::{AtomicU8, Ordering}; - static FEATURES: AtomicU8 = AtomicU8::new(0); - let known = FEATURES.load(Ordering::Relaxed); - if known & ASKED != 0 { - return known; + static AVX: AtomicU8 = AtomicU8::new(0); + match AVX.load(Ordering::Relaxed) { + 1 => false, + 2 => true, + _ => { + let ecx = core::arch::x86_64::__cpuid(1).ecx; + let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { + let (lo, _hi): (u32, u32); + // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. + unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; + lo & 0b110 == 0b110 + }; + AVX.store(if avx { 2 } else { 1 }, Ordering::Relaxed); + avx + } } - let ecx = core::arch::x86_64::__cpuid(1).ecx; - let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { - let (lo, _hi): (u32, u32); - // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. - unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; - lo & 0b110 == 0b110 - }; - let erms = core::arch::x86_64::__cpuid_count(7, 0).ebx & (1 << 9) != 0; - let answer = ASKED | if avx { AVX } else { 0 } | if erms { ERMS } else { 0 }; - FEATURES.store(answer, Ordering::Relaxed); - answer } /// Where a descending loop of `width`-byte stores starts: the offset of the From 019352cdd9af11475a28625315e386b5bb75b081 Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 2 Nov 2026 03:10:08 +0100 Subject: [PATCH 05/13] issues: a user copy a short way above its source modulo 4 KiB still costs What #1133 leaves, from round 5 (C1, C3 and C4 missed): the T14's routed copies at 1.4-1.7x of distance 0 and its heap pair at 1.56x, the AMD's stall above 1 MiB (4 MiB 3.4x, 64 MiB 2.0x), and one T14 boot's unexplained mem.memcpy.64. Exit: round 5's rules as written. Co-Authored-By: Claude Opus 5.5 --- ...ove-its-source-modulo-4-kib-still-costs.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md new file mode 100644 index 00000000000..d9f62361977 --- /dev/null +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -0,0 +1,51 @@ +--- +status: open +kind: defect +opened: 2026-11-02 +--- + +# A user copy a short way above its source modulo 4 KiB still costs + +`sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest +first when it has at least 64 bytes and at most 1 MiB, its destination is +1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint. +Every other copy is one `rep movsb`. Each load of a forward copy at such a +distance matches an older, unrelated store on address bits 11:0 (AMD +56305 rev. 3.02, §2.12). Before this route the copy was 10 to 29 times +slower. Round 5 of `osbench copyat` measured what is left. That was base +80421e74a against head 0f861df89, four boots per bench, p50s, with the +destination `d` bytes above the source: + +- **T14 (i5-1135G7), up to 1 MiB.** Routed 8 KiB copies take 82–103 ns + against 59 at d = 0, which is 1.4–1.7 times as long; base read 714–773 ns + at d = 16–64. The worst rows (s0.d16 at 101, s16.d32 at 103) are where the + loop's aligned 32-byte stores leave its loads 16 bytes off a 32-byte + boundary, so every other load is split. The two heap `Vec`s dlmalloc places + 16 apart copy in 92 ns, 1.56 times d = 0. +- **AMD (Ryzen 7 4800U), over 1 MiB.** Those copies keep `rep movsb`, which + stalls as before: 4 MiB at d = 16 takes 1.32 ms against 386 µs at d = 0 + (3.4 times), and 64 MiB at d = 16 takes 21.7 ms against 10.9 (2.0 times). + The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and + d = 256, a distance that does not stall, it lost: 597 µs against 385. + +One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The +other three boots read 16649–18226. That copy is page-aligned at d = 0 and +never takes the route. It is unexplained. + +Evidence: `ToyOSOrg/evidence` branch `pr-1133`, commit +`782346f21f39f6b2e87329546febaf433a8f7c2f`, `amd-memcpy/round-5-ab/` +(`judgement.md`, `table.txt` and every boot's rows) and +`amd-memcpy/round-5-ab-rule.txt`. + +**Exit**: round 5's rules as written, measured on both benches. +- C1: libc 8192 rows at d = 16..384, from both source offsets, within 1.3 + times libc.s0.d0, with 10 ns allowed on the AMD. +- C3: `copyat.8192.heap` within 1.5 times libc.s0.d0. +- C4: `copyat..libc.s0.d16` and `.d256`, for every size from 64 KiB to + 64 MiB, within 1.1 times base's same row, and on the AMD also within 1.1 + times base's d = 0 row of that size. + +The `mem.memcpy.64` outlier is reproduced and explained, or shown to be +independent of this copy. + +**Owner**: the orchestrator, for `sdk/libc/src/arch/x86_64.rs`. From 3c646d942bf4cff77575a062057337da60cda862 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 10:59:22 +0100 Subject: [PATCH 06/13] libc: the copy's record states each bench's measured window MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The comment cited 56305 by the wrong title: it is the Software Optimization Guide for AMD Family 17h Models 30h and Greater Processors, and its §2.12 (p. 39) says the processor uses address bits 11:0 to determine STLF eligibility. The comment also implied every distance under the 512 window stalls, which no bench showed on AMD. The issue now separates the two benches. On Zen 2, #1139's in-process rows stall at d = 8..31 and at nothing tried from 32 up, at 4 to 64 KiB. On the T14 the 512 window comes from copyat's rows alone; a heap pair 64 KiB + 16 apart and #1139's in-process copy at d = 16 did not stall there, so the Tiger Lake mechanism is recorded as not established. Co-Authored-By: Claude Opus 5.5 --- ...ove-its-source-modulo-4-kib-still-costs.md | 26 +++++++++++++++---- sdk/libc/src/arch/x86_64.rs | 17 ++++++------ 2 files changed, 30 insertions(+), 13 deletions(-) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index d9f62361977..e4ddcb53a1f 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -9,11 +9,27 @@ opened: 2026-11-02 `sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest first when it has at least 64 bytes and at most 1 MiB, its destination is 1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint. -Every other copy is one `rep movsb`. Each load of a forward copy at such a -distance matches an older, unrelated store on address bits 11:0 (AMD -56305 rev. 3.02, §2.12). Before this route the copy was 10 to 29 times -slower. Round 5 of `osbench copyat` measured what is left. That was base -80421e74a against head 0f861df89, four boots per bench, p50s, with the +Every other copy is one `rep movsb`. + +The two benches stall over different windows, and the route covers both: +- **AMD (Ryzen 7 4800U, Zen 2).** The processor judges store-to-load + forwarding eligibility by address bits 11:0 (Software Optimization Guide + for AMD Family 17h Models 30h and Greater Processors, 56305 rev. 3.02, + §2.12). A forward `rep movsb` from 4 KiB to 64 KiB stalls at d = 8, 16, + 24 and 31 and at no distance tried from 32 to 2048 (#1139's in-process + `kcopyat.movsb` rows, `ToyOSOrg/evidence` commit + `776c503ac402da3dc9c419fb2593a0b8ec79eabe`, `kmemcpy/round-3/`). Up to + 64 MiB it stalls at d = 16 and not at 256 (`copyat`, round 5). At 8 KiB + and d = 16 it takes 2615 ns against 90 at d = 0. +- **T14 (i5-1135G7, Tiger Lake).** `copyat`'s copies, whose ends are 1 MiB + plus the distance apart, read 714–773 ns at d = 16–64 and 206 at 256 + against 74 at d = 0, and 122 at 512. The 512 window is set by these rows. + Intel documents 4K aliasing on bits 11:0, but the same bench read 124 ns + for two heap `Vec`s 64 KiB + 16 apart and 136 for #1139's in-process copy + at d = 16, so what decides the T14's stall is not established. + +Round 5 of `osbench copyat` measured what is left after the route. That was +base 80421e74a against head 0f861df89, four boots per bench, p50s, with the destination `d` bytes above the source: - **T14 (i5-1135G7), up to 1 MiB.** Routed 8 KiB copies take 82–103 ns diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 3fd40bd2a79..9c1fbe85d0e 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -23,14 +23,15 @@ unsafe extern "C" fn _start() -> ! { /// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for /// any `dest` not above an overlapping `src`. /// -/// A destination less than [`SHORT_ALIAS`] above its source modulo 4 KiB makes -/// each load of a forward copy match an older, unrelated store on address bits -/// 11:0, which the load/store unit takes as a possible dependence: AMD's -/// Family 17h judges store-to-load forwarding by those bits alone (Software -/// Optimization Guide for AMD EPYC 7002, 56305 rev. 3.02, §2.12), as Intel's -/// 4K aliasing does (Intel 64 and IA-32 Architectures Optimization Reference -/// Manual). Copied highest first, every store such a load could match is -/// 4 KiB minus the distance behind it and long retired. +/// A forward copy whose destination is a short way above its source modulo +/// 4 KiB loads each byte soon after storing to an unrelated address with the +/// same bits 11:0, and the load waits on that store: AMD's Family 17h judges +/// store-to-load forwarding eligibility by those bits alone (Software +/// Optimization Guide for AMD Family 17h Models 30h and Greater Processors, +/// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and +/// IA-32 Architectures Optimization Reference Manual). Copied highest first, +/// every store such a load could match is 4 KiB minus the distance behind it +/// and long retired. pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { let distance = (dest as usize).wrapping_sub(src as usize) % 4096; // `distance` in 1..SHORT_ALIAS, as one compare. From 19f36d8fe1724f0c212caa7e215ec84aec6cc9ea Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 12:02:01 +0100 Subject: [PATCH 07/13] libc: only a copy of 4 KiB or more asks where its ends sit Round 7's AMD A/B showed every small copy paying 5-7% (mem.memcpy.64 0.93, .256 0.93, .1024 0.95 of base). The disassembly shows why: memcpy no longer inlined copy_forward and called it, with a second frame, and the distance arithmetic and two branches ran on every copy. copy_forward is now inlined into memcpy and memmove as main's rep movsb behind one length compare. Copies of ROUTED_LEAST (4 KiB) or more go out of line to copy_long, which alone computes the distance, the 1 MiB cap and the overlap guard. 4 KiB is the shortest length a stall has been measured at (AMD copyat 4096 d16 1312 ns against 60); whether shorter copies at those distances stall is unmeasured, and the issue records it. The tests now cover lengths across 4096 and to 8192, and the overlap test's every length reaches copy_long. Co-Authored-By: Claude Opus 5.5 --- sdk/libc/arch/src/lib.rs | 9 ++++---- sdk/libc/src/arch/x86_64.rs | 44 ++++++++++++++++++++++--------------- 2 files changed, 31 insertions(+), 22 deletions(-) diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 5ff1eff4178..1ccb0d2d45a 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -224,13 +224,13 @@ mod tests { /// 4 KiB, the source `at` bytes past a 64-byte boundary, both ends in one /// buffer: the whole buffer against `copy_within`. fn agrees_at(what: &str, copy: unsafe fn(*mut u8, *const u8, usize), distance: usize, at: usize, n: usize) { - let mut got = pattern(4 * 4096 + 256); + let mut got = pattern(6 * 4096 + 256); let base = got.as_ptr().align_offset(64); let (from, to) = (base + at, base + at + 8192 + distance); let mut want = got.clone(); want.copy_within(from..from + n, to); let p = got.as_mut_ptr(); - // SAFETY: both ranges are inside `got`, 8 KiB or more apart, `n` at most 4096. + // SAFETY: both ranges are inside `got`, 8 KiB or more apart, `n` at most 8192. unsafe { copy(p.add(to), p.add(from), n) }; assert_eq!(got, want, "{what}: n {n} at {at} distance {distance}"); } @@ -242,7 +242,7 @@ mod tests { fn a_forward_copy_agrees_at_every_page_distance() { for distance in (0..1024).chain([2048, 4032, 4095]) { for at in [0, 1, 17, 63] { - for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4096] { + for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4095, 4096, 4097, 5000, 8192] { agrees_at("copy_forward", arch::copy_forward, distance, at, n); } } @@ -278,7 +278,8 @@ mod tests { #[test] fn an_overlapping_forward_copy_a_page_apart_agrees() { for gap in [3584, 3600, 4000, 4080, 4095] { - for over in [1, 64, 500] { + // Every `n` at least 4096, the shortest copy that asks for the distance. + for over in [512, 600, 4096] { let n = gap + over; let base = pattern(n + gap); let mut want = base.clone(); diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 9c1fbe85d0e..b96d3d626ac 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -21,7 +21,19 @@ unsafe extern "C" fn _start() -> ! { } /// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for -/// any `dest` not above an overlapping `src`. +/// any `dest` not above an overlapping `src`. A copy shorter than +/// [`ROUTED_LEAST`] is one `rep movsb` behind one compare; only a longer one +/// asks where its ends sit. +#[inline(always)] +pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { + if n >= ROUTED_LEAST { + // SAFETY: as this function's caller promises. + return unsafe { copy_long(dest, src, n) }; + } + unsafe { movsb(dest, src, n) }; +} + +/// [`copy_forward`] from [`ROUTED_LEAST`] up, out of line. /// /// A forward copy whose destination is a short way above its source modulo /// 4 KiB loads each byte soon after storing to an unrelated address with the @@ -31,24 +43,16 @@ unsafe extern "C" fn _start() -> ! { /// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and /// IA-32 Architectures Optimization Reference Manual). Copied highest first, /// every store such a load could match is 4 KiB minus the distance behind it -/// and long retired. -pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { +/// and long retired. Only disjoint ranges go down. +#[inline(never)] +unsafe fn copy_long(dest: *mut u8, src: *const u8, n: usize) { let distance = (dest as usize).wrapping_sub(src as usize) % 4096; // `distance` in 1..SHORT_ALIAS, as one compare. - if n >= 64 && distance.wrapping_sub(1) < SHORT_ALIAS - 1 { - // SAFETY: as this function's caller promises. - return unsafe { copy_short_alias(dest, src, n) }; - } - unsafe { movsb(dest, src, n) }; -} - -/// [`copy_forward`] at a short distance: down where the ranges are disjoint, -/// out of line so the common copy carries none of it. -#[inline(never)] -unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) { - // SAFETY: `n` is at least 64; down only between disjoint ranges. + let short = distance.wrapping_sub(1) < SHORT_ALIAS - 1; + // SAFETY: `n` is at least `ROUTED_LEAST`, above the loops' 64; down only + // between disjoint ranges. unsafe { - if n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { + if !short || n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { movsb(dest, src, n); } else if has_avx() { copy_down_avx(dest, src, n); @@ -71,10 +75,14 @@ unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { } } -/// The distance under which [`copy_forward`] copies highest first. +/// The distance under which [`copy_long`] copies highest first. const SHORT_ALIAS: usize = 512; -/// The longest copy [`copy_forward`] copies highest first. A longer one runs +/// The shortest copy that asks where its ends sit, the shortest length a +/// stall has been measured at. Whether a shorter copy stalls is unmeasured. +const ROUTED_LEAST: usize = 4096; + +/// The longest copy [`copy_long`] copies highest first. A longer one runs /// from beyond the caches, where the loop's stores read each line before /// writing it and measured slower than `rep movsb` at distances that do not /// stall. From 643b89bc8c2fbdbae269577b918116249aaf3f2a Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 12:18:11 +0100 Subject: [PATCH 08/13] libc: copy_forward returns dest, so memcpy's short copy is main's With copy_long called, memcpy had to keep dest across the call: its prologue saved rbx and its epilogue restored it on every copy, short ones included. copy_forward now returns dest on both architectures, so memcpy and memmove end in a jump to copy_long and a copy under 4 KiB runs main's instructions plus one compare. Co-Authored-By: Claude Opus 5.5 --- sdk/libc/arch/src/lib.rs | 3 ++- sdk/libc/src/arch/aarch64.rs | 5 +++-- sdk/libc/src/arch/x86_64.rs | 12 +++++++----- sdk/libc/src/memory.rs | 7 +++---- 4 files changed, 15 insertions(+), 12 deletions(-) diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 1ccb0d2d45a..6b7509954fc 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -243,7 +243,8 @@ mod tests { for distance in (0..1024).chain([2048, 4032, 4095]) { for at in [0, 1, 17, 63] { for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4095, 4096, 4097, 5000, 8192] { - agrees_at("copy_forward", arch::copy_forward, distance, at, n); + // SAFETY: as `agrees_at` promises its copy. + agrees_at("copy_forward", |d, s, n| unsafe { _ = arch::copy_forward(d, s, n) }, distance, at, n); } } } diff --git a/sdk/libc/src/arch/aarch64.rs b/sdk/libc/src/arch/aarch64.rs index 2d7c0c27ce2..455ed9e10f3 100644 --- a/sdk/libc/src/arch/aarch64.rs +++ b/sdk/libc/src/arch/aarch64.rs @@ -27,8 +27,8 @@ unsafe extern "C" fn _start() -> ! { ); } -/// Copy `n` bytes from `src` to `dest`, lowest address first. -pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { +/// Copy `n` bytes from `src` to `dest`, lowest address first; returns `dest`. +pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { unsafe { core::arch::asm!( "2:", @@ -53,6 +53,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { options(nostack), ); } + dest } /// Copy `n` bytes from `src` to `dest`, highest address first: the order a diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index b96d3d626ac..744ce12744e 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -21,16 +21,17 @@ unsafe extern "C" fn _start() -> ! { } /// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for -/// any `dest` not above an overlapping `src`. A copy shorter than -/// [`ROUTED_LEAST`] is one `rep movsb` behind one compare; only a longer one -/// asks where its ends sit. +/// any `dest` not above an overlapping `src`. Returns `dest`, so `memcpy` +/// can end in a jump to [`copy_long`] and a shorter copy runs one `rep movsb` +/// behind one compare, with nothing held across a call. #[inline(always)] -pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { +pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { if n >= ROUTED_LEAST { // SAFETY: as this function's caller promises. return unsafe { copy_long(dest, src, n) }; } unsafe { movsb(dest, src, n) }; + dest } /// [`copy_forward`] from [`ROUTED_LEAST`] up, out of line. @@ -45,7 +46,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { /// every store such a load could match is 4 KiB minus the distance behind it /// and long retired. Only disjoint ranges go down. #[inline(never)] -unsafe fn copy_long(dest: *mut u8, src: *const u8, n: usize) { +unsafe fn copy_long(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { let distance = (dest as usize).wrapping_sub(src as usize) % 4096; // `distance` in 1..SHORT_ALIAS, as one compare. let short = distance.wrapping_sub(1) < SHORT_ALIAS - 1; @@ -60,6 +61,7 @@ unsafe fn copy_long(dest: *mut u8, src: *const u8, n: usize) { copy_down_sse2(dest, src, n); } } + dest } #[inline(always)] diff --git a/sdk/libc/src/memory.rs b/sdk/libc/src/memory.rs index e1805c835a1..436a6d84047 100644 --- a/sdk/libc/src/memory.rs +++ b/sdk/libc/src/memory.rs @@ -44,20 +44,19 @@ pub unsafe extern "C" fn posix_memalign(out: *mut *mut u8, align: usize, size: u #[allow(suspicious_runtime_symbol_definitions)] #[no_mangle] pub unsafe extern "C" fn memcpy(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { - crate::arch::copy_forward(dest, src, n); - dest + crate::arch::copy_forward(dest, src, n) } #[allow(suspicious_runtime_symbol_definitions)] #[no_mangle] pub unsafe extern "C" fn memmove(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { if (dest as usize) <= (src as usize) || (dest as usize) >= (src as usize) + n { - crate::arch::copy_forward(dest, src, n); + crate::arch::copy_forward(dest, src, n) } else { // Overlap with dest after src: copy backwards. crate::arch::copy_backward(dest, src, n); + dest } - dest } #[allow(suspicious_runtime_symbol_definitions)] From abfe08bf4cac77a6d0a0d8241cdea89642b7134d Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 12:26:54 +0100 Subject: [PATCH 09/13] issues: a copy under 4 KiB never asks the distance, and is unmeasured there Co-Authored-By: Claude Opus 5.5 --- ...t-way-above-its-source-modulo-4-kib-still-costs.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index e4ddcb53a1f..566eb079a26 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -7,9 +7,11 @@ opened: 2026-11-02 # A user copy a short way above its source modulo 4 KiB still costs `sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest -first when it has at least 64 bytes and at most 1 MiB, its destination is +first when it has at least 4 KiB and at most 1 MiB, its destination is 1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint. -Every other copy is one `rep movsb`. +Every other copy is one `rep movsb`, and a copy under 4 KiB never asks the +distance: round 7 measured every 64 B to 1 KiB copy 5-7% slower on the AMD +when every copy asked. The two benches stall over different windows, and the route covers both: - **AMD (Ryzen 7 4800U, Zen 2).** The processor judges store-to-load @@ -44,6 +46,11 @@ destination `d` bytes above the source: The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and d = 256, a distance that does not stall, it lost: 597 µs against 385. +- **Below 4 KiB, unmeasured.** No bench has timed a copy under 4 KiB at a + stalling distance, so whether one stalls, and what it costs, is unknown. + Round 8 adds unjudged `copyat` rows at 256, 1024 and 2048 bytes, d = 0 and + 16. + One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The other three boots read 16649–18226. That copy is page-aligned at d = 0 and never takes the route. It is unexplained. From 736ff8a93321eb763c3876eabf4c1bedb7331812 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 12:49:35 +0100 Subject: [PATCH 10/13] issues: copies under 4 KiB stall on the AMD, and the issue says how that ends Round 8 measured base copies at d = 16 against d = 0: 256 B 90 against 30 ns, 1 KiB 270 against 31, 2 KiB 511 against 40. They stall and are not routed. Exit C6 asks for them routed without any mem.memcpy row losing more than 10%, or a recorded decision to keep the stall. Review r8's suggestions: ROUTED_LEAST's comment no longer implies shorter lengths were measured clean, and the T14's 512 window is recorded as a choice within a taper. Co-Authored-By: Claude Opus 5.5 --- ...bove-its-source-modulo-4-kib-still-costs.md | 18 +++++++++++++----- sdk/libc/src/arch/x86_64.rs | 5 +++-- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index 566eb079a26..23b61576686 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -25,7 +25,8 @@ The two benches stall over different windows, and the route covers both: and d = 16 it takes 2615 ns against 90 at d = 0. - **T14 (i5-1135G7, Tiger Lake).** `copyat`'s copies, whose ends are 1 MiB plus the distance apart, read 714–773 ns at d = 16–64 and 206 at 256 - against 74 at d = 0, and 122 at 512. The 512 window is set by these rows. + against 74 at d = 0, then 122 at 512, 108 at 640 and 88 at 1024. The + cost tapers rather than ending, and 512 is a choice within the taper. Intel documents 4K aliasing on bits 11:0, but the same bench read 124 ns for two heap `Vec`s 64 KiB + 16 apart and 136 for #1139's in-process copy at d = 16, so what decides the T14's stall is not established. @@ -46,10 +47,12 @@ destination `d` bytes above the source: The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and d = 256, a distance that does not stall, it lost: 597 µs against 385. -- **Below 4 KiB, unmeasured.** No bench has timed a copy under 4 KiB at a - stalling distance, so whether one stalls, and what it costs, is unknown. - Round 8 adds unjudged `copyat` rows at 256, 1024 and 2048 bytes, d = 0 and - 16. +- **AMD, below 4 KiB.** These copies stall and are not routed. Round 8 + (base a049276a6, two boots, p50s identical in both) read, at d = 0 and + d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270, 2 KiB 40 and 511 (base's + 4 KiB: 60 and 992). Nothing below 256 B has been measured. Routing them + costs every copy from the new threshold up its out-of-line call: round + 8's 4 KiB copies at d = 0, which take that call, read 0.97 of main's. One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The other three boots read 16649–18226. That copy is page-aligned at d = 0 and @@ -68,6 +71,11 @@ Evidence: `ToyOSOrg/evidence` branch `pr-1133`, commit 64 MiB, within 1.1 times base's same row, and on the AMD also within 1.1 times base's d = 0 row of that size. +- C6: `copyat..libc.s0.d16` for 256 B, 1 KiB and 2 KiB within 1.5 + times that size's d = 0 on the AMD, with every `mem.memcpy.*` row within + 1.1 times main's; or a recorded decision that short copies keep the stall + because routing them costs more than it saves. + The `mem.memcpy.64` outlier is reproduced and explained, or shown to be independent of this copy. diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 744ce12744e..8fb8ea73e6c 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -80,8 +80,9 @@ unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { /// The distance under which [`copy_long`] copies highest first. const SHORT_ALIAS: usize = 512; -/// The shortest copy that asks where its ends sit, the shortest length a -/// stall has been measured at. Whether a shorter copy stalls is unmeasured. +/// The shortest copy that asks where its ends sit. Shorter copies stall too +/// (`issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md`) +/// but take one `rep movsb` behind one compare. const ROUTED_LEAST: usize = 4096; /// The longest copy [`copy_long`] copies highest first. A longer one runs From 97ae8d604f583c9fe1cab55c56ce2547684d8a7b Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 12:51:53 +0100 Subject: [PATCH 11/13] libc: a copy from 256 bytes asks its distance inline, and only a routed one jumps Round 8 measured copies under 4 KiB stalling on the AMD at d = 16 (256 B 3x, 1 KiB 8.7x, 2 KiB 12.8x) and left unrouted. Routing them through round 8's out-of-line copy_long would have cost every such copy a call (4 KiB read 0.97 of main's). The distance compare now sits inline in copy_forward, so memcpy and memmove ask it for every copy of 256 bytes or more without a call, and only a copy at a short distance jumps to copy_short_alias, which keeps the 1 MiB cap and the overlap guard. A copy under 256 bytes is main's rep movsb behind one compare. Co-Authored-By: Claude Opus 5.5 --- ...ove-its-source-modulo-4-kib-still-costs.md | 19 ++++--- sdk/libc/arch/src/lib.rs | 5 +- sdk/libc/src/arch/x86_64.rs | 53 +++++++++---------- 3 files changed, 37 insertions(+), 40 deletions(-) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index 23b61576686..9a1e2259037 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -7,11 +7,11 @@ opened: 2026-11-02 # A user copy a short way above its source modulo 4 KiB still costs `sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest -first when it has at least 4 KiB and at most 1 MiB, its destination is +first when it has at least 256 bytes and at most 1 MiB, its destination is 1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint. -Every other copy is one `rep movsb`, and a copy under 4 KiB never asks the -distance: round 7 measured every 64 B to 1 KiB copy 5-7% slower on the AMD -when every copy asked. +Every other copy is one `rep movsb`. A copy under 256 bytes never asks the +distance, and the check is inline for the rest: round 7 measured every 64 B +to 1 KiB copy 5-7% slower on the AMD when every copy made a call to ask. The two benches stall over different windows, and the route covers both: - **AMD (Ryzen 7 4800U, Zen 2).** The processor judges store-to-load @@ -47,12 +47,11 @@ destination `d` bytes above the source: The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and d = 256, a distance that does not stall, it lost: 597 µs against 385. -- **AMD, below 4 KiB.** These copies stall and are not routed. Round 8 - (base a049276a6, two boots, p50s identical in both) read, at d = 0 and - d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270, 2 KiB 40 and 511 (base's - 4 KiB: 60 and 992). Nothing below 256 B has been measured. Routing them - costs every copy from the new threshold up its out-of-line call: round - 8's 4 KiB copies at d = 0, which take that call, read 0.97 of main's. +- **AMD, below 4 KiB.** Round 8 (base a049276a6, two boots, p50s within + 10%) read, at d = 0 and d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270, + 2 KiB 40 and 511 (4 KiB: 60 and 992). Those copies are routed from 256 B + since round 9, which C6 judges. Nothing below 256 B is routed; round 9 + measures 64 and 128 B. One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The other three boots read 16649–18226. That copy is page-aligned at d = 0 and diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 6b7509954fc..2eb048c9e99 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -242,7 +242,7 @@ mod tests { fn a_forward_copy_agrees_at_every_page_distance() { for distance in (0..1024).chain([2048, 4032, 4095]) { for at in [0, 1, 17, 63] { - for n in [0, 1, 63, 64, 65, 127, 128, 200, 1000, 4095, 4096, 4097, 5000, 8192] { + for n in [0, 1, 63, 64, 65, 127, 128, 200, 255, 256, 257, 1000, 4096, 5000, 8192] { // SAFETY: as `agrees_at` promises its copy. agrees_at("copy_forward", |d, s, n| unsafe { _ = arch::copy_forward(d, s, n) }, distance, at, n); } @@ -279,8 +279,7 @@ mod tests { #[test] fn an_overlapping_forward_copy_a_page_apart_agrees() { for gap in [3584, 3600, 4000, 4080, 4095] { - // Every `n` at least 4096, the shortest copy that asks for the distance. - for over in [512, 600, 4096] { + for over in [1, 64, 500, 4096] { let n = gap + over; let base = pattern(n + gap); let mut want = base.clone(); diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index 8fb8ea73e6c..94e9496b9b2 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -21,39 +21,39 @@ unsafe extern "C" fn _start() -> ! { } /// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for -/// any `dest` not above an overlapping `src`. Returns `dest`, so `memcpy` -/// can end in a jump to [`copy_long`] and a shorter copy runs one `rep movsb` -/// behind one compare, with nothing held across a call. +/// any `dest` not above an overlapping `src`; returns `dest`. +/// +/// A forward copy whose destination is a short way above its source modulo +/// 4 KiB loads each byte soon after storing to an unrelated address with the +/// same bits 11:0, and the load waits on that store: AMD's Family 17h judges +/// store-to-load forwarding eligibility by those bits alone (Software +/// Optimization Guide for AMD Family 17h Models 30h and Greater Processors, +/// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and +/// IA-32 Architectures Optimization Reference Manual). Such a copy of at +/// least [`ROUTED_LEAST`] bytes jumps to [`copy_short_alias`]; every other copy +/// is one `rep movsb`, behind the length compare and, from `ROUTED_LEAST`, +/// the distance compare, both inline so no call is made. #[inline(always)] pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { - if n >= ROUTED_LEAST { + // The distance modulo 4 KiB in 1..SHORT_ALIAS, as one compare. + if n >= ROUTED_LEAST && ((dest as usize).wrapping_sub(src as usize) % 4096).wrapping_sub(1) < SHORT_ALIAS - 1 { // SAFETY: as this function's caller promises. - return unsafe { copy_long(dest, src, n) }; + return unsafe { copy_short_alias(dest, src, n) }; } unsafe { movsb(dest, src, n) }; dest } -/// [`copy_forward`] from [`ROUTED_LEAST`] up, out of line. -/// -/// A forward copy whose destination is a short way above its source modulo -/// 4 KiB loads each byte soon after storing to an unrelated address with the -/// same bits 11:0, and the load waits on that store: AMD's Family 17h judges -/// store-to-load forwarding eligibility by those bits alone (Software -/// Optimization Guide for AMD Family 17h Models 30h and Greater Processors, -/// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and -/// IA-32 Architectures Optimization Reference Manual). Copied highest first, -/// every store such a load could match is 4 KiB minus the distance behind it -/// and long retired. Only disjoint ranges go down. +/// [`copy_forward`] at a short distance, out of line: highest first, so every +/// store a load could match is 4 KiB minus the distance behind it and long +/// retired, where the ranges are disjoint and the copy at most +/// [`ROUTED_MOST`]; otherwise one `rep movsb`. #[inline(never)] -unsafe fn copy_long(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { - let distance = (dest as usize).wrapping_sub(src as usize) % 4096; - // `distance` in 1..SHORT_ALIAS, as one compare. - let short = distance.wrapping_sub(1) < SHORT_ALIAS - 1; +unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { // SAFETY: `n` is at least `ROUTED_LEAST`, above the loops' 64; down only // between disjoint ranges. unsafe { - if !short || n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { + if n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { movsb(dest, src, n); } else if has_avx() { copy_down_avx(dest, src, n); @@ -77,15 +77,14 @@ unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { } } -/// The distance under which [`copy_long`] copies highest first. +/// The distance under which [`copy_forward`] copies highest first. const SHORT_ALIAS: usize = 512; -/// The shortest copy that asks where its ends sit. Shorter copies stall too -/// (`issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md`) -/// but take one `rep movsb` behind one compare. -const ROUTED_LEAST: usize = 4096; +/// The shortest copy that asks where its ends sit; a shorter one pays one +/// compare over a bare `rep movsb`. +const ROUTED_LEAST: usize = 256; -/// The longest copy [`copy_long`] copies highest first. A longer one runs +/// The longest copy [`copy_short_alias`] copies highest first. A longer one runs /// from beyond the caches, where the loop's stores read each line before /// writing it and measured slower than `rep movsb` at distances that do not /// stall. From 7c67bbf527fc3eb662b8f0422abf26b9ea1fbca2 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 13:16:10 +0100 Subject: [PATCH 12/13] review of 97ae8d604: an overlap under 4 KiB the guard's absence reds, and C6 about short copies The overlap test's `over` of 64 could not catch a missing guard: the loops load the first 64 bytes before storing, so an overlap of 64 or less is never read clobbered. 200 makes n 3800-4295; with the guard removed the test now first fails at gap 3600 n 3800. The issue no longer names a pending round as a measurement, and C6 judges only the mem.memcpy rows whose copies ask the distance (256 B, 1 KiB, 4 KiB), so the open 256 KiB question cannot fail it. Co-Authored-By: Claude Opus 5.5 --- ...t-way-above-its-source-modulo-4-kib-still-costs.md | 11 ++++++----- sdk/libc/arch/src/lib.rs | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index 9a1e2259037..dd73181ca37 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -49,9 +49,9 @@ destination `d` bytes above the source: - **AMD, below 4 KiB.** Round 8 (base a049276a6, two boots, p50s within 10%) read, at d = 0 and d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270, - 2 KiB 40 and 511 (4 KiB: 60 and 992). Those copies are routed from 256 B - since round 9, which C6 judges. Nothing below 256 B is routed; round 9 - measures 64 and 128 B. + 2 KiB 40 and 511 (4 KiB: 60 and 992). Copies from 256 B are routed + (`ROUTED_LEAST`), which C6 judges. Nothing below 256 B is routed, and no + copy under 256 B has been timed at a stalling distance. One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The other three boots read 16649–18226. That copy is page-aligned at d = 0 and @@ -71,8 +71,9 @@ Evidence: `ToyOSOrg/evidence` branch `pr-1133`, commit times base's d = 0 row of that size. - C6: `copyat..libc.s0.d16` for 256 B, 1 KiB and 2 KiB within 1.5 - times that size's d = 0 on the AMD, with every `mem.memcpy.*` row within - 1.1 times main's; or a recorded decision that short copies keep the stall + times that size's d = 0 on the AMD, with `mem.memcpy.256`, `.1024` and + `.4096`, the rows whose copies are short and ask the distance, within 1.1 + times main's; or a recorded decision that short copies keep the stall because routing them costs more than it saves. The `mem.memcpy.64` outlier is reproduced and explained, or shown to be diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 2eb048c9e99..8e34e40ece4 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -279,7 +279,7 @@ mod tests { #[test] fn an_overlapping_forward_copy_a_page_apart_agrees() { for gap in [3584, 3600, 4000, 4080, 4095] { - for over in [1, 64, 500, 4096] { + for over in [1, 200, 500, 4096] { let n = gap + over; let base = pattern(n + gap); let mut want = base.clone(); From e93656701847a0db7407ff6fe438ae24781ecdc0 Mon Sep 17 00:00:00 2001 From: japabu Date: Thu, 5 Nov 2026 17:57:19 +0100 Subject: [PATCH 13/13] issues: mem.memcpy.262144 follows code placement, by round 10 and the ruling on it Co-Authored-By: Claude Opus 5.5 --- ...way-above-its-source-modulo-4-kib-still-costs.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md index dd73181ca37..06dfcfaae2c 100644 --- a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -53,6 +53,19 @@ destination `d` bytes above the source: (`ROUTED_LEAST`), which C6 judges. Nothing below 256 B is routed, and no copy under 256 B has been timed at a stalling distance. +- **AMD, `mem.memcpy.262144` follows code placement.** Rounds 7-9 read the + head at 0.906, 0.84 and 0.858 of base on that row, which never takes the + route. In round 10, a base whose `memcpy` was byte-identical but moved to + the head's address (by padding in `memcmp`, 4f14822ce) read 32030-35364 + MB/s, 0.80 of base's 38395-40764, against the head's 34397-35068. The rule + written for that round asked the shifted base to land inside the head's + range and failed because it landed below it. The orchestrator read that + by the rule's purpose, not its literal range clause: the move alone + reproduces the drop, so the row is sensitive to where `memcpy` sits, and + D3 passes on it. Which placement property matters is not established. + Evidence: `ToyOSOrg/evidence` commit + `6d717e2ad8683be5d02947ddc57dd3576e40ada6`, `amd-memcpy/round-10/`. + One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The other three boots read 16649–18226. That copy is page-aligned at d = 0 and never takes the route. It is unexplained.