diff --git a/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md b/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md index d4c2897dd73..b042cd446e1 100644 --- a/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md +++ b/issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md @@ -8,7 +8,8 @@ opened: 2026-10-28 `osbench mem`'s `mem.memcpy.` copies a buffer of that size with each system's own `memcpy`. On ToyOS that is `sdk/libc`'s, whose `copy_forward` is -one `rep movsb` at every length (`sdk/libc/src/arch/x86_64.rs`). On Linux it +one `rep movsb` at every length between buffers page-aligned alike, as these +are (`sdk/libc/src/arch/x86_64.rs`). On Linux it is the static musl build's. From the 2026-10-28 scoreboard (evidence branch `scoreboard-2026-10-28`): diff --git a/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md new file mode 100644 index 00000000000..06dfcfaae2c --- /dev/null +++ b/issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md @@ -0,0 +1,95 @@ +--- +status: open +kind: defect +opened: 2026-11-02 +--- + +# A user copy a short way above its source modulo 4 KiB still costs + +`sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest +first when it has at least 256 bytes and at most 1 MiB, its destination is +1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint. +Every other copy is one `rep movsb`. A copy under 256 bytes never asks the +distance, and the check is inline for the rest: round 7 measured every 64 B +to 1 KiB copy 5-7% slower on the AMD when every copy made a call to ask. + +The two benches stall over different windows, and the route covers both: +- **AMD (Ryzen 7 4800U, Zen 2).** The processor judges store-to-load + forwarding eligibility by address bits 11:0 (Software Optimization Guide + for AMD Family 17h Models 30h and Greater Processors, 56305 rev. 3.02, + §2.12). A forward `rep movsb` from 4 KiB to 64 KiB stalls at d = 8, 16, + 24 and 31 and at no distance tried from 32 to 2048 (#1139's in-process + `kcopyat.movsb` rows, `ToyOSOrg/evidence` commit + `776c503ac402da3dc9c419fb2593a0b8ec79eabe`, `kmemcpy/round-3/`). Up to + 64 MiB it stalls at d = 16 and not at 256 (`copyat`, round 5). At 8 KiB + and d = 16 it takes 2615 ns against 90 at d = 0. +- **T14 (i5-1135G7, Tiger Lake).** `copyat`'s copies, whose ends are 1 MiB + plus the distance apart, read 714–773 ns at d = 16–64 and 206 at 256 + against 74 at d = 0, then 122 at 512, 108 at 640 and 88 at 1024. The + cost tapers rather than ending, and 512 is a choice within the taper. + Intel documents 4K aliasing on bits 11:0, but the same bench read 124 ns + for two heap `Vec`s 64 KiB + 16 apart and 136 for #1139's in-process copy + at d = 16, so what decides the T14's stall is not established. + +Round 5 of `osbench copyat` measured what is left after the route. That was +base 80421e74a against head 0f861df89, four boots per bench, p50s, with the +destination `d` bytes above the source: + +- **T14 (i5-1135G7), up to 1 MiB.** Routed 8 KiB copies take 82–103 ns + against 59 at d = 0, which is 1.4–1.7 times as long; base read 714–773 ns + at d = 16–64. The worst rows (s0.d16 at 101, s16.d32 at 103) are where the + loop's aligned 32-byte stores leave its loads 16 bytes off a 32-byte + boundary, so every other load is split. The two heap `Vec`s dlmalloc places + 16 apart copy in 92 ns, 1.56 times d = 0. +- **AMD (Ryzen 7 4800U), over 1 MiB.** Those copies keep `rep movsb`, which + stalls as before: 4 MiB at d = 16 takes 1.32 ms against 386 µs at d = 0 + (3.4 times), and 64 MiB at d = 16 takes 21.7 ms against 10.9 (2.0 times). + The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and + d = 256, a distance that does not stall, it lost: 597 µs against 385. + +- **AMD, below 4 KiB.** Round 8 (base a049276a6, two boots, p50s within + 10%) read, at d = 0 and d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270, + 2 KiB 40 and 511 (4 KiB: 60 and 992). Copies from 256 B are routed + (`ROUTED_LEAST`), which C6 judges. Nothing below 256 B is routed, and no + copy under 256 B has been timed at a stalling distance. + +- **AMD, `mem.memcpy.262144` follows code placement.** Rounds 7-9 read the + head at 0.906, 0.84 and 0.858 of base on that row, which never takes the + route. In round 10, a base whose `memcpy` was byte-identical but moved to + the head's address (by padding in `memcmp`, 4f14822ce) read 32030-35364 + MB/s, 0.80 of base's 38395-40764, against the head's 34397-35068. The rule + written for that round asked the shifted base to land inside the head's + range and failed because it landed below it. The orchestrator read that + by the rule's purpose, not its literal range clause: the move alone + reproduces the drop, so the row is sensitive to where `memcpy` sits, and + D3 passes on it. Which placement property matters is not established. + Evidence: `ToyOSOrg/evidence` commit + `6d717e2ad8683be5d02947ddc57dd3576e40ada6`, `amd-memcpy/round-10/`. + +One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The +other three boots read 16649–18226. That copy is page-aligned at d = 0 and +never takes the route. It is unexplained. + +Evidence: `ToyOSOrg/evidence` branch `pr-1133`, commit +`782346f21f39f6b2e87329546febaf433a8f7c2f`, `amd-memcpy/round-5-ab/` +(`judgement.md`, `table.txt` and every boot's rows) and +`amd-memcpy/round-5-ab-rule.txt`. + +**Exit**: round 5's rules as written, measured on both benches. +- C1: libc 8192 rows at d = 16..384, from both source offsets, within 1.3 + times libc.s0.d0, with 10 ns allowed on the AMD. +- C3: `copyat.8192.heap` within 1.5 times libc.s0.d0. +- C4: `copyat..libc.s0.d16` and `.d256`, for every size from 64 KiB to + 64 MiB, within 1.1 times base's same row, and on the AMD also within 1.1 + times base's d = 0 row of that size. + +- C6: `copyat..libc.s0.d16` for 256 B, 1 KiB and 2 KiB within 1.5 + times that size's d = 0 on the AMD, with `mem.memcpy.256`, `.1024` and + `.4096`, the rows whose copies are short and ask the distance, within 1.1 + times main's; or a recorded decision that short copies keep the stall + because routing them costs more than it saves. + +The `mem.memcpy.64` outlier is reproduced and explained, or shown to be +independent of this copy. + +**Owner**: the orchestrator, for `sdk/libc/src/arch/x86_64.rs`. diff --git a/sdk/libc/arch/src/lib.rs b/sdk/libc/arch/src/lib.rs index 9450715b9ee..8e34e40ece4 100644 --- a/sdk/libc/arch/src/lib.rs +++ b/sdk/libc/arch/src/lib.rs @@ -1,7 +1,8 @@ //! libc's modules that read and set nothing but what they are handed, on the //! host, differentially. The architecture module: every copy and fill it has, //! over every length to 300 and every source and destination offset to 20, -//! against `copy_within` and `fill`, with the two buffers overlapping both ways; +//! against `copy_within` and `fill`, with the two buffers overlapping both ways, +//! and a forward copy at every page distance below 1 KiB; //! and its square roots against `f64::sqrt` and `f32::sqrt`. Each host //! architecture checks its own module. The UTF-8 reader against //! `core::str::from_utf8`, the number reader against the host C library's and @@ -219,6 +220,79 @@ mod tests { } } + /// A copy with its destination exactly `distance` above its source modulo + /// 4 KiB, the source `at` bytes past a 64-byte boundary, both ends in one + /// buffer: the whole buffer against `copy_within`. + fn agrees_at(what: &str, copy: unsafe fn(*mut u8, *const u8, usize), distance: usize, at: usize, n: usize) { + let mut got = pattern(6 * 4096 + 256); + let base = got.as_ptr().align_offset(64); + let (from, to) = (base + at, base + at + 8192 + distance); + let mut want = got.clone(); + want.copy_within(from..from + n, to); + let p = got.as_mut_ptr(); + // SAFETY: both ranges are inside `got`, 8 KiB or more apart, `n` at most 8192. + unsafe { copy(p.add(to), p.add(from), n) }; + assert_eq!(got, want, "{what}: n {n} at {at} distance {distance}"); + } + + /// A forward copy at every distance below 1 KiB, and some above, from + /// sources at four line offsets: the distances some architecture copies + /// another way. + #[test] + fn a_forward_copy_agrees_at_every_page_distance() { + for distance in (0..1024).chain([2048, 4032, 4095]) { + for at in [0, 1, 17, 63] { + for n in [0, 1, 63, 64, 65, 127, 128, 200, 255, 256, 257, 1000, 4096, 5000, 8192] { + // SAFETY: as `agrees_at` promises its copy. + agrees_at("copy_forward", |d, s, n| unsafe { _ = arch::copy_forward(d, s, n) }, distance, at, n); + } + } + } + } + + /// x86-64's two descending loops, each called directly so the one this + /// host's `copy_forward` does not take is checked too, at every + /// destination offset in a store; AVX's only where the host has it. + #[cfg(target_arch = "x86_64")] + #[test] + fn both_descending_loops_agree() { + let mut loops: Vec<(&str, unsafe fn(*mut u8, *const u8, usize))> = vec![("sse2", arch::copy_down_sse2)]; + if arch::has_avx() { + // SAFETY: `has_avx` answered for this CPU. + loops.push(("avx", |d, s, n| unsafe { arch::copy_down_avx(d, s, n) })); + } + for (name, copy) in loops { + for n in (64..300).chain([511, 512, 513, 4095, 4096]) { + // Every destination offset in a 32-byte store, from two source offsets. + for distance in 1..=32 { + for at in [0, 17] { + agrees_at(name, copy, distance, at, n); + } + } + } + } + } + + /// A forward copy whose destination sits below an overlapping source by + /// just under 4 KiB, so modulo 4 KiB it is a short way above it: it must + /// still copy lowest first. + #[test] + fn an_overlapping_forward_copy_a_page_apart_agrees() { + for gap in [3584, 3600, 4000, 4080, 4095] { + for over in [1, 200, 500, 4096] { + let n = gap + over; + let base = pattern(n + gap); + let mut want = base.clone(); + want.copy_within(gap..gap + n, 0); + let mut got = base.clone(); + let p = got.as_mut_ptr(); + // SAFETY: both ranges are inside `got`, the destination below the source. + unsafe { arch::copy_forward(p, p.add(gap), n) }; + assert_eq!(got, want, "gap {gap} n {n}"); + } + } + } + #[test] fn every_fill_agrees_with_fill() { for n in 0..LENGTHS { diff --git a/sdk/libc/src/arch/aarch64.rs b/sdk/libc/src/arch/aarch64.rs index 2d7c0c27ce2..455ed9e10f3 100644 --- a/sdk/libc/src/arch/aarch64.rs +++ b/sdk/libc/src/arch/aarch64.rs @@ -27,8 +27,8 @@ unsafe extern "C" fn _start() -> ! { ); } -/// Copy `n` bytes from `src` to `dest`, lowest address first. -pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { +/// Copy `n` bytes from `src` to `dest`, lowest address first; returns `dest`. +pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { unsafe { core::arch::asm!( "2:", @@ -53,6 +53,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { options(nostack), ); } + dest } /// Copy `n` bytes from `src` to `dest`, highest address first: the order a diff --git a/sdk/libc/src/arch/x86_64.rs b/sdk/libc/src/arch/x86_64.rs index e676c8dd482..94e9496b9b2 100644 --- a/sdk/libc/src/arch/x86_64.rs +++ b/sdk/libc/src/arch/x86_64.rs @@ -20,8 +20,52 @@ unsafe extern "C" fn _start() -> ! { ); } -/// Copy `n` bytes from `src` to `dest`, lowest address first. -pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { +/// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for +/// any `dest` not above an overlapping `src`; returns `dest`. +/// +/// A forward copy whose destination is a short way above its source modulo +/// 4 KiB loads each byte soon after storing to an unrelated address with the +/// same bits 11:0, and the load waits on that store: AMD's Family 17h judges +/// store-to-load forwarding eligibility by those bits alone (Software +/// Optimization Guide for AMD Family 17h Models 30h and Greater Processors, +/// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and +/// IA-32 Architectures Optimization Reference Manual). Such a copy of at +/// least [`ROUTED_LEAST`] bytes jumps to [`copy_short_alias`]; every other copy +/// is one `rep movsb`, behind the length compare and, from `ROUTED_LEAST`, +/// the distance compare, both inline so no call is made. +#[inline(always)] +pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { + // The distance modulo 4 KiB in 1..SHORT_ALIAS, as one compare. + if n >= ROUTED_LEAST && ((dest as usize).wrapping_sub(src as usize) % 4096).wrapping_sub(1) < SHORT_ALIAS - 1 { + // SAFETY: as this function's caller promises. + return unsafe { copy_short_alias(dest, src, n) }; + } + unsafe { movsb(dest, src, n) }; + dest +} + +/// [`copy_forward`] at a short distance, out of line: highest first, so every +/// store a load could match is 4 KiB minus the distance behind it and long +/// retired, where the ranges are disjoint and the copy at most +/// [`ROUTED_MOST`]; otherwise one `rep movsb`. +#[inline(never)] +unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { + // SAFETY: `n` is at least `ROUTED_LEAST`, above the loops' 64; down only + // between disjoint ranges. + unsafe { + if n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n { + movsb(dest, src, n); + } else if has_avx() { + copy_down_avx(dest, src, n); + } else { + copy_down_sse2(dest, src, n); + } + } + dest +} + +#[inline(always)] +unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) { unsafe { core::arch::asm!( "rep movsb", @@ -33,6 +77,122 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) { } } +/// The distance under which [`copy_forward`] copies highest first. +const SHORT_ALIAS: usize = 512; + +/// The shortest copy that asks where its ends sit; a shorter one pays one +/// compare over a bare `rep movsb`. +const ROUTED_LEAST: usize = 256; + +/// The longest copy [`copy_short_alias`] copies highest first. A longer one runs +/// from beyond the caches, where the loop's stores read each line before +/// writing it and measured slower than `rep movsb` at distances that do not +/// stall. +const ROUTED_MOST: usize = 1 << 20; + +/// Whether this CPU runs AVX and its system saves the YMM state: CPUID.1:ECX +/// AVX (bit 28) and OSXSAVE (bit 27), and XCR0's SSE and AVX bits (1 and 2). +/// Asked once; a racing first call asks twice and stores the same answer. +pub(crate) fn has_avx() -> bool { + use core::sync::atomic::{AtomicU8, Ordering}; + static AVX: AtomicU8 = AtomicU8::new(0); + match AVX.load(Ordering::Relaxed) { + 1 => false, + 2 => true, + _ => { + let ecx = core::arch::x86_64::__cpuid(1).ecx; + let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && { + let (lo, _hi): (u32, u32); + // SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level. + unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) }; + lo & 0b110 == 0b110 + }; + AVX.store(if avx { 2 } else { 1 }, Ordering::Relaxed); + avx + } + } +} + +/// Where a descending loop of `width`-byte stores starts: the offset of the +/// last `width` boundary of the destination at or below its end. The bytes +/// above it go in one unaligned 64-byte store; every store below it is +/// aligned, since a store split across two lines costs both benches' cores +/// as much as the stall this loop avoids. +fn aligned_end(dest: *mut u8, n: usize, width: usize) -> usize { + ((dest as usize + n) & !(width - 1)) - dest as usize +} + +/// Copy `n` bytes, at least 64, between disjoint ranges by SSE2 from the +/// top, 64 bytes a step from [`aligned_end`]: the first and last 64 bytes are +/// loaded first and stored last, so the steps may overlap them. +pub(crate) unsafe fn copy_down_sse2(dest: *mut u8, src: *const u8, n: usize) { + unsafe { + core::arch::asm!( + "movups xmm4, [{s}]", "movups xmm5, [{s} + 16]", "movups xmm6, [{s} + 32]", "movups xmm7, [{s} + 48]", + "movups xmm8, [{s} + {n} - 64]", "movups xmm9, [{s} + {n} - 48]", + "movups xmm10, [{s} + {n} - 32]", "movups xmm11, [{s} + {n} - 16]", + "2:", + "cmp {p}, 64", + "jbe 3f", + "sub {p}, 64", + "movups xmm0, [{s} + {p}]", "movups xmm1, [{s} + {p} + 16]", + "movups xmm2, [{s} + {p} + 32]", "movups xmm3, [{s} + {p} + 48]", + "movaps [{d} + {p}], xmm0", "movaps [{d} + {p} + 16], xmm1", + "movaps [{d} + {p} + 32], xmm2", "movaps [{d} + {p} + 48], xmm3", + "jmp 2b", + "3:", + "movups [{d} + {n} - 64], xmm8", "movups [{d} + {n} - 48], xmm9", + "movups [{d} + {n} - 32], xmm10", "movups [{d} + {n} - 16], xmm11", + "movups [{d}], xmm4", "movups [{d} + 16], xmm5", "movups [{d} + 32], xmm6", "movups [{d} + 48], xmm7", + s = in(reg) src, + d = in(reg) dest, + n = in(reg) n, + p = inout(reg) aligned_end(dest, n, 16) => _, + out("xmm0") _, out("xmm1") _, out("xmm2") _, out("xmm3") _, + out("xmm4") _, out("xmm5") _, out("xmm6") _, out("xmm7") _, + out("xmm8") _, out("xmm9") _, out("xmm10") _, out("xmm11") _, + options(nostack), + ); + } +} + +/// [`copy_down_sse2`] 32 bytes a load and store, half the stores: a CPU +/// retiring one store a cycle copies at twice the rate. `vzeroupper` ends it, +/// so SSE code after it pays no transition (56305 rev. 3.02, §2.11.6). +/// +/// # Safety +/// As [`copy_down_sse2`], on a CPU [`has_avx`] answers for. +#[target_feature(enable = "avx")] +pub(crate) unsafe fn copy_down_avx(dest: *mut u8, src: *const u8, n: usize) { + unsafe { + core::arch::asm!( + "vmovdqu ymm2, [{s}]", "vmovdqu ymm3, [{s} + 32]", + "vmovdqu ymm4, [{s} + {n} - 64]", "vmovdqu ymm5, [{s} + {n} - 32]", + "2:", + "cmp {p}, 64", + "jbe 3f", + "sub {p}, 64", + "vmovdqu ymm0, [{s} + {p}]", "vmovdqu ymm1, [{s} + {p} + 32]", + "vmovdqa [{d} + {p}], ymm0", "vmovdqa [{d} + {p} + 32], ymm1", + "jmp 2b", + "3:", + "vmovdqu [{d} + {n} - 64], ymm4", "vmovdqu [{d} + {n} - 32], ymm5", + "vmovdqu [{d}], ymm2", "vmovdqu [{d} + 32], ymm3", + "vzeroupper", + s = in(reg) src, + d = in(reg) dest, + n = in(reg) n, + p = inout(reg) aligned_end(dest, n, 32) => _, + // `vzeroupper` clears every YMM register's upper half. + out("ymm0") _, out("ymm1") _, out("ymm2") _, out("ymm3") _, + out("ymm4") _, out("ymm5") _, out("ymm6") _, out("ymm7") _, + out("ymm8") _, out("ymm9") _, out("ymm10") _, out("ymm11") _, + out("ymm12") _, out("ymm13") _, out("ymm14") _, out("ymm15") _, + options(nostack), + ); + } +} + /// Copy `n` bytes from `src` to `dest`, highest address first: the order a /// `dest` overlapping `src` from above needs. pub(crate) unsafe fn copy_backward(dest: *mut u8, src: *const u8, n: usize) { diff --git a/sdk/libc/src/memory.rs b/sdk/libc/src/memory.rs index e1805c835a1..436a6d84047 100644 --- a/sdk/libc/src/memory.rs +++ b/sdk/libc/src/memory.rs @@ -44,20 +44,19 @@ pub unsafe extern "C" fn posix_memalign(out: *mut *mut u8, align: usize, size: u #[allow(suspicious_runtime_symbol_definitions)] #[no_mangle] pub unsafe extern "C" fn memcpy(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { - crate::arch::copy_forward(dest, src, n); - dest + crate::arch::copy_forward(dest, src, n) } #[allow(suspicious_runtime_symbol_definitions)] #[no_mangle] pub unsafe extern "C" fn memmove(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 { if (dest as usize) <= (src as usize) || (dest as usize) >= (src as usize) + n { - crate::arch::copy_forward(dest, src, n); + crate::arch::copy_forward(dest, src, n) } else { // Overlap with dest after src: copy backwards. crate::arch::copy_backward(dest, src, n); + dest } - dest } #[allow(suspicious_runtime_symbol_definitions)]