Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ opened: 2026-10-28

`osbench mem`'s `mem.memcpy.<bytes>` copies a buffer of that size with each
system's own `memcpy`. On ToyOS that is `sdk/libc`'s, whose `copy_forward` is
one `rep movsb` at every length (`sdk/libc/src/arch/x86_64.rs`). On Linux it
one `rep movsb` at every length between buffers page-aligned alike, as these
are (`sdk/libc/src/arch/x86_64.rs`). On Linux it
is the static musl build's.

From the 2026-10-28 scoreboard (evidence branch `scoreboard-2026-10-28`):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
status: open
kind: defect
opened: 2026-11-02
---

# A user copy a short way above its source modulo 4 KiB still costs

`sdk/libc`'s `copy_forward` (`sdk/libc/src/arch/x86_64.rs`) copies highest
first when it has at least 256 bytes and at most 1 MiB, its destination is
1 to 511 bytes above its source modulo 4096, and the two ranges are disjoint.
Every other copy is one `rep movsb`. A copy under 256 bytes never asks the
distance, and the check is inline for the rest: round 7 measured every 64 B
to 1 KiB copy 5-7% slower on the AMD when every copy made a call to ask.

The two benches stall over different windows, and the route covers both:
- **AMD (Ryzen 7 4800U, Zen 2).** The processor judges store-to-load
forwarding eligibility by address bits 11:0 (Software Optimization Guide
for AMD Family 17h Models 30h and Greater Processors, 56305 rev. 3.02,
§2.12). A forward `rep movsb` from 4 KiB to 64 KiB stalls at d = 8, 16,
24 and 31 and at no distance tried from 32 to 2048 (#1139's in-process
`kcopyat.movsb` rows, `ToyOSOrg/evidence` commit
`776c503ac402da3dc9c419fb2593a0b8ec79eabe`, `kmemcpy/round-3/`). Up to
64 MiB it stalls at d = 16 and not at 256 (`copyat`, round 5). At 8 KiB
and d = 16 it takes 2615 ns against 90 at d = 0.
- **T14 (i5-1135G7, Tiger Lake).** `copyat`'s copies, whose ends are 1 MiB
plus the distance apart, read 714–773 ns at d = 16–64 and 206 at 256
against 74 at d = 0, then 122 at 512, 108 at 640 and 88 at 1024. The
cost tapers rather than ending, and 512 is a choice within the taper.
Intel documents 4K aliasing on bits 11:0, but the same bench read 124 ns
for two heap `Vec`s 64 KiB + 16 apart and 136 for #1139's in-process copy
at d = 16, so what decides the T14's stall is not established.

Round 5 of `osbench copyat` measured what is left after the route. That was
base 80421e74a against head 0f861df89, four boots per bench, p50s, with the
destination `d` bytes above the source:

- **T14 (i5-1135G7), up to 1 MiB.** Routed 8 KiB copies take 82–103 ns
against 59 at d = 0, which is 1.4–1.7 times as long; base read 714–773 ns
at d = 16–64. The worst rows (s0.d16 at 101, s16.d32 at 103) are where the
loop's aligned 32-byte stores leave its loads 16 bytes off a 32-byte
boundary, so every other load is split. The two heap `Vec`s dlmalloc places
16 apart copy in 92 ns, 1.56 times d = 0.
- **AMD (Ryzen 7 4800U), over 1 MiB.** Those copies keep `rep movsb`, which
stalls as before: 4 MiB at d = 16 takes 1.32 ms against 386 µs at d = 0
(3.4 times), and 64 MiB at d = 16 takes 21.7 ms against 10.9 (2.0 times).
The descending loop beat it at d = 16 there, 612 µs and 9.8 ms. At 4 MiB and
d = 256, a distance that does not stall, it lost: 597 µs against 385.

- **AMD, below 4 KiB.** Round 8 (base a049276a6, two boots, p50s within
10%) read, at d = 0 and d = 16: 256 B 30 and 90 ns, 1 KiB 31 and 270,
2 KiB 40 and 511 (4 KiB: 60 and 992). Copies from 256 B are routed
(`ROUTED_LEAST`), which C6 judges. Nothing below 256 B is routed, and no
copy under 256 B has been timed at a stalling distance.

- **AMD, `mem.memcpy.262144` follows code placement.** Rounds 7-9 read the
head at 0.906, 0.84 and 0.858 of base on that row, which never takes the
route. In round 10, a base whose `memcpy` was byte-identical but moved to
the head's address (by padding in `memcmp`, 4f14822ce) read 32030-35364
MB/s, 0.80 of base's 38395-40764, against the head's 34397-35068. The rule
written for that round asked the shifted base to land inside the head's
range and failed because it landed below it. The orchestrator read that
by the rule's purpose, not its literal range clause: the move alone
reproduces the drop, so the row is sensitive to where `memcpy` sits, and
D3 passes on it. Which placement property matters is not established.
Evidence: `ToyOSOrg/evidence` commit
`6d717e2ad8683be5d02947ddc57dd3576e40ada6`, `amd-memcpy/round-10/`.

One T14 head boot read `mem.memcpy.64` at 6873 MB/s in all 15 rounds. The
other three boots read 16649–18226. That copy is page-aligned at d = 0 and
never takes the route. It is unexplained.

Evidence: `ToyOSOrg/evidence` branch `pr-1133`, commit
`782346f21f39f6b2e87329546febaf433a8f7c2f`, `amd-memcpy/round-5-ab/`
(`judgement.md`, `table.txt` and every boot's rows) and
`amd-memcpy/round-5-ab-rule.txt`.

**Exit**: round 5's rules as written, measured on both benches.
- C1: libc 8192 rows at d = 16..384, from both source offsets, within 1.3
times libc.s0.d0, with 10 ns allowed on the AMD.
- C3: `copyat.8192.heap` within 1.5 times libc.s0.d0.
- C4: `copyat.<size>.libc.s0.d16` and `.d256`, for every size from 64 KiB to
64 MiB, within 1.1 times base's same row, and on the AMD also within 1.1
times base's d = 0 row of that size.

- C6: `copyat.<size>.libc.s0.d16` for 256 B, 1 KiB and 2 KiB within 1.5
times that size's d = 0 on the AMD, with `mem.memcpy.256`, `.1024` and
`.4096`, the rows whose copies are short and ask the distance, within 1.1
times main's; or a recorded decision that short copies keep the stall
because routing them costs more than it saves.

The `mem.memcpy.64` outlier is reproduced and explained, or shown to be
independent of this copy.

**Owner**: the orchestrator, for `sdk/libc/src/arch/x86_64.rs`.
76 changes: 75 additions & 1 deletion sdk/libc/arch/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
//! libc's modules that read and set nothing but what they are handed, on the
//! host, differentially. The architecture module: every copy and fill it has,
//! over every length to 300 and every source and destination offset to 20,
//! against `copy_within` and `fill`, with the two buffers overlapping both ways;
//! against `copy_within` and `fill`, with the two buffers overlapping both ways,
//! and a forward copy at every page distance below 1 KiB;
//! and its square roots against `f64::sqrt` and `f32::sqrt`. Each host
//! architecture checks its own module. The UTF-8 reader against
//! `core::str::from_utf8`, the number reader against the host C library's and
Expand Down Expand Up @@ -219,6 +220,79 @@ mod tests {
}
}

/// A copy with its destination exactly `distance` above its source modulo
/// 4 KiB, the source `at` bytes past a 64-byte boundary, both ends in one
/// buffer: the whole buffer against `copy_within`.
fn agrees_at(what: &str, copy: unsafe fn(*mut u8, *const u8, usize), distance: usize, at: usize, n: usize) {
let mut got = pattern(6 * 4096 + 256);
let base = got.as_ptr().align_offset(64);
let (from, to) = (base + at, base + at + 8192 + distance);
let mut want = got.clone();
want.copy_within(from..from + n, to);
let p = got.as_mut_ptr();
// SAFETY: both ranges are inside `got`, 8 KiB or more apart, `n` at most 8192.
unsafe { copy(p.add(to), p.add(from), n) };
assert_eq!(got, want, "{what}: n {n} at {at} distance {distance}");
}

/// A forward copy at every distance below 1 KiB, and some above, from
/// sources at four line offsets: the distances some architecture copies
/// another way.
#[test]
fn a_forward_copy_agrees_at_every_page_distance() {
for distance in (0..1024).chain([2048, 4032, 4095]) {
for at in [0, 1, 17, 63] {
for n in [0, 1, 63, 64, 65, 127, 128, 200, 255, 256, 257, 1000, 4096, 5000, 8192] {
// SAFETY: as `agrees_at` promises its copy.
agrees_at("copy_forward", |d, s, n| unsafe { _ = arch::copy_forward(d, s, n) }, distance, at, n);
}
}
}
}

/// x86-64's two descending loops, each called directly so the one this
/// host's `copy_forward` does not take is checked too, at every
/// destination offset in a store; AVX's only where the host has it.
#[cfg(target_arch = "x86_64")]
#[test]
fn both_descending_loops_agree() {
let mut loops: Vec<(&str, unsafe fn(*mut u8, *const u8, usize))> = vec![("sse2", arch::copy_down_sse2)];
if arch::has_avx() {
// SAFETY: `has_avx` answered for this CPU.
loops.push(("avx", |d, s, n| unsafe { arch::copy_down_avx(d, s, n) }));
}
for (name, copy) in loops {
for n in (64..300).chain([511, 512, 513, 4095, 4096]) {
// Every destination offset in a 32-byte store, from two source offsets.
for distance in 1..=32 {
for at in [0, 17] {
agrees_at(name, copy, distance, at, n);
}
}
}
}
}

/// A forward copy whose destination sits below an overlapping source by
/// just under 4 KiB, so modulo 4 KiB it is a short way above it: it must
/// still copy lowest first.
#[test]
fn an_overlapping_forward_copy_a_page_apart_agrees() {
for gap in [3584, 3600, 4000, 4080, 4095] {
for over in [1, 200, 500, 4096] {
let n = gap + over;
let base = pattern(n + gap);
let mut want = base.clone();
want.copy_within(gap..gap + n, 0);
let mut got = base.clone();
let p = got.as_mut_ptr();
// SAFETY: both ranges are inside `got`, the destination below the source.
unsafe { arch::copy_forward(p, p.add(gap), n) };
assert_eq!(got, want, "gap {gap} n {n}");
}
}
}

#[test]
fn every_fill_agrees_with_fill() {
for n in 0..LENGTHS {
Expand Down
5 changes: 3 additions & 2 deletions sdk/libc/src/arch/aarch64.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ unsafe extern "C" fn _start() -> ! {
);
}

/// Copy `n` bytes from `src` to `dest`, lowest address first.
pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) {
/// Copy `n` bytes from `src` to `dest`, lowest address first; returns `dest`.
pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 {
unsafe {
core::arch::asm!(
"2:",
Expand All @@ -53,6 +53,7 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) {
options(nostack),
);
}
dest
}

/// Copy `n` bytes from `src` to `dest`, highest address first: the order a
Expand Down
164 changes: 162 additions & 2 deletions sdk/libc/src/arch/x86_64.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,52 @@ unsafe extern "C" fn _start() -> ! {
);
}

/// Copy `n` bytes from `src` to `dest`, lowest address first.
pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) {
/// Copy `n` bytes from `src` to `dest` as a forward copy would: correct for
/// any `dest` not above an overlapping `src`; returns `dest`.
///
/// A forward copy whose destination is a short way above its source modulo
/// 4 KiB loads each byte soon after storing to an unrelated address with the
/// same bits 11:0, and the load waits on that store: AMD's Family 17h judges
/// store-to-load forwarding eligibility by those bits alone (Software
/// Optimization Guide for AMD Family 17h Models 30h and Greater Processors,
/// 56305 rev. 3.02, §2.12), and Intel calls it 4K aliasing (Intel 64 and
/// IA-32 Architectures Optimization Reference Manual). Such a copy of at
/// least [`ROUTED_LEAST`] bytes jumps to [`copy_short_alias`]; every other copy
/// is one `rep movsb`, behind the length compare and, from `ROUTED_LEAST`,
/// the distance compare, both inline so no call is made.
#[inline(always)]
pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 {
// The distance modulo 4 KiB in 1..SHORT_ALIAS, as one compare.
if n >= ROUTED_LEAST && ((dest as usize).wrapping_sub(src as usize) % 4096).wrapping_sub(1) < SHORT_ALIAS - 1 {
// SAFETY: as this function's caller promises.
return unsafe { copy_short_alias(dest, src, n) };
}
unsafe { movsb(dest, src, n) };
dest
}

/// [`copy_forward`] at a short distance, out of line: highest first, so every
/// store a load could match is 4 KiB minus the distance behind it and long
/// retired, where the ranges are disjoint and the copy at most
/// [`ROUTED_MOST`]; otherwise one `rep movsb`.
#[inline(never)]
unsafe fn copy_short_alias(dest: *mut u8, src: *const u8, n: usize) -> *mut u8 {
// SAFETY: `n` is at least `ROUTED_LEAST`, above the loops' 64; down only
// between disjoint ranges.
unsafe {
if n > ROUTED_MOST || (dest as usize).abs_diff(src as usize) < n {
movsb(dest, src, n);
} else if has_avx() {
copy_down_avx(dest, src, n);
} else {
copy_down_sse2(dest, src, n);
}
}
dest
}

#[inline(always)]
unsafe fn movsb(dest: *mut u8, src: *const u8, n: usize) {
unsafe {
core::arch::asm!(
"rep movsb",
Expand All @@ -33,6 +77,122 @@ pub(crate) unsafe fn copy_forward(dest: *mut u8, src: *const u8, n: usize) {
}
}

/// The distance under which [`copy_forward`] copies highest first.
const SHORT_ALIAS: usize = 512;

/// The shortest copy that asks where its ends sit; a shorter one pays one
/// compare over a bare `rep movsb`.
const ROUTED_LEAST: usize = 256;

/// The longest copy [`copy_short_alias`] copies highest first. A longer one runs
/// from beyond the caches, where the loop's stores read each line before
/// writing it and measured slower than `rep movsb` at distances that do not
/// stall.
const ROUTED_MOST: usize = 1 << 20;

/// Whether this CPU runs AVX and its system saves the YMM state: CPUID.1:ECX
/// AVX (bit 28) and OSXSAVE (bit 27), and XCR0's SSE and AVX bits (1 and 2).
/// Asked once; a racing first call asks twice and stores the same answer.
pub(crate) fn has_avx() -> bool {
use core::sync::atomic::{AtomicU8, Ordering};
static AVX: AtomicU8 = AtomicU8::new(0);
match AVX.load(Ordering::Relaxed) {
1 => false,
2 => true,
_ => {
let ecx = core::arch::x86_64::__cpuid(1).ecx;
let avx = ecx & (1 << 28) != 0 && ecx & (1 << 27) != 0 && {
let (lo, _hi): (u32, u32);
// SAFETY: OSXSAVE says XGETBV exists; XCR0 is readable at every privilege level.
unsafe { core::arch::asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") _hi, options(nomem, nostack, preserves_flags)) };
lo & 0b110 == 0b110
};
AVX.store(if avx { 2 } else { 1 }, Ordering::Relaxed);
avx
}
}
}

/// Where a descending loop of `width`-byte stores starts: the offset of the
/// last `width` boundary of the destination at or below its end. The bytes
/// above it go in one unaligned 64-byte store; every store below it is
/// aligned, since a store split across two lines costs both benches' cores
/// as much as the stall this loop avoids.
fn aligned_end(dest: *mut u8, n: usize, width: usize) -> usize {
((dest as usize + n) & !(width - 1)) - dest as usize
}

/// Copy `n` bytes, at least 64, between disjoint ranges by SSE2 from the
/// top, 64 bytes a step from [`aligned_end`]: the first and last 64 bytes are
/// loaded first and stored last, so the steps may overlap them.
pub(crate) unsafe fn copy_down_sse2(dest: *mut u8, src: *const u8, n: usize) {
unsafe {
core::arch::asm!(
"movups xmm4, [{s}]", "movups xmm5, [{s} + 16]", "movups xmm6, [{s} + 32]", "movups xmm7, [{s} + 48]",
"movups xmm8, [{s} + {n} - 64]", "movups xmm9, [{s} + {n} - 48]",
"movups xmm10, [{s} + {n} - 32]", "movups xmm11, [{s} + {n} - 16]",
"2:",
"cmp {p}, 64",
"jbe 3f",
"sub {p}, 64",
"movups xmm0, [{s} + {p}]", "movups xmm1, [{s} + {p} + 16]",
"movups xmm2, [{s} + {p} + 32]", "movups xmm3, [{s} + {p} + 48]",
"movaps [{d} + {p}], xmm0", "movaps [{d} + {p} + 16], xmm1",
"movaps [{d} + {p} + 32], xmm2", "movaps [{d} + {p} + 48], xmm3",
"jmp 2b",
"3:",
"movups [{d} + {n} - 64], xmm8", "movups [{d} + {n} - 48], xmm9",
"movups [{d} + {n} - 32], xmm10", "movups [{d} + {n} - 16], xmm11",
"movups [{d}], xmm4", "movups [{d} + 16], xmm5", "movups [{d} + 32], xmm6", "movups [{d} + 48], xmm7",
s = in(reg) src,
d = in(reg) dest,
n = in(reg) n,
p = inout(reg) aligned_end(dest, n, 16) => _,
out("xmm0") _, out("xmm1") _, out("xmm2") _, out("xmm3") _,
out("xmm4") _, out("xmm5") _, out("xmm6") _, out("xmm7") _,
out("xmm8") _, out("xmm9") _, out("xmm10") _, out("xmm11") _,
options(nostack),
);
}
}

/// [`copy_down_sse2`] 32 bytes a load and store, half the stores: a CPU
/// retiring one store a cycle copies at twice the rate. `vzeroupper` ends it,
/// so SSE code after it pays no transition (56305 rev. 3.02, §2.11.6).
///
/// # Safety
/// As [`copy_down_sse2`], on a CPU [`has_avx`] answers for.
#[target_feature(enable = "avx")]
pub(crate) unsafe fn copy_down_avx(dest: *mut u8, src: *const u8, n: usize) {
unsafe {
core::arch::asm!(
"vmovdqu ymm2, [{s}]", "vmovdqu ymm3, [{s} + 32]",
"vmovdqu ymm4, [{s} + {n} - 64]", "vmovdqu ymm5, [{s} + {n} - 32]",
"2:",
"cmp {p}, 64",
"jbe 3f",
"sub {p}, 64",
"vmovdqu ymm0, [{s} + {p}]", "vmovdqu ymm1, [{s} + {p} + 32]",
"vmovdqa [{d} + {p}], ymm0", "vmovdqa [{d} + {p} + 32], ymm1",
"jmp 2b",
"3:",
"vmovdqu [{d} + {n} - 64], ymm4", "vmovdqu [{d} + {n} - 32], ymm5",
"vmovdqu [{d}], ymm2", "vmovdqu [{d} + 32], ymm3",
"vzeroupper",
s = in(reg) src,
d = in(reg) dest,
n = in(reg) n,
p = inout(reg) aligned_end(dest, n, 32) => _,
// `vzeroupper` clears every YMM register's upper half.
out("ymm0") _, out("ymm1") _, out("ymm2") _, out("ymm3") _,
out("ymm4") _, out("ymm5") _, out("ymm6") _, out("ymm7") _,
out("ymm8") _, out("ymm9") _, out("ymm10") _, out("ymm11") _,
out("ymm12") _, out("ymm13") _, out("ymm14") _, out("ymm15") _,
options(nostack),
);
}
}

/// Copy `n` bytes from `src` to `dest`, highest address first: the order a
/// `dest` overlapping `src` from above needs.
pub(crate) unsafe fn copy_backward(dest: *mut u8, src: *const u8, n: usize) {
Expand Down
Loading