[Japabu]

Problem

sdk/libc's x86-64 memcpy and memmove's forward branch were one rep movsb at every length. On the AMD bench (Zen 2, no FSRM) mem.memcpy.64 read 0.61 of Linux's and mem.memcpy.256 0.90 (2026-11-11 scoreboard, issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md). In the diagnosis's candidate copies on that bench, rep movsb took 15.8 ns for 64 bytes against 5.4 by a 16-byte vector loop. On the T14 (FSRM) rep movsb was fastest, or within half a nanosecond of it, at every size (evidence).

Change

• A CPU that declares fast short rep movsb keeps one rep movsb for every copy. FSRM is CPUID.(EAX=7,ECX=0):EDX[4], the vendor's statement that a short rep movsb starts fast.
• Elsewhere a copy of up to 256 bytes is moves from each end that overlap in the middle:
  • below 16 bytes, two moves of 1, 2, 4 or 8 bytes;
  • from 16 bytes, one to eight unaligned SSE2 loads from each end.
  • Every load comes before any store, so any overlap copies right.
  • The moves are spelled out by a macro, not looped, so that the compiler cannot lower them to a call to memcpy.
• Longer copies stay one rep movsb.
• The CPU's answer is one word: the length below which a copy is moves. It is 0 with FSRM, 257 without, and usize::MAX until the process's first copy. That first copy asks CPUID, out of line, and then copies. Every later copy reads the word and compares its length with it; on a CPU without FSRM a short copy then also tests the word for usize::MAX.
• copy_forward now answers dest, on both architectures, so that the first copy is a tail call and memcpy's prologue keeps nothing across it.

With FSRM, memcpy runs three instructions more before rep movsb than main's did: a load of the word, the compare and the branch. It runs one more mov before it.

The kernel's copies are not changed here. The rule for the kernel side, which is integer-only, uses the same predicate, FSRM = CPUID.(EAX=7,ECX=0):EDX[4] (after checking CPUID.0:EAX >= 7), and the same 256-byte bound.

Commits

• 23d1129: the short copy.
• b6c8d00 and a7e5b1c: the guest build's disassembly of 23d1129 showed an out-of-line call for the FSRM question on every short copy. Its prologue saved three registers even on an FSRM CPU. These two commits take that cost back out of the hot path.

Checks

• Host checks at a7e5b1c, each exit 0 (logs):
  • cargo test -p toyos-libc-copies --target x86_64-apple-darwin, x86-64 under Rosetta: 105 passed.
  • cargo test -p toyos-libc-copies on aarch64: 105 passed.
  • cargo clippy -p toyos-libc-copies --all-targets, for x86-64 and for aarch64, with the adopted lints and -D warnings.
  • The aarch64 module changed only in answering dest. The stagings below also built the guest libc for both architectures.
• What the tests cover:
  • A new test calls the short copy directly, whatever the host CPU: every length 0 to 256, every overlap distance either way, at four alignments, against copy_within.
  • The existing tests cover copy_forward at every length to 300 and offset to 20, and now check that it answers dest.
  • Rosetta declares no FSRM, so under it the existing tests take the short path up to 256 bytes, rep movsb above, and the first copy's question.
  • Negative controls, against the head's module under Rosetta (evidence): a store before the last load, an arm one byte too wide, and a threshold past what the short copy covers. Each was caught, and the restored code passed.
• Guest code (disassembly), memcpy and memmove as each staging's osbench linked them:
  • no call to memcpy inside either;
  • the 129-256 byte arm keeps its sixteen loads in xmm0-15 without a spill;
  • the first copy's CPUID sits in a cold function reached by a tail jump.
• Guest run 43092282212 at the head 8267b8b, x86-64 and AArch64 together: virt_smp, std_threads_count_the_cpus_and_give_back_ended_threads and a_user_copy_faults_in_mid_copy_or_ends_as_bad_address. Every userland program in those boots copies through the new memcpy. The runner boots x86-64 under KVM with -cpu host, so which path it takes depends on the runner's CPU, and the host tests check both paths. It is green, at 8267b8b.
• Rebased onto main a7ca8ad as 61b3248. git range-diff shows the three libc commits and the review commit unchanged (=). Only the owed-on-metal record commit changed: it keeps An unmap walks no futex bucket while no thread of its space is in a futex wait #1223's candidates paragraph and this PR's owed list, both in the small-memcpy record. Main's libc changes since the tested base are in posix_io.rs and errcode.rs, outside the copies.
• Owed on metal, in issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md. The benches are down, so it is not a landing condition: an A/B on both benches, main from before and after, of osbench_mem and of osbench copyat. copyat is on the measurement branches, not on main. Main after must read:
  • on the AMD bench, mem.memcpy.64 and .256 faster;
  • on the T14, no mem.memcpy.* row slower;
  • on both, mem.memcpy.262144 and the eight copyat.place.* rows, judged together by each boot's mean, no slower. An AMD spread over 5% is inconclusive and asks for more rotated blocks.
• The speed-up this PR exists for is not yet measured on metal. The diagnosis timed candidate loops, not this code. The staged A/B (memperf-copy-ab-head3 bff283c against memperf-copy-ab-base 915c248) never booted.

Review r1 (0071258)

• Records:
  • issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md now says how a copy chooses, and what is owed on metal.
  • The memset record's cross-reference no longer points memcpy's rep movsb at the same start-up.
• Comment on the word: says "the rest" for every copy it does not send to moves, and cites SDM Vol. 1 Table 21-21 and the Optimization Reference Manual §3.7.6.1.
• Comments and records only, so the staged A/B images (bff283c) were still this code.
• Not taken, so that the build the A/B measures, code placement included, stays the one that lands:
  • Storing 1 where FSRM lacks fast zero-length rep movsb (CPUID.07H.01H:EAX[10]): the zero-length rep movsb cost is unmeasured on the T14.
  • Taking short backward memmoves through the short copy: a follow-up with its own A/B.
• Checks at 0071258: cargo test -p toyos-libc-copies --target x86_64-apple-darwin (105 passed) and its x86-64 clippy run. Each exited 0 (logs).

Not run, and why

• Full cargo run -- --ci host: under the owner's testing policy only the touched crate's tests and lints run; the merge queue runs the rest.
• Speed: guest timing is not evidence, so speed is measured only on metal.

🤖 Generated with Claude Code

[Japabu]

Review of a7e5b1c

What I read: the three commits and the full diff, sdk/libc/src/arch/x86_64.rs, aarch64.rs, mod.rs and memory.rs, the copies crate's tests, and the evidence at a33f124b: disassembly, host checks, and mutations at the head. Also memperf-copy-ab-head3 against memperf-copy-ab-base, #1133's closing record, both records about short copies, and Intel's SDM and Optimization Reference Manual.

Architecture

libc owns userland's memcpy, and choosing a path by a capability the vendor declares is ordinary hardware support. CPUs with FSRM keep main's instruction. The selection, the intrinsics and the asm all stay in the x86-64 architecture module. Production code grows by about 90 lines in that one module, and the tests by 27. The CPUID question is asked lazily, on the first copy, which keeps it self-contained: Rust programs reach libc's memcpy without running libc's _start, so an init hook would need two entry points.

Correctness for overlapping and unaligned buffers

• Every load comes before the first store.

  • ends reads both values, then writes both.
  • vectors! loads both tuples before any store. In the head's disassembly the compiler kept that order; it even moved the four shared head loads above the branch between the 65–128 and 129–256 arms.

  So an overlap in either direction copies right.

• Coverage without overreading. For 16k ≤ n ≤ 32k, [0, 16k) ∪ [n−16k, n) = [0, n), and the scalar arms work the same way. No byte outside [src, src+n) is read, so a copy never reads across into the next page.

• Unaligned buffers. Only loadu/storeu and read_unaligned/write_unaligned are used.

• Dispatch. n < MOVES_BELOW ≤ 257 means n ≤ 256 reaches copy_short. The _ arm (x86_64.rs:101) would accept any larger n but covers only 256 bytes; it relies on its one caller. The tests catch a wrong threshold (mutation 3).

• rep movsb path. It is unchanged and runs forward with DF = 0 by the ABI. memmove sends it only copies where dest ≤ src or the buffers are disjoint.

• Return value. dest survives the asm (=> _), and the disassembly keeps it in rax.

• Tests. The new test runs every length 0–256, every distance 0..n+1 in both directions and four offsets; its largest index is 784 of 788. Under Rosetta, which declares no FSRM, the existing 0–300 tests take the short path. 105 tests pass on x86-64 under Rosetta and 105 on aarch64 at the head. All three mutations were caught.

The FSRM predicate

__cpuid(0).eax >= 7 && __cpuid_count(7, 0).edx & (1 << 4) != 0 is correct.

• Intel SDM, combined volumes, 325462-093US, September 2026:
  • Vol. 1, Table 21-21, "Leaf 07H.00H Structured Extended Feature Flags Returned in EDX": EDX[4] FAST_SHORT_REP_MOVSB, "If 1, supports Fast Short REP MOVSB."
  • Vol. 1 §21.1.3: "If a value entered for CPUID.EAX is higher than the maximum input value … then the data for the highest basic information leaf is returned." So the check that EAX ≥ 7 is required.
  • In this edition, Vol. 2A's CPUID page (3-203) refers the leaf data to Vol. 1 Chapter 21.
• Intel Optimization Reference Manual Vol. 1, 248966-051US, October 2026, §3.7.6.1 "Fast Short REP MOVSB": from Ice Lake client, short REP MOVSB is enhanced for lengths of 1 to 128 bytes, enumerated by CPUID.(EAX=7H,ECX=0H):EDX[bit 4]. This is the vendor's statement that "a short one starts fast".
• AMD: APM Vol. 3 (24594), Appendix E, CPUID Fn0000_0007_EDX_x0 bit 4, FSRM. This is from my knowledge of the document: amd.com did not answer from this session, so I could not read it again. The AMD bench (Zen 2) reading no FSRM matches its slow measured short rep movsb.

The once-only CPUID read is thread-safe

• Threads racing on a first copy each compute the answer and store the same word.
• Relaxed is enough, because the word publishes nothing else. Every value a thread can read (UNASKED, 0, 257) gives a correct copy; the choice affects only speed. Even cores that answered differently could not make a copy wrong.
• There is no lock, so a signal handler can safely call memcpy during the first copy.
• The static is reached RIP-relative and first_copy by a direct call, so a copy made before the image relocates itself works too.

Code placement (the risk that closed #1133)

#1133's head dropped to 0.80–0.94 of base on the AMD 256 KiB placement rows in 5 of 5 blocks. Base with memcpy moved to the head's address did not drop: "the cost follows the routed code". This PR also adds code on the ≥257-byte path before the same rep movsb:

• a load of the word, a compare, a branch and one mov;
• rep movsb moves from memcpy+0x0a to +0x39;
• memcpy grows by about 0x1c0 bytes.

The risk is real. The A/B carries the right rows: copyat.place.p0–p7, mem.memcpy.262144, and copyat.code, whose 64-byte window still finds the rep movsb at +0x39.

The rule's power here is limited. With two boots per arm, "both head boots beyond both base boots" happens by chance in a third of rows, counting both directions. The 2% gate filters only rows steadier than that. The AMD 256 KiB row is bimodal from boot to boot (35.2–41.6 GB/s), so two boots per arm cannot rule out a drop of 6–20%. Before the results are read:

• Judge the eight placement rows together, per boot. libc: a forward copy a short way above its source modulo 4 KiB copies down #1133's drop moved every placement.
• Treat AMD arms whose ranges overlap by more than that size of drop as inconclusive, not as "no change", and run libc: a forward copy a short way above its source modulo 4 KiB copies down #1133-style rotated blocks before landing.

Two more points on the rule:

• A head boot that does not pass must be investigated before it is excluded, since every userland program's memcpy changed.
• The rule should say what happens to an arm left with one valid boot: no verdict, and the arm is run again.

Findings

BLOCKER: none in the code.

Required with the metal result:

• issues/a-small-memcpy-on-the-amd-bench-runs-at-as-little-as-six-tenths-of-linuxs.md:10-11 says libc's copy_forward "is one rep movsb at every length". That is false once this lands. The PR must update this record with the new selection and the A/B.
• It must also update the cross-reference in issues/a-short-memset-on-x86-64-costs-a-rep-stosb-start-up.md:27, "memcpy's rep movsb may share it".

The PR description does not list either update.

SUGGESTION

• x86_64.rs:50-55, first_copy — skip rep movsb for zero-length copies where it is slow. On an FSRM CPU without fast zero-length REP MOVSB, store 1 instead of 0; a zero-length memcpy then skips the start-up, with no added compare. That feature is CPUID.07H.01H:EAX[10] (SDM Vol. 1, leaf 07H.01H EAX; ORM §3.8.1). From Golden Cove on, a zero-length REP MOVSB takes as long as one of 1–128 bytes. The T14 predates that (issues/features-no-proving-machine-is-known-to-offer.md), and its zero-length rep stosb read about 50 ns (issues/a-short-memset-on-x86-64-costs-a-rep-stosb-start-up.md). It is unmeasured for movsb.
• memory.rs:56-58 — use the short copy for short backward memmoves. When dest overlaps src from above, memmove still runs std; rep movsb; cld at every length, and with DF = 1 no fast-string feature applies. copy_short is correct for any overlap, so copies of 256 bytes or less could take it on every CPU.
• Wording in the comments:
  • x86_64.rs:50 says "longer ones one rep movsb", but a copy exactly the word's length also takes rep movsb; "the rest" is accurate.
  • x86_64.rs:52 could cite SDM Vol. 1, Table 21-21 for the bit, and ORM §3.7.6.1 for "starts fast".
• PR description (the orchestrator's to correct): "Every later copy reads the word and makes one compare." On a CPU without FSRM a short copy also compares the word with UNASKED (cmp rcx,-1; je).

What metal must show

• AMD: mem.memcpy.64 and .256 are MOVED faster.
• T14: no REGRESSION in any mem.memcpy.* row. The FSRM path now runs four more instructions, and a 64-byte copy there takes about 14 cycles.
• Both benches: no REGRESSION in copyat.place.* or mem.memcpy.262144, judged as above. An inconclusive AMD result means more blocks are run.
• All head boots passed.
• Branches: the measurement commit is the same hunk on both arms. memperf-copy-ab-head3 is the head plus that commit, and memperf-copy-ab-base is f79df95 plus it.

Guest rows

None. The metal boots run every program's libc on a CPU without FSRM (AMD) and on one with it (T14), which no QEMU row adds to. The aarch64 change is only the return value, and the aarch64 host tests cover it.

LAND, pending metal and the record update above.

[Japabu]

Re-review of 0071258

Scope: whether round 1's findings are resolved, whether any code changed since a7e5b1c, and whether the records describe the path choice truthfully.

No code changed since a7e5b1c

git diff a7e5b1cb5 0071258b4 touches two records and the doc comment on MOVES_BELOW (x86_64.rs:50-55). copy_forward, first_copy, copy_short, ends, vectors! and the asm are identical. So the queued A/B at bff283c still measures what lands.

The records are true

• a-small-memcpy…md:9-15.
  • On a CPU that declares FSRM, MOVES_BELOW becomes 0, so every copy is one rep movsb; the first copy too, through first_copy → copy_forward.
  • Elsewhere, n ≤ 256 goes to copy_short, moves from each end, and longer copies go to rep movsb.
  • The T14 (i5-1135G7, Tiger Lake) has FSRM, which Intel's Optimization Reference Manual §3.7.6.1 gives from Ice Lake client. The AMD bench (Zen 2) lacks it.
  • "Until 2026-11 … which the tables below measure" is right: both tables measured the old routine (069cbbe, ad5db5a).
• The memset record's exit no longer points memcpy at the same start-up, and says how it selects.
• The word's comment says "the rest" and cites SDM Vol. 1 Table 21-21 and the Optimization Reference Manual §3.7.6.1.

Both required record updates from round 1 are done. Adding the A/B result to the memcpy record remains a landing condition.

Findings

BLOCKER: none.

SUGGESTION

• Short backward memmoves still run std; rep movsb; cld at every length. The PR calls this "a follow-up", but no record holds it. One sentence in the memcpy record, written with the A/B, keeps it from being lost.

Evidence

• Checks at 0071258: the x86-64 tests (105) and clippy, each exit 0 (f7d1bcf6). That suffices for a change to comments and records.

Landing conditions

1. The A/B by rule.txt as amended:
   • AMD: mem.memcpy.64 and .256 MOVED faster.
   • T14: no REGRESSION in any mem.memcpy.* row.
   • Both benches: no REGRESSION in the placement value (copyat.place.p0–p7 judged together, per boot) or in mem.memcpy.262144. On the AMD bench, a spread over 5% is INCONCLUSIVE, and more rotated blocks are run.
   • Every head boot passed, or its failure was investigated and shown to have a cause outside the change.
2. The A/B result goes into the memcpy record.
3. The orchestrator decides whether the boot-before-flash rule covers libc. Every program's memcpy, init's included, runs this code.

LAND, pending metal and the record update.

[Japabu]

Orchestrator: landing under the benches-down rule. Review r2 LAND. Head 61b3248 is the same libc commits rebased onto a7ca8ad (range-diff =; only the record commit changed, keeping both texts). Guest run 43092282212 at 8267b8b: success. The A/B is owed on metal; the path choice follows the CPU's declared FSRM, and the speed-up is not yet measured on metal.