Skip to content

Mutation testing on armv7 - #538

Merged
kixelated merged 10 commits into
mainfrom
quest/m3/confidence/mutants-armv7
Nov 12, 2026
Merged

kixelated merged 10 commits into
mainfrom
quest/m3/confidence/mutants-armv7

Conversation

@kixelated

@kixelated kixelated commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m3/confidence/mutants-armv7.md: admin/mutants runs on armv7 under qemu-arm, every armv7 survivor is killed by a test or recorded with its reason, and mutants.yml runs armv7 nightly.

Summary

  • admin/mutants --target armv7-unknown-linux-gnueabihf, in nix develop .#armv7, cross-builds and runs each test binary under qemu-arm. Its rows are sh/armv7.sh's tiers: qemu max (NEON and the crypto extension), a Cortex-A15 (NEON alone), and a Cortex-A15 with NEON denied. Each row sets RIND_TEST_ARM_NEON/RIND_TEST_ARM_CRYPTO_EXT, so a mutant that breaks CPU detection fails detects_the_expected_cpu.
  • armv7 mutants build with the release profile plus debug assertions and overflow checks, as sh/armv7.sh tests: the debug baseline under qemu took 703s, release 79s.
  • mutants.yml gains an armv7 matrix entry (-p rind --features wipe), and its artifacts are named by architecture (x86_64, aarch64, macos, armv7) so admin/mutants --survivors combines them all; the avx512 job now downloads mutants-x86_64.
  • mid/mod.rs swaps the portable aes_gcm/ed25519 modules with #[cfg] #[path] rather than cfg_attr(path), which cargo-mutants does not follow, so it mutates mid/portable.
  • New tests: the NEON Poly1305's freeze on h in [p, 2^130), which random inputs reach with probability about 2^-128; and zero_bytes's test moves from low/tests.rs (x86_64 and aarch64 only) to low/generic/zeroise.rs, so it runs on armv7 and the portable backend too.
  • .cargo/mutants.toml: records the equivalent survivors in the existing groups (ORs of disjoint bits in the bitsliced AES, Keccak interleaving, and GCM lengths; NEON-or-scalar path choices; freeze's limbs that are zero whenever kept; inner key types whose wrapper wipes the whole value), widens the ghash_nohw entries to its portable/../generic path, and drops the five mid/p384.rs "armv7 only" exclusions, which armv7 now catches.

Campaign

All 1,856 mutants in the files only armv7 reaches (low/arm, low/portable, mid/portable, low/generic/order.rs) and the shared files with arm or portable cfg blocks (low/generic/bignum.rs, mid/p384.rs, low/cpu.rs), on armv7 under qemu: 1,612 caught, 21 timeouts, 36 unviable, 187 missed. x86_64 catches the shared files' misses except those listed under Follow-ups. After this PR, re-running every miss on armv7 (with --features wipe) leaves none that are not recorded.

Public API

None. Test and CI only.

Validation

On the merge with main (just check on b3331bd; the armv7 and Miri runs on d923e66, the merge before moq-dev/rind#603, whose test-only and x86_64 changes CI's armv7 rows cover):

  • quest gate nix develop --command just check: passes.
  • quest gate nix develop .#armv7 --command sh/armv7.sh (clippy and all three tiers): passes.
  • Miri, sh/miri.sh's flags on x86_64 and s390x, filtered to the tests this PR adds or reaches (low::generic::zeroise, mid::aes_gcm, the ed25519 tests over mid/portable): all pass. The miri workflow's jobs hit the 6-hour limit here, as on main's nightly (run 42917952991) and every other PR that runs it: they hang on moq-dev/rind#555's acvp_* and cavp_ecdh vector tests, which sh/miri.sh does not skip, and low::guard_pages (moq-dev/rind#568) fails under Miri, which cannot mmap with PROT_NONE. Neither is this PR's.
  • The ctgrind workflow's build (--release) and binsec failures were main's, fixed by moq-dev/rind#607 and moq-dev/rind#604; other PRs with main merged pass them.
  • Earlier: admin/mutants --target armv7-unknown-linux-gnueabihf -p rind --features wipe --file rind/src/low/arm/poly1305_arm.rs --re freeze: baseline passes on every row, and all 42 mutants are caught. shellcheck admin/mutants and quest check: clean.

Follow-ups

  • quest/m2/confidence/mutants-wipe.md: mutants with --features rind/wipe on x86_64 and aarch64 too. Without it every Drop that wipes a key is a survivor there, such as mid/ed25519.rs's SigningKey::drop. No open decisions.

Decisions

  • Follow-ups, picked by the maintainer in the /quest-complete round: ✅ keep the wipe mutants plan, recorded as its own quest (rejected: drop it). ✅ Drop low/cpu.rs's DIT and verify_cpu_features survivors (now covered by main's macOS DIT campaign, and by CPUs that lack a required feature, which no host is) and bignum.rs's portable-only survivors, which quest/m2/confidence/portable-fuzz-mutants.md covers.
  • Merge order: moq-dev/rind#513 has landed, so the armv7 wipe baseline passes and no ordering remains.

Merge with main

main gained a macOS DIT campaign (moq-dev/rind#578), an AVX-512 campaign under SDE (moq-dev/rind#570), and PR-triggered runs (moq-dev/rind#575). admin/mutants keeps all three beside --target, selecting the campaign by ${target:-$(uname -sm)}. main also made zero_bytes an unsafe fn (moq-dev/rind#583), so the moved test calls it in unsafe. This PR finishes the last quest under quest/m3/confidence/, so it deletes that README and the m2 epic's pointer to it, plus TESTING.md's dead link to the finished quest/m3/confidence/vectors.md. .cargo/mutants.toml drops the DIT and AVX-512 entries main removed and takes main's shifted line numbers.

A second merge took main's split of the CPUs between parallel jobs' tests (moq-dev/rind#574) in admin/mutants, keeping this PR's --target in the runner it exports.

A third took moq-dev/rind#603, which deleted the mutants-edge-cases.md entries in .cargo/mutants.toml beside the p384 "armv7 only" entries this PR deletes; both stay deleted.

Models

Size: M
Implementer: unknown (starter session, first commit)
Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, M)
Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, M)
Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, M)

(Written by Claude Opus 5.5 with medium effort)

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, medium effort
  • Session: 01a1cbe4-aa98-7623-bd81-1e271956ee61, round 2
  • Head: b3331bd
  • Base: origin/main at f5f82bc
  • Score: 9/10, The update merges the base changes and preserves the previously reviewed branch changes without introducing actionable defects; builds and tests were not run as instructed.
  • Findings: none

kixelated and others added 5 commits November 9, 2026 16:56
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows pick qemu's CPU as sh/armv7.sh's tiers do and assert the detected
features, the NEON Poly1305's freeze is tested on h >= p, zero_bytes is
tested on every backend, and the equivalent survivors are recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 10, 2026 18:17
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 25 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6029ae9f-fe57-40d0-9926-e5f1b13f7a64
Commits

Reviewing files that changed from the base of the PR and between f5f82bc and b3331bd.

Files selected for processing: 14

  • .cargo/mutants.toml
  • .github/workflows/mutants.yml
  • TESTING.md
  • admin/mutants
  • flake.nix
  • quest/m2/confidence/README.md
  • quest/m2/confidence/mutants-wipe.md
  • quest/m2/confidence/portable-fuzz-mutants.md
  • quest/m3/confidence/README.md
  • quest/m3/confidence/mutants-armv7.md
  • rind/src/low/arm/poly1305_arm.rs
  • rind/src/low/generic/zeroise.rs
  • rind/src/low/tests.rs
  • rind/src/mid/mod.rs
Pre-merge checks: Inconclusive 2

Failed Failed checks: 2 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Description check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

The armv7 mutation-testing campaign, its CI wiring, and the tests that kill the boundary mutants it needed.

  • admin/mutants --target armv7-unknown-linux-gnueabihf: the new armv7 case cross-builds and runs each test binary under qemu-arm; its base run and two rows mirror sh/armv7.sh's three tiers exactly (QEMU_CPU=max with NEON and the crypto extension, a Cortex‑A15 with NEON alone, and one with NEON denied via RIND_CPU_DISABLE_neon), and the release-plus-debug-assertions profile matches what sh/armv7.sh exports. The native campaigns still behave: base/runner stay empty for them, and the --run handler keeps exiting before the rows on --list. shellcheck is clean.
  • mutants.yml: the include-only matrix and per-arch artifact names are right — the rename to matrix.name is required, since x86_64 and armv7 both run on ubuntu-latest and would collide under the old matrix.os names. nix develop ".#default" and .#armv7 both exist in flake.nix, the armv7 shell now carries cargo-mutants/cargo-nextest, and the survivors job's mutants/*/ glob picks up the new names.
  • mid/mod.rs: the #[cfg] + #[path] swap for the portable aes_gcm/ed25519 modules compiles on every backend — I checked the arm backend for armv7-unknown-linux-gnueabihf and the portable backend forced on both x86_64 and armv7 — and adding mid/portable to the x86_64/aarch64 skip lists correctly keeps those files from generating unreachable mutants now that cargo-mutants follows the #[path].
  • poly1305_arm.rs freeze test: I verified all five assertions against the 26-bit limb arithmetic by hand (h = p → 0, h = p+4 → 4, p−1 unchanged, the carry chain through [MAX+1, MAX, MAX, MAX, 0], and 2^130 → 5). It kills the line‑98 y0 ^= x0 mutant, and the recorded limbs‑1–4 exclusions are genuinely equivalent: those limbs are all-MAX exactly when the subtraction path runs, so y_i & ~x_i collapses to the correct value there. The Poly1305Neon::finish exclusion also holds — the 2^128 bit sits above the 16-byte tag.
  • zero_bytes test move: low/tests.rs is gated to x86_64/aarch64, so the move into low/generic/zeroise.rs (which already imports crate::low::zero_bytes) makes it run on armv7 and portable too; it passes natively (ran it).
  • .cargo/mutants.toml: every pinned line matches the current sources (low/arm/cpu.rs 38/40/70/83–88/96/104, poly1305_arm.rs 99–102), and no existing pin broke — mid/mod.rs keeps its line count and both new test modules append at file ends. The (83|85|86|87|88) &→|/^ exclusions are sound: the && chain means a single one only differs on a CPU with a partial crypto extension, which no qemu tier models. The widened ghash_nohw regex matches the path armv7 actually reaches the file by (low/portable/../generic), and the five dropped mid/p384.rs entries are arm|portable-cfg'd functions armv7 now compiles and tests — multiply_matches_double_and_add exercises them directly.
  • Quest docs: the quest file is closed and removed from the m3 README, portable-fuzz-mutants.md now records the armv7 campaign's remaining gaps, and no other doc still claims only two campaigns.

ℹ️ Sequencing and evidence

#513 is still open, so until it lands the nightly armv7 job fails its baseline (zeroing.rs's cipher test on tiers without the crypto extension) — the PR body already documents this and asks for that merge order, so this is only the reminder to hold it. Separately, there is no dispatched mutants.yml run on this branch, so the campaign numbers rest on the local runs; the first nightly after merge is where the recorded exclusions get their CI confirmation.

Technical details
# Sequencing and CI evidence for the armv7 campaign

## Affected sites
- .github/workflows/mutants.yml (armv7 matrix entry) — runs `-p rind --features wipe`,
  whose `rind/tests/zeroing.rs` baseline fails on non-crypto-ext armv7 tiers until #513 lands.
- .cargo/mutants.toml — the 187 recorded misses come from local campaign runs; no
  dispatched run artifacts exist for the branch to audit.

## Required outcome
- None before merge: the ordering constraint is the PR's own documented decision.
  If #513 lands second, expect nightly mutants alerts until it does.

## Open questions for the human (optional)
- Consider dispatching `mutants.yml` on this branch after #513 merges (or on a stack
  containing both) to leave CI evidence of the armv7 job wiring and the exclusions.

Pullfrog  | View workflow run | Using glm-5.3 | 𝕏

Keeps main's macOS DIT and AVX-512 campaigns beside the armv7 one, moves the zero_bytes test to call the now-unsafe fn, deletes the finished m3 confidence README, and records the kept wipe follow-up as quest/m2/confidence/mutants-wipe.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-11, /quest-run): ✅ take over with /quest-iterate and /quest-merge once green, keeping the follow-up quest/m2/confidence/mutants-wipe.md (rejected: iterate then ask again, leave it).

(Written by Claude Opus 5.5 with high)

…utants-armv7

# Conflicts:
#	.cargo/mutants.toml
@kixelated
kixelated added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit f6311c3 Nov 12, 2026
44 of 46 checks passed
@kixelated
kixelated deleted the quest/m3/confidence/mutants-armv7 branch November 12, 2026 00:25
@moq-bot moq-bot Bot mentioned this pull request Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant