[jtravs]

SEP 38 (Final): replace SEP 30's standard benchmark runner with a release-tracking command. SEP 37 (#30) is merged into the branch and the PR targets main; SEP 30 is marked Superseded. Since the draft: a --device mode (run end to end on an NVIDIA A40), the passive-bound threading policy, and threading policies that spell out their waiting so #33's library default cannot change them.

What changes

• python -m benchmarks track times complete runs of 13 representative cases at one thread and the production width. Each configuration runs in its own worker: build, translate, an untimed warm-up over 2 % of the guide, then 3 timed runs through run_problem (the loop of Simulation.run).
• Defaults: cases pass no tolerance or integrator, so they use their interface's defaults. Every noisy launch uses the fixed seed 20261027.
• Stepper override: --stepper default,multistep,multistep-interaction,rk selects steppers for A/B comparisons.
• History: --append-history writes benchmarks/history/<hardware>.csv. The hardware is named by CPU model and core count, never by host name. history summarises the file.
• ab now uses the same worker and gains --length FRACTION.
• Runnable cases: every SEP 30 case stays runnable by name, outside the default selection.
• Removed:
  • standard paired mode, qualification and references, native coverage, export;
  • the batch exercise;
  • their tests (about 5000 lines).
• Documentation: the manual (docs/dev/benchmarking.md), the rules (docs/dev/benchmarks.md), the SEP index, roadmap and changelog.

Calibration

One complete default run on 16 cores of an AMD EPYC 7763 node took about 50 min for 22 configurations. Repeats agree within 0.7 % on most cases. The full table is in the SEP's phase 1 report.

For review

• Q1: if the capillary default becomes RK, should ctc run longer or also at one thread?
• Q3: the 384² Kerr beam costs 134 s per run at 16 threads, about a fifth of the run. Keep it, or track the 192² beam at both widths instead?
• No history rows yet. The first ones wait for the release commit, once SEP 37: shot noise on by default, and an integrator and tolerances for each interface #30's provisional defaults are approved.

Checks

• Benchmark and SEP tests pass locally: 328 passed.
• ruff and basedpyright are clean on benchmarks/.
• The docs build passes with -W.
• CI runs the full gate.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 solprop | 🛠️ Build #35444586 | 📁 Comparing 89faeb1 against latest (c50e002)

  🔍 Preview build  

113 files changed · + 2 added · ± 111 modified

+ Added

• dev/seps/archive/sep-0037-noise-and-stepper-defaults.html
• dev/seps/archive/sep-0038-performance-tracking.html

± Modified

• changelog.html
• genindex.html
• sg_execution_times.html
• dev/backends.html
• dev/benchmarking.html
• dev/benchmarks.html
• dev/cpu-performance.html
• dev/decisions.html
• dev/dependencies.html
• dev/roadmap.html
• and 101 more...

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.