SEP 38: performance tracking — complete runs, about an hour per machine

AGENTS.md

@@ -53,7 +53,7 @@ rather than style, ask rather than guess.
 | Slow lane | `uv run poe test-slow` |
 | Coverage lane (100 % required) | `uv run poe cov` (kernel bodies interpreted first, remaining suite compiled) |
 | GPU tests | `uv run pytest -m gpu` (needs a GPU build of solprop-native and a device) |
-| Benchmarks | `uv run poe bench` (on an otherwise idle machine) |
+| Benchmarks | `uv run poe bench` plans; `uv run python -m benchmarks track` runs (on an otherwise idle machine) |
 | Build docs | `uv run poe docs` (never executes the worked examples) |
 | Run the worked examples | `uv run poe docs-examples [plot_NN_name]` (slow; commit `docs/auto_examples/`) |
 

CHANGELOG.md

@@ -146,7 +146,7 @@ The first public release. It contains:
     envelope grids.
   - Rotational and vibrational Raman responses, scalar and vector.
   - Quantum shot noise, on by default
-    ([SEP 37](https://solprop.readthedocs.io/en/latest/dev/seps/sep-0037-noise-and-stepper-defaults.html)):
+    ([SEP 37](https://solprop.readthedocs.io/en/latest/dev/seps/archive/sep-0037-noise-and-stepper-defaults.html)):
     one photon per mode with a random phase, added to the input field
     (`noise="input"`) and recorded with its seed and with how the model was chosen
     (`RunMetadata.noise_resolution`, output schema 26). `noise=None` switches it off,
@@ -314,6 +314,15 @@ The first public release. It contains:
   `"none"` shows nothing. Plane stores keep it as the `run_summary` annotation
   (`solprop.driver.summary`; a reporter receives it through the optional
   `SummarisingReporter.summary` call).
+- **Performance tracking** for developers
+  ([SEP 38](https://solprop.readthedocs.io/en/latest/dev/seps/archive/sep-0038-performance-tracking.html)):
+  `python -m benchmarks track` times complete runs of about a dozen representative
+  workloads, each with its interface's default integrator and tolerances, at one
+  thread and the production width in about an hour per machine, and records them in
+  `benchmarks/history/` by hardware; `--device` runs the same cases on a native GPU
+  build. It replaces SEP 30's standard suite, whose
+  reference qualification, native coverage and export are removed; `ab` compares two
+  builds on the same worker.
 - **Documentation.** An introduction with executable lessons, a user guide, a
   long-form primer on the physics and numerics, focused tutorials and worked examples,
   the API reference, and a developer's guide with the design records (SEPs and ADRs).

benchmarks/README.md

@@ -1,18 +1,19 @@
 # Benchmarks
 
-The standard benchmark suite
-([SEP 30](../docs/dev/seps/archive/sep-0030-standard-benchmark-suite.md)); its manual is
+Release performance tracking
+([SEP 38](../docs/dev/seps/archive/sep-0038-performance-tracking.md)); its manual is
 `docs/dev/benchmarking.md`. Run it on a quiet machine:
 
 ```bash
-uv run poe bench                                   # core set, quick mode, native CPU
-uv run poe bench run --mode standard --backend numba_cpu
-uv run python -m benchmarks compare RUN_A RUN_B
+uv run poe bench                                     # the plan and its estimate
+uv run python -m benchmarks track --append-history   # about an hour; records history/
+uv run python -m benchmarks history
 uv run python -m benchmarks ab --case CASE --baseline-python OLD/bin/python  # fast A/B
 ```
 
-Runs and machine profiles go to `.benchmarks/suite/`, which git ignores; the benchmark rules
-and indicative results are in `docs/dev/benchmarks.md`. What has already been tried for CPU
-performance is in `docs/dev/cpu-performance.md`. Earlier campaign harnesses, the retired
-version-2 suite and their raw data are kept with the project's development records, not
-in this repository.
+`history/` holds one CSV per hardware set, a few rows per release. Run bundles go to
+`.benchmarks/`, which git ignores; the benchmark rules and indicative results are in
+`docs/dev/benchmarks.md`. What has already been tried for CPU performance is in
+`docs/dev/cpu-performance.md`. Earlier campaign harnesses, the retired standard suite's
+baselines and their raw data are kept with the project's development records, not in
+this repository.

benchmarks/__init__.py

@@ -2,10 +2,10 @@
 #
 # SPDX-License-Identifier: GPL-3.0-or-later
 
-"""Collect bounded, independently qualified benchmarks (SEP 30).
+"""Track complete runs per hardware, and compare two builds (SEP 38).
 
-Importing the registry or planner never imports a scientific library. Workers
-load those libraries only after their thread environment has been established.
+Importing the registry or the controller never imports a scientific library.
+Workers load those libraries only after their thread environment is set.
 """
 
 __all__: list[str] = []