v0.7 pre-release candidate: benchmark report (v0.7-release @ 5f6ee09) — tree fit instruction regression 4-6% #506

What was measured

The v0.7 pre-release candidate (branch v0.7-release, commit 5f6ee09) was benchmarked with smartcore-benches run 40069239471, dispatched with smartcore_ref=v0.7-release.

• iai-callgrind (instruction counts): deterministic, machine-independent. This is the gate (fails above a 4% increase). Baseline: the published 0.6.16 from crates.io, measured with the same toolchain on the same day. Instruction counts do not depend on the runner, so this comparison is exact.
• criterion (wall clock): advisory. See the note below.

Verification: the run log shows Compiling smartcore v0.7.0 (.../smartcore), so the benches compiled the candidate, not the crates.io release. (Earlier runs did not: see "Harness fixes" below.)

Results: instruction counts (iai-callgrind)

Analysis

The regression is isolated to the regression-tree fit path (BaseTree/BaseForest for regressors). Classifier trees are unchanged and are flat. This matches the code that changed in the candidate:

• Use MSE criterion in base tree regressor #492 (@slievens): MSECriterion with a numerically stable improvement formula, plus removal of zero-weight rows. The CHANGELOG documents this as added work per split.
• Add feature importances to DecisionTreeRegressor, RandomForestRegressor and ExtraTreesRegressor #497 (@slievens): feature-importance bookkeeping stored during fit.
• Make BaseTree(Regressor) and BaseForest(Regressor) generic over criterion #502 (@slievens): generics over criterion and leaf output; possible code-generation differences.
• feat: add lazy normalization for Lasso and Elastic Net #480 (@jolars, lazy normalization) and feat: make ndarray std support conditional #472 (@mvanhorn, ndarray std) are neutral in the default configuration (±0.01% or less).

Suggested corrections

In order of preference:

1. Attribute the cost per PR. Bench Use MSE criterion in base tree regressor #492, Add feature importances to DecisionTreeRegressor, RandomForestRegressor and ExtraTreesRegressor #497 and Make BaseTree(Regressor) and BaseForest(Regressor) generic over criterion #502 one at a time against main. The +5.71% is the sum; one PR may carry most of it. Use the lazy/eager comparison from the benches README as a template.
2. Optimize the split loop. Profile the Criterion::improvement path (src/tree/criterion.rs, grow loop in `src/tree/base_tree.rs)). The stable formula costs extra operations per candidate split. Candidates: hoist loop-invariant sums, compute the improvement from running sums instead of per-row statistics, or filter zero-weight rows once before the recursion starts instead of per node.
3. Accept and document. If the cost buys the numerical stability and the feature importances, accept a one-time budget increase for the tree benches and record it in the CHANGELOG. The gate stays at 104% for everything else.
4. Revert the tree commits from the candidate (fallback). If the v0.7 timeline is short and the budget cannot be recovered, remove the regression-tree changes and move them to the next cycle. Because all PRs landed through the single squash cd217dd (v0.7 Release #504), the clean procedure is:

   git checkout -b v0.7-release-r2 origin/main
   git revert cd217dd            # removes tree work, lazy normalization, ndarray bump
   # re-land the neutral parts as a fresh PR:
   #   - #480 lazy normalization (jolars branch feat/lazy-normalization)
   #   - ndarray 0.17.2 requirement
   #   - CHANGELOG [v0.7] entries for the kept changes

   Then re-point v0.7-release and re-run the benches.

Criterion note (advisory)

The wall-clock numbers of this run are 8%–45% below the nightly baseline for all benchmarks, including unchanged paths. This is runner variance, not a real speedup. Do not use cross-run criterion numbers for decisions; use the instruction counts above. Within-run comparisons on the same runner are still useful.

Harness fixes (smartcore-benches)

• Before this report, a smartcore_ref run silently benched the crates.io release (patch was not used in the crate graph), because smartcore = \"0.6\" rejected the 0.7.0 patch. Fixed in smartcore-benches#12 (the repoint step now relaxes the version requirement to the ref's major.minor).
• The gh-pages history push failed on ref runs because the repoint step edits the tracked Cargo.toml. Fixed in smartcore-benches#13 (restore before the chart push). This run's results are in the run artifacts; the trend charts do not include it.
• The BENCHES_STATUS_PAT secret is not configured, so status checks cannot post back to smartcore commits. An admin must add it (fine-grained PAT, commit_status:write on smartcorelib/smartcore). The step is now continue-on-error so it cannot fail measurement runs.

Data

• Run: https://github.com/smartcorelib/smartcore-benches/actions/runs/40069239471 (artifact iai-json, logs contain the full criterion output)
• Trend charts: iai-dev · criterion dev
• Candidate ref: v0.7-release = 5f6ee09