Skip to content

Declare RR target pointers LIBINT2_RESTRICT - #449

Merged
evaleev merged 3 commits into
masterfrom
perf/restrict-rr-targets
Oct 25, 2026
Merged

evaleev merged 3 commits into
masterfrom
perf/restrict-rr-targets

Conversation

@evaleev

@evaleev evaleev commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Follow-up to #443. This hoists more loads out of generated kernels by telling the compiler that RR targets don't alias their sources.

What

Generated RR functions take LIBINT2_REALTYPE* target, const LIBINT2_REALTYPE* src0, ... with no aliasing information. Each store to target therefore forces the compiler to reload source values and inteval quantities it already loaded. In a scalar library with am ≤ 4, about 109k source reads in 1164 RR functions come after a store to target.

This PR:

  • adds a LIBINT2_RESTRICT macro to config.h;
  • declares the target argument of generated functions LIBINT2_RESTRICT. That covers 2214 RR functions and, through parent_stack, 587 _prereq functions;
  • declares the target argument of the generic RR templates (src/lib/libint/*.h) LIBINT2_RESTRICT as well;
  • adds a code-generation check (DirectedGraph::print_def) that no RR target overlaps any of its children.

Why it's sound. allocate_mem assigns a vertex its stack block before it releases its children's blocks, so a target never overlaps a live source. The new check enforces this: it validates over 10k target/child pairs even for an am-2 library. Inside RR functions every store goes through target; none of them uses stack scratch space or inteval->vstack. No _prereq function touches inteval->stack.

Pitfall. The macro must not expand to __restrict with GCC-compatible compilers, because macOS <sys/cdefs.h> defines __restrict to nothing in C++. A first attempt with __restrict produced identical machine code. LIBINT2_RESTRICT uses __restrict__ there, __restrict for MSVC, and nothing otherwise. In a g-shell OS VRR kernel, load instructions drop from 907 to 430.

Testing

Built and ran check (compiler coretest, eritest, unit tests with 20M assertions, hf, hf++) for both master (a1c3e63) and this branch in two configurations:

  • default: 1-body, ERI, ERI3 and ERI2 to am 4, 1st derivatives to am 3;
  • generic: the same plus LIBINT2_ENABLE_GENERIC_CODE=ON, LIBINT2_OPT_AM=1;1 and G12 to am 2. Here the OSVRR_xs_xs, OSVRR_xs_xs_deriv and GenericGaussDeriv templates are on the hot path.

All tests pass in all four builds. Benchmark checksums are bit-identical between master and the branch.

Not covered: no config instantiates OSVRR_sx_sx*, ITR_xs_xs, VRR_GTG_1d_xx_xx* or VRR_r12kg12_xs_xs. Syntax-checking all 12 template headers together gives the same 21 errors on master and on this branch, so the change adds none. Those errors are pre-existing, e.g. stray ) in OSVRR_sx_sx_deriv.h and an undeclared npts in VRR_GTG_1d_xx_xx_vec.h. Vectorized builds (LIBINT2_MAX_VECLEN > 1) were not tested.

Timings

Apple M-series, Apple clang -O3. Each case covers all unique shell sets. Libraries alternate within each round; the table shows the min over 3 rounds × 7 runs. Small cases (≤ 10 ms) vary by up to ~4% between rounds.

case master branch speedup master (generic) branch (generic) speedup
ERI (H2O)4 6-31G* 0.135 s 0.133 s 1.02× 0.148 s 0.142 s 1.04×
ERI (H2O)4 def2-SVP 0.198 s 0.194 s 1.02× 0.213 s 0.204 s 1.04×
ERI (H2O)4 cc-pVDZ 0.404 s 0.399 s 1.01× 0.431 s 0.420 s 1.03×
ERI H2O cc-pVTZ 0.0277 s 0.0264 s 1.05× 0.0346 s 0.0312 s 1.11×
ERI H2O cc-pVQZ 0.311 s 0.262 s 1.18× 0.414 s 0.366 s 1.13×
ERI H2O def2-QZVP 0.318 s 0.268 s 1.19× 0.425 s 0.375 s 1.13×
ERI 1st deriv (H2O)4 def2-SVP 0.718 s 0.676 s 1.06× 0.888 s 0.798 s 1.11×
ERI 1st deriv H2O cc-pVTZ 0.174 s 0.150 s 1.16× 0.239 s 0.205 s 1.17×
ERI3 (H2O)4 cc-pVDZ / cc-pVDZ-JKFIT 0.0451 s 0.0450 s 1.00× 0.0497 s 0.0482 s 1.03×
ERI3 H2O cc-pVQZ / cc-pVTZ-JKFIT 0.0093 s 0.0088 s 1.06× 0.0117 s 0.0108 s 1.08×
ERI3 1st deriv (H2O)4 cc-pVDZ / cc-pVDZ-JKFIT 0.114 s 0.112 s 1.01× 0.148 s 0.134 s 1.10×
nuclear (H2O)4 cc-pVTZ 0.0054 s 0.0054 s 1.00× 0.0060 s 0.0058 s 1.03×
nuclear 1st deriv (H2O)4 cc-pVTZ 0.0119 s 0.0120 s 0.99× 0.0189 s 0.0176 s 1.07×

The gains follow the share of time spent in RR kernels: 15–19% for quadruple-zeta and derivative ERIs, a few percent for small basis sets. Those are dominated by per-shell-set overhead.

Considered and dropped: top-level kernels. They re-read inteval->stack before every HRR call. Using the existing stack local, with or without restrict, changed timings by at most 0.2% (5 rounds × 15 runs), so it is not part of this PR.

Generated RR functions take the target and the source integral sets as
plain pointers, so every store to the target forces the compiler to
reload source values (and inteval quantities) it already loaded. Targets
never overlap their sources: allocate_mem assigns a vertex its block
before releasing its children's blocks. Declare the target (the stack, for
prerequisite functions) restrict, and check at code generation that no RR
target overlaps any of its children.

Adds the LIBINT2_RESTRICT macro to config.h. It expands to __restrict__
for GCC-compatible compilers because macOS <sys/cdefs.h> defines
__restrict to nothing in C++.
Same reasoning as for the generated RR functions, which forward their
arguments to these templates when LIBINT2_ENABLE_GENERIC_CODE is ON. With
an unqualified target the inner loops reload the inteval quantities
(oo2z, roz, PA, WP, ...) after every store to the target.
@evaleev
evaleev merged commit ac1a14c into master Oct 25, 2026
13 checks passed
@evaleev
evaleev deleted the perf/restrict-rr-targets branch October 25, 2026 15:26
@evaleev evaleev mentioned this pull request Oct 25, 2026
7 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant