Skip to content

TennoGmEval: fix Yukawa core integrals for U < 1e-7 at small T (#444); remove the Gaussian-fit fallback - #447

Merged
evaleev merged 5 commits into
masterfrom
fix/tenno-small-U
Oct 25, 2026
Merged

evaleev merged 5 commits into
masterfrom
fix/tenno-small-U

Conversation

@evaleev

@evaleev evaleev commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Fixes #444. Also removes the Gaussian-fit fallback from TennoGmEval: every regime now has a direct evaluator.

Yukawa core integrals for U < 1e-7 at small T (#444)

TennoGmEval::eval_yukawa() routed every U = zeta^2/(4 rho) below the interpolation range to the upward recursion regardless of T. The recursion amplifies rounding errors by ~(2m-1)/(2T) per step, so at small T the higher orders were garbage (up to 135 orders of magnitude at m = 36, T = 0.001).

Both eval_yukawa() and eval_slater() now use eval_small_U() for U < Umin, T <= Tmax: an expansion in powers of U about the Boys function,

G_m(T,U) = e^U [ sum_{k<=4} (-U)^k/k! F_{m-k}(T) + U^{m+1/2}/2 sum_{j<=4} Gamma(-m-j-1/2) (-TU)^j/j! ]

with the finite parts of F_n, n < 0, from the (stable) downward recursion. The truncation error is below 1e-20 for U <= 1e-7, T <= 1024 (2368-point grid against 50-digit quadrature, both operators; max abs error of the implementation 2.2e-16). U = 0 (zeta = 0) is now exact for any T (previously an assert/NaN).

No more Gaussian-fit fallback

The fallback was reached in two regimes; both now have direct evaluators:

  • U + 2 sqrt(U T) > 709 (e.g. U >= 240 at small T, inside the interpolation range; and T > Tmax, where the upward recursion used to return NaN): only the closed forms of G_0, G_{-1} overflowed, since exp(x^2 - T) and erfc(x) were formed separately. They are now computed as exp(-T) erfcx(x) for x > 25 (asymptotic series of erfcx), which never overflows.
  • U > Umax = 1e3: new eval_large_U(), the asymptotic (Watson) expansion of G_m = (e^{-T}/2) int_0^inf e^{-(U-T)s} (1+s)^{-m-3/2} e^{-T s^2/(1+s)} ds in 1/(U-T), with a 3-term recursion for the coefficients. G_{m-1} - G_m (Slater) has its own series, avoiding cancellation. Values are below 1e-300 unless U - T > 255, where the series reaches double precision in a few dozen terms.

Removed: the cache of Gaussian fits (which leaked by design), both GaussianGmEval members (not safe to use concurrently although the TennoGmEval instance is shared by all threads), the unconditional clog diagnostics, and the bool/optional failure plumbing. fit_slater() is kept as a public utility.

regime (mmax = 36) before after
U = 800, T = 0.3 Gaussian fit, 40 us interpolation, 6.6 us
U = 1e4 Gaussian fit, 40 us asymptotic series, 0.5 us, rel. err. <= 6e-16
T = 1280, U = 125 NaN 1.7585902911277e-295 (exact: 1.7585902911279e-295)

Tests

test-core-ints.cc:

  • small U: Yukawa at T = 1e-3, 1, 1024 and Slater at T = 1e-3, 1 with U = 1e-8; U = 0 against FmEval_Chebyshev7; T = 0, U = 1e-8 tolerances tightened from 1.4e-11 to 1.4e-14;
  • former fallback regimes, both operators: T = 0.3, U = 800 and T = 5, U = 1000 at the table's absolute precision; T = 0.3, U = 1e4, T = 10, U = 2000, T = 0, U = 1e6 at 1e-14 relative precision (40-digit references); T > Tmax overflow cases finite and negligible;
  • "TennoGmEval shared across threads": every path from 4 threads, bitwise agreement with the serial result (failed under ThreadSanitizer with the old fallback; clean now).

eval_yukawa() routed every U = zeta^2/(4 rho) below the interpolation range
(1e-7) to the upward recursion, which amplifies rounding errors by about
(2m-1)/(2T) per step and hence returns garbage for m >= 4 at small T.
eval_slater() fell back to the Gaussian fit of the kernel, which is accurate
but ~100x slower and uses scratch shared between threads.

Both now use the small-U expansion
  G_m(T,U) = e^U [ sum_k (-U)^k/k! F_{m-k}(T)
                   + U^{m+1/2}/2 sum_j Gamma(-m-j-1/2) (-TU)^j/j! ]
with the finite parts of F_n, n < 0, obtained by downward recursion from F_0;
the truncation error is below 1e-20 for U <= 1e-7 and T <= 1024 (checked
against 50-digit quadrature on a 2368-point grid). U = 0 (zeta = 0) is now
exact for any T.

Unit tests: Yukawa at T = 1e-3, 1, 1024 and Slater at T = 1e-3, 1 with
U = 1e-8; U = 0 against the Boys function; the T = 0, U = 1e-8 tolerances
tightened from 1.4e-11 to 1.4e-14.
…e Gaussian-fit fallback

eval_urr() computed its seed values G_{-1}, G_0 without the overflow guard
that eval_G0_and_maybe_Gm1() has, so for T > Tmax and U + 2 sqrt(U T) > 709
(e.g. zeta = 10, rho = 0.2, |PQ| = 80 bohr) exp overflowed, erfc underflowed,
and every G_m came out NaN. It now seeds from eval_G0_and_maybe_Gm1<true>()
and reports failure on overflow, which routes to the Gaussian fit; the exact
values there are below exp(U - 2 sqrt(U T)), i.e. negligible.

The Gaussian-fit fallback (U > Umax and the overflow cases) was not safe to
use concurrently although the TennoGmEval instance is shared by all threads:
its evaluators were created lazily without synchronization and the k=-1
evaluator used itself as scratch. The evaluators are now constructed in the
TennoGmEval constructor and the k=-1 evaluator is given per-call scratch.
Confirmed with ThreadSanitizer: 12 reports and wrong results before, none
after. The fallback's diagnostics are now printed only if libint2::verbose(),
like the other diagnostics in this file (they were printed on every call).

Unit tests: T > Tmax overflow cases must be finite and negligible;
"TennoGmEval shared across threads" evaluates every path from 4 threads and
requires bitwise agreement with the serial result.
The fallback was reached only in two regimes, both of which now have direct
evaluators:

- U + 2 sqrt(U T) > 709 (e.g. U >= 240 at small T inside the interpolation
  range): only the closed forms of G_0 and G_{-1} overflowed, because
  exp(x^2 - T) and erfc(x) were formed separately. They are now computed as
  exp(-T) erfcx(x) for x > 25 (asymptotic series of erfcx), which never
  overflows; this also makes the T > Tmax upward recursion overflow-free.
- U > Umax = 1e3: the new eval_large_U() uses the asymptotic (Watson)
  expansion of G_m = (e^{-T}/2) int_0^inf e^{-(U-T)s} (1+s)^{-m-3/2}
  e^{-T s^2/(1+s)} ds in 1/(U-T), with a 3-term recursion for the
  coefficients; G_{m-1} - G_m (for the Slater geminal) has its own series,
  avoiding cancellation. The values are below 1e-300 unless U - T > 255, where
  the series reaches double precision in a few dozen terms. Checked against
  quadrature for T in [0, 1023], U in [1e3, 1e6], m <= 36: max relative error
  6e-16. Cost ~0.5 us per call (mmax=36) vs ~40 us for the Gaussian fit.

Removed: the cache of Gaussian fits, both GaussianGmEval members, and the
bool/optional failure plumbing of interpolate_Gm(), eval_urr() and
eval_G0_and_maybe_Gm1(). fit_slater() is kept as a public utility.

Unit tests: overflow cases inside the interpolation range (T=0.3, U=800;
T=5, U=1000) at the table's absolute precision, and U > Umax cases
(T=0.3, U=1e4; T=10, U=2000; T=0, U=1e6) at 1e-14 relative precision, for
both operators.
@evaleev evaleev changed the title TennoGmEval: fix Yukawa core integrals for U < 1e-7 at small T (#444); no NaN on overflow, thread-safe fallback TennoGmEval: fix Yukawa core integrals for U < 1e-7 at small T (#444); remove the Gaussian-fit fallback Oct 25, 2026
The previous commit passed the exponents of exp(x^2 - T) erfc(x),
x = sqrt(U) -/+ sqrt(T), as U -/+ 2 sqrt(U T). Their rounding errors are then
uncorrelated with those of erfc(x), and G_0, which is the difference of the
two products divided by sqrt(T), lost precision for small T: at T = 1e-6,
U = 250 the error grew from 3.4e-14 (as on master) to 8.6e-13. Computing the
exponent as x*x - T from the same rounded x restores the previous behavior
for x <= 25; larger x use exp(-T) erfcx(x) as before.
@evaleev
evaleev merged commit 98ac030 into master Oct 25, 2026
13 checks passed
@evaleev
evaleev deleted the fix/tenno-small-U branch October 25, 2026 17:14
@evaleev evaleev added this to the 2.14 milestone Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TennoGmEval::eval_yukawa returns garbage for U < 1e-7 at small T (unstable upward recursion)

1 participant