Skip to content

fix(results): exact fractional confidence-level labels family-wide - #815

Merged
igerber merged 1 commit into
mainfrom
fix/fractional-confidence-label
Sep 3, 2026
Merged

fix(results): exact fractional confidence-level labels family-wide#815
igerber merged 1 commit into
mainfrom
fix/fractional-confidence-label

Conversation

@igerber

@igerber igerber commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • Every text surface that names a confidence level computed the percent with int((1 - alpha) * 100) (truncation), int(round((1 - alpha) * 100)) (rounding), or an :.0f format, so alpha=0.025 printed "97% Confidence Interval" or "98%" over a 97.5% interval. All 34 sites now route through one shared formatter in results_base (_coverage_level / _coverage_pct, plus _alpha_pct for the dCDH HonestDiD "Significant at" line, which printed 2% for alpha=0.025).
  • Surfaces covered: the 14 summary() headers (DiD/TWFE/MultiPeriod/SyntheticDiD, CallawaySantAnna, staggered and 2x2x2 TripleDifference, StackedDiD, ImputationDiD, TwoStageDiD, EfficientDiD, ContinuousDiD, dCDH, SunAbraham, TROP), the EventStudyResults / HAD / RDD / ETWFE / LWDiD / LPDiD / ChangesInChanges table headers, CS and dCDH sup-t band labels, WildBootstrapResults and LWDiD wild-cluster-bootstrap summaries, HonestDiDResults / PlaceboTestResults summaries, BusinessReport / DiagnosticReport prose, and the plot_event_study / plot_dose_response coverage labels.
  • BusinessReport headline.ci_level carries the exact level: int when integral (95 is byte-unchanged) and float otherwise (97.5). No schema-version bump; the decision is recorded as a REPORTING.md Note.
  • Default-alpha output is byte-identical: a before/after capture of 30 summary / report surfaces at the default alpha diffed empty (the only mover was TROP's unseeded bootstrap SE, unrelated to this change).
  • Retires the TODO.md "Fractional confidence-level display unification" row.

Methodology references (required if estimator / math changes)

  • Method name(s): N/A - display-only change; no estimator, inference, weighting, or default changes. Consistent with the REGISTRY fit-time-alpha summary() contract (stored inference is never recomputed or relabeled).
  • Paper / source link(s): N/A
  • Any intentional deviations from the source (and why): None

Validation

  • Tests added/updated: tests/test_coverage_label.py (formatter pins incl. np.float32 alpha; representative alpha=0.025 surfaces: DiD, CS with sup-t, WooldridgeDiD, TWFE event-study table header, dCDH HonestDiD block, BusinessReport ci_level type + prose, DiagnosticReport prose; a source guard that fails on any reintroduced inline percent computation, including float(...)-wrapped forms).
  • Targeted suites run locally: every module touched by the sweep plus the report / narrative drift tests and the tracking-file / changelog guards (4703 + 2617 passed across the two runs; test_trop.py -k summary 4 passed).
  • Backtest / simulation / notebook evidence (if applicable): N/A

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

Changelog

  • changelog.d/ fragment added (or N/A - no user-visible change): Yes (changelog.d/20260903-fractional-confidence-label.md)

Every text surface that names a confidence level computed the percent with
int((1 - alpha) * 100) (truncation), int(round(...)) (rounding), or :.0f, so
alpha=0.025 printed "97%" or "98%" over a 97.5% interval. All 34 sites now
route through one shared formatter in results_base (_coverage_level /
_coverage_pct, plus _alpha_pct for the dCDH "Significant at" line, which
printed 2% for alpha=0.025). Default-alpha output is byte-identical.

BusinessReport headline.ci_level carries the exact level: int when integral
(95 unchanged), float otherwise (97.5). No schema-version bump (REPORTING.md
Note). tests/test_coverage_label.py pins the formatter, the representative
surfaces at alpha=0.025, and adds a source guard against reintroducing an
inline percent computation. Retires the TODO.md row.
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Overall Assessment

Looks good — no unmitigated P0 or P1 findings.

Executive Summary

  • Reporting-only change; estimator math, inference, weighting, variance, and defaults are unchanged.
  • Fractional coverage labels now consistently use shared helpers.
  • Aligns with the fit-time-alpha contract in REGISTRY.md.
  • Changelog fragment and representative regression tests are included.

Methodology

  • P3 — Informational: The reporting behavior is documented in docs/methodology/REPORTING.md:L549-L558 and conforms to docs/methodology/REGISTRY.md:L1233.
    • Impact: No methodological deviation; stored inference is neither recomputed nor relabeled.
    • Concrete fix: None required.

Code Quality

No findings. The shared formatter removes inconsistent inline calculations.

Performance

No findings. Formatting overhead is negligible.

Maintainability

No findings. results_base.py remains dependency-safe, and the source guard discourages regression.

Tech Debt

No findings. Removing the completed TODO.md entry is appropriate.

Security

No findings or exposed secrets.

Documentation / Tests

No findings. The reporting note, release-note fragment, formatter tests, representative surface tests, and source guard are appropriate. Static parsing and changelog validation passed; runtime tests could not be executed because this environment lacks numpy and pytest.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Sep 3, 2026
@igerber
igerber merged commit e982340 into main Sep 3, 2026
39 of 40 checks passed
@igerber
igerber deleted the fix/fractional-confidence-label branch September 3, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant