Skip to content

N_EPOCH counts only PSF-validated epochs - #952

Merged
cailmdaley merged 8 commits into
developfrom
fix/n-epoch-psf-validated
Oct 10, 2026
Merged

cailmdaley merged 8 commits into
developfrom
fix/n-epoch-psf-validated

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

The final catalogue's N_EPOCH counts epochs with a validated PSF (point-spread function), the epochs available to ngmix.
There is one coverage-count column: the geometric overlap count remains internal and is not written to the final catalogue.
This PR is stacked on #942, with base feat/final-cat-hdf5.

PSF validation and epoch counts

tile_detect writes the sexcat's geometric N_EPOCH before any PSF exists, by projecting each object into exposure CCD footprints.
psfex_interp only interpolates on CCDs passing validation (ACCEPTED ≥ STAR_THRESH = 22, CHI2 ≤ 2), and ngmix uses those PSF vignets.
Previously, the final catalogue retained the geometric count even when a covering CCD failed validation.

On nibi's smk-g16 (25 tiles, 813,009 objects), 0.6–1.3% of objects have a +1 discrepancy against the PSF-validated exposure-count maps from #887.
Most form whole CCD footprints.
On tile 188.308, exposure 2603246 CCD 35 has CHI2 0.79 but only 21 accepted stars:

N_EPOCH − nexp on tile 188.308: the +1 block is one star-starved CCD

smk-g16: catalogue N_EPOCH against the nexp map (before this PR)

This change removes the +1 discrepancy caused by PSF rejection.
The −1s at CCD edges come from geometry: the map includes DATASEC's half-pixel border, while the catalogue uses strict bounds.
That convention is unchanged.

Catalogue design

  • make_cat overwrites N_EPOCH with the number of entries in each object's galaxy-PSF catalogue, or 0 for "empty".
    An epoch with a failed HSM PSF-shape fit still counts because its PSF model exists.
  • N_EPOCH is written even without SAVE_PSF_DATA; per-epoch slot columns still follow that setting.
  • Default slot sizing uses the sexcat's geometric max(N_EPOCH) + 1 as an upper bound, before overwriting the count.
    Explicit N_EPOCH_SLOTS is unchanged.
  • The final catalogue and parameter files contain no N_EPOCH_OVERLAP.
    PSF-gate rejections are studied per CCD through validation output and Exposure-level HealSparse maps: exposure count and flagged pixels #887 exposure maps, rather than through a second per-object count.
    One column avoids applying downstream cuts to the wrong count.
  • astra.yaml records this choice in epoch_membership_ccd_bounds and psf_acceptance_thresholds; their committed options are unchanged.

sp_validation's N_EPOCH >= 2 cut now requires two PSF-validated epochs.
An object whose second covering CCD has no validated PSF no longer passes that cut.
ngmix can reject additional epochs, so NGMIX_N_EPOCH <= N_EPOCH.

Catalogues made before this change

For catalogues written with SAVE_PSF_DATA, migrate in place by setting N_EPOCH := count of EXP_ID_n != -1 across the per-epoch slots.
No additional column is created, and no pipeline rerun is needed.
Catalogues without those slots need the galaxy-PSF catalogue to reconstruct the validated count.

Checked on nibi production wave 1 (b01–b05, 125 tiles, 5,015,937 objects): NGMIX_N_EPOCH <= N_EPOCH <= old N_EPOCH for every object; 99.42% unchanged.

Validation

In the develop SIF (shapepipe-develop-240b37e4.sif), with this checkout's src/ shadowed through PYTHONPATH:

  • 180 tests passed across make_cat, make_cat_mask_ext, hsm_column_seams, psf_grammar_properties, final_cat_merge_invariants, final_cat_reader, mask_ext_columns, decisions, and ngmix_range.
  • The runner regression checks sexcat counts [2,3,1] become final N_EPOCH [1,3,0], with no overlap column, even without per-epoch slots.
  • uvx astra-tools@0.2.17 validate passed for astra.yaml and universes/committed.yaml.
  • The workflow parameter-pin test could not run in this SIF because snakemake is not installed.

GPT-6.1 Sol via Claude Opus on behalf of Cail.

cailmdaley and others added 6 commits October 10, 2026 06:44
The tile SExtractor step counts every exposure CCD whose footprint holds an
object, before any PSF model exists. psfex_interp then drops CCDs that fail
PSF validation (STAR_THRESH, CHI2_THRESH), and ngmix fits only epochs with a
PSF, so the final N_EPOCH over-counted by one for 0.6-1.3% of objects on real
campaigns.

make_cat now saves the SExtractor count as N_EPOCH_OVERLAP and writes N_EPOCH
from the galaxy_psf catalogue: the number of epochs it holds for the object
(0 for "empty"). The runner always runs the psf pass for this column; the
per-epoch HSM/EXP_ID/CCD slot families still follow SAVE_PSF_DATA. The
default slot count is max(N_EPOCH_OVERLAP) + 1, an upper bound on every
object's PSF epoch count. The upstream SExtractor N_EPOCH, used by
vignetmaker and psfex_interp/mccd for array sizing, is unchanged. The final
catalogue param files list N_EPOCH_OVERLAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
Conflicts resolved keeping both intents:
- make_cat.py: #942's read_sextractor_data / in-memory dict catalogue, with
  the SExtractor N_EPOCH column kept as N_EPOCH_OVERLAP; _save_psf_data
  reads N_EPOCH_OVERLAP from self._final_cat for the default slot count and
  writes the PSF-validated N_EPOCH.
- make_cat_runner.py: #942's runner, with the psf pass always run for
  N_EPOCH and the per-epoch slots following SAVE_PSF_DATA.
- test_make_cat.py: #942's dict-based stubs and HDF5 runner test, keyed on
  N_EPOCH_OVERLAP; the N_EPOCH runner test reads the HDF5 output.
- test_make_cat_mask_ext.py: #942's dict API with develop's boolean halo test.
- params_pin.json: regenerated; only rules/tile_make_cat moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
…decision record cover N_EPOCH_OVERLAP and the mccd path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
Use the geometric sexcat count only as the internal slot-sizing upper bound. Record the single-column rationale and assert the overlap column is absent.

Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley changed the title N_EPOCH counts only PSF-validated epochs; geometric count kept as N_EPOCH_OVERLAP N_EPOCH counts only PSF-validated epochs Oct 10, 2026
Regenerate the workflow parameter pin in the develop SIF with host Snakemake 9.27.0; the regenerated pin matches the updated base.

Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley changed the base branch from feat/final-cat-hdf5 to develop October 10, 2026 15:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley merged commit f2ab8a5 into develop Oct 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant