Repository navigation
N_EPOCH counts only PSF-validated epochs - #952
Merged
Merged
Conversation
The tile SExtractor step counts every exposure CCD whose footprint holds an object, before any PSF model exists. psfex_interp then drops CCDs that fail PSF validation (STAR_THRESH, CHI2_THRESH), and ngmix fits only epochs with a PSF, so the final N_EPOCH over-counted by one for 0.6-1.3% of objects on real campaigns. make_cat now saves the SExtractor count as N_EPOCH_OVERLAP and writes N_EPOCH from the galaxy_psf catalogue: the number of epochs it holds for the object (0 for "empty"). The runner always runs the psf pass for this column; the per-epoch HSM/EXP_ID/CCD slot families still follow SAVE_PSF_DATA. The default slot count is max(N_EPOCH_OVERLAP) + 1, an upper bound on every object's PSF epoch count. The upstream SExtractor N_EPOCH, used by vignetmaker and psfex_interp/mccd for array sizing, is unchanged. The final catalogue param files list N_EPOCH_OVERLAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
…ange Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
Conflicts resolved keeping both intents: - make_cat.py: #942's read_sextractor_data / in-memory dict catalogue, with the SExtractor N_EPOCH column kept as N_EPOCH_OVERLAP; _save_psf_data reads N_EPOCH_OVERLAP from self._final_cat for the default slot count and writes the PSF-validated N_EPOCH. - make_cat_runner.py: #942's runner, with the psf pass always run for N_EPOCH and the per-epoch slots following SAVE_PSF_DATA. - test_make_cat.py: #942's dict-based stubs and HDF5 runner test, keyed on N_EPOCH_OVERLAP; the N_EPOCH runner test reads the HDF5 output. - test_make_cat_mask_ext.py: #942's dict API with develop's boolean halo test. - params_pin.json: regenerated; only rules/tile_make_cat moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
…epoch-psf-validated
…decision record cover N_EPOCH_OVERLAP and the mccd path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfF1SjWsjHQzZYjL8bbnLo
Use the geometric sexcat count only as the internal slot-sizing upper bound. Record the single-column rationale and assert the overlap column is absent. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
Regenerate the workflow parameter pin in the develop SIF with host Snakemake 9.27.0; the regenerated pin matches the updated base. Co-Authored-By: GPT-6.1 Sol via Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The final catalogue's
N_EPOCHcounts epochs with a validated PSF (point-spread function), the epochs available to ngmix.There is one coverage-count column: the geometric overlap count remains internal and is not written to the final catalogue.
This PR is stacked on #942, with base
feat/final-cat-hdf5.PSF validation and epoch counts
tile_detect writes the sexcat's geometric
N_EPOCHbefore any PSF exists, by projecting each object into exposure CCD footprints.psfex_interp only interpolates on CCDs passing validation (
ACCEPTED ≥ STAR_THRESH = 22,CHI2 ≤ 2), and ngmix uses those PSF vignets.Previously, the final catalogue retained the geometric count even when a covering CCD failed validation.
On nibi's smk-g16 (25 tiles, 813,009 objects), 0.6–1.3% of objects have a +1 discrepancy against the PSF-validated exposure-count maps from #887.
Most form whole CCD footprints.
On tile 188.308, exposure 2603246 CCD 35 has CHI2 0.79 but only 21 accepted stars:
This change removes the +1 discrepancy caused by PSF rejection.
The −1s at CCD edges come from geometry: the map includes DATASEC's half-pixel border, while the catalogue uses strict bounds.
That convention is unchanged.
Catalogue design
N_EPOCHwith the number of entries in each object's galaxy-PSF catalogue, or 0 for"empty".An epoch with a failed HSM PSF-shape fit still counts because its PSF model exists.
N_EPOCHis written even withoutSAVE_PSF_DATA; per-epoch slot columns still follow that setting.max(N_EPOCH) + 1as an upper bound, before overwriting the count.Explicit
N_EPOCH_SLOTSis unchanged.N_EPOCH_OVERLAP.PSF-gate rejections are studied per CCD through validation output and Exposure-level HealSparse maps: exposure count and flagged pixels #887 exposure maps, rather than through a second per-object count.
One column avoids applying downstream cuts to the wrong count.
astra.yamlrecords this choice inepoch_membership_ccd_boundsandpsf_acceptance_thresholds; their committed options are unchanged.sp_validation's
N_EPOCH >= 2cut now requires two PSF-validated epochs.An object whose second covering CCD has no validated PSF no longer passes that cut.
ngmix can reject additional epochs, so
NGMIX_N_EPOCH <= N_EPOCH.Catalogues made before this change
For catalogues written with
SAVE_PSF_DATA, migrate in place by settingN_EPOCH := count of EXP_ID_n != -1across the per-epoch slots.No additional column is created, and no pipeline rerun is needed.
Catalogues without those slots need the galaxy-PSF catalogue to reconstruct the validated count.
Checked on nibi production wave 1 (b01–b05, 125 tiles, 5,015,937 objects):
NGMIX_N_EPOCH <= N_EPOCH <= old N_EPOCHfor every object; 99.42% unchanged.Validation
In the develop SIF (
shapepipe-develop-240b37e4.sif), with this checkout'ssrc/shadowed throughPYTHONPATH:[2,3,1]become finalN_EPOCH [1,3,0], with no overlap column, even without per-epoch slots.uvx astra-tools@0.2.17 validatepassed forastra.yamlanduniverses/committed.yaml.snakemakeis not installed.GPT-6.1 Sol via Claude Opus on behalf of Cail.