diff --git a/astra.yaml b/astra.yaml index b1aa23b46..111d4e538 100644 --- a/astra.yaml +++ b/astra.yaml @@ -733,7 +733,9 @@ analyses: admits its upper endpoint, column 2080. [HARDCODED] The comparisons are strict in make_post_process; CCD_SIZE sets only the bounds. A WCS inversion failure skips that CCD, lowering N_EPOCH, which caps - how many exposures enter each galaxy's multi-epoch fit. + how many exposures enter each galaxy's multi-epoch fit. This geometric + count remains internal; the final catalogue's N_EPOCH counts only + epochs with a validated PSF. Values: SEXTRACTOR_RUNNER.CCD_SIZE = 33,2080,1,4612; SEXTRACTOR_RUNNER.MAKE_POST_PROCESS = True. @@ -1173,6 +1175,12 @@ analyses: 22-star floor to that accepted count. In the science path the CCD's epoch is dropped for every object on it, and an object left with no epoch has no shape. + The final catalogue's N_EPOCH counts the epochs with an + interpolated PSF (for psfex, those passing this gate; for mccd, + those with a model). Only this count is written as N_EPOCH: one column + avoids downstream cuts on the geometric count by mistake. + PSF-gate rejections are studied per CCD in validation output and + the exposure maps (#887), rather than per object. The pipeline has no minimum-epoch floor: NGMIX_N_EPOCH records what survived and epoch-count cuts happen downstream. Guinot+22 discards the CCD from PSF estimation rather than gating at interpolation. diff --git a/example/cfis_image_sims/final_cat.param b/example/cfis_image_sims/final_cat.param index 5f96d959e..3ac278819 100644 --- a/example/cfis_image_sims/final_cat.param +++ b/example/cfis_image_sims/final_cat.param @@ -31,7 +31,8 @@ NGMIX_MCAL_TYPES_FAIL NGMIX_G1_PSF_ORIG_NOSHEAR NGMIX_G2_PSF_ORIG_NOSHEAR -# Number of epochs (exposures) +# Number of epochs: N_EPOCH with a validated PSF model, +# NGMIX_N_EPOCH fitted by ngmix N_EPOCH NGMIX_N_EPOCH diff --git a/src/shapepipe/modules/make_cat_package/__init__.py b/src/shapepipe/modules/make_cat_package/__init__.py index cf1241168..dc9b93bdc 100644 --- a/src/shapepipe/modules/make_cat_package/__init__.py +++ b/src/shapepipe/modules/make_cat_package/__init__.py @@ -21,7 +21,9 @@ previous module runs. This gathers all relevant information on the measured galaxies for weak-lensing post-processing. This includes galaxy detection and basic measurement parameters, the PSF model at galaxy positions, and the -shape measurement. Every detected object is kept: the catalogue carries no +shape measurement. ``N_EPOCH`` counts the object's epochs with a validated +PSF model, the epochs ngmix can fit. +The geometric count of covering CCDs is used internally, not written out. Every detected object is kept: the catalogue carries no star/galaxy classification, which is done downstream. Each object is tagged with its source tile via the ``TILE_ID`` column and carries the survey-wide object ID ``TILE_UNIQUE_ID = tile_id * 10**6 + NUMBER`` @@ -43,7 +45,8 @@ (``HSM_*_PSF_n``, ``EXP_ID_n``, ``CCD_n``) when ``SAVE_PSF_DATA`` is ``True``; unfilled slots hold the family's sentinel, and an object with more epochs than slots raises an error. A fixed count gives every tile - the same schema. Default is the tile's maximum ``N_EPOCH`` plus one + the same schema. Default is the sexcat's maximum geometric ``N_EPOCH`` plus + one """ diff --git a/src/shapepipe/modules/make_cat_package/make_cat.py b/src/shapepipe/modules/make_cat_package/make_cat.py index ffd8f9e24..e6640d006 100644 --- a/src/shapepipe/modules/make_cat_package/make_cat.py +++ b/src/shapepipe/modules/make_cat_package/make_cat.py @@ -81,6 +81,12 @@ def read_sextractor_data(sexcat_path, remove_vignet=True): ``NUMBER``). The tile is read from the catalogue's file name, e.g. ``sexcat-301-279.fits``. + The SExtractor catalogue's ``N_EPOCH`` counts the exposure CCDs whose + footprint holds the object, before any PSF model exists; it is used + internally for slot sizing and overwritten before output. The final catalogue's ``N_EPOCH`` counts only the + epochs with a validated PSF and is written by + :meth:`SaveCatalogue.process` in ``psf`` mode. + Parameters ---------- sexcat_path : str @@ -214,6 +220,7 @@ def process( cat_path=None, moments=False, n_epoch_slots=None, + epoch_slots=True, ): """Process Catalogue. @@ -227,7 +234,10 @@ def process( Option to run ``ngmix`` mode with moments n_epoch_slots : int, optional Number of per-epoch slots in ``psf`` mode; if ``None``, the - tile's ``max(N_EPOCH) + 1`` + sexcat's geometric ``max(N_EPOCH) + 1`` + epoch_slots : bool, optional + In ``psf`` mode, write the per-epoch column families as well as + ``N_EPOCH``; default ``True`` Returns -------- @@ -242,7 +252,7 @@ def process( if mode == "ngmix": err_msg = self._save_ngmix_data(cat_path, moments) elif mode == "psf": - self._save_psf_data(cat_path, n_epoch_slots) + self._save_psf_data(cat_path, n_epoch_slots, epoch_slots) else: err_msg = ( f"Invalid process mode ({mode}) for " @@ -549,14 +559,24 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): return None - def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): + def _save_psf_data( + self, galaxy_psf_path, n_epoch_slots=None, epoch_slots=True + ): """Save PSF data. - Save the PSF catalogue into the final one, as per-epoch column + Save the PSF catalogue into the final one: the per-object epoch + count ``N_EPOCH`` and, if ``epoch_slots``, the per-epoch column families ``HSM_*_PSF_n`` (``psf_shape_cols``), ``EXP_ID_n`` and - ``CCD_n``. Slot ``n`` of every - family refers to the same epoch; slots no epoch fills keep the - family's sentinel. + ``CCD_n``. Slot ``n`` of every family refers to the same epoch; + slots no epoch fills keep the family's sentinel. + + ``N_EPOCH`` counts the object's entries in the PSF catalogue, i.e. + the epochs whose CCD passed PSF-model validation and so have an + interpolated PSF; ngmix fits no other epoch. It is at most + the sexcat's geometric ``N_EPOCH``, the count of CCDs whose footprint holds + the object, and 0 for an object the PSF catalogue marks + ``"empty"``. An epoch whose PSF shape fit failed + (``HSM_FLAG_PSF != 0``) still counts: its PSF exists. @sc [label:schema] psf-epoch-slot-columns The per-epoch families are ``psf_shape_cols`` plus ``EXP_ID``/``CCD``, @@ -570,7 +590,10 @@ def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): Path to the PSF catalogue to save n_epoch_slots : int, optional Number of slots written per family; if ``None``, the tile's - ``max(N_EPOCH) + 1`` + geometric ``max(N_EPOCH) + 1`` from the sexcat, which bounds every object's PSF + epoch count + epoch_slots : bool, optional + Write the per-epoch column families; default ``True`` Raises ------ @@ -580,12 +603,15 @@ def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): """ galaxy_psf_cat = SqliteDict(galaxy_psf_path) - n_epoch = self._final_cat["N_EPOCH"] - if n_epoch_slots is None: - n_slots = np.max(n_epoch) + 1 + n_obj = len(self._obj_id) + n_epoch = np.zeros(n_obj, dtype="int32") + + if not epoch_slots: + n_slots = 0 + elif n_epoch_slots is None: + n_slots = np.max(self._final_cat["N_EPOCH"]) + 1 else: n_slots = n_epoch_slots - n_obj = len(self._obj_id) # Per-epoch PSF shape columns copied from the producer's SHAPES dict: # (column, empty-slot fill, dtype). Fills are out of physical range @@ -613,15 +639,27 @@ def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): for idx, id_tmp in enumerate(self._obj_id): - obj_epochs = galaxy_psf_cat[str(id_tmp)] + try: + obj_epochs = galaxy_psf_cat[str(id_tmp)] + except KeyError: + galaxy_psf_cat.close() + raise KeyError( + f"Object {id_tmp} is missing from {galaxy_psf_path}; " + "the galaxy PSF catalogue and the SExtractor catalogue " + "do not match" + ) from None if obj_epochs == "empty": continue + n_epoch[idx] = len(obj_epochs) + if not epoch_slots: + continue + if len(obj_epochs) > n_slots: galaxy_psf_cat.close() raise ValueError( - f"Object {id_tmp} has {len(obj_epochs)} PSF epochs" - + f" (N_EPOCH={n_epoch[idx]}), more than the {n_slots}" + f"Object {id_tmp} has N_EPOCH={n_epoch[idx]} PSF epochs," + + f" more than the {n_slots}" + f" per-epoch slots (N_EPOCH_SLOTS={n_epoch_slots})" ) @@ -649,3 +687,5 @@ def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): ) galaxy_psf_cat.close() + + self._output_dict = {"N_EPOCH": n_epoch, **self._output_dict} diff --git a/src/shapepipe/modules/make_cat_runner.py b/src/shapepipe/modules/make_cat_runner.py index 540cb8afb..80e1e5756 100644 --- a/src/shapepipe/modules/make_cat_runner.py +++ b/src/shapepipe/modules/make_cat_runner.py @@ -76,8 +76,14 @@ def make_cat_runner( w_log.info(err_msg) return None, None - if save_psf: - sc_inst.process("psf", galaxy_psf_path, n_epoch_slots=n_epoch_slots) + # N_EPOCH (epochs with a validated PSF) always; the per-epoch PSF + # columns only when SAVE_PSF_DATA is set. + sc_inst.process( + "psf", + galaxy_psf_path, + n_epoch_slots=n_epoch_slots, + epoch_slots=save_psf, + ) # Optional per-band external healsparse mask lookup (UNIONS-WL/spherex#38): # add one MASK_ column per band, queried at each object's world diff --git a/tests/module/test_make_cat.py b/tests/module/test_make_cat.py index b5b48548b..6ef58b9e7 100644 --- a/tests/module/test_make_cat.py +++ b/tests/module/test_make_cat.py @@ -396,14 +396,20 @@ def _write_galaxy_psf_cat(path, per_obj): db.close() -def _run_save_psf(galaxy_psf_path, obj_id, n_epoch, n_epoch_slots=None): +def _run_save_psf( + galaxy_psf_path, obj_id, n_overlap, n_epoch_slots=None, epoch_slots=True +): """Drive ``_save_psf_data`` and return its populated output dict.""" inst = object.__new__(SaveCatalogue) inst._obj_id = np.asarray(obj_id) inst._output_dict = {} - inst._final_cat = {"N_EPOCH": np.asarray(n_epoch)} + inst._final_cat = {"N_EPOCH": np.asarray(n_overlap)} - inst._save_psf_data(str(galaxy_psf_path), n_epoch_slots=n_epoch_slots) + inst._save_psf_data( + str(galaxy_psf_path), + n_epoch_slots=n_epoch_slots, + epoch_slots=epoch_slots, + ) return inst._output_dict @@ -429,7 +435,7 @@ def test_save_psf_data_exp_id_ccd_align_with_hsm_psf_slots(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - out = _run_save_psf(galaxy_psf_path, [101], n_epoch=[2]) + out = _run_save_psf(galaxy_psf_path, [101], n_overlap=[2]) npt.assert_allclose(out["HSM_G1_PSF_1"], [0.03]) npt.assert_allclose(out["HSM_G1_PSF_2"], [0.01]) @@ -458,7 +464,7 @@ def test_save_psf_data_identity_survives_failed_hsm_fit(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - out = _run_save_psf(galaxy_psf_path, [202], n_epoch=[2]) + out = _run_save_psf(galaxy_psf_path, [202], n_overlap=[2]) npt.assert_allclose(out["HSM_G1_PSF_1"], [0.05]) # Sentinel: the epoch-2 HSM fit failed, so the pre-fill value stands. @@ -473,7 +479,7 @@ def test_save_psf_data_identity_survives_failed_hsm_fit(tmp_path): def test_save_psf_data_fills_sentinel_for_absent_epochs(tmp_path): """Unused epoch slots and "empty" objects keep the -1 sentinel. - ``max_epoch`` (from the largest ``N_EPOCH`` across the catalogue) can + ``max_epoch`` (from the largest geometric sexcat ``N_EPOCH``) can exceed a given object's own epoch count, and an object the PSF catalogue reports no epochs at all for is marked ``"empty"``; both cases must leave EXP_ID_n/CCD_n at -1, an exposure ID / CCD number no @@ -486,8 +492,8 @@ def test_save_psf_data_fills_sentinel_for_absent_epochs(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - # max(N_EPOCH) = 2 -> 3 slot columns, though obj 101 only fills slot 1. - out = _run_save_psf(galaxy_psf_path, [101, 303], n_epoch=[1, 2]) + # Geometric max(N_EPOCH) = 2 -> 3 slots; obj 101 only fills slot 1. + out = _run_save_psf(galaxy_psf_path, [101, 303], n_overlap=[1, 2]) assert out["EXP_ID_1"][0] == 2113864 assert out["CCD_1"][0] == 7 @@ -497,6 +503,52 @@ def test_save_psf_data_fills_sentinel_for_absent_epochs(tmp_path): assert out[col][1] == -1, col +def test_save_psf_data_n_epoch_counts_only_psf_validated_epochs(tmp_path): + """N_EPOCH counts the epochs with an interpolated PSF, not the overlaps. + + Object 101 overlaps three CCDs but one failed PSF-model validation, so + the PSF catalogue holds two epochs for it and ngmix sees two; object 202 + has an epoch whose PSF shape fit failed, which still counts (its PSF + exists); object 303 overlaps a CCD with no validated PSF at all and is + ``"empty"``. + """ + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 101: { + "2603237-12": _psf_epoch(0.01, 0.02, 0.5), + "2603241-12": _psf_epoch(0.03, 0.04, 0.6), + }, + 202: { + "2603237-13": _psf_epoch(0.05, 0.06, 0.7), + "2603241-13": _psf_epoch(-10.0, -10.0, 0.0, flag=5), + }, + 303: "empty", + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + out = _run_save_psf(galaxy_psf_path, [101, 202, 303], n_overlap=[3, 2, 1]) + + npt.assert_array_equal(out["N_EPOCH"], [2, 2, 0]) + assert out["EXP_ID_3"][0] == -1 + + +def test_save_psf_data_without_epoch_slots_writes_only_n_epoch(tmp_path): + """With per-epoch slots off, ``N_EPOCH`` is the only column written.""" + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 101: {"2603237-12": _psf_epoch(0.01, 0.02, 0.5)}, + 303: "empty", + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + out = _run_save_psf( + galaxy_psf_path, [101, 303], n_overlap=[2, 1], epoch_slots=False + ) + + assert list(out) == ["N_EPOCH"] + npt.assert_array_equal(out["N_EPOCH"], [1, 0]) + + # --- make_cat_runner: end-to-end catalogue assembly --- @@ -530,6 +582,7 @@ def test_make_cat_runner_ships_every_detection_unclassified(tmp_path): tile_sexcat_path = tmp_path / "tile_sexcat-350-100.fits" _write_sex_like_cat(tile_sexcat_path, _numbered_data(obj_ids)) galaxy_psf_path = tmp_path / "galaxy_psf-350-100.sqlite" + _write_galaxy_psf_cat(galaxy_psf_path, dict.fromkeys(obj_ids, "empty")) ngmix_path = tmp_path / "ngmix-350-100.fits" _write_ngmix_cat(ngmix_path, obj_ids) @@ -553,6 +606,54 @@ def test_make_cat_runner_ships_every_detection_unclassified(tmp_path): assert not [name for name in cat if name.startswith("MASK_")] +def test_make_cat_runner_n_epoch_excludes_unvalidated_psf_epochs(tmp_path): + """The final N_EPOCH counts only PSF-validated epochs. + + The SExtractor ``N_EPOCH`` counts every exposure CCD covering an object. + Object 1 lies on two CCDs of which one failed PSF validation, object 3 + on one such CCD only. Without SAVE_PSF_DATA the runner still writes + ``N_EPOCH`` from the PSF catalogue, and no per-epoch slot column. + """ + obj_ids = [1, 2, 3] + sexcat = np.array( + list(zip(obj_ids, [2, 3, 1])), + dtype=[("NUMBER", "i8"), ("N_EPOCH", "i8")], + ) + tile_sexcat_path = tmp_path / "tile_sexcat-350-100.fits" + _write_sex_like_cat(tile_sexcat_path, sexcat) + galaxy_psf_path = tmp_path / "galaxy_psf-350-100.sqlite" + _write_galaxy_psf_cat( + galaxy_psf_path, + { + 1: {"2603241-12": _psf_epoch(0.01, 0.02, 0.5)}, + 2: { + "2603237-12": _psf_epoch(0.05, 0.06, 0.7), + "2603241-12": _psf_epoch(0.07, 0.08, 0.9), + "2603246-12": _psf_epoch(0.03, 0.04, 0.6), + }, + 3: "empty", + }, + ) + ngmix_path = tmp_path / "ngmix-350-100.fits" + _write_ngmix_cat(ngmix_path, obj_ids) + + config = CustomParser() + config.read_dict({"MAKE_CAT_RUNNER": {"SHAPE_MEASUREMENT_TYPE": "ngmix"}}) + assert make_cat_runner( + [str(tile_sexcat_path), str(galaxy_psf_path), str(ngmix_path)], + {"output": str(tmp_path)}, + "-350-100", + config, + "MAKE_CAT_RUNNER", + _NullLogger(), + ) == (None, None) + + with h5py.File(tmp_path / "final_cat-350-100.hdf5", "r") as cat: + npt.assert_array_equal(cat["N_EPOCH"][()], [1, 3, 0]) + assert "N_EPOCH_OVERLAP" not in cat + assert "EXP_ID_1" not in cat + + def test_make_cat_runner_writes_one_hdf5_dataset_per_column(tmp_path): """All save stages land in one hdf5 file, one lzf dataset per column. @@ -620,10 +721,14 @@ def test_make_cat_runner_writes_one_hdf5_dataset_per_column(tmp_path): with h5py.File(out_dir / "final_cat-350-100.hdf5", "r") as cat: names = list(cat) assert names[:7] == [ - "NUMBER", "N_EPOCH", "XWIN_WORLD", "YWIN_WORLD", "FLUX_APER", - "TILE_ID", "TILE_UNIQUE_ID", + "NUMBER", "N_EPOCH", "XWIN_WORLD", "YWIN_WORLD", + "FLUX_APER", "TILE_ID", "TILE_UNIQUE_ID", ] - assert names.index("NGMIX_G1_NOSHEAR") < names.index("HSM_G1_PSF_1") + assert ( + names.index("N_EPOCH") + < names.index("NGMIX_G1_NOSHEAR") + < names.index("HSM_G1_PSF_1") + ) assert names[-1] == "MASK_r" for name in names: assert cat[name].shape[0] == len(obj_ids), name @@ -636,6 +741,7 @@ def test_make_cat_runner_writes_one_hdf5_dataset_per_column(tmp_path): assert cat["HSM_FLAG_PSF_1"].dtype == np.int16 assert cat["EXP_ID_2"].dtype == np.int32 npt.assert_array_equal(cat["NUMBER"][()], obj_ids) + npt.assert_array_equal(cat["N_EPOCH"][()], [1, 2, 0]) npt.assert_allclose(cat["HSM_G1_PSF_1"][()], [0.01, 0.05, -10.0]) npt.assert_array_equal(cat["EXP_ID_2"][()], [-1, 2358123, -1]) npt.assert_array_equal(cat["MASK_r"][()], [64, 64, -1]) @@ -763,7 +869,10 @@ def test_save_psf_data_fixed_slots_pad_every_family(tmp_path): # Driven through ``process``, the entry point the runner calls. n_slots = 7 - out = {"NUMBER": np.array([101, 202, 303]), "N_EPOCH": np.array([1, 2, 0])} + out = { + "NUMBER": np.array([101, 202, 303]), + "N_EPOCH": np.array([1, 2, 0]), + } sc = SaveCatalogue(out, 3, _NullLogger()) assert sc.process("psf", str(galaxy_psf_path), n_epoch_slots=n_slots) is None @@ -795,7 +904,7 @@ def test_save_psf_data_more_epochs_than_slots_raises(tmp_path): with pytest.raises(ValueError) as excinfo: _run_save_psf( - galaxy_psf_path, [101, 404], n_epoch=[1, 3], n_epoch_slots=2 + galaxy_psf_path, [101, 404], n_overlap=[1, 3], n_epoch_slots=2 ) msg = str(excinfo.value) assert "404" in msg @@ -814,15 +923,15 @@ def test_save_psf_data_exactly_slots_epochs_fits(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - out = _run_save_psf(galaxy_psf_path, [404], n_epoch=[2], n_epoch_slots=2) + out = _run_save_psf(galaxy_psf_path, [404], n_overlap=[2], n_epoch_slots=2) assert _slot_numbers(out, "EXP_ID") == [1, 2] assert out["EXP_ID_2"][0] == 2229900 npt.assert_allclose(out["HSM_G1_PSF_2"], [0.03]) -def test_save_psf_data_unset_slots_uses_tile_max_n_epoch_plus_one(tmp_path): - """Without N_EPOCH_SLOTS the slot count is the tile's max(N_EPOCH) + 1.""" +def test_save_psf_data_unset_slots_uses_tile_max_n_overlap_plus_one(tmp_path): + """Without N_EPOCH_SLOTS use the sexcat's geometric max(N_EPOCH) + 1.""" galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" per_obj = { 101: {"2113864-7": _psf_epoch(0.01, 0.02, 0.5)}, @@ -834,7 +943,7 @@ def test_save_psf_data_unset_slots_uses_tile_max_n_epoch_plus_one(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - out = _run_save_psf(galaxy_psf_path, [101, 202], n_epoch=[1, 3]) + out = _run_save_psf(galaxy_psf_path, [101, 202], n_overlap=[1, 3]) for family in _PSF_SLOT_SENTINELS: assert _slot_numbers(out, family) == [1, 2, 3, 4], family @@ -862,7 +971,7 @@ def test_save_psf_data_fixed_slots_keep_epoch_alignment(tmp_path): n_slots = 6 out = _run_save_psf( - galaxy_psf_path, [505], n_epoch=[3], n_epoch_slots=n_slots + galaxy_psf_path, [505], n_overlap=[3], n_epoch_slots=n_slots ) for n, (key, g1, g2, t, flag) in enumerate(epochs, start=1): @@ -898,7 +1007,7 @@ def test_save_psf_data_carries_fourth_moments_per_epoch(tmp_path): } _write_galaxy_psf_cat(galaxy_psf_path, per_obj) - out = _run_save_psf(galaxy_psf_path, [101], n_epoch=[2]) + out = _run_save_psf(galaxy_psf_path, [101], n_overlap=[2]) npt.assert_allclose(out["HSM_M4_1_PSF_1"], [0.11]) npt.assert_allclose(out["HSM_M4_2_PSF_1"], [-0.22]) diff --git a/workflow/CONTRACTS b/workflow/CONTRACTS index a940062d4..fe2e16c36 100644 --- a/workflow/CONTRACTS +++ b/workflow/CONTRACTS @@ -24,7 +24,7 @@ column list, in order: `final_cat_merge` writes those columns and no others, and the reader raises on a listed column a tile lacks. So a name belongs there only if `make_cat` writes it on EVERY tile. A per-epoch family (`EXP_ID_n`, `CCD_n`, `HSM_*_PSF_n`) qualifies only with a fixed slot count, since -`make_cat` otherwise sizes it from the tile's own maximum `N_EPOCH`. Enforced +`make_cat` otherwise sizes it from the sexcat's maximum geometric `N_EPOCH`. Enforced by tests/unit/test_final_cat_merge_invariants.py (the merge) and tests/module/test_psf_grammar_properties.py (the shipped names). diff --git a/workflow/config/cfis/config_tile_Mc.ini b/workflow/config/cfis/config_tile_Mc.ini index c6e08830c..695654523 100644 --- a/workflow/config/cfis/config_tile_Mc.ini +++ b/workflow/config/cfis/config_tile_Mc.ini @@ -82,7 +82,7 @@ SHAPE_MEASUREMENT_TYPE = ngmix # Save per-epoch PSF shapes and epoch identity (HSM_*_PSF_n, EXP_ID_n, # CCD_n). The campaign merge needs one column schema across all tiles, so -# the slot count is fixed rather than per-tile max(N_EPOCH)+1. It must +# the slot count is fixed rather than the sexcat's geometric max(N_EPOCH)+1. It must # exceed the survey's maximum epochs per object with margin (smk-g7: max 6 # over 8 inspected tiles); an object with more epochs than slots is an error. SAVE_PSF_DATA = True diff --git a/workflow/config/cfis/final_cat.param b/workflow/config/cfis/final_cat.param index bb1b2efe2..b1958794d 100644 --- a/workflow/config/cfis/final_cat.param +++ b/workflow/config/cfis/final_cat.param @@ -51,7 +51,8 @@ NGMIX_MCAL_FLAGS NGMIX_G1_PSF_ORIG_NOSHEAR NGMIX_G2_PSF_ORIG_NOSHEAR -# Number of epochs (exposures) +# Number of epochs: N_EPOCH with a validated PSF model, +# NGMIX_N_EPOCH fitted by ngmix N_EPOCH NGMIX_N_EPOCH diff --git a/workflow/config/cfis_image_sims/final_cat.param b/workflow/config/cfis_image_sims/final_cat.param index 9db095b66..8b5f34e2a 100644 --- a/workflow/config/cfis_image_sims/final_cat.param +++ b/workflow/config/cfis_image_sims/final_cat.param @@ -35,7 +35,8 @@ NGMIX_NEIGHBOUR_FLAG NGMIX_G1_PSF_ORIG_NOSHEAR NGMIX_G2_PSF_ORIG_NOSHEAR -# Number of epochs (exposures) +# Number of epochs: N_EPOCH with a validated PSF model, +# NGMIX_N_EPOCH fitted by ngmix N_EPOCH NGMIX_N_EPOCH diff --git a/workflow/scripts/ngmix_range.py b/workflow/scripts/ngmix_range.py index 6f2cbbf03..3bb816f5b 100644 --- a/workflow/scripts/ngmix_range.py +++ b/workflow/scripts/ngmix_range.py @@ -193,14 +193,15 @@ def object_epochs(run_dir: Path): discovered by name, never assumed — each with ``n_obj`` rows in ``LDAC_OBJECTS`` row order and ``CCD_N < 0`` where the object misses that exposure. Summing - ``CCD_N >= 0`` across them reproduces the final catalogue's ``N_EPOCH`` - column exactly — checked row by row against 186.307's + ``CCD_N >= 0`` across them reproduces the sexcat's geometric + ``N_EPOCH`` — checked row by row against the pre-change catalogue for 186.307, ``run_sp_tile_Mc/.../final_cat-186-307.fits``, all 35,298 of them, 7 extensions, 116,727 pairs, mean 3.31. The post-process is upstream of the whole tile_shape group, so the extensions always exist by the time ngmix runs; their absence is a broken tile, not a case to accommodate. - N_EPOCH is a GEOMETRIC count and mildly over-states the work, because ngmix + This is a GEOMETRIC count and mildly over-states the work, because ngmix + fits only epochs with a validated PSF (the final catalogue's N_EPOCH) and drops epochs it cannot fit. The same catalogue's NGMIX_N_EPOCH is lower for 2,203 of the 35,298 objects and never higher: 113,947 pairs against 116,727, 2.4%. That gap is the whole of the ~3.4% by which this cost model