From 0b3cae9bde626e87368a92fec523c983bab7657a Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Sun, 25 Oct 2026 01:16:50 +0000 Subject: [PATCH] decisions: the loop's reference from the rows whose overlapfields list the field, as dev; loop: the rule restated (ref-overlap) Co-Authored-By: Claude Opus 5.5 --- system/decisions.md | 12 +++++++----- system/loop.md | 28 ++++++++++++++++------------ 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/system/decisions.md b/system/decisions.md index 44dfb48..ee5cddc 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -999,10 +999,12 @@ The global default stays `zogy`, and OpenUniverse and GBTDS are unmeasured under **The difference's PSF-fit catalog fills its non-finite pixels from their neighbours for the finder and passes them as `mask=`**: a divergence from dev (dev runs DAOStarFinder and `PSFPhotometry` on the masked difference with its NaNs in place, no knob). Under photutils 3.0.0, DAOStarFinder convolves and measures with the NaNs in place, mask or not, so a star within about 4 px of one masked pixel was never found; part 12 step 6 measured a quarter of SCA 7's area blinded. The fill is `interpolate_replace_nans` with a Gaussian of σ 1 px (a hole wider than the kernel is 0), not a zero fill, which split a star with a masked peak into four rows; the fit ignores the masked pixels. Same bytes (`r00342-inj1` `001_0001` F146 passes 4 to 6, ZOGY, against run 72's and run 81's references): SCA 7 detected below AB 25 0.728 → **0.959** (within 5 px of a mask 9 → 81 of 93, beyond 275 → 293 of 297), false positives per epoch 174 (highest 186) → 165 (167), runs 115 → 117; SCA 8, the control, 542 → 545 of 600, false positives 214 (233) → 214 (233), runs 103 → 116. A star whose own peak pixel is masked is still missed. (finder-mask task 1, rapid PR 439, 2026-10-25) (decision-reference-by-field)= -**The loop builds a field's reference only from the L2 rows whose own field is that field**: a divergence from dev, recorded here for the first time (`rapidpipe.foundation.loop.plan`, no knob). dev's launcher (`awsBatchSubmitJobs_launchSingleReferenceImagePipeline.py`, `get_overlapping_l2files`) takes every L2 file whose `overlapfields` lists the field, ordered by `mjdobs` and then by distance from the tile centre. On fixed-pointing simulations the two rules agree. On flight program 1049's four dithered F158 WFI02 epochs, 0.04 degrees apart, they do not: +**The loop builds a field's reference from the L2 rows whose `overlapfields` list the field, as dev does**: this replaces the earlier rule, which took only the rows whose own field is that field (`rapidpipe.foundation.loop.plan`, no knob). dev's launcher (`awsBatchSubmitJobs_launchSingleReferenceImagePipeline.py`, `get_overlapping_l2files`) takes every L2 file whose `overlapfields` lists the field. -- The frames' centres fall in three fields: epoch 1 in 3014319, epochs 2 and 3 in 3014320, epoch 4 in 3010224. -- Every earlier epoch lists epoch 4's field in its `overlapfields`. -- Under the loop's rule no later epoch has two earlier frames in its own field, so with `min_frames` 2 the loop builds no reference epoch 4 can be differenced against. It builds field 3014320's from epochs 2 and 3 and holds the other two fields. +- **The frames.** A reference is planned for a field and filter that a selected row has as its own field; with a bound, a row at or after it. Its frames are the selected official rows of the filter before the bound whose `overlapfields` list the field, of any SCA, in `mjdobs` order. +- **What is not ported.** dev's second sort key, distance from the tile centre, is not ported, so this is membership parity, not full parity. The reference stage coadds in `mjdobs` order, so the order matters only past `max_frames` (25). +- **Why it changed.** On flight program 1049's four dithered F158 WFI02 epochs, 0.04 degrees apart, the frames' centres fall in three fields: epoch 1 in 3014319, epochs 2 and 3 in 3014320, epoch 4 in 3010224. Under the earlier rule the loop built no reference epoch 4 could be differenced against, and run 118's had to be submitted by hand. +- **Flight, measured.** Under this rule the loop alone (run 126) builds field 3010224's reference from epochs 1 to 3 and differences epoch 4 against it. Every measured number of epoch 4 matches run 118's hand-built reference: core +0.0013, ring −0.0050, usable fraction 0.781, 328 packets. No reference is built for the two fields nothing is differenced against. +- **HLWAS, measured.** On a fixed pointing the rule picks the same frames. The injected sample, `r00342-inj1` `001_0001` SCA 8 F146 passes 4 to 6 under ZOGY (run 127), builds its reference from passes 1 to 3, the frames run 81's reference holds. It detects 546 of 600 against run 116's 545, with false positives per epoch 213 mean and 233 highest against 214 and 233. -Run 118's reference for field 3010224 was submitted by hand with dev's selection, epochs 1 to 3, and epoch 4 was differenced against it. It covers 78.4% of epoch 4's pixels: 28.9% by two or more frames, 53% to 54% by epoch 1 or 3 alone. The code is unchanged. (flight-first task 1, 2026-10-25) +(flight-first task 1, 2026-10-25; ref-overlap task 1, rapid PR 450, 2026-10-25) diff --git a/system/loop.md b/system/loop.md index 7673e7d..544d225 100644 --- a/system/loop.md +++ b/system/loop.md @@ -201,12 +201,15 @@ Its order: taken, at most `--max-intake` per pass. Each names its filter's and SCA's science PSF. The first intake of an exposure expects the shared `exposures` key; the others leave it out. -3. **reference**: one job per field and filter with at least - `[reference.selection] min_frames` official L2 rows, no official - reference and none pending or failed, and only while no intake of - the run is pending, so a reference coadds every epoch the drain - admits rather than the first two to land. The rows are one SCA's, - the first in order with enough frames; the job's scope records which. +3. **reference**: one job per field and filter that a selected L2 row + has as its own field, with no official reference and none pending or + failed, and only while no intake of the run is pending, so a + reference coadds every epoch the drain admits rather than the first + two to land. Its frames are the selected official rows of the + filter, of any SCA, whose `overlapfields` list the field, at least + `[reference.selection] min_frames` of them, as `dev` selects them + ({ref}`ruling `). The job's scope records + the first SCA among the rows it serves. With a reference bound set, the rules change as described under [The reference bound](#the-reference-bound). 4. **difference**: one job per official L2 row whose field has an @@ -295,16 +298,17 @@ seconds after the last one started, or at once if the pass ran longer. ## The reference bound A run may set `[reference.selection] before_mjd`, a modified Julian -date. With it set, a field's reference is built only from the field's -epochs before the bound, and only the epochs at or after it are -differenced, so no epoch is subtracted from a reference that holds it. +date. With it set, a field's reference is built only from the epochs +before the bound that overlap the field, and only for a field an epoch +at or after the bound has as its own. Only the epochs at or after it +are differenced, so no epoch is subtracted from a reference that holds it. The bound is a date rather than a list of passes because a survey such as GBTDS has no passes ({ref}`ruling `). Absent, every epoch may build the reference and every epoch is differenced, as before. -With the bound set, a reference job names only the field's official L2 -rows with `mjdobs` before it, and waits until the selection's intake has +With the bound set, a reference job names only those official L2 rows +with `mjdobs` before it, and waits until the selection's intake has drained: every listed file the selection admits is published and no intake of the run is pending. No intake pending is not enough on its own, because a pass admits at most `--max-intake` files and a field's @@ -313,7 +317,7 @@ bound get no difference, association or packet; the field's chain starts at the bound. A field that, once the intake has drained, has fewer than `min_frames` -official rows before the bound, none at all included, gets no +official rows before the bound that overlap it, none at all included, gets no reference. The pass reports it under `held`, with the field and the reason, on every pass. A held field does not keep the loop from draining, and the verb exits 0: the gap is recorded, with no timeout.