From 8f371788f04016aa46be73dea05167a1d0c9a80a Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Sat, 24 Oct 2026 03:18:48 +0000 Subject: [PATCH] foundation, difference: the fixes' re-count in seven filters, the null test's figures pointed at it, the quality numbers and the noise-relative peak floor Co-Authored-By: Claude Opus 5.5 --- system/difference.md | 30 ++++++++++++++++++---- system/foundation.md | 61 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 79 insertions(+), 12 deletions(-) diff --git a/system/difference.md b/system/difference.md index 10e3838..cdff699 100644 --- a/system/difference.md +++ b/system/difference.md @@ -34,7 +34,8 @@ The reference image's PSF is not an input: it is derived data, measured by the job ({ref}`ruling `). Right after the resampled reference's background is subtracted, the job finds its stars (DAOStarFinder at 8σ on the pixels finite and covered at the -coverage threshold; peak 20 to 3,500 DN/s, no neighbour within 14 pixels, +coverage threshold; peak at least 400 times the clipped noise and at +most 3,500 DN/s, no neighbour within 14 pixels, the whole 31-pixel stamp usable, and a second-moment width between the science PSF's and 0.4 pixel more, since resampling only widens a star while defect pixels are narrower and galaxies wider), builds an @@ -51,10 +52,29 @@ sub-pixel resampling kernel leaves the wings unchanged. Fewer than ten stars sto exit 65 and `refpsf: N stars, 10 needed` on the job row; there is no fallback. The stamp is published as `refimage_psf.fits.gz`, and the manifest's `metrics` hold `refpsf_sigma_px`, `refpsf_stars`, -`sci_psf_sigma_px` and `psf_centroid_offset_px` (the reference stamp's -centroid minus the science stamp's). On run 17's pass 4 reference the -measurement takes 4.6 s and gives 1.080 pixels from 38 stars, against -the science ePSF's 0.947. +`sci_psf_sigma_px` and `psf_centroid_offset_x_px` and `_y_px` (the +reference stamp's centroid minus the science stamp's). On run 17's pass 4 +reference the measurement takes 4.6 s and gives 1.080 pixels from 38 +stars, against the science ePSF's 0.947. The peak floor was first 20 +DN/s, set on F146, where it is 886 times the noise; in F184 and F213 it +left 8 and 6 stars of 2,100 and 683 detections and failed every +difference, so it is stated in units of the noise (fixes task 5, 400σ: +45 and 19 stars on the resampled references). + +**Quality numbers.** Beside those, the manifest's `metrics` carry the +measurements that would have shown the null test's detections to be +artefacts on its first run, each from arrays the job already holds and +absent, never zero, when it cannot be made: the registration offset +(median centroid of science minus resampled reference at the reference +PSF's stars, before the subpixel shift), the reference stars' width and +its ratio to the science PSF's, the core (within 2.5 pixels) and ring +(2.5 to 6) fractions of the difference over star flux at the 30 brightest +isolated stars, the noise correlation, the usable fraction (finite and +covered pixels), the candidates before the cap and the sources after it, +those above S/N 20, and the share on a reference-catalog source within 3 +pixels. The database's `job_metrics` view exposes every job's metrics by +run, stage and scope, and `scripts/inspect_run.py` renders them per run, +with a gallery of the residual stars, before the run's checks. ## What it runs diff --git a/system/foundation.md b/system/foundation.md index 683d5f5..97357f7 100644 --- a/system/foundation.md +++ b/system/foundation.md @@ -1096,7 +1096,9 @@ the difference stage's 24 GiB for every stage. above S/N 20, none of them variability. They sit at and around stars, the residual a PSF mismatch leaves, and on unmasked pixel defects. SCAs 4 and 5 leave three to four times the artefact residuals of the - other SCAs. + other SCAs. These counts are artefact counts: an image study found + them to be the PSF mismatch and unmasked defects, and The fixes, + below, measures them removed. ### Housekeeping @@ -1129,7 +1131,8 @@ the same pixels. reference, which held each epoch; the same PSF and pixels, the reference the only change. The three epochs gave 551 packets against 447. The artefact fraction above S/N 20 rose from 0.144 to - 0.226. + 0.226. Every count here is of artefacts, not detections (The fixes, + below). Whether depth or disjointness does it is not separated (Not decided here, below). - **Packing has a cost in seconds.** An intake took 39 s with six @@ -1212,6 +1215,49 @@ NED and NED-LVS matches, whose model and catalog copies are in a bucket this account cannot read. sumrat needs no new data, but no stage computes it yet. +### The fixes + +Measured 2026-10-24 on SMDC (brief `fixes-2026-10-23`), on the wide-area +simulation's first pointing, SCA 8, in all seven filters, each filter a +dataset of its own with a new live run: the L2 data-quality flags masked +at intake (commissioning's rule without SATURATED), the reference PSF +measured in the difference job from the resampled reference's stars, the +SWarp kernel left at `dev`'s BILINEAR by a paired comparison, and every +job's quality numbers in its manifest, exposed by the `job_metrics` view +and rendered per run by `scripts/inspect_run.py` before the checks. The +before is the same pixels' official products from the breadth runs and, +for F146, the image study's run, read from the database by lineage. + +| Filter | Epochs | Above S/N 20 per epoch, before → after | Packets per epoch | Packets above S/N 20 | Core / ring of star flux after | +|---|---|---|---|---|---| +| F087 | 2 | 112 → 18 | 192 → 64 | 46 → 4 | -0.003 / +0.007 | +| F106 | 13 | 100 → 22 | 200 → 56 | 41 → 4 | +0.004 / +0.005 | +| F129 | 13 | 101 → 42 | 213 → 108 | 46 → 12 | +0.007 / +0.002 | +| F146 | 3 | 79 → 19 | 213 → 71 | 28 → 3 | +0.006 / -0.002, from +0.030 / -0.027 | +| F158 | 14 | 156 → 18 | 216 → 57 | 75 → 6 | +0.011 / -0.005 | +| F184 | 2 | 373 → 19 | 255 → 42 | 188 → 3 | +0.001 / +0.014 | +| F213 | 2 | 48 → 20 | 110 → 46 | 16 → 3 | +0.034 / +0.023 | + +- **What is gone**: the defects (most of F184's bright end) and the + core-and-ring residual the PSF mismatch left at every star. The share + of sources on a reference source fell from 69% to 43% in F146. +- **What remains** above S/N 20 is bright stars, none of it a defect: + on F146's SCA 8, 11 to 21 of 15 to 23 per epoch sit on a reference + source of aperture S/N above 100, with the PSF's diffraction-spike + pattern in the residual; F213 leaves a positive core and ring, a + flux-scale residual. Turning off `dev`'s science extreme-pixel repair + changed nothing, and the cause is open. SCAs 4 and 5 leave two to three + times SCA 8's. +- **Two defects found by the re-count, fixed**: the reference PSF's star + peak floor was F146's count rate and failed every F184 and F213 + difference (6 to 9 stars of 10 needed); it is now in units of the + noise. The reference job's star width read the galaxies, since + SExtractor's star classifier does not separate them on these mosaics; + it now takes the narrowest population of widths (F087 0.91 to F213 + 1.34 pixels). +- **Every run's checks** were clean: validation, the reader and the + consumer, over 4,336 packets. + ## Not decided here - The cost of following an object set's predecessor chain when a field @@ -1254,11 +1300,12 @@ computes it yet. source seen in two filters is two objects. Whether an object is per band or across bands is open, and with it forced photometry in the other bands at a source's position. -- The null test's artefacts: the PSF mismatch between the science PSF - and the resampled reference ([psf](psf), Not decided here) and pixel - defects the intake does not mask. Masking an L2 file's data-quality - flags at intake, and rejecting outliers in the reference coadd, are - not built. +- What the null test leaves after The fixes: residuals at bright + stars, now the whole bright end (about 20 per epoch above S/N 20 on + SCA 8, two to three times that on SCAs 4 and 5), with a flux-scale + residual in F213; their cause is not established. The quality gate, + a query over `job_metrics` that holds an epoch off its run's median, + is not built. - The 107.5 s exposures of the last two passes are differenced against a 294.1 s reference. Whether a short exposure gets a reference of its own exposure time, or is differenced at all, is open.