decisions, difference, psf: the reference PSF measured in the difference job (fixes refpsf)

system/decisions.md

@@ -79,6 +79,19 @@ science PSF is derived from the ePSF reference its own file names,
 which is the calibration's PSF and what flight data will carry; the
 simulation's rendering PSF is not a product. (Ben, 2026-10-22 13:08 PDT)
 
+(decision-reference-psf-measured)=
+**The reference PSF is derived data, measured in the difference job**:
+the PSF ZOGY uses for the reference is built from the stars of the
+resampled reference the job has just made and recorded in its manifest,
+not an input chosen at launch, because ZOGY needs the PSF of the image
+it is given and each resampling widens it: on run 17 (r00342 F146 SCA
+8) the resampled reference's stars were 15% wider than the science
+ePSF the `psf` stage had handed ZOGY for both, the cause of 73% of the
+null test's detections. Fewer than ten stars fails the job; there is no
+fallback. `dev` has no knob here (a static, offline-compiled reference
+PSF per detector), so this is a divergence from `dev`.
+(Ben, 2026-10-23 23:22 UTC, fixes-2026-10-23)
+
 (decision-spot-parked)=
 **Spot stays parked**: the rebuild's jobs run on demand; the cost
 figure is read and recorded, and nothing hangs on it.

system/difference.md

@@ -17,20 +17,38 @@ page states the records it works within, and the
 ## Scope and inputs
 
 The scope is `{"expid", "sca", "fid"}`, the epoch's key without the
-dataset, which is the run's. The job names exactly four `input` rows,
+dataset, which is the run's. The job names exactly three `input` rows,
 read in one short transaction and fetched through `prefix.get`:
 
 | Product | Which |
 |---|---|
 | `l2files` | the epoch's own row |
 | `refimages` | one reference of the scope's filter, imported or made by the [reference](reference) stage; its coverage and uncertainty maps sit beside the mosaic, and its mosaic catalog (`refimcatalogs`, `cattype` 1) gives the reference FWHM |
 | `psfs`, kind `science` | the science image's PSF for the scope's filter and SCA |
-| `psfs`, kind `reference` | the reference image's PSF for the same filter and SCA |
 
-The two PSFs are told apart by the `kind` in their key, read from the
-row, so the stage never reads a PSF file's header to decide which is
-which. Any other product, a second input of one kind, or an input whose
-key is not the scope's fails the job before a manifest is written.
+Any other product, a second input of one kind, a `psfs` row of another
+kind, or an input whose key is not the scope's fails the job before a
+manifest is written.
+
+The reference image's PSF is not an input: it is derived data, measured
+by the job ({ref}`ruling <decision-reference-psf-measured>`). Right after
+the resampled reference's background is subtracted, the job finds its
+stars (DAOStarFinder at 8σ on the pixels finite and covered at the
+coverage threshold; peak 20 to 3,500 DN/s, no neighbour within 14 pixels,
+the whole 31-pixel stamp usable, and a second-moment width between the
+science PSF's and 0.4 pixel more, since resampling only widens a star
+while defect pixels are narrower and galaxies wider), builds an
+effective PSF from the brightest 100 (`EPSFBuilder`, oversampling 4),
+evaluates it at the pixel centres of a 61-pixel stamp re-centred to
+0.01 pixel and normalised, and gives that to ZOGY, SFFT and the naive
+catalogs as the reference PSF. Fewer than ten stars stops the job with
+exit 65 and `refpsf: N stars, 10 needed` on the job row; there is no
+fallback. The stamp is published as `refimage_psf.fits.gz`, and the
+manifest's `metrics` hold `refpsf_sigma_px`, `refpsf_stars`,
+`sci_psf_sigma_px` and `psf_centroid_offset_px` (the reference stamp's
+centroid minus the science stamp's). On run 17's pass 4 reference the
+measurement takes 4.6 s and gives 1.080 pixels from 38 stars, against
+the science ePSF's 0.947.
 
 ## What it runs
 
@@ -74,11 +92,14 @@ for diagnosis; on the HLWAS ramp that was 72 files and 2.5 GB a job,
 | `awaicgen_output_mosaic_cov_map_resampled.fits.gz` | the mask: the resampled reference coverage map the difference is masked with |
 | `zogy_diffimage_masked_psfcat.parquet` | the catalog: ZOGY's positive PSF-fit catalog, uncapped, of which `sources` is the capped set |
 | `diffpsf.fits.gz` | the difference's PSF, for forced photometry on the difference |
+| `refimage_psf.fits.gz` | the reference PSF the job measured and gave ZOGY |
 | `bkg_subbed_science_image.fits.gz` | forced photometry on the science image, and the science cutout |
 | `awaicgen_output_mosaic_image_resampled_gainmatched.fits.gz` | the template cutout |
 
 The first four are the ruling's: the difference, its uncertainty, the
-mask and the catalog. The other three are what `packet` reads. Every
+mask and the catalog. Three of the others are what `packet` reads, and
+the reference PSF is the job's own measurement, kept for whoever checks
+the subtraction (15 KB). Every
 FITS file is gzipped at level 1 as the intake's are, the `diffimages`
 row names the `.fits.gz`, and its checksum is the MD5 of the gzipped
 file. Everything else, the SFFT and naive products, the negatives,

system/psf.md

@@ -4,8 +4,10 @@
 
 `psf` derives the PSF the difference stage reads for one filter and SCA
 of a dataset, from the CRDS ePSF reference the dataset's own L2 file
-names, and declares it twice, once as the science PSF and once as the
-reference PSF. Each image's PSF comes from that reference because it is
+names, and declares it as the science PSF. The reference image's PSF is
+not this stage's: the difference job measures it from the resampled
+reference it makes ({ref}`ruling <decision-reference-psf-measured>`).
+Each image's PSF comes from that reference because it is
 the calibration's PSF and what flight data will carry
 ({ref}`ruling <decision-psf-from-epsf>`); a simulation's rendering PSF
 is not a product. The reduction from the ePSF to a pipeline PSF is new
@@ -73,14 +75,13 @@ the stamp's centre, and the stage does not re-centre it.
 
 ## Declared outputs
 
-Two FITS files under the attempt's prefix, the same stamp in both:
+One FITS file under the attempt's prefix:
 
 | File | `PSFKIND` | `psfs` key |
 |---|---|---|
 | `sciimage_psf_<filter>_sca<NN>.fits` | `sciimage_psf` | (dataset, fid, sca, `science`) |
-| `refimage_psf_<filter>_sca<NN>.fits` | `refimage_psf` | (dataset, fid, sca, `reference`) |
 
-and `psfs.parquet` with the two rows, keyed as an imported PSF is
+and `psfs.parquet` with its row, keyed as an imported PSF is
 keyed, so the difference stage reads a derived PSF exactly as it reads
 a copied one. The `psfid` is drawn from the table's identity, and the
 `import` stage now draws its own the same way rather than writing a
@@ -161,13 +162,12 @@ reference per filter and SCA.
   detector centre. The reference holds a 3 x 3 grid of positions, so a
   PSF per quadrant, or one evaluated per source, is possible; which the
   difference stage should read is the plan's question.
-- **The reference PSF rule.** The reference PSF is the science PSF,
-  the same ePSF, because a reference coadds frames of the same filter
-  and SCA. The resampled coadd's stars are wider than that: second-moment
-  width 1.10 pixels against the science image's 0.96 on F146 SCA 8,
-  which leaves the core-and-ring residual at every star. A reference
-  PSF measured from the coadd's own stars, or built from its frames'
-  positions and dithers, is not built.
+- **The reference PSF.** Until 2026-10-24 this stage also wrote the
+  science stamp as the `reference` kind. The resampled coadd's stars are
+  wider than that (second-moment width 1.10 pixels against the science
+  image's 0.96 on F146 SCA 8), which left the core-and-ring residual at
+  every star, so the difference job now measures the reference PSF
+  itself ([difference](difference)).
 - **A reference-file part in the `psfs` key.** The key is (dataset,
   fid, sca, kind), with no reference name, because the simulation names
   one ePSF per filter and SCA. A dataset whose files name two