From 1db3a73461664d8c407badc464cd22d3fd4ed6c7 Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Sat, 24 Oct 2026 00:31:53 +0000 Subject: [PATCH] decisions, difference, psf: the reference PSF is derived data measured in the difference job Co-Authored-By: Claude Opus 5.5 --- system/decisions.md | 13 +++++++++++++ system/difference.md | 35 ++++++++++++++++++++++++++++------- system/psf.md | 24 ++++++++++++------------ 3 files changed, 53 insertions(+), 19 deletions(-) diff --git a/system/decisions.md b/system/decisions.md index 22d7ac2..f4326a7 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -79,6 +79,19 @@ science PSF is derived from the ePSF reference its own file names, which is the calibration's PSF and what flight data will carry; the simulation's rendering PSF is not a product. (Ben, 2026-10-22 13:08 PDT) +(decision-reference-psf-measured)= +**The reference PSF is derived data, measured in the difference job**: +the PSF ZOGY uses for the reference is built from the stars of the +resampled reference the job has just made and recorded in its manifest, +not an input chosen at launch, because ZOGY needs the PSF of the image +it is given and each resampling widens it: on run 17 (r00342 F146 SCA +8) the resampled reference's stars were 15% wider than the science +ePSF the `psf` stage had handed ZOGY for both, the cause of 73% of the +null test's detections. Fewer than ten stars fails the job; there is no +fallback. `dev` has no knob here (a static, offline-compiled reference +PSF per detector), so this is a divergence from `dev`. +(Ben, 2026-10-23 23:22 UTC, fixes-2026-10-23) + (decision-spot-parked)= **Spot stays parked**: the rebuild's jobs run on demand; the cost figure is read and recorded, and nothing hangs on it. diff --git a/system/difference.md b/system/difference.md index 282cc5d..2cad45f 100644 --- a/system/difference.md +++ b/system/difference.md @@ -17,7 +17,7 @@ page states the records it works within, and the ## Scope and inputs The scope is `{"expid", "sca", "fid"}`, the epoch's key without the -dataset, which is the run's. The job names exactly four `input` rows, +dataset, which is the run's. The job names exactly three `input` rows, read in one short transaction and fetched through `prefix.get`: | Product | Which | @@ -25,12 +25,30 @@ read in one short transaction and fetched through `prefix.get`: | `l2files` | the epoch's own row | | `refimages` | one reference of the scope's filter, imported or made by the [reference](reference) stage; its coverage and uncertainty maps sit beside the mosaic, and its mosaic catalog (`refimcatalogs`, `cattype` 1) gives the reference FWHM | | `psfs`, kind `science` | the science image's PSF for the scope's filter and SCA | -| `psfs`, kind `reference` | the reference image's PSF for the same filter and SCA | -The two PSFs are told apart by the `kind` in their key, read from the -row, so the stage never reads a PSF file's header to decide which is -which. Any other product, a second input of one kind, or an input whose -key is not the scope's fails the job before a manifest is written. +Any other product, a second input of one kind, a `psfs` row of another +kind, or an input whose key is not the scope's fails the job before a +manifest is written. + +The reference image's PSF is not an input: it is derived data, measured +by the job ({ref}`ruling `). Right after +the resampled reference's background is subtracted, the job finds its +stars (DAOStarFinder at 8σ on the pixels finite and covered at the +coverage threshold; peak 20 to 3,500 DN/s, no neighbour within 14 pixels, +the whole 31-pixel stamp usable, and a second-moment width between the +science PSF's and 0.4 pixel more, since resampling only widens a star +while defect pixels are narrower and galaxies wider), builds an +effective PSF from the brightest 100 (`EPSFBuilder`, oversampling 4), +evaluates it at the pixel centres of a 61-pixel stamp re-centred to +0.01 pixel and normalised, and gives that to ZOGY, SFFT and the naive +catalogs as the reference PSF. Fewer than ten stars stops the job with +exit 65 and `refpsf: N stars, 10 needed` on the job row; there is no +fallback. The stamp is published as `refimage_psf.fits.gz`, and the +manifest's `metrics` hold `refpsf_sigma_px`, `refpsf_stars`, +`sci_psf_sigma_px` and `psf_centroid_offset_px` (the reference stamp's +centroid minus the science stamp's). On run 17's pass 4 reference the +measurement takes 4.6 s and gives 1.080 pixels from 38 stars, against +the science ePSF's 0.947. ## What it runs @@ -74,11 +92,14 @@ for diagnosis; on the HLWAS ramp that was 72 files and 2.5 GB a job, | `awaicgen_output_mosaic_cov_map_resampled.fits.gz` | the mask: the resampled reference coverage map the difference is masked with | | `zogy_diffimage_masked_psfcat.parquet` | the catalog: ZOGY's positive PSF-fit catalog, uncapped, of which `sources` is the capped set | | `diffpsf.fits.gz` | the difference's PSF, for forced photometry on the difference | +| `refimage_psf.fits.gz` | the reference PSF the job measured and gave ZOGY | | `bkg_subbed_science_image.fits.gz` | forced photometry on the science image, and the science cutout | | `awaicgen_output_mosaic_image_resampled_gainmatched.fits.gz` | the template cutout | The first four are the ruling's: the difference, its uncertainty, the -mask and the catalog. The other three are what `packet` reads. Every +mask and the catalog. Three of the others are what `packet` reads, and +the reference PSF is the job's own measurement, kept for whoever checks +the subtraction (15 KB). Every FITS file is gzipped at level 1 as the intake's are, the `diffimages` row names the `.fits.gz`, and its checksum is the MD5 of the gzipped file. Everything else, the SFFT and naive products, the negatives, diff --git a/system/psf.md b/system/psf.md index e5194ca..ccee6f0 100644 --- a/system/psf.md +++ b/system/psf.md @@ -4,8 +4,10 @@ `psf` derives the PSF the difference stage reads for one filter and SCA of a dataset, from the CRDS ePSF reference the dataset's own L2 file -names, and declares it twice, once as the science PSF and once as the -reference PSF. Each image's PSF comes from that reference because it is +names, and declares it as the science PSF. The reference image's PSF is +not this stage's: the difference job measures it from the resampled +reference it makes ({ref}`ruling `). +Each image's PSF comes from that reference because it is the calibration's PSF and what flight data will carry ({ref}`ruling `); a simulation's rendering PSF is not a product. The reduction from the ePSF to a pipeline PSF is new @@ -73,14 +75,13 @@ the stamp's centre, and the stage does not re-centre it. ## Declared outputs -Two FITS files under the attempt's prefix, the same stamp in both: +One FITS file under the attempt's prefix: | File | `PSFKIND` | `psfs` key | |---|---|---| | `sciimage_psf__sca.fits` | `sciimage_psf` | (dataset, fid, sca, `science`) | -| `refimage_psf__sca.fits` | `refimage_psf` | (dataset, fid, sca, `reference`) | -and `psfs.parquet` with the two rows, keyed as an imported PSF is +and `psfs.parquet` with its row, keyed as an imported PSF is keyed, so the difference stage reads a derived PSF exactly as it reads a copied one. The `psfid` is drawn from the table's identity, and the `import` stage now draws its own the same way rather than writing a @@ -161,13 +162,12 @@ reference per filter and SCA. detector centre. The reference holds a 3 x 3 grid of positions, so a PSF per quadrant, or one evaluated per source, is possible; which the difference stage should read is the plan's question. -- **The reference PSF rule.** The reference PSF is the science PSF, - the same ePSF, because a reference coadds frames of the same filter - and SCA. The resampled coadd's stars are wider than that: second-moment - width 1.10 pixels against the science image's 0.96 on F146 SCA 8, - which leaves the core-and-ring residual at every star. A reference - PSF measured from the coadd's own stars, or built from its frames' - positions and dithers, is not built. +- **The reference PSF.** Until 2026-10-24 this stage also wrote the + science stamp as the `reference` kind. The resampled coadd's stars are + wider than that (second-moment width 1.10 pixels against the science + image's 0.96 on F146 SCA 8), which left the core-and-ring residual at + every star, so the difference job now measures the reference PSF + itself ([difference](difference)). - **A reference-file part in the `psfs` key.** The key is (dataset, fid, sca, kind), with no reference name, because the simulation names one ePSF per filter and SCA. A dataset whose files name two