docs: difference, reference, intake, datasets and foundation brought to the merged state through rapid PR 456 (consolidate-1025b)

system/datasets.md

@@ -70,7 +70,7 @@ simulation's key grammar, with a preview and a thumbnail PNG and a
 | With the science PSF on the reference, 346 to 418 core-and-ring residuals per epoch above S/N 20 | recorded with its gallery; this dataset's next measurement |
 | The difference job takes 820 to 1,036 s, 3.5 to 4.5 times HLWAS's, on 9,000 to 13,500 candidates per epoch | recorded |
 | The 9100 px reference peaks at 4,514 MiB of its 6,144 MiB ask (27% margin) | recorded |
-| SCA 2's intake Gaia offset `gaia_dy_px` sits at +0.23 to +0.38 px on every epoch (SCA 8 −0.08 to +0.05) | recorded; unread |
+| SCA 2's intake Gaia offset `gaia_dy_px` sits at +0.23 to +0.38 px on every epoch (SCA 8 −0.08 to +0.05) | recorded; not proper motion: carried to the frame's epoch it reads +0.297 against +0.232 on one file; SCA 2's distortion or the Gaia sample unread |
 | The input-catalog tiles over the bulge are stars only, 700 MB and 7.7 million rows each | recorded |
 | The Gaia archive returns at most 50,000 rows a job, silently | recorded; the cone is fetched in sub-boxes with the count checked (256,209 rows) |
 
@@ -115,7 +115,8 @@ whole sequence as FITS, 2.06 million files.
 | Transients live through the first visits: 348 of 761 lit transient rows in the sample have light in a reference frame | the reference stays the first visits under `before_mjd`; the count judges only transients that start after its last frame with no light in it, and reports the rest apart |
 | No Gaia stars in the simulation, and the packet stage needs a catalog | the run's catalog is the truth's star rows on Gaia's schema; the intake's WCS metric reads them, (−0.08, −0.07) px |
 | Every difference fills dev's cap of 500 sources (2,077 to 3,682 candidates, 261 to 500 kept above S/N 20) | recorded; it sets the faint end of the count, the next measurement |
-| The gain match scales the reference by 2.2 (HLWAS 0.99) | recorded; the flux of the subtracted transients is right to 0.95 |
+| The gain match scales the reference by 2.2 (HLWAS 0.99) | the shipped socsim reference zero point, `zprefimg_h158` 25.6549 against the file's 26.5113: 10^(0.4 × 0.8564) = 2.2006, 124 stars 2.214, not `sca_gain`; the flux of the subtracted transients is right to 0.95. On the science's own zero points it reads 1.018, and with `sca_gain = 1.0` too 1.011, with every check held but false positives 493 against 492 inside the capped 500, so the overlay keeps the socsim values, the tolerance Ben's |
+| `sca_gain` 2 is not this simulation's: the science sky's std is 1.535 times the gain-2 noise model and 0.980 times the gain-1 model | recorded; the overlay unchanged with the zero points above |
 
 The chain runs end to end under the reader. The sample is H158 on one
 SCA, whose live transients are the brightest of the seven filters, in
@@ -152,21 +153,24 @@ the program names a placeholder guide star.
 | The data-quality flags are roman_datamodels' pixel flags, as in the simulations | the overlay: HLWAS's mask `0x2110C01`, 0.231% of each frame; 1.36% carry any flag, the largest unmasked GW_AFFECTED_DATA (the guide window) 0.37% |
 | The file names its ePSF, `roman_wfi_epsf_0239` (CRDS `roman_0077`), on all four files: stpsf 2.0.0's pre-flight simulation, pedigree SIMULATION, no jitter | the psf stage reduces it by name, as for the SOC files; the ePSF per file is met |
 | `meta.photometry` is present | the zero point is the file's, ZPTMAG 25.801 |
-| The intake's Gaia offset reads (−0.08 to +0.16, −0.56 to −0.35) px, but that is Gaia DR3 at epoch 2016.0: propagated to 2026.81 the stars sit within 0.09 px | recorded; no WCS key; the metric reads proper motion as WCS on flight data |
+| At Gaia DR3's epoch 2016.0 the intake's Gaia offset reads (−0.08 to +0.16, −0.56 to −0.35) px, proper motion and not WCS | the overlay: `[intake] gaia_proper_motion = true` carries each star to the frame's mid-exposure; the offset reads dx −0.091 to −0.045, dy −0.060 to −0.038 px; no WCS key |
 | The program number is real | the overlay: `program_id = 1049` |
-| The dithers, 0.04 deg apart, put the four centres in three tessellation fields, and the loop builds a field's reference only from rows whose own field it is (dev: every file whose `overlapfields` lists it) | recorded ({ref}`decisions <decision-reference-by-field>`); e4's field reference was submitted by hand with dev's selection, e1 to e3 |
+| The dithers, 0.04 deg apart, put the four centres in three tessellation fields | the loop builds e4's field reference from e1 to e3, every row before the bound whose `overlapfields` list the field, as `dev` ({ref}`decisions <decision-reference-by-field>`) |
 | The reference covers 78.4% of the differenced frame | recorded; usable fraction 0.781, which the gate holds once judged |
 | **Hot pixels**: 3,116 pixels sit above 10 sigma of their local background at the same detector position in all four frames; 341 carry no flag | the mask removes 2,644; no single extra bit removes more than 72 of the other 472 (WARM), so not fixed; the ramp's JUMP_DET fires on 1,235 of them every exposure |
 | **The pointing**: the Gaia stars are 1.19 to 1.20 times the ePSF's width on every frame, nearly isotropic, with a small common elongation over the ePSF (e 0.025 to 0.043, PA +14 to +47°) | recorded; **pointing oscillation unmet, not observable in four 60 s frames**: broadening and a common elongation do not name jitter over charge diffusion or the model's error; the guide window's time series or a run of short exposures would show it |
 
-One run, 118, took the four epochs through the whole chain:
+Run 126 takes the four epochs through the whole chain under the loop
+alone (run 118, the first, had e4's reference submitted by hand under
+the loop's earlier field rule; e4 is identical in both):
 
 - the reference for e4's field from e1 to e3 (`before_mjd = 61337.4594`);
 - e4 differenced in 223 s, core +0.001 and ring −0.005, registration (−0.018, −0.013) px, S/N > 20 228 positive and 150 negative;
 - 328 packets on the topic, sent unjudged below the gate's `min_epochs`;
 - every job published on its first attempt and every check clean.
 
-The runbook's part 13 step flight-first holds the numbers.
+The runbook's part 13 steps flight-first, ref-overlap and gaia-pm hold
+the numbers.
 
 ## RimTimSim, 2026-06-22
 

system/difference.md

@@ -8,7 +8,8 @@ differencer's image and source set for the publisher: the one
 `[difference] diff_flavor` names, shipped as `zogy`
 ({ref}`ruling <decision-published-differencer>`). The other
 differencer's PSF-fit catalogs stay under the prefix as diagnostics, and
-the naive subtraction runs as `dev` runs it, for its outcome bits. The stage ports the science of `dev`'s
+the naive subtraction runs as `dev` runs it but makes no PSF-fit catalog
+({ref}`decision <decision-naive-no-psfcat>`). The stage ports the science of `dev`'s
 `pipeline/awsBatchSubmitJobs_runSingleSciencePipeline.py`, refreshed to
 `dev` at d6aac515, under the foundation's `run(job)` contract:
 `rapidpipe/stages/difference.py` and `rapidpipe/science/difference/` on
@@ -46,7 +47,7 @@ evaluates it at the pixel centres of a stamp the science PSF's size,
 re-centred to 0.01 pixel (a secant step, and a grid search where the
 step stalls, since the evaluated centroid's two axes are coupled), takes the science PSF's wings beyond 10 pixels (the measured
 core within 6, a cosine taper between), normalises it, and gives that to
-ZOGY, SFFT and the naive catalogs as the reference PSF. The wings are the
+ZOGY's and SFFT's catalogs as the reference PSF. The wings are the
 science PSF's because the ePSF is built on 31-pixel stamps and holds
 nothing beyond 15 pixels, where the science PSF still holds 3.3% of its
 flux: given the truncated stamp, ZOGY left a ring 10 to 20 pixels out at
@@ -125,8 +126,10 @@ by subprocess. Its difference and significance images are masked where
 the reference coverage is below threshold, the NaNs are restored, and
 negative copies are made. The stage then makes the difference's
 uncertainty image and the positive and negative SExtractor and
-Photutils PSF-fit catalogs. SFFT and the naive subtraction follow with
-their own catalogs.
+Photutils PSF-fit catalogs. SFFT follows with its own; the naive
+subtraction follows with its image, uncertainty and SExtractor catalogs
+and no PSF-fit catalog, so its two catalog-outcome bits (4 and 5) are
+never set.
 
 Every external tool runs checked, SFFT included: a nonzero exit fails
 the job with exit 70 and `ToolFailed: <tool> exited with code <n>; its
@@ -137,7 +140,7 @@ Photutils PSF fit: on an empty field photutils returns no table, which
 is the empty catalog, and it raises only on a fault, such as a zero or
 NaN uncertainty under a source, which fails the job.
 
-Every PSF-fit catalog, ZOGY's, SFFT's and the naive one's, positive and
+Every PSF-fit catalog, ZOGY's and SFFT's, positive and
 negative, takes the difference's non-finite pixels as its mask: the
 finder sees them filled from their neighbours (a Gaussian of σ 1 px; a
 hole wider than its 9 × 9 kernel is set to 0), and the finder and the
@@ -391,6 +394,13 @@ SCA 8 detected 542 → 545 of 600, false positives 214 (233) unchanged
 (run 116). Candidates before the cap rise where the difference has
 isolated masked pixels.
 
+The naive PSF-fit catalogs not made (rapid PR 446, `rapid-rebuild:79`,
+same bytes, RimTimSim F213 epoch 4571, runs 122 → 124): the difference
+job takes 2,645 s against 4,169 s, `sources` equal in all 27 columns,
+`diffimages` equal but `checksum`, the MD5 of a file whose header
+carries its job id and timestamps (pixels equal). ZOGY's negative and
+SFFT's positive PSF-fit catalogs are now 71% of that job.
+
 ## Not decided here
 
 - Source-level parity with `dev`. The two references are 25 different
@@ -408,6 +418,8 @@ isolated masked pixels.
   datasets' blocks are unchecked for the same effect. Under the shipped
   block `n_sources_snr_gt20` sits at the cap on every epoch, so the
   gate's count check cannot see such an SFFT epoch off-scale.
-- Whether the naive subtraction keeps running now that only its
-  outcome bits remain of it. Turning it off is a settings change with a
-  science consequence, the bits.
+- Whether the naive subtraction keeps running now that its PSF-fit
+  catalogs, and so its two catalog-outcome bits, are gone; its image
+  and SExtractor catalogs remain, and turning it off is a settings
+  change. The `[psfcat_diffimage] output_naive_psfcat_*` keys are read
+  by nothing.

system/foundation.md

@@ -1617,8 +1617,12 @@ cold queue.
   the field rule 13 packets carried a source split that way.
 - Forced photometry's cost at scale. It is about 3.6 s per window
   image, dominated by opening the images. A set's `forced` rows are its
-  packets times the window images on the trigger's SCA, so a field's
-  total grows as the square of its epochs inside one window; F158 at
+  objects times its window images (median ratio 1.000), so a field's
+  total grows as the square of its epochs inside one window: on the
+  HLWAS breadth runs, whose fields span under 10 days, quadratically
+  over epochs 1 to 26, copies 0.57 of run 38's rows and copies crossing
+  runs of one dataset; the table is 4.83 M rows, 1.02 GB, 211 B a row,
+  81% of it runs 36 to 38; past a 30-day span unmeasured; F158 at
   14 epochs held about 14 rows per packet and its packet job about
   300 s.
 - The cost of a large change group, about three round trips per item
@@ -1670,8 +1674,8 @@ cold queue.
 - Whether the published differencer returns to SFFT: with `dev`'s
   prefilter under HLWAS's OpenUniverse block it detects 530 of 600
   injected against ZOGY's 542 at half the false positives; and whether
-  the naive subtraction keeps running for its outcome bits
-  ([difference](difference)).
+  the naive subtraction keeps running now that it makes no PSF-fit
+  catalog ([difference](difference)).
 - Packets after a replacement stay as published with the old
   detections in their history; whether a loop should ever rebuild
   them is open. A campaign reruns from the replaced epoch.

system/intake.md

@@ -114,13 +114,18 @@ The run's `[intake]` table, with `dev`'s hard-coded values as keys:
 | `dq_mask_bits` | 0 | nothing masked, `dev`'s behaviour; a dataset's overlay names the bits whose pixels become NaN |
 | `crpix_shift` | `[0.0, 0.0]` | the file's WCS unchanged; a dataset's measured translation, in pixels, added to `CRPIX1` and `CRPIX2` |
 | `rederive_wcs` | false | the file's gWCS; true re-derives it from the file's `wcsinfo` and the distortion reference it names, before the SIP fit |
+| `gaia_proper_motion` | false | the Gaia offset's stars at the catalog's `ref_epoch`, where the SOC simulations place them; true carries each star by its `pmra` and `pmdec` to the frame's mid-exposure (`MJD-OBS` plus half `EXPTIME`) with the packet stage's propagation, as flight data needs; `dev` has no such metric |
 
 Each dataset's values, and the measurement behind each, are on the
 [datasets](datasets) page. When the job names its dataset's Gaia
 `catalogs` input, as the loop does once one is official, its metrics
 record `gaia_dx_px`, `gaia_dy_px` and `n_gaia_stars`: the median offset
 of the isolated Gaia stars on the frame, measured minus predicted
-through the published WCS, a check of the WCS on every file.
+through the published WCS, a check of the WCS on every file. Under
+`gaia_proper_motion` the stars are first carried to the frame's epoch,
+so on flight data the offset no longer reads proper motion as WCS
+(program 1049: dy −0.56 to −0.35 px at 2016.0, −0.060 to −0.038 px
+propagated).
 
 ## What it measured
 

system/reference.md

@@ -15,9 +15,10 @@ stage is.
 ## Scope and inputs
 
 The scope is `{"field", "fid", "sca"}`: the Roman tessellation tile,
-the filter id and the SCA. Every input is an `input` row of product
-`l2files` whose key has the scope's filter and SCA; any other product
-fails the job. The launcher chooses the frames and records them; the
+the filter id and the first SCA among the frames. Every input is an
+`input` row of product `l2files` whose key has the scope's filter, of
+any SCA, each row read by its own key's SCA; any other product or
+filter fails the job. The launcher chooses the frames and records them; the
 stage reads exactly those rows and their files through
 `prefix.get`, and never selects frames itself.