rusholme · Oct 24, 2026 · Oct 24, 2026
diff --git a/system/datasets.md b/system/datasets.md
@@ -0,0 +1,104 @@
+# Datasets
+
+**Status: DRAFT**
+
+A dataset is one body of input the pipeline takes in, with its own
+live run and its own products, since dataset is in every scientific
+key ([foundation](foundation)). This page states the three simulations
+the rebuild has read, what each one's data are, the quirks met in
+them, and what each quirk became: a key in the dataset's settings
+overlay, a reader, or a line recorded and moved past. A quirk is fixed
+on the overlay and not debugged to its root in every dataset. The
+overlays live in the pipeline repository under
+`rapidpipe/settings/datasets/`, one TOML file per dataset with a
+comment per key naming the measurement that earned it; the
+[foundation](foundation) page states how a run resolves them.
+
+| Dataset | Overlay | Survey | Read since |
+|---|---|---|---|
+| SOC simulation r00342, the High Latitude Wide Area Survey | `hlwas-r00342.toml` | HLWAS | 2026-10-22 |
+| SOC simulation r00340, the Galactic Bulge Time Domain Survey | `gbtds-r00340.toml` | GBTDS | 2026-10-24 |
+| OpenUniverse 2024, the time-domain set (TDS) | `openuniverse-tds.toml` | HLTDS | 2026-10-24 |
+
+## HLWAS, r00342
+
+STScI's SOC simulation of the wide-area survey: 54 pointings on a grid
+one SCA wide, seven filters, six passes, L2 ASDF files with a
+Gaia-fitted WCS and no time-variable sources, so every difference is a
+null test. The ramp, the breadth run over seven filters, the fixes and
+the second iteration's proofs ran on it.
+
+| Quirk | Disposition |
+|---|---|
+| The L2 data-quality flags mark defects the difference stage otherwise detects | the overlay: `dq_mask_bits = 0x2110C01`, commissioning's rule without SATURATED |
+| The L2 WCS sits (+0.12, −0.99) px from the simulation's own input-catalog stars, on every file | the overlay: `crpix_shift = [0.12, -0.99]`; the intake's Gaia-offset metric reads +0.127 / −0.986 px without it and +0.007 / +0.003 with it |
+| Each filter's passes differ in timing | the reference bound is the midpoint of the filter's third and fourth passes, set per run |
+| The last two passes' exposures are 107.5 s against 294.1 s | recorded ([foundation](foundation), Not decided here) |
+
+## GBTDS, r00340
+
+STScI's SOC simulation of the bulge survey, in the wide-area
+simulation's key grammar, with a preview and a thumbnail PNG and a
+`_wcs.asdf` sidecar beside every L2 file.
+
+| | r00340 |
+|---|---|
+| Span | 2027-10-01 to 2027-10-08, 7.8 days |
+| Pointings | 6, at position angle 90.46 throughout |
+| Passes | 5: the colour filters once, then F146 with F087 and F213 interleaved |
+| Exposure | 66.41 s, 5 resultants, every file |
+| Files per filter | F146 80,082; F213 3,348; F087 3,330; F062 432; F129, F158 and F184 216 each; F106 198 |
+| Epochs per pointing and SCA | F146 738 to 744, every 13.7 min; F087 and F213 31; F062 4; the others 2 |
+| Processing | romancal 0.22.0, CRDS context `roman_0046.pmap`, the same ePSF reference as HLWAS |
+
+| Quirk | Disposition |
+|---|---|
+| The L2 gWCS is tweakreg's spurious crowded-field match, 9 to 50 px off and different on every epoch | the intake re-derives the WCS (`rederive_wcs = true`): on one file the input catalog's stars sit at +0.030 / −0.032 px through it, 9,993 matched, against chance through the L2 gWCS |
+| Under today's CRDS context the WCS step takes a distortion reference that puts the frame about 1,770 px off | the re-derivation uses the distortion reference the file names, fetched by name |
+| STScI's recipe reads the L1 header, but the L2 files carry the L1 values unchanged (12 of 12 sampled) | recorded; the L1 file is not read |
+| The sidecar's `wcs_l1` is the L2 gWCS moved by the 4 px reference border | recorded |
+| A pointing is the visit modulo 6, so a pointing selection does not bound the epochs | recorded; a sample is a source root of chosen files |
+| The data-quality flags are HLWAS's | the overlay: HLWAS's mask |
+| The bulge is confusion-limited, so the reference PSF cannot be measured from its stars (none passes) | the overlay: `[difference] reference_psf = "science"`, the science PSF for both images |
+| With the science PSF on the reference, 346 to 418 core-and-ring residuals per epoch above S/N 20 | recorded with its gallery; this dataset's next measurement |
+| The difference job takes 820 s, 3.5 times HLWAS's, on 9,000 candidates per epoch | recorded |
+| The input-catalog tiles over the bulge are stars only, 700 MB and 7.7 million rows each | recorded |
+| The Gaia archive returns at most 50,000 rows a job, silently | recorded; the cone is fetched in sub-boxes with the count checked (256,209 rows) |
+
+One pointing's SCA 8, nine F146 epochs, ran through the whole chain:
+the reference from the first six, three epochs differenced, 515
+packets on the topic, every check clean.
+
+## OpenUniverse TDS
+
+The OpenUniverse 2024 time-domain simulation, romanisim's
+`simple_model` images. STScI's copy is ASDF, 13,256 files over seven
+filters in one patch of about 0.27 deg²; IRSA's public release holds the
+whole sequence as FITS, 2.06 million files.
+
+- **The pointing is a row of the observing sequence**, whose date is
+  the file's exposure start. A visit runs all seven filters over 2.7
+  days; visits come every 5 days. A point in the patch has 106 images
+  per filter at the median.
+- **The truth is in every file**: galaxies, point sources and
+  transients (one SCA held 164 transients, 113 core-collapse, 30 SN Ia,
+  20 SN Iax, at redshift 0.7 to 2.8). IRSA's index per image gives each
+  object's pixel position and realised flux, so completeness is a join
+  of each difference image's candidates to its file's transient rows.
+  It complements injection: real hosts, light curves and cadence,
+  where injection keeps dense control. It comes after HLWAS injection.
+
+| Quirk | Disposition |
+|---|---|
+| The tree is a plain mapping, not a data model; `data` is float64 counts per exposure, `dq` all zero | a reader of its own: `[intake] reader = "openuniverse"`, an adapter onto the SOC reader's conversion |
+| The WCS takes one-based pixels: stars sit at (−1.16, −1.06) px through it | the overlay: `crpix_shift = [-1, -1]`; the truth then sits within 0.0016 px |
+| The file's zero point is photons to counts, not AB | the overlay: the AB zero points for counts per second per filter |
+| `err` equals `data` on 90% of pixels | recorded; nothing reads it |
+| The gWCS cannot be inverted by today's gwcs | recorded; only the forward transform is used, matched by the SIP fit to 1.5e-9″ |
+| No ePSF reference is named | recorded; the proof imported a PSF stamp from the simulation's own model |
+
+One image went through intake under the reader, its truth tables
+carried as binary-table HDUs. A chain past intake needs three pieces
+not yet built: an ePSF source for the psf stage, the loop's file listing
+and the psf stage dispatching on the reader, and a reference from
+epochs before a transient starts.
diff --git a/system/decisions.md b/system/decisions.md
@@ -9,6 +9,31 @@ page where attribution matters.
 
 ## Rulings
 
+Ben's rulings of 2026-10-24 04:00 to 04:40 UTC on the second
+iteration (brief `night-2026-10-24`). The [foundation](foundation) and
+[difference](difference) pages state the rules they set.
+
+(decision-bright-star-flag)=
+**The bright-star residual is flagged, not fixed**: what the null test
+leaves above S/N 20 after the fixes, bright stars, gets an informational
+flag bit and a fraction in the metrics; nothing is dropped. SCAs 4 and
+5's excess gets one metric, the science stars' width against the ePSF,
+and F213's flux scale a metric and one check of the zero-point table.
+Measured: the radius law flags 21 of 23, 15 of 19 and 11 of 15 of
+run 46's sources above S/N 20 on F146 SCA 8; the science stars are 1.5
+to 3.3% wider than the ePSF on SCAs 4, 5 and 8 alike; the table agrees
+with the files to 0.035 mag. (Ben, 2026-10-24 UTC)
+
+(decision-spot-probe)=
+**Spot is a probe, not production**: a run that names a Spot queue
+sends a stage's backlog there with one attempt per submission, and
+every next attempt, and any straggler past a hedge factor, runs on
+demand; nothing is retried on Spot. It supersedes
+{ref}`Spot stays parked <decision-spot-parked>` for runs that opt in.
+Measured: the injected footprint's Spot twin finished 2.1 min after its
+on-demand run, with no reclaim, at $1.30 against $2.03 priced by lane.
+(Ben, 2026-10-24 UTC)
+
 Ben's rulings of 2026-10-22 on what follows the wide-area ramp, made
 stepwise between 12:57 and 13:23 PDT, and one of the same evening. The
 [foundation](foundation) page states the design they shaped, and the
@@ -44,6 +69,9 @@ pointings overlap and buy nothing. The loop admits in `mjdobs` order
 across every SCA of its selection; an earlier epoch that arrives after
 later ones is a rerun from that epoch, and the loop never
 re-associates. (Ben, 2026-10-22 PDT)
+The chain half is superseded by
+{ref}`association in the difference job <decision-association-in-difference>`
+(2026-10-24); the reference half stands.
 
 (decision-injection-outside)=
 **Injection is outside the pipeline, as a dataset**: a one-time job
@@ -117,6 +145,8 @@ night-2026-10-24 task 7)
 **Spot stays parked**: the rebuild's jobs run on demand; the cost
 figure is read and recorded, and nothing hangs on it.
 (Ben, 2026-10-22 13:23 PDT)
+Superseded for runs that opt in by
+{ref}`Spot as a probe <decision-spot-probe>` (2026-10-24).
 
 Ben's rulings of 2026-10-21 evening on the instance type and the plan
 in force. The [foundation](foundation) page states the design they

diff --git a/system/difference.md b/system/difference.md
@@ -40,8 +40,9 @@ the whole 31-pixel stamp usable, and a second-moment width between the
 science PSF's and 0.4 pixel more, since resampling only widens a star
 while defect pixels are narrower and galaxies wider), builds an
 effective PSF from the brightest 100 (`EPSFBuilder`, oversampling 4),
-evaluates it at the pixel centres of a 61-pixel stamp re-centred to
-0.01 pixel, takes the science PSF's wings beyond 10 pixels (the measured
+evaluates it at the pixel centres of a stamp the science PSF's size,
+re-centred to 0.01 pixel (a secant step, and a grid search where the
+step stalls, since the evaluated centroid's two axes are coupled), takes the science PSF's wings beyond 10 pixels (the measured
 core within 6, a cosine taper between), normalises it, and gives that to
 ZOGY, SFFT and the naive catalogs as the reference PSF. The wings are the
 science PSF's because the ePSF is built on 31-pixel stamps and holds
@@ -74,7 +75,30 @@ covered pixels), the candidates before the cap and the sources after it,
 those above S/N 20, and the share on a reference-catalog source within 3
 pixels. The database's `job_metrics` view exposes every job's metrics by
 run, stage and scope, and `scripts/inspect_run.py` renders them per run,
-with a gallery of the residual stars, before the run's checks.
+with a gallery of the residual stars, each cutout divided by its star's
+reference flux, before the run's checks. The metrics also hold the
+science stars' own width (`sci_star_sigma_px`) and its ratio to the
+science PSF's, the reference's scale factor (`scalefacref`), the
+sources near bright stars and in their rings above S/N 20, and every
+job's peak resident memory (`peak_rss_kib`, the process and its largest
+child). The {ref}`quality gate <foundation-quality-gate>` reads them.
+
+**The bright-star footprint flag.** What the null test leaves above S/N
+20 after the fixes is bright stars, and that residual is flagged, not
+fixed ({ref}`ruling <decision-bright-star-flag>`). The job sets bit
+`1 << 14` (16384) of `sources.flags` on every kept source inside the
+footprint of a bright reference source, a mosaic-catalog source with
+aperture S/N above 100, whose footprint is a disc of radius
+`max(4, 9.5 × 10^(0.2 (15 − m)))` pixels at its magnitude `m` on the
+reference's zero point. Nothing is dropped. The bit is informational:
+association and the packet stage clear it before their flag tests, so a
+flagged source is associated and triggers, and its packet carries the
+bit in `psfFitFlags` with no schema change. The metrics count the
+flagged sources and their share of those above S/N 20
+(`bright_footprint_fraction`), and the packet job counts its flagged
+packets. The radius law, derived on run 46's published sources, flags
+21 of 23, 15 of 19 and 11 of 15 of the sources above S/N 20 on F146 SCA
+8, the fixes' own bright-star counts on those epochs.
 
 ## What it runs
 
@@ -207,6 +231,7 @@ port or departs from `dev`:
 | `[difference.sfft] run_sfft`, `crossconv_flag` | true, false | SFFT runs, without cross-convolution for these simulations, as in `dev` |
 | `[difference.naive_diffimage] naive_diffimage_flag` | true | the naive subtraction runs, a diagnostic |
 | `[difference.fake_sources] inject_fake_sources_flag` | false | fake-source injection is not ported; true is refused |
+| `reference_psf` | `"measured"` | the reference PSF ZOGY is given: `"measured"`, from the resampled reference's stars, or `"science"`, the science PSF for both images, for a field whose reference stars cannot be measured (the GBTDS bulge, [datasets](datasets)) |
 | `[difference.statistics] clip_correction_seed` | 0 | seeds the clipped-statistics correction's random draw; negative is `dev`'s unseeded draw |
 
 With the seed pinned, the same inputs give the same pixels and rows,