psf, foundation, loop, intake, decisions, index: the psf stage, the six other filters as datasets of their own, the check script

AGENTS.md

@@ -200,7 +200,7 @@ authority; `decisions.md` records the team's rulings; `foundation.md`
 and `stage-contract.md` are the two model pages everything else builds
 on, and `foundation.md` governs where an older page still describes the
 retired run model; each pipeline stage has its own page (`reference`,
-`difference`, `intake`, `load`, `maintain`, `crossmatch`, `statistics`,
+`difference`, `intake`, `psf`, `load`, `maintain`, `crossmatch`, `statistics`,
 `finalize`, `alerts`, `prune`, `export`); `photometry.md` is not a
 stage page, since the rebuild carries no photometry stage, and holds
 the forced-photometry note instead; `tool.md`, `loop.md` and

system/decisions.md

@@ -23,7 +23,8 @@ injection first, were named and not taken. (Ben, 2026-10-22 PDT)
 (decision-breadth-before-injection)=
 **The other filters before injection**: after the housekeeping and the
 disjoint reference, the other filters run without injection, each
-image's PSF from its file's CRDS ePSF reference, and injection follows.
+filter a dataset of its own with its own live run, each image's PSF
+from its file's CRDS ePSF reference, and injection follows.
 This brings the first half of the breadth step forward, for experience
 running the pipeline over several filters. (Ben, 2026-10-22 16:03 PDT)
 

system/index.md

@@ -17,12 +17,13 @@ repository's history. The rebuild starts from this page.
 | Document | Status | Scope |
 |---|---|---|
 | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided |
-| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review |
-| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; the thin chain, with the `intake`, `import`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound); the ten validation checks, the script that runs six of them and their outcomes; the night's, the ramp's and the housekeeping's measurements with the disjoint reference's counts; what is not yet decided |
+| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked; the other filters as datasets of their own), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review |
+| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; the thin chain, with the `intake`, `import`, `psf`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound), and the breadth run over the six other filters, one dataset each, and why; the ten validation checks, the script that runs six of them, the check script for a run and their outcomes; the night's, the ramp's, the housekeeping's and the breadth run's measurements, with the null test's residuals labelled as artefacts; what is not yet decided, cross-band association among it |
 | [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest, instance type, Batch job id and peak memory; exit codes, a refusal because the run is not live exiting 0, and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract |
 | [`reference.md`](reference) | DRAFT | The `reference` stage on `run(job)`: its scope and `l2files` inputs, the awaicgen coadd and SExtractor catalog in `dev`'s order, the header stamp, the three declared tables and the drawn `rfid`, its settings, and what it measured |
 | [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, one differencer published, the gzipped files it keeps and why, its settings and determinism, and what it measured against `dev` |
-| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings, and what the ramp measured |
+| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the check that its science PSF was made from the file's own ePSF reference, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings, and what the ramp measured |
+| [`psf.md`](psf) | DRAFT | The `psf` stage on `run(job)`: the CRDS ePSF reference a file names, fetched by that name; the fixed reduction to a pipeline PSF; the science and reference kinds it declares and the `EPSFREF` the intake checks; its settings; the F146 comparison with `dev`'s copied PSF and the job's measurements; the position, the reference PSF rule and a reference part in the key, not yet decided |
 | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block |
 | [`maintain.md`](maintain) | DRAFT | The `maintain` stage: the once-per-date CLUSTER and ANALYZE of a `sources` child table moved out of `load`, its `detector-date` unit, its manifest and exit codes |
 | [`crossmatch.md`](crossmatch) | DRAFT | The `crossmatch` stage as ported from `dev`: one field's two passes, the catalog a pass reads (base plus delta), the per-field tables and their set-scoped keys, what lands in `astroobjects` and `merges`, the association set, its settings and exit codes |
@@ -33,7 +34,7 @@ repository's history. The rebuild starts from this page.
 | [`photometry.md`](photometry) | DRAFT | Forced photometry: the alert-driven measurement inside the `packet` stage, and `dev`'s standalone `forcedPhotometryForField.py`, not ported |
 | [`export.md`](export) | DRAFT | The `export` stage as ported from `dev`'s `generateSourceHATSCatalog.py`: the named source sets it reads, the CSV dump and the `hats-import` build, the `catalog-export` bundle and registration, its settings and exit codes; the light-curve catalog is not ported |
 | [`tool.md`](tool) | DRAFT | The command-line tool `rapidpipe`, every subcommand and its flags, the operations mapped from the specification's Tools sentence onto subcommands, the input-set composer, the tool's own exit codes, personal submission from a workstation, and where it runs |
-| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from intake to Kafka with the listing by the selection, the Batch reconcile and one flush, pending, failed and drained, the reference bound and held fields, the per-field chains in admission order, its exit codes and what it measured; it replaces the run model's processing-date loop |
+| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from the psf submission and hold through intake to Kafka with the listing by the selection, the Batch reconcile, association from the field's official chain and a flush per set, pending, failed and drained, the reference bound and held fields, the per-field chains in admission order, its exit codes and what it measured, six filters at once among it; it replaces the run model's processing-date loop |
 | [`operations.md`](operations) | DRAFT | Operations proposals the team has not yet ruled on: what they assume about the mission, three shapes of the run model and the recommended named stream of batch runs, replacement scope and the chain switch, batch cadence against the latency requirement, a proposed production check policy, reference eligibility and selection, staged inputs of a production run, capacity lanes |
 | [`observability.md`](observability) | DRAFT | The `rapidpipe` log line and where it goes, the record of an attempt with and without CloudWatch, monitoring with `rapidpipe` alone, queue, execution and orchestration time per Batch job, profiling a stage, and today's job-duration baseline against a 30-minute target |
 
@@ -47,6 +48,7 @@ stage-contract
 reference
 difference
 intake
+psf
 load
 maintain
 crossmatch

system/intake.md

@@ -27,7 +27,8 @@ source root is read anonymously when `[intake] unsigned` is true; a
 signed read is not tried after an anonymous one fails.
 
 The job names one input by key: the `psfs` set holding the science PSF
-of the file's filter and SCA, used for the limiting magnitude. The
+of the file's filter and SCA, used for the limiting magnitude and for
+the check that it was made from the file's own ePSF reference. The
 `_wcs.asdf` sidecar SOC delivers beside each file is not read, since
 the calibrated file carries the same gWCS, fit results and metadata.
 
@@ -46,11 +47,17 @@ the calibrated file carries the same gWCS, fit results and metadata.
    widened; and the tables as binary tables. The uncompressed file is
    made in a temporary directory and only the gzipped one is kept.
 4. Gzip at `[intake] gzip_level`.
-5. Measure the limiting magnitude with the science PSF, as `import`
+5. Check that the science PSF was made from this file's own ePSF
+   reference: the basename of the file's `meta.ref_file.epsf` card must
+   be the PSF file's `EPSFREF`, which the [psf](psf) stage writes and
+   `dev`'s copied F146 files also carry. A mismatch fails the job before
+   any manifest, naming both; a PSF or a file that names no reference
+   is accepted.
+6. Measure the limiting magnitude with the science PSF, as `import`
    does.
-6. Derive the image centre, corners, tessellation field, overlapping
+7. Derive the image centre, corners, tessellation field, overlapping
    fields and HEALPix indexes, as `import` does.
-7. Write the rows and the manifest.
+8. Write the rows and the manifest.
 
 The fitted SIP reproduces the gWCS to well under the fit's own 0.1
 pixel bound: on a real file, to below 1e-5 arcsec at the corners and