foundation, intake, loop, stage-contract, decisions, index: the HLWAS run from SOC L2 files to the Kafka topic

AGENTS.md

@@ -200,7 +200,7 @@ authority; `decisions.md` records the team's rulings; `foundation.md`
 and `stage-contract.md` are the two model pages everything else builds
 on, and `foundation.md` governs where an older page still describes the
 retired run model; each pipeline stage has its own page (`reference`,
-`difference`, `load`, `maintain`, `crossmatch`, `statistics`,
+`difference`, `intake`, `load`, `maintain`, `crossmatch`, `statistics`,
 `finalize`, `alerts`, `prune`, `export`); `photometry.md` is not a
 stage page, since the rebuild carries no photometry stage, and holds
 the forced-photometry note instead; `tool.md`, `loop.md` and

system/alerts.md

@@ -311,6 +311,11 @@ position within it and within the container.
 
 ## Kafka
 
+The foundation publishes packets to Kafka with its own verb, `rapidpipe
+publish`, after the packet stage; the [foundation](foundation) page
+states it under Publication to Kafka. What follows is the retired
+stage's record.
+
 `[publish] kafka` defaults to `false`. Setting it `true` exits 64,
 "Kafka publication is not enabled in this build": the same refusal
 `dev`'s own pipeline path takes, since `dev` never constructs a Kafka

system/decisions.md

@@ -9,6 +9,26 @@ page where attribution matters.
 
 ## Rulings
 
+Ben's rulings of 2026-10-21 evening on the instance type and the plan
+in force. The [foundation](foundation) page states the design they
+shaped, and the [stage contract](stage-contract) page the execution
+record.
+
+(decision-instance-type)=
+**The instance type is recorded and no family is pinned**: every job
+records the instance type it ran on beside its image digest, and the
+compute environment is not pinned to one instance family.
+(Ben, 2026-10-21 20:41 PDT)
+
+(decision-hlwas-direction)=
+**The exercised HLWAS pipeline is the plan in force**: the direction
+question is set aside; the plan is a heavily exercised pipeline from
+ASDF intake through Kafka publication, with the data entirely within
+SMDC and nothing imported from another centre, starting with the SOC
+HLWAS simulation without source injection, and iterating and ramping
+once it runs end to end. The test is that the chain has the right
+shape and functions on simple data. (Ben, 2026-10-21 20:50 PDT)
+
 Ben's rulings of 2026-10-21 on the night through the chain. The
 [foundation](foundation) page states the design they shaped, and the
 [stage contract](stage-contract) page the retry and the PSF input.

system/index.md

@@ -17,22 +17,23 @@ repository's history. The rebuild starts from this page.
 | Document | Status | Scope |
 |---|---|---|
 | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided |
-| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review |
-| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes and a job on Batch with its retry rule; the thin chain, with the `import`, `associate` and `packet` stages and the reader; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure and the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo); the ten validation checks a night meets and their outcomes; the night's measurements; what is not yet decided |
-| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest and peak memory; exit codes and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract |
+| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review |
+| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes and a job on Batch with its retry rule and instance type; the thin chain, with the `intake`, `import`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs); the ten validation checks, the script that runs six of them and their outcomes; the night's and the ramp's measurements; what is not yet decided, source injection as the next step among it |
+| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest, instance type and peak memory; exit codes, a refusal because the run is not live exiting 0, and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract |
 | [`reference.md`](reference) | DRAFT | The `reference` stage on `run(job)`: its scope and `l2files` inputs, the awaicgen coadd and SExtractor catalog in `dev`'s order, the header stamp, the three declared tables and the drawn `rfid`, its settings, and what it measured |
 | [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, one differencer published, its settings and determinism, and what it measured against `dev` |
+| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the declared `exposures` and `l2files` rows with the packed `expid` and the shared exposures key, its settings, and what the ramp measured |
 | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block |
 | [`maintain.md`](maintain) | DRAFT | The `maintain` stage: the once-per-date CLUSTER and ANALYZE of a `sources` child table moved out of `load`, its `detector-date` unit, its manifest and exit codes |
 | [`crossmatch.md`](crossmatch) | DRAFT | The `crossmatch` stage as ported from `dev`: one field's two passes, the catalog a pass reads (base plus delta), the per-field tables and their set-scoped keys, what lands in `astroobjects` and `merges`, the association set, its settings and exit codes |
 | [`statistics.md`](statistics) | DRAFT | The `statistics` stage as ported from `dev`: per-object position and flux statistics over an association set's base-plus-delta membership, what lands in `astroobjectsmeta`, the statistics-set result set, its settings and exit codes |
 | [`finalize.md`](finalize) | DRAFT | The `finalize` stage as ported from `dev`'s post-processing pipeline: the chain order, the manifest it republishes as new instances, the header stamp, its settings and exit codes |
-| [`alerts.md`](alerts) | DRAFT | The `alerts` stage as ported from `dev`'s alert production: the input-set manifest, what it reads from the source, association and statistics sets, the container and the alert outbox, its registration, settings and exit codes |
+| [`alerts.md`](alerts) | DRAFT | The retired `alerts` stage as ported from `dev`'s alert production: the input-set manifest, what it reads from the source, association and statistics sets, the container and the alert outbox, its registration, settings and exit codes; its Kafka section points at the foundation's publication |
 | [`prune.md`](prune) | DRAFT | The `prune` stage: the not-best merge exclusion ported from `dev`'s `pruneNotBestMerges`, its `pruned-set` result set and `prunedmerges` rows, the run model's own-run exclusion clause, its settings and exit codes |
 | [`photometry.md`](photometry) | DRAFT | Forced photometry: the alert-driven measurement inside the `packet` stage, and `dev`'s standalone `forcedPhotometryForField.py`, not ported |
 | [`export.md`](export) | DRAFT | The `export` stage as ported from `dev`'s `generateSourceHATSCatalog.py`: the named source sets it reads, the CSV dump and the `hats-import` build, the `catalog-export` bundle and registration, its settings and exit codes; the light-curve catalog is not ported |
 | [`tool.md`](tool) | DRAFT | The command-line tool `rapidpipe`, every subcommand and its flags, the operations mapped from the specification's Tools sentence onto subcommands, the input-set composer, the tool's own exit codes, personal submission from a workstation, and where it runs |
-| [`loop.md`](loop) | DRAFT | The processing-date loop: the operations-registry trigger, the venue and spec it runs from, the production run and stage chain per date, base-catalog binding across dates, promotion, its own records, and release binding |
+| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from intake to Kafka, pending, failed and drained, the per-field chains, its exit codes and what it measured; it replaces the run model's processing-date loop |
 | [`operations.md`](operations) | DRAFT | Operations proposals the team has not yet ruled on: what they assume about the mission, three shapes of the run model and the recommended named stream of batch runs, replacement scope and the chain switch, batch cadence against the latency requirement, a proposed production check policy, reference eligibility and selection, staged inputs of a production run, capacity lanes |
 | [`observability.md`](observability) | DRAFT | The `rapidpipe` log line and where it goes, the record of an attempt with and without CloudWatch, monitoring with `rapidpipe` alone, queue, execution and orchestration time per Batch job, profiling a stage, and today's job-duration baseline against a 30-minute target |
 
@@ -45,6 +46,7 @@ foundation
 stage-contract
 reference
 difference
+intake
 load
 maintain
 crossmatch

system/intake.md

@@ -0,0 +1,131 @@
+# The intake stage
+
+**Status: DRAFT**
+
+The `intake` stage turns one SOC L2 file, a calibrated Roman image in
+ASDF, into the L2 product the rest of the chain reads: a gzipped FITS
+file in `dev`'s layout, one `l2files` row and one `exposures` row. It is
+where a run's data enter the record, so a run that starts from intake
+owns every byte from the raw file to the packet. The
+[foundation](foundation) page states the records it works within, and
+the [stage contract](stage-contract) page what a stage is.
+
+The stage is `rapidpipe/stages/intake.py` on the pipeline repository's
+`rebuild` branch. Its conversion, `rapidpipe/science/intake.py`, is
+ported from `dev`'s `pipeline/ingestL2Files.py` with `dev`'s logic
+unchanged.
+
+## Scope and inputs
+
+A job's scope is one file of one dataset: `{"dataset": ..., "file":
+<stem>}`, where the stem is the SOC file name without its
+`_cal.asdf` suffix. The file itself is read from the run's source
+root, `[intake] source_root`, a URL prefix given in the run's settings
+overlay at `run create` and never in the shipped settings. A public
+source root is read anonymously when `[intake] unsigned` is true; a
+signed read is not tried after an anonymous one fails.
+
+The job names one input by key: the `psfs` set holding the science PSF
+of the file's filter and SCA, used for the limiting magnitude. The
+`_wcs.asdf` sidecar SOC delivers beside each file is not read, since
+the calibrated file carries the same gWCS, fit results and metadata.
+
+## What it runs
+
+1. Fetch `<source_root><stem>_cal.asdf` through `prefix.get`.
+2. Parse the stem's program, plan, pass, segment, observation, visit,
+   exposure, SCA and filter, and check them against the file's own
+   observation metadata and detector. A contradiction fails the job
+   before any manifest.
+3. Convert with `dev`'s `build_fits_file`: a 16-HDU FITS whose primary
+   header holds every scalar of the ASDF metadata as a `HIERARCH` card;
+   the science array in DN with a degree-5 SIP distortion fitted to the
+   gWCS, every coefficient to fifth order present; the other arrays
+   scaled by their exposure-time exponent, half-precision arrays
+   widened; and the tables as binary tables. The uncompressed file is
+   made in a temporary directory and only the gzipped one is kept.
+4. Gzip at `[intake] gzip_level`.
+5. Measure the limiting magnitude with the science PSF, as `import`
+   does.
+6. Derive the image centre, corners, tessellation field, overlapping
+   fields and HEALPix indexes, as `import` does.
+7. Write the rows and the manifest.
+
+The fitted SIP reproduces the gWCS to well under the fit's own 0.1
+pixel bound: on a real file, to below 1e-5 arcsec at the corners and
+the centre. The difference stage reformats the science image itself,
+so intake does not.
+
+## Declared outputs
+
+| Output | Key | Notes |
+|---|---|---|
+| the gzipped FITS | | under the attempt's prefix; the `l2files` row's `filename` is its URL |
+| `l2files` | dataset, expid, sca, fid | the file's footprint, field, overlapping fields, HEALPix indexes, WCS columns and limiting magnitude |
+| `exposures` | dataset, expid | one per exposure |
+
+The manifest records the file's provenance: its observation
+identifiers, romancal's version and the summary of its Gaia WCS fit
+(status, geometry, matches, rms and mean absolute error, rotation),
+with the conversion's bytes and step times.
+
+**The `expid`.** `dev` draws an `expid` from a sequence keyed by the
+observation date. In the foundation the key is the producer's, and a
+SOC exposure's own identifier, 23 digits, fits no `integer` column, so
+intake packs the identifiers that vary within a program and plan:
+`pass * 10**6 + observation * 10**4 + visit * 10**2 + exposure`, each of
+observation, visit and exposure below 100. The 18 SCAs of one exposure
+share it, as they share `dev`'s one `Exposures` row.
+
+**The shared `exposures` key.** Because the 18 SCAs share an
+`exposures` key, only one job may expect it with no open binding. The
+loop has the first intake of an exposure expect it and every other
+intake of that exposure leave it out of its expects. Every intake still
+writes the row, so 17 of each exposure's 18 rows are unbound
+candidates.
+
+The `SWVERS` keyword is `job <id>`: the job names its run, and the run
+its code revision.
+
+## Settings
+
+The run's `[intake]` table, with `dev`'s hard-coded values as keys:
+
+| Key | Value | `dev` |
+|---|---|---|
+| `source_root` | the run's overlay only | |
+| `unsigned` | false | |
+| `sip_degree` | 5 | `SIPDISTORTIONDEGREE`'s default |
+| `sip_max_pix_error` | 0.1 | the `to_fits_sip` argument |
+| `gzip_level` | 1 | `L2GZIPLEVEL`'s default |
+| `program_id` | -1 | the `SOCSIMS` program id |
+| `[intake.nominal_zptmag]` | `dev`'s fallback zeropoints | used only when the file has no photometry metadata |
+| `limmag_nsigma`, `limmag_clip_sigma` | 5.0, 3.0 | as `[import]` has them |
+
+## What it measured
+
+On one 204 MB F146 file, in a Batch container: the anonymous fetch
+took 0.8 s, the conversion 2.7 s to a 421 MB FITS, gzip at level 1 5.5
+s to 195 MB, the limiting magnitude 3.5 s, the rows 1.0 s, upload and
+publish 1.5 s; the stage peaked at 906 MiB. On a host whose working
+directory is network storage, conversion and gzip took about 4 s each.
+
+Over the F146 ramp on SCA 8, 312 intakes ran in 17.1 to 24.6 s each,
+median 20.8 s, at 910 MiB, writing 60.9 GB; their median queue wait was
+144 s, because the compute environment scales up from nothing and every
+job reserves the difference stage's 24 GiB. The 426 intakes of the run
+all published on their first attempt.
+
+## Not decided here
+
+- Gzip is single-threaded and is the stage's largest step on Batch; a
+  parallel compressor, or no compression, since the difference stage
+  unpacks the file, would roughly halve the stage.
+- Intake peaks at under a twentieth of what its job definition
+  reserves. A smaller reservation for intake would pack many more per
+  instance, and several files per job would cut the queue wait that
+  dominates it.
+- Whether an intake that does not expect the `exposures` key should
+  write the row at all.
+- The `l2files` row and the PSF lookup are built by both `import` and
+  `intake`; one helper would hold them once.