From f9474178540c3bbff84b7fbc8ec166ad7f758ddd Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Fri, 23 Oct 2026 16:07:28 +0000 Subject: [PATCH] psf, foundation, loop, intake, decisions, index: the psf stage from each file's CRDS ePSF reference, the six other filters as datasets of their own, the check script, and their measurements with the null test's residuals labelled as artefacts Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- system/decisions.md | 3 +- system/foundation.md | 232 ++++++++++++++++++++++++++++++++++++------- system/index.md | 10 +- system/intake.md | 15 ++- system/loop.md | 72 +++++++++++--- system/psf.md | 180 +++++++++++++++++++++++++++++++++ 7 files changed, 451 insertions(+), 63 deletions(-) create mode 100644 system/psf.md diff --git a/AGENTS.md b/AGENTS.md index 090b56a..c87f2a6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -200,7 +200,7 @@ authority; `decisions.md` records the team's rulings; `foundation.md` and `stage-contract.md` are the two model pages everything else builds on, and `foundation.md` governs where an older page still describes the retired run model; each pipeline stage has its own page (`reference`, -`difference`, `intake`, `load`, `maintain`, `crossmatch`, `statistics`, +`difference`, `intake`, `psf`, `load`, `maintain`, `crossmatch`, `statistics`, `finalize`, `alerts`, `prune`, `export`); `photometry.md` is not a stage page, since the rebuild carries no photometry stage, and holds the forced-photometry note instead; `tool.md`, `loop.md` and diff --git a/system/decisions.md b/system/decisions.md index 58cf7a8..22d7ac2 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -23,7 +23,8 @@ injection first, were named and not taken. (Ben, 2026-10-22 PDT) (decision-breadth-before-injection)= **The other filters before injection**: after the housekeeping and the disjoint reference, the other filters run without injection, each -image's PSF from its file's CRDS ePSF reference, and injection follows. +filter a dataset of its own with its own live run, each image's PSF +from its file's CRDS ePSF reference, and injection follows. This brings the first half of the breadth step forward, for experience running the pipeline over several filters. (Ben, 2026-10-22 16:03 PDT) diff --git a/system/foundation.md b/system/foundation.md index f2179cb..683d5f5 100644 --- a/system/foundation.md +++ b/system/foundation.md @@ -6,18 +6,19 @@ This page states how RAPID records what it made and which of it is official, and the narrow chain of stages built on that record, from stored images to alert packets. It is the design the `rebuild` branch of the pipeline repository implements: `database/schema.sql`, the -package `rapidpipe.foundation`, the stages `intake`, `import`, +package `rapidpipe.foundation`, the stages `intake`, `import`, `psf`, `reference`, `difference`, `associate` and `packet`, the loop, the Kafka publication, the reader `scripts/read_packets.py`, the run -validation `scripts/validate_run.py` and the queries in -`database/queries.sql`. +validation `scripts/validate_run.py`, the check script +`scripts/check_run.sh` and the queries in `database/queries.sql`. The [stage contract](stage-contract) page states what a stage is under it, and the [decisions](decisions) page records the rulings it rests on. The chain has run end to end on SMDC's database and AWS Batch over one field's whole night, 69 epochs, and then from SOC L2 ASDF files to the Kafka topic over 426 epochs of a simulated wide-area survey, and again after the housekeeping that made each epoch cheaper and gave the -reference a bound; the last sections give the procedure, the checks +reference a bound, and then over the simulation's six other filters, +each with PSFs the pipeline derives itself; the last sections give the procedure, the checks they met and what they measured. ## In plain terms @@ -386,6 +387,7 @@ measured peak, and the difference stage keeps the definition's 24 GiB: | Stage | Asked, MiB | Peak measured on the ramp, MiB | |---|---|---| | `intake` | 2,048 | 910 | +| `psf` | 2,048 | 333, at breadth | | `reference` | 6,144 | 2,831 | | `difference` | 24,576 | 16,125 | | `associate` | 1,024 | 238 | @@ -421,9 +423,12 @@ Each commit that runs on Batch has its own image, tagged registered by the infrastructure repository's release hooks. A run's revision names the commit; a fix to the code is a new commit, a new image, and a new run whose jobs are submitted afresh, so the run and -job records always name the code that ran. A Batch stage names only -files in the inputs bucket and outputs under the scratch bucket's root, -the two places the job role can read. +job records always name the code that ran. The new run, made live for +the same dataset, continues from the dataset's official rows, so a fix +in the middle of a drain resumes where the old run stopped. A Batch +stage reads only files in the inputs bucket, the run's public source +root and the CRDS server, and writes only under the scratch bucket's +root. ## The thin chain @@ -433,8 +438,9 @@ image. Each simplification keeps the shape of the full stage. | Stage | Reads (named in `job_inputs`) | Writes (declared outputs) | Left out until breadth | |---|---|---|---| -| `intake` | one SOC L2 `_cal.asdf` file under the run's source root; the science PSF of its filter and SCA | the gzipped 16-HDU FITS converted as `dev` converts it; `l2files`, and `exposures` when the job expects that key, with limiting magnitude, overlapping fields and sky indices computed | other filters, for want of their PSFs | +| `intake` | one SOC L2 `_cal.asdf` file under the run's source root; the science PSF of its filter and SCA | the gzipped 16-HDU FITS converted as `dev` converts it; `l2files`, and `exposures` when the job expects that key, with limiting magnitude, overlapping fields and sky indices computed | nothing for want of a PSF: every filter's is derived by `psf` | | `import` | stored products for one field: L2 files, difference images, PSFs, the reference and its mosaic catalog, the Gaia copy; the cap on sources per image | copies of the image files; `exposures`, `l2files`, `diffimages`, `psfs`, `refimages`, `refimcatalogs`, `catalogs` and `sources` (the brightest N per difference image by flux over its error), with limiting magnitude, overlapping fields and sky indices computed; the source system's ids kept; provenance marked unknown | other fields and epochs; recovering provenance | +| `psf` | one SOC L2 file's metadata and the CRDS ePSF reference it names | the science and the reference PSF, one stamp twice; two `psfs` rows | a PSF per position on the detector; a reference PSF of the coadd's own | | `reference` | the field's `l2files` rows for one filter and SCA, before the run's reference bound when it has one | the awaicgen mosaic with its coverage and uncertainty maps and SExtractor catalog; `refimages`, `refimimages` (one row per coadded L2 row) and `refimcatalogs` | `dev`'s later reference changes (infobits, status, the 9100-pixel mosaic); the Photutils catalog | | `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and one `psfs` row of each kind, `science` and `reference`; the cap on sources | the gzipped ZOGY difference image, its uncertainty, the mask, the uncapped catalog and the files `packet` reads beside it, SFFT and naive run only for their outcome bits; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s | sumrat, real/bogus and the extra SFFT catalog columns; `dev`'s loader prefilter | | `associate` | the epoch's source sets for the field and its eight neighbours; the predecessor object set, or none; the exclusions | `objects.parquet` (every object of the field, id carried forward or position-hashed, founding position, sufficient sums) and `matches.parquet` (this epoch's memberships) | `rubrat`, `sumrat`; field parallelism | @@ -469,6 +475,15 @@ from the file's name, and its settings. Where `import` copies products another system made, `intake` makes the L2 product from the raw file, so a run that starts from intake owns its data from the first stage. +### psf + +The [psf](psf) page states `psf`: its scope (one file of the filter and +SCA it serves), the ePSF reference fetched by the name the file +carries, the fixed reduction to a 61-pixel stamp, the two `psfs` rows +it declares, and the `EPSFREF` keyword the intake checks against each +file's own reference. The loop submits one for each filter and SCA that +lacks its PSFs, so a dataset needs no PSF imported. + ### reference and difference The [reference](reference) page states `reference`: its scope (field, @@ -552,10 +567,17 @@ the packet's own block, so each message decodes on its own and carries its schema. There is no schema registry. The producer waits for every acknowledgement from all in-sync replicas, then stamps the set's job with the time (`jobs.kafka_published_at`). One call opens one database -connection and one producer for every set it finds, sends them all, -flushes once, and then stamps each set whose every acknowledgement came -back. A set with a failed acknowledgement stays unstamped, the others -are stamped, and the verb exits 70 naming it. A crash between the last +connection and one producer, then takes its sets one at a time: it +sends a set's messages, flushes within 120 s, and stamps the set when +every acknowledgement came back. A set with a failed acknowledgement +stays unstamped, the others are stamped, and the verb exits 70 naming +it. A flush that does not finish in time leaves its set and every later +one unstamped and stops the call, which exits non-zero, and the next +call sends them again; closing the producer is bounded the same way. +This replaced one flush per call: with five loops publishing at once, +batches waited in the client past its delivery timeout, and the +client's expiry path then killed its own sender thread and left the +flush waiting with no end. A crash between the last acknowledgement and the stamp offers the set again on the next call, so a consumer may see a packet twice; replay is allowed and there is no exactly-once delivery. A set that loses its binding before it is @@ -570,13 +592,21 @@ the shipped settings file or the one `--settings` names, not from the run's body. `scripts/consume_packets.py --run ` is the consumer check. It reads -every partition from its earliest offset to the end offset at its -start, with no consumer group; keeps the messages whose key names one -of the run's stamped sets; and requires each to decode as one record -valid against its index row's schema, to equal the outbox bytes for its -key, and every stamped packet to be seen at least once. Zero kept is a -failure. It holds one message and one set's outbox at a time, so its -memory is one set's, whatever the topic's size. +each partition from the first offset at or after the earliest end time +of the run's stamped packet jobs, since a set is published only after +its job has ended, to the end offset at its start, with no consumer +group; keeps the messages whose key names one of the run's stamped +sets; and requires each to decode as one record valid against its index +row's schema, to equal the outbox bytes for its key, and every stamped +packet to be seen at least once. Zero kept is a failure, so a run with +no stamped set fails. The client's own record checksum is switched off: +every kept message is compared byte for byte with the outbox anyway, +and the client computes that checksum in pure Python, which held the +read to about 50 messages a second against 805 without it. The check +holds one message and one set's outbox at a time, so its memory is one +set's, whatever the topic's size. It holds only within the topic's +retention, seven days: a run whose messages have aged out fails it as +never read. Today the producer runs where the loop runs, on a host, after the packet stage has published. Producing from the packet stage's own Batch @@ -602,7 +632,8 @@ outside the image overrides them in its own body. Data locations are never in the shipped file: the import's source directory is its job's scope, the intake's source root is the run's own `[intake] source_root`, given in the settings overlay at `run create`, and the -outputs root is the environment's. `[batch.memory_mib]` holds each +outputs root is the environment's. `[psf]` names the CRDS server and +the stamp's size ([psf](psf)). `[batch.memory_mib]` holds each stage's memory ask (A job on Batch, above). `[reference.selection] before_mjd`, the reference bound, is documented in the shipped file and absent there: a run sets it in its overlay, and a run without it @@ -656,7 +687,7 @@ minutes. The same observations on synthetic rows run in CI under The SMDC database `rapid` holds the foundation in its `public` schema, and the pipeline repository's `docs/foundation-smdc-step.md` is the -procedure, one command and then its check per step. It has eight parts. +procedure, one command and then its check per step. It has nine parts. **The schema.** `public` is dropped and recreated from `schema.sql` in one transaction, through the pooler, as the database owner. Everything @@ -846,6 +877,50 @@ the GBTDS run stayed live and untouched, and nothing was deleted. the recorded inputs naming a catalog gained their dataset; the bound products were unchanged. +**Breadth.** The ninth part ran the simulation's six other filters, +F087, F106, F129, F158, F184 and F213, without injection +({ref}`ruling `), first on SCA 8 of +all 54 pointings and then on the 18 SCAs of the first pointing. + +- *The PSF from each file's reference.* The [psf](psf) stage was proven + on F146 first, under a dataset of its own with no PSF imported: the + loop's first pass submitted one psf job and no intake, the next + admitted the pointing's intakes, and the chain drained with every + check clean. Every filter's PSFs were then derived by the loop, 109 + pairs over seven datasets, and every intake's check of its file's own + reference held. +- *One dataset per filter.* Each filter ran as a dataset of its own + with its own live run and its own reference bound, the midpoint of + that filter's third and fourth passes, so every field has exactly + three reference frames in every filter. A filter's chains, fixes and + resumes are its own: three fixes resumed five filters while the + sixth, already drained, stayed as it was. The shape avoids cross-band + association by construction. A source seen in two filters is two + objects in two datasets, and no packet carries forced photometry from + another band; whether an object should span bands is open (Not + decided here, below). +- *SCA 8 over the footprint.* The six loops drained 3,456 epochs: 324 + references, 2,484 differences, associates and packet sets, and + 503,964 packets on the topic. Three failures seen at this width were + fixed in code and each resumed by a new live run of the same + datasets: the parameter store throttled 3,400 jobs starting in one + minute; one flush per publishing pass hung the producer (Publication + to Kafka, above); and the loop's association rule read only its own + run's jobs, so a resumed run associated every epoch a second time. + That run was taken off live, the state before it restored in one + change group with nothing deleted, and the rule now follows the + field's official chain ([loop](loop)). A validation false failure on a + source catalogued twice was a tolerance below the one-pass variance's + rounding floor, and was corrected. Sixteen jobs that Batch failed to + start were rerun by hand. +- *The 18 SCAs of one pointing.* The same six runs took 1,088 more + epochs, with 102 psf jobs in the first pass, no code change, and one + transient failure: the CRDS server answered 72 of 102 fetches at once + with gateway timeouts, and the 72 published when rerun in waves. +- *The checks.* Every run of the six filters is clean by the check + script, after the consumer check stopped reading the topic from its + beginning (Validation, below). + ## Validation A night is one coherent account when it meets ten checks. Each is a @@ -908,6 +983,14 @@ Parquet and decodes every packet set eight at a time, each worker on its own connection, with the same assertions: on the wide-area run it took 666 s against 1,512 s. +`scripts/check_run.sh --run ` runs a run's three checks in turn, +`validate_run.py`, the reader over the run's bound packet sets eight at +a time, and the consumer check, streams their output, and prints one +line, `Check run : CLEAN|FAIL validate / reader +/ consumer / `, exiting non-zero +when any of the three fails. Every run of the breadth part was checked +with it, and every one was clean. + ## Measurements Measured 2026-10-21 on SMDC, field 4658155, SCA 8, filter W146, every @@ -1009,10 +1092,11 @@ the difference stage's 24 GiB for every stage. minutes, at most 99 containers at once, 11,711 packets, median queue waits 86 s for intake and 98 s for difference. - **The null test.** Per epoch the difference stage found 798 to 1,207 - detections before the cap of 500, median 954, with 53 to 119 above - S/N 20. They sit at and around stars, the residual a PSF made for - another simulation leaves; none is variability. SCAs 4 and 5 leave - three to four times the bright residuals of the other SCAs. + artefact detections before the cap of 500, median 954, with 53 to 119 + above S/N 20, none of them variability. They sit at and around stars, + the residual a PSF mismatch leaves, and on unmasked pixel defects. + SCAs 4 and 5 leave three to four times the artefact residuals of the + other SCAs. ### Housekeeping @@ -1038,12 +1122,14 @@ the same pixels. | `reference` in the container | 76.4 s, 6 frames | 46.0 s, 3 frames | - **The disjoint reference.** On one pointing's SCA 8, the three epochs - of the fourth to sixth passes left 113, 114 and 128 residuals above - S/N 20 against a three-frame reference made from the first three + of the fourth to sixth passes left 113, 114 and 128 artefact + residuals above S/N 20, PSF mismatch and pixel defects rather than + detections, against a three-frame reference made from the first three passes, against 72, 75 and 100 under the wide-area run's six-frame reference, which held each epoch; the same PSF and pixels, the reference the only change. The three epochs gave 551 packets - against 447. The bright-residual fraction rose from 0.144 to 0.226. + against 447. The artefact fraction above S/N 20 rose from 0.144 to + 0.226. Whether depth or disjointness does it is not separated (Not decided here, below). - **Packing has a cost in seconds.** An intake took 39 s with six @@ -1055,6 +1141,70 @@ the same pixels. partition from the earliest offset, so its time grows with the topic, whatever the run's size. +### Breadth + +Measured 2026-10-23 on SMDC over the six other filters of the wide-area +simulation, one dataset each, SCA 8 of the 54 pointings and then the 18 +SCAs of the first pointing, every PSF derived by the `psf` stage. + +The residual counts below are artefact counts, not detections. The +simulation has no time-variable sources, and an image study of one F146 +epoch found about three quarters of its packets on stars, where the +science PSF and the resampled reference's differ (the reference's stars +are about 15% wider, and the stage hands ZOGY one PSF for both), and +most of the brightest on unmasked pixel defects; noise accounted for +essentially none of them, and the pattern repeats from epoch to epoch. +The table gives the median per differenced epoch of these artefact +residuals above S/N 20 and their fraction of the 500 kept sources, for +the 294.1 s exposures, with the 107.5 s exposures of the last two passes +as "short": + +| Filter | SCA 8, 54 pointings | 18 SCAs, first pointing | SCA 4 / SCA 5 | Packets | Run prefixes | +|---|---|---|---|---|---| +| F146 | 79 (0.158), one pointing's three epochs | 79 (0.158), the wide-area run, a reference holding the epoch and `dev`'s PSF | 352 / 256 | | | +| F087 | 110 (0.220) | 103 (0.207) | 401 / 324 | 27,644 | 154 GB | +| F106 | 97 (0.194); short 44 (0.088) | 97 (0.195); short 49 (0.098) | 393 / 229 | 180,389 | 534 GB | +| F129 | 92 (0.184); short 44 (0.088) | 118 (0.235); short 54 (0.108) | 500 / 180 | 183,535 | 535 GB | +| F158 | 148 (0.296); short 78 (0.156) | 161 (0.322); short 86 (0.172) | 500 / 270 | 203,892 | 571 GB | +| F184 | 333 (0.665) | 214 (0.428) | 500 / 500 | 31,913 | 155 GB | +| F213 | 33 (0.101), 333 kept | 44 (0.127), 394 kept | 66 / 166 | 14,631 | 148 GB | + +- **What the table shows**, descriptively, since filter, depth and PSF + change together across its rows: F087, F106 and F129 leave artefact + fractions near F146's; F158 leaves more and F184 three to four times + as many; F213 is the shallowest band and does not reach the cap. The + short exposures halve the artefact fraction in each band that has + them and carry about a quarter fewer packets per epoch. SCA 4 holds + the most artefact residuals of the 18 in five of the six filters, and + every kept source of SCA 4 is above S/N 20 in F129, F158 and F184. + SCA 9, differenced against SCA 8's reference because the two share a + field, leaves more artefact residuals than SCA 8 in five of the six, + where in F146 the two were level. +- **In all**: 4,544 epochs ingested, 3,266 differenced, 643,004 + packets, 2.1 TB of run prefixes (0.45 to 0.46 GB per epoch), and a + database of 2.74 GB. +- **Per stage on SCA 8**, medians across the six filters: intake 61 to + 79 s in the container after a 244 to 305 s queue wait, the six + filters' 3,456 intakes submitted at once; reference 99 to 125 s on + three frames; difference 181 to 289 s, longer with more containers + sharing an instance; associate 4 to 9 s; packet 24 to 31 s at chain + depth 2 and 128 to 141 s at depths 13 and 14, its memory flat at under + 900 MiB. Peak memory stayed within the F146 figures in every stage. The + compute environment chose 2xlarge to 8xlarge instances of the m6a, + r6i and r7i families. +- **Forced rows grow with the chain**: about 300 per epoch at depth 2 + and 1,400 at depth 13, 3.2 million over SCA 8. +- **Kafka**: about 30 packets a second per loop, one set at a time, + with up to six loops publishing together. The topic keeps seven days, + unbounded by size. +- **The checks.** The consumer check had read the topic from its + beginning at about 40 messages a second, which made one run's check + four hours and growing; the cause was the client's pure-Python record + checksum. With the checksum off and the read starting at the run's + first packet, a run's consumer check took 4 to 109 minutes, by how + much of the topic followed its first packet. `validate_run.py` grows + with the chain, 1,850 to 2,710 s at depths 13 and 14. + **Not measured, for want of data on the host that ran it**: the matcher and the difference stage at the plan's density; a thirty-day window of real cadence; completeness against the injected sources; and the rubrat, @@ -1091,23 +1241,29 @@ computes it yet. simulation has six F146 epochs per pointing, too few for a six-frame reference disjoint from three. The PSF mismatch at stars is under both. -- Where the reference bound goes for each filter of a footprint. The - midpoint between the third and fourth passes leaves three F146 epochs - on each side. +- Where the reference bound goes in general. On the HLWAS simulation + each filter's bound is the midpoint of its own third and fourth + passes, which leaves three reference epochs before it in every filter; + a survey without passes needs another rule. - The grant and the path for producing packets to Kafka from the packet stage's own Batch job ({ref}`ruling `). The compute environment and the brokers share a security group that admits itself; the Batch job role and its permission boundary hold no Kafka action. -- PSFs for the other filters. Only F146's exist on SMDC; each image's - PSF is to come from the CRDS ePSF reference its file names - ({ref}`ruling `), and the conversion from ePSF to the - PSF the difference stage reads is new code with no `dev` original. +- Cross-band association. Each filter is a dataset of its own, so a + source seen in two filters is two objects. Whether an object is per + band or across bands is open, and with it forced photometry in the + other bands at a source's position. +- The null test's artefacts: the PSF mismatch between the science PSF + and the resampled reference ([psf](psf), Not decided here) and pixel + defects the intake does not mask. Masking an L2 file's data-quality + flags at intake, and rejecting outliers in the reference coadd, are + not built. +- The 107.5 s exposures of the last two passes are differenced against + a 294.1 s reference. Whether a short exposure gets a reference of its + own exposure time, or is differenced at all, is open. - Source injection, outside the pipeline as a dataset of its own ({ref}`ruling `), and its completeness measure. -- The `psfs` identity sequence meets ids an import copied from a source - system: an import that draws an id already written fails at the - publish once, and its rerun draws past it. - Whether SFFT and the naive subtraction keep running now that only their outcome bits remain ([difference](difference)). - Packets after a replacement stay as published with the old diff --git a/system/index.md b/system/index.md index 742f95b..5765392 100644 --- a/system/index.md +++ b/system/index.md @@ -17,12 +17,13 @@ repository's history. The rebuild starts from this page. | Document | Status | Scope | |---|---|---| | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided | -| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | -| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; the thin chain, with the `intake`, `import`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound); the ten validation checks, the script that runs six of them and their outcomes; the night's, the ramp's and the housekeeping's measurements with the disjoint reference's counts; what is not yet decided | +| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked; the other filters as datasets of their own), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | +| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; the thin chain, with the `intake`, `import`, `psf`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound), and the breadth run over the six other filters, one dataset each, and why; the ten validation checks, the script that runs six of them, the check script for a run and their outcomes; the night's, the ramp's, the housekeeping's and the breadth run's measurements, with the null test's residuals labelled as artefacts; what is not yet decided, cross-band association among it | | [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest, instance type, Batch job id and peak memory; exit codes, a refusal because the run is not live exiting 0, and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract | | [`reference.md`](reference) | DRAFT | The `reference` stage on `run(job)`: its scope and `l2files` inputs, the awaicgen coadd and SExtractor catalog in `dev`'s order, the header stamp, the three declared tables and the drawn `rfid`, its settings, and what it measured | | [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, one differencer published, the gzipped files it keeps and why, its settings and determinism, and what it measured against `dev` | -| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings, and what the ramp measured | +| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the check that its science PSF was made from the file's own ePSF reference, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings, and what the ramp measured | +| [`psf.md`](psf) | DRAFT | The `psf` stage on `run(job)`: the CRDS ePSF reference a file names, fetched by that name; the fixed reduction to a pipeline PSF; the science and reference kinds it declares and the `EPSFREF` the intake checks; its settings; the F146 comparison with `dev`'s copied PSF and the job's measurements; the position, the reference PSF rule and a reference part in the key, not yet decided | | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block | | [`maintain.md`](maintain) | DRAFT | The `maintain` stage: the once-per-date CLUSTER and ANALYZE of a `sources` child table moved out of `load`, its `detector-date` unit, its manifest and exit codes | | [`crossmatch.md`](crossmatch) | DRAFT | The `crossmatch` stage as ported from `dev`: one field's two passes, the catalog a pass reads (base plus delta), the per-field tables and their set-scoped keys, what lands in `astroobjects` and `merges`, the association set, its settings and exit codes | @@ -33,7 +34,7 @@ repository's history. The rebuild starts from this page. | [`photometry.md`](photometry) | DRAFT | Forced photometry: the alert-driven measurement inside the `packet` stage, and `dev`'s standalone `forcedPhotometryForField.py`, not ported | | [`export.md`](export) | DRAFT | The `export` stage as ported from `dev`'s `generateSourceHATSCatalog.py`: the named source sets it reads, the CSV dump and the `hats-import` build, the `catalog-export` bundle and registration, its settings and exit codes; the light-curve catalog is not ported | | [`tool.md`](tool) | DRAFT | The command-line tool `rapidpipe`, every subcommand and its flags, the operations mapped from the specification's Tools sentence onto subcommands, the input-set composer, the tool's own exit codes, personal submission from a workstation, and where it runs | -| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from intake to Kafka with the listing by the selection, the Batch reconcile and one flush, pending, failed and drained, the reference bound and held fields, the per-field chains in admission order, its exit codes and what it measured; it replaces the run model's processing-date loop | +| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from the psf submission and hold through intake to Kafka with the listing by the selection, the Batch reconcile, association from the field's official chain and a flush per set, pending, failed and drained, the reference bound and held fields, the per-field chains in admission order, its exit codes and what it measured, six filters at once among it; it replaces the run model's processing-date loop | | [`operations.md`](operations) | DRAFT | Operations proposals the team has not yet ruled on: what they assume about the mission, three shapes of the run model and the recommended named stream of batch runs, replacement scope and the chain switch, batch cadence against the latency requirement, a proposed production check policy, reference eligibility and selection, staged inputs of a production run, capacity lanes | | [`observability.md`](observability) | DRAFT | The `rapidpipe` log line and where it goes, the record of an attempt with and without CloudWatch, monitoring with `rapidpipe` alone, queue, execution and orchestration time per Batch job, profiling a stage, and today's job-duration baseline against a 30-minute target | @@ -47,6 +48,7 @@ stage-contract reference difference intake +psf load maintain crossmatch diff --git a/system/intake.md b/system/intake.md index 7838dde..211ab7e 100644 --- a/system/intake.md +++ b/system/intake.md @@ -27,7 +27,8 @@ source root is read anonymously when `[intake] unsigned` is true; a signed read is not tried after an anonymous one fails. The job names one input by key: the `psfs` set holding the science PSF -of the file's filter and SCA, used for the limiting magnitude. The +of the file's filter and SCA, used for the limiting magnitude and for +the check that it was made from the file's own ePSF reference. The `_wcs.asdf` sidecar SOC delivers beside each file is not read, since the calibrated file carries the same gWCS, fit results and metadata. @@ -46,11 +47,17 @@ the calibrated file carries the same gWCS, fit results and metadata. widened; and the tables as binary tables. The uncompressed file is made in a temporary directory and only the gzipped one is kept. 4. Gzip at `[intake] gzip_level`. -5. Measure the limiting magnitude with the science PSF, as `import` +5. Check that the science PSF was made from this file's own ePSF + reference: the basename of the file's `meta.ref_file.epsf` card must + be the PSF file's `EPSFREF`, which the [psf](psf) stage writes and + `dev`'s copied F146 files also carry. A mismatch fails the job before + any manifest, naming both; a PSF or a file that names no reference + is accepted. +6. Measure the limiting magnitude with the science PSF, as `import` does. -6. Derive the image centre, corners, tessellation field, overlapping +7. Derive the image centre, corners, tessellation field, overlapping fields and HEALPix indexes, as `import` does. -7. Write the rows and the manifest. +8. Write the rows and the manifest. The fitted SIP reproduces the gWCS to well under the fit's own 0.1 pixel bound: on a real file, to below 1e-5 arcsec at the corners and diff --git a/system/loop.md b/system/loop.md index c9ac818..c27b1a3 100644 --- a/system/loop.md +++ b/system/loop.md @@ -19,8 +19,9 @@ classification or commands, this page governs. The loop looks at the run's source root and at the database, decides what can run now, submits it to Batch without waiting, and goes to -sleep. Two minutes later it looks again. A file nobody has taken in -gets an intake job; a field with all its images taken in gets a +sleep. Two minutes later it looks again. A filter and SCA with no PSF +gets a job that derives one, and its files wait for it; a file nobody +has taken in gets an intake job; a field with all its images taken in gets a reference; an image with a reference gets a difference; and each field's association and packets advance one epoch at a time, oldest first. At the end of every pass, whatever packet sets became official @@ -67,12 +68,19 @@ full. Over the HLWAS simulation that is 23,398 keys instead of 272,160. Its order: -1. **intake**: one job per listed file the selection admits that no +1. **psf**: the pass's new files are grouped by filter and SCA, and a + pair without both an official science and an official reference PSF + gets one [psf](psf) job, named by the pair's first new file, unless a + psf job of the run already serves it. That SCA's files are held from + intake until both rows are official; they still count as new, so the + pass is not drained while they wait. A selection needs no PSF + imported: its first pass derives what is missing. +2. **intake**: one job per listed file the selection admits that no published intake of the dataset and no intake job of this run has taken, at most `--max-intake` per pass. Each names its filter's and SCA's science PSF. The first intake of an exposure expects the shared `exposures` key; the others leave it out. -2. **reference**: one job per field and filter with at least +3. **reference**: one job per field and filter with at least `[reference.selection] min_frames` official L2 rows, no official reference and none pending or failed, and only while no intake of the run is pending, so a reference coadds every epoch the drain @@ -80,27 +88,32 @@ Its order: the first in order with enough frames; the job's scope records which. With a reference bound set, the rules change as described under [The reference bound](#the-reference-bound). -3. **difference**: one job per official L2 row whose field has an +4. **difference**: one job per official L2 row whose field has an official reference and whose key has no official difference and no difference job of the run pending or failed. With a bound, only rows at or after it. -4. **associate**: per field, one job for the earliest official source - set, by `mjdobs`, that no associate of the run has named, with the - field's open object set as its predecessor. It waits while an +5. **associate**: per field, one job for the earliest official source + set, by `mjdobs`, that is not yet associated, with the field's open + object set as its predecessor. A source set is associated when it is + in the field's official chain, walked back from the official object + set through each associate's recorded predecessor, whatever run made + it, or when an associate of this run naming it has not yet + published. So a new live run of the same dataset continues the + chains where the last one stopped and never associates an epoch + twice. It waits while an associate of the field is pending, while an earlier selected L2 row of the field lacks its source set, and while any intake of the run is pending, since an intake's field is known only once it publishes. -5. **packet**: per field, one job for the earliest associated epoch +6. **packet**: per field, one job for the earliest associated epoch without a packet set, once every earlier epoch's packet set is official, because a packet set reuses its predecessors' forced rows. It names the reference, the run's Gaia catalog, the science PSF, the object set, the epoch's source set, the field's earlier packet sets, and the field's difference images and L2 rows within the stage's forced window. -6. **Kafka**: `publish` over the run's official, unstamped packet sets +7. **Kafka**: `publish` over the run's official, unstamped packet sets ([foundation](foundation), Publication to Kafka), on one connection - with one flush for every set the pass made official. An idle pass - opens no producer. + and one producer, one set at a time. An idle pass opens no producer. Fields advance independently, so one pass can carry many fields at different depths, and a packet job for epoch *n* goes out beside the @@ -201,6 +214,30 @@ On SMDC, 2026-10-22, over the HLWAS simulation's F146 data: chain took 20 minutes for about 21 seconds of associate containers, and the same across 52 fields at once. +On SMDC, 2026-10-23, the six other filters of the same simulation, one +dataset and one loop each, all six at once: + +| Selection | Epochs | Jobs | Loop start to drained | +|---|---|---|---| +| every pointing, SCA 8, six filters | 3,456 | 12,686, 1,448 of them intakes failed and re-planned | 3 h 21 min for all six, with three code fixes and their redeploys; one filter, interrupted once, 49 min | +| one pointing, all 18 SCAs, six filters | 1,088 more | 3,632 | 1 h 22 min, 9 min of it psf reruns | + +- **A psf job** adds one queue wait before a pair's first intake: two + passes, about four minutes, against an imported PSF. +- **A pass can submit thousands of jobs.** The six loops' second passes + submitted 3,456 intakes in a minute, and 1,448 of them failed reading + the database parameters from the parameter store, which throttled + them; each job's read now backs off for minutes, with botocore's own + retry setting, and none failed after. `--max-intake` is the lever on + the burst. +- **Publishing at depth sets the pace.** About 30 packets a second per + loop, one set at a time; a pass of 54 sets of about 205 packets took + about six minutes, so a chain of 13 or 14 advanced one link each six + minutes on 54 fields at once, and about two minutes on 17. +- **One failed job stops a loop**: it exits 2, and every + other SCA of that filter waits for the rerun. 72 psf jobs failing at + once on the CRDS server stopped all six loops for nine minutes. + ## Not decided here - Where the loop runs on a schedule, and at what cadence, once real @@ -210,9 +247,14 @@ On SMDC, 2026-10-22, over the HLWAS simulation's F146 data: one Batch job. At GBTDS depth this is the loop's latency. - An unbounded run's field with fewer than `min_frames` rows still waits without a `held` report; only a bounded run reports it. -- Where the bound goes for each filter of a footprint, and whether - a reference disjoint from its epochs costs residuals by its depth - or by its disjointness ([foundation](foundation), Not decided here). +- Where the bound goes in general. On the HLWAS simulation each + filter's bound is the midpoint of its own third and fourth passes at + the first pointing, a run setting per filter; a survey without passes + needs another rule. Whether a reference disjoint from its epochs + costs residuals by its depth or by its disjointness is open too + ([foundation](foundation), Not decided here). +- Whether a failed job should stop only its own filter and SCA rather + than the whole loop. - A reconcile of a Batch job that Batch no longer describes is unit-tested only; it has not been seen live. - The mission's own delivery layout and manifest, once the interface is diff --git a/system/psf.md b/system/psf.md new file mode 100644 index 0000000..e5194ca --- /dev/null +++ b/system/psf.md @@ -0,0 +1,180 @@ +# The psf stage + +**Status: DRAFT** + +`psf` derives the PSF the difference stage reads for one filter and SCA +of a dataset, from the CRDS ePSF reference the dataset's own L2 file +names, and declares it twice, once as the science PSF and once as the +reference PSF. Each image's PSF comes from that reference because it is +the calibration's PSF and what flight data will carry +({ref}`ruling `); a simulation's rendering PSF +is not a product. The reduction from the ePSF to a pipeline PSF is new +science with no `dev` original. The stage is `rapidpipe/stages/psf.py` +on the pipeline repository's `rebuild` branch. The +[foundation](foundation) page states the records it works within, the +[stage contract](stage-contract) page what a stage is, and the +[loop](loop) page when a job of it is submitted. + +## Scope and inputs + +The scope is `{"dataset": ..., "file": }`, where the stem is any +one SOC calibrated file of the filter and SCA the job serves; the loop +names the first new stem of the pair. The job names no `input` rows. +It reads two things from outside the record: + +- the calibrated file, from the run's `[intake] source_root`, anonymously + when `[intake] unsigned` is true, as the [intake](intake) stage reads + it. Only its metadata is read: the filter, the detector, the CRDS + context it was calibrated under, and `meta.ref_file.epsf`, the ePSF + reference it names (`crds://roman_wfi_epsf_0276.asdf` for F146 on SCA + 8 of the HLWAS simulation, under context `roman_0046.pmap`); +- that reference, fetched **by the name the file carries** from the + CRDS server in `[psf] crds_server`, at + `/unchecked_get/references/roman/`. There is no rmap + match, no context lookup and no `crds` package: the file has already + chosen its reference, and the stage takes that choice as given. + +The reference is one ASDF file per filter and SCA, 178.6 MB. Its +`roman.psf` array has shape (3 defocus values, 3 spectral types, 9 +positions, 361, 361): a 3 x 3 grid of positions on the detector, +oversampled four times, made with stpsf 2.0.0. The stage refuses the +job before a manifest when the reference's `meta.instrument.detector` +or `optical_element` is not the calibrated file's. + +The reference is fetched into a temporary directory beside the job's +working directory, not into it, because everything in the working +directory is uploaded and nothing downstream reads the reference. It is +removed when the stage returns. Nothing caches it between jobs. + +## The reduction + +The conversion is fixed, not configured. The stage takes the in-focus +plane for a G2V star, `psf[focus 0, G2V]`, as a photutils +`GriddedPSFModel` over the reference's nine positions, the recipe +romanisim 0.14.0 uses, built directly so that romanisim is not a +dependency. It evaluates the model at the detector centre, (2048, 2048), +on `oversample x size` points per axis at sub-pixel centres, offset by +`(j + 0.5) / oversample - size / 2` from the centre, so the PSF's centre +is the stamp's centre: the central pixel for an odd `size`, the central +corner for an even one. Each `oversample x oversample` block is summed +into one pixel, and the stamp is normalised to unit sum. `oversample` is +the reference's own `meta.oversample`, 4; `size` is the one setting. + +The block sum is right for these files because the stored grid holds +flux per oversampled sample, not a surface brightness. In the F146 SCA +8 reference the in-focus G2V image at the centre sums to 0.979 over its +361 x 361 samples, and the same model sampled once per detector pixel +sums to 0.066, about one sixteenth. Summing a pixel's sixteen samples +gives that pixel's flux; sampling at pixel centres would not. + +The centroid is kept where the reference puts it. At the detector +centre of the F146 SCA 8 reference it sits 0.027 and 0.043 pixel from +the stamp's centre, and the stage does not re-centre it. + +## Declared outputs + +Two FITS files under the attempt's prefix, the same stamp in both: + +| File | `PSFKIND` | `psfs` key | +|---|---|---| +| `sciimage_psf__sca.fits` | `sciimage_psf` | (dataset, fid, sca, `science`) | +| `refimage_psf__sca.fits` | `refimage_psf` | (dataset, fid, sca, `reference`) | + +and `psfs.parquet` with the two rows, keyed as an imported PSF is +keyed, so the difference stage reads a derived PSF exactly as it reads +a copied one. The `psfid` is drawn from the table's identity, and the +`import` stage now draws its own the same way rather than writing a +source system's id, so a derived PSF never meets an id an import wrote. + +Each file's header names the reference (`EPSFREF`, the keyword `dev`'s +copied F146 files already carry), the calibrated file's CRDS context +(`CRDSCTX`), the position (`PSFX`, `PSFY`), the oversampling +(`EPSFOVS`) and the block (`EPSFBIN`). The manifest adds the focus, the +spectral type, the size, the reference's bytes, the flux the stamp held +before normalising, the centroid offset and the time of each step. + +`EPSFREF` is a contract with the [intake](intake) stage: an intake +refuses a file, before a manifest, whose own `meta.ref_file.epsf` is +not the reference its science PSF names. So "each file's own +reference" is checked on every file. A PSF that names no +reference, or a file that names none, is accepted. + +## Settings + +| Setting | Value | Meaning | +|---|---|---| +| `[psf] crds_server` | `https://roman-crds.stsci.edu` | the CRDS server the reference is fetched from, the public server by default | +| `[psf] size` | 61 | the stamp's side in detector pixels, the `NAXIS1` of `dev`'s copied F146 science PSF | +| `[batch.memory_mib] psf` | 2,048 | the stage's memory ask on Batch | + +The focus, the spectral type, the position and the block are not +settings: they are the conversion, and changing one is a change of +science. + +## What it measured + +On SMDC on 2026-10-23, over the HLWAS simulation. + +**The stamp.** For F146 on SCA 8 the 61 x 61 stamp holds 0.9710 of the +model's flux before normalising; the reference's own 361-sample grid +holds 0.979, and a 181-pixel stamp 0.9793, so 61 pixels hold 99.15% of +what the model holds. Beside `dev`'s copied F146 PSF, which is an +ensemble over twelve sub-pixel phases: + +| | Peak | Centroid offset x, y (pixel) | Encircled within 1, 2, 3, 5 pixels | +|---|---|---|---| +| derived by this stage | 0.344 | -0.027, -0.043 | 0.569, 0.719, 0.820, 0.874 | +| `dev`'s copied PSF | 0.240 | +0.094, +0.081 | 0.534, 0.711, 0.816, 0.874 | + +The derived core is sharper and the wings agree. + +**The F146 comparison.** Two runs differenced the same three epochs of +one pointing's SCA 8, passes four to six, against the same three-frame +reference of passes one to three under the same bound, one with `dev`'s +copied PSF and one with the derived PSF, the PSF the only change. The +residuals above S/N 20 per epoch fell from 113, 114 and 128 to 80, 79 +and 75, and the fraction of the 500 kept rows above S/N 20 from 0.23 to +0.26 to 0.15 to 0.16; every kept row now clears S/N 5 (500 of 500, +against 414 to 454), and the three epochs gave 628 packets against 551. +These are artefact counts, not detections: the simulation has no +time-variable sources, and an image study of one of these epochs found +about three quarters of its packets on stars, where the science image's +PSF and the resampled reference's differ, and most of the brightest on +unmasked pixel defects, with noise accounting for essentially none. The +derived PSF's lower count is a smaller PSF mismatch at the stars, not a +cleaner image. It is within the factor of two the comparison allowed. + +**The job.** Alone on Batch: the reference fetched in 3.18 s, the +calibrated file's metadata read in 2.05 s, the model built in 2.93 s, +the reduction 0.12 s and the rows 0.26 s, 10.9 s in the container +after a 186 s queue wait, at a peak of 302 MiB. The prefix holds 43.7 +KB. With six fetching at once the container took 38 to 40 s; with 102 +at once, 53 to 118 s, and 72 of the 102 failed on the CRDS server's +gateway timeouts (70 HTTP 504, two read timeouts) and published when +rerun in waves of 12 and 24. Over seven datasets, 109 jobs derived 109 +pairs, and every intake's `EPSFREF` check held: the simulation names one +reference per filter and SCA. + +## Not decided here + +- **The position.** Every image's PSF is the reference's model at the + detector centre. The reference holds a 3 x 3 grid of positions, so a + PSF per quadrant, or one evaluated per source, is possible; which the + difference stage should read is the plan's question. +- **The reference PSF rule.** The reference PSF is the science PSF, + the same ePSF, because a reference coadds frames of the same filter + and SCA. The resampled coadd's stars are wider than that: second-moment + width 1.10 pixels against the science image's 0.96 on F146 SCA 8, + which leaves the core-and-ring residual at every star. A reference + PSF measured from the coadd's own stars, or built from its frames' + positions and dithers, is not built. +- **A reference-file part in the `psfs` key.** The key is (dataset, + fid, sca, kind), with no reference name, because the simulation names + one ePSF per filter and SCA. A dataset whose files name two + references for one filter and SCA would see its first intake refused + on `EPSFREF`, and that refusal is what would earn the key change. +- **The fetch.** Each job fetches the whole 178.6 MB reference for a + 42 MB array of which it reads one plane, and every dataset fetches + its references again. A server error under a wide fan-out fails the + job; mapping it to exit 75, which Batch retries, or reading the + reference once per reference rather than once per job, is open.