diff --git a/system/decisions.md b/system/decisions.md index 67db063..7f4ed16 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -9,6 +9,31 @@ page where attribution matters. ## Rulings +Ben's rulings of 2026-10-21 on the night through the chain. The +[foundation](foundation) page states the design they shaped, and the +[stage contract](stage-contract) page the retry and the PSF input. + +(decision-latency-later)= +**Latency is later**: latency and performance optimisation are a later +task. The current focus is coherent, validated machinery end to end; +every run takes metrics and records improvements it sees, and no +measurement gates anything. (Ben, 2026-10-21) + +(decision-psf-kind)= +**The PSF key gains a kind**: a `psfs` row is `science` or `reference`, +and the kind is in its scientific key, so the two PSFs of one filter +and SCA are distinct products. (Ben, 2026-10-21) + +(decision-read-role)= +**The read role selects**: the team's read role has `SELECT` on every +table and view of the foundation's schema, and on those created later. +(Ben, 2026-10-21) + +(decision-batch-retry)= +**Batch retries exit 75 and nothing else**: three attempts, the first +run and two retries, on the stage contract's temporary dependency +failure; every other exit ends the job. (Ben, 2026-10-21) + Ben's ruling of 2026-10-21 on the SMDC step. The [foundation](foundation) page states the procedure it shaped. diff --git a/system/difference.md b/system/difference.md index 4900777..f4ad094 100644 --- a/system/difference.md +++ b/system/difference.md @@ -2,196 +2,145 @@ **Status: DRAFT** -This page describes the stage as ported to the older contract. It now -runs on the foundation's `run(job)` contract, which the -[foundation](foundation) page's `difference` section states; where this -page describes registration, custody, runs or promotion, that page -governs. - -The `difference` stage takes one detector image and the reference for -its field, subtracts them with ZOGY exactly as the `dev` science pipeline -does, and writes the difference, uncertainty and significance images -and four source catalogs. SFFT and a plain subtraction also run, as in -`dev`. The stage writes no database rows and reads the database only -for the read guard; `register` records its manifest -afterwards. - -The stage lands on the pipeline repository's `rebuild` branch -(`rapidpipe/stages/difference.py`, `rapidpipe/settings/difference.toml`, -`rapidpipe/products/diffimage.py`, `rapidpipe/db/diffimages.py`). Every -ported stage minimises differences from `dev` and designs in, off by -default, anything that could be left unused. The [foundation](foundation) -page fixes the vocabulary and field list used here. - -## Inputs - -The `--inputs` manifest is an input set: its `outputs` list the products -the attempt consumes, with their files beside the manifest. - -| Entry | Members | Registration block | -|---|---|---| -| one `l2-image` | `image`, the delivered file (gzipped or not) | as `admit` writes it; the stage reads the exposure time, the info bits, the centre and the corners | -| one `reference-image` | `image`, `coverage`, `uncertainty` | `infobits`; `rfid`, the legacy `refimages` row of a reference registered by `dev`, else null | -| one `reference-catalog` | `catalog`, the reference's SExtractor catalog | none; the stage reads its FWHM column | -| two `psf` | `psf` | none; each input-set entry's role, `applies_to`, is `science` or `reference` | +`difference` subtracts one epoch's L2 image from the reference its job +names, with ZOGY, catalogs the result, and declares the difference +image and its source set for the publisher. SFFT and the naive +subtraction run beside ZOGY as `dev` runs them, and their files stay +under the prefix for diagnosis. The stage ports the science of `dev`'s +`pipeline/awsBatchSubmitJobs_runSingleSciencePipeline.py`, refreshed to +`dev` at d6aac515, under the foundation's `run(job)` contract: +`rapidpipe/stages/difference.py` and `rapidpipe/science/difference/` on +the pipeline repository's `rebuild` branch. The [foundation](foundation) +page states the records it works within, and the +[stage contract](stage-contract) page what a stage is. + +## Scope and inputs + +The scope is `{"expid", "sca", "fid"}`, the epoch's key without the +dataset, which is the run's. The job names exactly four `input` rows, +read in one short transaction and fetched through `prefix.get`: + +| Product | Which | +|---|---| +| `l2files` | the epoch's own row | +| `refimages` | one reference of the scope's filter, imported or made by the [reference](reference) stage; its coverage and uncertainty maps sit beside the mosaic, and its mosaic catalog (`refimcatalogs`, `cattype` 1) gives the reference FWHM | +| `psfs`, kind `science` | the science image's PSF for the scope's filter and SCA | +| `psfs`, kind `reference` | the reference image's PSF for the same filter and SCA | -Every member's size and SHA-256 is checked before use; a mismatch, a -missing entry or a malformed block exits 65. +The two PSFs are told apart by the `kind` in their key, read from the +row, so the stage never reads a PSF file's header to decide which is +which. Any other product, a second input of one kind, or an input whose +key is not the scope's fails the job before a manifest is written. ## What it runs The steps follow `dev`'s order, one module each under -`rapidpipe.science.difference`. External tools run in the stage's work -directory with `dev`'s file names and command lines, so each tool sees -what it saw in `dev`. - -The delivered image is moved to the primary HDU, padded to an odd size -and converted to DN/s, with a simple-model uncertainty image. SExtractor -catalogs the science image for its FWHM. The science image's SIP -distortion is rewritten as PV, and SWarp resamples the reference image, -coverage map and uncertainty image onto the science grid. bkgest then -subtracts the science background. - -Gain matching compares the two images' SExtractor catalogs to find the -reference's scale factor and the median offsets between the images. -It uses the science image's own zero point and the reference's `MAGZP` -header keyword, stamped on the coadd by the reference stage, unless -`[awaicgen] zprefimg` overrides that value. With too few matched -sources, it falls back to these zero points and zero offsets. - -NaNs in ZOGY's inputs are replaced and extreme artifact pixels in the -science image are repaired; the reference is shifted by the median -offsets. ZOGY runs by subprocess, as in `dev`. Its difference and -significance images are masked where the reference coverage is below -threshold, the replaced NaNs are restored, and negative copies are made. -The stage then makes the difference's uncertainty image and the positive -and negative SExtractor and Photutils catalogs. - -SFFT then runs in its own environment, and the naive subtraction after -it, each with its own catalogs. An SFFT failure is not fatal: the stage -continues and notes the failure in the execution record's `notes`. - -## Outputs - -ZOGY produces one `difference-image` instance with members `difference`, -`uncertainty`, `significance` and `psf`, and four `source-catalog` -entries naming it: SExtractor positive and negative, Photutils positive -and negative. A Photutils catalog that could not be made has no entry; -its bit in the catalog-outcome mask records the failure. - -`[sfft] register_sfft` is on by default: when SFFT -succeeds, a second `difference-image` instance carries its result, with -members `difference`, `uncertainty`, and `psf` and `kernel` where SFFT -wrote them, and its own four catalogs. Turning the setting off keeps -SFFT's files as diagnostics only. A failed SFFT run adds nothing either -way: never a partial bundle. Promotion chooses which registered instance -is current downstream; that choice is outside this stage. The naive -subtraction's files stay diagnostics, never registered as an instance. - -Every working file stays under `diff/` in the attempt's output location, -as `dev` uploads its intermediates for diagnosis. - -## Registration - -`register` maps the difference-image entry's `registration` block to -columns as follows. The `source-catalog` block is `{"source_count": n}`; -only its instance row is written before `load`. - -| Block field | Meaning | Column | -|---|---|---| -| `centre`, `corners` | the image centre and four corners (RA, Dec), bottom-left first; the l2 instance's values, as `dev` registers the science image's | `ra0`, `dec0` to `ra4`, `dec4` | -| `catalog_outcome_bits` | `dev`'s six-bit mask, one bit per differencer and sign with no Photutils catalog | `infobitssci` | -| `infobits_science` | the l2 image's quality bits | carried, not registered | -| `infobits_reference` | the reference's info bits | `infobitsref` | -| `source_counts` | `{sextractor, photutils}` by `{positive, negative}`; null where a catalog was not made | `diffimmeta.source_counts`; the SExtractor positive count also in `nsexcatsources` | -| `registration_residual` | `x_rms`, `y_rms`: the measured astrometric residual RMS from gain matching, not the value ZOGY itself is fed; `x_median`, `y_median`: the measured offsets applied | `dxrmsfin`, `dyrmsfin`, `dxmedianfin`, `dymedianfin` | -| `reference_scale_factor` | the gain-matching scale factor applied to the reference | `scalefacref` | -| `detection_role` | the member the catalogs were detected on | carried | -| `md5` | the primary member's MD5 | `checksum` | -| `reference_rfid` | the legacy `refimages` row, for a reference registered by `dev` | used to resolve `rfid` | - -The remaining columns come from lookups and allocations: - -| Column | Source | +`rapidpipe.science.difference`, and every external tool runs in the +work directory with `dev`'s file names and command lines. + +The L2 image is moved to the primary HDU and converted to DN/s, with a +simple-model uncertainty image. SExtractor catalogs it for the FWHM of +its point sources. Its SIP distortion is rewritten as PV, and SWarp +resamples the reference, its coverage and its uncertainty onto the +science grid. bkgest subtracts the background of the science image and +of the resampled reference. Gain matching compares the two images' +SExtractor catalogs for the reference's scale factor and the median +offsets, using the science image's own zero point and the reference's +`MAGZP`; with too few matched sources it falls back to the zero points +and zero offsets. + +NaNs in ZOGY's inputs are replaced, extreme artifact pixels are +repaired, and the reference is shifted by the median offsets. ZOGY runs +by subprocess. Its difference and significance images are masked where +the reference coverage is below threshold, the NaNs are restored, and +negative copies are made. The stage then makes the difference's +uncertainty image and the positive and negative SExtractor and +Photutils PSF-fit catalogs. SFFT and the naive subtraction follow with +their own catalogs; an SFFT failure is recorded in the manifest and is +not fatal. + +## Declared outputs + +Every working file stays under `diff/` in the attempt's prefix, as +`dev` uploads its intermediates; `packet` reads the ZOGY difference +image's neighbours there (its uncertainty map, `diffpsf.fits`, the +background-subtracted science image and the gain-matched reference). +The manifest declares two tables: + +| Table | Rows | |---|---| -| `rid`, `expid`, `sca`, `field`, `fid`, `jd` | the l2 instance's `l2files` row; `jd` is its `mjdobs` plus 2400000.5, as `dev`'s `addDiffImage` computes it | -| `rfid` | the reference instance's `refimages` row, through the `instance` column the migration `20260923-02-refimages-instance.sql` adds; for a reference registered by `dev`, the block's `reference_rfid` | -| `ppid` | the differencer: `zogy` is 15, the science pipeline's row, as in `dev`; `sfft` is 16, priority 6, script `sfft_rapid_rimtimsim.py`, its own `pipelines` row added by the migration `20260924-01-pipelines-sfft.sql` | -| `hp6`, `hp9` | derived from the centre, on both tables | -| `version` | the next number for (`rid`, `ppid`) within the run | -| `svid` | the `swversions` row whose `cvstag` is the run's code revision, made on first use | -| `vbest`, `status` | 0 and 0 | -| `filename` | the primary member under the attempt's output location | -| `run`, `attempt`, `instance` | the manifest and the registering attempt, on both tables | +| `diffimages` | one, the ZOGY image, keyed (dataset, expid, sca, fid); `rid`, `rfid`, `field`, `jd`, the centre and four corners, `infobitssci` (`dev`'s catalog-outcome mask, one bit per differencer and sign with no PSF-fit catalog), `infobitsref`, `status` 1, the file's MD5 | +| `sources` | the ZOGY positive PSF-fit catalog, mapped onto the `sources` columns exactly as the import maps `dev`'s, the brightest `max_sources` by flux over its error (0 keeps every source) | + +The stage draws `pid` from the `diffimages` identity sequence while it +reads its inputs, so the source rows carry it. A run has one +differencer, so the SFFT and naive images are never published as rows. +A missing ZOGY PSF-fit catalog publishes an empty source set with its +outcome bit set, where `dev`'s loader skips the job. The manifest also +records the zero point and where it came from, both FWHMs, the scale +factor, the registration residuals, the catalog counts, the count of +sources before the cap, and SFFT's outcome. ## Settings -`rapidpipe/settings/difference.toml` holds every `dev` setting a step -reads, with `dev`'s value from -`cdf/awsBatchSubmitJobs_launchSingleSciencePipeline.ini` on `dev`. -The table below lists them except for the four tool tables -(`[sextractor_sciimage]`, `[sextractor_gainmatch]`, -`[sextractor_diffimage]`, `[swarp]`), which copy `dev`'s sections -verbatim. Settings new with the port are marked. +The run's `[difference]` table in `rapidpipe/settings/foundation-thin.toml` +holds `max_sources` (500) and `dev`'s fit-position bounds, and one +subtable per `dev` ini section with `dev`'s values. The tool tables +(`[difference.sextractor_sciimage]`, `_gainmatch`, `_diffimage`, +`_diffimage_sfft`, `_diffimage_naive` and `[difference.swarp]`) copy +`dev`'s sections verbatim. The others, where a value is new with the +port or departs from `dev`: -| Setting | Default | Meaning | +| Setting | Value | Meaning | |---|---|---| -| `[paths] rapid_sw`, `cfg_path` | `/code`, `/code/cdf` | where the pipeline image puts the repository and its tool configuration files | -| `[paths] python` | empty | new: the interpreter for `py_zogy.py`; empty is the stage's own (`dev` hard-codes `/usr/bin/python3.11`) | -| `[paths] zogy_code` | `/code/modules/zogy/v21Aug2018/py_zogy.py` | ZOGY | -| `[paths] bkgest_code`, `bkgest_include_dir` | `bkgest`, `/opt/rapid/share/bkgest` | bkgest, on `PATH`, and its include files as the pipeline image installs them; `dev` uses `/code/c/bin/bkgest` and `/code/c/include`, absent from the image (rapid #93) | -| `[paths] sextractor`, `swarp` | `sex`, `swarp` | the tools, on `PATH` | -| `[paths] work_subdirectory` | `diff` | new: the working directory under the attempt's output location | -| `[instrument] sca_gain`, `sca_readout_noise` | 2.0, 9.4 | the socsims values | -| `[sci_image] saturation_level` | 2500000.0 | DN | -| `[sci_image] repair_extreme_artifact_pixels`, `extreme_artifact_threshold` | true, 10000.0 | artifact repair before differencing | -| `[ref_image] saturation_level` | 100000.0 | `dev` reads `[SEXTRACTOR_REFIMAGE] sextractor_SATUR_LEVEL` | -| `[awaicgen] zprefimg` | empty | an explicit override of the reference zero point gain matching uses; empty (the default) reads it from the reference image's own `MAGZP` header keyword instead (see [reference](reference)) | -| `[zogy] astrometric_uncert_x`, `astrometric_uncert_y` | 0.05, 0.05 | gain matching's fallback RMS | -| `[zogy] astrometric_sigma` | 0.0 | new: ZOGY's own astrometric inputs, always 0.0 as `dev`'s override; `dxrmsfin` and `dyrmsfin` register the measured residual RMS from gain matching instead, not this setting | -| `[zogy] post_zogy_keep_diffimg_lower_cov_map_thresh` | 0.5 | the coverage threshold for masking | -| `[zogy] zogy_sn_sr_from_uncertainty_maps` | true | ZOGY's noise arguments from the uncertainty maps | -| `[zogy] zogy_output_*_file` | `zogy_diffimage.fits`, `diffpsf.fits`, `scorrimage.fits` | ZOGY's output names | -| `[zogy] detection_role` | `significance` | new: the member ZOGY's catalogs detect on | -| `[sfft] run_sfft`, `crossconv_flag` | true, false | SFFT runs as in `dev`; cross-convolution is forced off for rimtimsim data, as in `dev` | -| `[sfft] sfft_bsmask_value`, `sfft_bsmask_radius`, `sfft_use_gainmatch_catalogs`, `sfft_use_segmentation` | `20000.0`, `30.0`, false, false | the socsims block; an empty `sfft_bsmask_value` selects `dev`'s file-name fallback | -| `[sfft] sfft_code` | `/code/modules/sfft/sfft_rapid_rimtimsim.py` | hard-coded in `dev` | -| `[sfft] python_cmd`, `activate_cmd` | empty, empty | new: empty `python_cmd` selects the stage's own interpreter, the same convention as `[paths] python`; empty `activate_cmd` runs SFFT in the stage's own environment rather than activating one. Neither is a venv gap: the base image resolves sfft 1.7.3 into the main conda environment, `/sfft_env` exists nowhere, and `dev`'s `python3.11` is an smdc-layer alias for 3.14. `dev`'s values, `python3.11` and `source /sfft_env/bin/activate`, remain selectable. | -| `[sfft] register_sfft` | true | new: register SFFT's result as its own instance, alongside ZOGY's | -| `[sfft] detection_role` | `difference` | new: the member SFFT's catalogs detect on (the cross-convolved image with `crossconv_flag`, as in `dev`) | -| `[naive_diffimage] naive_diffimage_flag`, `naive_output_diffimage_file` | true, `naive_diffimage_masked.fits` | the naive subtraction, a diagnostic | -| `[bkgest]` | `dev`'s values | bkgest's options and output names | -| `[gainmatch]` | `dev`'s values | the gain-matching thresholds | -| `[psfcat_diffimage]` | `dev`'s values | the Photutils catalog's settings and output names | -| `[fake_sources] inject_fake_sources_flag` | false | fake-source injection is not ported; true exits 64 | -| `[statistics] clip_correction_seed` | -1 | new: seeds the clipped-statistics correction's random draw; negative is `dev`'s unseeded draw | - -## Fixture and selftest - -`rapidpipe selftest --stage difference`, with `--real-tools`, runs as -an ordinary AWS Batch job inside the deployed pipeline image, against -its digest (rapid #103, #104, #105). This replaces the earlier -`rapid-admin` docker fixture venue. The fixture's real-tool expectations -come from a measured run and are deterministic across docker and Batch. -Catalog row counts are checked within `max(3 sources, 2%)`: SExtractor -and Photutils catalogs vary by a few sources between runs on -pixel-identical images. - -On dev's pid-1105 inputs, with dev's settings of the time (gain 1.0, -read noise 8.5, ZOGY noise from image scatter), the rebuilt stage -reproduces dev's ZOGY difference to numerical equivalence: 0.93% of -pixels differ from dev's by more than 1e-3 DN, and the median relative -difference is 1.5e-6. Every difference above 0.011 DN sits within about -300 pixels of two science-image pixels of 10,000 DN or more. Dev's -artifact-repair step (`625b8dcf`, ported by the rebuild) now removes -those pixels; the comparison run predates that dev change. - -Gain matching, the background subtraction, the resampled and gain-matched -reference, the naive difference and ZOGY's own PSF are identical or differ only at -float precision. With the rebuild's defaults, differences from that -same comparison come from later dev changes also ported by the rebuild: -the instrument's gain and read noise (`8428314b`) and ZOGY's noise -inputs from the uncertainty maps rather than image scatter (`df5117c3`). - -The real-tool run against fixed inputs and the IMSS comparison remain -the team's gate before operational use, and that gate has not run. +| `[difference.paths] python`, `[difference.sfft] python_cmd`, `activate_cmd` | empty | ZOGY and SFFT run in the stage's own interpreter; `dev` hard-codes `python3.11` and its own SFFT environment | +| `[difference.paths] bkgest_code`, `bkgest_include_dir` | `bkgest`, `/opt/rapid/share/bkgest` | bkgest as the pipeline image installs it, where `dev`'s `/code/c` tree is absent | +| `[difference.paths] work_subdirectory` | `diff` | the working directory under the attempt's prefix | +| `[difference.awaicgen] zprefimg` | empty | an override of the reference zero point; empty reads the reference's `MAGZP` | +| `[difference.zogy] astrometric_sigma` | 0.0 | ZOGY's astrometric inputs, 0.0 as `dev` overrides them | +| `[difference.sfft] run_sfft`, `crossconv_flag` | true, false | SFFT runs, without cross-convolution for these simulations, as in `dev` | +| `[difference.naive_diffimage] naive_diffimage_flag` | true | the naive subtraction runs, a diagnostic | +| `[difference.fake_sources] inject_fake_sources_flag` | false | fake-source injection is not ported; true is refused | +| `[difference.statistics] clip_correction_seed` | 0 | seeds the clipped-statistics correction's random draw; negative is `dev`'s unseeded draw | + +With the seed pinned, the same inputs give the same pixels and rows, +apart from the FITS `DATE` cards and bkgest's time-stamped tables. The +unseeded draw moves every clipped standard deviation by about 3e-4 and, +through the PSF-fit threshold, the source count by a few: 5,539 to +5,544 for one epoch over four runs. Two seeded executions of one epoch +on the same instance type were identical row for row. A third on a +different CPU family gave the same 5,536 sources with 2,156 rows +differing in some column, most by less than a microarcsecond; the cause +is likely floating point that depends on the CPU, and is not verified. + +## Measured + +On SMDC on 2026-10-21, on Batch with one vCPU and 24 GiB, 69 epochs of +field 4658155, SCA 8, W146 against the 25-frame reference took 1,091 to +1,181 s each in the container (median 1,124 s), against 1,324 to +1,369 s on a two-frame reference; the cause of the difference was not +probed. The largest child process peaked at 14.54 GiB whatever the +reference, and each job wrote 72 files, about 2.8 GB. Uncapped, the +epochs gave 4,708 to 5,748 sources, 365,104 in all, 0.975 of `dev`'s +own catalogs' total. + +Against `dev`'s night, which used a reference of 25 frames outside it, +the port's difference images correlate with `dev`'s at 0.961 to 0.978, +with a median absolute deviation of 4.79 against `dev`'s 4.87. Source +positions mostly do not coincide: about 21% of the port's sources lie +within 0.055″ of one of `dev`'s, and the two brightest-500 lists share +12%. With the same reference as `dev`, an earlier host run of one epoch +correlated with `dev`'s difference at 0.99989. The real-tool run +against fixed inputs and the IMSS comparison remain the team's gate +before operational use. + +## Not decided here + +- Source-level parity with `dev`. The two references are 25 different + frames, so their noise differs, and most of a chip's sources sit near + the 5σ PSF-fit threshold, where which noise peaks cross it would + differ while the images agree; that reading fits every figure and is + not verified. A check needs the same reference frames or injected + sources with known positions. +- sumrat, real/bogus and the extra SFFT catalog columns, and `dev`'s + loader prefilter, left out of the thin chain. diff --git a/system/foundation.md b/system/foundation.md index 619aaa7..4636142 100644 --- a/system/foundation.md +++ b/system/foundation.md @@ -11,8 +11,9 @@ package `rapidpipe.foundation`, the stages `import`, `reference`, `scripts/read_packets.py` and the queries in `database/queries.sql`. The [stage contract](stage-contract) page states what a stage is under it, and the [decisions](decisions) page records the rulings it rests -on. The chain has run end to end on SMDC's database and AWS Batch; the -last two sections give the procedure and what it measured. +on. The chain has run end to end on SMDC's database and AWS Batch over +one field's whole night, 69 epochs; the last sections give the +procedure, the checks the night met and what it measured. ## In plain terms @@ -73,7 +74,7 @@ distinct live streams: | `exposures` | dataset, expid | `exposures` | | `l2files` | dataset, expid, sca, fid | `l2files` | | `diffimages` | dataset, expid, sca, fid | `diffimages` | -| `psfs` | dataset, fid, sca | `psfs` | +| `psfs` | dataset, fid, sca, kind | `psfs` | | `refimages` | dataset, field, fid | `refimages`, with children `refimimages` (members) and `refimcatalogs` (mosaic catalogs) | | `sources` (source set) | dataset, expid, sca, fid | `sources` | | `objects` (object set) | dataset, survey, field | `objects`, with child `memberships` | @@ -87,7 +88,11 @@ exactly one key of that set's product. The pipeline id that `dev` carries in nearly every image query is not in any key: a run has one differencer, and a second differencer is -another run. +another run. A PSF's kind is in its key, `science` or `reference` +({ref}`ruling `): the science image's PSF and the +reference image's PSF of one filter and SCA are different products, +which the key without the kind merged. `dev`'s `psfs` has no such +column; this is the identity rule applied. ### Inputs are chosen at launch and recorded @@ -146,7 +151,8 @@ and returns. The publisher then, serialised on the job's row: opens the job's; otherwise it rolls back to the savepoint and records the refusal on the job; 5. commits, all of the above in one short transaction, then runs - `ANALYZE` on the tables it loaded. + `ANALYZE` on the tables it loaded, and on `official` when its + binding wrote there. A refused job's rows stay published as an unbound candidate, visibly, and a later change group can bind them. A retry finds its job published @@ -296,8 +302,9 @@ read. **One schema file, applied fresh.** `schema.sql` holds every table, view and operation, with the q3c extension and its indexes in a block that continues where q3c is absent, as on a developer's container. The -apply is one `psql -f`. There is no migration machinery until the first -change after real data; from then on changes are incremental SQL. +apply is one `psql -f`. There is no migration machinery: the PSF kind +was the first change after real data, and it and every later change +are incremental SQL run once. Immutability is of published rows and their bytes, not of every field: bindings close and job state changes. Retention is deferred, and its @@ -352,9 +359,18 @@ records the job, then submits one Batch job whose command is `job run --job `, with the outputs root (which must be `s3://`) in the container's environment; the job definition's parameter tree names the database. The container runs one attempt and publishes it, exactly as -`job run` does on a host. A retry is the same Batch request submitted -again: `job run` stops on a published job and otherwise takes the next -attempt. Running the job records the job definition's image digest on +`job run` does on a host. Batch retries exit 75 and nothing else +({ref}`ruling `): the job definition allows three +attempts, the first run and two retries, and its rules retry on exit +75, the stage contract's temporary dependency failure, and end the job +on any other exit code or on a failure with no exit code, such as a +reclaimed instance. Exits 64, 65 and 70 are faults a retry would only +repeat. A retry after a late commit is safe because `job run` stops on +a published job and otherwise takes the next attempt. Batch places a +retry within seconds, with no backoff, so the three attempts ride out a +database outage only as long as the connection's own retries inside +each attempt, about 8 s each. A job can still be resubmitted by hand, +the same Batch request again. Running the job records the job definition's image digest on the job row, and its printed result carries the peak resident memory of the stage's process and of its largest child, which is how a toolkit binary's memory is seen; Batch itself reports no memory figure. @@ -378,7 +394,7 @@ image. Each simplification keeps the shape of the full stage. |---|---|---|---| | `import` | stored products for one field: L2 files, difference images, PSFs, the reference and its mosaic catalog, the Gaia copy; the cap on sources per image | copies of the image files; `exposures`, `l2files`, `diffimages`, `psfs`, `refimages`, `refimcatalogs`, `catalogs` and `sources` (the brightest N per difference image by flux over its error), with limiting magnitude, overlapping fields and sky indices computed; the source system's ids kept; provenance marked unknown | other fields and epochs; recovering provenance | | `reference` | the field's `l2files` rows for one filter and SCA | the awaicgen mosaic with its coverage and uncertainty maps and SExtractor catalog; `refimages`, `refimimages` (one row per coadded L2 row) and `refimcatalogs` | `dev`'s later reference changes (infobits, status, the 9100-pixel mosaic); the Photutils catalog | -| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, the science PSF and the reference PSF; the cap on sources | the ZOGY difference image and the files `packet` reads beside it, SFFT and naive as `dev` runs them; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s | sumrat, real/bogus and the extra SFFT catalog columns; `dev`'s loader prefilter | +| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and one `psfs` row of each kind, `science` and `reference`; the cap on sources | the ZOGY difference image and the files `packet` reads beside it, SFFT and naive as `dev` runs them; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s | sumrat, real/bogus and the extra SFFT catalog columns; `dev`'s loader prefilter | | `associate` | the epoch's source sets for the field and its eight neighbours; the predecessor object set, or none; the exclusions | `objects.parquet` (every object of the field, id carried forward or position-hashed, founding position, sufficient sums) and `matches.parquet` (this epoch's memberships) | `rubrat`, `sumrat`; field parallelism | | `packet` | the epoch's source set; the object set to use, and through it its chain and exclusions; the window's difference and science images with their PSFs; the predecessor packet sets; the Gaia product; the reference and its mosaic catalog | `packets/--.avro` at schema 00.06, the `packets.parquet` index, `forced.parquet`, `external.parquet` | NED, NED-LVS and solar-system matches | | reader, outside the pipeline | a packet set's Avro and its official index and forced rows | a validation report | nothing | @@ -397,46 +413,21 @@ a source set's job binds one key. The kept sources are mapped onto the `sources` columns with the old load stage's join, and their HEALPix and tessellation columns are computed. A row's `filename` is the URL of the copy under the job's prefix; every file is reached through its row. - -### reference - -`reference` coadds the L2 images its job names into one reference for -a field, filter and SCA, with `dev`'s steps in `dev`'s order: each -frame reformatted and scaled to the filter's zero point, the awaicgen -mosaic with its coverage and uncertainty maps, the coverage and image -measurements `dev` keeps, SExtractor on the mosaic, and a header stamp -naming the run, job, attempt and stage. Frames are coadded in order of -observation time; fewer than `min_frames` fails the job, and frames -past `max_frames` are left out and named in the manifest. - -The stage draws the reference's `rfid` from the table's own sequence -while it reads its inputs, so the member and catalog rows carry it -before publication. That `rfid` is the foundation's; an imported -reference keeps the source system's id as an import-only value. The -mosaic carries the filter's zero point as `MAGZP` (26.863 for W146), -where the imported reference from `dev`'s August run carries the scalar -fallback, 17.0. - -### difference - -`difference` subtracts one epoch's L2 image from the reference its job -names, with ZOGY, and runs SFFT and the naive subtraction beside it as -`dev` does. The science was refreshed from `dev` before the port: the -image FWHM from point sources, the reference background subtracted by -the science image's background estimator, the detection overrides of -the SFFT and naive catalogs, SFFT's coverage mask, and `dev`'s current -SExtractor tables. Gain matching reads the reference's zero point from -its own `MAGZP`, so an imported and a coadded reference both work. - -ZOGY needs two PSFs, the science image's and the reference image's. -Both are `psfs` rows under the same key, and the stage tells them apart -by the `PSFKIND` keyword in each file, failing unless there is exactly -one of each. A run has one differencer, so only the ZOGY image is -published as a `diffimages` row; the SFFT and naive files stay under -the prefix for diagnosis. The `sources` set is the ZOGY positive PSF-fit -catalog, capped at `max_sources` by flux over its error (0 keeps every -source), through the same mapping the import uses. A missing catalog -publishes an empty set with its outcome bit set in `infobitssci`. +A PSF's `psf` scope declares its kind; the stage reads the file's +`PSFKIND` (`dev` writes `sciimage_psf` and `refimage_psf`) and fails +before any manifest if it names the other kind. A file with no keyword +imports under the declared kind, and the manifest says so. + +### reference and difference + +The [reference](reference) page states `reference`: its scope (field, +filter and SCA), the `l2files` rows it coadds in `dev`'s order, the +three tables it declares with the `rfid` it draws, and its settings. +The [difference](difference) page states `difference`: its scope (one +epoch's key), its four inputs (the L2 row, a reference, and one PSF of +each kind), the ZOGY image and source set it declares with the `pid` it +draws, and its settings. A run has one differencer, so only the ZOGY +image is published; the SFFT and naive files stay under the prefix. ### associate @@ -511,7 +502,8 @@ cap `max_sources` (500); `associate.match_radius` (1.528e-05 degrees), and `[difference]` carry `dev`'s ini sections as subtables, one per section, with `dev`'s values (`min_frames` 2 and `max_frames` 25 among them); `[difference]` adds `max_sources` (500) with the import's -meaning. The paths in those tables are the pipeline image's; a run +meaning, and `[difference.statistics] clip_correction_seed` (0) makes +the stage's output the same on the same inputs. The paths in those tables are the pipeline image's; a run outside the image overrides them in its own body. Data locations are never settings: the import's source directory is its job's scope and the outputs root is the environment's. @@ -564,7 +556,7 @@ minutes. The same observations on synthetic rows run in CI under The SMDC database `rapid` holds the foundation in its `public` schema, and the pipeline repository's `docs/foundation-smdc-step.md` is the -procedure, one command and then its check per step. It has five parts. +procedure, one command and then its check per step. It has six parts. **The schema.** `public` is dropped and recreated from `schema.sql` in one transaction, through the pooler, as the database owner. Everything @@ -604,104 +596,218 @@ row count is unchanged except `official`'s, so nothing is deleted. **Uncapped.** The same chain at `max_sources = 0`, against the same reference, measured per stage, then its own change group and undo. +**The night.** The sixth part takes one field's whole simulated night, +69 epochs of one SCA in one filter, through the chain twice: once in +the shape of the live loop and once as a campaign, then makes the +campaign official and exercises the operations on it. + +- *The PSF kind and the read role* went in as the first incremental + SQL, in one transaction as the database owner: the `kind` column, + each existing row's kind read from its file's keyword, the check + constraint, the unique key and the view replaced, and the kind + backfilled into the stored keys of `official` and `job_inputs`, which + compare a key as a whole. The reference PSF, unbound until then + because its key collided with the science PSF's, was bound by one + group. The team's read role gained `SELECT` on every table and view + of `public`, and the same by default on anything created later + ({ref}`ruling `). +- *The cutover* was one change group of two rows: it closed the + baseline run's live binding and opened one for a new run, created + not live a moment before. Every product binding stayed on the + baseline's jobs, because a cutover carries only what it names. +- *The live loop* imported the night's 67 further epochs under the new + live run, one host job per epoch in order of `mjdobs`, each bound by + the publisher as it landed. `associate` and `packet` then ran on + Batch for each epoch in turn: each `associate` named the current + official object set as its predecessor, and each `packet` named the + earlier packet sets' `forced.parquet` among its inputs, since a set's + forced rows cover only the objects that triggered in it. All 134 jobs + succeeded on their first attempt and were bound automatically. The + night's last object set holds 1,504 objects with 2,547 memberships + across 69 sets, and 69 packet sets hold 2,488 packets, all read clean + by the reader. +- *The campaign*, a run that is not live and keeps every source, + coadded a reference from the night's first 25 epochs (none of the + 25 frames of `dev`'s reference is in the night), differenced all 69 + epochs on Batch at once and one epoch a second time, and ran + `associate` and `packet` over the first five epochs. Everything + published unbound. +- *One change group made the campaign official*: it opened 145 + bindings, the reference, 69 difference images, 69 source sets, the + last object set and five packet sets, and closed the 209 it replaced, + including the live loop's packet sets for epochs 6 to 69 with nothing + opened in their place. Exposures, L2 files, PSFs, the Gaia copy and + the live binding did not move. Of the two executions of one epoch, + the group bound the first; the other stays a visible candidate. +- *A mid-night epoch was replaced*: the third epoch of the campaign's + chain was differenced again, an `associate` job named the new source + set, the night's last object set as predecessor and the old source + set as an exclusion, a `packet` job followed for that epoch, and one + group bound the four keys together, so there was no moment when the + sources had moved and the objects had not. The packets of epochs 4 + and 5 stay as published, and three of them carry an epoch-3 detection + of the old set in their history; they are valid against their + recorded inputs. A loop does nothing about them, since a packet is a + record at its trigger time; a campaign that wants them consistent + reruns from the replaced epoch. +- *Undo* of the replacement group restored the open bindings row for + row. The undo of the night's group then refused, because the first + undo had reopened four of its keys as new rows and an undo never + reopens a closed row; an explicit group that closed the current + binding on each key the night's group opened and reopened what it + closed restored the earlier state row for row. The night's group was + then applied again, so the campaign's products are official and the + live run is unchanged. +- *Two change groups closing the same binding at once, through the + pooler*: the second blocked on the row lock, then refused naming the + binding, and nothing was half-applied. + +## Validation + +A night is one coherent account when it meets ten checks. Each is a +function over the views, the tables, the Parquet under the prefixes and +the Avro read with the reader's own functions, and prints a pass line +or its first counter-example. Checks 1 to 6 run on a whole run's +products; 7 to 10 on the operations. + +1. **Lineage closes.** Every published row names a published job and + every job a run; every `job_inputs` row resolves to a published row + of its product and key; every binding, open or closed, names a + published job's rows. No orphans either way. +2. **Parquet and tables agree.** Each declared output's row count is + the table's count for that job, and the manifest's checksums match + the bytes in the prefix. +3. **The chain.** `members_official` equals the union of + `matches.parquet` over the object set's predecessor chain, less + exclusions, row for row; every object's sums reproduce its members' + mean and standard deviation; every object of a set is in the next. +4. **Packets.** Against the packet job's own recorded inputs, never the + current official sets: every history detection precedes the + trigger, `nDiaSources` counts the memberships up to the trigger, the + three cutouts open, every forced row equals the row of the + `forced.parquet` its inputs name, and every index row resolves to + its Avro record. The reader passes on every set. +5. **Forced photometry is measured once.** Each (object, image) pair is + measured in one packet set; a set that carries a reused row carries + the same measurement, value for value. +6. **Two executions of one key** both publish with equal rows, and at + most one is official while the other is a visible candidate. +7. **One group makes a night official**: every view the group names + moves to the new jobs, nothing it does not name moves, and every + table's row count is unchanged except `official`'s. +8. **A replacement conserves membership**: the official members are + the predecessor's, less every membership of the old source set, + plus the replacement's matches, row for row, and only the touched + objects change. +9. **Undo restores** the open bindings row for row, deleting nothing. +10. **Two concurrent groups on one binding**: one commits, the other + refuses naming the binding, and nothing is half-applied. + +On 2026-10-21 the night met all ten. Checks 1 to 6 held on the live +loop's run, 69 sets deep, and again on the campaign's after it became +official, five sets deep; 7 to 10 held on the campaign. Check 9 held +by the rule's own refusal: after a later group has moved one of its +keys, an undo refuses and an explicit group restores. + ## Measurements -Measured 2026-10-21 on SMDC, two epochs (expid 84613 and 84624) of -field 4658155, SCA 8, filter W146, every stage on Batch with one vCPU -and 15 GiB per job. The pipeline repository's runbook carries the full -tables, ids and commands; these are its results. They are observations, -not gates. +Measured 2026-10-21 on SMDC, field 4658155, SCA 8, filter W146, every +Batch job with one vCPU. The pipeline repository's runbook carries the +full tables, ids and commands; these are its results. They are +observations, not gates: latency and performance are a later task +({ref}`ruling `). -| Stage | Epoch | In container, cap 500 | In container, uncapped | Peak memory, process / largest child | Outputs | +| Stage | Jobs | In container | Queue wait | Peak memory, process / largest child | Output per job | |---|---|---|---|---|---| -| `reference` | both | 36.8 s | (the same reference) | not recorded | three 196 MB images, an 8.3 MB catalog | -| `difference` | 84613 | 1,348 s | 1,369 s | 1.37 GiB / 14.35 GiB | 72 files, 2.82 GB | -| `difference` | 84624 | 1,324 s | 1,353 s | 1.31 GiB / 14.35 GiB | 72 files, 2.82 GB | -| `associate` | 1 | 2.8 s | 3.2 s | 227 MiB | 10.3 KB | -| `associate` | 2 | 2.7 s | 2.9 s | 228 MiB | 13.5 KB | -| `packet` | 84613 | 21.5 s | 14.1 s | 844 MiB | 15.7 MB | -| `packet` | 84624 | 17.8 s | 18.5 s | 845 MiB | 14.1 MB | - -Peak memory is from the uncapped run, the first to report it. A cold -queue added about two minutes before the first job of each run; jobs -that found an instance running waited under ten seconds. The baseline's -`job local` jobs on a host took 1 to 10 s each, and the schema replace -took 1 s. - -- **Sources per image before the cap**: 5,703 and 5,544, against 5,450 - and 5,586 in `dev`'s own catalogs of the same images. The same code on - the same inputs counted 5,539 for 84624 in the capped run, so the - stage is not reproducible run to run by a few sources; the cause is - not established. The plan's figure of 16 to 43 thousand sources per - image is not reproduced on these two images, by the port or by `dev`. -- **Objects, memberships and packets**: 155 objects (82 founded by the - first epoch, 73 by the second), 82 and 73 memberships, 82 and 73 - packets, all read CLEAN by the reader. Removing the cap changes none - of these: `associate` accepts 199 and 158 sources by flags and - signal-to-noise, all inside the brightest 500. -- **The matcher** took about 1 s per epoch at 227 MiB on those sources. -- **The member chain** is two sets deep on this field. A synthetic - chain on a local database, with 1,000 memberships per set, measured - the packet stage's reads: at 10,000 sets and 10 million memberships - the chain walk took 9.9 ms, the trigger lookup 12 ms, and 500 - objects' history 0.92 s for 1.58 million rows. The history read grows - with the rows it returns, not the depth. -- **A change group** of eight keys took 19.3 ms through the pooler and - its undo 15.3 ms; the command around each took 0.40 s, mostly - start-up. The local curve's 0.14 ms per key predicts 1.1 ms; the rest - is round trips to the pooler. -- **The Batch-made sources against `dev`'s**: 1 to 2% of the capped 500 - match `dev`'s within the match radius, and the difference images - correlate at 0.71 to 0.73. The Batch reference coadds the two epochs - being differenced, so each subtraction removes half of itself; `dev`'s - reference is 25 other frames. Against `dev`'s reference, the port's - host difference image correlates with `dev`'s at 0.99989. With two - epochs the Batch result proves the chain, not the science. +| `import`, on a host | 67 | 5.9 to 7.1 s, mean 6.4 s | | | about 327 MB | +| `reference`, 25 frames | 1 | 227 s | 115 s, from a cold queue | 1.35 / 1.41 GiB | 595 MB | +| `difference` | 71 | 1,091 to 1,181 s, median 1,124 s | 2.4 to 222 s, median 166 s | at most 1.39 / 14.54 GiB | 72 files, about 2.8 GB | +| `associate`, live loop | 67 | 2.4 to 10.0 s, mean 2.7 s | median 6.9 s | 227 to 243 MiB | a few KB | +| `packet`, live loop | 67 | 28 s at 3 window images to 362 s at 69, mean 189 s | median 7.1 s | 844 to 1,036 MiB | 469 MB of Avro in all | +| `associate`, campaign | 5 | 2.7 to 3.0 s | | 224 to 230 MiB | a few KB | +| `packet`, campaign | 5 | 12.7 to 37.8 s | | 844 to 848 MiB | 61 MB in all | + +- **The fan-out.** All 70 difference jobs of the campaign started within + 240 s of the first submit and ran at once; the last published 23 min + 19 s after the first was submitted. The largest child process peaked + at 14.54 GiB, against the job definition's 24 GiB. +- **The live loop's chain**, first submit to last packet, took 3 h 53 + min. The `associate` chain alone took 27 min; the `packet` chain is + the critical path, because each packet job waits for the one before + in order to reuse its forced rows. Forced photometry is 69% of the + packet stage's time, about 3.6 s per window image, and a newly + triggering object is measured on every earlier image of the night. + The night produced 87,404 forced rows, 62,427 of them measured. +- **Chain depth.** At 69 sets the chain walk took 1.8 ms and + `members_official` 32 ms; at 5 sets, 1.8 ms and 5.1 ms. +- **Change groups.** The cutover, two rows, took 4.1 ms. The night's + group, 145 opens and 209 closes, took 257 ms in the operation, 1.23 + ms per key, and 0.58 s with process start-up; eight keys had taken + 2.4 ms per key, and a local database 0.14 ms. The replacement's + group of four keys took 12.1 ms, its undo 8.9 ms. +- **The ten checks** took 255 s on the live loop's run and 145 s on the + campaign's, mostly reading S3. +- **The database** is 239 MB: `sources` 102 MB, `objects` 19 MB, + `forced` 19 MB, and every other table under 2 MB. `official` holds + 961 rows. All 23 statements of `database/queries.sql` run under 30 ms + at this size. +- **Storage.** The live loop's run holds 22.4 GB under the scratch root, + almost all the imported images; the campaign's holds 201 GB, 197 GB + of it the difference stage's working files. +- **The retry.** A job that could not reach the database exited 75 on + each of three attempts, 33 s from the first start to the last stop; a + job with a bad parameter tree exited 64 once. +- The source counts and the comparison with `dev`'s night are on the + [difference](difference) and [reference](reference) pages. **Against the hour per image.** The provisional requirement is an alert current within an hour of an image's delivery -({ref}`ruling `). On this chip, one image's chain on -Batch took about 25 minutes: two of cold queue, 22.5 to 23 of -difference, and under half a minute for `associate` and `packet`. The -difference stage is 95% of it, the cap does not change its cost, and -its largest child process peaked 0.65 GiB under the job definition's -15 GiB. These figures cover two epochs and one chip, with a reference -that includes the images it is subtracted from. - -**Not measured, for want of data on the host that ran it**: the field's -other epochs and a second field; the matcher and the difference stage -at the plan's density; a night's fan-out across 18 chips; packets with -a thirty-day window; a reference built from other frames; completeness -against the injected sources; and the rubrat, NED and NED-LVS matches, -whose model and catalog copies are in a bucket this account cannot -read. sumrat needs no new data, but no stage computes it yet. +({ref}`ruling `). On this chip one epoch's chain on +Batch takes about 19 minutes of difference, seconds of `associate`, and +a packet job that grows with the window, from under a minute to six +minutes over one night; the difference stage dominates until the +window is long. Shortening either is the later task the latency +ruling names. + +**Not measured, for want of data on the host that ran it**: a second +field; a night's fan-out across 18 chips; the matcher and the +difference stage at the plan's density; a thirty-day window of real +cadence; completeness against the injected sources; and the rubrat, +NED and NED-LVS matches, whose model and catalog copies are in a bucket +this account cannot read. sumrat needs no new data, but no stage +computes it yet. ## Not decided here - The cost of following an object set's predecessor chain when a field - has tens of thousands of sets. The synthetic measurement above finds - no need for compaction at ten thousand; if it bites later, a periodic - set holding the full membership compacts the chain, a job like any - other. -- Forced photometry's cost at scale. On the thin field it is two to - four seconds per window image, dominated by opening the images, not - by the fits. + has tens of thousands of sets. A synthetic chain of ten thousand sets + walked in 9.9 ms and the night's 69 in 1.8 ms; if it bites later, a + periodic set holding the full membership compacts the chain, a job + like any other. +- Forced photometry's cost at scale. It is about 3.6 s per window + image, dominated by opening the images, and it orders the packet + jobs into one chain. Measuring a night's new objects per image across + jobs, or naming predecessor sets for reuse without waiting on them, + would change both. +- The cost of a large change group, about three round trips per item + through the pooler: 1.23 ms per key at 145 keys. A set-based close + and open would remove most of it. +- Whether an undo after a later group has moved one of its keys should + have a verb. Today it refuses, which is the safety the design asks + for, and an explicit group restores. +- A published job of a run that is not live, launched with `expects`, + has its binding refused, exits 1, and Batch reports it failed though + its rows are published. Launching without `expects`, or exiting 0 + when the refusal is the run's liveness, would make the two agree. +- The difference stage's rows differ in some columns between CPU + families on the same inputs, by unverified cause. Recording the + instance type beside the image digest would make a re-execution + comparable. +- Packets after a replacement stay as published with the old + detections in their history; whether a loop should ever rebuild + them is open. A campaign reruns from the replaced epoch. - Dense-field memory for `associate` and packet counts per chip. -- The cost of a large change group. The operation is linear at about - 0.14 ms per key on a local database and dominated by round trips - through the pooler; a set-based close and open would remove most of - it. On an `official` table never analysed, the planner picks the - wrong partial index, and the publisher analyses only the tables it - loads. -- Whether the PSF key needs a kind. The science and reference PSFs of - one filter and SCA share a key today, and `difference` tells them - apart by a keyword in the file. -- Whether the team's read role reads the foundation's tables, or only - the official views. It has `USAGE` on the schema and no `SELECT`. - The meaning of `nDiaSources` in the 00.06 schema's own documentation, which is `dev`'s; changing it is a schema change for the team. - `packet.edge_pixels` is in the thin run's settings and read by no stage, since no two-pixel edge rule exists in `dev`'s alerts code. -- The imported reference has no `refimimages` rows, since the source - system's rows do not name its members; a reference `reference` makes - has them. diff --git a/system/index.md b/system/index.md index 192c9da..b61ad84 100644 --- a/system/index.md +++ b/system/index.md @@ -17,11 +17,11 @@ repository's history. The rebuild starts from this page. | Document | Status | Scope | |---|---|---| | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided | -| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-20 and 2026-10-21 on the foundation and its SMDC step, his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | -| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes and a job on Batch; the `import`, `reference`, `difference`, `associate` and `packet` stages and the reader; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure and the chain's measurements on Batch; what is not yet decided | -| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; running a job locally, again or on Batch, with its image digest and peak memory; exit codes; `admit` and `finalize`, still on the older contract | -| [`reference.md`](reference) | DRAFT | The `reference` stage as ported from `dev`'s reference pipeline: its input set and selection rule, the awaicgen coadd and SExtractor catalog, the header stamp, identity, the four registration tables, and its settings | -| [`difference.md`](difference) | DRAFT | The `difference` stage as ported from `dev`: its input set, the steps it runs, its outputs, the registration fields and columns, and its settings | +| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | +| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes and a job on Batch with its retry rule; the thin chain, with the `import`, `associate` and `packet` stages and the reader; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure and the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo); the ten validation checks a night meets and their outcomes; the night's measurements; what is not yet decided | +| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest and peak memory; exit codes and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract | +| [`reference.md`](reference) | DRAFT | The `reference` stage on `run(job)`: its scope and `l2files` inputs, the awaicgen coadd and SExtractor catalog in `dev`'s order, the header stamp, the three declared tables and the drawn `rfid`, its settings, and what it measured | +| [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, one differencer published, its settings and determinism, and what it measured against `dev` | | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block | | [`maintain.md`](maintain) | DRAFT | The `maintain` stage: the once-per-date CLUSTER and ANALYZE of a `sources` child table moved out of `load`, its `detector-date` unit, its manifest and exit codes | | [`crossmatch.md`](crossmatch) | DRAFT | The `crossmatch` stage as ported from `dev`: one field's two passes, the catalog a pass reads (base plus delta), the per-field tables and their set-scoped keys, what lands in `astroobjects` and `merges`, the association set, its settings and exit codes | diff --git a/system/reference.md b/system/reference.md index 18e0e2e..ed74ae0 100644 --- a/system/reference.md +++ b/system/reference.md @@ -2,295 +2,119 @@ **Status: DRAFT** -This page describes the stage as ported to the older contract. It now -runs on the foundation's `run(job)` contract, which the -[foundation](foundation) page's `reference` section states; where this -page describes registration, custody, runs or promotion, that page -governs. - -The `reference` stage coadds already-admitted science frames for one -field and filter with `awaicgen`, catalogs the coadd with SExtractor, -and stamps bookkeeping keywords on the result. It ports `dev`'s chain -as a transform stage: an input set names the frames, the tools run in -the same order, and the stage writes one reference-image bundle and one -reference-catalog instance. It declares `custody` database access: it -reads the database only for the read guard and writes no rows. `register` records both products afterwards, the same -division `difference` and `finalize` already use. - -The stage is planned from the inventory of `dev`'s reference pipeline, -`ppid` 12 -(`pipeline/awsBatchSubmitJobs_launchSingleReferenceImagePipeline.py`, -`pipeline/referenceImageSubs.py`), onto the pipeline -repository's `rebuild` branch as `rapidpipe/stages/reference.py`, -`rapidpipe/science/reference/`, `rapidpipe/settings/reference.toml` and -the packaged `cdf/rapidSex*RefImage*` files. Every ported stage -minimises differences from `dev` and designs in, off by default, -anything that could be left unused. The [foundation](foundation) page fixes -the vocabulary. - -## Inputs - -Stage `reference` has unit kind `field` and unit id `/` -(for example `4711398/W146`). It builds one reference per (field, -filter) per run. The filter is part of the logical key, so a run may -build several filters of one field. - -Filter names are RAPID's own spelling, `W146` not `F146`: FITS `FILTER` -headers and the `filters` table carry `W146`, and lookups against -`filters` match that name exactly. A unit id or manifest may give the -Roman spelling instead; it is normalised to the RAPID spelling with -`dev`'s own `roman_to_rapid_filter_names` map before any comparison or -lookup, so `4711398/F146` and `4711398/W146` name the same unit. - -`--inputs` is an input-set manifest listing N `l2-image` entries, -member role `image`, the delivered `.fits.gz`, all of one filter. The -stage checks each frame's `FILTER` header value, normalised, against -the unit's filter, checks `N >= [selection] min_frames` (`dev` 2) and -coadds at most `[selection] max_frames` (`dev` 25) frames in manifest -order; any violation of those three checks exits 65. - -The launcher selects overlapping frames to build the manifest, using -`dev`'s `get_overlapping_l2files`: science images of the same filter, -`overlapfields @> field`, `vbest > -0`, `mjdobs` in `[start, end)`, ordered by `mjdobs` then distance from -the tile centre. `reference` does not check this selection rule. -Eligibility and selection beyond it remain undecided (see -[specification](specification), "Not decided here"). +`reference` coadds the L2 images its job names into one reference image +for a field, filter and SCA, catalogs the mosaic with SExtractor, and +declares three tables for the publisher. It runs `dev`'s reference +pipeline (`ppid` 12, `pipeline/referenceImageSubs.py`) in `dev`'s order +under the foundation's `run(job)` contract: `rapidpipe/stages/reference.py` +and `rapidpipe/science/reference/` on the pipeline repository's +`rebuild` branch. The [foundation](foundation) page states the records +it works within, and the [stage contract](stage-contract) page what a +stage is. + +## Scope and inputs + +The scope is `{"field", "fid", "sca"}`: the Roman tessellation tile, +the filter id and the SCA. Every input is an `input` row of product +`l2files` whose key has the scope's filter and SCA; any other product +fails the job. The launcher chooses the frames and records them; the +stage reads exactly those rows and their files through +`prefix.get`, and never selects frames itself. + +Frames are coadded in order of `mjdobs`. Fewer than `min_frames` fails +the job; frames past `max_frames` are left out and named in the +manifest as `not_coadded`. Every frame's `FILTER`, normalised to RAPID's +spelling (`W146`, not `F146`) by the one map in +`rapidpipe.products.filters`, must be the scope's filter. ## What it runs -The steps are `dev`'s, in `dev`'s order, one module each under -`rapidpipe.science.reference`: - -1. **Per-frame preparation.** Divide HDU 1's data by `EXPTIME` - (`BUNIT` becomes `DN/s`), then scale it to the filter zero point by - `× 10^(0.4 × (zprefimg − ZPTMAG))`. Build an uncertainty image from - a gain and read-noise model, - `sqrt(|data_norm| · exptime / gain + rn²) / exptime × scale`. Both - are written as PRIMARY-HDU FITS files carrying the frame's SCI - header. Accumulate the JD range and total exposure time across the - frames. -2. **awaicgen.** Run from the stage's work directory over the prepared - image and uncertainty lists, tile-centred, at the mosaic geometry in - `[mosaic]` and 0.11 arcsec pixels: - - | Flag | Meaning | - |---|---| - | `-f1` | the input-image list file | - | `-f3` | the input-uncertainty list file | - | `-X`, `-Y` | mosaic size in pixels, `naxis1`, `naxis2` | - | `-R`, `-D` | mosaic centre, RA and Dec | - | `-C` | mosaic rotation | - | `-pa` | pixel scale, arcsec | - | `-wf` | inverse-variance weighting flag | - | `-sf` | pixel-flux-scale flag | - | `-sc` | simple-coadd flag | - | `-nt` | thread count | - | `-o1`, `-o2`, `-o3` | output mosaic image, coverage map, uncertainty image | - | `-v` | verbose | - -3. **SExtractor.** Runs on the mosaic image with the uncertainty image - as its weight, the same convention `dev`'s - `generateSExtractorReferenceImageCatalog` uses, against the packaged - `SEXTRACTOR_REFIMAGE` configuration. -4. **Measurements.** `cov5percent`, the median coverage, the median - pixel uncertainty, the clipped image statistics and the catalog FWHM - (median, min, max, read from the SExtractor catalog as `dev`'s - `parse_ascii_text_sextractor_catalog` does) are computed exactly as - `dev` computes them for `refimmeta`. - -## Outputs and the header - -One `reference-image` bundle instance: primary member role `image` -(`awaicgen_output_mosaic_image.fits`), members `coverage` and -`uncertainty`. One `reference-catalog` instance, member role `catalog` -(`awaicgen_output_mosaic_refimsexcat.txt`), key `{"reference": , "catalog_type": "sextractor"}`. - -Header keywords are written on the mosaic image and the uncertainty -image as `dev`'s `addKeywordsToReferenceImageHeader` writes them: -`BUNIT`, `FIELD`, `FILTER`, `COV5PERC`, `NFRAMES`, `JDSTART`, `JDEND`, -`MAGZP`, `TOTEXPTM`, and one `INFIL`*nnn* per input frame, plus the -run-model stamp `finalize` already uses (`RUN`, `ATTEMPT`, `INSTANCE`, -`STAGE`). Both files are rewritten with astropy's `checksum=True`, so -`CHECKSUM` and `DATASUM` are recomputed. - -`FID` is not stamped: it is a database id that `register` derives. -The other departures from `dev` concern optional work. `[psfcat]`, -`dev`'s Photutils reference catalog, is designed in and off because it -needs a PSF input this step does not produce. Fake-source injection, -a test-only branch in `dev`, is not ported. - -Gain matching in `difference` reads `zero_point` from the reference -image's `MAGZP` header keyword. Its `[awaicgen] zprefimg` setting -remains available only as an explicit override of that header value. - -## Identity - -The reference-image logical key is `{"field": "", "filter": -"", "recipe": "awaicgen", "version": ""}`. The -selection digest is the full hex-encoded SHA-256 digest over the sorted -constituent `l2-image` instance ids, joined by newlines, plus the -resolved settings hash. Rebuilding the same selection makes another -instance of the same logical product; a different selection makes a -new product. Truncating the digest offers no benefit and adds collision -risk, so the full digest is kept. - -`refimages.version` is a separate legacy per-(`field`, `fid`, `ppid`) -counter, allocated at registration as [foundation](foundation) describes -for every legacy version column. `dev`'s unchanged `addRefImage` -allocates `coalesce(max(version), 0) + 1` over the whole `refimages` -table for that triple, across all runs. The row records `run`, -`attempt` and `instance`, but the run does not bound the counter. This -corrects the earlier "within the run" wording. - -A transaction-level advisory lock, -`pg_advisory_xact_lock(hashtext('refimages:::'))`, -serialises attempts registering the same `(field, fid, ppid)`. It is -taken before `addRefImage` allocates the version and released -automatically at transaction end, preventing a race on -`MAX(version) + 1`. Concurrency beyond that lock remains undecided -(see Not decided here). - -## Registration - -`register` handles both new kinds through -`rapidpipe/products/refimage.py` and `rapidpipe/db/refimages.py`. - -The `reference-image` registration block: `md5` (the primary member), -`status` 1, `infobits` 0 (`dev`'s TODO; no code sets bits yet), `field` -(int), `filter`, `ra_center` and `dec_center` (the mosaic centre, for -`hp6`/`hp9`), `constituents` (the ordered list of coadded `l2-image` -instance ids), `nframes`, `mjdobs_min`, `mjdobs_max`, `jd_start`, -`jd_end`, `total_exptime`, `zero_point` (`zprefimg`), `cov5percent`, -`medncov`, `medpixunc`, `npixnan`, `clmean`, `clstddev`, `clnoutliers`, -`gmedian`, `datascale`, `gmin`, `gmax`, `fwhmmedpix`, `fwhmminpix`, -`fwhmmaxpix`, `nsexcatsources`, `npucatsources`, `settings_hash`. The -block's `nsexcatsources` field maps to `dev`'s column -`refimmeta.nsxcatsources`, whose spelling differs. The measurements -keep `dev`'s `refimmeta` names because they are the target columns. -`npucatsources` is null when `[psfcat]` is off; the migration needed -for that null is described below. - -The `reference-catalog` block: -`md5`, `status` 1, `catalog_type` (`sextractor` maps to `cattype` 1, -`psf` to 2), `source_count`. - -Registration writes four tables: - -- `refimages`, one row, through `dev`'s unchanged `addRefImage`: `field`, - `hp6`/`hp9` from the centre, `fid` from the `filters` table by name, - `ppid` 12, `status`, `filename` (the primary member's location), - `checksum` (`md5`), `infobits`, `svid`, then `run` and `instance` set - as in `psfs.py`. `attempt` is the *producing* `reference` attempt - carried in the manifest it wrote. - `psfs.py`, `diffimages` and `l2files` all record the *registering* - attempt in their own `attempt` columns instead; `refimages` is the one - exception. `vbest` stays 0; the promotion ruling governs promotion. -- `refimmeta`, one row, through `registerRefImMeta` with the block's - measurements. -- `refimimages`, one `(rfid, rid)` row per constituent whose - `l2files.instance` matches an admitted frame. A constituent without - a matching `l2files` row fails registration, exit 65: the run must - register its admitted frames before it can register a reference built - from them. -- `refimcatalogs`, one row, through `registerRefImCatalog`, with `rfid` - resolved from the reference instance (registered earlier in the same - manifest, or already present in `refimages`), `ppid` 12, `cattype`, - and `field`/`hp6`/`hp9`/`fid` copied from the `refimages` row. - -Replay checks the registration block as well as `register_manifest`'s -identity and member-metadata comparison. If an already-registered -instance's stored block matches the manifest's, replay is a no-op and -writes no satellite row twice. A different block is an error: -`addRefImage` and `registerRefImCatalog` would otherwise silently update -an existing row. +Each step is a module of `rapidpipe.science.reference`: + +1. **Preparation.** Each frame's HDU 1 is divided by `EXPTIME` and + scaled to the filter's zero point, with an uncertainty image from the + gain and read-noise model. +2. **awaicgen** coadds the prepared frames, tile-centred, at the + `[mosaic]` geometry, into a mosaic with its coverage and uncertainty + maps. +3. **Measurements**: `cov5percent`, the median coverage and pixel + uncertainty and the clipped image statistics, as `dev` computes them + for `refimmeta`. +4. **SExtractor** runs on the mosaic with the uncertainty map as weight, + and gives the catalog's FWHM statistics. +5. **The header stamp** on the mosaic and its uncertainty map: `dev`'s + `BUNIT`, `FIELD`, `FILTER`, `COV5PERC`, `NFRAMES`, `JDSTART`, + `JDEND`, `MAGZP`, `TOTEXPTM` and one `INFIL`*nnn* per frame, then + `RPRUN`, `RPJOB`, `RPATTMPT` and `RPSTAGE` naming the run, job, + attempt and stage. + +`MAGZP` is the filter's zero point (26.863 for W146), which the +[difference](difference) stage's gain matching reads from the file. The +job fails when awaicgen or SExtractor leaves no output, or when the +catalog has no source with a usable FWHM, where `dev` checks no exit +code. + +## Declared outputs + +The files go under `ref/` in the attempt's prefix: the mosaic, its +coverage and uncertainty maps under awaicgen's names, and the +SExtractor catalog. The manifest declares three tables: + +| Table | Rows | +|---|---| +| `refimages` | one, keyed (dataset, field, fid); `nframes`, `cov5percent`, `status` 1, `infobits` 0, and the stamped mosaic's MD5 | +| `refimimages` | one per coadded L2 row, (`rfid`, `rid`) | +| `refimcatalogs` | one, the mosaic catalog, `cattype` 1, which `packet` reads for reference-star matches | -Two migrations are needed. `20260923-02-refimages-instance.sql` already -covers `refimages`; `20260924-09` drops the baseline schema's `NOT NULL` -constraint on `refimmeta.npucatsources` before a null can be written. -The three satellite tables have no run column and are reached only -through their owning `refimages` row's `rfid`. That relationship alone -does not provide deletion handling. The [foundation](foundation) page's deletion -section fixes cleanup of a scratch run's own reference, amending its -cleanup-set sentence. Any needed stored function missing from the trial -database is ported inline into the register code, as `sources.py` -already does, and recorded. +The stage draws `rfid` from the `refimages` identity sequence while it +reads its inputs, so the child rows carry it before publication. An +imported reference keeps the source system's id as an import-only +value. The manifest also records `dev`'s `refimmeta` measurements, the +JD range, the total exposure time and the zero point. ## Settings -`rapidpipe/settings/reference.toml` holds every `dev` setting the stage -reads, taken from `awsBatchSubmitJobs_launchSingleReferenceImagePipeline.ini` -on `dev`: - -| Setting | Default | Meaning | -|---|---|---| -| `[selection] min_frames` | 2 | fewest frames the stage will coadd | -| `[selection] max_frames` | 25 | most frames the stage will coadd; extra manifest entries beyond this are not read | -| `[instrument] sca_gain`, `sca_readout_noise` | 2.0, 9.4 | `dev`'s science ini values, as `difference.toml` | -| `[mosaic] naxis1`, `naxis2` | 7000, 7000 | mosaic size, pixels | -| `[mosaic] pixel_scale_arcsec` | 0.11 | mosaic pixel scale | -| `[mosaic] rotation` | 0.0 | mosaic rotation | -| `[mosaic] ra_center`, `dec_center` | empty, empty | empty means the tile centre from the closed-form tessellation the rebuild already ships (`RomanTessellationClosedForm`), the same lookup `crossmatch`'s cone uses | -| `[awaicgen] inv_var_weight_flag` | 0 | `dev`'s value | -| `[awaicgen] pixelflux_scale_flag` | 1 | `dev`'s value | -| `[awaicgen] simple_coadd_flag` | 1 | `dev`'s value | -| `[awaicgen] num_threads` | 2 | `dev`'s value | -| `[awaicgen]` list and output file names | `dev`'s names | `awaicgen_output_mosaic_image_file`, `_cov_map_file`, `_uncert_image_file` and the input list file names, `dev`'s values | -| `[awaicgen] zprefimg_` | per filter, `dev`'s ini | the zero point this stage normalises frames to before coadding; a filter with no entry falls back to `zprefimg` 17.0, `dev`'s scalar default. Stamped as `MAGZP` on the coadd, which `difference`'s gain matching reads by default (see [difference](difference)); `difference.toml`'s own, separate `[awaicgen] zprefimg` is now an override of that header, not a second source of truth | -| `[sextractor]` | `dev`'s `SEXTRACTOR_REFIMAGE` section | params, filter and star/galaxy classifier files, the packaged `cdf/rapidSexParamsRefImage.inp`, `cdf/rapidSexRefImageFilter.conv`, `cdf/rapidSexRefImageStarGalaxyClassifier.nnw` | -| `[psfcat] enabled` | false | `dev`'s Photutils reference catalog; designed in, off, since it needs a PSF input this step does not produce | -| `[paths]` | as `difference.toml` | `rapid_sw`, `cfg_path`, and the external tools (`awaicgen`, `sextractor`) on `PATH` | -| `[fake_sources] inject_fake_sources_flag` | false | fake-source injection is not ported; true exits 64, the same convention `difference.toml` uses | - -## Exit codes +The run's `[reference]` table in `rapidpipe/settings/foundation-thin.toml` +holds one subtable per `dev` ini section, with `dev`'s values: -| Code | When | +| Table | Holds | |---|---| -| 0 | built: the bundle and the catalog were written and the manifest published | -| 64 | a setting is missing, empty or invalid, including a true `[fake_sources] inject_fake_sources_flag` | -| 65 | the input set is not all one filter, does not meet `[selection] min_frames`, exceeds `[selection] max_frames`, or a member fails its size or SHA-256 check | -| 70 | any unclassified error, including a failure in awaicgen or SExtractor | -| 75 | not used: the stage's tools run locally, with no temporary dependency to retry | - -## Local execution - -The invocation form is shared by every stage -([stage-contract](stage-contract)): - -``` -rapidpipe stage reference --run --unit / --attempt \ - --inputs --outputs \ - [--settings ] [--dry-run] -``` - -A selftest fixture is planned under `rapidpipe/selftest/fixtures/reference/`: -three tiny synthetic frames under 1 MB, a fake `awaicgen` that mean-stacks -the reformatted inputs onto a small TAN mosaic, and the existing fake -`sex` pattern; `--real-tools` runs the image's own `awaicgen` and `sex` -on the same frames. The fixture is registered in `selftest/runner.py`'s -`STAGE_NAMES` and the Makefile as `stage-reference`, following the other -stages' fixture pattern. +| `[reference.selection]` | `min_frames` 2, `max_frames` 25 | +| `[reference.instrument]` | `sca_gain` 2.0, `sca_readout_noise` 9.4 | +| `[reference.mosaic]` | `naxis1`, `naxis2` 7000, `cdelt1` and `cdelt2` as `dev` writes them, `rotation` 0.0; an empty `ra_center` and `dec_center` take the tile centre from the closed-form tessellation | +| `[reference.awaicgen]` | every awaicgen option `dev`'s launcher passes, the output names, and `zprefimg_` per filter with the scalar `zprefimg` 17.0 as fallback | +| `[reference.sextractor_refimage]` | `dev`'s `SEXTRACTOR_REFIMAGE` section verbatim | +| `[reference.paths]` | the repository and tool configuration paths of the pipeline image, the tools on `PATH`, and `work_subdirectory` `ref` | +| `[reference.psfcat]` | `enabled` false: `dev`'s Photutils reference catalog needs a reference PSF input and is not produced; true is refused | +| `[reference.fake_sources]` | `inject_fake_sources_flag` false; fake-source injection is not ported, and true is refused | +| `[reference.statistics]` | `clip_correction_seed` -1, `dev`'s unseeded draw for the clipped-statistics correction | + +Three keys hold `dev`'s values from before its latest reference changes +(the 9100-pixel mosaic and two SExtractor values), left for the stage's +own refresh from `dev`. + +## Measured + +On SMDC on 2026-10-21, on Batch with one vCPU, the stage coadded 25 +frames of field 4658155, SCA 8, W146 in 227 s in the container, peaking +at 1.35 GiB, and wrote 595 MB: three 196 MB maps and a catalog of 5,248 +sources. Against the reference `dev` built from 25 other frames, the +mosaic's finite fraction (0.32712 against 0.32711) and `cov5percent` +(32.682 against 32.683) agree, the clipped residual after rescaling by +the zero-point ratio has median 3.2e-5 against `dev`'s own scatter of +1.3e-3, and the pixels correlate at 0.9962. ## Not decided here -- Eligibility and the selection rule beyond dev's SQL: which frames a - launcher may choose to overlap a field, and when a reference should be - rebuilt. -- Where the reference PSF is resolved from, for the Photutils reference - catalog `[psfcat]` designs in. -- The Photutils reference catalog itself: `dev`'s script that produces - one, `scripts/generate_refim_psfs.py`, is not called from - `referenceImageSubs.py` in `dev` either, and how it would be wired in - is not decided. -- `reference_sets`: whether a named, reusable grouping of reference - instances is needed beyond the logical key's own identity. -- Concurrency of registration beyond the advisory lock: the lock - serialises the `(field, fid, ppid)` version allocation, but `svid` - allocation (the latest global software-revision row) and the general - question of two runs registering references for the same field and - filter at once are not otherwise addressed here. -- The Roman-to-RAPID filter map exists twice, once in - `rapidpipe/science/reference/prep.py` and once in - `rapidpipe/products/refimage.py`; unifying the two into one place is - left for later. +- Eligibility and selection: which frames a launcher may choose for a + field, and when a reference is rebuilt. The night's reference took the + first 25 epochs by `mjdobs` because none of `dev`'s 25 frames was in + the night. +- `[reference.statistics] clip_correction_seed` is still -1, so the + catalog's source count varies run to run by a few sources as the + difference stage's did before its seed was pinned. +- The Photutils reference catalog, and where its PSF would come from: + `dev`'s script that makes one is not called from `dev`'s reference + pipeline either. +- The imported reference has no `refimimages` rows, since the source + system's rows do not name its members. diff --git a/system/stage-contract.md b/system/stage-contract.md index 396058c..98fff81 100644 --- a/system/stage-contract.md +++ b/system/stage-contract.md @@ -63,7 +63,10 @@ replace with fakes. its own table; - `inputs`: the job's recorded `job_inputs` rows (`kind`, `product`, `key`, `source_job`, `binding`), the authority for what the stage - reads; + reads. An input is told apart from another of its product by its key + alone: a `psfs` key carries its `kind`, `science` or `reference`, so + `difference` names one of each and never reads a file to decide + which is which; - `attempt` and `prefix`: the attempt number and its URL, `/////`, and `path`, the local directory the stage writes into; @@ -142,9 +145,15 @@ definition. | 1 | The job was published, but its automatic binding was refused; its rows are an unbound candidate | | 64 | Usage: bad arguments, no outputs root, or a stage with no `run(job)` | | 70 | The stage or the publication raised; the job is marked `failed` | - -Nothing retries by itself. A retry is `job run` on the same job, or a -Batch resubmission of it, and lands in a new attempt. +| 75 | The database could not be reached after the connection's own retries: a temporary dependency failure | + +On Batch, exit 75 and nothing else is retried: the job definition +allows three attempts, and any other exit code, or a failure with no +exit code, ends the job. A bad parameter tree is 64 and is not retried. +Off Batch nothing retries by itself. Every retry, whether Batch's, a +`job run` on the same job or a resubmission, lands in a new attempt, +and one that finds the job published stops at once. The +[foundation](foundation) page states the rule and why. ## Stages on the older contract