From f8ea4a73410ce8bb674b09717698800cfdd25a98 Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Sat, 24 Oct 2026 10:22:51 +0000 Subject: [PATCH] docs: the night's foundation (settings layers, association amendment), the quality gate, Spot as a probe, the datasets page, the night's figures (night docs) foundation: Part 1's settings statement (three layers resolved at run create), the association amendment through the inputs, views, why, thin chain, packet and validation sections, the quality gate, the packet job's own send with the recovery sweep and the direct consumer check, Spot's four rules, the procedure proof's steps, On SMDC parts 10 and 11, the second iteration's measurements (the null test with the bright end flagged, completeness and purity per magnitude with the purity floor, association timing, Spot's data points) and Not decided here brought up to date. datasets: new, HLWAS, GBTDS and OpenUniverse quirks with dispositions. decisions: the bright-star flag and Spot as a probe (Ben, 2026-10-24), two superseded notes. difference, intake, loop: the flag, the new metrics and settings, the loop's operations. Every page stays DRAFT. Co-Authored-By: Claude Opus 5.5 --- system/datasets.md | 104 +++++++ system/decisions.md | 30 ++ system/difference.md | 31 ++- system/foundation.md | 637 ++++++++++++++++++++++++++++++------------- system/index.md | 12 +- system/intake.md | 11 + system/loop.md | 40 ++- 7 files changed, 662 insertions(+), 203 deletions(-) create mode 100644 system/datasets.md diff --git a/system/datasets.md b/system/datasets.md new file mode 100644 index 0000000..278d149 --- /dev/null +++ b/system/datasets.md @@ -0,0 +1,104 @@ +# Datasets + +**Status: DRAFT** + +A dataset is one body of input the pipeline takes in, with its own +live run and its own products, since dataset is in every scientific +key ([foundation](foundation)). This page states the three simulations +the rebuild has read, what each one's data are, the quirks met in +them, and what each quirk became: a key in the dataset's settings +overlay, a reader, or a line recorded and moved past. A quirk is fixed +on the overlay and not debugged to its root in every dataset. The +overlays live in the pipeline repository under +`rapidpipe/settings/datasets/`, one TOML file per dataset with a +comment per key naming the measurement that earned it; the +[foundation](foundation) page states how a run resolves them. + +| Dataset | Overlay | Survey | Read since | +|---|---|---|---| +| SOC simulation r00342, the High Latitude Wide Area Survey | `hlwas-r00342.toml` | HLWAS | 2026-10-22 | +| SOC simulation r00340, the Galactic Bulge Time Domain Survey | `gbtds-r00340.toml` | GBTDS | 2026-10-24 | +| OpenUniverse 2024, the time-domain set (TDS) | `openuniverse-tds.toml` | HLTDS | 2026-10-24 | + +## HLWAS, r00342 + +STScI's SOC simulation of the wide-area survey: 54 pointings on a grid +one SCA wide, seven filters, six passes, L2 ASDF files with a +Gaia-fitted WCS and no time-variable sources, so every difference is a +null test. The ramp, the breadth run over seven filters, the fixes and +the second iteration's proofs ran on it. + +| Quirk | Disposition | +|---|---| +| The L2 data-quality flags mark defects the difference stage otherwise detects | the overlay: `dq_mask_bits = 0x2110C01`, commissioning's rule without SATURATED | +| The L2 WCS sits (+0.12, −0.99) px from the simulation's own input-catalog stars, on every file | the overlay: `crpix_shift = [0.12, -0.99]`; the intake's Gaia-offset metric reads +0.127 / −0.986 px without it and +0.007 / +0.003 with it | +| Each filter's passes differ in timing | the reference bound is the midpoint of the filter's third and fourth passes, set per run | +| The last two passes' exposures are 107.5 s against 294.1 s | recorded ([foundation](foundation), Not decided here) | + +## GBTDS, r00340 + +STScI's SOC simulation of the bulge survey, in the wide-area +simulation's key grammar, with a preview and a thumbnail PNG and a +`_wcs.asdf` sidecar beside every L2 file. + +| | r00340 | +|---|---| +| Span | 2027-10-01 to 2027-10-08, 7.8 days | +| Pointings | 6, at position angle 90.46 throughout | +| Passes | 5: the colour filters once, then F146 with F087 and F213 interleaved | +| Exposure | 66.41 s, 5 resultants, every file | +| Files per filter | F146 80,082; F213 3,348; F087 3,330; F062 432; F129, F158 and F184 216 each; F106 198 | +| Epochs per pointing and SCA | F146 738 to 744, every 13.7 min; F087 and F213 31; F062 4; the others 2 | +| Processing | romancal 0.22.0, CRDS context `roman_0046.pmap`, the same ePSF reference as HLWAS | + +| Quirk | Disposition | +|---|---| +| The L2 gWCS is tweakreg's spurious crowded-field match, 9 to 50 px off and different on every epoch | the intake re-derives the WCS (`rederive_wcs = true`): on one file the input catalog's stars sit at +0.030 / −0.032 px through it, 9,993 matched, against chance through the L2 gWCS | +| Under today's CRDS context the WCS step takes a distortion reference that puts the frame about 1,770 px off | the re-derivation uses the distortion reference the file names, fetched by name | +| STScI's recipe reads the L1 header, but the L2 files carry the L1 values unchanged (12 of 12 sampled) | recorded; the L1 file is not read | +| The sidecar's `wcs_l1` is the L2 gWCS moved by the 4 px reference border | recorded | +| A pointing is the visit modulo 6, so a pointing selection does not bound the epochs | recorded; a sample is a source root of chosen files | +| The data-quality flags are HLWAS's | the overlay: HLWAS's mask | +| The bulge is confusion-limited, so the reference PSF cannot be measured from its stars (none passes) | the overlay: `[difference] reference_psf = "science"`, the science PSF for both images | +| With the science PSF on the reference, 346 to 418 core-and-ring residuals per epoch above S/N 20 | recorded with its gallery; this dataset's next measurement | +| The difference job takes 820 s, 3.5 times HLWAS's, on 9,000 candidates per epoch | recorded | +| The input-catalog tiles over the bulge are stars only, 700 MB and 7.7 million rows each | recorded | +| The Gaia archive returns at most 50,000 rows a job, silently | recorded; the cone is fetched in sub-boxes with the count checked (256,209 rows) | + +One pointing's SCA 8, nine F146 epochs, ran through the whole chain: +the reference from the first six, three epochs differenced, 515 +packets on the topic, every check clean. + +## OpenUniverse TDS + +The OpenUniverse 2024 time-domain simulation, romanisim's +`simple_model` images. STScI's copy is ASDF, 13,256 files over seven +filters in one patch of about 0.27 deg²; IRSA's public release holds the +whole sequence as FITS, 2.06 million files. + +- **The pointing is a row of the observing sequence**, whose date is + the file's exposure start. A visit runs all seven filters over 2.7 + days; visits come every 5 days. A point in the patch has 106 images + per filter at the median. +- **The truth is in every file**: galaxies, point sources and + transients (one SCA held 164 transients, 113 core-collapse, 30 SN Ia, + 20 SN Iax, at redshift 0.7 to 2.8). IRSA's index per image gives each + object's pixel position and realised flux, so completeness is a join + of each difference image's candidates to its file's transient rows. + It complements injection: real hosts, light curves and cadence, + where injection keeps dense control. It comes after HLWAS injection. + +| Quirk | Disposition | +|---|---| +| The tree is a plain mapping, not a data model; `data` is float64 counts per exposure, `dq` all zero | a reader of its own: `[intake] reader = "openuniverse"`, an adapter onto the SOC reader's conversion | +| The WCS takes one-based pixels: stars sit at (−1.16, −1.06) px through it | the overlay: `crpix_shift = [-1, -1]`; the truth then sits within 0.0016 px | +| The file's zero point is photons to counts, not AB | the overlay: the AB zero points for counts per second per filter | +| `err` equals `data` on 90% of pixels | recorded; nothing reads it | +| The gWCS cannot be inverted by today's gwcs | recorded; only the forward transform is used, matched by the SIP fit to 1.5e-9″ | +| No ePSF reference is named | recorded; the proof imported a PSF stamp from the simulation's own model | + +One image went through intake under the reader, its truth tables +carried as binary-table HDUs. A chain past intake needs three pieces +not yet built: an ePSF source for the psf stage, the loop's file listing +and the psf stage dispatching on the reader, and a reference from +epochs before a transient starts. diff --git a/system/decisions.md b/system/decisions.md index 7bdedfd..d8c577a 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -9,6 +9,31 @@ page where attribution matters. ## Rulings +Ben's rulings of 2026-10-24 04:00 to 04:40 UTC on the second +iteration (brief `night-2026-10-24`). The [foundation](foundation) and +[difference](difference) pages state the rules they set. + +(decision-bright-star-flag)= +**The bright-star residual is flagged, not fixed**: what the null test +leaves above S/N 20 after the fixes, bright stars, gets an informational +flag bit and a fraction in the metrics; nothing is dropped. SCAs 4 and +5's excess gets one metric, the science stars' width against the ePSF, +and F213's flux scale a metric and one check of the zero-point table. +Measured: the radius law flags 21 of 23, 15 of 19 and 11 of 15 of +run 46's sources above S/N 20 on F146 SCA 8; the science stars are 1.5 +to 3.3% wider than the ePSF on SCAs 4, 5 and 8 alike; the table agrees +with the files to 0.035 mag. (Ben, 2026-10-24 UTC) + +(decision-spot-probe)= +**Spot is a probe, not production**: a run that names a Spot queue +sends a stage's backlog there with one attempt per submission, and +every next attempt, and any straggler past a hedge factor, runs on +demand; nothing is retried on Spot. It supersedes +{ref}`Spot stays parked ` for runs that opt in. +Measured: the injected footprint's Spot twin finished 2.1 min after its +on-demand run, with no reclaim, at $1.30 against $2.03 priced by lane. +(Ben, 2026-10-24 UTC) + Ben's rulings of 2026-10-22 on what follows the wide-area ramp, made stepwise between 12:57 and 13:23 PDT, and one of the same evening. The [foundation](foundation) page states the design they shaped, and the @@ -44,6 +69,9 @@ pointings overlap and buy nothing. The loop admits in `mjdobs` order across every SCA of its selection; an earlier epoch that arrives after later ones is a rerun from that epoch, and the loop never re-associates. (Ben, 2026-10-22 PDT) +The chain half is superseded by +{ref}`association in the difference job ` +(2026-10-24); the reference half stands. (decision-injection-outside)= **Injection is outside the pipeline, as a dataset**: a one-time job @@ -117,6 +145,8 @@ night-2026-10-24 task 7) **Spot stays parked**: the rebuild's jobs run on demand; the cost figure is read and recorded, and nothing hangs on it. (Ben, 2026-10-22 13:23 PDT) +Superseded for runs that opt in by +{ref}`Spot as a probe ` (2026-10-24). Ben's rulings of 2026-10-21 evening on the instance type and the plan in force. The [foundation](foundation) page states the design they diff --git a/system/difference.md b/system/difference.md index 89fc4ab..f454afc 100644 --- a/system/difference.md +++ b/system/difference.md @@ -40,8 +40,9 @@ the whole 31-pixel stamp usable, and a second-moment width between the science PSF's and 0.4 pixel more, since resampling only widens a star while defect pixels are narrower and galaxies wider), builds an effective PSF from the brightest 100 (`EPSFBuilder`, oversampling 4), -evaluates it at the pixel centres of a 61-pixel stamp re-centred to -0.01 pixel, takes the science PSF's wings beyond 10 pixels (the measured +evaluates it at the pixel centres of a stamp the science PSF's size, +re-centred to 0.01 pixel (a secant step, and a grid search where the +step stalls, since the evaluated centroid's two axes are coupled), takes the science PSF's wings beyond 10 pixels (the measured core within 6, a cosine taper between), normalises it, and gives that to ZOGY, SFFT and the naive catalogs as the reference PSF. The wings are the science PSF's because the ePSF is built on 31-pixel stamps and holds @@ -74,7 +75,30 @@ covered pixels), the candidates before the cap and the sources after it, those above S/N 20, and the share on a reference-catalog source within 3 pixels. The database's `job_metrics` view exposes every job's metrics by run, stage and scope, and `scripts/inspect_run.py` renders them per run, -with a gallery of the residual stars, before the run's checks. +with a gallery of the residual stars, each cutout divided by its star's +reference flux, before the run's checks. The metrics also hold the +science stars' own width (`sci_star_sigma_px`) and its ratio to the +science PSF's, the reference's scale factor (`scalefacref`), the +sources near bright stars and in their rings above S/N 20, and every +job's peak resident memory (`peak_rss_kib`, the process and its largest +child). The {ref}`quality gate ` reads them. + +**The bright-star footprint flag.** What the null test leaves above S/N +20 after the fixes is bright stars, and that residual is flagged, not +fixed ({ref}`ruling `). The job sets bit +`1 << 14` (16384) of `sources.flags` on every kept source inside the +footprint of a bright reference source, a mosaic-catalog source with +aperture S/N above 100, whose footprint is a disc of radius +`max(4, 9.5 × 10^(0.2 (15 − m)))` pixels at its magnitude `m` on the +reference's zero point. Nothing is dropped. The bit is informational: +association and the packet stage clear it before their flag tests, so a +flagged source is associated and triggers, and its packet carries the +bit in `psfFitFlags` with no schema change. The metrics count the +flagged sources and their share of those above S/N 20 +(`bright_footprint_fraction`), and the packet job counts its flagged +packets. The radius law, derived on run 46's published sources, flags +21 of 23, 15 of 19 and 11 of 15 of the sources above S/N 20 on F146 SCA +8, the fixes' own bright-star counts on those epochs. ## What it runs @@ -207,6 +231,7 @@ port or departs from `dev`: | `[difference.sfft] run_sfft`, `crossconv_flag` | true, false | SFFT runs, without cross-convolution for these simulations, as in `dev` | | `[difference.naive_diffimage] naive_diffimage_flag` | true | the naive subtraction runs, a diagnostic | | `[difference.fake_sources] inject_fake_sources_flag` | false | fake-source injection is not ported; true is refused | +| `reference_psf` | `"measured"` | the reference PSF ZOGY is given: `"measured"`, from the resampled reference's stars, or `"science"`, the science PSF for both images, for a field whose reference stars cannot be measured (the GBTDS bulge, [datasets](datasets)) | | `[difference.statistics] clip_correction_seed` | 0 | seeds the clipped-statistics correction's random draw; negative is `dev`'s unseeded draw | With the seed pinned, the same inputs give the same pixels and rows, diff --git a/system/foundation.md b/system/foundation.md index 97357f7..ded62d2 100644 --- a/system/foundation.md +++ b/system/foundation.md @@ -7,7 +7,7 @@ official, and the narrow chain of stages built on that record, from stored images to alert packets. It is the design the `rebuild` branch of the pipeline repository implements: `database/schema.sql`, the package `rapidpipe.foundation`, the stages `intake`, `import`, `psf`, -`reference`, `difference`, `associate` and `packet`, the loop, the +`reference`, `difference` and `packet`, the loop, the quality gate, the Kafka publication, the reader `scripts/read_packets.py`, the run validation `scripts/validate_run.py`, the check script `scripts/check_run.sh` and the queries in `database/queries.sql`. @@ -18,8 +18,11 @@ one field's whole night, 69 epochs, and then from SOC L2 ASDF files to the Kafka topic over 426 epochs of a simulated wide-area survey, and again after the housekeeping that made each epoch cheaper and gave the reference a bound, and then over the simulation's six other filters, -each with PSFs the pipeline derives itself; the last sections give the procedure, the checks -they met and what they measured. +each with PSFs the pipeline derives itself, and then with association +moved into the difference job, injected sources counted, Spot measured +and two more simulations taken in ([datasets](datasets)); the last +sections give the procedure, the checks they met and what they +measured. ## In plain terms @@ -53,6 +56,25 @@ the one survey and dataset the run is for. There is no settings table and no resolution of settings at run time: a stage reads its own table from the run's body. +`run create` resolves the body from three layers, low to high. The +shipped defaults, `rapidpipe/settings/foundation-thin.toml`, hold +`dev`'s values and `dev`'s behaviour, with the measured value and its +reason where a behaviour is new to the rebuild. A dataset's overlay, +`rapidpipe/settings/datasets/.toml` (`--overlay`), holds only +the keys that differ for that dataset, each with a comment naming the +measurement that earned it; it is the recorded divergence from `dev`, +and what the next dataset copies and edits. Command-line overrides +(`--set TABLE.key=value`) carry one run's experiment. The run stores the +resolved body, its hash, and the name and hash of each layer, which +`run show` prints as `layers`. A middle layer per survey or instrument +waits until two datasets share a quirk. A setting exists only where a +measured case needed a different value; otherwise the code does one +thing. Datasets differ at intake in kind (a reader per input format, a +WCS correction, the data-quality mask) and downstream only in value. A +code divergence from `dev` where `dev` has no knob is one line on the +[decisions](decisions) page with its measurement. The overlays and what +earned each key are on the [datasets](datasets) page. + ### Jobs A job is one execution of one stage under a run. The `jobs` row holds @@ -86,12 +108,15 @@ distinct live streams: | `psfs` | dataset, fid, sca, kind | `psfs` | | `refimages` | dataset, field, fid | `refimages`, with children `refimimages` (members) and `refimcatalogs` (mosaic catalogs) | | `sources` (source set) | dataset, expid, sca, fid | `sources` | -| `objects` (object set) | dataset, survey, field | `objects`, with child `memberships` | +| `objects` (the object catalog, not bound) | dataset, survey, field, object id, founding job | `objects`, with `memberships` declared beside each source set | | `packets` (packet set) | dataset, expid, sca, fid | `packets`, with children `forced` and `external` | | `catalogs` (reference catalog) | dataset, name, version | `catalogs` | A source is identified by its set and its source id, an object by its -set and its object id, a packet by its set and its source. A set is +founding job and its object id, a packet by its set and its source. +Objects and memberships carry no binding of their own: a membership is +official when its source set is, and an object when one of its +memberships is. A set is identified by the job that wrote it, so a job that writes a set binds exactly one key of that set's product. @@ -105,13 +130,10 @@ column; this is the identity rule applied. ### Inputs are chosen at launch and recorded -`job_inputs` holds, for each job, rows of three kinds. An `input` row -names a set the job reads, by product, key and the job that wrote it; -an object-set job's predecessor is its `input` row of product -`objects`. An `expects` row names, for one output key, the open binding -the job expects to replace, or none. An `excludes` row, for `associate` -only, names a source set of the predecessor's chain that is no longer -official, whose members the job drops. +`job_inputs` holds, for each job, rows of two kinds. An `input` row +names a set the job reads, by product, key and the job that wrote it. +An `expects` row names, for one output key, the open binding the job +expects to replace, or none. The launcher chooses these. For the live run it chooses the current official sets; a campaign chooses its own previous sets, and none for @@ -120,6 +142,18 @@ current official" one, which is what lets a recipe that is not live, or a correction of one downstream product, read sets that are not official. +The one exception is the field's object catalog +({ref}`ruling `). The difference +job reads its field's objects of its own run as they stand when it +runs, and the packet job reads the trigger's objects, their official +members and the forced-photometry window the same way. Each records +exactly what it read, one `input` row per founding job or set seen, +written by the publisher from the manifest's `reads`. The run scope +keeps a recipe from reading another recipe's objects; the recorded read +gives the same rows again. The same night run in another order founds +some objects from a different epoch, so their ids differ, and a +comparison across recipes goes through source memberships. + ### One table of official bindings `official` holds two kinds of binding. A product binding points a @@ -139,9 +173,10 @@ row. Each product table has a view `_official`: its rows joined to their open bindings, with the `binding` and `run` columns added. There -is no stored flag. `members_official` follows an official object set's -predecessor chain and returns its memberships, except those whose -source set any set in the chain excluded. +is no stored flag. `memberships_official` holds the memberships whose +source set is official, so a superseded source set's memberships leave +with it, and `objects_official` holds every object with at least one +official membership, its statistics summed over them. ### Publish last @@ -219,12 +254,11 @@ only downstream products is either not made live, its results bound by an explicit group, or made live for everything. **Automatic binding compares the expected binding, not the input.** -For `associate` the set it read and the binding it replaces coincide; -for `packet` the input is an object set and the binding to replace is -the previous packet set of that image. Two jobs that both read set A: -the first publishes B and binds; the second publishes C and stays an -unbound candidate. A late job from a run that is no longer live stays -unbound for the same reason. +For `packet` the input is a source set and the binding to replace is +the previous packet set of that image. Two jobs that both expect to +replace binding A: the first publishes and binds; the second publishes +and stays an unbound candidate. A late job from a run that is no longer +live stays unbound for the same reason. **Concurrency is the unique index and the row lock.** Two groups that close the same binding: the second blocks on the row, finds it closed, @@ -240,8 +274,8 @@ holds no connection while it computes and writes no catalog row. The old crossmatch matched each exposure against rows it had itself written for the previous one, inside one transaction under an advisory lock; with outputs going to Parquet there is nothing for SQL to query -mid-job, so `associate` matches in memory, and its parity with the SQL -matcher on the same field is a test in CI. +mid-job, so the difference job matches in memory, and its parity with +the SQL matcher on the same field is a test in CI. **Declared outputs, not every file.** A stage's manifest names which of its files are catalog rows, and the publisher loads those and only @@ -263,25 +297,38 @@ close, and the group refuses. contain the job id and the attempt, with their checksums. A retry writes new files and never overwrites a referenced one. -**Association products are sets, and sets are deltas.** An object set -is one job's output for one field. `objects.parquet` holds every object -of the field with its founding position and its statistics as -sufficient sums (count, and the sum and sum of squares of position and -flux), so the next epoch updates statistics from the predecessor's -objects and this epoch's sources without reading history. -`matches.parquet` holds only this epoch's new memberships. A full -membership snapshot per epoch would grow with the square of the epoch -count, and a GBTDS field has tens of thousands of epochs. Object ids -are position hashes of the founding source, so the same founding source -gives the same id under any recipe, and a join on object id works -across recipes for most objects. - -**The object set is the pruned set.** `dev`'s prune deleted members -whose difference image was no longer best. Here an `associate` job -learns at launch which of the predecessor's source sets are no longer -official, records them as exclusions, and drops their members from its -sums; `members_official` omits them. Nothing is deleted, and exclusions -are rare because a superseded source set is rare. +**The object catalog is append-only per run and field, and association +runs in the difference job** +({ref}`ruling `). Observations do +not arrive in order, and an alert is the best information at the time. +The per-field chain it replaced, one `associate` job per epoch, each +waiting for its predecessor's set, assumed arrival in order and paid a +queue wait and a loop pass for three seconds of work; F158's +fourteen-deep chains took hours while the fan-out stages took minutes. +Now the difference job holds the epoch's sources in memory, reads the +field's objects as they stand, matches, founds an object for each +unmatched source, and publishes `objects.parquet` (founded here) and +`memberships.parquet` (each membership with its source's offsets from +the object's founding position and its flux) beside its sources. An +object's statistics are the sums of its official memberships' shares, +through `objects_official`: sufficient sums are additive, so the order +of arrival does not matter. Object ids are position hashes of the +founding source and never change after publication. + +**Two epochs in flight at once are reconciled at publish.** Both can +see no object at a new source and both found one. The publisher, +serialised per run and field by a transaction-scoped lock, matches the +job's founded objects against those founded since its recorded read, +and attaches the duplicate's memberships to the existing object, +rebased to its origin, instead of inserting a second. `dev`'s route, +tolerate duplicates and merge later, changes object ids after alerts +have carried them, and is not taken. + +**A superseded source set's memberships leave with it.** `dev`'s prune +deleted members whose difference image was no longer best. Here +official-ness of a membership is its source set's, so a re-differenced +epoch's old memberships drop out of `memberships_official` and its new +ones come in. Nothing is deleted. **A packet set is per difference image.** The alert unit is one difference image, so a packet set's key is its image key, never the @@ -298,10 +345,12 @@ carries nothing from epoch 2. **Forced photometry is a packet-set output with named inputs.** `dev` caches forced rows in a per-object store that every job rewrites and no -recorded input can describe. Here a packet job names its window's -images and its predecessor packet sets' `forced.parquet` as inputs, -measures only the (object, image) pairs those do not already hold, and -publishes its own `forced.parquet` with `dev`'s row id. +recorded input can describe. Here a packet job reads its window's +images and the official `forced` rows that exist when it runs, +records that read, measures only the (object, image) pairs those do not +already hold, and publishes its own `forced.parquet` with `dev`'s row +id. Two packet sets in flight at once may each measure a pair neither +could read; the later of them copies one of the two. **Reference catalogs are products.** Gaia, NED, NED-LVS and the solar-system predictions are copies whose version no packet recorded. @@ -333,15 +382,16 @@ and `dev`'s per-object forced-photometry store. | Verb | Does | |---|---| -| `rapidpipe run create --who --why --revision --settings FILE --survey --dataset` | records a run and prints its id | +| `rapidpipe run create --who --why --revision [--settings FILE] [--overlay FILE] [--set TABLE.key=VALUE ...] --survey --dataset` | resolves the three layers, records a run with its body and layers, and prints its id | | `rapidpipe official change --group FILE --who --why` | applies a change group from a JSON list of `close` and `open` items; refuses naming the binding | | `rapidpipe official undo --who --why` | applies a group's inverse under the same rule | | `rapidpipe run show `, `rapidpipe job show ` | print a run with its live binding and jobs, or a job with its inputs and bindings, as JSON | -| `rapidpipe job local --run --stage --scope JSON [--inputs JSON] [--expects JSON] [--excludes JSON]` | records a job and runs it in this process, then publishes | +| `rapidpipe job local --run --stage --scope JSON [--inputs JSON] [--expects JSON]` | records a job and runs it in this process, then publishes | | `rapidpipe job run --job ` | runs an existing job's next attempt, then publishes; stops at once if the job is published | | `rapidpipe job submit ... --job-definition ` | records a job and submits one Batch job whose command is `job run`; with `--job `, submits an existing job's next attempt instead | -| `rapidpipe publish --run [--settings FILE]` | produces the run's official, unstamped packet sets to the Kafka topic and stamps each (Publication to Kafka, below) | -| `rapidpipe loop --run --filters --scas --pointings --job-definition` | carries a selection one stage forward per pass until it drains ([loop](loop)) | +| `rapidpipe job wait --job N [N ...] [--timeout S]` | blocks until every named job is published or failed, reconciling with Batch each check; exits 75 at the timeout | +| `rapidpipe publish --run [--settings FILE]` | the recovery sweep: produces the run's official packet sets that their own jobs did not send and the gate does not hold, and stamps each (Publication to Kafka, below) | +| `rapidpipe loop --run --filters --scas --pointings [--passes] --job-definition [--job-queue]` | carries a selection one stage forward per pass until it drains ([loop](loop)) | A refusal exits 1 with the binding named. The exception is a refusal because the job's run is not live: the job is published and its rows @@ -390,7 +440,6 @@ measured peak, and the difference stage keeps the definition's 24 GiB: | `psf` | 2,048 | 333, at breadth | | `reference` | 6,144 | 2,831 | | `difference` | 24,576 | 16,125 | -| `associate` | 1,024 | 238 | | `packet` | 2,048 | 867 | With one reservation for every stage, the ramp ran every job on @@ -400,10 +449,13 @@ jobs onto six instances; every peak stayed under its ask. The container runs one `job run` does on a host. Batch retries exit 75 and nothing else ({ref}`ruling `): the job definition allows three attempts, the first run and two retries, and its rules retry on exit -75, the stage contract's temporary dependency failure, and end the job -on any other exit code or on a failure with no exit code, such as a -reclaimed instance. Exits 64, 65 and 70 are faults a retry would only -repeat. A retry after a late commit is safe because `job run` stops on +75, the stage contract's temporary dependency failure, and on a +container that failed to start (Batch's reason `Task failed to start`, +sixteen of them at breadth), and end the job on any other exit code or +on a failure with no exit code, such as a reclaimed instance. Exits 64, +65 and 70 are faults a retry would only repeat. The loop adds one +rerun of its own: a container Batch ended without `job run` recording +it is resubmitted once, and its rerun's death fails the loop. A retry after a late commit is safe because `job run` stops on a published job and otherwise takes the next attempt. Batch places a retry within seconds, with no backoff, so the three attempts ride out a database outage only as long as the connection's own retries inside @@ -430,6 +482,35 @@ stage reads only files in the inputs bucket, the run's public source root and the CRDS server, and writes only under the scratch bucket's root. +### Spot, a probe + +Spot is a probe, never production +({ref}`ruling `). A run whose `[batch.spot]` table +names a Spot queue (`queue`, `backlog`, `hedge_factor`; the table is +absent by default, so every job runs on demand) is run under four +rules: + +1. a stage's submission goes to the Spot queue while more than + `backlog` of that stage's jobs remain to submit in the pass, and + otherwise to the prompt queue the loop's `--job-queue` names, so a + stage's last wave runs on demand; +2. a Spot submission has one attempt: it overrides the job + definition's retry strategy; +3. a Spot job that ends unpublished, for any reason, is the job's one + rerun, on the prompt queue, so nothing is retried on Spot; +4. a Spot job still running past `hedge_factor` times its stage's + median `container_seconds` is terminated and rerun on the prompt + queue. + +`jobs.attempts` records every submission (queue, times, instance type, +zone, Batch's reason), each loop pass reports `queues` and `attempts`, +and `job_metrics` gains `container_seconds`, `batch_submissions`, +`reclaims`, `hedges` and `lost_seconds`. Capturing EventBridge's +two-minute interruption warning needs a CloudWatch Logs resource policy +not yet granted, so a reclaim is seen today only as a job that ended +unpublished. The first data points are under Measurements, below; the +infrastructure repository's job-shape page holds the series. + ## The thin chain One narrow path through every stage, the smallest working version of @@ -442,14 +523,12 @@ image. Each simplification keeps the shape of the full stage. | `import` | stored products for one field: L2 files, difference images, PSFs, the reference and its mosaic catalog, the Gaia copy; the cap on sources per image | copies of the image files; `exposures`, `l2files`, `diffimages`, `psfs`, `refimages`, `refimcatalogs`, `catalogs` and `sources` (the brightest N per difference image by flux over its error), with limiting magnitude, overlapping fields and sky indices computed; the source system's ids kept; provenance marked unknown | other fields and epochs; recovering provenance | | `psf` | one SOC L2 file's metadata and the CRDS ePSF reference it names | the science and the reference PSF, one stamp twice; two `psfs` rows | a PSF per position on the detector; a reference PSF of the coadd's own | | `reference` | the field's `l2files` rows for one filter and SCA, before the run's reference bound when it has one | the awaicgen mosaic with its coverage and uncertainty maps and SExtractor catalog; `refimages`, `refimimages` (one row per coadded L2 row) and `refimcatalogs` | `dev`'s later reference changes (infobits, status, the 9100-pixel mosaic); the Photutils catalog | -| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and one `psfs` row of each kind, `science` and `reference`; the cap on sources | the gzipped ZOGY difference image, its uncertainty, the mask, the uncapped catalog and the files `packet` reads beside it, SFFT and naive run only for their outcome bits; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s | sumrat, real/bogus and the extra SFFT catalog columns; `dev`'s loader prefilter | -| `associate` | the epoch's source sets for the field and its eight neighbours; the predecessor object set, or none; the exclusions | `objects.parquet` (every object of the field, id carried forward or position-hashed, founding position, sufficient sums) and `matches.parquet` (this epoch's memberships) | `rubrat`, `sumrat`; field parallelism | -| `packet` | the epoch's source set; the object set to use, and through it its chain and exclusions; the window's difference and science images with their PSFs; the predecessor packet sets; the Gaia product; the reference and its mosaic catalog | `packets/--.avro` at schema 00.06, the `packets.parquet` index, `forced.parquet`, `external.parquet` | NED, NED-LVS and solar-system matches | +| `difference` | one epoch's `l2files` row, a `refimages` row with its mosaic catalog, and the science `psfs` row; the field's objects of its run, read at execution and recorded | the gzipped ZOGY difference image, its uncertainty, the mask, the uncapped catalog and the files `packet` reads beside it, SFFT and naive run only for their outcome bits; `diffimages` and the `sources` set, mapped exactly as the import maps `dev`'s; `objects` founded here and this epoch's `memberships` | sumrat, real/bogus and the extra SFFT catalog columns; `dev`'s loader prefilter; `rubrat` | +| `packet` | the epoch's source set, the science PSF, the Gaia product, the reference and its mosaic catalog; read at execution and recorded: the triggers' objects and official members, the window's difference and science images and the official `forced` rows | `packets/--.avro` at schema 00.06, the `packets.parquet` index, `forced.parquet`, `external.parquet`; as its last act, the set on the Kafka topic | NED, NED-LVS and solar-system matches | | reader, outside the pipeline | a packet set's Avro and its official index and forced rows | a validation report | nothing | Every stage reads its inputs the same way whoever made them, so -`associate` and `packet` read a Batch-made epoch exactly as an imported -one, and `difference` never cares whether its reference was imported +`packet` reads a Batch-made epoch exactly as an imported one, and `difference` never cares whether its reference was imported or coadded. ### import @@ -496,45 +575,46 @@ draws, the files it publishes and why each is kept, and its settings. A run has one differencer, so only the ZOGY image is published as a row; the SFFT and naive files are not uploaded at all. -### associate +### Association -`associate` replaces `crossmatch`, `statistics`, `prune` and -`maintain`. It accepts sources whose `flags` and signal-to-noise meet -the run's `[associate]` settings, then matches them in memory with the -semantics of the SQL matcher it replaces: every object within the match -radius matches, not only the nearest, so a source may join several -objects; an unmatched source founds an object whose id is the -`radec_index` of its position, and coincident founders share one -object. Exposures are taken in ascending MJD, and objects founded by -one exposure are seen by the next, not by their own. A second pass over -the eight neighbouring fields' sources adds memberships only. +The [difference](difference) page states the association the +difference job runs: the read of the field's objects, `dev`'s two +crossmatch passes in memory (every object within the match radius +matches, not only the nearest, so a source may join several objects; +an unmatched source founds an object whose id is the `radec_index` of +its position; the neighbouring fields' sources join objects only), and +the reconcile at publish. The `associate` stage it replaced, and with +it `crossmatch`, `statistics`, `prune` and `maintain`, is retired. The sums are of offsets from the founding position, not of raw degrees: at RA 268 an object spreads by about 1e-5 degrees, and a variance from raw-degree sums would be lost to cancellation. The mean is `ra0 + sumra / n`, and the standard deviation in RA carries one -cosine, of the mean declination. An excluded source set's members are -subtracted once; an object left with no members stays with -`nsources = 0`. +cosine, of the mean declination. A membership attached by the +reconcile is rebased to the surviving object's origin. ### packet `packet` is a port of `dev`'s alerts package at schema 00.06, one job -per difference image. Every `flags = 0` source the named object set -holds triggers a packet; a source in several objects takes the lowest -object id, as `dev` does, and `dev`'s cap per chip is not carried. Each -packet carries: +per official source set, in any order. Every source the run's +`[associate]` table accepts that is a member of an object triggers a +packet; a source in several objects takes the lowest object id, as +`dev` does, and `dev`'s cap per chip is not carried. Each packet +carries the best information at the time it is made +({ref}`ruling `), what the job +read as it ran and recorded: - `diaSource` from the source row; -- `prvDiaSources`, the object's members through the chain, less - exclusions, with `mjdobs` before the trigger and within the history - window; -- `diaObject` from the set's position and sums, `nDiaSources` counting - members up to and including the trigger; +- `prvDiaSources`, the object's official members at the read, with + `mjdobs` before the trigger and within the history window, so a late + epoch appears in packets issued after it arrives and never in those + already sent; +- `diaObject` from the object's position and summed memberships, + `nDiaSources` counting members up to and including the trigger; - `prvDiaForcedSources`, `dev`'s PSF photometry at the object's founding - position on every window image that holds it, difference and science - image merged into one row per object and image, rows a predecessor - set holds copied rather than measured; + position on every window image that holds it at the read, difference + and science image merged into one row per object and image, rows an + official set already holds copied rather than measured; - `gaiaMatches` with proper motion propagated to the trigger, and `refStarMatches` and `refGalaxyMatches` from the reference's mosaic catalog; @@ -557,62 +637,80 @@ stated size with the stated fill; that every forced value equals the precedes its trigger; and that every index row decodes to its record and names the job's recorded inputs. +(foundation-quality-gate)= +### The quality gate + +The gate decides whether an official packet set goes to the topic. +`rapidpipe.foundation.gate.decide` is one query over `job_metrics`: it +holds the set when its difference job's `n_sources_snr_gt20` is above +`snr_gt20_factor` times the run's median, or `|core_fraction|` is above +`core_fraction_max_abs`, or `|ring_fraction|` above +`ring_fraction_max_abs`, once the run has `min_epochs` published +differences; otherwise it sends. The run's `[gate]` table holds 3.0, +0.025, 0.015 and 5, the healthy ranges the fixes measured across seven +filters: the worst epoch's count was 2.6 times its run's median, the +worst core 0.022. A held set is neither sent nor stamped, its reason +names the metric and the value, and each loop pass reports the run's +held sets as `held_sets`. Nothing is deleted, and there is no verb +that releases a held set. Every official set of the +fixes' seven runs, 55 of them, was sent at these values; the footprint +runs held one in 162 (ring fraction +0.016), and the 18 SCAs of one +pointing held seven, all on `core_fraction` +0.026 to +0.028, SCAs 16 +and 17 entire. + ### Publication to Kafka -`rapidpipe publish --run ` takes every packet set of the run that is -official and not yet stamped, and produces one message per packet to the -topic. The key is the packet set's job and the packet's source id; the -value is a one-record Avro container, the set's container header with -the packet's own block, so each message decodes on its own and carries -its schema. There is no schema registry. The producer waits for every -acknowledgement from all in-sync replicas, then stamps the set's job -with the time (`jobs.kafka_published_at`). One call opens one database -connection and one producer, then takes its sets one at a time: it -sends a set's messages, flushes within 120 s, and stamps the set when -every acknowledgement came back. A set with a failed acknowledgement -stays unstamped, the others are stamped, and the verb exits 70 naming -it. A flush that does not finish in time leaves its set and every later -one unstamped and stops the call, which exits non-zero, and the next -call sends them again; closing the producer is bounded the same way. -This replaced one flush per call: with five loops publishing at once, -batches waited in the client past its delivery timeout, and the -client's expiry path then killed its own sender thread and left the -flush waiting with no end. A crash between the last -acknowledgement and the stamp offers the set again on the next call, so -a consumer may see a packet twice; replay is allowed and there is no -exactly-once delivery. A set that loses its binding before it is -stamped is not offered. +The packet job sends its own set as its last act, once its publisher +has loaded and bound it. `rapidpipe.foundation.publish_kafka.deliver` +is the one send boundary: the gate answers first, and a held set goes +nowhere. Otherwise one message per packet goes to the topic, keyed by +the run's dataset, so a dataset's packets share a partition and keep +their order, with the set and the source id in the `rapid-set` and +`rapid-sid` headers. The value is a one-record Avro container, the +set's container header with the packet's own block, so each message +decodes on its own and carries its schema. There is no schema +registry. The producer waits for every acknowledgement from all in-sync +replicas, flushes once, and stamps the set's job with the time, the +partition and the first and last offsets +(`jobs.kafka_published_at`, `kafka_partition`, `kafka_first_offset`, +`kafka_last_offset`). The producer is not idempotent: idempotence +asks the cluster for a permission the job role does not hold, and +nothing here is exactly-once. + +`rapidpipe publish --run `, and the loop's publish in each pass, +is the recovery sweep and the verb that publishes again: it takes every +official, unstamped set whose job ended more than 300 s ago, through +the same boundary. A set inside that window is its container's to send, +is reported as `sets_waiting`, and holds the loop's drain. One call +opens one database connection and one producer and takes its sets one +at a time, each flushed within 120 s; a set with a failed +acknowledgement stays unstamped and the verb exits 70 naming it, and a +flush that does not finish in time leaves its set and every later one +unstamped and stops the call. A crash between the last acknowledgement +and the stamp offers the set again, so a consumer may see a packet +twice; replay is allowed. A job killed between its database publish and +its send, proven on a live run, is sent by the next pass. A set that +loses its binding before it is stamped is not offered. The cluster's bootstrap servers, the topic and the largest request the producer may send are named in the settings file's `[kafka]` table as -the deployment's parameter names, never their values, and the -connection authenticates with the caller's own cloud credentials. The -transport is not part of a run's recipe: `publish` reads the table from -the shipped settings file or the one `--settings` names, not from the -run's body. +the deployment's parameter names, never their values. On Batch their +values, and the database's, are in the container's environment, so a +job reads no parameter store; the connection authenticates with the +caller's own cloud credentials. The transport is not part of a run's +recipe. `scripts/consume_packets.py --run ` is the consumer check. It reads -each partition from the first offset at or after the earliest end time -of the run's stamped packet jobs, since a set is published only after -its job has ended, to the end offset at its start, with no consumer -group; keeps the messages whose key names one of the run's stamped -sets; and requires each to decode as one record valid against its index -row's schema, to equal the outbox bytes for its key, and every stamped -packet to be seen at least once. Zero kept is a failure, so a run with -no stamped set fails. The client's own record checksum is switched off: -every kept message is compared byte for byte with the outbox anyway, -and the client computes that checksum in pure Python, which held the -read to about 50 messages a second against 805 without it. The check -holds one message and one set's outbox at a time, so its memory is one -set's, whatever the topic's size. It holds only within the topic's -retention, seven days: a run whose messages have aged out fails it as -never read. - -Today the producer runs where the loop runs, on a host, after the -packet stage has published. Producing from the packet stage's own Batch -job, as its last act, is to be the last code milestone -({ref}`ruling `); `publish` stays as the verb that -publishes again. +only the offset ranges the run's stamped sets recorded, overlapping +ranges merged, with no consumer group; keeps a set's messages by key +and headers; requires every stamped packet to be seen; and decodes and +byte-compares against the outbox one set per field, the field's latest. +Zero kept is a failure, so a run with no stamped set fails. The +client's own record checksum is switched off, since it is pure Python +and held the read to about 50 messages a second. A run's check takes +about 4 s at one pointing and 388 s over 33,650 packets, whatever the +topic's size. It holds only within the topic's retention, seven days: +a run whose messages have aged out fails it as never read. ### Settings of the thin run @@ -634,7 +732,10 @@ scope, the intake's source root is the run's own `[intake] source_root`, given in the settings overlay at `run create`, and the outputs root is the environment's. `[psf]` names the CRDS server and the stamp's size ([psf](psf)). `[batch.memory_mib]` holds each -stage's memory ask (A job on Batch, above). `[reference.selection] +stage's memory ask (A job on Batch, above), `[batch.spot]` the Spot +probe's three values when a run names them, and `[gate]` the quality +gate's bounds. `[associate]` holds the association's values, which the +difference job reads. `[reference.selection] before_mjd`, the reference bound, is documented in the shipped file and absent there: a run sets it in its overlay, and a run without it coadds every epoch of a field ([loop](loop), the reference bound). @@ -645,8 +746,8 @@ The official views answer eight queries without reading files: - the current image and source set for a dataset, exposure, SCA and filter; -- the current object set of a field and its members; -- an object's detection and forced history within a named set; +- a field's official objects and their members; +- an object's detection and forced history as a packet read it; - packet history by observation, by source and by versioned object; - a candidate set against the official one, with unmatched detections shown; @@ -664,30 +765,29 @@ objects in a cone, a chip's unflagged sources, the usable reference, the L2 time-window work list, L2 images without a difference image, frame counts, a difference image by its product id, and L2 images with a usable reference. All of them are `database/queries.sql`, one named -statement each. A query within a named set reads base tables through -that set's recorded chain; every other query reads the official views. +statement each. A query as a job read it reads base tables through +that job's recorded inputs; every other query reads the official views. ## The procedure proof -`tests/db/proof/thin_proof.py` runs the thirteen observations of the -procedure proof on the thin field's real data, through the verbs and -`job local` only, from an empty database. They cover the object binding -moved by the publisher alone; two jobs racing for one binding; a recipe -that is not live beside the live one; a cutover group and its undo; a -job launched before a cutover and published after it; a correction run -of one epoch's packets; a job killed before and after its commit; two -recipes from two commits at once, joined on scientific key and on -object id; two change groups on one binding; publication racing a -cutover; an undo refused because a binding moved; and a replacement -source set excluded downstream. The first passing run took about two -minutes. The same observations on synthetic rows run in CI under -`tests/db/proof/`. +`tests/db/proof/test_steps.py` asserts the procedure proof's +observations on synthetic rows in CI, one test per step, read back +through the official views: a cutover as one group; its undo restoring +the first recipe; a publication interrupted before and after its +commit; two change groups on one binding at once; publication racing a +cutover; and an undo refused because a later group moved a binding. +Steps 1, 2 and 13 are the association's: two epochs of +one field differenced at once both publish and bind their own sources, +the second's founded objects attach to the first's, and a superseded +source set's memberships drop from the view. The steps that need the +real stages (3, 5, 7 and 9) are proven on SMDC's live runs instead. The real-data +script of the chain, `thin_proof.py`, is retired with `associate`. ## On SMDC The SMDC database `rapid` holds the foundation in its `public` schema, and the pipeline repository's `docs/foundation-smdc-step.md` is the -procedure, one command and then its check per step. It has nine parts. +procedure, one command and then its check per step. It has eleven parts. **The schema.** `public` is dropped and recreated from `schema.sql` in one transaction, through the pooler, as the database owner. Everything @@ -921,6 +1021,51 @@ all 54 pointings and then on the 18 SCAs of the first pointing. script, after the consumer check stopped reading the topic from its beginning (Validation, below). +**The fixes.** The tenth part removed the null test's two measured +artefact sources on the wide-area simulation's first pointing in all +seven filters (Measurements, The fixes, below). + +**The second iteration.** The eleventh part, one unattended night, +changed the chain's shape and took it to two more simulations. Every +proof ran under a dataset of its own with its own live run, and nothing +was deleted. + +- *Datasets.* The GBTDS simulation went through intake with its WCS + re-derived, and one pointing through the chain under its own overlay; + the OpenUniverse TDS set was read and one image taken in by a second + intake reader. Their quirks and dispositions are on the + [datasets](datasets) page. +- *The bright end flagged.* The bright-star footprint flag and the + metrics the re-count asked for went into the difference job + ([difference](difference)); a stamp twice the size changed nothing + that matters. +- *The loop's operations.* The HLWAS WCS translation at intake with a + Gaia-offset metric on every file, `--passes`, `job wait`, one + automatic rerun and a submission cap per pass, the check scoped to a + loop's SCAs, the quality gate, the ePSF from a copy and no parameter + store read in a job ([loop](loop), The quality gate, above). +- *The producer.* The packet job sends its own set, offsets on the + stamp; a job killed between its publish and its send was sent by the + next pass (Publication to Kafka, above). +- *The guards.* A schema test that every bound product carries the + dataset in its key and its view, and an import test that no import + supplies an identity column but the three source-system ids kept. +- *Association.* Moved into the difference job and reconciled at + publish; the chain's rows kept as `objects_chain` and + `memberships_chain`. Two epochs of one field in flight at once, a + replayed reconcile equal to the sequential result, a late epoch and a + superseded set were each proven on run 69. +- *Spot.* The four rules proven on one pointing, then a twin of the + injected footprint on Spot beside its on-demand run (Measurements, + below). +- *Injection.* 200 constant sources per image, AB 21 to 27, injected + into SCA 8's 162 epochs after the reference bound and the 18 SCAs of + the first pointing, as datasets of their own + ({ref}`ruling `); completeness and purity + per magnitude are under Measurements, below. The footprint earned two + fixes: the reference ePSF's centring where the secant step stalls, + and the forced check under packet sets in flight at once. + ## Validation A night is one coherent account when it meets ten checks. Each is a @@ -936,10 +1081,12 @@ products; 7 to 10 on the operations. 2. **Parquet and tables agree.** Each declared output's row count is the table's count for that job, and the manifest's checksums match the bytes in the prefix. -3. **The chain.** `members_official` equals the union of - `matches.parquet` over the object set's predecessor chain, less - exclusions, row for row; every object's sums reproduce its members' - mean and standard deviation; every object of a set is in the next. +3. **The catalog.** Every official membership is within the match + radius of its object; every accepted source of an image's field is a + member of some object; no two founders share one object; every + official object's sums equal its official members'. (Until + 2026-10-24, the chain: `members_official` equal to the union of the + predecessor chain's matches, less exclusions.) 4. **Packets.** Against the packet job's own recorded inputs, never the current official sets: every history detection precedes the trigger, `nDiaSources` counts the memberships up to the trigger, the @@ -954,10 +1101,9 @@ products; 7 to 10 on the operations. 7. **One group makes a night official**: every view the group names moves to the new jobs, nothing it does not name moves, and every table's row count is unchanged except `official`'s. -8. **A replacement conserves membership**: the official members are - the predecessor's, less every membership of the old source set, - plus the replacement's matches, row for row, and only the touched - objects change. +8. **A replacement conserves membership**: the official memberships + lose every membership of the old source set and gain the + replacement's, row for row, and only the touched objects change. 9. **Undo restores** the open bindings row for row, deleting nothing. 10. **Two concurrent groups on one binding**: one commits, the other refuses naming the binding, and nothing is half-applied. @@ -989,7 +1135,9 @@ a time, and the consumer check, streams their output, and prints one line, `Check run : CLEAN|FAIL validate / reader / consumer / `, exiting non-zero when any of the three fails. Every run of the breadth part was checked -with it, and every one was clean. +with it, and every one was clean, as was every run of the second +iteration: the injected footprint's on-demand run validated in 475 s +and read 33,650 packets back in 388 s. ## Measurements @@ -1258,18 +1406,132 @@ for F146, the image study's run, read from the database by lineage. - **Every run's checks** were clean: validation, the reader and the consumer, over 4,336 packets. +### The second iteration + +Measured 2026-10-24 on SMDC (brief `night-2026-10-24`), on the +wide-area simulation's F146 data unless named; the runbook's part 11 +holds the jobs, commands and raw tables. + +**The null test, with the bright end flagged.** On pass 4 of the first +pointing, after the HLWAS WCS translation: + +| | SCA 8 | SCA 5 | SCA 4 | +|---|---|---|---| +| Candidates / above S/N 20 | 802 / 29 | 884 / 69 | 640 / 36 | +| In a bright-star footprint, share of those above S/N 20 | 0.79 | 0.74 | 0.75 | +| Packets / above S/N 20 / flagged | 79 / 8 / 49 | 124 / 16 / 75 | 98 / 11 / 65 | +| Science stars' width over the ePSF's | 1.024 | 1.015 | 1.033 | +| Core / ring of star flux | +0.0001 / +0.0009 | +0.0114 / −0.0015 | +0.0058 / +0.0024 | + +- **SCAs 4 and 5's excess is not a science-PSF width mismatch**: the + science stars are 1.5 to 3.3% wider than the ePSF on all three, and + SCA 5, with the most bright-star residuals, has the smallest + mismatch. +- **The stamp size is not the residual's cause**: a 121-pixel reference + PSF stamp against the 61-pixel one moved the ePSF flux held from + 0.971 to 0.979 and changed the counts by one. +- **F213's flux scale.** The zero-point table agrees with each filter's + own file metadata to within 0.035 mag and is not corrected; the stored + reference scale factor is 0.958 in F184 and 0.939 in F213 against + 0.99 to 1.01 in the others, which fits F213's positive core and ring. + +**Association in the difference job.** On one pointing's SCA 8, the +last difference to the last packet took 31 s, against 5.2 min through +the chain of run 46; the read of a field's 200 objects took 0.9 ms +against 422 ms through the chain's view. Two epochs differenced at once +founded no duplicate, and a replayed reconcile from an empty read +attached its 105 memberships exactly as the sequential run had. + +**Completeness and purity, injected.** 200 constant sources per image, +AB 21 to 27 uniform, in SCA 8's 162 epochs over the footprint after +the reference bound; matched within 0.3″ through the corrected WCS. +Every 0.5 mag bin is in the runbook; every other bin here: + +| AB | Injected | Detected | Alerted | Alerted, of those detected in their image's own field | Purity, packets at this measured magnitude | +|---|---|---|---|---|---| +| 21.0 | 2,595 | 0.958 | 0.398 | 0.951 | 0.93 | +| 22.0 | 2,808 | 0.975 | 0.431 | 0.969 | 0.91 | +| 23.0 | 2,670 | 0.966 | 0.381 | 0.974 | 0.75 | +| 24.0 | 2,910 | 0.968 | 0.415 | 0.981 | 0.59 | +| 25.0 | 2,853 | 0.944 | 0.395 | 0.958 | 0.35 | +| 26.0 | 2,721 | 0.754 | 0.298 | 0.960 | 0.20 | +| 26.5 | 2,640 | 0.233 | 0.109 | 0.976 | 0.13 | + +- **Detection is 0.94 to 0.98 to AB 25**, 0.75 at 26 and 0.23 at 26.5, + with positions within 0.011″ and fluxes at 0.91 to 0.95 of the + injected to that depth. +- **The alerted fraction, 0.40, is the association's field rule**, not + the difference stage: a source outside its image's own field only + joins an object already within the radius, and where none exists it + never triggers. Of the unflagged detections inside their image's own + field, 12,069 of 12,070 were alerted. +- **The purity floor.** Purity, packets on injected sources over all + packets at that magnitude, is 0.91 to 0.93 at AB 21 to 22 and falls to + 0.20 at 26, where the artefacts and real faint residuals dominate. + Per epoch the packets on anything but an injected source are a median + 95 (33 to 182): 52 on bright-footprint-flagged sources and 38 on + others. The bright end of the flagged share is the null test's floor, + about 20 per epoch above S/N 20 on SCA 8. +- **History.** Of 11,576 earlier alerted epochs the alerted injected + sources' later packets should carry, 11,048 are present, 524 arrived + after the packet's read, as the alert contract allows, and 4 are + missing, where two epochs in flight founded one source's objects + apart. +- **The 18 SCAs of the first pointing**: detection below AB 25 from + 0.67 (SCA 9) and 0.73 (SCA 7) to 0.99 (SCAs 8 and 17), with SCAs 4 and 5 + at 0.93 and 0.89; per-epoch false positives a median 153 against SCA + 8's 95; the gate held seven of the 51 sets. + +**Spot, the first data points** +({ref}`ruling `). On one pointing (14 jobs, 7 on +Spot) there was no reclaim and no hedge, and Spot cost wall time: its +differences started 168 to 292 s after submission against 11 s on a +warm on-demand host. At footprint scale, the injected SCA 8 footprint +ran twice on the same bytes and truth, once on demand with 512 vCPU +held warm and once as a Spot twin: + +| | On demand, warm | Spot twin | +|---|---|---| +| First submission to last job end | 28.1 min | 30.2 min | +| Batch submissions | 709 on demand | 589 Spot, 129 on demand | +| Reclaims / hedges | 0 / 0 | 0 / 0 | +| Median start wait, intake / difference | 65 s / 15 s | 160 s / 223 s | +| Job-hours | 17.4 | 13.4 Spot + 3.8 on demand | +| Priced by lane, Spot at Spot rates | $2.03 | $1.30 | +| Completeness | | the same to three detections | + +The prices are each job's memory share of its host at published rates, +unverified against the bill; the warm capacity, up to about 59 +instance-hours for the hour it was held, is not in them. Both runs lost +the same 13 minutes to the reference ePSF's centring failure, fixed in +the middle of the drain, so the spans compare like for like. + +**GBTDS's one pointing**, with the science PSF on both images because +the bulge's reference stars cannot be measured, leaves 346 to 418 +sources above S/N 20 per epoch, each bright one a core-and-ring +residual at a star, against about 19 on HLWAS; its difference job takes +820 s, 3.5 times HLWAS's ([datasets](datasets)). + ## Not decided here -- The cost of following an object set's predecessor chain when a field - has tens of thousands of sets. A synthetic chain of ten thousand sets - walked in 9.9 ms and the night's 69 in 1.8 ms; if it bites later, a - periodic set holding the full membership compacts the chain, a job - like any other. +- The association's field rule. A source outside its difference + image's own field and its neighbours is never associated, and one in + a neighbouring field only joins an existing object, so a field no + image is centred on founds none: on the injected footprint this + costs three fifths of the detected sources their alert. It was so + under the chain too, and it is the next science question. +- Whether an object's statistics are materialised per field. The view + read a field's 200 objects in 0.9 ms; at GBTDS depth it is unmeasured. +- A source that a job in flight founded nearby is not added to an + object the other job read, as a sequential run would have. On the + footprint, 4 of 11,576 expected history entries were missing where + two epochs in flight founded one source's objects apart. - Forced photometry's cost at scale. It is about 3.6 s per window - image, dominated by opening the images, and it orders the packet - jobs into one chain. Measuring a night's new objects per image across - jobs, or naming predecessor sets for reuse without waiting on them, - would change both. + image, dominated by opening the images. A set's `forced` rows are its + packets times the window images on the trigger's SCA, so a field's + total grows as the square of its epochs inside one window; F158 at + 14 epochs held about 14 rows per packet and its packet job about + 300 s. - The cost of a large change group, about three round trips per item through the pooler: 1.23 ms per key at 145 keys. A set-based close and open would remove most of it. @@ -1291,32 +1553,35 @@ for F146, the image study's run, read from the database by lineage. each filter's bound is the midpoint of its own third and fourth passes, which leaves three reference epochs before it in every filter; a survey without passes needs another rule. -- The grant and the path for producing packets to Kafka from the - packet stage's own Batch job ({ref}`ruling `). The - compute environment and the brokers share a security group that - admits itself; the Batch job role and its permission boundary hold no - Kafka action. +- Exactly-once delivery. The producer is not idempotent, because the + job role holds no idempotent-write permission; granting it, or not, + is open. +- The Spot interruption warning's capture, which waits on a logs + resource policy, and what a reclaim costs, since none has been seen. - Cross-band association. Each filter is a dataset of its own, so a source seen in two filters is two objects. Whether an object is per band or across bands is open, and with it forced photometry in the other bands at a source's position. -- What the null test leaves after The fixes: residuals at bright - stars, now the whole bright end (about 20 per epoch above S/N 20 on - SCA 8, two to three times that on SCAs 4 and 5), with a flux-scale - residual in F213; their cause is not established. The quality gate, - a query over `job_metrics` that holds an epoch off its run's median, - is not built. +- The cause of the bright-star residual (about 20 per epoch above S/N + 20 on SCA 8, two to three times that on SCAs 4 and 5), now flagged + and not fixed; neither the science PSF's width nor the stamp size is + it. F213's flux-scale residual fits its references' scale factor of + 0.94. - The 107.5 s exposures of the last two passes are differenced against a 294.1 s reference. Whether a short exposure gets a reference of its own exposure time, or is differenced at all, is open. -- Source injection, outside the pipeline as a dataset of its own - ({ref}`ruling `), and its completeness measure. +- Completeness on real hosts and light curves, the OpenUniverse TDS + set's, after HLWAS injection; the chain past its intake needs the + three pieces the [datasets](datasets) page lists. +- GBTDS's core-and-ring residual with the science PSF on both images, + 346 to 418 per epoch above S/N 20. - Whether SFFT and the naive subtraction keep running now that only their outcome bits remain ([difference](difference)). - Packets after a replacement stay as published with the old detections in their history; whether a loop should ever rebuild them is open. A campaign reruns from the replaced epoch. -- Dense-field memory for `associate` and packet counts per chip. +- Packet counts per chip in a dense field: GBTDS's 9,000 candidates + per epoch kept 500 sources and 169 to 173 packets. - The meaning of `nDiaSources` in the 00.06 schema's own documentation, which is `dev`'s; changing it is a schema change for the team. - `packet.edge_pixels` is in the thin run's settings and read by no diff --git a/system/index.md b/system/index.md index 5765392..e484f50 100644 --- a/system/index.md +++ b/system/index.md @@ -17,12 +17,13 @@ repository's history. The rebuild starts from this page. | Document | Status | Scope | |---|---|---| | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided | -| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked; the other filters as datasets of their own), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | -| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; the thin chain, with the `intake`, `import`, `psf`, `associate` and `packet` stages, the reader and the publication to Kafka with its consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound), and the breadth run over the six other filters, one dataset each, and why; the ten validation checks, the script that runs six of them, the check script for a run and their outcomes; the night's, the ramp's, the housekeeping's and the breadth run's measurements, with the null test's residuals labelled as artefacts; what is not yet decided, cross-band association among it | +| [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-10-24 on the second iteration (the bright-star residual flagged, not fixed; Spot as a probe), the association in the difference job; Ben's rulings of 2026-10-22 on what follows the wide-area ramp (housekeeping first, the other filters before injection, the reference bound by `mjdobs`, a reference per dataset, field and filter with the chain in admission order, injection outside the pipeline, the container-side producer last, the `catalogs` key with its dataset, difference outputs trimmed, GBTDS after injection, the PSF from the CRDS ePSF reference, Spot parked; the other filters as datasets of their own), his rulings of 2026-10-21 evening (the instance type recorded with no family pinned; the exercised HLWAS pipeline as the plan in force), his rulings of 2026-10-20 and 2026-10-21 on the foundation, its SMDC step and the night through the chain (latency later, the PSF kind, the read role, the Batch retry), his rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | +| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs with settings resolved from three layers, jobs, recorded inputs with the object catalog read at execution, scientific keys, official bindings and change groups, publish-last; the PSF key's kind and the `catalogs` key's dataset; the decisions with their reasons; the verbs, the `file://` and `s3://` prefixes with the working directory removed after upload, and a job on Batch with its retry rule, instance type, Batch job id and per-stage memory; Spot as a probe and its four rules; the thin chain, with the `intake`, `import`, `psf` and `packet` stages, association in the difference job, the reader, the quality gate, and the publication to Kafka from the packet job with the recovery sweep and the direct consumer check; the eight queries and the team's spatial shapes; the procedure proof; the SMDC procedure, the night run on it (cutover, live loop, campaign, one group making it official, a mid-night replacement, undo) and the HLWAS run from SOC L2 files to the topic (the live run beside the GBTDS one, one pointing, the loop, the ramp to 426 epochs) and the housekeeping after it (bytes, Batch, the pass, the reference bound), and the breadth run over the six other filters, one dataset each, and why, the fixes and the second iteration; the ten validation checks, the script that runs six of them, the check script for a run and their outcomes; the night's, the ramp's, the housekeeping's, the breadth run's, the fixes' and the second iteration's measurements, with the null test's residuals labelled as artefacts, the bright end flagged, completeness and purity of injected sources per magnitude, and Spot's first data points; what is not yet decided, cross-band association among it | +| [`datasets.md`](datasets) | DRAFT | The simulations the rebuild reads, one section each (HLWAS r00342, GBTDS r00340, OpenUniverse TDS): their data, their quirks and each quirk's disposition on the dataset's settings overlay, a reader or a recorded line | | [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; inputs told apart by key, the PSF kind among them; running a job locally, again or on Batch, with its image digest, instance type, Batch job id and peak memory; exit codes, a refusal because the run is not live exiting 0, and the Batch retry of exit 75; `admit` and `finalize`, still on the older contract | | [`reference.md`](reference) | DRAFT | The `reference` stage on `run(job)`: its scope and `l2files` inputs, the awaicgen coadd and SExtractor catalog in `dev`'s order, the header stamp, the three declared tables and the drawn `rfid`, its settings, and what it measured | -| [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, one differencer published, the gzipped files it keeps and why, its settings and determinism, and what it measured against `dev` | -| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the check that its science PSF was made from the file's own ePSF reference, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings, and what the ramp measured | +| [`difference.md`](difference) | DRAFT | The `difference` stage on `run(job)`: its scope and four inputs with the two PSF kinds, the ZOGY, SFFT and naive steps in `dev`'s order, the declared `diffimages` row and source set with the drawn `pid`, the association with its objects and memberships, the bright-star footprint flag and the quality numbers, one differencer published, the gzipped files it keeps and why, its settings and determinism, and what it measured against `dev` | +| [`intake.md`](intake) | DRAFT | The `intake` stage on `run(job)`: one SOC L2 ASDF file under the run's source root, the conversion ported from `dev`, the check that its science PSF was made from the file's own ePSF reference, the declared `l2files` row and the `exposures` row of the one SCA that expects it, with the packed `expid`, its settings (the reader, the DQ mask, the WCS translation and re-derivation) and the Gaia-offset metric, and what the ramp measured | | [`psf.md`](psf) | DRAFT | The `psf` stage on `run(job)`: the CRDS ePSF reference a file names, fetched by that name; the fixed reduction to a pipeline PSF; the science and reference kinds it declares and the `EPSFREF` the intake checks; its settings; the F146 comparison with `dev`'s copied PSF and the job's measurements; the position, the reference PSF rule and a reference part in the key, not yet decided | | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block | | [`maintain.md`](maintain) | DRAFT | The `maintain` stage: the once-per-date CLUSTER and ANALYZE of a `sources` child table moved out of `load`, its `detector-date` unit, its manifest and exit codes | @@ -34,7 +35,7 @@ repository's history. The rebuild starts from this page. | [`photometry.md`](photometry) | DRAFT | Forced photometry: the alert-driven measurement inside the `packet` stage, and `dev`'s standalone `forcedPhotometryForField.py`, not ported | | [`export.md`](export) | DRAFT | The `export` stage as ported from `dev`'s `generateSourceHATSCatalog.py`: the named source sets it reads, the CSV dump and the `hats-import` build, the `catalog-export` bundle and registration, its settings and exit codes; the light-curve catalog is not ported | | [`tool.md`](tool) | DRAFT | The command-line tool `rapidpipe`, every subcommand and its flags, the operations mapped from the specification's Tools sentence onto subcommands, the input-set composer, the tool's own exit codes, personal submission from a workstation, and where it runs | -| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments, one pass's order from the psf submission and hold through intake to Kafka with the listing by the selection, the Batch reconcile, association from the field's official chain and a flush per set, pending, failed and drained, the reference bound and held fields, the per-field chains in admission order, its exit codes and what it measured, six filters at once among it; it replaces the run model's processing-date loop | +| [`loop.md`](loop) | DRAFT | The loop, `rapidpipe loop`: the selection as arguments with the passes, one pass's order from the psf submission and hold through intake to the Kafka recovery sweep with the listing by the selection, no field chains, the submission cap and the Spot queue, the Batch reconcile, pending with one automatic rerun, failed and drained, the reference bound and held fields, the per-field chains' history, its exit codes and what it measured, six filters at once among it; it replaces the run model's processing-date loop | | [`operations.md`](operations) | DRAFT | Operations proposals the team has not yet ruled on: what they assume about the mission, three shapes of the run model and the recommended named stream of batch runs, replacement scope and the chain switch, batch cadence against the latency requirement, a proposed production check policy, reference eligibility and selection, staged inputs of a production run, capacity lanes | | [`observability.md`](observability) | DRAFT | The `rapidpipe` log line and where it goes, the record of an attempt with and without CloudWatch, monitoring with `rapidpipe` alone, queue, execution and orchestration time per Batch job, profiling a stage, and today's job-duration baseline against a 30-minute target | @@ -44,6 +45,7 @@ repository's history. The rebuild starts from this page. specification decisions foundation +datasets stage-contract reference difference diff --git a/system/intake.md b/system/intake.md index 211ab7e..62d1b8c 100644 --- a/system/intake.md +++ b/system/intake.md @@ -104,12 +104,23 @@ The run's `[intake]` table, with `dev`'s hard-coded values as keys: |---|---|---| | `source_root` | the run's overlay only | | | `unsigned` | false | | +| `reader` | `"soc-l2"` | the only format `dev` reads; `"openuniverse"` reads OpenUniverse 2024's `simple_model` ASDF through the same conversion | | `sip_degree` | 5 | `SIPDISTORTIONDEGREE`'s default | | `sip_max_pix_error` | 0.1 | the `to_fits_sip` argument | | `gzip_level` | 1 | `L2GZIPLEVEL`'s default | | `program_id` | -1 | the `SOCSIMS` program id | | `[intake.nominal_zptmag]` | `dev`'s fallback zeropoints | used only when the file has no photometry metadata | | `limmag_nsigma`, `limmag_clip_sigma` | 5.0, 3.0 | as `[import]` has them | +| `dq_mask_bits` | 0 | nothing masked, `dev`'s behaviour; a dataset's overlay names the bits whose pixels become NaN | +| `crpix_shift` | `[0.0, 0.0]` | the file's WCS unchanged; a dataset's measured translation, in pixels, added to `CRPIX1` and `CRPIX2` | +| `rederive_wcs` | false | the file's gWCS; true re-derives it from the file's `wcsinfo` and the distortion reference it names, before the SIP fit | + +Each dataset's values, and the measurement behind each, are on the +[datasets](datasets) page. When the job names its dataset's Gaia +`catalogs` input, as the loop does once one is official, its metrics +record `gaia_dx_px`, `gaia_dy_px` and `n_gaia_stars`: the median offset +of the isolated Gaia stars on the frame, measured minus predicted +through the published WCS, a check of the WCS on every file. ## What it measured diff --git a/system/loop.md b/system/loop.md index 80c1382..219fcc1 100644 --- a/system/loop.md +++ b/system/loop.md @@ -32,11 +32,16 @@ nothing to submit, the selection is drained and the loop exits. ``` rapidpipe loop --run --filters F146 --scas 8 --pointings all \ - --job-definition [--max-intake 100] [--interval 120] [--once] + [--passes 1,2,3] --job-definition [--job-queue ] \ + [--max-intake 100] [--interval 120] [--once] ``` -The selection is three arguments: filters, SCAs, and pointings, either -`all` or a list of `VVV_EEEE`, a visit and an exposure. A pointing is +The selection is three arguments and an optional fourth: filters, SCAs, +pointings, either `all` or a list of `VVV_EEEE`, a visit and an +exposure, and the survey's passes, applied to the listed files and the +L2 rows alike. A GBTDS pointing is the visit modulo six, so it does not +bound that survey's epochs; a sample there is a source root of chosen +files ([datasets](datasets)). A pointing is the same (visit, exposure) pair in every pass of a filter; the observation number is not part of it, since it can change between passes of one filter. The selection is the operator's instruction for @@ -100,9 +105,22 @@ Its order: catalog, the science PSF and the source set; the objects, their members, the forced window's images and the official packet sets' forced rows it reads as they stand when it runs, and records. -6. **Kafka**: `publish` over the run's official, unstamped packet sets - ([foundation](foundation), Publication to Kafka), on one connection - and one producer, one set at a time. An idle pass opens no producer. +6. **Kafka**: the recovery sweep. A packet job sends its own set as its + last act; the pass's `publish` takes only the official, unstamped + sets whose jobs ended more than 300 s ago and the quality gate does + not hold ([foundation](foundation), Publication to Kafka), on one + connection and one producer, one set at a time. A set inside the + window is reported as `sets_waiting` and keeps the pass from + draining; a held set is reported in `held_sets` with its metric and + value. An idle pass opens no producer. + +A pass submits at most the run's `[batch] max_submissions_per_pass` +(1,000), in stage order, and reports the rest as `deferred`: the +breadth night's 3,456 intakes submitted at once throttled the parameter +store and 1,448 failed. With `--job-queue` the submissions go to that +queue; a run whose `[batch.spot]` names a Spot queue sends each stage's +backlog there under the four rules of +[Spot, a probe](foundation) and reports `queues` and `attempts`. There are no field chains and no admission order ({ref}`ruling `): every epoch with a @@ -111,8 +129,11 @@ packet job at once, so the loop's only waits are the reference's frames and the queue. **Pending and failed.** A job is pending while it is `launched` or -`running`. A failed job is reported and never resubmitted: its key -counts as taken until a person reruns it, with `job run --job ` on +`running`. A job whose container Batch ended without `job run` +recording it is resubmitted once, its Batch name ending `-rerun`, and +held pending; if the rerun's container dies too, the job is failed. A +failed job is reported and never resubmitted: its key counts as taken +until a person reruns it, with `job run --job ` on the host or `job submit --job ` on Batch, and the loop then carries on from it. @@ -122,7 +143,8 @@ unpublished Batch jobs, a hundred ids a call, and reads the answer two ways. A Batch job that has ended while its row still says `launched` or `running` is a container that ended without `job run` recording it: the row is marked `failed`, with Batch's exit code and its status reason in -the row's `refusal`, and the pass reports it like any failed job. A row +the row's `refusal`, and it gets its one rerun (Pending and failed, +above). A row marked `failed` whose Batch job is still queued or running is Batch's own retry of exit 75 in flight, and the pass holds it pending rather than exiting. Once Batch has ended a job, the job's own record is the