diff --git a/AGENTS.md b/AGENTS.md index d4a24af..3afdb09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,15 +196,16 @@ on anything outside it. ## The system section `system/` holds one page per register: `specification.md` is the design -authority; `decisions.md` records the team's rulings; `products.md`, -`runs.md` and `stage-contract.md` are the three model pages everything -else builds on; each pipeline stage has its own page (`reference`, +authority; `decisions.md` records the team's rulings; `foundation.md` +and `stage-contract.md` are the two model pages everything else builds +on, and `foundation.md` governs where an older page still describes the +retired run model; each pipeline stage has its own page (`reference`, `difference`, `load`, `maintain`, `crossmatch`, `statistics`, `finalize`, `alerts`, `prune`, `export`); `photometry.md` is not a stage page, since the rebuild carries no photometry stage, and holds -the unported-forced-photometry note instead; `releases.md`, -`tool.md`, `loop.md`, `checks.md` and `observability.md` cover the -mechanisms that run stages rather than a stage itself; and +the forced-photometry note instead; `tool.md`, `loop.md` and +`observability.md` cover the mechanisms that run stages rather than a +stage itself; and `operations.md` holds operations proposals still open. Dated reviews go in the log, not in `system/`. `index.md`'s table is the map: its Scope column names what each page owns, and a page's own content should never diff --git a/log/2026-09-26-direction-review.md b/log/2026-09-26-direction-review.md index 7a337b9..2da2ddc 100644 --- a/log/2026-09-26-direction-review.md +++ b/log/2026-09-26-direction-review.md @@ -154,7 +154,7 @@ These go beyond consistency: each removes a mechanism. - **One notion of "current".** `vbest` on the `dev` tables and `current_selection` both answer "which one do consumers see", and promotion maintains both. Consumers moving to `current_selection` is - already the recorded direction ([products](../system/products.md)); + already the recorded direction ([foundation](../system/foundation.md)); retiring `vbest` maintenance waits for the team's existing queries to move. - **The transitional cleanup grant.** The workstation role still carries @@ -268,7 +268,7 @@ field per source changes no row count: it writes one column's value. The lead's approval of that stamping (2026-09-26) is recorded on [load](../system/load.md). -Two consequences for the pages. [products](../system/products.md) says ZOGY +Two consequences for the pages. [foundation](../system/foundation.md) says ZOGY registers "as in `dev`", but `dev`'s own run of 2026-08-09 registered its SFFT difference for this exposure; which differencer `dev` registers is a question for the lead. And the rebuild's execution records hold `{}` for diff --git a/system/alerts.md b/system/alerts.md index 0937af8..5ee8eb9 100644 --- a/system/alerts.md +++ b/system/alerts.md @@ -2,6 +2,11 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + The `alerts` stage builds an Avro container for one finalized difference image and records each alert in an outbox. It lands on the pipeline repository's `rebuild` branch @@ -12,7 +17,7 @@ repository's `rebuild` branch `pipeline/produceAlertsForProcDate.py` and `alerts/` (schema `rapid.v00_04`). Every ported stage minimises differences from `dev` and leaves optional features designed in but off by default. The -[products](products) page fixes the vocabulary; this page states how +[foundation](foundation) page fixes the vocabulary; this page states how the stage meets it and amends two of its paragraphs below. ## In plain terms @@ -176,7 +181,7 @@ pruned-set rule above. "Any result set" is restricted here to the source sets named by the association chains (each key's `source_sets`). Every history and -association read must satisfy the [products](products) page's reading +association read must satisfy the [foundation](foundation) page's reading rule: the source set must be complete, retained, and either the alerts run's own or `candidate`/`current` from a selected attempt. A chain naming a scratch or unselected source set exits 65. Only the named sets @@ -300,7 +305,7 @@ cutout are small enough to share a block. `UNIQUE (instance, candidate)` is the row's own idempotency key, the same pair a rerun's recovery reads back. -This amends the general locator description in [runs.md](runs)'s +This amends the general locator description in [foundation](foundation)'s storage-layout paragraph: the locator is an Avro block plus the record's position within it and within the container. diff --git a/system/checks.md b/system/checks.md deleted file mode 100644 index c7cf151..0000000 --- a/system/checks.md +++ /dev/null @@ -1,199 +0,0 @@ -# Candidate checks - -**Status: DRAFT** - -A check examines one product instance in the database and records -whether it passes. A check policy names and versions a list of required -and advisory checks with their bounds. Promotion runs the required -checks and refuses if any is missing or failed. This page defines how -checks run without changing how promotion treats their results, meeting -the requirement in the [runs](runs) page's Promotion eligibility -paragraph and the [specification](specification)'s Promotion section. - -Every promotion in the rebuild so far, including every demonstration on -this page and the [runs](runs) page, has used `rebuild-trial@1` for a -person's `run promote` by hand. Its trial approval permits manual -promotion. Automatic promotion is built and tested but stays off in -production: no shipped policy has the team's approval to permit it. - -## What a check is - -A check is a named, versioned Python function over one product -instance. It is registered in `rapidpipe/checks/`, keyed by -`name@version`, with a declaration such as -`@check("difference-image-statistics", "1")`. Its signature is -`(conn, instance_id, params: dict) -> CheckResult(outcome 'passed'|'failed', detail dict)`. -Versions are strings; a changed threshold or measurement requires a -new version. Both shipped checks read only the database; future checks -may read S3. - -Running a check always records a `checks` row: instance, check name, -version, the policy's required flag, outcome, and a detail document -containing the measurements, bounds, params and reason for the outcome. -A check that raises records `failed` with the error in `detail`. -Policies can override a shipped check's bounds, so the promotion gate -counts only rows whose recorded params match the resolving policy's -params for that check. - -## The two checks - -`difference-image-statistics@1` runs over a `difference-image` -instance. It reads that instance's `diffimmeta` row through -`diffimages.instance` and passes when `scalefacref` falls within a -policy-supplied `[lo, hi]`, `dxrmsfin` and `dyrmsfin` are each at most -`rms_max`, `|dxmedianfin|` and `|dymedianfin|` are each at most -`median_max`, `nsexcatsources` falls within `[n_min, n_max]`, and, when -`source_counts` is present, the positive-to-negative SExtractor count -ratio falls within `[ratio_lo, ratio_hi]`. All bounds come from the -policy's `params` for that check. - -`catalog-counts-vs-reference@1` runs over a `source-set` result-set -instance. It finds its reference by slot instead of walking dependencies -by hand. The slot names the candidate's exposure, detector and catalog -type ([products](products) page). The reference is another current -`source-set` instance in the same slot, from any differencer, or, when -`params` names a `reference_run`, that run's instance for the same slot. -A candidate or would-be reference without a slot is not a reference. -The runner fills every candidate's slot first, so a set lacks one only -when its producer is still unresolved. The check compares the candidate's -`result_sets.row_count` against the reference's and passes when -`|candidate − reference| / reference` is at most `params.tolerance`. -When no reference instance exists, the outcome follows -`params.missing_reference`, recorded in `detail`; policy -`rebuild-trial@1` sets that to pass. - -## Check policies - -A check policy is a named, versioned TOML file shipped in the package, -at `rapidpipe/checks/policies/@.toml`, loaded by the -same `name@version` key as a check. Its fields are `name`, `version`, -`approval` (`none`, `trial` or `team`), `approved_by` (nullable; who -granted the recorded approval level), `auto_promote` (boolean), and a -`[[checks]]` table per check listing its name, version, kind, required -flag and params. A policy is immutable once landed; changing any field -requires a new version. - -Manual and automatic promotion refuse a policy whose `approval` is -`none`. A `trial`-approved policy can gate a person's promotion; -automatic promotion requires `team` and `auto_promote` true. - -The rebuild ships one policy, `rebuild-trial@1`. It marks -`difference-image-statistics` required and `catalog-counts-vs-reference` -advisory, required false. Its bounds come from the control run's measured -values with generous margins: -`scalefacref` in `[1e-3, 1e5]`, `dxrmsfin` and `dyrmsfin` each at most -2.0 pixels, `|dxmedianfin|` and `|dymedianfin|` each at most 1.0 pixel, -`nsexcatsources` in `[1000, 1e6]`, and the SExtractor positive-to-negative -ratio in `[0.1, 10]`. It carries `approval: trial`, `approved_by` -recording who granted it, and `auto_promote` false. This trial approval -suffices to demonstrate the manual-promotion mechanism required by the -runs and specification pages; the team's own sign-off remains open. - -`rebuild-strict@1` is a refusal test fixture using the same two checks -with bounds the control run cannot meet: `scalefacref` in -`[0.99, 1.01]`, the same two RMS measurements each at most 0.01 pixel, -and `nsexcatsources` at most 1000. It does not ship; naming it to a -command is an unknown policy. - -There is no policy table in the database. The promotion row durably -records what ran through the policy's version string and the ids of the -check rows it relied on. The TOML file defines what that version meant. - -## The check commands - -`run_policy_checks`, the runner called by both `check run` and the -promotion gate, fills every candidate's slot before running any check. -Thus `catalog-counts-vs-reference` never sees a slot the fill could have -resolved. Checks run on the workstation and reach the database through -the instance role; they do not touch the pipeline image. The [tool](tool) -page gives the full command surface for `check list`, `check run` and -`check show`, including every flag, print format and exit code. - -## The promotion gate - -`promote()` takes an optional check policy. `promote_run` and `run -promote` resolve which one to validate against in this order: an -explicit `--check-policy`, then the run's own `check_policy_ref`, then -the default `rebuild-trial@1`. Promotion refuses an unapproved policy -(`approval: none`) before reading any check result. - -For each after-instance, every check in the approved policy whose kind -matches the instance must have a latest `checks` row for its name, -version and policy params. Required checks must have outcome `passed`. -Ties in `happened_at` break by row id. The read locks the rows it relies -on; a check recorded after the read starts is not counted. - -A missing or failed required check refuses the whole promotion, exit 1, -with a message naming the instance, check and outcome. A failed or -missing advisory check is recorded but does not refuse. The promotion -row records `check_policy_version` and `check_result_ids`, the rows it -relied on. A kind with no check in the policy passes trivially. There is -no unchecked-exception flag: a deliverable of a kind covered by the -policy always goes through this gate. - -Acceptance means required checks pass, nothing more ({ref}`acceptance -ruling `). There is no separate acceptance record: -the gate re-reads the latest result of every required check each time. -The `checks` row is the durable record of what let a candidate through; -no judgement is stored on the instance. - -### Dependencies and rollback - -Promotion walks every dependency to its roots. -`_refuse_unpromoted_ancestor` follows `dependencies` recursively for -each after-instance, with a cycle guard and no depth cap. It reads the -same instance states that `run show` prints ([runs](runs) page, -"Rules"). A `current` or `superseded` ancestor passes because its chain -was judged when it was promoted. The walk still continues past it, so a -`candidate` grandparent behind a current parent refuses promotion. - -An ancestor included among the same request's after-instances also -passes the walk, but must independently meet the same eligibility rule -and check policy as every after-instance. Any ancestor that is neither -`current` nor `superseded` nor an after-instance in the request refuses -the whole promotion, exit 1. The message names the after-instance, -ancestor, kind and state: "after instance '01...' depends on '01...' -(difference-image, candidate: not current or superseded; promote it -first or in the same request); refusing", the hint appearing only for -a `candidate` ancestor. A refusal changes no selection or custody; the -transaction raises before anything is written. - -Rollback skips the check-policy gate and the ancestor walk because it -restores a selection an earlier promotion admitted, the same treatment -the released-image check gives it. Each direct dependency of the -restored selection must still be complete, retained and in project -custody; only the walk past those direct dependencies is skipped. - -### Accepted defect - -The gate's read of the latest check rows takes its lock `FOR SHARE`; a -failed check row committed after that read is not seen by the -promotion it should have refused. The dependency walk reads no check -rows: an ancestor's state is custody, completeness and selection. This -is an accepted defect, not fixed. - -## Automatic promotion - -`run create --auto-promote [--check-policy P]` is accepted only when the named -policy carries the team's approval, recorded as `approval: team` in the -policy file, and its `auto_promote` is true; otherwise it is refused, -exit 1, with a message naming the policy and stating that team -approval is pending. - -At the end of a `run start` walk, once every unit is complete, the tool -calls `maybe_auto_promote(conn, run_id)`: with the run's `auto_promote` flag -true, it runs the resolved policy's checks over the run's candidates and -promotes on a pass; otherwise it prints `auto-promote off (policy -)` and does nothing further, immediately before `start`'s own -final `run= state=complete` line. Every run created so far prints -this line, since no shipped policy permits automatic promotion. -Tests exercise the path end to end with a fixture policy. In production -it can currently only print: no policy carries the approval to act. - -## Not decided here - -- The team's own sign-off on a policy, beyond `rebuild-trial@1`'s trial - approval, whether an approved policy permits automatic promotion, and - the content of any policy approved for it. These are scientific - questions for the team. -- Checks that read S3 rather than only the database. diff --git a/system/crossmatch.md b/system/crossmatch.md index 2464f62..b922904 100644 --- a/system/crossmatch.md +++ b/system/crossmatch.md @@ -2,9 +2,14 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + `crossmatch` assigns newly loaded sources to astronomical objects and records the associations in `astroobjects_` and `merges_`. -The [products](products) page fixes the vocabulary. +The [foundation](foundation) page fixes the vocabulary. ## In plain terms @@ -236,7 +241,7 @@ recorded disposition. It does not reuse a different attempt's set if that attempt committed rows and then failed, or still has no disposition. Those rows stay the run's own, but a fresh attempt writes a fresh set. The check uses `db/objects.find_complete_result_set`, joining `attempts`; -the [runs](runs) page has the general rule. The gap between a committed +the [foundation](foundation) page has the general rule. The gap between a committed write and a published manifest remains a stage-contract matter, not closed here. diff --git a/system/decisions.md b/system/decisions.md index 2881b1d..4e87829 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -9,6 +9,16 @@ page where attribution matters. ## Rulings +Ben's ruling of 2026-10-20 on the foundation. The +[foundation](foundation) page states the design it opened. + +(decision-foundation-dev-prior-art)= +**`dev`'s shapes are prior art**: the rule that the rebuild coexist with +`dev`'s schema, and the principle that every port minimise its +differences from `dev`, are revoked. `dev`'s table shapes and column +names are kept where they cost nothing and the team's queries read +them. (Ben, 2026-10-20) + Ben's rulings of 2026-10-08 on the pipeline's Python environment, the rapid-python RPM, after a review of the environment. The infrastructure repository's rapid-python README states the design they produced. @@ -144,25 +154,25 @@ makes them the team's; each page states the rule. - 2026-09-21: keep the `dev` schema: rename or drop nothing; add columns and tables where the vocabulary needs them - ([products](products)). The rebuild's schema coexists with `dev`'s and - writes it ([runs](runs)). + ([foundation](foundation)). The rebuild's schema coexists with `dev`'s and + writes it ([foundation](foundation)). - 2026-09-21: retain run records in full; deleting a run's data changes - state, not history ([runs](runs)). + state, not history ([foundation](foundation)). - 2026-09-21: current result sets are readable by any run as frozen inputs, by instance id; scratch result sets only within their own run - ([products](products)). + ([foundation](foundation)). - 2026-09-22: every ported stage minimises differences from `dev` and makes anything that could be left unused optional and off by default (every stage page). The catalog stage's detection member is per-differencer and recorded with the instance; `vbest` stays the query surface - ([products](products)). + ([foundation](foundation)). - 2026-09-22: all promotions take one transaction-scoped advisory lock - ([runs](runs)). + ([foundation](foundation)). - 2026-09-23: `dev`'s once-per-date CLUSTER and ANALYZE move to their own `maintain` stage ([maintain](maintain), [load](load)). - 2026-09-23: only one producer's frozen inputs bind at a time - ([runs](runs)); rebuild deletion and cleanup never touch anything - `dev` wrote ([runs](runs)). + ([foundation](foundation)); rebuild deletion and cleanup never touch anything + `dev` wrote ([foundation](foundation)). - 2026-09-23: no per-child UNIQUE on `(pid, id, isdiffpos)`; SFFT loads against its own `pid`; a psf instance's `version` equals the allocated legacy number and `psfs`' key stays global ([load](load)). @@ -190,7 +200,7 @@ removes it here. - **Trial policy.** `rebuild-trial@1`'s `approval: trial` is a trial approval, not the team's sign-off; promotion refuses a policy with `approval: none`, and automatic promotion needs a team approval and - `auto_promote: true` ([checks](checks)). (2026-09-24, 2026-09-25) + `auto_promote: true` ([foundation](foundation)). (2026-09-24, 2026-09-25) - Every check records a row; one that raises records `failed`. (2026-09-24) - `catalog-counts-vs-reference@1` finds its reference by slot; `rebuild-trial@1` marks `difference-image-statistics` required and @@ -235,7 +245,7 @@ removes it here. (2026-09-24) - Run deletion locks the run, verifies the owner, and refuses while a frozen input binding depends on it or a reference row would be - orphaned; the science-row cleanup set is fixed on [runs](runs). + orphaned; the science-row cleanup set is fixed on [foundation](foundation). (2026-09-24) - Every submission binds its input set before allocating the attempt. A name that resolves to nothing is logged, not refused. An unreadable @@ -319,7 +329,7 @@ removes it here. ([stage-contract](stage-contract)) (2026-09-27) - **Running checks against a run.** That code lives in `rapidpipe.runs`, not `rapidpipe.checks`, since `checks` sits below `runs` in the layer - order. ([stage-contract](stage-contract), [runs](runs)) (2026-09-27) + order. ([stage-contract](stage-contract), [foundation](foundation)) (2026-09-27) - **`run --help` grouping.** `rapidpipe run --help` groups its subcommands under five titled sections, lifecycle, recovery, promotion, housekeeping and inspection ([tool](tool)); no subcommand @@ -357,7 +367,7 @@ removes it here. ### Releases -- [Releases](releases) defines the release command, tag form +- [foundation](foundation) defines the release command, tag form (`rebuild-v0.`, annotated, remote authoritative), and `releases` and `release_deployments` tables. Account-specific steps are hooks. Migrations are applied with recorded checksums before a cut records. @@ -397,7 +407,7 @@ removes it here. (decision-slot-identity)= - **Association-set identity.** The slot-identity table on - [products](products) is the sole statement. An association set's + [foundation](foundation) is the sole statement. An association set's identity adds the field, crossmatch settings hash, sorted source-set identities, and a hash of its base's own identity; the proposal's shorter wording, which omitted the base, is withdrawn with the table copy it @@ -406,7 +416,7 @@ removes it here. (decision-pending-sharing-rule)= - **Sharing rule widening.** The specification lets any run read *current* result sets and keeps scratch within its run; the eligibility - table on [products](products) also lets another run's stage read a + table on [foundation](foundation) also lets another run's stage read a *selected candidate*, whatever its check outcome. The team must decide whether selected candidates are readable across runs, then align the specification or the table. (2026-09-26) diff --git a/system/difference.md b/system/difference.md index 1641a58..cc18427 100644 --- a/system/difference.md +++ b/system/difference.md @@ -2,6 +2,10 @@ **Status: DRAFT** +This stage still runs on the older contract ([stage-contract](stage-contract), +"Stages on the older contract"). Where this page describes registration, +custody, runs or promotion, the [foundation](foundation) page governs. + The `difference` stage takes one detector image and the reference for its field, subtracts them with ZOGY exactly as the `dev` science pipeline does, and writes the difference, uncertainty and significance images @@ -14,7 +18,7 @@ The stage lands on the pipeline repository's `rebuild` branch (`rapidpipe/stages/difference.py`, `rapidpipe/settings/difference.toml`, `rapidpipe/products/diffimage.py`, `rapidpipe/db/diffimages.py`). Every ported stage minimises differences from `dev` and designs in, off by -default, anything that could be left unused. The [products](products) +default, anything that could be left unused. The [foundation](foundation) page fixes the vocabulary and field list used here. ## Inputs diff --git a/system/export.md b/system/export.md index 0fa725d..7137320 100644 --- a/system/export.md +++ b/system/export.md @@ -2,10 +2,15 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + The `export` stage builds a HATS (Hierarchical Adaptive Tiling Scheme) source catalog from named, completed source sets and publishes it as one `catalog-export` instance through the manifest, like other stages' -products. The [products](products) page fixes the vocabulary. +products. The [foundation](foundation) page fixes the vocabulary. The stage lands on the pipeline repository's `rebuild` branch (`rapidpipe/stages/export.py`, `rapidpipe/settings/export.toml`, @@ -164,7 +169,7 @@ never the unnamed set or the association set. `pipeline/generateLightCurveHATSCatalog.py`, not ported; `[export] catalog_type = "light-curves"` exits 64. - Delivery: where a `catalog-export` product is read from once made, - since [products](products) marks it "none; exported." + since [foundation](foundation) marks it "none; exported." - Catalog naming: `[hats] catalog_name` fixes one directory name per settings profile; how two exports of the same field and export type, or a real deployment serving several catalogs, would be named apart is diff --git a/system/finalize.md b/system/finalize.md index 616c8f0..b3b7d57 100644 --- a/system/finalize.md +++ b/system/finalize.md @@ -2,6 +2,10 @@ **Status: DRAFT** +This stage still runs on the older contract ([stage-contract](stage-contract), +"Stages on the older contract"). Where this page describes registration, +custody, runs or promotion, the [foundation](foundation) page governs. + The `finalize` stage stamps a difference image's header and republishes its bundle as new instances. It lands on the pipeline repository's `rebuild` branch @@ -10,7 +14,7 @@ its bundle as new instances. It lands on the pipeline repository's ported from `dev`'s post-processing pipeline, `ppid` 17 (`pipeline/awsBatchSubmitJobs_runSinglePostProcPipeline.py`). Every ported stage minimises differences from `dev` and leaves optional -features designed in but off by default. The [products](products) page +features designed in but off by default. The [foundation](foundation) page fixes the vocabulary. ## In plain terms diff --git a/system/foundation.md b/system/foundation.md new file mode 100644 index 0000000..1e15c52 --- /dev/null +++ b/system/foundation.md @@ -0,0 +1,491 @@ +# The foundation and the thin chain + +**Status: DRAFT** + +This page states how RAPID records what it made and which of it is +official, and the narrow chain of stages built on that record, from +stored images to alert packets. It is the design the `rebuild` branch +of the pipeline repository implements: `database/schema.sql`, the +package `rapidpipe.foundation`, the stages `import`, `associate` and +`packet`, the reader `scripts/read_packets.py` and the queries in +`database/queries.sql`. The [stage contract](stage-contract) page +states what a stage is under it, and the [decisions](decisions) page +records the rulings it rests on. + +## In plain terms + +A run is a recipe: who launched it, why, from which code, with which +settings, for which survey and dataset. A job is one execution of one +stage under a run. Every catalog row the pipeline publishes names the +job that wrote it, and nothing published is ever changed. + +Which rows are official is a separate record. One table holds, for +each product key, the binding that says which job's rows are official, +and for each survey and dataset, which run is live. Bindings change +only in recorded groups, and every group can be undone. Each product +table has a view that shows its official rows. + +A stage reads what its job was given at launch, computes in memory, +writes files under a prefix that names the job and attempt, and stops. +The publisher then loads the declared rows in one short transaction +and, when the run is live and nothing has moved in the meantime, makes +them official. A retry is a new attempt of the same job; once a job is +published, a retry does nothing. + +## The foundation + +Six statements define it. Each is one mechanism. + +### Runs + +A run is a recipe someone launched for a reason. The `runs` row holds +who, when, why, the code revision, the full settings body inline, and +the one survey and dataset the run is for. There is no settings table +and no resolution of settings at run time: a stage reads its own table +from the run's body. + +### Jobs + +A job is one execution of one stage under a run. The `jobs` row holds +the stage, its scope, its state (`launched`, `running`, `published` or +`failed`), the image digest it ran under (none for a job run from a +checkout), the settings hash, the attempt whose outputs were published, +its manifest, the launched, started and ended times, the exit code, and +the reason an automatic binding was refused, if one was. Local and +Batch executions both get a job row. A killed execution is a job row +that never reached `published`. + +### Published rows name their job + +Every product row carries `job`, never null, and each scientific key is +unique per job. Two executions over the same input therefore coexist +and neither overwrites the other. A row reaches its run through its +job, and the official views expose both. + +Each product has one scientific key, and dataset is in every key, so a +source-injected variant and its original are distinct products with +distinct live streams: + +| Product | Scientific key | Rows | +|---|---|---| +| `exposures` | dataset, expid | `exposures` | +| `l2files` | dataset, expid, sca, fid | `l2files` | +| `diffimages` | dataset, expid, sca, fid | `diffimages` | +| `psfs` | dataset, fid, sca | `psfs` | +| `refimages` | dataset, field, fid | `refimages`, with children `refimimages` (members) and `refimcatalogs` (mosaic catalogs) | +| `sources` (source set) | dataset, expid, sca, fid | `sources` | +| `objects` (object set) | dataset, survey, field | `objects`, with child `memberships` | +| `packets` (packet set) | dataset, expid, sca, fid | `packets`, with children `forced` and `external` | +| `catalogs` (reference catalog) | name, version | `catalogs` | + +A source is identified by its set and its source id, an object by its +set and its object id, a packet by its set and its source. A set is +identified by the job that wrote it, so a job that writes a set binds +exactly one key of that set's product. + +The pipeline id that `dev` carries in nearly every image query is not +in any key: a run has one differencer, and a second differencer is +another run. + +### Inputs are chosen at launch and recorded + +`job_inputs` holds, for each job, rows of three kinds. An `input` row +names a set the job reads, by product, key and the job that wrote it; +an object-set job's predecessor is its `input` row of product +`objects`. An `expects` row names, for one output key, the open binding +the job expects to replace, or none. An `excludes` row, for `associate` +only, names a source set of the predecessor's chain that is no longer +official, whose members the job drops. + +The launcher chooses these. For the live run it chooses the current +official sets; a campaign chooses its own previous sets, and none for +its first epoch. A stage reads the sets its rows name and never "the +current official" one, which is what lets a recipe that is not live, +or a correction of one downstream product, read sets that are not +official. + +### One table of official bindings + +`official` holds two kinds of binding. A product binding points a +(product, key) at one job's rows; a partial unique index allows at most +one open binding per key. A live binding points a (survey, dataset) at +one run, again at most one open. Every binding row records who opened +it, when, why and in which change group, and once closed, who closed +it, when, why and in which group. A group's full effect is read from +the rows it opened and the rows it closed. + +One operation, `change`, applies a group in one transaction: it closes +bindings, then opens them, and refuses the whole group if a binding it +expected to close is no longer open. `undo` reads a group's two sides, +builds the inverse (close what it opened, reopen as new rows what it +closed) and applies it under the same rule; it never reopens a closed +row. + +Each product table has a view `_official`: its rows joined to +their open bindings, with the `binding` and `run` columns added. There +is no stored flag. `members_official` follows an official object set's +predecessor chain and returns its memberships, except those whose +source set any set in the chain excluded. + +### Publish last + +A stage computes, writes its files and its declared Parquet outputs +under an immutable prefix, `/////`, +and returns. The publisher then, serialised on the job's row: + +1. stops at once, before reading any file, if the job is already + published; +2. verifies every declared file against its checksum; +3. loads the attempt's declared rows, and writes the job's state, + published attempt and manifest; +4. inside a savepoint, attempts the automatic binding: if the job's run + is live for its survey and dataset, and for each output key the open + binding is the one the job's `expects` row named, it closes those and + opens the job's; otherwise it rolls back to the savepoint and records + the refusal on the job; +5. commits, all of the above in one short transaction, then runs + `ANALYZE` on the tables it loaded. + +A refused job's rows stay published as an unbound candidate, visibly, +and a later change group can bind them. A retry finds its job published +and stops; a retry after a failed attempt writes a new attempt, so a +published prefix is never rewritten. + +## Why it is this way + +**`dev`'s shapes are prior art, not a constraint.** The rule that the +rebuild coexist with `dev`'s schema is revoked +({ref}`ruling `). The per-SCA row +shape and the scientific column names stay because they are the +instrument's shape and cost nothing. The columns kept per table are +the ones the team's queries read. The static tables (filters, SCAs, +surveys, fields) stay plain tables. The sky tessellation is computed in +closed form by `roman_tessellation.py` and is not a table. `dev`'s +`pipelines` and `swversions` are not carried, because the run's +revision and settings record what they recorded. + +**There is no `vbest`.** It was a second, mutable copy of the truth +with its own write path, on four tables. The binding replaces it, and +an explicit change group with a reason is the recorded act that `dev`'s +human lock (`vbest = 2`) stood for. + +**Joins between products are on scientific keys, never on another +product's row id.** `dev` joins a difference image to its L2 row by row +id, and under coexistence that row may not be the official one. "L2 +images with no official difference image" is an anti-join of two +official views on (dataset, expid, sca, fid). + +**A source is official when its set is.** `dev` has no flag on sources; +association, export and light curves were correct only because prune +deleted superseded rows first. Here `sources_official` joins source +sets to the open bindings of their image key, and no catalog row is +deleted. + +**Spatial queries stay in SQL where the team runs them.** Cone +searches on image centres, containment on overlapping fields and the +forced-photometry image search are most of the team's query shapes. +The base tables keep q3c indexes on image and object centres and a GIN +index on `overlapfields`, and every official view is a plain join so +the planner reaches them. A failed execution publishes nothing, so +`status` is a column a query filters on, not part of a view. + +**Columns `dev` backfills are computed before publish.** `dev` fills +limiting magnitude, overlapping fields and sky indices into published +L2 rows in place. Here the stage that publishes a row computes them; a +later correction is a new job's row, rebound. + +**One live run per survey and dataset, for every stage.** A retune is a +new run, which becomes live for every stage; stages whose values did +not change behave as before. A live pointer per stage would buy only a +patchwork of recipes nothing asked for. A correction run that touches +only downstream products is either not made live, its results bound by +an explicit group, or made live for everything. + +**Automatic binding compares the expected binding, not the input.** +For `associate` the set it read and the binding it replaces coincide; +for `packet` the input is an object set and the binding to replace is +the previous packet set of that image. Two jobs that both read set A: +the first publishes B and binds; the second publishes C and stays an +unbound candidate. A late job from a run that is no longer live stays +unbound for the same reason. + +**Concurrency is the unique index and the row lock.** Two groups that +close the same binding: the second blocks on the row, finds it closed, +and refuses. Two groups that open a binding on the same key: the +partial unique index refuses the second. There is no table lock and no +lock table. The publisher's live check is a plain read: a cutover that +does not touch a job's expected bindings is equivalent to one committed +after the job, and one that does is caught by the row lock on the +binding. + +**Stages read, compute in memory, and the publisher writes.** A stage +holds no connection while it computes and writes no catalog row. The +old crossmatch matched each exposure against rows it had itself written +for the previous one, inside one transaction under an advisory lock; +with outputs going to Parquet there is nothing for SQL to query +mid-job, so `associate` matches in memory, and its parity with the SQL +matcher on the same field is a test in CI. + +**Declared outputs, not every file.** A stage's manifest names which of +its files are catalog rows, and the publisher loads those and only +those. Parquet is the row format because the stages, the publisher and +an archive export all read it without conversion. + +**The database is authoritative for bookkeeping.** Runs, failed jobs +and official changes with their reasons live only in the database, and +its backups are what protect them. Catalog rows can be reloaded from a +job's Parquet; the bookkeeping cannot. + +**A cutover is one change group.** It rebinds the affected products and +the live binding together. A campaign that carries live results forward +names them in its group; nothing is inherited. A live epoch that lands +while a cutover is being prepared moves a binding the group expects to +close, and the group refuses. + +**Bytes are immutable.** Published rows reference files whose paths +contain the job id and the attempt, with their checksums. A retry +writes new files and never overwrites a referenced one. + +**Association products are sets, and sets are deltas.** An object set +is one job's output for one field. `objects.parquet` holds every object +of the field with its founding position and its statistics as +sufficient sums (count, and the sum and sum of squares of position and +flux), so the next epoch updates statistics from the predecessor's +objects and this epoch's sources without reading history. +`matches.parquet` holds only this epoch's new memberships. A full +membership snapshot per epoch would grow with the square of the epoch +count, and a GBTDS field has tens of thousands of epochs. Object ids +are position hashes of the founding source, so the same founding source +gives the same id under any recipe, and a join on object id works +across recipes for most objects. + +**The object set is the pruned set.** `dev`'s prune deleted members +whose difference image was no longer best. Here an `associate` job +learns at launch which of the predecessor's source sets are no longer +official, records them as exclusions, and drops their members from its +sums; `members_official` omits them. Nothing is deleted, and exclusions +are rare because a superseded source set is rare. + +**A packet set is per difference image.** The alert unit is one +difference image, so a packet set's key is its image key, never the +field. The current-packet view keeps every epoch, and a correction of +one epoch replaces only that epoch's set. The packet stage writes Avro +and a `packets.parquet` index that locates each record by file, offset +and length; an object's alert history is a query over the index, and a +packet's lineage is its job's recorded inputs. + +**A packet's history is bounded by its trigger.** Previous detections +are those before the trigger, and the detection count is the count as +of the trigger, so a packet rebuilt for epoch 1 after epoch 2 is loaded +carries nothing from epoch 2. + +**Forced photometry is a packet-set output with named inputs.** `dev` +caches forced rows in a per-object store that every job rewrites and no +recorded input can describe. Here a packet job names its window's +images and its predecessor packet sets' `forced.parquet` as inputs, +measures only the (object, image) pairs those do not already hold, and +publishes its own `forced.parquet` with `dev`'s row id. + +**Reference catalogs are products.** Gaia, NED, NED-LVS and the +solar-system predictions are copies whose version no packet recorded. +Each is a product keyed by (name, version), and a job names the one it +read. + +**One schema file, applied fresh.** `schema.sql` holds every table, +view and operation, with the q3c extension and its indexes in a block +that continues where q3c is absent, as on a developer's container. The +apply is one `psql -f`. There is no migration machinery until the first +change after real data; from then on changes are incremental SQL. + +Immutability is of published rows and their bytes, not of every field: +bindings close and job state changes. Retention is deferred, and its +rule when it comes is that a row that was ever official keeps its whole +input closure. Stacked inputs fit the identity model as a product with +its own key and a membership table, on the reference-image pattern. + +Given up on purpose: scratch and production custody, releases before +delivery, check policies, plan files, custom retries (Batch retries, +and publish-last makes a retry safe), a sharing rule, catalog copies, +expiry, reconstruction from storage alone, a uniform file-only stage +interface, a stored `vbest`, a live pointer per stage, a lock on the +official table, the branch's `alert_outbox` table and `maintain` stage, +and `dev`'s per-object forced-photometry store. + +## The verbs + +| Verb | Does | +|---|---| +| `rapidpipe run create --who --why --revision --settings FILE --survey --dataset` | records a run and prints its id | +| `rapidpipe official change --group FILE --who --why` | applies a change group from a JSON list of `close` and `open` items; refuses naming the binding | +| `rapidpipe official undo --who --why` | applies a group's inverse under the same rule | +| `rapidpipe run show `, `rapidpipe job show ` | print a run with its live binding and jobs, or a job with its inputs and bindings, as JSON | +| `rapidpipe job local --run --stage --scope JSON [--inputs JSON] [--expects JSON] [--excludes JSON]` | records a job and runs it in this process, then publishes | +| `rapidpipe job run --job ` | runs an existing job's next attempt, then publishes; stops at once if the job is published | +| `rapidpipe job submit ... --job-definition ` | records a job and submits one Batch job whose command is `job run` | + +A refusal exits 1 with the binding named; a missing row or unreadable +file exits 64. The outputs root comes from `RAPIDPIPE_OUTPUTS_ROOT`, a +`file://` or `s3://` URL. + +## The thin chain + +One narrow path through every stage, the smallest working version of +each, run first on one field's two epochs with the brightest sources per +image. Each simplification keeps the shape of the full stage. + +| Stage | Reads (named in `job_inputs`) | Writes (declared outputs) | Left out until breadth | +|---|---|---|---| +| `import` | stored products for one field: L2 files, difference images, PSFs, the reference and its mosaic catalog, the Gaia copy; the cap on sources per image | copies of the image files; `exposures`, `l2files`, `diffimages`, `psfs`, `refimages`, `refimcatalogs`, `catalogs` and `sources` (the brightest N per difference image by flux over its error), with limiting magnitude, overlapping fields and sky indices computed; the source system's ids kept; provenance marked unknown | other fields and epochs; recovering provenance | +| `associate` | the epoch's source sets for the field and its eight neighbours; the predecessor object set, or none; the exclusions | `objects.parquet` (every object of the field, id carried forward or position-hashed, founding position, sufficient sums) and `matches.parquet` (this epoch's memberships) | `rubrat`, `sumrat`; field parallelism | +| `packet` | the epoch's source set; the object set to use, and through it its chain and exclusions; the window's difference and science images with their PSFs; the predecessor packet sets; the Gaia product; the reference and its mosaic catalog | `packets/--.avro` at schema 00.06, the `packets.parquet` index, `forced.parquet`, `external.parquet` | NED, NED-LVS and solar-system matches | +| reader, outside the pipeline | a packet set's Avro and its official index and forced rows | a validation report | nothing | +| `reference`, `difference` | L2 files, a reference or its members | images and their source catalogs, published by the publisher | nothing; their port to this contract comes before they run on Batch | + +### import + +The import stage takes stored products into the record. One job +imports the shared products (the reference with its mosaic catalog, the +science PSF and the Gaia copy) and one job imports each epoch, because +a source set's job binds one key. The kept sources are mapped onto the +`sources` columns with the old load stage's join, and their HEALPix and +tessellation columns are computed. A row's `filename` is the URL of the +copy under the job's prefix; every file is reached through its row. + +### associate + +`associate` replaces `crossmatch`, `statistics`, `prune` and +`maintain`. It accepts sources whose `flags` and signal-to-noise meet +the run's `[associate]` settings, then matches them in memory with the +semantics of the SQL matcher it replaces: every object within the match +radius matches, not only the nearest, so a source may join several +objects; an unmatched source founds an object whose id is the +`radec_index` of its position, and coincident founders share one +object. Exposures are taken in ascending MJD, and objects founded by +one exposure are seen by the next, not by their own. A second pass over +the eight neighbouring fields' sources adds memberships only. + +The sums are of offsets from the founding position, not of raw +degrees: at RA 268 an object spreads by about 1e-5 degrees, and a +variance from raw-degree sums would be lost to cancellation. The mean +is `ra0 + sumra / n`, and the standard deviation in RA carries one +cosine, of the mean declination. An excluded source set's members are +subtracted once; an object left with no members stays with +`nsources = 0`. + +### packet + +`packet` is a port of `dev`'s alerts package at schema 00.06, one job +per difference image. Every `flags = 0` source the named object set +holds triggers a packet; a source in several objects takes the lowest +object id, as `dev` does, and `dev`'s cap per chip is not carried. Each +packet carries: + +- `diaSource` from the source row; +- `prvDiaSources`, the object's members through the chain, less + exclusions, with `mjdobs` before the trigger and within the history + window; +- `diaObject` from the set's position and sums, `nDiaSources` counting + members up to and including the trigger; +- `prvDiaForcedSources`, `dev`'s PSF photometry at the object's founding + position on every window image that holds it, difference and science + image merged into one row per object and image, rows a predecessor + set holds copied rather than measured; +- `gaiaMatches` with proper motion propagated to the trigger, and + `refStarMatches` and `refGalaxyMatches` from the reference's mosaic + catalog; +- `cutoutDifference`, `cutoutScience` and `cutoutReference`, each 129 + pixels square, pixels off the chip filled with 0.0. + +`ssMatches`, `nedMatches` and `lvsMatches` are null: in the schema's +contract null means the match did not run, and these are deferred. The +difference image is measured with its own PSF and uncertainty map, +where `dev` measures the SFFT difference image whatever the named +flavour. + +### The reader + +`scripts/read_packets.py` reads what `packet` published and exits +non-zero on any failure. It checks that every record is valid against +the schema its index row names; that every cutout opens as FITS at the +stated size with the stated fill; that every forced value equals the +`forced_official` row with its id; that every previous detection +precedes its trigger; and that every index row decodes to its record +and names the job's recorded inputs. + +### Settings of the thin run + +The run's body is `rapidpipe/settings/foundation-thin.toml`, one table +per stage, with `dev`'s values where `dev` hard-codes them: the import +cap `max_sources` (500); `associate.match_radius` (1.528e-05 degrees), +`source_flags` (0) and `min_snr` (5); `packet.stamp_half_width` (64), +`prv_window_days` (365.25), `forced_window_days` (30), +`gaia_radius_arcsec` (1) with `gaia_max_matches` (3), and +`refcat_radius_arcsec` (5) with `refcat_max_matches` (3). Data +locations are never settings: the import's source directory is its +job's scope and the outputs root is the environment's. + +## The queries + +The official views answer eight queries without reading files: + +- the current image and source set for a dataset, exposure, SCA and + filter; +- the current object set of a field and its members; +- an object's detection and forced history within a named set; +- packet history by observation, by source and by versioned object; +- a candidate set against the official one, with unmatched detections + shown; +- the ancestors of a packet set, and the descendants of a bad + reference; +- failed and unbound jobs; +- the bindings before and after a named change group. + +The first four have precedent in the team's code; the other four are +new with this design. Beside them, the team's twelve spatial and status +shapes run on the views with the q3c and GIN indexes used: a cone over +difference images, L2 images in a cone overlapping a field, difference +images in a cone with their reference, overlap of `overlapfields`, +objects in a cone, a chip's unflagged sources, the usable reference, +the L2 time-window work list, L2 images without a difference image, +frame counts, a difference image by its product id, and L2 images with +a usable reference. All of them are `database/queries.sql`, one named +statement each. A query within a named set reads base tables through +that set's recorded chain; every other query reads the official views. + +## The procedure proof + +`tests/db/proof/thin_proof.py` runs the thirteen observations of the +procedure proof on the thin field's real data, through the verbs and +`job local` only, from an empty database. They cover the object binding +moved by the publisher alone; two jobs racing for one binding; a recipe +that is not live beside the live one; a cutover group and its undo; a +job launched before a cutover and published after it; a correction run +of one epoch's packets; a job killed before and after its commit; two +recipes from two commits at once, joined on scientific key and on +object id; two change groups on one binding; publication racing a +cutover; an undo refused because a binding moved; and a replacement +source set excluded downstream. The first passing run took about two +minutes. The same observations on synthetic rows run in CI under +`tests/db/proof/`. + +## Not decided here + +- The cost of following an object set's predecessor chain when a field + has tens of thousands of sets. If it bites, a periodic set holding the + full membership compacts the chain, a job like any other. +- Forced photometry's cost at scale. On the thin field it is about two + seconds per window image, dominated by opening the images, not by the + fits. +- Dense-field memory for `associate` and packet counts per chip. +- The cost of a large change group. The operation is linear at about + 0.14 ms per key; a set-based close and open would remove most of it. + On an `official` table never analysed, the planner picks the wrong + partial index, and the publisher analyses only the tables it loads. +- The meaning of `nDiaSources` in the 00.06 schema's own documentation, + which is `dev`'s; changing it is a schema change for the team. +- `packet.edge_pixels` is in the thin run's settings and read by no + stage, since no two-pixel edge rule exists in `dev`'s alerts code. +- The imported reference has no `refimimages` rows, since the source + system's rows do not name its members. +- How the `reference` and `difference` stages port to this contract, + with `dev`'s later changes to differencing taken in first. diff --git a/system/index.md b/system/index.md index b2d5333..f99749d 100644 --- a/system/index.md +++ b/system/index.md @@ -18,9 +18,8 @@ repository's history. The rebuild starts from this page. |---|---|---| | [`specification.md`](specification) | DRAFT | Purpose and outcome; the pipelines and their stages; the stage contract; runs, the three output states, promotion, attempts and deletion; tools; the three repositories and their boundary; releases; the manifest edges; constraints; sequencing; what is not yet decided | | [`decisions.md`](decisions) | DRAFT | The team's rulings, one dated line each with its author: Ben's rulings of 2026-09-27, his rulings of 2026-10-06 on weekly maintenance on SMDC, earlier rulings, and the rulings carried from the build pending team review | -| [`products.md`](products) | DRAFT | Product kinds and result-set kinds, logical key versus instance id, bundles, reading across runs, registration metadata with one source per field, one complete worked manifest | -| [`runs.md`](runs) | DRAFT | The run-model tables beside the kept `dev` schema; unit and attempt state machines; instances and the three custody states; promotion under one lock with before and after per key; guarded deletion as the only deleter; storage layout; identifiers | -| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; the stage declaration, one invocation form, attempts, the manifest, six exit codes, settings; local fixtures; what it replaces | +| [`foundation.md`](foundation) | DRAFT | The foundation and the thin chain: runs, jobs, recorded inputs, scientific keys, official bindings and change groups, publish-last; the decisions with their reasons; the verbs; the `import`, `associate` and `packet` stages and the reader; the eight queries and the team's spatial shapes; the procedure proof; what is not yet decided | +| [`stage-contract.md`](stage-contract) | DRAFT | The `rapidpipe` package and its dependency direction; a stage as `run(job)` and its `JobContext`; settings from the run's body; the manifest; running a job locally, again or on Batch; exit codes; the stages still on the older contract | | [`reference.md`](reference) | DRAFT | The `reference` stage as ported from `dev`'s reference pipeline: its input set and selection rule, the awaicgen coadd and SExtractor catalog, the header stamp, identity, the four registration tables, and its settings | | [`difference.md`](difference) | DRAFT | The `difference` stage as ported from `dev`: its input set, the steps it runs, its outputs, the registration fields and columns, and its settings | | [`load.md`](load) | DRAFT | The `load` stage as ported from `dev`: its inputs, the child tables, what lands in `sources`, the source-set result set, its settings and exit codes; the `psf` registration block | @@ -30,12 +29,10 @@ repository's history. The rebuild starts from this page. | [`finalize.md`](finalize) | DRAFT | The `finalize` stage as ported from `dev`'s post-processing pipeline: the chain order, the manifest it republishes as new instances, the header stamp, its settings and exit codes | | [`alerts.md`](alerts) | DRAFT | The `alerts` stage as ported from `dev`'s alert production: the input-set manifest, what it reads from the source, association and statistics sets, the container and the alert outbox, its registration, settings and exit codes | | [`prune.md`](prune) | DRAFT | The `prune` stage: the not-best merge exclusion ported from `dev`'s `pruneNotBestMerges`, its `pruned-set` result set and `prunedmerges` rows, the run model's own-run exclusion clause, its settings and exit codes | -| [`photometry.md`](photometry) | DRAFT | Not ported: forced photometry stays `dev`'s `forcedPhotometryForField.py`, and the rebuild carries no photometry stage | +| [`photometry.md`](photometry) | DRAFT | Forced photometry: the alert-driven measurement inside the `packet` stage, and `dev`'s standalone `forcedPhotometryForField.py`, not ported | | [`export.md`](export) | DRAFT | The `export` stage as ported from `dev`'s `generateSourceHATSCatalog.py`: the named source sets it reads, the CSV dump and the `hats-import` build, the `catalog-export` bundle and registration, its settings and exit codes; the light-curve catalog is not ported | -| [`releases.md`](releases) | DRAFT | The release tag and record, the `cut` command and its hook contract, migrations at release time, the launcher reading a release instead of a branch, promotion's released-image eligibility, deployed pins, and what is not yet decided | | [`tool.md`](tool) | DRAFT | The command-line tool `rapidpipe`, every subcommand and its flags, the operations mapped from the specification's Tools sentence onto subcommands, the input-set composer, the tool's own exit codes, personal submission from a workstation, and where it runs | | [`loop.md`](loop) | DRAFT | The processing-date loop: the operations-registry trigger, the venue and spec it runs from, the production run and stage chain per date, base-catalog binding across dates, promotion, its own records, and release binding | -| [`checks.md`](checks) | DRAFT | What a check is, the two candidate checks and their measurements and bounds, check policies as versioned TOML files, the promotion gate they validate, and automatic promotion's design; the check commands are on the tool page | | [`operations.md`](operations) | DRAFT | Operations proposals the team has not yet ruled on: what they assume about the mission, three shapes of the run model and the recommended named stream of batch runs, replacement scope and the chain switch, batch cadence against the latency requirement, a proposed production check policy, reference eligibility and selection, staged inputs of a production run, capacity lanes | | [`observability.md`](observability) | DRAFT | The `rapidpipe` log line and where it goes, the record of an attempt with and without CloudWatch, monitoring with `rapidpipe` alone, queue, execution and orchestration time per Batch job, profiling a stage, and today's job-duration baseline against a 30-minute target | @@ -44,9 +41,8 @@ repository's history. The rebuild starts from this page. specification decisions +foundation stage-contract -products -runs reference difference load @@ -58,10 +54,8 @@ alerts prune photometry export -releases tool loop -checks operations observability ``` diff --git a/system/load.md b/system/load.md index 5a276a3..6bf1871 100644 --- a/system/load.md +++ b/system/load.md @@ -2,6 +2,11 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + The `load` stage reads the two Photutils PSF-fit catalogs, positive and negative, made by the difference stage for one registered difference image. It joins each to its finder catalog, rejects sources fitted @@ -18,7 +23,7 @@ The stage lands on the pipeline repository's `rebuild` branch `rapidpipe/db/psfs.py`, migrations `20260923-04` to `-07`), ported from `dev`'s `pipeline/loadPSFCatIntoDBSourcesTable.py`. Every ported stage minimises differences from `dev` and designs in, off by default, -anything that could be left unused. The [products](products) page fixes +anything that could be left unused. The [foundation](foundation) page fixes the vocabulary used here. The `psf` registration block landed beside the stage and is described below. @@ -184,7 +189,7 @@ committed rows and then failed, or if another attempt still has no disposition. The earlier rows remain the run's own but are unreachable: no later stage selects an unsucceeded attempt's output. `db/sources.find_complete_source_set` enforces this by joining -`attempts`; the [runs](runs) page has the general rule. +`attempts`; the [foundation](foundation) page has the general rule. ## Settings @@ -215,7 +220,7 @@ replaced by the launcher and manifests. `register` records the field list below for the products page's `psf` kind (filter, detector, version) in `psfs`. Registration is through -`psf-import`, the [runs](runs) page's import-run pattern. The block is +`psf-import`, the [foundation](foundation) page's import-run pattern. The block is designed in and unused: `psf-import` is not built, nothing emits a `psf` entry yet, and the difference stage consumes its input `psf` entries without registering them. diff --git a/system/loop.md b/system/loop.md index 42905dd..5b60a96 100644 --- a/system/loop.md +++ b/system/loop.md @@ -2,6 +2,10 @@ **Status: DRAFT** +Parts of this page describe the run model that the [foundation](foundation) +page replaced: units, attempts, custody, promotion, checks and releases. +Where the two differ, the foundation page governs. + `rapidpipe loop run` runs regular operations from one scheduled trigger; it is not a continuously running service. The [specification](specification)'s Sequencing item 7 states the requirement. @@ -81,7 +85,7 @@ date's run records it. The run's code and the image its units execute therefore use the same tag, never a branch head. The release is the first tag cut at a rebuild head carrying the loop's code and migration, allocated by `next_tag` from the laptop using the -[releases](releases) page's recipe. +[foundation](foundation) page's recipe. ## The loop spec @@ -260,7 +264,7 @@ exclusion keeps an inferior instance out of the chain a later date reads. Each field's record stores `base_promoted`, whether the date supplying its base had been promoted at bind time. -Field discovery and base selection both obey the [products](products) +Field discovery and base selection both obey the [foundation](foundation) page's cross-run reading rule. Field discovery reads a source set only after the rule passes; an unreadable own source set fails the date. A candidate base that fails the rule (another run's scratch set, or @@ -276,7 +280,7 @@ field-level input selection. The tool page's question of which reference to use among several eligible ones remains open (below). A schedule's first date has no base. If field association sets for that -sky are already current, the [runs](runs) page's association-set +sky are already current, the [foundation](foundation) page's association-set ancestor rule refuses scheduler promotion until a chain switch to the earlier sets exists: the before instance must be reachable from the new set through `base`. A proof or new stream over the same sky @@ -289,7 +293,7 @@ When every unit in a date's run is complete, the loop resolves the check policy from the spec, then the run, then the default. It checks the run's candidates and records each result as `scheduler`. `promote_run` is called only for a team-approved policy with -`auto_promote` true, the same gate used by `run start` ([checks](checks), +`auto_promote` true, the same gate used by `run start` ([foundation](foundation), "Automatic promotion"). No shipped policy permits this. Every scheduled date so far has used `rebuild-trial@1`, which leaves @@ -303,7 +307,7 @@ which text was recorded. A candidate date still supplies the next date's base by instance; the next date's field entry `base_promoted` (Base catalog, above) records `false`. A person promotes candidates with `rapidpipe run promote`, in date -order. The [runs](runs) page's ancestor rule refuses promotion of a +order. The [foundation](foundation) page's ancestor rule refuses promotion of a later date's association set while the earlier date's run it depends on remains a candidate. @@ -320,7 +324,7 @@ complete units, attaching to running attempts and re-attempting ready units. A `failed` row is skipped by default. For a failed date whose run has a failed or cancelled unit, -`loop run --retry-failed` seeds a replacement run using the [runs](runs) +`loop run --retry-failed` seeds a replacement run using the [foundation](foundation) page's `run create --seed --only-failed` recovery. The date row points to the new run and retains the old id in `record.previous_runs`. The loop walks the date again, re-running only the seed's non-complete diff --git a/system/maintain.md b/system/maintain.md index c704882..a87bd45 100644 --- a/system/maintain.md +++ b/system/maintain.md @@ -2,6 +2,11 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + For one observation date and detector, `maintain` clusters the `sources` child table on its position index and analyzes it. It runs once, after the date's last `load` unit and before `crossmatch` reads the table, to @@ -16,7 +21,7 @@ recluster the table on every load. The implementation is on the pipeline repository's `rebuild` branch: `rapidpipe/stages/maintain.py`, `rapidpipe/db/sources.py`'s `cluster_and_analyze`, and migration `20260924-02`. The -[products](products) page fixes the vocabulary. +[foundation](foundation) page fixes the vocabulary. ## Unit diff --git a/system/observability.md b/system/observability.md index 092db63..618918f 100644 --- a/system/observability.md +++ b/system/observability.md @@ -2,6 +2,10 @@ **Status: DRAFT** +Parts of this page describe the run model that the [foundation](foundation) +page replaced: units, attempts, custody, promotion, checks and releases. +Where the two differ, the foundation page governs. + An attempt's logs and database record show what failed and where time went. Every `rapidpipe` log line has one shape and goes to stderr; stdout carries only command data. Stages also publish a log file beside diff --git a/system/operations.md b/system/operations.md index ee1a95c..890ea6e 100644 --- a/system/operations.md +++ b/system/operations.md @@ -1,10 +1,10 @@ # Operations: live processing, reprocessing and development together **Status: DRAFT, a proposal.** The team has not yet ruled on these -operations proposals. Landed behaviour is on [loop](loop), -[products](products), [runs](runs) and [checks](checks). This page changes +operations proposals. Landed behaviour was on [loop](loop) +and the run-model pages the [foundation](foundation) replaced. This page changes no code, schema or other page's rules. [Specification](specification), -[runs](runs), [products](products) and [loop](loop) remain the design +[foundation](foundation) and [loop](loop) remain the design authority; proposed changes identify the sentence they would change. ## What this page assumes about the mission @@ -14,7 +14,7 @@ Edges). The recommendation depends on the following assumptions. - **Delivery manifest.** Each delivery arrives with a manifest naming, per file, the exposure, detector, delivered version and checksum: the - shape `admit` already reads ([products](products), the l2 image). + shape `admit` already reads ([foundation](foundation), the l2 image). RAPID discovers files by listing their delivery location; it does not need a message. - **Re-delivery signalling.** A corrected file carries a higher delivered @@ -52,7 +52,7 @@ This shape is rejected. Changing the input set after creation loses the frozen provenance protected by the specification's Runs section. Whole-run promotion becomes meaningless, so every promotion is piecemeal. A release change -puts two releases in one run, against [releases](releases)' "a run never +puts two releases in one run, against the retired releases page's "a run never spans a release". `run create --seed --only-failed` seeds a replacement for all operations instead of one date. The per-date association base becomes a same-run read, skipping @@ -73,7 +73,7 @@ identity: the `processing-date-loop` registry row, its spec and its The stream discovers unadmitted deliveries instead of reading them from the spec. A date may have several batches (Closure and latency); promotion supersedes by science slot (Replacement scope); publication -follows [products](products), "Reading across runs", and its eligibility +follows [foundation](foundation), "Reading across runs", and its eligibility table. Runs, units, attempts, custody, the promotion lock, release binding, recovery, the base-catalog rule, deletion and the scratch path stay unchanged. @@ -94,7 +94,7 @@ the same mechanism as a new date. ## Replacement scope Promotion supersedes by science slot. Each kind's identity key and slot -are on [products](products), "Identity". [Runs](runs), "Rules", defines +were on the retired products page, "Identity"; its "Rules" defined slot replacement, including the rule that an `association-set` promotion replaces only an ancestor of the set it promotes. @@ -190,7 +190,7 @@ others. The proposed `rebuild-production@1` requests `auto_promote = true`; team approval is pending. A policy is immutable once landed -([checks](checks)), so its file must land in a team-approved pull +([foundation](foundation)), so its file must land in a team-approved pull request carrying `approval = "team"` and `approved_by` set to the approver's login. Until then, its content exists only in the table below, with no file to edit. No other path raises a policy to `team`. @@ -217,7 +217,7 @@ production batches under that ruling, not from `dev`. The catalog-count check stays advisory until earlier production runs provide current source sets for the same frames. It compares another -run's source set, never a reference catalog ([checks](checks)). +run's source set, never a reference catalog ([foundation](foundation)). ## Reference eligibility and selection diff --git a/system/photometry.md b/system/photometry.md index c9dfdcd..29083e6 100644 --- a/system/photometry.md +++ b/system/photometry.md @@ -1,14 +1,18 @@ -# Unported forced photometry +# Forced photometry -**Status: DRAFT.** Forced photometry is not ported; the rebuild carries -no photometry stage. +**Status: DRAFT.** Forced photometry for alerts is part of the `packet` +stage; `dev`'s standalone forced photometry for a list of positions is +not ported. -Forced photometry remains `dev`'s standalone script, -`pipeline/forcedPhotometryForField.py`. It takes a field and a CSV of -`reqid, ra, dec` sky positions. It has no `ppid` row in `dev`'s pipeline -table, and the inventory did not find where or how it is scheduled in -production. +The `packet` stage measures forced photometry at each alerted object's +founding position on the difference and science images of its window, +with `dev`'s algorithm, and publishes the rows as its packet set's +`forced.parquet`. The [foundation](foundation) page states the design: +a packet job names the window's images and its predecessor packet +sets' forced rows as inputs, and measures only what those do not hold. -`ExitCode.NOT_IMPLEMENTED` (69) remains reserved in the stage contract's -exit-code table ([stage-contract](stage-contract)). No stage in this -build returns it. The ruling is on [decisions](decisions). +`dev`'s standalone script, `pipeline/forcedPhotometryForField.py`, +takes a field and a CSV of `reqid, ra, dec` sky positions. It has no +`ppid` row in `dev`'s pipeline table, and the inventory did not find +where or how it is scheduled in production. Nothing outside packets +needs it yet, so it stays unported. diff --git a/system/products.md b/system/products.md deleted file mode 100644 index 7439b9e..0000000 --- a/system/products.md +++ /dev/null @@ -1,514 +0,0 @@ -# Products - -**Status: DRAFT** - -The [stage contract](stage-contract)'s `consumes` and `produces` -declarations use the product vocabulary defined here. Each kind has a -uniqueness rule and manifest metadata that `register` uses to record it. -The tables below map this vocabulary onto the `dev` schema. Existing -tables and columns keep their names and meanings; missing columns or -tables are added, and nothing is renamed or dropped. - -## In plain terms - -A product is something a stage makes for another stage or a consumer: -an image, catalog file, alert file, or set of database rows. Each kind -has a fixed name and required manifest fields, so registration never -needs to open the file. Making the same thing twice gives two instances -of one logical product. The instance id tells them apart; promotion -selects the one consumers see. - -## Identity - -Every instance carries three keys. The provenance key is the manifest's -`key`: for a difference image, the detector image, reference, -differencer and settings. It names the exact input instances consumed, -so reprocessings have different provenance keys even when their science -is identical. Stages write this key unchanged; it remains in -`product_instances.logical_key`. Existing readers, including a base -association set's ancestor walk and a retry's reuse lookup, need no -change. - -The identity key (column `identity`) contains every scientific -distinction: delivered facts and science choices, never instance ids. -The slot (column `slot`) is the subset a consumer selects on, "the -current X for Y". Both columns are nullable JSON. Stages write neither; -the database derives them from the provenance key through -`product_identity_fill()`. It first resolves each named producer's -identity, then derives the instance's identity and finally its slot. -The manifest is unchanged, so stage images from any earlier release -still register as before; the database fills the new columns afterward. - -In the table, `k` is the provenance key, `V(x)` is producer instance -`x`'s identity, and `P(x)` projects the slot fields from that identity. -Every slot is a subset of its kind's identity fields. Because `P(x)` -reads `x`'s identity, a collision that withholds the producer's slot -(below) does not block descendants: each row derives independently -from identities alone. A missing producer or null producer identity -leaves the row null until resolved. - -| Kind | Slot | Identity adds | -|---|---|---| -| `l2-image` | exposure, detector, from `k` | version | -| `psf` | filter, detector, from `k` | version | -| `reference-image` | field, filter, from `k` | recipe, version (the selection digest) | -| `reference-catalog` | `P(k.reference)` plus catalog type | `V(k.reference)` plus catalog type | -| `difference-image` | `P(k.l2)` plus differencer | exposure, detector, version from `V(k.l2)`, differencer, the reference's `V(k.reference)`, settings hash | -| `source-catalog` | `P(k.difference)` plus catalog type, sign | `V(k.difference)` plus catalog type, sign | -| `source-set` | `P(k.difference)` plus catalog type | `V(k.difference)` plus catalog type | -| `alert-container`, `alert-set` | `P(k.difference)` (an alert set shares its container's slot; one alert set per difference) | `V(k.difference)` plus schema version | -| `association-set` | field, from `k` | field, settings hash, the sorted identities of `k.source_sets`, plus a hash of the base's own identity (below) | -| `pruned-set` | `P(k.base)`, which is field | the base association set's identity, settings hash | -| `statistics-set` | `P(k.membership)` plus which kind of set `k.membership` names | the membership set's identity, plus that kind | -| `light-curve` | field, request id, from `k`, when both are present, else null (declared, not yet ported) | nothing beyond the slot | -| `catalog-export` | field, export type, from `k` | the selection digest, as today | - -An unknown kind resolves to null and is counted unresolved, the same as -a missing producer. - -Association sets built from equal source sets over different bases -must have different identities. Each therefore includes the SHA-256 -digest of its base identity's canonical JSON text, or null when there is no -base. This hash stays bounded however deep the chain runs, without -introducing an instance id. Pruned and statistics sets inherit the -distinction through their membership's identity. - -The fill has two stages. For every kind, slot NOT NULL implies identity -NOT NULL. The identity stage repeatedly resolves rows whose producers -already have identities, stopping when a pass resolves no more. There -is no fixed pass count, only a very large cycle guard. If it trips, -the call assigns no slots and counts every remaining row unresolved. - -Once identities settle, one further pass derives every still-null slot -from its row's identity. The collision rule applies once across all -current slots, held or newly derived. Both members of a duplicated -pair are withheld together at any chain depth, including pruned or -statistics sets built on a duplicated association pair. The fill never -rewrites an existing slot or assigns one another current instance -holds. A withheld row keeps its identity and null slot, counted as -`duplicate_current` alongside `unresolved`. Re-running the fill -resolves rows completed by later registrations and leaves filled rows -alone. - -An instance with a null slot or identity cannot be promoted until a -later registration supplies the missing producer or an operator -corrects it. A partial unique index allows at most one current instance -per kind and slot. The older (kind, provenance key) index remains under -the additive migration rule ([releases](releases)). - -A reference to another product (a reference version, source set or base -association set) always means its complete instance id, never a bare -version number. - -## File products - -| Kind | Unit | Provenance key | Format | Made by | Today's table | -|---|---|---|---|---|---| -| `l2-image` | detector-image | exposure, detector, delivered version | FITS or ASDF as delivered | `admit` | `l2files` | -| `psf` | detector-image | filter, detector, version | FITS | `psf-import` | `psfs` | -| `reference-image` | field | field, filter, reference recipe, version | FITS bundle: image, coverage map, uncertainty | `reference` | `refimages`, `refimimages`, `refimmeta` | -| `reference-catalog` | field | reference instance, catalog type | table, format per catalog type | `reference` | `refimcatalogs` | -| `difference-image` | detector-image | l2 instance, reference instance, differencer, settings hash | FITS bundle, roles declared per differencer | `difference` | `diffimages`, `diffimmeta` | -| `source-catalog` | detector-image | difference instance, catalog type, sign | table | `difference` | none until `load` | -| `alert-container` | detector-image | difference instance, alert schema version | Avro object container plus JSON summary | `alerts` | the outbox | -| `light-curve` | field | field, object set instance, request id | Parquet | none (not ported) | none; exported | -| `catalog-export` | field | field, export type, selection digest of the named source sets | HATS | `export` | none; exported | - -A bundle is one product with several files. Its manifest entry names -the primary member and lists every member's role, size and SHA-256. -Member paths resolve against the attempt's output location. - -Each differencer declares its bundle roles. `difference` and -`uncertainty` are always present; `significance` is present where the -differencer produces one: ZOGY does, SFFT does not. The optional roles are `psf` -(the difference PSF) for any differencer and `kernel` (the -matching-kernel solution) for SFFT. A declared role that is not -delivered fails registration. The catalog stage's detection member is -a per-differencer setting recorded with the instance: significance for -ZOGY, difference for SFFT. - -The difference-stage port minimises differences to the `dev` branch; -improvements come later. Mechanisms that can be designed in but left -unused are included, off by default. - -The reference recipe names the pipeline and settings, giving each way -of building a reference for a field its own version namespace. -Constituent inputs are `l2-image` instance ids, each fixing exposure, -detector and delivered version. The logical key's `version` is the -full, hex-encoded SHA-256 digest of the sorted constituent instance ids -plus the resolved settings hash. Rebuilding the same selection makes -another instance of one logical product; a different selection makes -a new logical product. - -`refimages.version` remains the legacy per-(field, fid, ppid) counter. -Like every other legacy version column below, it is allocated at -registration, globally across runs rather than within one. It is -separate from the selection digest (see the reference-image field list -and [reference](reference)). - -`finalize` reads an immutable instance and writes a new instance of the -same kind in its own attempt location. Its manifest records the input -instance, output revision, and finalized file sizes and checksums. -`register` consumes the selected finalized instance. The -[finalize](finalize) page fixes the chain, stamped header and -source-catalog republication. - -Alert names are not a file product. `alerts` writes one outbox record -per alert (name, candidate id, first-seen time, position) alongside the -container's byte range; `register` records the container. The outbox -locator is a block offset and length plus the record's ordinal -position, not a per-alert byte range. `alerts` registers both the -container and its alert-set itself; `register` validates and replays -them. The locator and registration are fixed on the [alerts](alerts) -page. - -## Database result sets - -Catalog stages produce rows, not files. A result set is the named, -versioned group of rows one stage attempt wrote. Its instance id identifies the next -stage's input. Completion is recorded on the result set, so an empty -set can be complete. - -| Kind | Unit | Provenance key | Rows in | Made by | -|---|---|---|---|---| -| `source-set` | detector-image | difference instance, catalog type | `sources` | `load` | -| `association-set` | field | field, frozen input selection, crossmatch settings hash | `merges`, `astroobjects` | `crossmatch` | -| `pruned-set` | field | base association set, pruning settings hash | membership table | `prune` | -| `statistics-set` | field | membership set (association or pruned) | `astroobjectsmeta` | `statistics` | - -An association set's frozen input selection records the exact source-set -instance ids read, catalog versions, any seed object set, and the -epoch-selection rule that produced the list. Field membership is -complete when every named source set is complete. - -A pruned set excludes an explicit list of (object, source) pairs from -its base association set without mutating the base. A statistics set -names the exact membership it describes; its input determines whether -it is before or after pruning, with no separate flag. Delivered -statistics describe the association set, following `dev`'s order: -`dev` runs crossmatch, then statistics, then prune. The `statistics-set` -row accepts either membership kind, but pruned-set input is designed in -and left unused. - -Every row carries its run id, writing attempt id and result-set id. -Keys are set-scoped: statistics for one object in two sets are two -rows, `(statistics_set, object)`. Stages read result sets by id, never -"whatever is current". Promotion atomically selects compatible file and -database-result instances and records the previous selection for -reversal. Scratch-run deletion removes only result sets with no active -attempt or retained output depending on them. - -### Reading across runs - -Crossmatch needs earlier epochs' sources from earlier runs to accumulate -a catalog across processing dates. Current result sets are readable by any run -as frozen inputs, by instance id; scratch result sets are readable only -within their own run. This amends the specification's sharing rule. -Cross-run reads of selected candidates, allowed by the table below, -remain pending team review -({ref}`sharing rule `). - -A stage may read an input it cannot publish from. The table separates -reading from promotion for all product instances, both files and result -sets. - -| Input's state | A stage of another run may read it | A product built from it may be promoted | -|---|---|---| -| Scratch, own run | yes | no, scratch never leaves scratch | -| Scratch, another run | no, exit 65 | no | -| Candidate, from an unselected attempt | no, exit 65 | no | -| Candidate, selected | yes | yes, once it is itself promoted, earlier or in the same request, where it passes the gate on its own ([checks](checks) page has the gate) | -| Current | yes | yes | -| Superseded (current before, candidate now) | yes, as any candidate from a selected attempt | yes | - -Every read requires completeness and retention. A same-run read needs -nothing more, so a run may name its own orphaned set. A cross-run read -also requires a selected producing attempt for every instance in the -dependency chain. Checked and unchecked candidates are equally -readable: a failed or missing check leaves a candidate, even when it -refuses promotion. Promotion requires complete, retained dependencies -that are current, superseded, or members of the same promotion request, -throughout the chain, not just the directly named instance. The -[checks](checks) page describes the walk. - -Each instance has one of seven states, computed once by -`rapidpipe.runs.eligibility.instance_state`: `deleted`, `incomplete`, -`scratch` and `unselected` rule it out first, using only custody, -completeness and selection facts. The remaining states are `current`, -`superseded`, and `candidate`. The promotion walk and `run show` use -this state without re-deriving it. The [checks](checks) page defines the -states; the [tool](tool) page gives `run show`'s format. - -`register_manifest` applies the read rule to every dependency edge, -refusing foreign file products on the same terms as foreign result -sets. Every reader uses -`rapidpipe.db.objects.assert_readable_instance`. Its result-set wrapper, -`assert_readable_result_set`, is called by `source_set_table`, -`association_chain`, and set resolution in `statistics`, `prune`, -`alerts` and `export`. This defines which foreign instances the sharing -rule permits and keeps a still-running scratch attempt's half-written -outputs, including files, out of production inputs. - -The stage guard, `rapidpipe.runs.readguard.assert_inputs_readable`, uses -the same rule. After parsing the input manifest and before fetching any -other object, `run_stage` calls it over every named instance. Direct -invocation, the local launcher and Batch all reach `run_stage`, so none -of the three bypasses the guard. - -The rule refuses ids that name no product instance; the guard handles -those separately. An unregistered entry with no members (a -result-set-style entry) is readable because it names nothing to check. -For a file-product entry, each member's path and SHA-256 must match -either no registered instance or at least one readable instance. A -member matching only unreadable instances refuses the entire entry, -regardless of its other members. A fresh id therefore cannot stand in -for another run's scratch files. - -The guard requires a database connection to check named inputs. A -stage without one, including a dry run, refuses execution. For S3 -inputs, only the manifest is fetched before the guard runs. Afterward, -only its named member files are fetched, one at a time; the rest of the -prefix is never fetched. The [tool](tool) page gives the exit codes. - -## Registration metadata - -Each target column has one source: a manifest value, an immutable -parent-instance lookup, a deterministic derivation, or a database -allocation or default. The manifest carries the stage's measurements -and omits anything registration can derive (spatial indexes from a -position) or allocate (row ids, current flags). An unavailable measurement -is never replaced with zero, and a missing required value fails validation. - -Checksums use SHA-256, stored with the algorithm named, except for the -legacy MD5 columns below. External identifiers, such as the -observatory's exposure id, are stored as delivered and mapped to -internal ids at admission. - -Field lists and their columns are fixed one kind at a time; the -[runs](runs) page describes how the run model attaches to existing -tables. Two lists are fixed so far: the difference image, the first -kind the rebuild registers, and the l2 image, because `admit` is the -first stage built and `register` needs its list. Remaining kinds follow -with their stages. - -Three rules preserve the `dev` tables' meaning across all kinds: - -- Legacy checksum columns keep MD5. `l2files.checksum`, - `refimages.checksum` and `diffimages.checksum` are 32-character MD5 - columns. Stages compute the primary member's MD5 alongside its SHA-256 - and carry it as `md5` in the registration block. SHA-256 goes to - `product_members`, so each name describes what it stores. Dropping - MD5 would require relaxing `l2files.checksum`'s NOT NULL constraint. -- Legacy version columns follow the team's procedures: the next number - for the table's logical pair, except for delivered versions (the l2 - image). When registration retains a `dev` stored function unchanged, - as `refimages` does through `addRefImage`, allocation uses `dev`'s - `coalesce(max(version), - 0) + 1` for the pair across the whole table and every run. The row - records the run, but the counter is not run-scoped. Concurrent - allocations for the same pair are serialised by a transaction-level advisory - lock, `pg_advisory_xact_lock(hashtext('refimages:::'))` - for `refimages`, taken before the allocation and released - automatically at the transaction's end. -- Legacy current flags are never set at registration. Every row a run - writes has `vbest` 0; custody lives on the instance row. Promotion - maintains `vbest` on the `dev` tables alongside `current_selection` - for existing queries. Moving consumers to `current_selection` is a - later improvement. - -### Difference image - -`difference` makes the image; `register` records it. - -| Field | Source | Column | -|---|---|---| -| l2 instance | manifest identity | `diffimages.rid`, with `expid` and `sca` copied from that `l2files` row | -| reference instance | manifest identity | `diffimages.rfid`: that instance's `refimages` row, through the `instance` column added with the `difference` stage; for a reference registered by `dev`, which has no instance, the legacy rfid the registration block carries as `reference_rfid` | -| differencer | manifest identity | `diffimages.ppid`: the `pipelines` row for the differencer; the name-to-row mapping is fixed with the `difference` stage. When the stage runs both ZOGY and SFFT, both register, each its own `difference-image` instance with its own `diffimages` row and `ppid` (ZOGY 15, SFFT 16); `[sfft] register_sfft` turns SFFT's registration off. Which instance consumers read is a promotion choice under the planned slot supersession, not a code default. `dev`'s own stored sources for the control exposure it processed are its SFFT difference's, not ZOGY's, so the rebuild does not copy `dev`'s choice here; it registers both. The naive subtraction is an optional diagnostic file, never a registered instance. | -| settings hash | manifest identity | the instance's provenance key only; no legacy column | -| field, filter, observation time | lookup on the l2 instance | `field`, `fid`, `jd` (from that row's `mjdobs`), on `diffimages` and `diffimmeta` | -| image centre and four corners (RA, Dec) | manifest, from the difference WCS | `ra0`, `dec0` to `ra4`, `dec4` | -| reference info bits | manifest, `infobits_reference` | `infobitsref`: the reference instance's info bits | -| catalog-outcome mask | manifest, `catalog_outcome_bits`, set per job when no Photutils catalog was produced | `infobitssci`, its `dev` meaning kept: a six-bit mask, one bit per differencer and sign (see below). The manifest also carries `infobits_science`, the l2 image's quality bits; only the mask is registered. | -| source counts per catalog type and sign | manifest | `diffimmeta.source_counts`, all of them as the manifest carries them; `diffimmeta.nsexcatsources` holds the SExtractor positive count for the team's existing queries. Both catalog families, SExtractor and Photutils, are retained in the rebuild. | -| registration residuals: x and y RMS and median | manifest | `dxmedianfin`, `dymedianfin`: the measured offsets. `dxrmsfin`, `dyrmsfin`: the measured astrometric residual RMS from gain matching, not the value ZOGY itself is fed; ZOGY's own astrometric input (`[zogy] astrometric_sigma`, a named stage setting) stays 0.0 as `dev`'s override. | -| reference scale factor | manifest | `scalefacref` | -| spatial indexes | derived from the centre at registration | `hp6`, `hp9` on both tables | -| file path | manifest primary member, resolved against the attempt's output location | `filename` | -| checksum | manifest registration block, `md5` | `checksum` | -| version | allocation: the next number for (`rid`, `ppid`) within the run | `version` | -| software version | allocation from the run's code revision | `svid`: one `swversions` row per code revision, made on first use | -| run, attempt, instance ids | enclosing manifest and allocation | `run`, `attempt`, `instance` on both tables | -| current flag, status | allocation; never current at registration | `vbest` 0, `status` 0 | -| written by later stages | not registration | `avid`, `archivestatus`, `nalertpackets` | - -`infobitssci`'s six bits, kept from `dev`: - -- bit 0: ZOGY positive -- bit 1: ZOGY negative -- bit 2: SFFT positive -- bit 3: SFFT negative -- bit 4: naive positive -- bit 5: naive negative - -### Reference image - -`reference` makes the image; `register` records it. - -| Field | Source | Column | -|---|---|---| -| constituent l2-image instances | manifest, `registration.constituents` | `refimimages`: one `(rfid, rid)` row per constituent whose `l2files.instance` matches; a constituent without a matching `l2files` row fails registration | -| filter | manifest identity | `refimages.fid`: lookup in `filters` by name | -| field | manifest, `registration.field` | `refimages.field` | -| recipe | manifest identity, fixed `awaicgen` | `refimages.ppid`: 12, the `pipelines` row `dev` used | -| selection digest (the key's `version`) | manifest identity, the instance's provenance key only | no legacy column; `refimages.version` is the separate legacy counter, allocated globally per `(field, fid, ppid)` under an advisory lock, not scoped to the run (above) | -| mosaic centre (RA, Dec) | manifest, `registration.ra_center`, `dec_center` | derived to `hp6`, `hp9` on `refimages` and `refimmeta` | -| frame count | manifest, `registration.nframes` | `refimmeta.nframes` | -| observation time range | manifest, `registration.mjdobs_min`, `mjdobs_max` | `refimmeta.mjdobsmin`, `mjdobsmax` | -| JD range, total exposure time, zero point | manifest, `registration.jd_start`, `jd_end`, `total_exptime`, `zero_point` | carried into the header stamp (`JDSTART`, `JDEND`, `TOTEXPTM`, `MAGZP`), not registered: no matching `refimmeta` or `refimages` column | -| coverage and pixel-uncertainty measures | manifest, `registration.cov5percent`, `medncov`, `medpixunc` | `refimmeta`, same names | -| bad-pixel count | manifest, `registration.npixnan` | `refimmeta.npixnan` | -| clipped image statistics | manifest, `registration.clmean`, `clstddev`, `clnoutliers`, `gmedian`, `datascale`, `gmin`, `gmax` | `refimmeta`, same names | -| catalog FWHM | manifest, `registration.fwhmmedpix`, `fwhmminpix`, `fwhmmaxpix` | `refimmeta`, same names | -| source counts | manifest, `registration.nsexcatsources`, `npucatsources` | `refimmeta.nsxcatsources` (`dev`'s column spelling; the block field is `nsexcatsources`), `npucatsources`: null when `[psfcat]` is off, needing migration `20260924-09` to drop that column's `NOT NULL` | -| settings hash | manifest identity | the instance's provenance key only; no legacy column | -| infobits | manifest, `registration.infobits` | `refimages.infobits`: 0, `dev`'s TODO; no code sets bits | -| checksum | manifest registration block, `md5` | `refimages.checksum` | -| file path | manifest primary member | `refimages.filename` | -| software version | allocation from the run's code revision | `refimages.svid`: one `swversions` row per code revision, made on first use, as `difference` allocates | -| current flag, status | allocation; never current at registration | `refimages.vbest` 0, `status`: the block's `status`, 1 | -| run, attempt, instance ids | enclosing manifest and allocation | `run`, `attempt`, `instance` on `refimages`, the columns migration `20260923-02-refimages-instance.sql` added; `attempt` is the *producing* attempt (the `reference` attempt named in the manifest), not the attempt running `register` -- `psfs`, `diffimages` and `l2files` record the registering attempt instead, so `refimages` is the one exception | - -### Reference catalog - -`reference` makes the catalog; `register` records it. - -| Field | Source | Column | -|---|---|---| -| catalog type | manifest identity, `key.catalog_type` | `refimcatalogs.cattype`: `sextractor` maps to 1, `psf` to 2 | -| reference instance | manifest identity, `key.reference` | `refimcatalogs.rfid`, resolved from the reference instance: registered earlier in the same manifest, or already present in `refimages` | -| field, filter | lookup on the `refimages` row | `field`, `hp6`, `hp9`, `fid`, copied from that row | -| checksum | manifest registration block, `md5` | `refimcatalogs.checksum` | -| file path | manifest primary member | `refimcatalogs.filename` | -| status | manifest registration block | `status`: 1 | -| differencer/recipe id | allocation | `ppid`: 12, the same row the reference image uses | -| source count | manifest registration block, `source_count` | carried, not registered: `registerRefImCatalog` takes no such column, the same treatment the difference stage's `source-catalog` block gets until `load` | -| run, attempt, instance ids | enclosing manifest and allocation | as `refimages` | - -### L2 image - -`admit` makes the image; `register` records it. Admission alone reads -the delivered header, so its manifest carries every header value the -tables need. `admit` has no upstream stage. Whoever stages the delivered -file writes its input delivery manifest: stage `delivery`, one -`l2-image` entry of format version `delivered`, a key naming the -exposure, detector and delivered version, and the delivered file as -its member. - -`admit` verifies the delivered bytes, copies the file into its attempt's -output location, and publishes a new instance. The registration block -keeps the delivery's instance id and source. A delivery is not a -registered product, so it is not a dependency. - -| Field | Source | Column | -|---|---|---| -| observatory exposure id | manifest identity, stored as delivered | `exposures`: an external-id column added with the `admit` stage; `l2files.expid` stays the internal id | -| detector | manifest identity | `sca` on `l2files` and `l2filemeta` | -| delivered version | manifest identity | `version`, stored as delivered; the legacy procedure allocated it per (`expid`, `sca`) | -| exposure record | lookup, or allocation on first admission of the exposure | `expid`; the `exposures` row carries the header's date, filter, exposure time and MJD | -| filter | manifest, from the header | `fid` on both tables: lookup in `filters` | -| observation time, exposure time | manifest, from the header | `dateobs`, `mjdobs` (both tables), `exptime` | -| info bits | manifest | `infobits` | -| verification | manifest: 1 when the stage verified the delivered DATASUM and CHECKSUM keywords, else 0 | `status` | -| WCS as delivered: reference point, pixel matrix, axis types and units, SIP coefficients, equinox | manifest, from the header | `crval1` to `cunit2`, `a_order` and `a_*`, `b_order` and `b_*`, `equinox` | -| target position | manifest, from the header | `ra`, `dec` | -| orientation and photometric calibration | manifest, from the header | `paobsy`, `pafpa`, `zptmag`, `skymean` | -| image centre and four corners (RA, Dec) | manifest, from the WCS | `l2filemeta.ra0`, `dec0` to `ra4`, `dec4` | -| spatial indexes | derived from the centre at registration | `field`, `hp6`, `hp9` on both tables; `l2filemeta.x`, `y`, `z`; `l2files.overlapfields` from the corners | -| file path | manifest primary member, resolved against the attempt's output location | `filename` | -| checksum | manifest registration block, `md5` | `checksum` | -| run, attempt, instance ids | enclosing manifest and allocation | `run`, `attempt`, `instance` on both tables, added with the `admit` stage | -| current flag | allocation; never current at registration | `vbest` 0 | - -## A complete manifest - -```json -{ - "schema_version": "1", - "run": "r-2026-09-21-0007", - "unit": {"kind": "detector-image", "id": "e20260821001234/SCA07"}, - "stage": "difference", - "attempt": "a-01J8Y6QZ3M", - "execution_record": "exec/a-01J8Y6QZ3M.json", - "inputs": { - "manifest": "s3:///manifest.json", - "products": { - "l2-image": "pi-l2-0000123456", - "reference-image": "pi-ref-0000004321", - "psf": "pi-psf-0000000789" - }, - "result_sets": [] - }, - "outputs": [ - { - "kind": "difference-image", - "format_version": "1", - "instance": "pi-diff-0000998877", - "key": { - "l2": "pi-l2-0000123456", - "reference": "pi-ref-0000004321", - "differencer": "zogy", - "settings_hash": "sha256:4f2a9c0e1b7d6a5f3e2c1d0b9a8f7e6d5c4b3a291807f6e5d4c3b2a1f0e9d8c7" - }, - "primary": "diff/e20260821001234_SCA07_zogy.fits", - "members": [ - {"role": "difference", "path": "diff/e20260821001234_SCA07_zogy.fits", "bytes": 201326592, "sha256": "sha256:9a1f0e2d3c4b5a69788796a5b4c3d2e1f00112233445566778899aabbccddeef"}, - {"role": "uncertainty", "path": "diff/e20260821001234_SCA07_zogy_unc.fits", "bytes": 201326592, "sha256": "sha256:0b2e1f3d4c5a6b7988a7b6c5d4e3f2019900aabbccddeeff0011223344556677"}, - {"role": "significance", "path": "diff/e20260821001234_SCA07_zogy_scorr.fits", "bytes": 201326592, "sha256": "sha256:1c3f2e4d5b6a7c8a99b8c7d6e5f4a3b20aa11bbccddeeff00112233445566778"} - ], - "registration": { - "centre": {"ra": 269.4521, "dec": -28.7710}, - "corners": [[269.39, -28.83], [269.51, -28.83], [269.51, -28.71], [269.39, -28.71]], - "catalog_outcome_bits": 0, - "infobits_science": 0, - "infobits_reference": 0, - "source_counts": {"sextractor": {"positive": 412, "negative": 388}, "photutils": {"positive": 405, "negative": 391}}, - "registration_residual": {"x_rms": 0.0, "y_rms": 0.0, "x_median": 0.004, "y_median": -0.002}, - "reference_scale_factor": 0.998, - "detection_role": "significance", - "reference_rfid": null, - "md5": "9e107d9d372bb6826bd81d3542a419d6" - } - }, - { - "kind": "source-catalog", - "format_version": "1", - "instance": "pi-cat-0000998878", - "key": {"difference": "pi-diff-0000998877", "catalog_type": "sextractor", "sign": "positive"}, - "primary": "cat/e20260821001234_SCA07_zogy_pos.sexcat", - "members": [{"role": "catalog", "path": "cat/e20260821001234_SCA07_zogy_pos.sexcat", "bytes": 88214, "sha256": "sha256:2d4a3f5e6c7b8d9aaac9d8e7f6a5b4c31bb22ccddeeff0011223344556677889"}], - "registration": {"source_count": 412} - } - ] -} -``` - -`register` validates each entry against its kind's schema and enclosing -manifest, then resolves upstream instance ids. It writes rows from -manifest fields, execution provenance, documented lookups and defaults, -without reading product files. - -## Not decided here - -- Registration field lists for source catalog and `light-curve`, each - fixed with its stage. `light-curve` has no stage in this build; - [photometry](photometry) records its state pending the real port. - `catalog-export`'s field list is fixed with the `export` stage on the - [export](export) page. The [load](load) page holds the source set's - rows and result-set record and the `psf` block; [alerts](alerts) holds - the `alert-container` registration block and `alert-set` result set. -- Storage layout beneath the run: the path scheme under the attempt's - output location. -- Closed by the Identity section above: a derived product's science - identity, and the slot it supersedes by, are now derived for every - kind, not only `l2-image`. diff --git a/system/prune.md b/system/prune.md index 513a1e9..bc70e03 100644 --- a/system/prune.md +++ b/system/prune.md @@ -2,11 +2,16 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + For one field, `prune` excludes association pairs whose sources came from difference images that are not best for their positions. It records the excluded pairs in a new table, `prunedmerges`. The resulting `pruned-set` is its base association set minus those pairs; the base is -never mutated, and a stage never drops a table. The [products](products) +never mutated, and a stage never drops a table. The [foundation](foundation) page fixes this vocabulary and rule: "A pruned set is its base association set minus an explicit list of excluded pairs ... the base is never mutated". @@ -51,7 +56,7 @@ excludes nothing for a production run's own, not-yet-promoted images. A later `prune` attempt excludes pairs whose sources' difference images have since been superseded (reassigned to another run, still `vbest = 0`), as `dev`'s `vbest = 0` does once a newer image is promoted current. -Promotion maintains `vbest` as a current-membership flag ([runs](runs) +Promotion maintains `vbest` as a current-membership flag ([foundation](foundation) page), so both clauses are exercised now: the own-run clause keeps this run's images, and the `vbest` clause excludes a pair once another run's promotion supersedes its difference image. @@ -127,7 +132,7 @@ retained pruned set already written in this run with the same key (same base and settings hash) and writes nothing. Reuse requires the producing attempt to be the calling attempt or to have disposition `succeeded` (`db/objects.find_complete_result_set`, joining `attempts`; -the [runs](runs) page has the general rule). +the [foundation](foundation) page has the general rule). A set left by an attempt that committed rows and then failed, or by another attempt still without a disposition, is not reused. A retry diff --git a/system/reference.md b/system/reference.md index 63b1a45..1c35938 100644 --- a/system/reference.md +++ b/system/reference.md @@ -2,6 +2,10 @@ **Status: DRAFT** +This stage still runs on the older contract ([stage-contract](stage-contract), +"Stages on the older contract"). Where this page describes registration, +custody, runs or promotion, the [foundation](foundation) page governs. + The `reference` stage coadds already-admitted science frames for one field and filter with `awaicgen`, catalogs the coadd with SExtractor, and stamps bookkeeping keywords on the result. It ports `dev`'s chain @@ -19,7 +23,7 @@ repository's `rebuild` branch as `rapidpipe/stages/reference.py`, `rapidpipe/science/reference/`, `rapidpipe/settings/reference.toml` and the packaged `cdf/rapidSex*RefImage*` files. Every ported stage minimises differences from `dev` and designs in, off by default, -anything that could be left unused. The [products](products) page fixes +anything that could be left unused. The [foundation](foundation) page fixes the vocabulary. ## Inputs @@ -131,7 +135,7 @@ new product. Truncating the digest offers no benefit and adds collision risk, so the full digest is kept. `refimages.version` is a separate legacy per-(`field`, `fid`, `ppid`) -counter, allocated at registration as [products](products) describes +counter, allocated at registration as [foundation](foundation) describes for every legacy version column. `dev`'s unchanged `addRefImage` allocates `coalesce(max(version), 0) + 1` over the whole `refimages` table for that triple, across all runs. The row records `run`, @@ -205,7 +209,7 @@ covers `refimages`; `20260924-09` drops the baseline schema's `NOT NULL` constraint on `refimmeta.npucatsources` before a null can be written. The three satellite tables have no run column and are reached only through their owning `refimages` row's `rfid`. That relationship alone -does not provide deletion handling. The [runs](runs) page's deletion +does not provide deletion handling. The [foundation](foundation) page's deletion section fixes cleanup of a scratch run's own reference, amending its cleanup-set sentence. Any needed stored function missing from the trial database is ported inline into the register code, as `sources.py` diff --git a/system/releases.md b/system/releases.md deleted file mode 100644 index ba17449..0000000 --- a/system/releases.md +++ /dev/null @@ -1,163 +0,0 @@ -# Releases - -**Status: DRAFT** - -Operations run a release while `main` moves every day. A release is a -commit that has been tagged, migrated, built into an image and deployed, -with a database row recording each step. Cutting a release performs -those steps in order and records success only when each step has the -required evidence. The [specification](specification)'s Releases -section states the requirement; this page defines the tag, record and -cut, and what the launcher and promotion require of them. - -## The command - -`rapidpipe release cut|show|list|verify`, also runnable as -`python -m rapidpipe.release`, and wrapped unchanged by the future -`rapidctl`. The full flag surface for `cut`, `show`, `list` and `verify` -is on the [tool](tool) page. The Python interface is -`rapidpipe.release.core`: `next_tag`, `cut`, `show` and `verify`, and -the `Release` dataclass that represents the release record. - -## The tag - -A release tag is annotated, on the `rebuild` branch, and named -`rebuild-v0.`. Here `n` is one more than the highest tag already -pushed: remote tags are authoritative. The scheme becomes `v1.` -once the rebuild replaces `main`, as sequenced in the specification. - -The tag message is the release's initial manifest: the tag, its source -revision, the tagged tree's schema version, and who cut it and when. -Once written, the tag is never amended, moved or deleted. Later -evidence, including images built and consumers deployed, goes in the -release record. The image registry tag equals the git tag. Registry -tag immutability makes a second build under the same tag fail outright, -meeting the specification's requirement without a separate check in the -cut. - -## The record - -Two tables carry a release's state. `releases` holds one row per tag: -source revision, schema version, image digest, image reference, who cut -it and when, and when it completed. Its state advances through -`migrated`, `built`, `deployed` and `complete`. -`release_deployments` holds one row per consumer the release reaches: -the release, consumer, deployed job definition name and revision, and -who deployed it and when. - -An attempt's execution record carries the release identity its job -definition was deployed under, read from the deploy-time environment -variable `RAPID_RELEASE_IDENTITY`. A job definition never deployed -under a release reports `unreleased`; reconcile stores whichever value -it finds. - -## The order of a cut - -`cut` runs six steps in order: tag, migrate, record, build, deploy, -pins. The systems repository supplies executables, called hooks, for -the account-specific steps: migrate, build, deploy and pins. `cut` -invokes them in order with the release's tag, source revision and schema -version in their environment. All account-specific details live in the -hooks; the pipeline repository names no account, host or bucket. - -Each hook reports its result as one JSON object on its last line of -output, and `cut` reads only that line: - -| Hook | Reports | -|---|---| -| migrate | the schema version reached and the migration files applied | -| build | the image digest and its registry reference | -| deploy | each consumer's job definition revision | -| pins | the number of rows written | - -Hooks are idempotent. `--resume TAG` re-enters a cut at its recorded -state without re-tagging. `cut` never retries a hook: a failure leaves -the record at its last completed step for a person to fix and resume. -`--skip HOOK` omits one step whose account side was already done by -hand. `--dry-run` prints the planned order and each hook's command -without running them or writing the record. - -The final pins step appends one row per rebuild consumer to the -operations pin table. It reads each consumer's live job definition and -records the release identity beside it, so the database states which -release is deployed. The existing daily pin sweep over the legacy -pipeline's consumer set is unchanged; the cut's pin step covers the -rebuild's consumers alongside it. - -CI on `rebuild` gates the source before tagging. The cut stops at -deploy and pins. Selftests run afterwards on Batch under the newly -deployed revision as evidence that the image behaves; the cut does not -wait for them. - -## Migrations at release time - -A release's schema version is the greatest migration filename in its -tagged tree. The migrate hook applies that tree's migrations to the -release's database target. `cut` refuses to record the release until -every file is applied with the checksum the migration applier recorded -for it. Migration always precedes image deployment, so a job definition -revision never runs code ahead of its expected schema. - -Migrations must stay additive (new tables and nullable columns; no drop -or rename while an earlier release's runs are open) and compatible with -the readers and writers of every release whose runs remain open. This -is a constraint of the migration rule. Together with the fixed release -binding below, it makes cutting a release while other runs remain open -safe. - -## The launcher reads the release - -A run's release is fixed at creation by `run create --release`. It -copies the source revision and image digest from the release record, -without reading git or the environment. Every unit submits to the job -definition revision that the release's `release_deployments` row -records for the run's kind, verified `ACTIVE` before submission. There -is no fallback to the latest revision. - -A later cut repoints the job definition to a new revision, so operations -must keep earlier revisions alive for runs still using them. -`SkipDeregisterOnUpdate` keeps a revision `ACTIVE` past its own -release. Retention of superseded revisions belongs to the systems -repository. Every attempt reads the same release's recorded revision: -a run never picks up a later cut mid-flight, and its execution records -and `schema_version` all carry the release it was created under. - -A run started without a release submits by the unversioned job -definition name and may pick up a revision a later deploy repoints it -to. This carries the specification's "scratch runs may use any commit" -through to the job definition. - -The processing-date loop's binding to a release is resolved on the -[loop](loop) page: the loop's spec names the release, and the scheduled -operation checks that tag out before running each date. - -## Promotion eligibility - -Promotion checks executed provenance. Every deliverable's selected -producing attempt must have an execution record whose image digest -matches a complete release's digest. Its recorded release, when present, -must be that release's tag; a tag on a commit alone does not suffice. -Failure refuses promotion unless the operator passes the explicit -unreleased exception, which is then recorded in the promotion's request -context. This closes the trial exception recorded earlier on the -[runs](runs) page: a missing released-image check refuses promotion -unless waived. - -## Concurrent cuts are serialised - -Before any fetch or tag, `cut` refuses to start while any `releases` -row has a state other than `complete`, unless `--resume` names that -row. The refusal names the tag and state and exits 1 (the -[tool](tool) page's table). To continue a cut that failed partway, use -`--resume` for the row already in flight. - -The row is written only after the tag is pushed. Two cuts started at -the same instant can therefore both pass the check before either has a -row. The rule serialises cuts once a record exists; it is not a lock. -Closing that window remains open. - -## Not decided here - -- Signing of tags or images. -- The `v1` cutover and whether the production database becomes a - release target. diff --git a/system/runs.md b/system/runs.md deleted file mode 100644 index 49753ca..0000000 --- a/system/runs.md +++ /dev/null @@ -1,623 +0,0 @@ -# Runs, attempts and custody - -**Status: DRAFT** - -Runs, units of work, attempts, product instances, result sets and the -three custody states have database records and storage locations. This -page defines them, alongside the specification's Runs section, the -[stage contract](stage-contract) and the [products](products) page. - -## In plain terms - -A run records who started a pass over which inputs, with which code -and settings, and whether the outputs belong to a person or the -project. Each piece of work is a unit; each try is an attempt with its -own output folder. Every product a finished attempt makes, whether a -file or database rows, has an instance row. Its custody state says -whose it is and whether consumers see it. - -Promotion changes those rows under one lock and records the changes -so they can be undone. Files never move. Scratch-run deletion is one -guarded operation; nothing else deletes run data. - -## Tables - -| Table | One row per | Key columns | -|---|---|---| -| `runs` | run | id, kind (`scratch`, `production`), owner login, created, state (`open`, `finished`, `deleting`, `deleted`), purpose, selected stages, code revision, image digest, release, schema version, settings overlay ref, input selection ref, lane, resource profile, database target, max attempts per unit, auto-promote flag, check-policy ref, seed run, expires at, pinned, deleted at | -| `units` | piece of work in a run | run, stage, unit kind, unit id, state (`pending`, `ready`, `running`, `complete`, `failed`, `cancelled`), selected attempt, cancel reason, seeded-from unit | -| `unit_inputs` | frozen input binding | unit, producer instance | -| `attempts` | execution of a unit | id, run, stage, unit, started, ended, exit code, disposition (null while queued or running; `succeeded`, `failed`, `transient`, `killed`, `lost`), output location, inputs location, settings location, execution record, scheduler job id | -| `execution_records` | attempt | attempt, source revision, working-copy patch, image digest, release, schema version, resolved settings, settings hash, scheduler metadata; contents stored, not referenced into the run prefix | -| `product_instances` | product an attempt made, file or result set | instance id, kind, provenance key, slot, identity (the latter two nullable, database-derived; [products](products) page), run, producing stage, producing attempt, registering attempt, custody (`scratch`, `candidate`, `current`), format version, primary location, manifest ref, published at, deletion state | -| `product_members` | file in a bundle | instance, role, path, bytes, sha256 | -| `result_sets` | one-to-one extension of an instance that is rows | instance, complete flag, row count | -| `promotions` | promotion action | id, who, when, reason, check-policy version, check result ids, request context | -| `promotion_changes` | affected slot, or provenance key for a legacy row, in a promotion | promotion, kind, provenance key, slot (nullable), before instance (nullable), after instance (nullable) | -| `checks` | verification result | instance, check name, version, required flag, outcome, when, detail | -| `dependencies` | provenance edge | consumer instance, producer instance | - -The `dev` schema's product tables (`l2files`, `refimages`, `diffimages`, -`sources`, `merges`, `astroobjects`, `astroobjectsmeta` and their -companions) are kept with their names and columns. Each gains `run`, -`attempt` and `instance` (or `result_set`) columns, and where a -uniqueness constraint would block two runs holding the same logical -product it is widened to include the run or set. Nothing is renamed and -nothing is dropped. - -## Runs - -A run's kind and input selection are fixed at creation. Scratch runs -make scratch outputs; production runs make candidates. Production uses -the one production database, while scratch runs may name a trial -database. A run may bind another run's complete current result-set -instance as a frozen input. The dependency retains that instance, so -the binding stays valid after supersession. Scratch result sets are -usable only within their own run. - -A run is eligible to finish once every unit is terminal. Finishing -requires an explicit `finish`; a finished run is never reopened. -Seeding copies configuration and permitted input selections without -authorising reuse of another run's scratch outputs. A run created by -the processing-date loop takes its owner from the spec and records the -spec's location as its `input_selection_ref`. A loop promotion records -`who = scheduler` (the [loop](loop) page). - -### Recovery from a seed - -`run create --seed --only-failed` creates a recovery run. What it -copies depends on the seed's kind. For a production seed, it copies -the release (or code revision and image digest), settings and input -refs, lane, resource profile, database target, max attempts and -check-policy ref. It sets `purpose` to name the seed it recovers and -takes `selected_stages` from the seed's list, starting at the earliest -stage with a non-complete unit: `failed`, `cancelled`, or `running` -with a job-less or `lost` attempt. - -At that stage and every later one, each non-complete seed unit becomes -a new `pending` unit carrying `units.seeded_from_unit` and a copy of -its frozen `unit_inputs`. Each copied binding names the same producer -instance, protected by the existing deletion fence like any frozen -input of unfinished work. `run start` skips stages the seed completed -and goes straight to the seeded units. - -A scratch seed's recovery run recreates every unit from the first -stage. It carries over only that stage's inputs and settings; later -stages must regenerate their inputs within the new run, as in a fresh -run. Recovery refuses with exit 64 if the seed has no non-complete -unit or is deleting or deleted. `--only-failed` without `--seed` is a -usage error. - -### Expiry - -Scratch runs receive a default `expires_at` of fourteen days after -creation. `pinned` holds a run past that date. The expiry sweep deletes -unpinned scratch runs past their `expires_at` through the same -operation as `run delete`. It acts under explicit authorisation, -rather than as the run's owner. Under the run lock, it re-checks kind, -pin and expiry and refuses deletion if any condition no longer holds. -The warning mechanics are not decided here. - -## Units - -A unit is pending until its declared input set is complete and its -upstream attempts are selected. It then becomes ready, and running -when an attempt is allocated. Selecting a successful completed attempt -makes it complete. A retryable failure returns it to ready while -attempts remain; otherwise it becomes failed. Cancellation or a -permanently failed required dependency cancels an unstarted unit with -a recorded reason. Complete, failed and cancelled are terminal. Unit -state is stored, not derived. Inputs are bound in `unit_inputs` before -execution and retained for retries. - -`register`'s unit id is `/`, for -example `admit/e20260821001234/SCA07` or -`difference/e20260821001234/SCA07`, derived from the manifest -`register` reads. One `register` unit follows every producer, with no -occurrence counter, and is runnable singly for one producer at a time. -Rows written under an earlier unit-id form stay unchanged. - -A unit created by `run create --seed --only-failed` carries -`units.seeded_from_unit`, identifying the seed unit it re-runs. It -enters the same state machine as an ordinary new `pending` unit. - -## Input binding and composition - -A stage declares its input set by product kind and role. The launcher -resolves each entry from this run's own instance of that kind for the -unit first, then from the run's frozen input selection. It uses -`dev`'s selection rules: the current reference by field and filter, -and the current PSFs by filter and detector. It never resolves an entry -from an unrelated run unless the selection explicitly names that -instance. - -The binding is written to `unit_inputs` before execution, and the -stage's input manifest is generated from it. Hand-composed manifests -are a test path; production runs assemble inputs through the launcher. - -### Submission - -Before any submission write (`run submit`, `run start`, `run local`), -the launcher reads `manifest.json` at the local or `s3://` `--inputs` -location. It collects every output entry's `instance` and every -`inputs.result_sets` entry. It excludes `inputs.products`, which -names what the upstream attempt read rather than this unit's inputs. -It then creates the unit, binds the registered product instances -through `bind_unit_inputs`, commits both together and allocates the -attempt. - -Unregistered names, such as a delivery manifest or dev-era template -entry, bind nothing and are logged rather than refused. An absent, -invalid or unreadable manifest refuses submission before any write: -exit 75 for a network error, exit 65 otherwise. Binding is idempotent -per (unit, instance): a retry and a seeded `--only-failed` re-run both -re-read the manifest and bind nothing new. - -Binding takes each producer instance's run row for share and refuses -with exit 65 if that run is deleting or deleted, as -`register_manifest` does. An uncommitted binding therefore holds the -producer's run against deletion. The fence also covers `run inputs` -and the loop's binding. These bindings are the deletion guard's basis. - -### Composition - -One primitive backs `run inputs`, `run start`'s `compose_inputs` and -the loop's maintain, crossmatch and alerts composers: -`rapidpipe.runs.binding.bind_input_set(conn, storage, *, run_id, stage, -unit_kind, unit_id, dest, compose, reuse_existing=True)`. Submission-time -binding (`run submit`, `run start`, `run local`, through -`bind_registered_inputs`) remains a separate path, sharing only the id -rule with the primitive. - -The primitive first checks whether `manifest.json` exists at `dest`. -`run inputs` refuses reuse with exit 64; `run start` and the loop reuse -it. Next, `add_unit` admits the unit through the run fence. This is the -first write-side check, before copying anything, reading template or -producer manifests, or parsing an existing manifest. A finished, -deleting or deleted run is therefore refused with exit 64 before its -inputs are inspected. - -The composer reads the existing manifest or calls its `compose` -callable to copy inputs and build one. The `manifest_instances` rule -collects every output entry's `instance` plus every -`inputs.result_sets` entry. The composer binds registered instances -through `bind_registered_inputs`, commits, then writes the manifest to -storage only if it did not already exist. Reuse commits an idempotent -re-bind of the existing manifest's ids to the consuming unit without -rewriting the manifest. A stored manifest therefore means its bindings -were committed. If the write fails after commit, the next call finds -no manifest and composes again. - -Both `compose_inputs` and the loop's composer call this primitive; a -test fails if either binds or writes a manifest independently. Thus -`run inputs` binds a template's `inputs.result_sets` and output -entries, and exits 65 if a producer is deleting or deleted at binding. - -## Attempts and retries - -Each try has a fresh attempt id and an exclusive output location. An -attempt has no disposition while queued or running. Success requires -a valid completion manifest and complete declared outputs; exit zero -alone is insufficient. Selection locks the unit row and atomically -sets its selected attempt and complete state. The selected attempt -belongs to that unit and never changes. Late results cannot reopen a -terminal unit or replace its selection. - -The run records a maximum attempts per unit, counting the first try -and every retry. The launcher retries; a Batch job runs its container -once. An attempt ending `transient` returns the unit to ready for -submission as a new attempt. Once one of a unit's attempts is lost to -a Spot reclaim, in the run or in a run it was seeded from, every -further attempt of that unit is submitted to the reclaim queue -(`RAPIDPIPE_BATCH_RECLAIM_QUEUE`) when that is set and differs from -the job queue; otherwise it goes to the job queue like any other -retry. Only code 75 and the approved -infrastructure failures listed in the [stage contract](stage-contract), -"Exit codes", are recorded `transient`. Exhausting the allowance fails -the unit. `lost` means the scheduler lost the job: unresolved -execution, treated as a possible writer until resolved. - -`submit_unit` records the resolved input-set and settings locations on -`attempts.inputs_location` and `attempts.settings_location`, alongside -the frozen bindings in `unit_inputs`. - -### Job-less attempts - -`run reconcile --resolve-jobless [--older-than -SECONDS]`, default 600 seconds, first looks for a scheduler job under -the attempt's deterministic job name. Exactly one match repairs the -attempt by recording the job id; an ambiguous match is left alone. -Once past that age, an attempt with no disposition, scheduler job or -repair is locked and recorded as `lost`, with a null exit code and a -reconcile note explaining why. Its unit returns to `ready` if attempts -remain under the allowance, or `failed` otherwise. This resolves -job-less running attempts instead of leaving them open indefinitely. - -### Reusing result sets - -A database-writing stage's `done_check` decides whether to reuse a -complete result set or write a new one. It checks the producing -attempt's disposition as well as completeness. Reuse requires a -complete, retained set in this run with the same kind and provenance -key, produced either by the calling attempt or by an attempt whose -disposition is `succeeded`. - -A set left by an attempt that committed rows and then failed, or by -another attempt still without a disposition, is not reused. The retry -writes a new set under its own instance. The orphaned rows remain the -run's own until deletion, but are unreachable because their producer -is never the selected attempt. - -`load`'s, `crossmatch`'s, `statistics`' and `prune`'s `done_check` -all use this rule through `db/sources.find_complete_source_set` and -`db/objects.find_complete_result_set`. Both join `attempts` for the -producing attempt's disposition; the per-stage pages record each -`done_check`'s key. - -## Instances and custody - -`register` preserves the instance ids and producing run, stage and -attempt from the manifest, recording the registering attempt -separately. Replaying an identical manifest is a no-op; conflicting -content for an existing instance id is an error. Database-writing -stages create their own attempt-scoped result-set instances and record -completion, including empty completion. After an uncertain commit, the -same attempt recovers its recorded results instead of inserting -duplicates. - -Published identity, content metadata and provenance are immutable. -Custody and deletion metadata change only through promotion and -deletion. Scratch never leaves scratch. Promotion makes a candidate -current; supersession returns it to candidate. - -A product registered by `dev` enters the run model through an import -run: `reference-import` for a reference, `psf-import` for a PSF. Each -is a production run whose one unit registers the product with its own -run, attempt and instance. The instance is a candidate in project -custody and can be named as a frozen input. Production runs that -depend on it can therefore be promoted: `dev`'s registered products -belong to the project, while a scratch import would refuse every -dependent production promotion. - -Two partial unique indexes govern current selection. One allows at -most one current instance per kind and provenance key across all -current rows. The other allows at most one per kind and slot wherever -the fill has resolved a slot ([products](products) page). Current -selection is a view over the instance table, covering file products -and result sets alike. A consumer resolves a related selection from -one database snapshot. - -## Promotion - -### Eligibility - -Automatic and manual promotion require completed, selected outputs, -a recorded image digest identifying a released artifact, and passing -results for every required check under the resolved check-policy -version. Policy resolution uses an explicit `--check-policy`, then the -run's `check_policy_ref`, then the default `rebuild-trial@1`. There is -no unchecked exception: every deliverable of a kind the policy covers -passes the gate ([checks](checks) page). Missing or failed required -checks refuse promotion with exit 1. - -Every provenance dependency, followed to its roots through the whole -chain, must be `current`, `superseded`, or an after-instance of the -same promotion request that passes validation in its own right. -Anything else refuses promotion with exit 1, naming the ancestor and -its state ([checks](checks) page has the states and walk). `run show` -ends its listing with a `state:` block, one line per candidate or -current instance in those states, after `units:`, `attempts:` and any -`promotions:` ([checks](checks) page has the format). - -The replacement's kind and slot must match the requested selector. -It must be retained, and a result set must be complete -([products](products) page has the identity). Every deliverable's -selected producing attempt must have an execution record whose image -digest belongs to a complete release. Otherwise promotion is refused -unless the operator passes the explicit unreleased exception -(the [releases](releases) page has the record and check). Automatic -promotion stays disabled until the team approves its policy. -Reprocessing is a production run with auto-promote off. - -### Selecting replacements - -`promote_run` first fills the run's rows through -`product_identity_fill()` ([products](products) page). A candidate -registered before its slot resolved is therefore not refused for that -alone. The default deliverable list contains every candidate the run -produced through its unit's selected attempt, grouped by (kind, slot). -It excludes intermediate revisions and unselected attempts. - -A candidate whose slot or identity remains null is refused by name: -a slot never exists without its identity ([products](products) page). -Two candidates sharing a slot are refused because each (kind, slot) -needs exactly one replacement. Piecemeal promotion names an explicit -subset of the list and passes the same validation. - -A change may withdraw a slot by naming no replacement. The replaced -instance returns to candidate, and the promotion record's after-instance -is null. Expected-before for a slot is its current instance, or none. -A current instance with a null slot is invisible to slot promotion: -it can be neither replaced nor withdrawn until an operator resolves it -by hand. - -A replacement supersedes its predecessor whenever they share (kind, -slot), regardless of differing settings or upstream instances. -Reprocessing with changed settings therefore replaces old difference -images in their slots instead of adding instances beside them. - -### Frozen plans - -`run promote-plan ` reads the run's slot groupings under the lock -and prints them without writing anything. Later, -`run promote --plan ` applies that file under a fresh lock. -It refuses with exit 1, naming the first slot whose actual current -instance or candidate no longer matches the plan, and writes nothing -([tool](tool) page has the commands). - -The plan file must contain a non-empty JSON list of -`{kind, slot, before, after}` entries. A JSON `null`, an empty list or -any other shape exits 64 before the file is read as a plan or anything -is written. - -### Applying changes - -A change into an `association-set` slot with a non-null expected-before -is refused unless that instance is an ancestor of the after-instance, -following the provenance key's `base` field. This prevents a live -batch from replacing a reprocessing campaign's chain head with its -own older chain. Rollback's recorded inverse is the sole exception: -it undoes a change that already passed this rule. - -Ordinary slot replacement cannot substitute for the chain switch -operations.md describes, the promotion that moves a whole stream -between catalog chains. That switch is not built, and nothing writes -`loop_dates.kind = 'switch'`. - -Only production outputs can be promoted; scratch never leaves -scratch. On rows a run wrote, `vbest` is 1 while the instance is -current and 0 otherwise. Rows `dev` wrote with `run` null keep -`dev`'s flag, including rows an import run links through `instance`. -A mapped kind whose instance has no row refuses promotion. - -Every promotion takes one transaction-scoped advisory lock. Under it, -the transaction checks all expected previous selections, including -expected absence, against the actual selection. Any mismatch refuses -the whole request. It then validates dependencies, updates custody and -records the action. Each affected slot has a before and after instance, -either nullable; `promotion_changes` stores the slot alongside the -provenance key. - -### Rollback - -Rollback inverts the exact recorded change by slot: the recorded -after-instance becomes the expected before, and the recorded -before-instance, possibly null, becomes the new after. A change -recorded before slot identity, with no slot, is refused as not -reversible ({ref}`legacy compatibility ruling -`); every promotion `rapid_rebuild` has -made was recorded after slot identity, so this refusal is reachable -only against a disposable database. - -Reversal is itself a promotion, carrying the inverse mapping and -`request_context.rollback_of`. It is refused if the recorded -after-selection is no longer current. `promote()` accepts only a slot -selector; anywhere else, a selector naming an instance whose own slot -is null is refused. - -## Deletion - -### Guarding the run - -An owner may `run delete` a scratch run whether or not it is finished. -A finished run admits no new unit, attempt or input binding, but can -still be deleted. In one transaction, deletion locks the run, verifies -the owner, checks for blockers and marks the run deleting. - -Deletion is refused if: - -- An attempt is queued, running or unresolved, meaning it has no - disposition. -- A frozen input binding of unfinished work or a provenance dependency - of a retained output outside the run points into it. -- A row in `xsources` references one of the run's rows. That table is - outside the cleanup set, so deletion would leave an orphan. -- A row in `refimimages`, `refimcatalogs` or `refimmeta` belonging - to another run's reference points to one of this run's rows. Its own - reference's satellite rows are removed during cleanup. - -`lost` is a recorded resolution of uncertainty. Reconcile writes it -only after the scheduler no longer returns the job, or after -`run reconcile --resolve-jobless` finds no job under the attempt's -name. A `lost` attempt therefore does not by itself block deletion. -Attempt allocation, input binding and result acceptance use the same -run fence and refuse a deleting or deleted run. - -The guard counts only live consumers outside the run. A `unit_inputs` -binding counts only when its unit's run is not `deleted`; a -`dependencies` edge counts only when the consumer instance's -`deletion_state` is not `deleted`. Tombstones are never removed. -A consumer run still `deleting` blocks deletion of its producer's run -until the consumer finishes deleting. `run delete` refuses with exit 64. - -Bindings recorded through `bind_unit_inputs` before an attempt starts -make the unit a live consumer immediately. The deletion guard sees it -before it produces any output, which makes submission-time binding -safe. - -### Storage and row cleanup - -Before storage is touched, every attempt's output location must lie in -the scratch bucket under `runs//`; otherwise the whole delete -is refused. Cleanup removes the run's object versions first. A -per-object storage deletion error leaves the run deleting and touches -no row; a retry resumes at storage. - -After clean storage removal, one transaction removes the science rows, -marks the instance rows deleted (`deletion_state`) and marks the run -deleted. The science-row cleanup set is `l2files`, `l2filemeta`, -`refimages`, `diffimages`, `diffimmeta`, `psfs` and `sources` -(the parent delete reaches the children), each where `run` equals the -run being deleted. `l2files` and `l2filemeta` are included because a -scratch run owns its admitted rows, and the fence already refuses -deletion when another run binds them. - -`refimimages`, `refimcatalogs` and `refimmeta` have no `run` column: -they are reached through the `refimages` row's `rfid`. The same -transaction joins them through this run's `refimages` rows and deletes -them before those parent rows. Another run's reference is untouched; -the guard prevents deleting rows it references. Rows with `run` NULL, -everything `dev` wrote, are never touched. - -Failure before the final transaction leaves the run deleting. -Re-running cleanup resumes where it stopped. Custody stays separate -from deletion state. Run, unit, attempt and instance rows remain as -tombstones with their provenance. Scratch expiry calls the same -operation after the expiry date, with a warning first and a pin to -hold a run. - -## Storage layout - -Two buckets separate personal and project custody: -`s3:///rapidpipe` for scratch runs and -`s3:///rapidpipe` for production runs. At submission, -the launcher chooses the root from the run's kind. Scratch reads -`RAPIDPIPE_OUTPUTS_ROOT_SCRATCH`, falling back to -`RAPIDPIPE_OUTPUTS_ROOT` when unset. Production requires -`RAPIDPIPE_OUTPUTS_ROOT_PRODUCTION` and never falls back to the -unsuffixed variable. - -Runs written under `rapidpipe-firstrun/` stay there; their attempt -rows retain that location. Candidate and current objects share the -project bucket and never move at promotion. Ordinary execution roles -cannot delete completed objects. Only the cleanup role deletes, and -only through `run delete`. Within either bucket: - -``` -runs/////manifest.json -runs///// -``` - -Admitted inputs and reference images use this scheme in the project -bucket, in the run that admitted or built them. Consumers read current -products through the selection, never by guessing paths. Alert -containers use the same scheme. The outbox row holds the container's -location and each alert's block locator: the offset and size of its -Avro block, plus the record's position within the block and container. -Avro compresses records per block, so a single record has no byte range -of its own (the [alerts](alerts) page has the outbox's full column list). - -### Execution and cleanup credentials - -Each run kind has its own Batch job definition. Scratch uses -`rapid-rebuild`, whose job role writes only to the scratch bucket. -Production uses `rapid-rebuild-production`, whose job role can write -`rapidpipe/*` in the products bucket but never delete. It also reads -the scratch bucket's `rapidpipe*` prefixes, which hold staged inputs, -settings overlays and input-set manifests. - -Scratch reads `RAPIDPIPE_BATCH_JOB_DEFINITION_SCRATCH`, falling back to -`RAPIDPIPE_BATCH_JOB_DEFINITION` when unset. Production requires -`RAPIDPIPE_BATCH_JOB_DEFINITION_PRODUCTION` and never falls back to the -unsuffixed variable. - -`run delete` runs launcher-side under the caller's credentials, not a -Batch job's. On a workstation, the instance role holds -`rapid-scratch-cleanup`, which can delete object versions only under -the scratch bucket's `rapidpipe*/runs/*` prefix. The deletion code -refuses any object outside the scratch bucket. The [tool](tool) page -names the dedicated cleanup principal for workstation submission. - -## Identifiers - -Runs, attempts and product instances receive globally unique, -time-ordered identifiers (ULID) before execution or manifest -publication. Whoever creates the row allocates the id; the local -runner and launcher need no database for allocation, and registration -preserves it. Date-and-sequence names such as `r-20260921-007` are -display labels only. Unit identity is unique within a run and stage -and uses the product vocabulary's unit identifiers. - -## Schema - -The run-model tables land as one migration beside the baseline. -Additive columns and widened keys on the `dev` product tables land -one kind at a time, difference image first, with the stage that writes -each kind. The `dev` schema coexists and is kept as far as possible; -the rebuild adds without renaming or dropping. - -## Trial database - -A scratch run's `database target` may name a trial database instead of -the one production database. A trial database lives beside `rapid` -on the production PostgreSQL instance, built with rapid's own schema. -It is neither a separate server nor a clone. - -rapid's migration applier (`database/apply-migrations.sh`, -`database/migrations/`, files named `YYYYMMDD-NN-name.sql`) builds -and updates it from inside the pipeline container image. The applier -records each file's sha256; an applied migration is never edited. -Clients connect through pgbouncer, whose wildcard `[databases]` entry -routes any database name to the one PostgreSQL port. Adding a trial -database therefore needs no pooler config change. - -The pipeline authenticates as a per-tier service login, for example -`rapid_rebuild_pipeline`, with SELECT/INSERT/UPDATE/DELETE on the trial -database's tables. Guarded migrations in rapid's stream grant these -permissions; they no-op where the role does not exist, so CI can apply -the stream from empty. People read through `rapid_read`. That role's -identity (its NOLOGIN cluster role, secret and SSM tree) is provisioned -separately by the system repo's migration stream: the role is -cluster-wide, while rapid's stream owns only the trial database's tables. - -A job definition's `RAPID_PARAMETER_PATH` selects the tier. It points -at the SSM tree `/rapid//db/*` (name, host, port, secret id), -which resolves the trial database's name, host and port. The service -login's password is in Secrets Manager under -`rapid/db/service/-pipeline`. Each tier has its own Batch job -definition, revision-pinned to an image digest. The rebuild tier uses -database `rapid_rebuild`, service login `rapid_rebuild_pipeline`, -tree `/rapid/rebuild` and job definition `rapid-rebuild`. - -A run created under a release submits to the revision that release's -record deployed for the run's kind, regardless of what revision the -job definition currently pins (the [releases](releases) page has the -mechanism). - -## Python interface - -`rapidpipe.runs`, in its `repository`, `cleanup` and `checking` modules, -`rapidpipe.launch.batch`, and `rapidpipe.checks.policy` are the -interface the command-line tool and the scheduler build on, as the -specification's Tools section names them. - -| Function | Does | -|---|---| -| `create_run(conn, kind, owner, purpose, selected_stages, code_revision, image_digest, schema_version, settings_overlay_ref, input_selection_ref, lane, resource_profile, database_target, max_attempts_per_unit, auto_promote, check_policy_ref, seed_run=None, expires_at=None) -> run_id` | Inserts the run row and returns its id; validates that `check_policy_ref` names a real policy and refuses `auto_promote` unless that policy permits it (`CheckPolicyRefused`). | -| `submit_unit(conn, *, run_id, stage, unit_kind, unit_id, inputs_location, settings_location=None, outputs_root=None, job_definition=None, client=None) -> BatchSubmission` | Binds the unit's inputs (`bind_unit_inputs`, below) before allocating the attempt, then starts it on Batch; when not given, the outputs root and job definition are resolved from the run's kind, production failing closed if its configuration is missing rather than falling back to scratch's; records the resolved `inputs_location`/`settings_location` on the attempt. | -| `bind_unit_inputs(conn, *, unit_id, manifest) -> list[str]`, in `rapidpipe.runs.repository` | Writes a `unit_inputs` row for every `instance` the manifest names, in `inputs.products` and `inputs.result_sets`, that is a registered product instance; a name that resolves to no instance binds nothing and is logged, not refused; called by `submit_unit` and `run local` before allocating the attempt. | -| `reconcile(conn, run_id, ...)` | Records the attempts Batch has finished since the last call and selects among them. | -| `cancel(conn, *, attempt_id, reason)` | Terminates a queued or running attempt. | -| `promote(conn, who, reason, changes, check_policy: Policy \| None = None, request_context=None, *, allow_unreleased=False) -> promotion_id` | Runs one promotion from an explicit `changes` list of `(kind, selector, expected_before, after)`, either instance nullable; `selector` is `{"slot": {...}}`. Validated under `check_policy` ([checks](checks) page). | -| `promote_run(conn, run_id, who, reason, *, kinds=None, check_policy: Policy \| str \| None = None, allow_unreleased=False, plan=None) -> promotion_id` | Fills the run's rows, builds its default deliverable list grouped by slot, optionally narrowed to `kinds`, and calls `promote` with it; given `plan` (the list `run promote-plan` printed), refuses under `StalePlan` if the actual current selection or the run's own candidates have moved since the plan was made. | -| `rollback_promotion(conn, promotion_id, who, reason) -> promotion_id` | Submits a past promotion's inverse mapping as a new promotion. | -| `finish_run(conn, run_id) -> None` | Marks a run finished; refuses unless every unit is terminal. | -| `mark_run_deleting(conn, run_id, requested_by, *, expiry=False) -> None` | The deletion fence: locks the run, runs the pre-deletion checks, and marks it deleting. `delete_run` calls it first; `expiry=True` marks the caller as the expiry sweep rather than the run's owner. | -| `delete_run(conn, run_id, requested_by, *, s3_client=None, scratch_bucket=None, expiry=False) -> DeletionReport` | Runs guarded deletion on a scratch run, committing internally rather than inside the caller's transaction; the report lists the objects, versions and per-table rows removed and the instances marked deleted. | -| `expire_runs(conn, *, now=None, s3_client=None) -> list[DeletionReport]` | Calls `delete_run` with `expiry=True` for every unpinned scratch run past its `expires_at`. | -| `pin_run(conn, run_id, pinned) -> None` | Sets or clears a run's `pinned` flag. | -| `resolve_run_policy(conn, run_id, explicit=None) -> Policy`, in `rapidpipe.runs.checking` | Resolves the policy `run promote` and `run start`'s auto-promote validate against: an explicit ref, then the run's `check_policy_ref`, then `rebuild-trial@1`. | -| `run_policy_checks(conn, run_id, policy, *, instance=None, check=None, param_overrides=None, who=None) -> list[RecordedCheck]`, in `rapidpipe.runs.checking` | Runs every applicable policy check over the run's candidate instances from their selected attempts, or one named instance or check, and records a `checks` row for each; `check run` builds on it. | -| `recorded_checks(conn, run_id, *, instance=None) -> list[RecordedCheck]`, in `rapidpipe.runs.checking` | Returns the run's recorded check results, newest first; `check show` prints them. | -| `maybe_auto_promote(conn, run_id, *, who="auto-promote") -> AutoPromoteOutcome`, in `rapidpipe.runs.checking` | Called at the end of a `run start` walk once every unit is complete; with the run's `auto_promote` flag true, resolves the policy, runs its checks over the run's candidates and calls `promote_run` on a pass; the outcome's status is `off`, `skipped`, `refused` or `promoted`. | -| `failed_rerun_plan(conn, seed_run) -> FailedRerunPlan`, in `rapidpipe.runs.repository` | Computes the recovery plan `run create --seed --only-failed` executes: the earliest non-complete position, the seed's stage list, and the non-complete units to seed, or, for a scratch seed, the first stage's units to recreate. | -| `seed_failed_units(conn, *, seed_run, new_run) -> list[str]`, in `rapidpipe.runs.repository` | Creates the new run's seeded units from a `FailedRerunPlan`, copying each seed unit's `unit_inputs` bindings, and returns the created unit ids. | -| `record_attempt_locations(conn, attempt_id, inputs_location, settings_location) -> None`, in `rapidpipe.runs.repository` | Records the input-set and settings locations `submit_unit` resolved for an attempt, on `attempts.inputs_location` and `attempts.settings_location`. | -| `resolve_jobless(conn, *, run_id, older_than_seconds, client=None) -> list[Reconciled]`, in `rapidpipe.launch.batch` | Implements `run reconcile --resolve-jobless`: for each job-less attempt, looks first for a scheduler job under its deterministic job name and repairs the attempt on exactly one match, otherwise records it `lost` once past the age threshold. | - -The command-line tool's `run` and `check` subcommands are thin wrappers -over these; the [tool](tool) page has the command surface. - -## Not decided here - -- The scratch expiry warning mechanics. diff --git a/system/specification.md b/system/specification.md index 24eee98..2f6824e 100644 --- a/system/specification.md +++ b/system/specification.md @@ -2,6 +2,13 @@ **Status: DRAFT** +The [foundation](foundation) page states the record the rebuild is built +on and the thin chain of stages over it. It replaces this page's Runs and +Releases sections and the candidate checks: runs are recipes, jobs are +executions, and official bindings changed in recorded groups take the +place of units, attempts, custody, promotion, check policies and +releases. Where the two pages differ, the foundation page governs. + This specification replaces the earlier design corpus, retained in the repository's history. The team owns the design and every ruling recorded on the [decisions](decisions) page. The team reviews this page at the @@ -222,7 +229,7 @@ A candidate becomes current only if its recorded image is a released artifact; tagging related source afterwards is not enough. Before operations move to a release, schema deployment documents migration order, compatibility with active runs, and recovery. The -[releases](releases) page defines the mechanism, record and command. +[foundation](foundation) page defines the mechanism, record and command. ## Edges @@ -326,10 +333,10 @@ on the `rebuild` branch, documented here: | Part | Scope | Pages | |---|---|---| | 2 | Stage entrypoints | [stage-contract](stage-contract); [reference](reference), [difference](difference), [load](load), [maintain](maintain), [crossmatch](crossmatch), [statistics](statistics), [finalize](finalize), [alerts](alerts), [prune](prune), [export](export) | -| 3 | Runs, attempts, the three output states | [runs](runs) | +| 3 | Runs, attempts, the three output states | [foundation](foundation) | | 4 | Command-line tool | [tool](tool) | -| 5 | Releases | [releases](releases) | -| 6 | Candidate checks, promotion and recovery by hand | [checks](checks); recovery rules on [runs](runs) | +| 5 | Releases | [foundation](foundation) | +| 6 | Candidate checks, promotion and recovery by hand | [foundation](foundation) | | 7 | Scheduled processing-date loop | [loop](loop) | | 8 | Slower pipelines and exports | [reference](reference), [export](export) | @@ -344,7 +351,7 @@ SMDC cutover; none is ADOPTED. The rebuild ships `rebuild-trial@1`, a trial-level policy requiring promotion by hand. A team-approved policy and whether it permits automatic promotion remain open - ([checks](checks) page). + ([foundation](foundation) page). - The boundary between mission-supplied data and RAPID-derived products, and what each side's retention and provenance owe. - Admission rules for duplicate, incomplete or corrected inputs: the diff --git a/system/stage-contract.md b/system/stage-contract.md index 0e08bd2..77b9ee0 100644 --- a/system/stage-contract.md +++ b/system/stage-contract.md @@ -2,252 +2,166 @@ **Status: DRAFT** -A stage implementation is accepted only if it meets this contract. -It expands the specification's "The stage contract" section, drawing on -the specification, the `dev` stage inventory and the `smdc` salvage. +A stage implementation is accepted only if it meets this contract. The +[foundation](foundation) page states the records a stage works within: +runs, jobs, recorded inputs, official bindings and the publisher. ## In plain terms -A stage is one program that takes one declared piece of work, reads the -inputs a manifest lists, does one job, and writes its outputs plus a -manifest saying what it wrote. The same program accepts a local -directory or an S3 location. Image-transform stages use the database -only to check that their run may read the supplied inputs; catalog -stages declare what they read and write. Each execution is an attempt -with its own output location, so a retry cannot overwrite a finished -one. The caller controls retries. +A stage is one Python function. It is given a job: the run's settings, +the inputs chosen for the job at launch, and a folder whose path names +the job and the attempt. It reads only what the job names, computes in +memory, writes its files and a manifest that says which of them are +catalog rows, and returns. It never writes a catalog row and never +decides what is official. The runner around it then calls the +publisher, which loads the declared rows in one transaction. A retry is +a new attempt of the same job in a new folder, and a job that is +already published is never run again. ## The package -The repository builds one Python distribution, `rapid-pipeline`, -with import package `rapidpipe`. The word `rapid` alone means the -project. Development workstations use an editable installation; -release images install the distribution built from the recorded -source commit. +The repository builds one Python distribution, `rapid-pipeline`, with +import package `rapidpipe`. The word `rapid` alone means the project. | Subpackage | Holds | |---|---| -| `rapidpipe.products` | Product identifiers, kinds, manifest types, the storage layout beneath a run, and the pure sky-partition geometry `db` needs. | -| `rapidpipe.db` | Persistence: typed repositories, the connection module, the migrations applier. | -| `rapidpipe.science` | Algorithms the stages call: differencing, coaddition, photometry. Pure functions and wrappers around the C tools. | -| `rapidpipe.checks` | The check registry and the shipped policies. Running checks against a run's candidates, and automatic promotion, is `rapidpipe.runs`, not this package. | -| `rapidpipe.runs` | Runs, units of work, attempts, the three output states, promotion, and recording a run. | -| `rapidpipe.stages` | One module per stage, each directly runnable. | -| `rapidpipe.launch` | Turning a run into Batch jobs, enforcing dependencies, reading results back, and the run walk that carries a unit through its selected stages. | -| `rapidpipe.selftest` | Each stage's packaged fixture and the runner that drives it. | +| `rapidpipe.products` | Product identifiers, the manifest types, the storage layout, and the pure sky-partition geometry (HEALPix indexes and tessellation fields). | +| `rapidpipe.db` | One connection from the `PG*` environment, a secret or a parameter tree. | +| `rapidpipe.science` | Algorithms the stages call: differencing, coaddition, limiting magnitude, the in-memory matcher, sufficient sums, alert records, cutouts, forced photometry. Pure functions and wrappers around the C tools. | +| `rapidpipe.foundation` | The operations over `database/schema.sql`: runs and jobs, `change` and `undo`, `publish`, the stage contract (`JobContext`, `run_job`), and reading and writing `file://` and `s3://` prefixes. | +| `rapidpipe.stages` | One module per stage. | +| `rapidpipe.launch` | Submitting a recorded job to Batch. | +| `rapidpipe.selftest` | The packaged fixtures of the stages still on the older contract, and the runner that drives them. | | `rapidpipe.cli` | The command-line tool: parses the command line and prints, with no orchestration of its own. | A unit test enforces the dependency order across every module, -including lazy and relative imports: a unit imports only units -strictly below it. From lowest to highest, the layers are the leaf -modules (`exitcodes`, `log`, `revision`, `seams`, with no -`rapidpipe` import), `products`, `db` and `science` (which do not -import each other), `checks`, `runs`, `stages`, `launch`, -`selftest`, then `cli`. `release` sits beside this stack: it -imports only the leaves and `db`, and only `cli` imports it. Stage -modules do not import other stage modules, `launch` or `cli`. - -The sky-partition geometry comprises the HEALPix indexes and -tessellation fields in a registered image's row. It lives in -`products` so `db` can derive it without importing `science`. - -Stage entrypoints live in `rapidpipe.stages`, reusable algorithms in -`rapidpipe.science`, and database access and migration code in -`rapidpipe.db`. Existing code is retained only where it implements -this contract. The container build pins C tool versions and build inputs. +including lazy and relative imports: a unit imports only units strictly +below it. From lowest to highest: the leaf modules (`exitcodes`, `log`, +`revision`, `seams`, with no `rapidpipe` import), `products`, then +`db`, `science` and `foundation` (which do not import one another), +`stages`, `launch`, `selftest`, then `cli`. A stage module never imports +another stage module. ## The stage contract -### Declaration - -Every stage exports a declaration containing its name, unit of work, -settings schema, versioned input and output product kinds, database -reads and writes, and supported exit codes. Importing it performs no -I/O. Resources come from the Batch job definition, not the declaration. -Dependencies specify the required product kinds, their mapping to -upstream units, and the condition that makes each input set complete. -A field-wide stage waits for all its declared field inputs. - -Stage names are a stable list: `admit`, `reference`, `difference`, -`finalize`, `register`, `load`, `maintain`, `crossmatch`, -`statistics`, `prune`, `alerts`, `export` (`maintain` has its -own page, [maintain](maintain)). Units of work are `exposure`, -`detector-image`, `field`, `processing-date` and `detector-date` -(`detector-date` is `maintain`'s unit; see -[maintain](maintain), "Unit"). - -### Database access - -A declaration specifies one of four access levels. `none` never -connects, and the shared runner skips its read guard. `custody` -connects only for that guard, which judges the custody of every -registered instance named in the input manifest (see "Invocation"); -it reads or writes nothing else. `read` and `read-write` also pass -through the guard. Every shipped stage reads an input manifest, so -none declares `none`. - -The transform stages (`admit`, `reference`, `difference`, -`finalize`) declare `custody` and need a reachable database whenever -their input manifest names a registered instance. `register` records -file products from manifests; `load` loads source rows. `maintain` -clusters and analyzes a `sources` child table once per observation -date and detector, reading named source sets but writing none of its own. - -`crossmatch`, `statistics` and `prune` read named, completed -database result sets and write new run-scoped sets; their manifests -identify both. The run records the catalog versions and epoch range -crossmatch used. Each downstream stage names its exact predecessor -result set, and statistics identify the association set they describe. - -`alerts` and `export` also read named, completed database result -sets. `export` writes files only, with no database writes of its own. -No stage changes another run's results or the current selection. - -### Invocation - -Each stage exposes `main(argv)` and is directly runnable as -`python -m rapidpipe.stages.`; `rapidpipe stage ` calls -the same entrypoint. One invocation form: - -``` -rapidpipe stage --run --unit --attempt \ - --inputs --outputs \ - [--settings ] [--dry-run] -``` - -`--inputs` names a local directory or S3 prefix containing -`manifest.json`; the stage reads only the immutable inputs it lists. -`--outputs` names the attempt's exclusive output location. Each -execution receives a unique attempt ID: the local runner allocates -local IDs, and the launcher allocates one per Batch job it submits. -A Batch job runs its container once, so a retry is a new submission -with a new attempt. The launcher selects at most one completed attempt -per run, stage and unit. - -Deployment supplies account-specific locations and connection settings -through the environment; explicit input and output locations may -appear on the command line. Credentials never appear in arguments or -committed settings. - -`--dry-run` validates arguments, settings and the input manifest, -then prints the planned inputs and outputs without executing science -code or writing files or database rows. Exit 0 means validation passed; -no completion manifest is written. - -The shared runner, `run_stage` in `rapidpipe/stages/contract.py`, -returns on `--dry-run` before the stage's body runs. A stage that -needs kind-specific input validation before that return passes a -`validate_inputs` callable to -`run_stage(..., validate_inputs=)`. The runner calls it with -the built `StageContext` after generic validation and before the -`--dry-run` return, outside the body. +### A stage is `run(job)` + +A stage is `rapidpipe/stages/.py` with one function, +`run(job: JobContext) -> None`. The stage named `import` lives in +`import_products.py`, since `import` is a Python keyword. The stages on +this contract are `import`, `associate` and `packet`; the +[foundation](foundation) page describes each. + +`JobContext` carries: + +- `job`, `run`, `stage`, `scope`, `survey`, `dataset`: the job's + identity and the scope it was launched with; +- `settings`: the run's whole settings body, from which the stage reads + its own table; +- `inputs`: the job's recorded `job_inputs` rows (`kind`, `product`, + `key`, `source_job`, `binding`), the authority for what the stage + reads; +- `attempt` and `prefix`: the attempt number and its URL, + `/////`, and `path`, the local + directory the stage writes into; +- `connect()`: a context manager yielding a database connection for + short reads, closed on exit. + +A stage reads the rows and files its inputs name, in short +transactions, and never the official views: what is current is decided +by whoever launched the job. It computes in memory, writes its files +under `path`, and writes the manifest with +`job.write_manifest([(table, file), ...], **fields)` last. It returns +without publishing. ### Settings -Each stage ships default settings and a settings schema under -`settings/.toml`. `--settings` supplies an overlay: tables merge -recursively, while supplied scalar and array values replace defaults. -Unknown keys and invalid values fail with code 64. Before execution, -the runner retains the resolved settings and their canonical hash. -Algorithm parameters, including random seeds where used, belong in -settings; deployment locations and credentials do not. +A stage's settings are its table in the run's settings body, for +example `[associate]` in `rapidpipe/settings/foundation-thin.toml`. The +body is stored whole on the run when the run is created, so there is no +overlay and no resolution at run time; a retune is a new run. +Algorithm parameters belong in settings, with `dev`'s values where +`dev` hard-codes them. Data locations and credentials do not: a source +directory is a job's scope, and the outputs root is the environment's +`RAPIDPIPE_OUTPUTS_ROOT`. ### The manifest -A stage publishes its manifest only after its outputs are complete. -Downstream stages use only the completed attempt the launcher selected. -Database writes are attempt-scoped and retry-safe: a retry after an -uncertain commit recovers completed results or discards incomplete ones -without duplicating them. Writing a manifest does not promote products -to current. - -The manifest records its schema version; run, unit, stage and attempt -IDs; a reference to the execution record; references to the input -manifest and any database result sets read; and each output's identity, -kind, format version and location, with byte size and SHA-256 for -files. The execution record, kept by `rapidpipe.runs`, retains the -source revision, working-copy changes if any, image digest when -applicable, database schema version, and resolved settings. Input -provenance identifies exact reference versions and their constituent -exposures. - -Each product kind defines the registration metadata its manifest entry -must carry. `register` validates that metadata and writes product rows -without reading product contents. It remains independently runnable; -sharing a Batch job with a transform changes nothing about attempt -identity, completion or retry safety. +`manifest.json` under the attempt's prefix declares the files the +publisher loads: + +```json +{"outputs": [{"table": "objects", "path": "objects.parquet", "checksum": "sha256:<64 hex>"}]} +``` + +`table` is one of the product tables, `path` is relative to the +prefix, and `checksum` is the file's SHA-256. Each Parquet file holds +the table's columns without `job`, which the publisher sets; columns +the file leaves out take their defaults. Any further fields a stage +passes (counts, timings, provenance) are kept with the manifest on the +job row. Files the manifest does not declare are the stage's own +products, reached through the URL a declared row records. + +### Running a job + +`rapidpipe.foundation.run_job` runs one attempt of one job, and every +way of running a stage goes through it: + +- `rapidpipe job local --run --stage --scope JSON + [--inputs JSON] [--expects JSON] [--excludes JSON]` records the job + and runs it in this process; +- `rapidpipe job run --job ` runs an existing job's next attempt; +- `rapidpipe job submit` records the job and submits one Batch job + whose command is `rapidpipe job run --job `. + +`run_job` reads the job and stops at once, touching no file, if it is +already published. Otherwise it claims the next attempt, marks the job +`running` with its start time, closes its connection, and calls the +stage's `run(job)`. Under an `s3://` outputs root the stage writes into +a local working directory, which is uploaded to the attempt's prefix, +manifest last, before publication. Then it calls `publish` and records +the end time and exit code. Batch runs one attempt at a time; two +concurrent attempts of one job are not supported on `s3://`. ### Exit codes -| Code | Meaning | Caller's action | -|---|---|---| -| 0 | Success, manifest published | none | -| 64 | Usage: bad arguments, invalid settings, missing environment | fail, no retry | -| 65 | Declared input absent, corrupt or incompatible after its storage was reached | fail, no retry | -| 69 | Declared, not implemented in this build | fail, no retry | -| 70 | Unclassified stage error; stop for investigation | fail, no retry | -| 75 | Recognised temporary dependency failure; repeating the same work may succeed | retry within the limit | - -These six codes are the stage subset (`STAGE_EXIT_CODES` in -`rapidpipe/stages/contract.py`) of the vocabulary in -`rapidpipe/exitcodes.py`. The [tool](tool) page has the full table; -a stage never exits 1 or 2. The entrypoint maps argument errors to 64 -and unhandled exceptions to 70. - -Code 69 is `sysexits`' `EX_UNAVAILABLE`. A stub stage validates -arguments and settings, then declines to run: 64 would misreport a -correct invocation as a usage error, while 70 calls for investigation. -`disposition_for` maps 69 to `failed`. No stage in this build -returns it; it remains reserved for a future stage declared but not -yet implemented (see [decisions](decisions)). - -The launcher reports forced termination. Batch does not retry: -the rebuild's job definitions carry one attempt. The launcher retries -as a fresh attempt, within the run's maximum attempts per unit, after -exit 75 or an approved infrastructure failure. These are jobs with no -container exit code whose status reason starts `Host EC2` (the host -was reclaimed) or whose container reason starts `Cannot` or -`DockerTimeoutError` (the container never started). It records other -terminations without a stage exit code, and unexpected codes, as not -retried. - -## Local execution - -Each stage fixture under `tests/fixtures//` includes minimal -input files, settings, required database seed data, and expected -products with documented comparison tolerances. It ships as package -data under 1 MB. - -`make db` starts a local PostgreSQL with Q3C and applies the -migrations; fixture setup loads the seed data. `make stage-` -prepares an isolated fixture, runs the stage without account -credentials, and checks its products and manifest. Provenance fields -are validated for shape, not compared with fixed IDs or paths. CI uses -the same commands. - -The same fixture also runs through `rapidpipe selftest ---stage ` -on Batch as an ordinary job against the deployed image and digest. -The submitted job's execution record is evidence that it ran, distinct -from the real-tool fixture gate below. Each rebuilt science stage also -has an IMSS comparison on fixed inputs, with differences and tolerances -approved by the team before operational use. +| Code | Meaning | +|---|---| +| 0 | The job was published, or was already published | +| 1 | The job was published, but its automatic binding was refused; its rows are an unbound candidate | +| 64 | Usage: bad arguments, no outputs root, or a stage with no `run(job)` | +| 70 | The stage or the publication raised; the job is marked `failed` | + +Nothing retries by itself. A retry is `job run` on the same job, or a +Batch resubmission of it, and lands in a new attempt. + +## Stages on the older contract + +`admit`, `reference`, `difference` and `finalize` still run as +`rapidpipe stage --run --unit --attempt --inputs --outputs +[--settings] [--dry-run]`, through the shared runner `run_stage` in +`rapidpipe/stages/contract.py`, with their default settings under +`rapidpipe/settings/.toml` and their fixtures under `rapidpipe +selftest`. They write files and a manifest but publish nothing, and +their exit codes are the six of `STAGE_EXIT_CODES` (0, 64, 65, 69, 70, +75). Each moves to `run(job)` when it is ported, declaring the rows its +registration block carries today (`l2files`, `refimages` with its +children, `diffimages`, `psfs`, and for `difference` the `sources` set) +as Parquet outputs, and computing limiting magnitude, overlapping +fields and sky indices before publication. ## What this replaces -This contract replaces the per-stage scripts on `dev` -(`awsBatchSubmitJobs_*`, the launcher and register scripts, the -operator loop) and the `smdc` branch's single dispatching entrypoint -with manifest-driven job types. - -The launcher enforces dependencies and controls retries. -`rapidpipe.runs` owns attempt allocation and the execution record. -Stages execute the supplied attempt and report its outputs. +This contract replaces the run model's stage invocation: the +`--run/--unit/--attempt` arguments, the execution record, custody +checks before a stage reads, `register` writing rows from manifests, +and stages writing catalog rows themselves. The run model's database +stages (`register`, `load`, `maintain`, `crossmatch`, `statistics`, +`prune`, `alerts`, `export`) were retired with its schema; `import`, +`associate` and `packet` replace the ones the thin chain needs. ## Not decided here -- Each remaining kind's registration metadata; the vocabulary and the - difference-image example are on the [products](products) page. -- Whether delivered statistics describe associations before or after - pruning. A science decision left to the team. +- The port of `reference` and `difference` to `run(job)`, and how + `dev`'s later changes to differencing are taken in first. - The C tool packaging beneath `rapidpipe.science`. diff --git a/system/statistics.md b/system/statistics.md index 4cab296..d7b2c95 100644 --- a/system/statistics.md +++ b/system/statistics.md @@ -2,9 +2,14 @@ **Status: DRAFT** +The module this page describes was retired from the `rebuild` branch with +the run model. The [foundation](foundation) page states what replaced it; +this page stays as the record of the port until the replacement's own page +supersedes it. + `statistics` computes object positions, fluxes and source counts from a field's association set and records them in `astroobjectsmeta`. The -[products](products) page fixes the vocabulary. +[foundation](foundation) page fixes the vocabulary. The stage is ported from `dev`'s `pipeline/computeStatisticsForAstroObjects.py` onto the pipeline @@ -82,7 +87,7 @@ The stage runs in one transaction: statistics set with the same key already written in this run only if its producer is the calling attempt or has disposition `succeeded`. The attempt-disposition join is - `db/objects.find_complete_result_set`; the [runs](runs) page has the + `db/objects.find_complete_result_set`; the [foundation](foundation) page has the general rule. A fresh attempt writes its own set if the earlier attempt committed rows then failed, or is another attempt still without a disposition. @@ -194,7 +199,7 @@ replace them. Delivered statistics describe the association set, following `dev`'s order: crossmatch, statistics, prune. Pruned-set input is designed in -but unused ([products](products) page): the key can already name either +but unused ([foundation](foundation) page): the key can already name either kind of set. Enabling it later needs a settings change and the code behind it. diff --git a/system/tool.md b/system/tool.md index c868eeb..4884009 100644 --- a/system/tool.md +++ b/system/tool.md @@ -2,6 +2,11 @@ **Status: DRAFT** +Parts of this page describe the run model that the [foundation](foundation) +page replaced: units, attempts, custody, promotion, checks and releases. +Where the two differ, the foundation page governs; its "The verbs" section +lists the commands the tool has today. + `rapidpipe` is the team's single command-line tool, shipped in the pipeline repository and already the image's entrypoint for stage invocations. It meets the [specification](specification)'s Tools @@ -19,19 +24,19 @@ arguments keep their names. | Operation | Subcommand | |---|---| -| Create a run | `run create` (`--seed ` records lineage only: it inherits no settings and no input references from the seeding run, unless paired with `--only-failed`, which makes it a recovery run over the seed's non-complete units instead (the [runs](runs) page has the mechanics); `--release ` is the [releases](releases) page's; `--auto-promote --check-policy P` is refused unless P permits automatic promotion (the [checks](checks) page has the gate)) | +| Create a run | `run create` (`--seed ` records lineage only: it inherits no settings and no input references from the seeding run, unless paired with `--only-failed`, which makes it a recovery run over the seed's non-complete units instead (the [foundation](foundation) page has the mechanics); `--release ` is the [foundation](foundation) page's; `--auto-promote --check-policy P` is refused unless P permits automatic promotion (the [foundation](foundation) page has the gate)) | | Start a stage or the whole loop | `run start` | | Rerun part of a run | `run start --stage ` | -| Watch progress | `run status [--watch]`, `run show` (a `state:` block at the end, after `units:`, `attempts:` and any `promotions:`; the [checks](checks) page has the states), `run timings [--stage S] [--json]` (queue, execution and orchestration time per attempt; the [observability](observability) page) | +| Watch progress | `run status [--watch]`, `run show` (a `state:` block at the end, after `units:`, `attempts:` and any `promotions:`; the [foundation](foundation) page has the states), `run timings [--stage S] [--json]` (queue, execution and orchestration time per attempt; the [observability](observability) page) | | Cancel and restart from failure | `run cancel `, then `run start` | | List and compare runs | `run list`, `run compare ` (each `instance` line carries the slot column beside the provenance key) | -| Promote a candidate | `run promote --check-policy P` (defaults to the run's own `check_policy_ref`, then `rebuild-trial@1`); `run promote-plan ` prints the run's slot groupings as a frozen plan, without promoting; `run promote --plan ` applies that plan, refusing (exit 1) if the current selection or the run's candidates have moved since, or refusing (exit 64) if the file is not a non-empty JSON list of `{kind, slot, before, after}` entries (JSON `null` included); `run rollback` ([checks](checks) page has the gate) | +| Promote a candidate | `run promote --check-policy P` (defaults to the run's own `check_policy_ref`, then `rebuild-trial@1`); `run promote-plan ` prints the run's slot groupings as a frozen plan, without promoting; `run promote --plan ` applies that plan, refusing (exit 1) if the current selection or the run's candidates have moved since, or refusing (exit 64) if the file is not a non-empty JSON list of `{kind, slot, before, after}` entries (JSON `null` included); `run rollback` ([foundation](foundation) page has the gate) | | Delete scratch | `run delete`, `run expire`, `run pin` / `run unpin` | -| One unit by hand | `run submit`, `run reconcile` (`--resolve-jobless [--older-than SECONDS]` records a job-less attempt `lost`; the [runs](runs) page has the mechanics), `run local` | -| Verify a candidate | `check list`, `check run`, `check show` (the check commands below; the [checks](checks) page) | +| One unit by hand | `run submit`, `run reconcile` (`--resolve-jobless [--older-than SECONDS]` records a job-less attempt `lost`; the [foundation](foundation) page has the mechanics), `run local` | +| Verify a candidate | `check list`, `check run`, `check show` (the check commands below; the [foundation](foundation) page) | | A stage directly | `stage run …` (a synonym of `stage …`, the frozen invocation form the container's entrypoint calls), `stage list`, `stage describe ` | | Fixtures | `selftest --stage `, one of `reference`, `difference`, `finalize`, `load`, `maintain`, `crossmatch`, `alerts`, `statistics`, `prune`, `export`; `admit` and `register` have no fixture | -| Releases | `release cut\|show\|list\|verify` (the release commands below; the [releases](releases) page) | +| Releases | `release cut\|show\|list\|verify` (the release commands below; the [foundation](foundation) page) | | The processing-date loop | `loop run\|plan\|show` (the loop commands below; the [loop](loop) page) | `rapidpipe run --help` groups the subcommands above under five titled @@ -72,7 +77,7 @@ restarts one attempt by hand. A run created with `--release ` submits to that release's recorded job-definition revisions rather than the current pins. Its outputs are promotable without the unreleased-image exception -([releases](releases) page). +([foundation](foundation) page). Each stage resolves inputs in this order: explicit command-line `--inputs`, a `--template` composed through `run inputs` (below), then @@ -128,7 +133,7 @@ for a network error. Binding is idempotent per (unit, instance), so a retry or seeded `--only-failed` re-run binds nothing new. The unit becomes a live consumer of its declared inputs at submission, before producing an -output that depends on them ([runs](runs) page, "Units", "Deletion"). +output that depends on them ([foundation](foundation) page, "Units", "Deletion"). This resolves an explicit whole input set. Which reference a field should use among several eligible ones remains open (below). @@ -210,7 +215,7 @@ unaffected. among several eligible ones): still open, and not resolved by the [loop](loop) page's own field-level binding either. - The automatic check gate `run promote` consults is on the - [checks](checks) page; the team-approved policy content behind it is + [foundation](foundation) page; the team-approved policy content behind it is scientific and stays undecided there. - Removal of the transitional direct cleanup attachment on the workstation role, once the assumed-role path is proven.