From 3f46b7e28ba961b963176f4cfad040f1c4c347a2 Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Sat, 24 Oct 2026 06:55:45 +0000 Subject: [PATCH] docs: association in the difference job, reconciled at publish; the loop without chains; the alerts contract, best information at the time (night association) Co-Authored-By: Claude Opus 5.5 --- system/alerts.md | 16 +++++++++++ system/decisions.md | 21 +++++++++++++++ system/difference.md | 35 +++++++++++++++++++++++- system/loop.md | 64 +++++++++++++++++--------------------------- 4 files changed, 96 insertions(+), 40 deletions(-) diff --git a/system/alerts.md b/system/alerts.md index ee2720a..8f87514 100644 --- a/system/alerts.md +++ b/system/alerts.md @@ -7,6 +7,22 @@ the run model. The [foundation](foundation) page states what replaced it; this page stays as the record of the port until the replacement's own page supersedes it. +## The contract: best information at the time (2026-10-24) + +An alert carries the best information at the time it is made, latency +over completeness ({ref}`ruling `). +The `packet` stage that replaced this one reads the object catalog as it +stands when it runs: the trigger's object and its members whose source +sets are official, and the difference images of the forced-photometry +window that exist then, recording exactly what it read. A packet's +history is the detections known at its read and before its trigger. A +late epoch appears in alerts issued after it arrives, never in those +already sent; an epoch replaced later leaves the history of alerts made +after the replacement, not of those already sent. Observations do not +arrive in order, and nothing waits for them to. + +## The retired stage + The `alerts` stage builds an Avro container for one finalized difference image and records each alert in an outbox. It lands on the pipeline repository's `rebuild` branch diff --git a/system/decisions.md b/system/decisions.md index f4326a7..7bdedfd 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -92,6 +92,27 @@ fallback. `dev` has no knob here (a static, offline-compiled reference PSF per detector), so this is a divergence from `dev`. (Ben, 2026-10-23 23:22 UTC, fixes-2026-10-23) +(decision-association-in-difference)= +**Association runs in the difference job, reconciled at publish**: the +difference job matches its sources against its field's objects of its +run as they stand when it reads them, founds an object for each +unmatched source, and publishes sources, memberships and founded objects +together; the per-field chain of associate jobs, its admission order and +its exclusions are retired, and an object's statistics are the sums of +its official memberships, read through a view. Two epochs of a field in +flight at once can both found an object at one new source, so the +publisher, serialised per run and field, attaches a job's founded +objects to any founded since its read, the memberships rebased to the +surviving origin; object ids never change after publication. `dev`'s +route, crossmatch after difference against the catalog as it stands, +is the one taken; its duplicate-tolerant merge pass (`xmerges`), which +changes object ids after alerts have carried them, is named and not +taken. Measured: run 46's chained passes (`r00342-fx-f146`, SCAs 4, 5 +and 8, three epochs) held nine associate jobs of 1.2 s each across 19.3 +minutes; the read of the old chain view on one of its fields took 422 +ms. (Ben, the plan's amendment of 2026-10-24 00:55 UTC; +night-2026-10-24 task 7) + (decision-spot-parked)= **Spot stays parked**: the rebuild's jobs run on demand; the cost figure is read and recorded, and nothing hangs on it. diff --git a/system/difference.md b/system/difference.md index cdff699..89fc4ab 100644 --- a/system/difference.md +++ b/system/difference.md @@ -137,12 +137,14 @@ FITS, reads as before. Trimming and compressing change no pixel: the packets of four epochs published both ways had byte-equal cutouts, 781 of 781. -The manifest declares two tables: +The manifest declares four tables: | Table | Rows | |---|---| | `diffimages` | one, the ZOGY image, keyed (dataset, expid, sca, fid); `rid`, `rfid`, `field`, `jd`, the centre and four corners, `infobitssci` (`dev`'s catalog-outcome mask, one bit per differencer and sign with no PSF-fit catalog), `infobitsref`, `status` 1, the file's MD5 | | `sources` | the ZOGY positive PSF-fit catalog, mapped onto the `sources` columns exactly as the import maps `dev`'s, the brightest `max_sources` by flux over its error (0 keeps every source) | +| `objects` | the objects this job founded: one per accepted source of the image's field that matched no object of the field as the job read it, its id the position hash of that source | +| `memberships` | this epoch's matches, onto objects it read and onto those it founded: each with the source's offsets from the object's founding position and its flux, the membership's share of the object's sufficient sums | The stage draws `pid` from the `diffimages` identity sequence while it reads its inputs, so the source rows carry it. A run has one @@ -154,6 +156,37 @@ factor, the registration residuals, the catalog counts, the count of sources before the cap, SFFT's outcome, and the published files with their plain and gzipped bytes, the gzip's seconds and what was removed. +## Association + +The difference job associates its own sources +({ref}`ruling `). After the +catalog, it reads its field's objects of its run in one short +transaction, the objects every published difference job of the run has +founded there, and records that read exactly: one `objects` input per +founding job it saw, in the manifest's `reads`, which the publisher +writes to `job_inputs`. This is the one input read at execution rather +than named at launch. It then runs `dev`'s two crossmatch passes in +memory: the sources the run's `[associate]` table accepts (fit flags +zero once the informational bright-footprint bit is cleared, S/N at +least `min_snr`) in the image's field match every object within +`match_radius`, a source that matches none founds an object, and the +sources in the neighbouring fields join objects only, inside the field's +inclusion cone when it has eight neighbours. + +At publish, under a transaction-scoped lock on the run and field, the +publisher reads the objects founded since the job's read, by jobs not in +its recorded read. A founded object within the radius of one of them is +not inserted; its memberships go to that object, each one whose source +is within the radius of it, rebased to its origin. The manifest records +the outcome as `reconcile`. Two epochs of one field in flight at once +therefore publish one object per star, whichever finishes first giving +it its id. A membership onto an object the job read is kept as matched; +a source that a job in flight founded nearby is not added to it, as a +sequential run would have, an edge case the reconcile does not cover. + +The metrics add `n_objects_read`, `n_objects_founded` and +`n_memberships`. + ## Settings The run's `[difference]` table in `rapidpipe/settings/foundation-thin.toml` diff --git a/system/loop.md b/system/loop.md index c27b1a3..80c1382 100644 --- a/system/loop.md +++ b/system/loop.md @@ -92,32 +92,23 @@ Its order: official reference and whose key has no official difference and no difference job of the run pending or failed. With a bound, only rows at or after it. -5. **associate**: per field, one job for the earliest official source - set, by `mjdobs`, that is not yet associated, with the field's open - object set as its predecessor. A source set is associated when it is - in the field's official chain, walked back from the official object - set through each associate's recorded predecessor, whatever run made - it, or when an associate of this run naming it has not yet - published. So a new live run of the same dataset continues the - chains where the last one stopped and never associates an epoch - twice. It waits while an - associate of the field is pending, while an earlier selected L2 row - of the field lacks its source set, and while any intake of the run - is pending, since an intake's field is known only once it publishes. -6. **packet**: per field, one job for the earliest associated epoch - without a packet set, once every earlier epoch's packet set is - official, because a packet set reuses its predecessors' forced rows. - It names the reference, the run's Gaia catalog, the science PSF, the - object set, the epoch's source set, the field's earlier packet sets, - and the field's difference images and L2 rows within the stage's - forced window. -7. **Kafka**: `publish` over the run's official, unstamped packet sets + The difference job associates its own sources with the field's + objects as they stand ([difference](difference), Association). +5. **packet**: one job per official source set of the selection with + no official packet set and no packet job of the run pending or + failed, in any order. It names the reference, the run's Gaia + catalog, the science PSF and the source set; the objects, their + members, the forced window's images and the official packet sets' + forced rows it reads as they stand when it runs, and records. +6. **Kafka**: `publish` over the run's official, unstamped packet sets ([foundation](foundation), Publication to Kafka), on one connection and one producer, one set at a time. An idle pass opens no producer. -Fields advance independently, so one pass can carry many fields at -different depths, and a packet job for epoch *n* goes out beside the -associate for epoch *n + 1*. +There are no field chains and no admission order +({ref}`ruling `): every epoch with a +reference is differenced at once, and every official source set gets its +packet job at once, so the loop's only waits are the reference's frames +and the queue. **Pending and failed.** A job is pending while it is `launched` or `running`. A failed job is reported and never resubmitted: its key @@ -173,22 +164,17 @@ reference. The pass reports it under `held`, with the field and the reason, on every pass. A held field does not keep the loop from draining, and the verb exits 0: the gap is recorded, with no timeout. -## The per-field chain - -A field's object sets form one chain, one `associate` per epoch, each -naming the last as its predecessor; its packet sets form a second chain -behind it. The chain is in the loop's admission order, and the loop -admits in `mjdobs` order across every SCA of its selection -({ref}`ruling `). The loop holds a field's -association while any earlier epoch of the field is still upstream. -When two SCAs cover one field, as adjacent pointings do on a grid one -SCA wide, both SCAs' epochs share one reference, since a reference is -per dataset, field and filter, and the SCA is in the PSF key the -difference stage reads per epoch. A loop over every SCA admits each -exposure's SCAs together, so their epochs join the chain in time order; -the HLWAS ramp, which drained SCA 8 before the other SCAs, chained one -such field out of order. An earlier epoch that arrives after later ones -is a rerun from that epoch; the loop never re-associates. +## No chain (2026-10-24) + +Until 2026-10-24 a field's object sets formed one chain, one `associate` +per epoch in `mjdobs` order, and its packet sets a second chain behind +it; an earlier epoch that arrived late was a rerun from that epoch. The +association now runs in the difference job against the field's objects +as they stand, so data arriving out of order needs nothing: a late +epoch's memberships appear in the packets issued after it arrives and +never in those already sent +({ref}`ruling `). Two SCAs on one +field still share one reference. ## What it measured