Skip to content

foundation, loop, difference, intake, stage-contract, decisions, index: the housekeeping of 2026-10-22 and the reference bound - #92

Merged
rusholme merged 2 commits into
mainfrom
foundation/housekeeping
Oct 23, 2026
Merged

rusholme merged 2 commits into
mainfrom
foundation/housekeeping

Conversation

@rusholme

Copy link
Copy Markdown
Collaborator

What and why

Task 5 of the unattended run housekeeping-2026-10-22 (specification): rapid_docs records what tasks 1 to 4 landed, as system behaviour with its numbers, and Ben's rulings of 2026-10-22 in decisions.md. Every earlier task's ledger line is TASK DONE, so everything is documented as landed; nothing is reported as not landed.

Pull requests documented (Caltech-IPAC/rapid, rebuild): task 1 bytes #312 (intake writes exposures only when expected), #313 (working directory removed after put), #314 (difference outputs trimmed and gzipped; packet reads either suffix), runbook #315; task 2 batch #316 (jobs.batch_job_id), #317 (the loop's reconcile, job submit --job), #318 ([batch.memory_mib]), runbook #319; task 3 pass #320 (listing by the selection's observations), #321 (one connection and one flush per publish pass), #322 (validate_run.py eight at a time), #323 (consumer check one set at a time), runbook #324; task 4 reference #325 ([reference.selection] before_mjd, held, the catalogs key's dataset), runbook #326 with "Results of part 8".

Files

  • system/difference.md: the published files, each with the reader that earns it (the ruling's four: difference, uncertainty, mask, catalog; plus the three packet reads), gzipped FITS, the checksum of the gzipped file, no retention setting, a variant made again by a rerun, either-suffix reads, the cutout parity (781 of 781); SFFT and naive run for their bits only; the trimmed bytes in Measured; a "Not decided" bullet on whether SFFT/naive keep running.
  • system/foundation.md: intro; the job row's Batch job id and the reconcile's reason; the catalogs key with its dataset in Part 1's key table; job submit --job in the verbs; the working directory's removal under "Where outputs live"; per-stage memory as a table under "A job on Batch"; the thin chain's intake, reference and difference rows; publish's one connection and flush, the consumer's one-set memory, the producer pointing at its ruling; the settings paragraph names [batch.memory_mib] and before_mjd; "On SMDC" has eight parts, the shared-field bullet points at its ruling, and a new "Housekeeping" part with five bullets; Validation gains the bounded script; Measurements gains "Housekeeping" (before and after, from the PR bodies and Results of part 8) with the disjoint reference's counts and three observations; "Not decided here": the reference-per-SCA, disjoint-reference and catalogs bullets leave (settled), the producer, other-filters and injection bullets are restated against their rulings, and enter: depth vs disjointness, the bound per filter, the psfs sequence against copied ids, SFFT/naive.
  • system/loop.md: the reconcile before the snapshot, the listing by observation, the reference rule (per field and filter, one SCA's rows) and the difference rule under a bound, Kafka on one connection and one flush, job submit --job as a rerun, held and the reconcile in the pass line, a new "The reference bound" section, the per-field chain in admission order, the measurements before and after; "Not decided here" loses the four items that landed and gains three.
  • system/stage-contract.md: the Batch job id and memory ask on submission, --job, refusal cleared at an attempt's start, the working directory removed after upload.
  • system/intake.md: the conditional exposures row (the page said every intake writes it); its "Not decided" bullets on the reservation and the duplicate row updated.
  • system/decisions.md: eleven rulings of 2026-10-22, one dated line each.
  • system/index.md: scope cells for decisions, foundation, stage-contract, difference, intake, loop.

Divergences from the brief, with the reason

  • intake.md is edited, though the brief does not name it: it stated that every intake writes the exposures row and listed that as undecided, both false after #312. The repository wins.
  • One more ruling than the brief lists, decision-breadth-before-injection: Ben ruled on rapid-rusholme on 2026-10-22 at 23:03 UTC (16:03 PDT), after the brief was written, that the other filters run un-injected, PSF from each file's CRDS ePSF reference, before injection (recorded in the work log). Leaving it out would make decision-housekeeping-first the latest word on the order when it is not.
  • Times on the rulings: PLAN.md v7 gives the stepwise session as 12:57 to 13:08 PDT with times only for the PSF (13:08) and Spot (13:23); the others carry the date alone.
  • The reference per (dataset, field, filter) and the chain in admission order are one entry, as the brief lists them as one item.
  • No run ids, job ids, Batch ids or bucket names on the site; the runbook (part 8) carries them. Instance types and the dataset name are used, as in foundation, intake, loop, stage-contract, decisions, index: the HLWAS run from SOC L2 files to the Kafka topic #91.

Build

Not built on the host (no Sphinx; provisioning is an install). docs.yml is the check. Locally: scripts/check-public-safety.sh OK; every {ref} decision target added resolves; the in-page anchor #the-reference-bound is a level-2 heading under myst_heading_anchors = 3; the added text carries no em or en dashes and none of AGENTS.md's excess-vocabulary words.

Measurements

Documentation only: no operation ran. The figures are tasks 1 to 4's PR bodies and the runbook's "Results of part 8".

Improvements seen (not applied)

  • reference.md still states its scope as (field, filter, SCA) without saying the product key has no SCA and which SCA's rows the loop picks; the loop and foundation pages now say it, and a sentence there would close it.
  • foundation.md's "The wide-area ramp" storage bullet says 42 uncompressed FITS per epoch while #314 counted 72 objects in a run 12 difference job; the bytes agree, the count does not. Left as the ramp's own record.
  • specification.md, tool.md and operations.md still describe the processing-date loop (carried from foundation, intake, loop, stage-contract, decisions, index: the HLWAS run from SOC L2 files to the Kafka topic #91).

What the next task must know

  • New ruling anchors: decision-housekeeping-first, decision-breadth-before-injection, decision-reference-bound, decision-chain-admission-order, decision-injection-outside, decision-producer-last, decision-catalogs-dataset, decision-difference-trimmed, decision-gbtds-after-injection, decision-psf-from-epsf, decision-spot-parked.
  • Status stays DRAFT on every page.

Left

Nothing in this task's sentence.

🤖 Generated with Claude Code

rusholme and others added 2 commits October 23, 2026 04:21
…x: the housekeeping of 2026-10-22 and the reference bound, with their measurements

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant