[rusholme]

What and why

Milestone 0, task 1 of the foundation plan (PLAN.md v5, 2026-10-20, held outside the repository until its rapid_docs page lands after milestone 3): the one schema file of Part 1, applied to an empty PostgreSQL 18 by psql -v ON_ERROR_STOP=1 -f database/schema.sql, a CI workflow that proves it on every push, and tests/db failing rather than skipping without a database. No code reads the new schema yet; task 2 adds the publisher and the change operation against it.

Files

• database/schema.sql (new): static tables with dev's data inline, runs, jobs, job_inputs, official, the scientific tables, the _official views, members_official, the q3c block.
• database/README.md: the apply command, once.
• .github/workflows/foundation.yml (new): copied in shape from db-migrations.yml (postgres:18 service, Q3C install, client, empty rapid database), then the apply, then python -m pytest tests/db/test_schema.py -q -rs.
• .github/workflows/db-migrations.yml: its pytest step gets --ignore=tests/db/test_schema.py. cli-behaviour.yml is untouched (it runs tests/cli). Task 3 extends the exclusion to tests/db/proof/, test_publish.py and test_change.py; task 4 deletes the old workflow.
• tests/db/conftest.py: conn fails with "PGHOST is not set; database tests need a database".
• tests/db/test_connection_live.py: its own PGHOST skip becomes the same failure, so no test in tests/db skips for want of a database.
• tests/db/test_schema.py (new): the schema applies to a fresh empty database (its own scratch database, created and dropped); the plan's tables and views exist; the partial unique index refuses a second open binding on one key and allows one after the first closes; a product row with a null job is refused on job; members_official returns a three-set chain's memberships and omits a source set named in an excludes row, whose rows still exist; under CI=true q3c is present (skipped with that reason otherwise).

Names task 2 codes against

Keys. official.key and job_inputs.key are jsonb objects whose names are the table's column names; the views build them with jsonb_build_object, so the publisher must build the same object:

product (official.product)	key	product rows
exposures	dataset, expid	exposures
l2files	dataset, expid, sca, fid	l2files
diffimages	dataset, expid, sca, fid	diffimages
psfs	dataset, fid, sca	psfs
refimages	dataset, field, fid	refimages; children refimimages, refimcatalogs
sources (source set)	dataset, expid, sca, fid	sources
objects (object set)	dataset, survey, field	objects; child memberships
packets (packet set)	dataset, expid, sca, fid	packets; children forced, external
catalogs	name, version	catalogs

Product views join open official rows on (product, key, job). A set is identified by its job, so the set-child views (refimimages, refimcatalogs, sources, memberships, packets, forced, external) join through the set's open binding on (product, job): a set job must bind at most one key of its set product; the publisher should hold to that.

Bookkeeping.

• runs (id, who, created, why, revision, settings jsonb, survey text, dataset text), id an identity.
• jobs (id, run, stage, scope jsonb, state, image_digest, settings_hash, published_attempt, manifest jsonb, refusal, launched, started, ended, exit_code). state is one of launched (default), running, published, failed; a CHECK ties state = 'published' to published_attempt IS NOT NULL, so the publisher sets both in one update. refusal (text) is where the publisher records a refused automatic binding; the brief needed it on the job row and named no column.
• job_inputs (job, kind, product, key jsonb, source_job, binding), kind in input, expects, excludes. An object-set job's predecessor is its input row of product objects (source_job = the predecessor's job); members_official follows exactly that. expects: binding = the open binding id the job expects to replace, null for none. excludes: product sources, the source set's key, source_job = that source set's job.
• official (id, product, key, job, run, survey, dataset, opened_group, opened_by, opened_at, opened_why, closed_group, closed_by, closed_at, closed_why). A product binding has product, key, job and no run; the live binding has run, survey, dataset and no product, key or job (CHECK). A second CHECK makes the four closed_* fields all null or all set. Partial unique indexes: official_open_product_key (product, key) WHERE closed_at IS NULL AND job IS NOT NULL, official_open_live (survey, dataset) WHERE closed_at IS NULL AND run IS NOT NULL. opened_at defaults to now().
• change_groups: a bigint sequence for group ids (nextval('change_groups')), the smallest way to give each change group one new id.

Views. Each <table>_official is t.* plus binding (the open official.id) and run (from jobs). members_official is memberships.* plus set_job (the official object set the row belongs to), binding and run; job is the set in the chain that recorded the membership.

Scientific tables. Every row has job bigint NOT NULL REFERENCES jobs. Image row ids (rid, pid, rfid, psfid, rfcatid) are integer GENERATED BY DEFAULT AS IDENTITY, so the publisher may omit them or supply them. A source is (job, sid) with sid bigint assigned by the producer; an object is (job, aid) with aid bigint the position hash; a membership is (job, aid, source_job, sid); a packet is (job, source_job, sid); a forced row (job, forced_id); an external match (job, source_job, sid, catalog_job, catid).

Column choices that are mine

• l2filemeta folded into l2files (centre ra0, dec0 and corners ra1..dec4; its rid, fid, sca, hp6, hp9 are already there): one row per image, one binding, one view.
• refimmeta folded into refimages (nframes, cov5percent, the two columns the team reads), nullable because not every reference recipe computes them.
• dataset text, survey text (a survey such as a program title spans several program_id values; surveys stays dev's table keyed by program_id, and nothing references it yet).
• expid is the exposure's scientific id from its producer, no sequence (joins are on scientific keys).
• diffimages.rid, rfid, refimimages.rfid, rid, refimcatalogs.rfid, sources.pid, forced.rid, pid: kept as dev's lineage columns with no foreign keys; joins between products go by scientific key (Part 1).
• checksum kept on every file-bearing table where dev had it (immutable bytes).
• objects: aid, dataset, survey, field, ra0, dec0 (founding position) and the sums nsources, sumra, sumra2, sumdec, sumdec2, sumflux, sumflux2; dev's mean and stdev are derived from them, not stored.
• memberships (job, aid, source_job, sid); packets (job, dataset, expid, sca, fid, object_job, aid, source_job, sid, trigger_mjd, band, schema_version, filename, byte_offset, byte_length): offset and length renamed to avoid the SQL keyword.
• forced: dev's FORCED_TABLE_DTYPE columns (forced_id, aid, rid, pid, expid, sca, fid, band, mjdobs, ra, dec, x, y, psf_flux, psf_fluxerr, science_flux, science_fluxerr, flags, time_proc).
• external: the match envelope every dev match record opens with (catid, ra, dec, sep, pa) plus source_job, sid, catalog_job; Gaia's other fields come with the packet-stage port.
• catalogs (job, name, version, location): no dataset column, since its key is (name, version) per the brief.
• Filters, scas and surveys data inline in schema.sql (11 rows and a generate_series), so there is no second data file.
• Indexes: only the per-table job indexes, the view-join index official (product, job) WHERE closed_at IS NULL, the group indexes undo reads, the GIN index on overlapfields, and dev's field/mjd/jd indexes on the image tables. No grants or roles: SMDC's grants arrive with milestone 1.

Dev columns dropped, with the reason

• vbest, version, ppid everywhere: the binding, the job and the run replace them (Part 1).
• l2files: version, vbest as above; dev's other header columns (ctype*, cunit*, the SIP a_*/b_* terms, equinox, paobsy, pafpa, skymean) and l2filemeta's x, y, z, mjdobs: no team query reads them (LEDGER-queries.md (d)).
• exposures: field, hp6, hp9, fid, mjdobs, status, infobits, created and the ASDF observation columns: not read by any query.
• refimages: hp6, hp9, svid, avid, archivestatus; refimmeta except nframes, cov5percent; refimcatalogs: ppid, field, hp6, hp9, fid, svid, status, created, archivestatus, avid (its reference key is its reference's job).
• diffimages: hp6, hp9, svid, avid, archivestatus, nalertpackets.
• psfs: version, vbest.
• sources: npix, rb, nneg, nbad, sumrat, rblabel (stored but never read by name; rubrat and sumrat are left out until breadth, Part 2).
• diffimmeta, alertnames, socprocs, procreqs, archiveversions, dev's jobs: not in Part 1; jobs is the new one.

Divergences from PLAN.md, with the reason

• pipelines and swversions are not carried: the run's revision and settings replace them; a divergence from the plan's static-table list, made on purpose per the brief.
• skytiles, vskytiles, decbins are not created. The repository disagrees with the plan on a fact here, and the repository wins. Neither the branch's baseline migration nor dev's rapidOpsTables.sql creates them in PostgreSQL: dev kept them in a SQLite file (database/schema/roman_tessellation_nside512.txt on dev), and the branch computes the tessellation in closed form with no database access (database/modules/utils/roman_tessellation.py, RomanTessellationClosedForm; database/README.md says so). Loading 6,291,458 generated rows would need a load script, which this task forbids. Empty tables that nothing fills or reads would only cost something. If a later milestone wants them in SQL, they are one COPY from gen_rows() and gen_decbins().
• q3c indexes are on centres only (l2files, diffimages, objects, fields on ra0, dec0), not corners and not refimages: every team q3c query addresses ra0, dec0 (q3c_radial_query(ra0, dec0, ...), q3c_dist(ra0, dec0, ...) in dev's rapid_db.py and alerts/), and q3c indexes points, so corner indexes would not be used. Dev's refimages has no position columns, so there is nothing to index there. The block catches only feature_not_supported and undefined_file (q3c absent), so any other error in it still fails the apply.
• Set children join on (product, job), not (product, key, job): the brief says children carry their set's job and their own id, and a set's identity is its job. Above, under "Names task 2 codes against", is what the publisher must hold to.

Measurements

• Applied locally to PostgreSQL 18.6 (podman container on rr, no q3c): 22 tables, 15 views, one NOTICE for q3c.
• tests/db/test_schema.py locally: 5 passed, 1 skipped (the q3c test, not CI). With PGHOST unset: 5 errors "PGHOST is not set; database tests need a database", 1 skipped (the q3c test's own skip, which is about the environment and not about a missing database).
• tests/unit: 1735 passed.
• The other tests/db modules need the migration stream, whose baseline needs q3c, so they run only in db-migrations.yml.

Left for the next task

• Task 2 (operations): publish, change, undo, create_run, create_job, run_show, job_show against the names above; group ids from nextval('change_groups'); record a binding refusal in jobs.refusal; set state and published_attempt together; add test_publish.py and test_change.py to foundation.yml and to db-migrations.yml's --ignore list.
• tests/cli/conftest.py still skips without PGHOST; it goes with tests/cli in task 4.

🤖 Generated with Claude Code