[juaristi22]

Microcosm UK publishes each dataset as a release line (PolicyEngine/microcosm#823). This PR lets the Calibration fit page show every published cut of a line, including the local-area dense release. It also makes the UK page open on the current national cut instead of the June 2023 Populace build. Finally, it lets the Staging candidates page show the per-target fit of UK local-area candidates again.

Problem

A Microcosm line keeps one release id and one folder, releases/<release id>/. Each cut is an immutable Hugging Face tag, <release id>-<YYYYMMDDTHHMMSSZ>-<attempt>. The national line's current cut is named by latest-national.json. The repository-wide latest.json stays on the June 2023 build.

The publisher assumed that a release id has one commit and that a tag's files sit under releases/<tag>/. It also read only latest.json. As a result:

• microcosm-uk-2024-25-national is still stored as the 4 Oct cut. Each later cut changed the folder's commit, so UK release reconciliation now stops with "Released dataset … changed".
• Each cut tag was reported as "Release has no calibration diagnostics". This includes microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe, which exists only as a tag.
• The UK default stays on the June 2023 build.

Changes

• A cut tag is its own release build. Its manifest entry uses the tag as releaseId and source ID. A new optional field, releaseDirectory, records the line folder. Readers fetch releases/<releaseDirectory>/ at the entry's commit. Entries without the field read releases/<releaseId>/ as before.
• Cut verification. A tag counts as a cut only when releases/<release id>/release_manifest.json at that tag names it in build.cut_tag. Any other cut-shaped tag is reported ineligible with a reason, and the run continues.
• Line folders. A line folder on main is not published again once its cuts are. A build already published from the folder stays pinned at its own commit, so it still audits and repairs. Tags that the manifest already pins at the same commit are not probed again.
• UK default. The country registration gains pointer_path, which is latest-national.json for the UK. The publisher reads it and checks it: schema_version 1, a line that matches the file name, and a revision that is a cut of release_id. That revision becomes the latest release.
• Release picker. /api/microcosm/releases adds every published cut beside the folders on main. It labels each cut by line and attempt, for example 2026-10-19 08:33Z · 2024-25-national · 6cba5d71. It hides a line folder unless the latest release still points at it. Each cut's role comes from its own release manifest at its commit, so the dense release keeps the existing "local area · experimental" flag.
• Pinned source links. The summary's source links and the target investigation packet cite the line folder, at the cut's commit.
• Upload injection. publishRelease takes an options object, and the manifest store can inject the bundle upload, so the reconciliation test runs without a dry run.
• Staging page: local candidates. The staged-bundle reader looked only for calibration_diagnostics.json. Microcosm's UK local lane stages its outputs under the dataset's own stem, so its diagnostics arrive as microcosm_uk_2024_25_local.diagnostics.json. Every local candidate since early October therefore showed "has not uploaded calibration diagnostics yet", and staging reconciliation reported them ineligible. The reader now uses calibration_diagnostics.json when the receipt records it. Otherwise it uses the one <dataset>.diagnostics.json that the receipt records. If the receipt records several stem files, or a name that contains a path, the run has no target view. The digest check is unchanged.

US and Belgium have no cut tags and no pointer_path. Their inventory, picker, labels and latest selection are unchanged by construction, and tests check this. The manifest change is one optional entry field. The current reader already accepts and keeps unknown optional fields.

Rollout

The old frontend would read a cut entry from releases/<tag>/ and get a 404. Publishing therefore pauses until the new frontend is deployed:

1. Before merging, set CALIBRATION_TREE_BUILD_ENABLED=FALSE.
2. Merge, then wait for Deploy production to succeed.
3. Re-enable CALIBRATION_TREE_BUILD_ENABLED.
4. Dispatch Publish calibration tree one run at a time, because the shared slot cancels pending runs:
   • UK manual, dry run. Expect:
     • the national cuts and the dense cut published;
     • the microcosm-uk-2024-25-national folder build complete;
     • latest moves to the cut that latest-national.json names.
   • UK manual, real run.
   • UK staging, to catch up on webhooks skipped while publishing was paused.
   • US and BE manual dry runs, expecting every build complete.
5. Check that the UK picker lists the cuts and dense, and that the UK default is the current national cut.

The first UK run builds the dense bundle. Locally, from the published files, it took about 2.5 minutes and produced 42 files. The largest file is 16.2 MB raw, against the 100 MB limit. Building the summary for that release took 0.3 seconds.

Related

• The 1 Jan 1970 date on the stored national entry came from builds published before Date a release from its history, not its id #200. The migrate-timestamps run from Supply and qualify private release credentials before deployment #211 corrected it on 19 Oct, so it now shows 4 Oct 2026. New cut entries are dated from their tag's commit.
• @anth-volk: Extract the calibration tree builder into a self-contained workspace package #220 moves most of these files into frontend/packages/calibration-tree/src/. Whichever PR merges second will need a rebase; git follows the renames.

Validation

• cd frontend && bun test: 617 pass. New tests:
  • an end-to-end UK reconciliation with a legacy folder entry, two cut tags (one of them tag-only), a stray cut-shaped tag and the line pointer;
  • the pointer's checks;
  • the manifest field;
  • the release locator;
  • inventory merging and cut roles;
  • the investigation packet;
  • the releases route;
  • labels;
  • the staged diagnostics file rule.
• bun run lint (tsc --noEmit) is clean.
• bun run build passes.
• I ran the staged-bundle reader locally against three real UK runs. The 19 Oct local candidate now loads from its stem file. The 19 Sep local candidate and the 19 Oct national attempt load from the same files as before.
• The hosted Python · dependencies · source APIs check fails here, and it would fail on main too. backend/requirements.lock pins policyengine-us==1.764.6, which has been deleted; Pin .py 6.4.1 instead of the deleted policyengine-us 1.764.6 #222 replaces the pin. This PR changes no Python.

Review fixes (added by @anth-volk, with Claude Code)

Merged main into this branch (649a299, one import-list conflict in latest-artifact.test.ts: both sides kept). That fixes the hosted Python check.

Four fixes for bugs a Codex review of this PR found, one commit each. Each red count is from the new test(s) before the fix; each green count is the whole test file afterwards, rerun after the commit.

1. --latest checks the pointer's tag against its release manifest (8a01c06). Reconciliation publishes a tag only when releases/<release id>/release_manifest.json at that tag names it in build.cut_tag. --latest skipped that check: given a latest-national.json whose revision is a tag whose manifest names another build, it published that tag and promoted it. readUpstreamLatest now applies the same check before any mode publishes or promotes from the pointer, and requires the tag's files to be under the folder named by the pointer's release_id. Reconciliation passes its manifest, so a tag already pinned there at the same commit is not probed again.
   • Red: 1 of 12 failed. readUpstreamLatest and runPublisher({ country: "uk", mode: "latest" }) accepted the tag.
   • Green: 12 of 12.
2. The national pointer file is checked even when it names no revision (5fea679). A country whose registration names its own pointer file (the UK's pointer_path) now always gets the schema_version, line and tag checks, and a file without revision is refused. Countries that use latest.json still read release_id alone. The reconciliation metadata test used a UK pointer without a revision, so it now names a published tag of the same folder.
   • Red: 2 of 14 failed ({"schema_version":99,"line":"dense",...} and a pointer with no revision were both accepted).
   • Green: 14 of 14.
3. A published tag's comparison metadata names the tag (3614ca7). publishRelease copied the diagnostics' release_id, which is the folder name, into index.targetComparison.releaseId, so two builds of one release compared as the same release. A tag published from its folder now records the tag there, as index.build.releaseId already did. A folder release keeps the id its diagnostics carry, so its bytes are unchanged.
   • Red: 1 of 15 failed (comparison metadata named microcosm-uk-2024-25-national and microcosm-uk-2024-25-dense).
   • Green: 15 of 15.
4. A staged receipt is refused when one of its stem names contains a path (1079fa9). stagedBundleDiagnosticsFile dropped such names before counting, so {local.diagnostics.json, nested/dense.diagnostics.json} returned the first file. The count now includes every stem name, and the single stem file must also be one path segment.
   • Red: 1 of 30 failed.
   • Green: 30 of 30.

Typecheck (tsc --noEmit) is clean after each commit. Only the changed test files were run locally; CI runs the rest.

Tree publishing is paused (CALIBRATION_TREE_BUILD_ENABLED=FALSE, set 2026-10-19) until this stack is deployed, so rollout step 1 is already done.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 19, 2026 9:06pm UTC