Project Orrery artifacts without the content store, and write the execution overlay as its own file

changelog.d/1294-execution-overlay-file.added.md

@@ -0,0 +1 @@
+Add `microcosm.graph.execution_overlay.execution_overlay_bytes` and `load_execution_overlay`, which write and read the execution overlay (`microcosm.graph.execution-evidence.v1`) as its own canonical JSON file. Projecting a schema with the overlay read back from that file gives byte-identical Orrery JSON, and the reader needs only the standard library. It refuses non-regular files, files over the schema's 32 MiB serialized limit, duplicate keys, non-finite numbers and other protocols.

changelog.d/1294-orrery-projection-without-store.changed.md

@@ -0,0 +1 @@
+Load `microcosm.graph`'s runtime exports on first access, and take the execution overlay's protocol name and the SHA-256 helper from modules that need only the standard library. `microcosm.graph.orrery` now imports and projects a saved schema and execution overlay in a process without numpy, pandas, microcosm-frame or the content store; the package's declared dependencies are unchanged.

changelog.d/1294-uk-execution-overlay-output.added.md

@@ -0,0 +1 @@
+UK finalize now writes the execution overlay it projects as `graph.execution.json`, beside `graph.orrery.json`, and registers it as a build output: in the dataset manifest, in a refused build's evidence manifest and in a failed build's bundle. Projecting `graph.schema.json` with it reproduces `graph.orrery.json` byte for byte. It stays out of the Orrery publication set (`orrery-publication/` and `orrery.upload.json`), and every existing file keeps its bytes, apart from the new entry in the manifest that registers it.

docs/orrery-adapter.md

@@ -24,6 +24,14 @@ Microcosm executes. The direct API performs steps 2 and 3 in memory. A saved
 schema lets another process repeat step 3 without importing the original graph
 construction code.
 
+That process needs none of the build runtime either. `microcosm.graph.orrery`
+and the modules it imports use only the standard library, and `microcosm.graph`
+loads its runtime exports (the executor, content store, kernels, codecs and
+evidence capture, which need numpy, pandas and microcosm-frame) on first
+access. A test runs the projection, with an execution overlay, and the command
+in an interpreter that refuses to import numpy, pandas, microcosm-frame or the
+store. The package's declared dependencies are unchanged.
+
 A kernel is the implementation of one graph operation, with a declared input,
 output, numerical and execution contract.
 
@@ -243,6 +251,39 @@ npx orrery --input build/graph.orrery.json --output build/graph.orrery.html
 Native evidence links and saved-schema relative references are rebased to the
 command's output directory; keep the referenced files available with the HTML.
 
+`collect_execution_evidence` returns the execution overlay: the
+`microcosm.graph.execution-evidence.v1` object that the projection embeds at
+`metadata.execution`. `microcosm.graph.execution_overlay` saves it as its own
+file, so a later projection needs only the schema and that file, not the store:
+
+```python
+from microcosm.graph.execution_overlay import (
+    execution_overlay_bytes,
+    load_execution_overlay,
+)
+
+Path("build/graph.execution.json").write_bytes(execution_overlay_bytes(execution))
+
+# Later, in a process without the store, numpy, pandas or microcosm-frame:
+schema = json.loads(Path("build/graph.schema.json").read_bytes())
+overlay = load_execution_overlay(Path("build/graph.execution.json"))
+Path("build/graph.orrery.json").write_text(
+    orrery_json_from_schema(schema, execution=overlay), encoding="utf-8"
+)
+```
+
+The file is the overlay's canonical JSON: UTF-8, sorted keys and no
+whitespace. It has none of the transport tags for numbers JavaScript cannot
+represent; the projection adds those. Reading it back gives an equal object,
+so the projection is byte-identical to projecting the in-memory overlay. The
+reader takes only a regular file within the schema's 32 MiB serialized limit,
+which evidence collection and projection already apply to the overlay. It
+refuses duplicate keys, non-finite numbers, values outside the schema's JSON
+bounds and any other protocol. The overlay's graph binding and records are
+checked when it is projected. Its native-evidence links are relative to the
+directory it was collected for (`reference_base`), so project into that
+directory, beside the linked files.
+
 Other countries supply the same three inputs: a presentation dictionary (groups,
 scope, optional labels and references), an artifact-summary registry keyed by
 artifact type, and driver wiring that calls `save_graph_schema`,
@@ -256,6 +297,11 @@ that case, pass the attempt's `graph.schema.json` and
 `graph-evidence/execution.evidence.json`, and the same store. The spine sidecar
 identifies its native index under the checkpoint directory; use that directory's
 `graph.schema.json` and `node-graph` store. Orrery JSON and HTML remain opt-in.
+When UK finalize exports `graph.orrery.json` from recorded execution, it also
+writes the overlay it projected as `graph.execution.json` and registers it as a
+build output, so `graph.schema.json` and `graph.execution.json` reproduce the
+export byte for byte without the store. The overlay is not part of the Orrery
+publication set (see [UK graph publication](orrery-publication.md)).
 
 The final national schema, declaration and inventory all include the appended
 `uk.full.national.readback` operation. Earlier endpoint runs remain separate

docs/orrery-publication.md

@@ -6,6 +6,14 @@ public-safe file inventory in the attempt directory, attempt publication before
 Hugging Face artifact staging, and register graph/evidence files and an
 `orrery.publication.json` receipt snapshot in the artifact manifest.
 
+When execution was recorded, they also write the execution overlay that
+`graph.orrery.json` embeds as `graph.execution.json` and register it in the
+same manifest. A failed build's bundle carries it too. It is not copied to the
+preserved `orrery-publication/` directory or listed in `orrery.upload.json`:
+the emitter's file roles and the runs service's contract do not admit it yet.
+Its content is already inside `graph.orrery.json`, so registering it discloses
+nothing new.
+
 Microcosm's metadata adapter starts one local emitter service with two independent
 workers: live telemetry and completed graph publication. The `microcosm-emitter` package
 owns sockets, authentication, persistence, retries and resource sampling.

packages/microcosm-build/src/microcosm/build/uk_runtime/orrery_publication.py

@@ -26,11 +26,13 @@
 from microcosm.graph import ContentStore
 from microcosm.graph.canonical import canonical_json
 from microcosm.graph.evidence import collect_execution_evidence, load_run_evidence
+from microcosm.graph.execution_overlay import execution_overlay_bytes
 from microcosm.graph.orrery import orrery_json_from_schema
 
 RECEIPT_NAME = "orrery.publication.json"
 INVENTORY_NAME = "orrery.upload.json"
 EVIDENCE_MANIFEST_NAME = "orrery.evidence-manifest.json"
+EXECUTION_OVERLAY_NAME = "graph.execution.json"
 FAILURE_MARKER_NAME = "orrery.failure.json"
 FAILURE_BUNDLE_DIRECTORY = "orrery-failure"
 # How long a build waits for a first upload result after the durable enqueue.
@@ -141,6 +143,18 @@ def finalize_graph(
                         roles[run[field]["path"]] = role
             payload = orrery_json_from_schema(schema, execution=execution).encode()
             materialize_bytes(payload, output / "graph.orrery.json")
+            # The overlay is the projection's only input besides the schema,
+            # so a later projection can start from these two files. It is a
+            # registered build output but not part of the Orrery publication:
+            # the emitter's file roles and the runs service's contract do not
+            # admit it yet.
+            unpublished: list[str] = []
+            if execution is not None:
+                materialize_bytes(
+                    execution_overlay_bytes(execution),
+                    output / EXECUTION_OVERLAY_NAME,
+                )
+                unpublished.append(EXECUTION_OVERLAY_NAME)
             for url in _urls(json.loads(payload)):
                 name = url.split("#", 1)[0]
                 if re.fullmatch(r"upstream-[a-f0-9]{64}\.json", name):
@@ -149,7 +163,7 @@ def finalize_graph(
                     roles[name] = "upstream"
             # Keep exact sources outside the build's temporary directory and
             # outside HF's immutable receipt snapshot, including when offline.
-            names = list(roles)
+            names = [*roles, *unpublished]
             preserved = Path(args.attempt_evidence) / "orrery-publication"
             for name in roles:
                 materialize_bytes((output / name).read_bytes(), preserved / name)
@@ -169,7 +183,7 @@ def finalize_graph(
                 raise RuntimeError("Local graph inventory service is unavailable.")
             materialize_bytes(canonical_json(inventory), preserved / INVENTORY_NAME)
             materialize_bytes(canonical_json(inventory), output / INVENTORY_NAME)
-            names = [*roles, INVENTORY_NAME]
+            names = [*roles, *unpublished, INVENTORY_NAME]
             receipt = {
                 "version": 1,
                 "publication_id": inventory["publication_id"],

packages/microcosm-build/tests/engine_free/uk/test_uk_orrery_publication.py

@@ -1,19 +1,36 @@
 """UK exports and staging order, independent of the runs application's code."""
 
+import hashlib
 import json
 from argparse import Namespace
+from pathlib import Path
 
 import pytest
 
+import test_support.microcosm_build.uk_full_build_cli as full_build_support
 from microcosm.build.uk_runtime import (
     full_build_cli,
     orrery_publication,
     rowwise_staging,
 )
 from microcosm.build.uk_runtime.orrery_contract import save_uk_graph_schema
-from microcosm.graph import Graph, Node, Owned, SourceRef, StructuralDelta
+from microcosm.graph import (
+    ContentStore,
+    Graph,
+    Node,
+    Owned,
+    SourceRef,
+    StructuralDelta,
+    collect_execution_evidence,
+    load_run_evidence,
+)
+from microcosm.graph.execution_overlay import load_execution_overlay
+from microcosm.graph.orrery import orrery_json_from_schema
 from test_support.microcosm_build.telemetry import fake_publication_inventory
 
+#: The execution overlay's file name, which the collector's inputs will name.
+OVERLAY_NAME = "graph.execution.json"
+
 
 def fixture(tmp_path, **options):
     out = tmp_path / "temporary"
@@ -323,3 +340,110 @@ def publish_graph(self, directory, inventory, *, wait_seconds):
     assert (args.out / "graph.orrery.json").is_file()
     assert (args.out / "orrery.upload.json").is_file()
     assert (args.out / "orrery.evidence-manifest.json").is_file()
+
+
+def synthetic_build_request(tmp_path, monkeypatch, failure=None):
+    """A synthetic dense build request whose run records execution."""
+    pytest.importorskip("tables")
+    full_build_support.patch_certification(monkeypatch)
+    args = full_build_support.arguments(tmp_path)
+    return args, full_build_support.prepared(tmp_path, failure)
+
+
+def synthetic_full_build(tmp_path, monkeypatch, failure=None):
+    """Run a synthetic dense build; return its arguments and exit status."""
+    args, build = synthetic_build_request(tmp_path, monkeypatch, failure)
+    return args, full_build_support.cli.execute_full_build(build, args)
+
+
+def test_execution_overlay_reprojects_to_the_exported_orrery_artifact(
+    tmp_path, monkeypatch
+):
+    args, status = synthetic_full_build(tmp_path, monkeypatch)
+    assert status == 0
+    schema = json.loads((args.out / "graph.schema.json").read_bytes())
+    store = ContentStore(args.out / ".graph-store", create=False)
+    collected = collect_execution_evidence(
+        schema,
+        runs=load_run_evidence(args.out / "execution.evidence.json", store=store),
+        store=store,
+    )
+
+    overlay = load_execution_overlay(args.out / OVERLAY_NAME)
+
+    assert overlay == collected
+    assert (
+        orrery_json_from_schema(schema, execution=overlay).encode()
+        == (args.out / "graph.orrery.json").read_bytes()
+    )
+
+
+def test_execution_overlay_is_registered_but_not_published(tmp_path, monkeypatch):
+    args, status = synthetic_full_build(tmp_path, monkeypatch)
+    assert status == 0
+    overlay_path = args.out / OVERLAY_NAME
+    manifest = json.loads(
+        (args.out / full_build_support.cli.MANIFEST_FILENAME).read_text()
+    )
+    registered = manifest["outputs"][f"orrery/{OVERLAY_NAME}"]
+    assert Path(registered["path"]) == overlay_path
+    assert registered["sha256"] == hashlib.sha256(overlay_path.read_bytes()).hexdigest()
+    assert registered["bytes"] == overlay_path.stat().st_size
+    preserved = args.attempt_evidence / "orrery-publication"
+    inventory = json.loads((preserved / "orrery.upload.json").read_bytes())
+    assert (preserved / "graph.orrery.json").is_file()
+    assert not (preserved / OVERLAY_NAME).exists()
+    assert OVERLAY_NAME not in {entry["name"] for entry in inventory["files"]}
+
+
+def test_refused_build_registers_its_execution_overlay_in_its_evidence_record(
+    tmp_path, monkeypatch
+):
+    args, status = synthetic_full_build(
+        tmp_path, monkeypatch, "uk_local_geography_ladder_post_calibration"
+    )
+    assert status == 1
+    evidence = json.loads(
+        (args.out / orrery_publication.EVIDENCE_MANIFEST_NAME).read_bytes()
+    )
+    overlay_path = args.out / OVERLAY_NAME
+    assert evidence["orrery_publication"]["recorded_execution"] is True
+    assert evidence["outputs"][OVERLAY_NAME]["sha256"] == (
+        hashlib.sha256(overlay_path.read_bytes()).hexdigest()
+    )
+
+
+def test_failed_build_keeps_its_execution_overlay_in_the_failure_bundle(
+    tmp_path, monkeypatch
+):
+    args, build = synthetic_build_request(tmp_path, monkeypatch)
+
+    def refuse_manifest(*arguments, **options):
+        raise RuntimeError("manifest projection failed")
+
+    monkeypatch.setattr(
+        full_build_support.cli, "rowwise_candidate_manifest_from_graph", refuse_manifest
+    )
+    with pytest.raises(RuntimeError, match="manifest projection failed"):
+        full_build_support.cli.execute_full_build(build, args)
+    bundle = args.attempt_evidence / orrery_publication.FAILURE_BUNDLE_DIRECTORY
+    evidence = json.loads(
+        (bundle / orrery_publication.EVIDENCE_MANIFEST_NAME).read_bytes()
+    )
+    assert evidence["orrery_publication"]["recorded_execution"] is True
+    assert Path(evidence["outputs"][OVERLAY_NAME]["path"]) == bundle / OVERLAY_NAME
+    assert load_execution_overlay(bundle / OVERLAY_NAME)["phases"]
+    assert not (args.attempt_evidence / "orrery-publication" / OVERLAY_NAME).exists()
+
+
+def test_build_without_recorded_execution_writes_no_execution_overlay(
+    tmp_path, monkeypatch
+):
+    args = fixture(tmp_path)
+    monkeypatch.setattr(rowwise_staging, "_ACTIVE_EMITTER", None)
+    manifest = {"outputs": {}}
+    result = orrery_publication.finalize_graph(args, manifest=manifest)
+    assert result.receipt["recorded_execution"] is False
+    assert not (args.out / OVERLAY_NAME).exists()
+    assert OVERLAY_NAME not in result.files
+    assert f"orrery/{OVERLAY_NAME}" not in manifest["outputs"]