Stop trusting the compromised eMudhra CA certificates

changelog.d/emudhra-ca-trust.fixed.md

@@ -0,0 +1 @@
+Stop trusting the compromised eMudhra (emSign) CA certificates, whose intermediate private keys were disclosed on 2026-10-20 (https://emudhra.we-have-a-problem.com). The lock moves certifi from 2026.5.20 to 2026.10.19, the first certifi release without emSign ECC Root CA - G3; nothing else in the lock moves, and `APPROVED_UV_LOCK_SHA256` is re-pinned for the new lock. The Modal stage image (`tools/modal_us_stage.py`) installs uv 0.13.4 instead of 0.11.7, because uv verifies HTTPS against a CA list compiled into it; requires certifi 2026.10.19 or later beside the runner's `huggingface_hub`; and upgrades `ca-certificates` to Debian's current release, which git and the stdlib's ssl module read.

docs/us-modal-stage-runbook.md

@@ -62,7 +62,11 @@ release directory onto the runs volume. Publication stays the human step in
    three times the list price (see "Preemption and restarts").
 2. **The image is the commit.** It starts from `debian_slim` with Python
    3.14, the minor version the local US builds record (`runtime.python`
-   3.14.4 in the #974 build manifest; Modal served 3.14.2). It adds git and uv 0.11.7, makes a shallow
+   3.14.4 in the #974 build manifest; Modal served 3.14.2). It adds git,
+   upgrades `ca-certificates` to Debian's current release, and installs
+   uv 0.13.4 and, for the runner, `huggingface_hub` with certifi 2026.10.19
+   or later: the first releases whose CA lists drop the compromised eMudhra
+   roots (https://emudhra.we-have-a-problem.com). It then makes a shallow
    clone of the plan's commit from GitHub, checks out the plan's branch name,
    and asserts `HEAD` equals the commit. It then runs
    `uv sync --all-packages --extra us --frozen` against that tree's own

packages/microcosm-build/src/microcosm/build/us_runtime/worker_identity.py

@@ -28,7 +28,7 @@
 PRIMARY_QRF_WORKER_MODULE = "microcosm.build.us_runtime.puf_qrf_worker"
 PRIMARY_QRF_INTERPRETER_PLACEHOLDER = "{python_interpreter}"
 APPROVED_UV_LOCK_SHA256 = (
-    "ceffa14029b2f4818fa1749684361a6de4d38358c27d3173cc2b0d47721ef3a4"
+    "dcb1bfb5bba39ec47cf52139e0260230bbadb892c950d1b9028eb29eda9c9dd0"
 )
 LEGACY_CAMPAIGN_UV_LOCK_SHA256 = (
     "27f47e385cfa35e2644a37410d1804b361ad9aee123577551c8421547bda65ee"

tools/modal_us_stage.py

@@ -85,13 +85,18 @@ def _local_plan() -> tuple[dict, plan_lib.Plan] | None:
 def _image() -> modal.Image:
     image = (
         modal.Image.debian_slim(python_version=plan_lib.IMAGE_PYTHON_VERSION)
-        .apt_install("git", "build-essential")
+        # Naming ca-certificates upgrades the base image's copy to the
+        # distribution's current release, which drops the compromised
+        # eMudhra roots; git's GitHub fetch and the stdlib's ssl module
+        # verify against that system CA store.
+        .apt_install("git", "build-essential", "ca-certificates")
         # The runner itself (not the build) runs on the image's interpreter;
         # it needs the Hub client to stage hf:// inputs. The build runs in
         # IMAGE_VENV, synced from the pinned tree's own lock.
         .pip_install(
             f"uv=={plan_lib.IMAGE_UV_VERSION}",
             f"huggingface_hub=={plan_lib.RUNNER_HF_HUB_VERSION}",
+            f"certifi>={plan_lib.RUNNER_CERTIFI_MINIMUM_VERSION}",
         )
         .env(
             {

tools/modal_us_stage_plan.py

@@ -70,8 +70,17 @@
 # too. compare-lineage checks the outputs the local run left (its first two
 # outer stages), not the stages after them.
 IMAGE_PYTHON_VERSION = "3.14"
-IMAGE_UV_VERSION = "0.11.7"
+# uv verifies HTTPS against a CA list compiled into it. 0.13.4 is the first
+# release whose list drops the compromised eMudhra (emSign) roots; earlier
+# releases trust certificates anyone can issue with the disclosed keys
+# (https://emudhra.we-have-a-problem.com).
+IMAGE_UV_VERSION = "0.13.4"
 RUNNER_HF_HUB_VERSION = "1.18.0"
+# The runner's Hub client verifies HTTPS against certifi's CA list, and
+# 2026.10.19 is the first certifi release without the eMudhra roots. Modal
+# reuses a cached layer while its definition is unchanged, so the floor is
+# stated rather than left to whatever certifi PyPI served at the first build.
+RUNNER_CERTIFI_MINIMUM_VERSION = "2026.10.19"
 WORK_ROOT = "/work"
 
 # Modal list prices for standard (non-sandbox) compute, read from