[anth-volk]

Fixes #1351

The incident

On 2026-10-20 it was disclosed that the private keys of eight eMudhra (emSign) intermediate CA certificates were generated so weakly that anyone can recover them from the public certificates (https://emudhra.we-have-a-problem.com). Anyone holding a key can issue a certificate for any domain, so a client that trusts emSign ECC Root CA - G3 (or Sectigo's cross-signature of the same intermediates) can have its HTTPS traffic intercepted. Browsers are protected; software that carries its own CA list, or reads a system CA store that has not been updated, is not.

What in this repository trusted the affected roots

1. uv.lock pinned certifi 2026.5.20, which still contains emSign ECC Root CA - G3. requests 2.34.2 and httpx 0.28.1 (used by huggingface_hub 1.18.0) verify against certifi, and the build tools use them to download Census, Hugging Face and other sources. Measured: requests with certifi 2026.5.20 got HTTP 200 from https://dv-g3b-emudhra.we-have-a-problem.com/; with certifi 2026.10.19 it raised CERTIFICATE_VERIFY_FAILED.
2. The Modal stage image (tools/modal_us_stage.py) installed uv 0.11.7. uv verifies HTTPS against a CA list compiled into it, and the image's uv sync --frozen downloads every locked wheel with it. Measured: uv 0.11.7 completed the TLS handshake with the test endpoint above (the server's plain-text response reached uv's index parser); uv 0.13.4 refused it with invalid peer certificate: UnknownIssuer. The lock's hashes still protected the wheel contents; the connection itself did not.
3. The same image's runner environment installed certifi without a version, as a dependency of huggingface_hub==1.18.0. Modal reuses a cached layer while its definition is unchanged, so the runner kept whichever certifi PyPI served when that layer was first built. The runner downloads hf:// inputs with it, sending HF_TOKEN when an input lives in a private or access-restricted Hub repository.
4. The image never refreshed Debian's ca-certificates package. git's fetch from GitHub and the stdlib ssl module (urllib) verify against that system store. Measured in debian:bookworm-slim: with ca-certificates 20230311+deb12u1, curl got HTTP 200 from both test endpoints; after apt-get install ca-certificates (which upgraded it to 20250419~deb12u2 from bookworm-security), curl refused both with unable to get local issuer certificate. debian:trixie-slim upgrades to 20250419+deb13u1. In both, emSign ECC Root CA - G3 is gone from /usr/share/ca-certificates/mozilla.

Checked and left unchanged:

• hf-xet 1.5.1 in the lock uses the system CA store (hf-xet 1.4.3 and later), not a compiled-in list. The lock has no grpcio.
• No code disables verification or sets a custom CA bundle (searched for verify=False, SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS, GRPC_DEFAULT_SSL_ROOTS_FILE_PATH, CERT_NONE and similar).
• CI: astral-sh/setup-uv@v6 has no version input and the repository declares no required-version, so it installs the newest uv; main's lint job on 2026-10-21 installed uv 0.13.4. actions/setup-node asks for the range "24", not an exact release, and runs only the Orrery parser contract test. The runners' system CA store is maintained by GitHub.
• docs/evidence/us-release-reimputation-1191/scripts/platform/Dockerfile (uv 0.9.26, python:3.14.2-slim-bookworm) is the committed record of a past offline replay. Its requirements are hash-pinned, nothing deploys it, and changing it would stop it matching the measurement it records.
• experiments/us-acs-local-l2-basis-20260928/modal_sweep.py is a dated experiment whose containers make no HTTPS requests of their own.

Changes

• uv.lock: certifi 2026.5.20 -> 2026.10.19. Nothing else in the lock moves.
• microcosm.build.us_runtime.worker_identity: APPROVED_UV_LOCK_SHA256 re-pinned to the new lock's digest (60ba4a85…), as every lock change requires.
• tools/modal_us_stage_plan.py: IMAGE_UV_VERSION 0.11.7 -> 0.13.4, and a new RUNNER_CERTIFI_MINIMUM_VERSION = "2026.10.19".
• tools/modal_us_stage.py: the image's apt_install also names ca-certificates, so the build upgrades it to Debian's current release; the runner's pip_install adds certifi>=2026.10.19.
• docs/us-modal-stage-runbook.md: the image description follows.
• changelog.d/emudhra-ca-trust.fixed.md.

Checks

• uvx uv@0.13.4 lock --check (the uv CI installs): passes. I generated the lock with uv 0.12.23 (uv lock --upgrade-package certifi), which changes only the certifi entry. uv 0.13.4's re-resolution writes the same certifi entry but also rewrites 49 dependency markers and drops the Windows wheels of the CUDA packages; uv 0.13.4 accepts the certifi-only lock as current, so I kept the smaller diff.
• uvx uv@0.13.4 sync --all-packages --locked: succeeds and installs certifi 2026.10.19, whose bundle contains neither emSign ECC Root CA - G3 nor AAA Certificate Services.
• ruff check and ruff format --check (ruff 0.15.16 from the lock) on the three changed Python files: pass.
• python3 tools/ci_test_plan.py verify: ok.
• pytest -n 4 --dist loadfile on test_us_stacked_spine.py, test_us_multispine_pool_h5_io.py, test_us_spine_blindness.py (the tests that use APPROVED_UV_LOCK_SHA256) and the three test_us_modal_stage_* modules: 1337 passed, 6 skipped.
• Test-endpoint probes for certifi, uv and Debian ca-certificates, as listed above.
• Not run: a Modal image build, which needs Modal credentials and a plan. The tests replace modal with a stub, so they do not exercise the image definition.

Operator actions after merge

• Modal stage runs: the image definition changed, so the next modal run tools/modal_us_stage.py from a checkout that includes this change rebuilds the base layers; no forced rebuild is needed. The runner code comes from the operator's checkout, but /opt/venv is synced from the plan commit's own uv.lock, so a plan pinned to a commit before this merge still installs certifi 2026.5.20 for the stage tool. Point new plans at a commit that includes this change.
• First Modal build: confirm that the image's ca-certificates is 20250419~deb12u2 (bookworm) or 20250419+deb13u1 (trixie). I could not check which Debian release and apt sources Modal's debian_slim uses for this workspace.
• Local and long-running build environments (developer checkouts, worktrees made by tools/route_a/route_a.sh): run uv sync at a commit that includes this change.

Environment-variable verification

This PR changes the Modal image definition, so here is every environment variable that code reads or declares:

Variable	Provider	How it reaches the code	Status
UV_PROJECT_ENVIRONMENT, UV_PYTHON_DOWNLOADS, UV_LINK_MODE, PYTHONDONTWRITEBYTECODE	Literal values in the image's .env({...}) (unchanged here)	Modal applies .env to the later run_commands build steps (uv sync) and to the running container	Verified in code
MICROCOSM_MODAL_PLAN	The operator's local shell	Read only while defining the app locally (modal.is_local())	Verified in code
MICROCOSM_MODAL_HF_SECRET	The operator's local shell; names a Modal secret	Attached with modal.Secret.from_name only when set	Unverified: whether the named Modal secret exists in the workspace
HF_TOKEN	The Modal secret named above (per the runbook)	Injected into the container by the secret attachment; read by huggingface_hub	Unverified
GIT_TERMINAL_PROMPT	Literal "0" in the runner	Passed to its git subprocesses	Verified in code

Per-plan tool variables come from the plan file (plan.env) and are not changed by this PR. The CI workflow is not changed.

🤖 Generated with Claude Code