Skip to content

Stop trusting the compromised eMudhra CA certificates - #697

Merged
anth-volk merged 2 commits into
mainfrom
emudhra-ca-trust-update
Oct 22, 2026
Merged

anth-volk merged 2 commits into
mainfrom
emudhra-ca-trust-update

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #698

Incident

On 2026-10-20 it was disclosed that eight eMudhra (emSign) intermediate CA certificates have private keys below 256, so anyone can recover them from the public certificates and issue a certificate for any domain. The TLS intermediates chain to the root "emSign ECC Root CA - G3" and were also cross-signed under Sectigo's "AAA Certificate Services". Browsers and current operating systems already reject them; software that carries its own CA list is still exposed until that list is updated. Details: https://emudhra.we-have-a-problem.com

What in this repository trusted the affected roots

Component Where it is pinned How it is used How I established it
certifi 2025.8.3 uv.lock certify-uk-release.yaml installs the lock with uv sync --frozen --extra uk --extra dev in the prepare and qualify jobs (scheduled every two hours), which send HUGGING_FACE_TOKEN to Hugging Face. The docs targets in the Makefile (uv run) and local uv sync also install from the lock. With requests 2.32.5 and certifi 2025.8.3, https://dv-g3b-emudhra.we-have-a-problem.com/ returned HTTP 200. With certifi 2026.10.19 it failed with CERTIFICATE_VERIFY_FAILED.
hf-xet 1.1.9 uv.lock (dependency of huggingface-hub 0.34.4) Same jobs, for Hugging Face downloads from Xet storage. The 1.1.9 manylinux x86_64 binary contains the subject string "emSign ECC Root CA - G3" (a compiled-in root list), so it trusts the root whatever certifi says. The 1.7.0 binary contains no compiled-in roots and uses rustls-platform-verifier, which reads the operating-system CA store.
uv 0.12.14 the setup-uv step of every workflow Every install and lock step in CI. uv uses its own compiled-in CA list by default. uv pip install --index-url https://dv-g3b-emudhra.we-have-a-problem.com/simple/ ... with uv 0.12.14 completed the TLS handshake and failed only on the response's content type. With uv 0.13.4 it failed with invalid peer certificate: UnknownIssuer.

Not changed, and why:

  • The published package: its dependency ranges (for example requests>=2.31.0) are unchanged, and users resolve their own certifi.
  • The uv pip install steps in every workflow do not read uv.lock; they resolve from PyPI and already get certifi 2026.10.19 and hf-xet 1.7.0.
  • pypa/gh-action-pypi-publish@release/v1 already pins certifi 2026.10.19 (Upgrade certifi and sigstore pypa/gh-action-pypi-publish#427, merged 2026-10-20).
  • The Docs job in push.yaml sets up Node.js 18.x, which has no fixed release, but make docs runs only quarto render and nothing in the repository runs Node.js. Left as it is.
  • The repository has no Dockerfile, Modal app, Vercel project or other deployment configuration, and no code that sets a custom CA bundle or disables certificate verification.

Changes

  • uv.lock: uv lock --upgrade-package certifi --upgrade-package hf-xet with uv 0.13.4. certifi 2025.8.3 → 2026.10.19, hf-xet 1.1.9 → 1.7.0, and the lock's revision 3 → 5 (written by uv 0.13.4). No other package changed.
  • .github/workflows/{push,pr_code_changes,pr_docs_changes}.yaml and UV_VERSION in certify-uk-release.yaml: uv 0.12.14 → 0.13.4. docs/release-bundles.md requires the uv pin and the lock to change together in one reviewed PR, because the lock's marker spelling depends on the uv version.
  • changelog.d/698.fixed.md: changelog fragment.

Checks

  • uv lock --locked with uv 0.13.4: passes. uv 0.12.14 also reads the new lock (uv lock --locked passes), so local installs with the older uv still work.
  • python scripts/release_lock.py with uv 0.13.4 on PATH (the reviewed-lock check run by BundleVerification, Versioning and Publish): passes.
  • Versioning simulation: in a copy with the version raised to 6.10.1, python scripts/release_lock.py --refresh with uv 0.13.4 changed only the root package's version in uv.lock, as the release requires.
  • pytest tests/test_release_lock.py with uv 0.13.4 on PATH: 25 passed.
  • uv sync --frozen --extra uk --extra dev --python 3.13 --dry-run with uv 0.13.4: plans certifi 2026.10.19 and hf-xet 1.7.0.
  • huggingface-hub 0.34.4 with hf-xet 1.7.0: downloaded model.safetensors from hf-internal-testing/tiny-random-gpt2 through Xet storage.
  • certifi 2026.10.19 and uv 0.13.4 still connect to https://pypi.org/ and https://huggingface.co/.
  • towncrier check, ruff format --check ., ruff check .: pass. All workflow files parse as YAML.
  • The full test suite was not run locally; CI runs it.

After merge

  • As for every merge to main, push.yaml publishes a patch release (6.10.1) and dispatches version-bump PRs to consumer repositories. The wheel's contents differ only in the version and the bundle metadata that records it.
  • No deployment needs rebuilding: this repository publishes a library and deploys nothing.
  • Developers with an environment installed from the old lock should run uv sync to install certifi 2026.10.19 and hf-xet 1.7.0.
  • No UK candidate PR is open now. Any candidate branch created before the merge carries the old lock; promotion fast-forwards main only to a candidate built on the current main, so such a candidate is rebuilt from the new lock.

Environment-variable verification

This PR changes CI workflows, so I checked every environment variable the changed code reads or declares:

  • UV_VERSION (changed value only): declared as a literal in the workflow-level env: block of certify-uk-release.yaml. Every job inherits it, and the setup-uv steps of the resolve, prepare, qualify, replay, catch-up, accept and promote jobs read it through ${{ env.UV_VERSION }}, as before. Verified.
  • The other changed lines are the version: input of setup-uv, not environment variables. setup-uv downloads uv using its github-token input, which defaults to the automatic github.token; unchanged.
  • No secret, env: binding or job permission changed. The steps that install from the lock still receive HUGGING_FACE_TOKEN from the GitHub Actions secret, unchanged.

The repository's PR guidance asks for a tracking issue and a Fixes # first line. This PR came from an organization-wide check for the eMudhra certificates, and no issue was opened for it; add one if you want it tracked.

🤖 Generated with Claude Code

Eight eMudhra (emSign) intermediate CA certificates have private keys
that can be recovered from the public certificates. Software that trusts
the root "emSign ECC Root CA - G3" accepts certificates issued with them.

uv.lock pinned certifi 2025.8.3, which includes that root, and hf-xet
1.1.9, whose compiled-in CA list also includes it. Upgrade only those
two packages: certifi 2026.10.19 removes the root, and hf-xet 1.7.0
verifies against the operating-system CA store. The certification
workflow installs this lock with `uv sync --frozen`.

Every workflow pinned uv 0.12.14, whose compiled-in CA list includes the
root. Pin uv 0.13.4, which removes it, and regenerate the lock with that
uv, as docs/release-bundles.md requires when the uv pin changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review October 22, 2026 17:24
@anth-volk
anth-volk merged commit 8d76fff into main Oct 22, 2026
13 checks passed
@anth-volk
anth-volk deleted the emudhra-ca-trust-update branch October 22, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop trusting the compromised eMudhra CA certificates

1 participant