Skip to content

Roll out eMudhra incident fixes (Go, base images, trust stores) #78

Closed

Description

@cbrgm

Why

eMudhra intermediates are compromised -> https://go.dev/blog/emudhra-incident. Everything that ships its own trust store keeps trusting them until its updated and redeployed:

  • Go binaries: verifier is statically linked, Go 1.27.3 / 1.26.10 block the intermediates in crypto/x509. Linux is the important case, there Go uses its own verifier, not the OS.
  • Base images: ca-certificates bundle baked into the image
  • Python certifi, Rust webpki-roots: ship their own root lists (emSign roots)
  • CI tooling / installers that bundle certs

What

Update every trust path we ship, then cut and roll out new images.

  • Every repo below is on Go >= 1.27.3 / 1.26.10, a current base image and trust stores without the compromised emSign ECC Root CA - G3 (other emSign roots are still in every current CA bundle, and that's fine)
  • New images are released and deployed

Where

Go toolchain

  • artifact-conduit: #522
  • dependency-controller: #204
  • quota-controller: #63
  • apiserver-kit: #151
  • solution-arsenal: #898
  • machine-controller-manager-provider-hcloud: #3 (1.26.10)
  • gardener-extension-provider-hcloud: #87 (1.26.10)
  • ocm-kit: #109, #112, #113 v0.2.0 released
  • krop-controller: #55

Base images

  • machine-controller-manager-provider-hcloud: #3 (distroless debian13)
  • gardener-extension-provider-hcloud: #87 (static-debian13), #88 (hcloud CSI / CCM images)

Python / Rust trust stores

  • artifact-conduit: #538 (certifi)
  • solution-arsenal: #906 (certifi)
  • knot: #43 (webpki-roots)

CI / tooling

Notes

  • List comes from PR search, not from checking each repo's go.mod, Dockerfiles or lockfiles, so repos without a fix PR may be missing. Since then every non-archived org repo (forks included) was swept for go.mod, Dockerfile FROMs, certifi and webpki-roots, and the CA bundles of the pinned base images were checked for G3. Beyond the out-of-scope list, only inactive forks are behind: ocm-community's ocm-kit (Go 1.26.4, base image with G3, inactive since September) and external-dns-management (Go 1.25.6, last build January). They only matter if something deployed still runs them. Not checked: GHCR packages, actual deployments, Node lockfiles

Activity

  1. added
    needs-triageIssue that has not been reviewed
    choreA routine task or common potentially re-occurring task
    on Oct 29, 2026
  2. changed the title [-]Roll out Go 1.27.3 / 1.26.10 for the eMudhra incident[/-] [+]Roll out eMudhra incident fixes (Go, base images, trust stores)[/+] on Oct 29, 2026
  3. added theissue type on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

choreA routine task or common potentially re-occurring task

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions