Repository navigation
security: per deployment EL/CL JWT - #520
Merged
Merged
Conversation
jwt/jwt.hex was tracked, so every default deployment shared one Engine API secret. A one-shot jwt-init service now writes a random secret before the EL and CL start, never replacing an existing one, and the file is untracked. A label on each EL and CL service makes the next `docker compose up -d` recreate both so they load the new secret together.
The generated secret was a world-readable file in the repo directory. jwt-init now writes it to a jwt volume that only the execution and consensus clients mount, read-only. An operator-provided jwt/jwt.hex is loaded instead, the old public secret is ignored, and an invalid file stops the clients from starting. The mount change recreates both clients on upgrade, so the label is gone.
The README covered what the secret is, upgrading, a refused git pull, using your own secret and printing it in one paragraph. Each is now its own point.
|
aly-obol
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
jwt/jwt.hexwas tracked in this repo, so every deployment used the same, public Engine API secret. The Engine API port (8551) is published on127.0.0.1by default and is reachable by every container on the compose network, so anyone who can reach it could drive the execution client.Change
jwt-initservice (busybox, no network) generates 32 random bytes from/dev/urandomas 64 hex characters, with no0xand no newline, into ajwtDocker volume. It writes to a temporary file and renames it, and never replaces an existing secret.nethermind,lighthouse,el-nethermind,el-reth,cl-grandine,cl-lighthouse,cl-teku,cl-lodestar,cl-prysm,cl-nimbus) wait for it (depends_on: service_completed_successfully) and mount the volume read-only at the path they used before../jwt, so no host user can read it without root. Inside the volume it is0444 root:root: Teku and Nimbus run as UID 1000 and the other clients as root, so no single owner or mode below world-readable works without changing users, and only these clients mount the volume.jwt/jwt.hex(for an execution client outside the stack) is loaded into the volume instead. It must be one line of 64 hex characters, otherwisejwt-initfails and the clients do not start. The old public secret is ignored if it is left there.jwt/jwt.hexis deleted from git and ignored. README and CLAUDE.md updated.Upgrade behaviour
git pullanddocker compose up -d, as documented. The changed mount makes compose recreate the execution and consensus clients, and both start with the new secret. The secret is rotated on purpose.git pullanddocker compose up -dthe old containers keep running on the old secret, which they hold in memory. If one of them restarts in that window it finds no file: Nethermind, Reth and Lighthouse then write their own, the others exit. The pair recovers ondocker compose up -d, and a file written this way is loaded like an operator-provided one.git pullrefuses to run ifjwt/jwt.hexwas changed locally. The README says what to do.docker compose run --rm jwt-init cat /jwt/jwt.hex.Testing
Done, without Docker:
jwt-init, mounts thejwtvolume read-only at the directory its flag points to, and that nothing else mounts./jwt..github/scripts/check_exposure.pypasses.jwt-initscript was run underdashwith GNU tools for: first run, re-run, stale temporary file, own secret (plain,0xand newline, mode 0600), old public secret, and invalid files (short, CRLF, two lines, empty).git pullover a clean and a locally changedjwt/jwt.hexwas simulated with a toy repo.Not done:
ashand its applets (od -An -tx1 -v -N32,grep -x,cmp -s), that compose recreates both clients on upgrade, that each client reads the secret from the volume, Docker Desktop, rootless Docker and podman.Manual test checklist
On a Linux Docker host, with
.envfrom.env.sample.hoodi:docker compose up -d.docker compose logs jwt-initsaysGenerated a new Engine API secret,docker compose ps -a jwt-initshows exit 0, anddocker compose run --rm jwt-init ls -ln /jwtshows-r--r--r-- ... 0 0 64 ... jwt.hex.jwt/holds only.gitkeepandgit statusis clean. The consensus client reports the execution client as online.docker compose run --rm jwt-init sha256sum /jwt/jwt.hex. It is unchanged, and the clients are not recreated, afterdocker compose up -d, afterdocker compose restart, and afterdocker compose down && docker compose up -d. Afterdocker compose down -v && docker compose up -dit is different and the clients connect.mainwith the clients connected.git pullthis branch: the containers keep running.docker compose up -drecreates the execution and consensus clients, they connect, and the secret is not7074a5bf....jwt/jwt.hex:git pullrefuses. Follow the README.jwt-initlogsLoaded the Engine API secret from jwt/jwt.hexand the volume holds that secret.jwt/jwt.hex:jwt-initlogsIgnoring jwt/jwt.hexand a new secret is generated.jwt/jwt.hex(for exampleecho zz > jwt/jwt.hex):jwt-initexits 1 with a message and the clients are not started.EL=el-nethermindandEL=el-reth, each withCL=cl-lighthouse,cl-grandine,cl-teku,cl-lodestar,cl-prysmandcl-nimbus. The consensus client starts, reads the secret and reports the execution client as online. Teku and Nimbus matter most, as they do not run as root. Also the default services with no.env(nethermind,lighthouse).docker compose exec <el> touch /root/jwt/xfails (read-only). A non-root host user cannot read/var/lib/docker/volumes/<project>_jwt/_data/jwt.hex.