From fea00202dbb5502112be2b0ef637a98ad58e6cd4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Mon, 9 Nov 2026 21:50:39 +0000 Subject: [PATCH 1/3] security: generate the Engine API JWT per deployment jwt/jwt.hex was tracked, so every default deployment shared one Engine API secret. A one-shot jwt-init service now writes a random secret before the EL and CL start, never replacing an existing one, and the file is untracked. A label on each EL and CL service makes the next `docker compose up -d` recreate both so they load the new secret together. --- .gitignore | 3 +++ CLAUDE.md | 2 +- README.md | 1 + compose-cl.yml | 30 ++++++++++++++++++++++++++++++ compose-el.yml | 10 ++++++++++ docker-compose.yml | 36 ++++++++++++++++++++++++++++++++++++ jwt/.gitkeep | 0 jwt/jwt.hex | 1 - 8 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 jwt/.gitkeep delete mode 100644 jwt/jwt.hex diff --git a/.gitignore b/.gitignore index 03b84cb9..262e94f3 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,6 @@ commit-boost/config.toml # Allow test fixtures !scripts/edit/**/test/fixtures/.charon/ !scripts/edit/**/test/fixtures/.charon/** + +# Engine API secret, generated per deployment by the jwt-init service in docker-compose.yml +jwt/jwt.hex* diff --git a/CLAUDE.md b/CLAUDE.md index 60caeffc..d587f6bc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -178,7 +178,7 @@ When modifying this repository: ## Important Notes - **Never commit `.env` files** - they contain operator-specific configuration -- **JWT secret** in `jwt/jwt.hex` must be shared between EL and CL clients +- **JWT secret** in `jwt/jwt.hex` must be shared between EL and CL clients. It is not tracked: the `jwt-init` service generates it per deployment before the EL and CL start - **Cluster lock** in `.charon/cluster-lock.json` is critical - back it up before any edit operations - **Validator keys** in `.charon/validator_keys/` must be kept secure and never committed - **Data directory compatibility**: When switching VCs, verify the new client can handle existing key state diff --git a/README.md b/README.md index 127662f6..dad5730a 100644 --- a/README.md +++ b/README.md @@ -211,6 +211,7 @@ docker compose up cl-grandine -d - **Grafana is only published on `127.0.0.1`.** To view it from another machine, use an SSH tunnel: `ssh -L 3000:127.0.0.1:3000 @`, then open `http://localhost:3000`. Anonymous visitors get read-only access, and password logins are disabled because dashboards and datasources are provisioned from this repo. - **Docker-published ports bypass host firewalls such as `ufw`.** Only bind a port to a non-loopback address (e.g. `MONITORING_IP_GRAFANA`, `EL_IP_HTTP`) if a provider or network firewall restricts who can reach it. By default only the P2P ports are published on all interfaces. - **Avoid descriptive hostnames.** DNS names and TLS certificates naming the network or what the node runs (e.g. `mainnet`, `validator`) make a node easy to find with internet-wide scans. +- **The Engine API secret is unique to each deployment.** On first start, the `jwt-init` service writes a random secret to `jwt/jwt.hex`, which the execution and consensus clients share. It never replaces an existing secret. Older releases shipped the same `jwt/jwt.hex` to every operator, so `git pull` deletes that file and a new secret is generated. On the next `docker compose up -d`, both clients are recreated and load the new secret together. If they keep running on the old secret, restart them together: `docker compose down && docker compose up -d`. If you start the clients without `jwt-init`, for example with your own compose files, create the secret first: `openssl rand -hex 32 > jwt/jwt.hex`. - **If you ran a release before the Grafana hardening,** Grafana now starts from a fresh Docker volume and no longer uses `data/grafana`. After upgrading, run `./grafana/security-check.sh` (it may need `sudo`). It reports whether Grafana is published on a public address. It lists anything in the old `data/grafana` that this stack does not create, such as service accounts, tokens, users, datasources, plugins, organizations, dashboards, alert rules and contact points. It also lists changed or added files under `grafana/`, which older releases mounted writable. Report findings to security@obol.tech, then delete `data/grafana`. Grafana stores secrets with a default encryption key, so if your Grafana was reachable, rotate any credentials you stored in it, such as webhook URLs or tokens used by contact points. - **Upgrading leaves the old Grafana database behind.** Anything created in the Grafana UI, such as custom dashboards, alert rules, contact points or users, stays in the unused `data/grafana` and does not carry over to the fresh Grafana. Before upgrading, export any custom dashboards (Share > Export) and note your alert rules. Do not copy the old database across, as it may contain unauthorized changes. To set customizations up again, add dashboards as JSON files under `grafana/dashboards/`, or re-enable logins with your own admin password in a `docker-compose.override.yml`, adding `:docker-compose.override.yml` to the end of `COMPOSE_FILE` in `.env` so it is loaded: diff --git a/compose-cl.yml b/compose-cl.yml index c5d0008e..6a7d09c7 100644 --- a/compose-cl.yml +++ b/compose-cl.yml @@ -13,9 +13,14 @@ services: cl-grandine: profiles: [ cl-grandine ] image: sifrai/grandine:${CL_GRANDINE_VERSION:-${GRANDINE_VERSION:-3.0.0-rc.2}} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_GRANDINE_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: - --data-dir=/root/.grandine - --eth1-rpc-urls=http://${EL}:8551 @@ -48,9 +53,14 @@ services: cl-lighthouse: profiles: [ cl-lighthouse ] image: sigp/lighthouse:${LIGHTHOUSE_VERSION:-v8.3.3} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_LIGHTHOUSE_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | lighthouse bn --network=${NETWORK} @@ -85,9 +95,14 @@ services: cl-teku: profiles: [ cl-teku ] image: consensys/teku:${CL_TEKU_VERSION:-26.10.3} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_TEKU_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | --network=${NETWORK} --checkpoint-sync-url=${LIGHTHOUSE_CHECKPOINT_SYNC_URL} @@ -121,9 +136,14 @@ services: cl-lodestar: profiles: [ cl-lodestar ] image: chainsafe/lodestar:${CL_LODESTAR_VERSION:-v1.50.1} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_LODESTAR_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | beacon --network=${NETWORK} @@ -157,9 +177,14 @@ services: cl-prysm: profiles: [ cl-prysm ] image: gcr.io/prysmaticlabs/prysm/beacon-chain:${CL_PRYSM_VERSION:-v7.3.1} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_PRYSM_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: - --accept-terms-of-use - --${NETWORK} @@ -192,9 +217,14 @@ services: cl-nimbus: profiles: [ cl-nimbus ] image: statusim/nimbus-eth2:${CL_NIMBUS_VERSION:-multiarch-v26.10.2} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped labels: - "alloy-monitored=${CL_NIMBUS_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" # Nimbus has no runtime checkpoint-sync flag — to checkpoint-sync from an # empty data dir, run the `trustedNodeSync` subcommand once before `up`: # docker compose run --rm --no-deps cl-nimbus trustedNodeSync \ diff --git a/compose-el.yml b/compose-el.yml index 938e6adc..c5346d7a 100644 --- a/compose-el.yml +++ b/compose-el.yml @@ -12,6 +12,9 @@ services: el-nethermind: profiles: [el-nethermind] image: nethermind/nethermind:${NETHERMIND_VERSION:-${EL_NETHERMIND_VERSION:-2.2.2}} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped ports: - ${EL_PORT_P2P:-30303}:30303/tcp # P2P TCP @@ -20,6 +23,8 @@ services: - ${EL_IP_ENGINE:-127.0.0.1}:${EL_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${EL_NETHERMIND_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | --config=${NETWORK} --data-dir=/nethermind/data @@ -50,6 +55,9 @@ services: el-reth: profiles: [el-reth] image: ghcr.io/paradigmxyz/reth:${RETH_VERSION:-${EL_RETH_VERSION:-v2.8.1}} + depends_on: + jwt-init: + condition: service_completed_successfully restart: unless-stopped ports: - ${EL_PORT_P2P:-30303}:30303/tcp # P2P TCP @@ -58,6 +66,8 @@ services: - ${EL_IP_ENGINE:-127.0.0.1}:${EL_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${EL_RETH_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | node --full diff --git a/docker-compose.yml b/docker-compose.yml index 26faa465..0ab1a8b1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,6 +3,32 @@ # ${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment. services: + # _ _ + # (_)_ _| |_ + # | \ \ /\ / / __| + # | |\ V V /| |_ + # _/ | \_/\_/ \__| + # |__/ + + # Creates the Engine API secret shared by the execution and consensus clients, once per deployment. + # Runs before them and never replaces an existing jwt/jwt.hex. + jwt-init: + image: busybox:${BUSYBOX_VERSION:-1.37.0} + network_mode: none + volumes: + - ./jwt:/jwt + command: + - sh + - -ec + - | + if [ -s /jwt/jwt.hex ]; then echo "Using existing jwt/jwt.hex"; exit 0; fi + od -An -tx1 -v -N32 /dev/urandom | tr -d ' \n' > /jwt/jwt.hex.tmp + grep -qE '^[0-9a-f]{64}$$' /jwt/jwt.hex.tmp + # Teku and Nimbus run as non-root users, so the secret must be readable by all. + chmod 644 /jwt/jwt.hex.tmp + mv /jwt/jwt.hex.tmp /jwt/jwt.hex + echo "Created a new random jwt/jwt.hex" + # _ _ _ _ # _ __ ___| |_| |__ ___ _ __ _ __ ___ (_)_ __ __| | # | '_ \ / _ \ __| '_ \ / _ \ '__| '_ ` _ \| | '_ \ / _` | @@ -11,6 +37,9 @@ services: nethermind: image: nethermind/nethermind:${NETHERMIND_VERSION:-2.2.2} + depends_on: + jwt-init: + condition: service_completed_successfully profiles: [""] restart: unless-stopped ports: @@ -20,6 +49,8 @@ services: - ${NETHERMIND_IP_ENGINE:-127.0.0.1}:${NETHERMIND_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${NETHERMIND_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | --config=${NETWORK} --datadir=data @@ -50,6 +81,9 @@ services: lighthouse: image: sigp/lighthouse:${LIGHTHOUSE_VERSION:-v8.3.3} + depends_on: + jwt-init: + condition: service_completed_successfully profiles: [""] ports: - ${LIGHTHOUSE_PORT_P2P:-9000}:9000/tcp # P2P TCP @@ -58,6 +92,8 @@ services: - ${LIGHTHOUSE_IP_HTTP:-127.0.0.1}:${LIGHTHOUSE_PORT_HTTP:-5052}:5052 # BN TCP labels: - "alloy-monitored=${LIGHTHOUSE_ALLOY_MONITORED:-true}" + # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. + - "jwt-secret=per-deployment" command: | lighthouse bn --network=${NETWORK} diff --git a/jwt/.gitkeep b/jwt/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/jwt/jwt.hex b/jwt/jwt.hex deleted file mode 100644 index bc2136a8..00000000 --- a/jwt/jwt.hex +++ /dev/null @@ -1 +0,0 @@ -7074a5bf6bd6dae368fa598249d57edfcbccc67a1205b2c8d5d2fe7b800663aa \ No newline at end of file From c3b536815a495028feeb2527c4117c20c4333093 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Tue, 10 Nov 2026 15:17:34 +0000 Subject: [PATCH 2/3] security: keep the Engine API secret in a Docker volume The generated secret was a world-readable file in the repo directory. jwt-init now writes it to a jwt volume that only the execution and consensus clients mount, read-only. An operator-provided jwt/jwt.hex is loaded instead, the old public secret is ignored, and an invalid file stops the clients from starting. The mount change recreates both clients on upgrade, so the label is gone. --- .gitignore | 2 +- CLAUDE.md | 4 ++-- README.md | 2 +- compose-cl.yml | 24 ++++++---------------- compose-el.yml | 8 ++------ docker-compose.yml | 50 ++++++++++++++++++++++++++++++++-------------- 6 files changed, 47 insertions(+), 43 deletions(-) diff --git a/.gitignore b/.gitignore index 262e94f3..716447d7 100644 --- a/.gitignore +++ b/.gitignore @@ -27,5 +27,5 @@ commit-boost/config.toml !scripts/edit/**/test/fixtures/.charon/ !scripts/edit/**/test/fixtures/.charon/** -# Engine API secret, generated per deployment by the jwt-init service in docker-compose.yml +# Your own Engine API secret, if you provide one. The jwt-init service in docker-compose.yml generates one otherwise. jwt/jwt.hex* diff --git a/CLAUDE.md b/CLAUDE.md index d587f6bc..53a926e4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -133,7 +133,7 @@ docker compose exec charon charon alpha test --publish # infra | peers - `.charon/`: Cluster configuration and validator keys (created by DKG or add-validators) - `data/`: Persistent data for all clients (execution, consensus, validator databases) -- `jwt/`: JWT secret for execution<->consensus authentication +- `jwt/`: optional operator-provided JWT secret for execution<->consensus authentication (generated in the `jwt` volume otherwise) - `grafana/`: Monitoring dashboards and configuration - `prometheus/`: Metrics collection configuration - `scripts/`: Automation scripts for cluster operations @@ -178,7 +178,7 @@ When modifying this repository: ## Important Notes - **Never commit `.env` files** - they contain operator-specific configuration -- **JWT secret** in `jwt/jwt.hex` must be shared between EL and CL clients. It is not tracked: the `jwt-init` service generates it per deployment before the EL and CL start +- **JWT secret** must be shared between EL and CL clients. It is not tracked: the `jwt-init` service generates it per deployment in the `jwt` volume before the EL and CL start, or loads an operator-provided `jwt/jwt.hex`. Every EL and CL service needs `depends_on: jwt-init` (`service_completed_successfully`) and a read-only mount of the `jwt` volume - **Cluster lock** in `.charon/cluster-lock.json` is critical - back it up before any edit operations - **Validator keys** in `.charon/validator_keys/` must be kept secure and never committed - **Data directory compatibility**: When switching VCs, verify the new client can handle existing key state diff --git a/README.md b/README.md index dad5730a..b7cf794d 100644 --- a/README.md +++ b/README.md @@ -211,7 +211,7 @@ docker compose up cl-grandine -d - **Grafana is only published on `127.0.0.1`.** To view it from another machine, use an SSH tunnel: `ssh -L 3000:127.0.0.1:3000 @`, then open `http://localhost:3000`. Anonymous visitors get read-only access, and password logins are disabled because dashboards and datasources are provisioned from this repo. - **Docker-published ports bypass host firewalls such as `ufw`.** Only bind a port to a non-loopback address (e.g. `MONITORING_IP_GRAFANA`, `EL_IP_HTTP`) if a provider or network firewall restricts who can reach it. By default only the P2P ports are published on all interfaces. - **Avoid descriptive hostnames.** DNS names and TLS certificates naming the network or what the node runs (e.g. `mainnet`, `validator`) make a node easy to find with internet-wide scans. -- **The Engine API secret is unique to each deployment.** On first start, the `jwt-init` service writes a random secret to `jwt/jwt.hex`, which the execution and consensus clients share. It never replaces an existing secret. Older releases shipped the same `jwt/jwt.hex` to every operator, so `git pull` deletes that file and a new secret is generated. On the next `docker compose up -d`, both clients are recreated and load the new secret together. If they keep running on the old secret, restart them together: `docker compose down && docker compose up -d`. If you start the clients without `jwt-init`, for example with your own compose files, create the secret first: `openssl rand -hex 32 > jwt/jwt.hex`. +- **The Engine API secret is unique to each deployment.** On first start, the `jwt-init` service generates a random secret in the `jwt` Docker volume, which only the execution and consensus clients mount, so it is not a file in this directory. Older releases shipped the same `jwt/jwt.hex` to every operator. After `git pull`, the next `docker compose up -d` recreates both clients with a new secret, and nothing else is needed. Start them together: a client still running from before the update cannot authenticate to one that has the new secret. If `git pull` stops because you changed `jwt/jwt.hex`, run `git checkout jwt/jwt.hex` and pull again. To use your own secret, for example with an execution client outside this stack, put it in `jwt/jwt.hex` as 64 hex characters (copy it aside before the `git checkout`), then run `docker compose up -d --force-recreate`. The secret that older releases shipped is ignored. To print the secret in use: `docker compose run --rm jwt-init cat /jwt/jwt.hex`. - **If you ran a release before the Grafana hardening,** Grafana now starts from a fresh Docker volume and no longer uses `data/grafana`. After upgrading, run `./grafana/security-check.sh` (it may need `sudo`). It reports whether Grafana is published on a public address. It lists anything in the old `data/grafana` that this stack does not create, such as service accounts, tokens, users, datasources, plugins, organizations, dashboards, alert rules and contact points. It also lists changed or added files under `grafana/`, which older releases mounted writable. Report findings to security@obol.tech, then delete `data/grafana`. Grafana stores secrets with a default encryption key, so if your Grafana was reachable, rotate any credentials you stored in it, such as webhook URLs or tokens used by contact points. - **Upgrading leaves the old Grafana database behind.** Anything created in the Grafana UI, such as custom dashboards, alert rules, contact points or users, stays in the unused `data/grafana` and does not carry over to the fresh Grafana. Before upgrading, export any custom dashboards (Share > Export) and note your alert rules. Do not copy the old database across, as it may contain unauthorized changes. To set customizations up again, add dashboards as JSON files under `grafana/dashboards/`, or re-enable logins with your own admin password in a `docker-compose.override.yml`, adding `:docker-compose.override.yml` to the end of `COMPOSE_FILE` in `.env` so it is loaded: diff --git a/compose-cl.yml b/compose-cl.yml index 6a7d09c7..68f95fa8 100644 --- a/compose-cl.yml +++ b/compose-cl.yml @@ -19,8 +19,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_GRANDINE_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: - --data-dir=/root/.grandine - --eth1-rpc-urls=http://${EL}:8551 @@ -40,7 +38,7 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/cl-grandine:/root/.grandine - - ./jwt:/jwt:ro + - jwt:/jwt:ro networks: [ dvnode ] # _ _ _ _ _ @@ -59,8 +57,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_LIGHTHOUSE_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | lighthouse bn --network=${NETWORK} @@ -83,7 +79,7 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/lighthouse:/opt/app/beacon # Keep data in lighthouse and not cl-lighthouse for backwards compatibility - - ./jwt:/opt/jwt + - jwt:/opt/jwt:ro networks: [ dvnode ] # _ _ @@ -101,8 +97,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_TEKU_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | --network=${NETWORK} --checkpoint-sync-url=${LIGHTHOUSE_CHECKPOINT_SYNC_URL} @@ -124,7 +118,7 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/cl-teku:/opt/teku/data - - ./jwt:/jwt:ro + - jwt:/jwt:ro networks: [ dvnode ] # _ _ _ @@ -142,8 +136,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_LODESTAR_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | beacon --network=${NETWORK} @@ -165,7 +157,7 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/cl-lodestar:/opt/lodestar/data - - ./jwt:/jwt:ro + - jwt:/jwt:ro networks: [ dvnode ] # _ __ _ __ _ _ ___ _ __ ___ @@ -183,8 +175,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_PRYSM_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: - --accept-terms-of-use - --${NETWORK} @@ -205,7 +195,7 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/cl-prysm:/data - - ./jwt:/jwt:ro + - jwt:/jwt:ro networks: [ dvnode ] # _ _ @@ -223,8 +213,6 @@ services: restart: unless-stopped labels: - "alloy-monitored=${CL_NIMBUS_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" # Nimbus has no runtime checkpoint-sync flag — to checkpoint-sync from an # empty data dir, run the `trustedNodeSync` subcommand once before `up`: # docker compose run --rm --no-deps cl-nimbus trustedNodeSync \ @@ -249,5 +237,5 @@ services: - ${CL_IP_HTTP:-127.0.0.1}:${CL_PORT_HTTP:-5052}:5052 volumes: - ./data/cl-nimbus:/data - - ./jwt:/jwt:ro + - jwt:/jwt:ro networks: [ dvnode ] diff --git a/compose-el.yml b/compose-el.yml index c5346d7a..379e72e5 100644 --- a/compose-el.yml +++ b/compose-el.yml @@ -23,8 +23,6 @@ services: - ${EL_IP_ENGINE:-127.0.0.1}:${EL_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${EL_NETHERMIND_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | --config=${NETWORK} --data-dir=/nethermind/data @@ -44,7 +42,7 @@ services: networks: [dvnode] volumes: - ./data/nethermind:/nethermind/data - - ./jwt:/root/jwt + - jwt:/root/jwt:ro # _ _ # _ __ ___| |_| |__ @@ -66,8 +64,6 @@ services: - ${EL_IP_ENGINE:-127.0.0.1}:${EL_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${EL_RETH_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | node --full @@ -83,4 +79,4 @@ services: networks: [dvnode] volumes: - ./data/reth:/reth/data - - ./jwt:/root/jwt + - jwt:/root/jwt:ro diff --git a/docker-compose.yml b/docker-compose.yml index 0ab1a8b1..f32ce7bb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -11,23 +11,46 @@ services: # |__/ # Creates the Engine API secret shared by the execution and consensus clients, once per deployment. - # Runs before them and never replaces an existing jwt/jwt.hex. + # It lives in the jwt volume, which only those clients mount, so it is not a file in this directory. + # To use your own secret, for example with an external execution client, put it in jwt/jwt.hex. jwt-init: image: busybox:${BUSYBOX_VERSION:-1.37.0} network_mode: none volumes: - - ./jwt:/jwt + - jwt:/jwt + - ./jwt:/custom:ro command: - sh - - -ec + - -euc - | - if [ -s /jwt/jwt.hex ]; then echo "Using existing jwt/jwt.hex"; exit 0; fi + # One line of 64 hex characters, which every client accepts with or without 0x and a newline. + is_secret() { grep -qxE '(0x)?[0-9a-fA-F]{64}' "$$1" && [ "$$(grep -c '' "$$1")" = 1 ]; } + # Read-only for everyone: Teku and Nimbus do not run as root, the other clients do. + store() { chmod 444 /jwt/jwt.hex.tmp && mv -f /jwt/jwt.hex.tmp /jwt/jwt.hex; } + rm -f /jwt/jwt.hex.tmp + if [ -s /custom/jwt.hex ]; then + if ! is_secret /custom/jwt.hex; then + echo "jwt/jwt.hex must contain 32 bytes as 64 hex characters" >&2 + exit 1 + elif grep -qi 7074a5bf6bd6dae368fa598249d57edfcbccc67a1205b2c8d5d2fe7b800663aa /custom/jwt.hex; then + echo "Ignoring jwt/jwt.hex: older releases shipped this secret to everyone, so it is public. Delete the file." + else + if ! cmp -s /custom/jwt.hex /jwt/jwt.hex; then + cp /custom/jwt.hex /jwt/jwt.hex.tmp + store + echo "Loaded the Engine API secret from jwt/jwt.hex. If the execution or consensus client is running, recreate both." + fi + exit 0 + fi + fi + if [ -s /jwt/jwt.hex ]; then exit 0; fi od -An -tx1 -v -N32 /dev/urandom | tr -d ' \n' > /jwt/jwt.hex.tmp - grep -qE '^[0-9a-f]{64}$$' /jwt/jwt.hex.tmp - # Teku and Nimbus run as non-root users, so the secret must be readable by all. - chmod 644 /jwt/jwt.hex.tmp - mv /jwt/jwt.hex.tmp /jwt/jwt.hex - echo "Created a new random jwt/jwt.hex" + if ! is_secret /jwt/jwt.hex.tmp; then + echo "Could not generate the Engine API secret" >&2 + exit 1 + fi + store + echo "Generated a new Engine API secret" # _ _ _ _ # _ __ ___| |_| |__ ___ _ __ _ __ ___ (_)_ __ __| | @@ -49,8 +72,6 @@ services: - ${NETHERMIND_IP_ENGINE:-127.0.0.1}:${NETHERMIND_PORT_ENGINE:-8551}:8551 # ENGINE-API labels: - "alloy-monitored=${NETHERMIND_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | --config=${NETWORK} --datadir=data @@ -70,7 +91,7 @@ services: networks: [dvnode] volumes: - ./data/nethermind:/nethermind/data - - ./jwt:/root/jwt + - jwt:/root/jwt:ro # _ _ _ _ _ # | (_) __ _| |__ | |_| |__ ___ _ _ ___ ___ @@ -92,8 +113,6 @@ services: - ${LIGHTHOUSE_IP_HTTP:-127.0.0.1}:${LIGHTHOUSE_PORT_HTTP:-5052}:5052 # BN TCP labels: - "alloy-monitored=${LIGHTHOUSE_ALLOY_MONITORED:-true}" - # Changing this label recreates the client, so the EL and CL load jwt/jwt.hex together. - - "jwt-secret=per-deployment" command: | lighthouse bn --network=${NETWORK} @@ -113,7 +132,7 @@ services: networks: [dvnode] volumes: - ./data/lighthouse:/opt/app/beacon - - ./jwt:/opt/jwt + - jwt:/opt/jwt:ro restart: unless-stopped # _ @@ -246,4 +265,5 @@ networks: monitoring: volumes: + jwt: grafana-data: From fe34c3245e00fc62805f95cafdf20128a6b62a1a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ois=C3=ADn=20Kyne?= Date: Tue, 10 Nov 2026 16:37:23 +0000 Subject: [PATCH 3/3] docs: split the Engine API secret note into short points The README covered what the secret is, upgrading, a refused git pull, using your own secret and printing it in one paragraph. Each is now its own point. --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index b7cf794d..a2099dc2 100644 --- a/README.md +++ b/README.md @@ -211,7 +211,11 @@ docker compose up cl-grandine -d - **Grafana is only published on `127.0.0.1`.** To view it from another machine, use an SSH tunnel: `ssh -L 3000:127.0.0.1:3000 @`, then open `http://localhost:3000`. Anonymous visitors get read-only access, and password logins are disabled because dashboards and datasources are provisioned from this repo. - **Docker-published ports bypass host firewalls such as `ufw`.** Only bind a port to a non-loopback address (e.g. `MONITORING_IP_GRAFANA`, `EL_IP_HTTP`) if a provider or network firewall restricts who can reach it. By default only the P2P ports are published on all interfaces. - **Avoid descriptive hostnames.** DNS names and TLS certificates naming the network or what the node runs (e.g. `mainnet`, `validator`) make a node easy to find with internet-wide scans. -- **The Engine API secret is unique to each deployment.** On first start, the `jwt-init` service generates a random secret in the `jwt` Docker volume, which only the execution and consensus clients mount, so it is not a file in this directory. Older releases shipped the same `jwt/jwt.hex` to every operator. After `git pull`, the next `docker compose up -d` recreates both clients with a new secret, and nothing else is needed. Start them together: a client still running from before the update cannot authenticate to one that has the new secret. If `git pull` stops because you changed `jwt/jwt.hex`, run `git checkout jwt/jwt.hex` and pull again. To use your own secret, for example with an execution client outside this stack, put it in `jwt/jwt.hex` as 64 hex characters (copy it aside before the `git checkout`), then run `docker compose up -d --force-recreate`. The secret that older releases shipped is ignored. To print the secret in use: `docker compose run --rm jwt-init cat /jwt/jwt.hex`. +- **The Engine API secret is unique to each deployment.** On first start, the `jwt-init` service generates a random secret in the `jwt` Docker volume, which only the execution and consensus clients mount, so it is not a file in this directory. + - **Upgrading:** older releases shipped the same `jwt/jwt.hex` to every operator. After `git pull`, the next `docker compose up -d` recreates the execution and consensus clients with a new secret, and nothing else is needed. Start them together: a client still running from before the update cannot authenticate to one that has the new secret. + - **If `git pull` stops because you changed `jwt/jwt.hex`,** run `git checkout jwt/jwt.hex` and pull again. + - **To use your own secret,** for example with an execution client outside this stack, put it in `jwt/jwt.hex` as 64 hex characters (copy it aside before the `git checkout`), then run `docker compose up -d --force-recreate`. The secret that older releases shipped is ignored. + - **To print the secret in use:** `docker compose run --rm jwt-init cat /jwt/jwt.hex`. - **If you ran a release before the Grafana hardening,** Grafana now starts from a fresh Docker volume and no longer uses `data/grafana`. After upgrading, run `./grafana/security-check.sh` (it may need `sudo`). It reports whether Grafana is published on a public address. It lists anything in the old `data/grafana` that this stack does not create, such as service accounts, tokens, users, datasources, plugins, organizations, dashboards, alert rules and contact points. It also lists changed or added files under `grafana/`, which older releases mounted writable. Report findings to security@obol.tech, then delete `data/grafana`. Grafana stores secrets with a default encryption key, so if your Grafana was reachable, rotate any credentials you stored in it, such as webhook URLs or tokens used by contact points. - **Upgrading leaves the old Grafana database behind.** Anything created in the Grafana UI, such as custom dashboards, alert rules, contact points or users, stays in the unused `data/grafana` and does not carry over to the fresh Grafana. Before upgrading, export any custom dashboards (Share > Export) and note your alert rules. Do not copy the old database across, as it may contain unauthorized changes. To set customizations up again, add dashboards as JSON files under `grafana/dashboards/`, or re-enable logins with your own admin password in a `docker-compose.override.yml`, adding `:docker-compose.override.yml` to the end of `COMPOSE_FILE` in `.env` so it is loaded: