From 753feea214872fa7b82ca8a064ed77d32bfc4ec0 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 14 Nov 2026 00:27:28 +0000 Subject: [PATCH] docs(phoenix): key-only sshd and a default-deny inbound policy (ADR-0117) phoenix holds the only key the lab domain accepts, and the repository never said who may log in to phoenix itself. The 2026-11-13 audit found the installer's password login and no host firewall. ADR-0117 decides the following, and build-the-jumpbox.md section 6a applies it by hand, since nothing converges phoenix: - sshd takes keys only; - ufw admits 22 from Hicks and the tunnel's peers, plus the WireGuard port; - the tunnel's routing is left to morpheus, as ADR-0042 has it. security.md's list of phoenix's secrets gains the gandalf token and four LAB_* passwords it was missing. Refs #1496. Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +- ...nd-build-phoenix-as-the-deployment-host.md | 6 + ...-sshd-and-a-default-deny-inbound-policy.md | 130 ++++++++++++++++++ docs/architecture.md | 2 +- docs/network.md | 6 +- docs/roadmap.md | 6 + docs/runbooks/build-the-jumpbox.md | 73 ++++++++++ docs/runbooks/build-the-lab-templates.md | 14 +- docs/runbooks/open-the-remote-path.md | 9 ++ docs/security.md | 13 +- 10 files changed, 252 insertions(+), 11 deletions(-) create mode 100644 docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md diff --git a/README.md b/README.md index a4480640..627648f0 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ documents for different readers. at all. Every pinned digest is also scanned weekly for fixable HIGH and CRITICAL CVEs, and each image with findings has an open issue until a scan finds it clean. -- **Documented decisions and runbooks.** 116 ADRs covering what was chosen +- **Documented decisions and runbooks.** 117 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 54 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. Every critical alert links to @@ -235,7 +235,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md changelog.md │ ├── diagrams/ # the network diagram (SVG) and its predecessors -│ ├── adr/ # 116 ADRs — architecture decision records +│ ├── adr/ # 117 ADRs — architecture decision records │ ├── exercises/ # purple-team exercise records — see its README │ └── runbooks/ # 54 runbooks; successor-handover.md is the front door └── Makefile # make help diff --git a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md index faf80a0e..f1086e04 100644 --- a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md +++ b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md @@ -35,6 +35,12 @@ and amends the first consequence of > one host instead of the segment. Nothing here is amended, per ADR-0001; > [`build-the-playground.md`](../runbooks/build-the-playground.md) §4 carries > the procedure and the finding. +> +> [!NOTE] +> Amended by [ADR-0117](0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): +> `phoenix`'s own `sshd` is key-only, and its inbound policy is default-deny +> with `ufw`. It admits `22` from Hicks and the tunnel's peers only. The text +> here is left as written, per ADR-0001 and ADR-0115. ## Context diff --git a/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md b/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md new file mode 100644 index 00000000..fdfdb3ee --- /dev/null +++ b/docs/adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md @@ -0,0 +1,130 @@ +# ADR-0117: Give `phoenix` a key-only `sshd` and a default-deny inbound policy + +**Status:** Accepted · 2026-11 · amends [ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) + +## Context + +[ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +made `phoenix` the one machine whose purpose is to hold credentials for other +machines. It holds the Proxmox tokens for `Saruman` and `gandalf`, the +domain's passwords, `tofu/`'s state passphrase, and the only key that +ADR-0029's six accept for administration +([ADR-0077](0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). +[`security.md`](../security.md) accepts the residual: whoever owns `phoenix` +owns the lab domain. + +That acceptance assumes the way into `phoenix` is narrow, but nothing in the +repository says what it is. The runbook that built it, +[`build-the-jumpbox.md`](../runbooks/build-the-jumpbox.md), installs OpenSSH +and says nothing more. The six, which `phoenix` configures, are held to more +than that: `sshd` refuses passwords, and one rule admits `10.0.30.70` alone. +The audit of the host on 2026-11-13 +([#1496](https://github.com/Gerrrt/HomeLab/issues/1496)) found what that gap +leaves in place: + +- the Ubuntu installer's `PasswordAuthentication yes`, in + `sshd_config.d/50-cloud-init.conf`; +- no host firewall; +- `sshd` answering every address that can route to it. + +Those addresses are all of VLAN 30, all of Hicks +([`firewall-claims.yaml`](../firewall-claims.yaml): nothing blocks Hicks from +ImaginationLAN), and the tunnel's peers. A password guessed or reused from any +of those owns the lab domain. The key-only rule on the six then protects +nothing, because the key is on the machine the password opens. + +Three things must still reach `phoenix`: + +- **The operator,** from a Hicks workstation or through the tunnel. The + Mac's `deploy-agent.sh` run and the CA copy in + [`build-the-lab-templates.md`](../runbooks/build-the-lab-templates.md) both + come from Hicks. +- **WireGuard's handshake,** from the internet through `morpheus`'s one + `rdr` ([ADR-0042](0042-terminate-the-remote-path-on-the-lab-and-route-it.md)). +- **A Debian or Kali installer under construction,** fetching its preseed from + Packer's HTTP server on `8800`, for the length of that build + ([ADR-0090](0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)). + +Nothing on VLAN 30 needs to log in to `phoenix`. Its work all flows outward: +API calls to `8006`, SSH and WinRM to the guests it builds and configures, +and Alloy's push to `alexander`. + +## Decision + +**`sshd` on `phoenix` is key-only.** A drop-in, +`/etc/ssh/sshd_config.d/10-phoenix.conf`, sorts ahead of the installer's +`50-cloud-init.conf`. `sshd` keeps the first value it reads, so the drop-in +wins without the installer's file being edited. It sets: + +- `PasswordAuthentication no` and `KbdInteractiveAuthentication no`; +- `PermitRootLogin no`; +- `X11Forwarding no`. + +Every workstation that logs in has its key in `authorized_keys` before the +drop-in is applied. That ordering is the whole risk of the change, so the +runbook makes it a check rather than a reminder. + +**Inbound is default-deny, with `ufw`.** `ufw` because +[`build-the-lab-templates.md`](../runbooks/build-the-lab-templates.md) +already opens `8800` with it. One tool, not two. The rules: + +- `22/tcp` from Hicks, `10.0.50.0/24`; +- `22/tcp` arriving on `wg0` from the peers, `172.31.0.0/24`; +- the WireGuard listen port, `udp`, from anywhere. `morpheus` decides who + reaches it, and the port number stays out of this repository; +- `8800/tcp` from `10.0.30.0/24`, added for a Debian or Kali build and + deleted after it, as that runbook already says; +- outbound allowed, as `ufw` defaults. + +**Routing from the tunnel is left as ADR-0042 made it.** `ufw`'s forward +policy is `DROP`, so one route rule, `wg0` to `ens18`, carries the peers' +traffic as before, with replies admitted as established. It does not narrow +the destinations. ADR-0042 makes `morpheus` the control and its blocks the +attribution. A second filter on `phoenix` would stop that traffic before +`morpheus` could log it, and the remote-path runbook's negative tests would +then prove the wrong firewall. `ufw` does not touch `net.ipv4.ip_forward`, so +the systemd drop-in still scopes forwarding to the tunnel's lifetime. + +**Nothing converges `phoenix`, so this is applied by hand** from +[`build-the-jumpbox.md`](../runbooks/build-the-jumpbox.md) §6a, and a rebuild +applies it in the same place. The runbook ends with the checks that prove +each rule from the side it applies to. + +This amends ADR-0043, which placed `phoenix` and named what it holds but not +how it is reached. + +## Alternatives considered + +- **`nftables` directly,** with a ruleset file in the repository. It is more + exact, and [ADR-0093](0093-run-openvas-on-garuda-on-demand-scoped-to-the-domain-by-nftables.md) + uses it on `garuda`. Declined because the template runbook's `ufw` step + would then be a second tool on the same host. The policy here is a short + allow-list, which `ufw` states in five lines. +- **Admit `22` from VLAN 30 as well.** It is the segment `phoenix` is on, and + the rule would be shorter. Declined: nothing on it needs to log in. VLAN 30 + includes the lab domain and the guests `phoenix` builds, which are the + machines least trusted to reach the one that owns them. +- **Narrow the routed traffic on `phoenix` to `10.0.30.0/24`.** Declined + for the reason above: it moves ADR-0042's control and its evidence off + `morpheus`. +- **Leave passwords on and rely on the firewall.** Declined. Hicks is a + whole segment of household machines, and the firewall admits all of it. + +## Consequences + +- **A lost key now locks a workstation out,** where before it fell back to a + password. The recovery path is `phoenix`'s console in `Saruman`'s web UI, + reached from Hicks, where a local login still takes a password. +- **A rebuilt `phoenix` is closed before its first login from anywhere but + Hicks and the tunnel.** `deploy-agent.sh` from the Mac is unaffected. +- **Peers keep their access to `phoenix`'s `22`,** as before. A peer is a + device with a key and no second factor (`security.md`), and `sshd` asks for + a second key. +- **`8800` is a step again, not a no-op.** The template runbook's "if `ufw` is + active" becomes unconditional. +- **Reopened by:** + - anything on VLAN 30 that needs to reach `phoenix` inbound; + - `phoenix` gaining a converger, which would carry this config instead of + the runbook; + - Hicks being narrowed by `firewall-claims.yaml`, which would let the + `22` rule name workstations instead of the segment. diff --git a/docs/architecture.md b/docs/architecture.md index 9856f4d3..7de40230 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -201,7 +201,7 @@ hole from the monitoring VLAN into the monitored one. | `dot-gentoo` (10.0.30.96) | 🟢 30 | *(none — a test machine)* | Built 2026-10-09. Gentoo Linux on the 23.0 systemd profile, cloned from `tpl-gentoo` (908), which Packer made from Gentoo's own signed cloud image, for the dotfiles' Gentoo layer (whose default is OpenRC). Secure Boot is off. A guest on `Saruman`, VMID 196, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | | `dot-nixos` (10.0.30.97) | 🟢 30 | *(none — a test machine)* | Built 2026-10-09. NixOS 26.05, cloned from `tpl-nixos` (909), which Packer installed from the minimal ISO with a committed `configuration.nix`, for the dotfiles' NixOS layer. Secure Boot is off. A guest on `Saruman`, VMID 197, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | | `dot-windows` (10.0.30.98) | 🟢 30 | *(none — Windows)* | Built 2026-10-08. Windows 11 Pro, cloned from `tpl-win11-pro` (911) and left unactivated, for the dotfiles' Windows layer. A guest on `Saruman`, VMID 198, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy | -| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and runs no continuous convergence (what it configures, it configures on demand, below); it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). The first toolchain on it is [`packer/`](../packer/README.md), the lab's VM templates at VMIDs 901–912, with 903–909 for the dotfiles OS layers ([ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)) ([ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md), [`build-the-lab-templates.md`](runbooks/build-the-lab-templates.md)). The second is [`ansible/`](../ansible/README.md), which configures ADR-0029's six over SSH and touches nothing else ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md), [#448](https://github.com/Gerrrt/HomeLab/issues/448)). Alloy agent (native package, `scripts/deploy-agent.sh`) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and runs no continuous convergence (what it configures, it configures on demand, below); it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). The first toolchain on it is [`packer/`](../packer/README.md), the lab's VM templates at VMIDs 901–912, with 903–909 for the dotfiles OS layers ([ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)) ([ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md), [`build-the-lab-templates.md`](runbooks/build-the-lab-templates.md)). The second is [`ansible/`](../ansible/README.md), which configures ADR-0029's six over SSH and touches nothing else ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md), [#448](https://github.com/Gerrrt/HomeLab/issues/448)). Its own way in is decided by [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): `sshd` takes keys only, and `ufw` admits `22` from Hicks and the tunnel's peers, the tunnel's port, and `8800` from VLAN 30 for the length of a Debian or Kali template build. Applying it on the host is [#1496](https://github.com/Gerrrt/HomeLab/issues/1496). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `golem` (10.0.30.80) | 🟢 30 | *(none — an appliance, no Docker)* | Proxmox Backup Server 4.2, built 2026-10-03 by [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) ([#485](https://github.com/Gerrrt/HomeLab/issues/485)): the lab's backups, placed by [ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md). A guest on `Saruman`, VMID 180, whose only datastore, `erebor`, is `smaug`'s `erebor/pbs` over NFSv4 through one `2049` pass, with `atime` on because garbage collection depends on it, and the mountpoint made immutable so an unmounted share cannot fill the OS disk. Every backup is encrypted on `Saruman` before it leaves, with a key made on `Saruman` and copied, on 2026-10-03, into `secrets/lab.sops.yaml` and onto paper (the runbook's §7), so a lost `Saruman` does not take the backups' readability with it; `Saruman` writes through a token that can back up and cannot prune. PBS prunes (7 daily, 4 weekly, 3 monthly), collects garbage on Saturdays and verifies on Sundays; TrueNAS keeps fourteen daily snapshots of the dataset as the copy PBS cannot prune. The nightly job takes `odin` and the domain's six; `alexander`, `phoenix`, `fenrir` and `golem` itself are rebuilt from this repository and stay out. `scripts/collect-pbs-task-state.py` publishes every job's last outcome and the snapshots' verify state hourly, and seven rules in `stacks/lab` read them; the first nightly run, 2026-10-03, was verified the next morning. Its Alloy pushes to `alexander`, never to VLAN 99, since 2026-10-03. Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | [`stacks/wiki`](../stacks/wiki) | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)), hand-run containers until [#251](https://github.com/Gerrrt/HomeLab/issues/251) wrote them down as a stack deployed by hand from `main`, like `smaug`'s. **Cut over 2026-10-07** ([#847](https://github.com/Gerrrt/HomeLab/issues/847), [ADR-0082](adr/0082-serve-the-wiki-over-tls-and-deploy-it-from-a-verified-checkout.md)): deployed by hand from a checkout whose signature is verified first, and served over https from a Caddy on a leaf from the estate's CA, with port 80 only redirecting ([`stacks/wiki/README.md`](../stacks/wiki/README.md#tls)); its database is pulled to `prometheus` nightly as a `pg_dump` (`make backup-wiki`, [ADR-0065](adr/0065-pull-the-wikis-database-to-prometheus-as-a-dump.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Holding real data since 2026-09-28, Immich's first 615 photographs, which passed the #404 step 10 gate while most of it was open. The library is copied nightly to `oracle`, off-host and not off-estate, until [#455](https://github.com/Gerrrt/HomeLab/issues/455) ([ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) run beside them, with the household's page of links to them at `home.matrix.elysium` and its live numbers on read-only tokens ([#137](https://github.com/Gerrrt/HomeLab/issues/137)) — Homepage until [ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md) made it a page of Glance's. ntfy ([#136](https://github.com/Gerrrt/HomeLab/issues/136)) is the in-house receiver for Alertmanager's three real channels, **deployed and cut over 2026-09-29** by [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy), with `urgent` and `security` also sent to ntfy.sh for a phone away from home. Miniflux, a feed reader with a Postgres of its own, is the tier's first service beyond ADR-0008's nine, **deployed 2026-09-29** ([#147](https://github.com/Gerrrt/HomeLab/issues/147), [ADR-0057](adr/0057-add-miniflux-to-the-sensitive-tier-with-its-fetcher-kept-off-winterfell.md)): the one container here that reaches outward on a timer, with its fetcher refused every private address. Memos, the household's notes, is the second, decided by [ADR-0059](adr/0059-add-memos-to-the-sensitive-tier-for-notes-and-keep-documentation-in-docs.md) ([#145](https://github.com/Gerrrt/HomeLab/issues/145)) and deployed behind Caddy the same day, all sixteen containers healthy after that `make up`. Miniflux holds no real data; Memos holds the household's first real notes since 2026-10-22, ahead of ADR-0023's off-estate copy by the operator's accepted exception, with the nightly volume set on `oracle` its only copy off the host, since the household copy under [#455](https://github.com/Gerrrt/HomeLab/issues/455) does not carry `memos-data` yet. Mealie, the household's recipes at `recipes.matrix.elysium`, is the third, **deployed 2026-09-29** ([#146](https://github.com/Gerrrt/HomeLab/issues/146), [ADR-0060](adr/0060-add-mealie-to-the-sensitive-tier-as-recipes.md)), behind the same Caddy with no new rule, its URL import refused every private address. linkding, a bookmark manager, is the fourth, **deployed 2026-09-29** ([#144](https://github.com/Gerrrt/HomeLab/issues/144), [ADR-0061](adr/0061-add-linkding-to-the-sensitive-tier-behind-one-factor.md)). Actual ([#142](https://github.com/Gerrrt/HomeLab/issues/142)), the household's budget, is the fifth beyond ADR-0008's nine, **deployed 2026-09-29** ([ADR-0062](adr/0062-add-actual-to-the-sensitive-tier.md)), claimed from SOPS before its first start. It holds no real budget before #404 step 10. Stirling-PDF, the household's PDF editor at `pdf.matrix.elysium`, is the sixth, **deployed 2026-09-29** ([#143](https://github.com/Gerrrt/HomeLab/issues/143), [ADR-0063](adr/0063-add-stirling-pdf-to-the-sensitive-tier-and-keep-its-documents-in-memory.md)): its documents live on a tmpfs and never reach a disk. Glance, the operator's browser start page at `start.matrix.elysium`, is **deployed 2026-10-21** ([#1223](https://github.com/Gerrrt/HomeLab/issues/1223), [ADR-0098](adr/0098-add-glance-as-the-operators-start-page-beside-homepage.md)): the estate's alerts, probes and targets read from `prometheus` with the reader token, and feeds read from a Miniflux user of its own. Since [ADR-0100](adr/0100-grow-glance-into-a-homelab-page-and-keep-the-operators-lists-in-sops.md) it has a page per subject — Home, Read, Watch, Sports & Markets, Homelab, Network & Storage and Maintenance — the estate's read from `prometheus`, and the operator's own lists come from SOPS. It also serves the household's page at `home.matrix.elysium` ([ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md)), which took over from Homepage. Mosquitto and ring-mqtt bring the Ring Alarm into Home Assistant, the broker on an internal network that only those two join, **deployed 2026-10-22** ([ADR-0104](adr/0104-bring-the-ring-alarm-into-home-assistant-through-ring-mqtt.md)). Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | diff --git a/docs/network.md b/docs/network.md index 262c2cbc..bcdabcd5 100644 --- a/docs/network.md +++ b/docs/network.md @@ -715,7 +715,11 @@ Where things get broken on purpose. [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s WireGuard tunnel terminates, built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the peers reach the - lab through it and nothing else. Its Alloy pushes to + lab through it and nothing else. Inbound, it is default-deny + ([ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md), + applied by [#1496](https://github.com/Gerrrt/HomeLab/issues/1496)). `22` + admits Hicks and the peers, and nothing else on this segment. `8800` opens + to the segment only while a Debian or Kali template builds. Its Alloy pushes to `alexander` and not to Winterfell, and it gets no pass out of this segment: a rule from it into 99 would make it the bastion ADR-0002 and ADR-0012 declined. Everything diff --git a/docs/roadmap.md b/docs/roadmap.md index 5d7ed27f..696db030 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -102,6 +102,12 @@ Closes when it is empty. `smaug`.** Decided by [ADR-0114](adr/0114-rip-discs-on-smaug-with-makemkv-started-only-to-rip.md): deploy it, measure what capabilities it needs, and create the `5800` pass. +- **[#1496](https://github.com/Gerrrt/HomeLab/issues/1496) Close `phoenix`'s + way in.** Decided by + [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): + give every workstation that logs in a key, then run + [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md) §6a on the host. + First of these, because `phoenix` owns the lab domain. The rest of the milestone has no order between its issues. diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 05e19a21..52658a47 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -384,6 +384,79 @@ to check the pass to `10.0.30.40:9443`, the answer is that there is no pass to place — go back to §5, because that message was written for hosts on another VLAN and this one is not. +## 6a. Close the way in + +[ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): +`sshd` takes keys only, and inbound is default-deny. Nothing converges this +host, so this section is how it gets there, on a rebuild as on the first +build. + +**First, every workstation that logs in has a key.** This is the step that +locks you out if it is skipped. From each Hicks machine that uses `phoenix`, +add its public key to `~/.ssh/authorized_keys` on `phoenix`, then prove it +with passwords refused on the client side: + +```bash +ssh -o PasswordAuthentication=no -o KbdInteractiveAuthentication=no @10.0.30.70 true && echo key-ok +``` + +Every machine prints `key-ok` before you go on. Keep one session open on +`phoenix` until the end of this section. + +**Then `sshd`.** On `phoenix`. The `10-` prefix sorts ahead of the installer's +`50-cloud-init.conf`, and `sshd` keeps the first value it reads: + +```bash +sudo tee /etc/ssh/sshd_config.d/10-phoenix.conf >/dev/null <<'EOF' +# ADR-0117: phoenix owns the lab domain, so its own login is key-only. +PasswordAuthentication no +KbdInteractiveAuthentication no +PermitRootLogin no +X11Forwarding no +EOF +sudo sshd -t && sudo systemctl reload ssh +sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|x11forwarding) ' +``` + +All four lines read `no`. A fresh login from a second terminal, with the +first still open, is the check that the reload did not shut you out. + +**Then the firewall.** `ufw` does not touch `net.ipv4.ip_forward`, and its +forward policy is `DROP`, so the tunnel needs the one `route` line. On a +first build the tunnel does not exist yet: leave out the three lines marked +`tunnel`, and [`open-the-remote-path.md`](open-the-remote-path.md) §3 adds +them. + +```bash +sudo ufw default deny incoming +sudo ufw default allow outgoing +sudo ufw default deny routed +sudo ufw allow from 10.0.50.0/24 to any port 22 proto tcp comment 'Hicks' +sudo ufw allow in on wg0 from 172.31.0.0/24 to any port 22 proto tcp comment 'tunnel peers' # tunnel +sudo ufw allow /udp comment 'WireGuard' # tunnel +sudo ufw route allow in on wg0 out on ens18 comment 'ADR-0042: morpheus filters' # tunnel +sudo ufw enable +sudo ufw status verbose +``` + +`` is the one `open-the-remote-path.md` §0 chose. It stays out +of this repository. `ens18` is the NIC `ip -br addr` shows with +`10.0.30.70`. Port `8800` is not here: a Debian or Kali template build opens +it for its own length +([`build-the-lab-templates.md`](build-the-lab-templates.md) §5). + +**Check each rule from the side it applies to.** + +- From Hicks: `ssh @10.0.30.70 true` succeeds. +- From a VLAN 30 host, `alexander` for example: + `nc -zv -w 3 10.0.30.70 22` times out. +- From a peer, with the tunnel up: `ssh @10.0.30.70 true` succeeds. + `ping -c 1 10.0.30.40` answers at `ttl=63`, which shows `phoenix` still + forwards. Then run [`open-the-remote-path.md`](open-the-remote-path.md) §8's + negative tests, which must still be blocked and logged on `morpheus`. +- On `phoenix`: §4's `curl` still lists the guests, and §7's Alloy checks + still pass. Both are outbound, and `ufw` allows outbound. + ## 7. Verify — including the thing that fails quietly ```bash diff --git a/docs/runbooks/build-the-lab-templates.md b/docs/runbooks/build-the-lab-templates.md index d642656a..e9b8e0f6 100644 --- a/docs/runbooks/build-the-lab-templates.md +++ b/docs/runbooks/build-the-lab-templates.md @@ -505,11 +505,15 @@ DHCP server dhcpcd adds a `169.254.x.x` address and a link-local default route ([`build-the-playground.md`](build-the-playground.md) §8). Rebuild 902 and 910 with this section and §5b to clear it. -1. Admit the guest to that port on `phoenix` for the length of the build. If - `ufw` is active, run `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`, - and delete the rule afterwards. Debian's build (§4) needs this step and - no other here. On 2026-11-09 `ufw` was inactive on `phoenix`, so there - was nothing to open. +1. Admit the guest to that port on `phoenix` for the length of the build: + `sudo ufw allow from 10.0.30.0/24 to any port 8800 proto tcp`. Afterwards, + `sudo ufw delete allow from 10.0.30.0/24 to any port 8800 proto tcp`. + Debian's build (§4) needs this step and no other here. `ufw` is + default-deny on `phoenix` + ([ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md)), + so without the rule the installer times out fetching its preseed. On + 2026-11-09, before that ADR, `ufw` was inactive and there was nothing to + open. 2. Give `phoenix` a user on `gandalf`, and trust `gandalf`'s CA on `phoenix`, as §2 did for `Saruman`. Unless `gandalf` joins `Saruman` in a cluster, it is its own API with its own users, so `phoenix@pve` there is a diff --git a/docs/runbooks/open-the-remote-path.md b/docs/runbooks/open-the-remote-path.md index d37d1a74..09c3a3b9 100644 --- a/docs/runbooks/open-the-remote-path.md +++ b/docs/runbooks/open-the-remote-path.md @@ -269,6 +269,15 @@ sudo sysctl -w net.ipv4.ip_forward=0 \ # expect 1, then 0, then 1 ``` +Then the jumpbox's own firewall. `ufw` is default-deny on `phoenix` +([ADR-0117](../adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md)), +so add the three lines +[`build-the-jumpbox.md`](build-the-jumpbox.md) §6a marks `tunnel`: the listen +port, `22` from the peers on `wg0`, and the `route` rule from `wg0` to the +lab. Without the `route` rule the handshake succeeds and nothing forwards, +the same symptom as the CAUTION above. `sudo ufw status verbose` lists all +three. + ## 4. The client configuration On the **client device**, generate its own keypair and build this. Send the diff --git a/docs/security.md b/docs/security.md index 91674877..722e91b0 100644 --- a/docs/security.md +++ b/docs/security.md @@ -17,7 +17,7 @@ What this network is actually built to survive: | A lost or stolen tunnel peer | **Accepted.** A WireGuard peer is a device with a key and no second factor ([ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a `/32` pin on the server and a preshared key per peer. Revocation is a `wg0.conf` edit on `phoenix` ([runbook](runbooks/open-the-remote-path.md#rollback)), proportionate at two or three devices and recorded as the thing that stops being so | | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | -| Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them | +| Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them. `phoenix`'s own way in is narrowed to match by [ADR-0117](adr/0117-give-phoenix-a-key-only-sshd-and-a-default-deny-inbound-policy.md): its `sshd` takes keys only, and its `ufw` admits `22` from Hicks and the tunnel's peers and nothing from the rest of VLAN 30. Until [#1496](https://github.com/Gerrrt/HomeLab/issues/1496) applies it on the host, the installer's password login and an open segment are what stand there | | Someone on the lab segment reading the domain's metrics | **Scoped, unauthenticated, and accepted.** `windows_exporter` listens on `9182` on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits `10.0.30.40` (`alexander`) alone ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) §7). That rule is the difference between an endpoint a `/24` sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns `alexander` reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as `node_exporter` on the NAS below | | Someone on the lab segment fetching a Velociraptor client package | **Scoped by address, and accepted.** `odin`'s `caddy` service (container `soc-agent-msi`) serves the two agent MSIs `ansible/` installs, and the two Debian packages `garuda` installs, on `8448` ([#1068](https://github.com/Gerrrt/HomeLab/issues/1068), [ADR-0092](adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)). Its Caddyfile answers the six domain addresses, `.50`–`.55`, and `garuda`'s `.62`, and refuses everyone else, Hicks's route from VLAN 50 included. The Velociraptor MSI and the Velociraptor `.deb` each carry the client config: the server URL, its CA, and the enrolment nonce. Whoever holds either can enrol a rogue client, which is why #864 took the MSI out of SYSVOL. The residual: an address on this segment can be spoofed, and any of the seven can fetch either package. What the nonce buys is noise in Velociraptor's client list, the same class as the Wazuh enrolment password. Rotating it means a new server config, a rebuilt MSI with the role's pin moved, and the staged `.deb` deleted so that `scripts/stage-agent-msis.sh` builds it again. The two Wazuh packages carry no secret | | An OpenVAS plugin on `garuda` reaching a non-target | **Fenced at layer 2** ([ADR-0093](adr/0093-run-openvas-on-garuda-on-demand-scoped-to-the-domain-by-nftables.md)). OpenVAS scans the domain's six only. While a scan runs, the scanner runs in a network namespace whose one route to the segment is a macvlan, `scan0` at `10.0.30.63`, behind an nftables `netdev` egress hook that drops every frame — IP, ARP (`send_arp_request`) and raw (`send_frame`) — to anything but the six. So the AF_PACKET paths that bypass an IP filter are caught here. Proved: a GVM scan of an out-of-scope host raised the egress drop counter with 0 open ports, and an `arping` from the namespace to an out-of-scope host was dropped while an in-scope host answered. `gvmd`, `notus`, `redis` and `mosquitto` stay on the host; only MQTT crosses to the namespace, on a veth that routes nowhere else | @@ -636,7 +636,10 @@ assumption consistent with what they are. genuinely encrypted, and that no state, plan or `.terraform/` is tracked. - **`phoenix` is a secret-bearing host, and holds no age key.** Its secrets are all mode 600 and owned by its operator: - - the Proxmox API token `phoenix@pve!builder`; + - the Proxmox API token `phoenix@pve!builder` on `Saruman`, and its + namesake on `gandalf`, the range hypervisor, which is in `phoenix.env` as + `TF_VAR_gandalf_api_token` and in `gandalf.env` for the curl scripts + ([ADR-0116](adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)); - the Windows build password, which is also a fresh clone's Administrator password until #448 rotates it; - `tofu/`'s state passphrase, all three in `~/.config/proxmox/phoenix.env` @@ -650,6 +653,12 @@ assumption consistent with what they are. and `LAB_POPULATION_SEED`, from which every population password is derived ([ADR-0078](adr/0078-populate-the-lab-domain-from-a-committed-file-and-a-seed.md)). The seed adds no reach: the first of the three already owns the domain. + Four more, also in `phoenix.env`, add none either + ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md)): + `LAB_TIER_ADMIN_PASSWORD` and `LAB_ENDPOINT_ADMIN_PASSWORD`, which a + rebuild creates accounts with, `LAB_WEAK_PASSWORD`, the deliberately + crackable one (ADR-0083), and `LAB_WAZUH_REGISTRATION_PASSWORD`, a copy + of the SOC manager's enrolment password. The state is the one that would have been quiet. A Terraform state holds every value a provider touched in cleartext, the cloud-init password of every