From d55b5b8a301d8a1b5478eda2cee4df685545315b Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 13 Nov 2026 14:19:39 -0800 Subject: [PATCH 1/3] docs(shiva): record what the 2026-11-13 Redfish re-read found The 2026-09-09 hardening holds, but the BMC also runs HTTP, SSDP and the unauthenticated XML reply, holds a second all-privilege local account, and has the lab's internet egress through its DHCP gateway. The current-state docs said its one path out of the segment was deleted; they now say what is true. Co-Authored-By: Claude Opus 5.5 --- SECURITY.md | 2 +- docs/changelog.md | 24 ++++++++++++++++++++++++ docs/hardware.md | 2 +- docs/security.md | 2 +- 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index c91892de..8204e9a5 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -51,7 +51,7 @@ is a very different thing from an overlooked one. Full detail in | The lab's Prometheus and Loki published to VLAN 30 with no authentication | **Open until the rollout lands.** The fix is authored ([#834](https://github.com/Gerrrt/HomeLab/issues/834)), and none of the four hosts converges, so it is live only once `stacks/lab/README.md`'s ordered rollout has run and its three `curl`s return 401. `alexander` published both to the segment that exists to hold attackers (#436, 2026-09). Anything on it could `POST /-/quit`, forge series and delete log ranges in the lab's evidence store. Once rolled out, they are on `127.0.0.1`, behind `stacks/lab/Caddyfile`: one token each for `odin`, `phoenix` and `fenrir`, and a reader token, all the lab's own and never the estate's. **Two residuals.** The tokens cross VLAN 30 as plain HTTP, where ARP spoofing is expected (#764's question). And nothing pages if the proxy is bypassed, because the lab has no Alertmanager (ADR-0020, [#858](https://github.com/Gerrrt/HomeLab/issues/858)). | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | | Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. | -| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. | +| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. Re-read 2026-11-13 over Redfish: the 2026-09-09 settings hold, and the control is weaker than this row described. A second local account with every privilege, the unauthenticated XML reply, SSDP and HTTP are on, and the BMC has the lab's outbound internet through its DHCP gateway. The tripwire watches the other segments, not the internet. Recorded in [`docs/hardware.md`](docs/hardware.md#management); the fixes are on the BMC and on `morpheus`, by hand. | | `smaug` ships no logs, and no absence rule can say so | **Accepted residual, not a fix in progress.** [ADR-0016](docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) reverses the monitoring direction for CasaBonita (40) so that nothing on that segment initiates upward: `prometheus` scrapes `10.0.40.30` over `99 → 40:9100`, and `smaug` runs no Alloy agent. Loki has no pull — ingest is a push or it is nothing — so the NAS ships no logs. Its SMART attributes went the same way until [ADR-0047](docs/adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) put them under the scrape — a root cron job on the host writes a textfile the exporter serves, nothing initiating from 40 — and its patch state is deliberately not collected, an appliance having no `apt` to ask ([#483](https://github.com/Gerrrt/HomeLab/issues/483)). It is the one host in the estate whose compromise is invisible: Prometheus shows it alive and busy, and nothing shows what it did — on the segment whose stated assumption is that everything on it is already compromised. That is already true with nothing deployed on it: the blind spot is a property of the direction rather than of what runs there, and Jellyfin's network-facing indexer will land on a host that was invisible before it arrived. **Collecting the logs costs more than the gap.** A `40 → 99:3100` rule would be the first upward path in the estate, and it would put an agent token, which is write access to the log store, on the one segment whose stated assumption is that everything on it is already compromised. The ingest ports are authenticated since #182; that makes the token the thing a compromise of `smaug` would take, and it does not make handing one over cheaper. What still watches the segment is the `igc0.40` tripwire and nothing else — Suricata runs on Skids and Degens, not on 40. **No new alert is warranted, and that is the second half of this row.** `TerminalSegmentReachedInternalNetwork` fires on a `pass` from a terminal VLAN toward `30`, `50`, `99` or the switch LAN, which catches a compromise *leaving*; anything that stays on CasaBonita or egresses to the internet leaves no record at all, and a rule cannot be written against a record that does not exist. Decided under [#255](https://github.com/Gerrrt/HomeLab/issues/255); the detail is in [`docs/security.md`](docs/security.md). | **The switch was the honest gap, and it closed on 2026-11-05** with the diff --git a/docs/changelog.md b/docs/changelog.md index 173089c2..d5cf98d3 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -17,6 +17,30 @@ roadmap as it read that day, and the *Done* entries keep the shape they had there. `check_docs.py` does not check this file, for the reason its module docstring gives: it is a record, not a claim about now. +## 2026-11-13 + +- **`shiva` re-read over Redfish, from `Saruman`.** This was a read-only audit: + GET requests only, plus a handshake and a port check. Nothing was installed and + nothing was set. The 2026-09-09 hardening holds. SSH (`22`) and IPMI-over-LAN + (`623/udp`) are off on the iLO and closed on the wire. Federation management + is off. SNMP answers only the `prometheus` v3 user, with SHA and AES and no + community. Firmware 2.82 is still HPE's last iLO 4 release (2023-03-02), as + ADR-0033 assumed. Health is clean: fans, both supplies, all eight DIMMs, the + four drives and the Smart Storage Battery, which reads `Ok` after the + August–September failures in the IML. What the hardening never recorded: + - a second local account with every privilege, older than the retained event + log; + - the unauthenticated XML reply (`/xmldata`), SSDP with Federation's + multicast discovery, and HTTP on `80`; + - a DHCP gateway, so the BMC has the lab's internet egress, which + `docs/security.md`'s "its one path out of the segment deleted" overstated; + - no NTP server, so the iLO's clock restarts at 1970 after each power + removal, and the event log logs only every third failed login. + + `docs/hardware.md`, `docs/security.md` and `SECURITY.md` now say so. The + fixes are on the iLO and on `morpheus`, by hand, and are not done in this + entry. + ## 2026-11-12 - **#1473 is deployed on `alexander` and verified.** This completes the #1430 diff --git a/docs/hardware.md b/docs/hardware.md index 9eade0ae..e0c0453a 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -324,7 +324,7 @@ version component in either table. | Host | BMC | Address | Notes | | --- | --- | --- | --- | -| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | +| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. **Read again 2026-11-13** over Redfish: the 2026-09-09 settings hold, and these are on beside them, never recorded until then: HTTP on `80`, virtual media on `17988`, SSDP on `1900/udp` with Federation's multicast discovery, the unauthenticated XML reply (`/xmldata`, which serves the serial number and every NIC's MAC to the segment), and the serial CLI behind a login. It has **two** local accounts, both with every privilege: the one this row has meant, and a second one older than the retained event log (2025-10-12), which nothing here records. It takes its gateway, DNS and DDNS registration by DHCP and has no NTP server, so its clock restarts at 1970 after every power removal and many event-log entries carry no date. The certificate is HPE's factory self-signed one, to 2031-01-13 Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | The BMC and the host it manages carry different names and different addresses: `shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.20`. Earlier diff --git a/docs/security.md b/docs/security.md index 91674877..867fe1cb 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it; the lab tripwire logs anything that gets past the blocks ([#234](https://github.com/Gerrrt/HomeLab/issues/234)) | | A lost or stolen tunnel peer | **Accepted.** A WireGuard peer is a device with a key and no second factor ([ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a `/32` pin on the server and a preshared key per peer. Revocation is a `wg0.conf` edit on `phoenix` ([runbook](runbooks/open-the-remote-path.md#rollback)), proportionate at two or three devices and recorded as the thing that stops being so | -| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | +| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its rule to the monitoring host deleted, so nothing on the lab interface lets it reach another segment. It still takes the segment's gateway, so it has the lab's outbound internet like every lab host, and the 2026-11-13 re-read found services and a second administrator account the hardening never recorded ([`hardware.md`](hardware.md#management)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | | Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them | | Someone on the lab segment reading the domain's metrics | **Scoped, unauthenticated, and accepted.** `windows_exporter` listens on `9182` on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits `10.0.30.40` (`alexander`) alone ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) §7). That rule is the difference between an endpoint a `/24` sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns `alexander` reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as `node_exporter` on the NAS below | From abf4c6b5ed4277bf6f84da115a42bcf6c5b75a83 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 13 Nov 2026 16:43:45 -0800 Subject: [PATCH 2/3] docs(changelog): remove conflict markers left by the da2d97c merge Co-Authored-By: Claude Opus 5.5 --- docs/changelog.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 1812c51f..a20df371 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,7 +19,6 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-13 -<<<<<<< HEAD - **`shiva` re-read over Redfish, from `Saruman`.** This was a read-only audit: GET requests only, plus a handshake and a port check. Nothing was installed and nothing was set. The 2026-09-09 hardening holds. SSH (`22`) and IPMI-over-LAN @@ -41,8 +40,7 @@ docstring gives: it is a record, not a claim about now. `docs/hardware.md`, `docs/security.md` and `SECURITY.md` now say so. The fixes are on the iLO and on `morpheus`, by hand, and are not done in this entry. -||||||| 4aca054 -======= + - **`ragnarok`'s outbound scope is a filter on the VM, as ADR-0014 decided.** ADR-0014 says the attack VM's "own outbound scope is limited on the VM itself, to `10.0.30.0/24` and the target subnet". A read-only audit of @@ -61,7 +59,6 @@ docstring gives: it is a record, not a claim about now. lift both for a package install, and §8 check three proves it. Not yet run against `ragnarok`: the scope holds once `finish-attack-vm.sh` has run again from `phoenix`. ->>>>>>> origin/main - **An audit of `golem`, read-only, found its last verify failed and a 19-day gap in the lab's backups.** It was read on the box against this From d60f969c82a2e6828487234225515353bf473f29 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 13 Nov 2026 22:38:47 -0800 Subject: [PATCH 3/3] docs(shiva): record the iLO as re-hardened after the 2026-11-13 re-read The second all-privilege account is deleted, the XML reply, SSDP and Federation's multicast discovery are off, SNTP comes from 10.0.30.1, every failed login is logged, and morpheus blocks the iLO from anything off the segment. hardware.md, network.md, security.md, SECURITY.md and the changelog entry now describe that state rather than the findings. Co-Authored-By: Claude Opus 5.5 --- SECURITY.md | 2 +- docs/changelog.md | 12 +++++++++--- docs/hardware.md | 2 +- docs/network.md | 8 +++++++- docs/security.md | 2 +- 5 files changed, 19 insertions(+), 7 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 8204e9a5..28ff5425 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -51,7 +51,7 @@ is a very different thing from an overlooked one. Full detail in | The lab's Prometheus and Loki published to VLAN 30 with no authentication | **Open until the rollout lands.** The fix is authored ([#834](https://github.com/Gerrrt/HomeLab/issues/834)), and none of the four hosts converges, so it is live only once `stacks/lab/README.md`'s ordered rollout has run and its three `curl`s return 401. `alexander` published both to the segment that exists to hold attackers (#436, 2026-09). Anything on it could `POST /-/quit`, forge series and delete log ranges in the lab's evidence store. Once rolled out, they are on `127.0.0.1`, behind `stacks/lab/Caddyfile`: one token each for `odin`, `phoenix` and `fenrir`, and a reader token, all the lab's own and never the estate's. **Two residuals.** The tokens cross VLAN 30 as plain HTTP, where ARP spoofing is expected (#764's question). And nothing pages if the proxy is bypassed, because the lab has no Alertmanager (ADR-0020, [#858](https://github.com/Gerrrt/HomeLab/issues/858)). | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | | Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. | -| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. Re-read 2026-11-13 over Redfish: the 2026-09-09 settings hold, and the control is weaker than this row described. A second local account with every privilege, the unauthenticated XML reply, SSDP and HTTP are on, and the BMC has the lab's outbound internet through its DHCP gateway. The tripwire watches the other segments, not the internet. Recorded in [`docs/hardware.md`](docs/hardware.md#management); the fixes are on the BMC and on `morpheus`, by hand. | +| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. Re-read 2026-11-13 over Redfish: the 2026-09-09 settings held, and the control had been weaker than this row described, because the tripwire watches the other segments and not the internet. A second local account with every privilege was deleted; the unauthenticated XML reply, SSDP and Federation's multicast discovery were turned off; the clock now comes from SNTP and every failed login is logged; and a block on `morpheus` takes away the outbound internet the BMC had through its DHCP gateway. Detail in [`docs/hardware.md`](docs/hardware.md#management). | | `smaug` ships no logs, and no absence rule can say so | **Accepted residual, not a fix in progress.** [ADR-0016](docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) reverses the monitoring direction for CasaBonita (40) so that nothing on that segment initiates upward: `prometheus` scrapes `10.0.40.30` over `99 → 40:9100`, and `smaug` runs no Alloy agent. Loki has no pull — ingest is a push or it is nothing — so the NAS ships no logs. Its SMART attributes went the same way until [ADR-0047](docs/adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) put them under the scrape — a root cron job on the host writes a textfile the exporter serves, nothing initiating from 40 — and its patch state is deliberately not collected, an appliance having no `apt` to ask ([#483](https://github.com/Gerrrt/HomeLab/issues/483)). It is the one host in the estate whose compromise is invisible: Prometheus shows it alive and busy, and nothing shows what it did — on the segment whose stated assumption is that everything on it is already compromised. That is already true with nothing deployed on it: the blind spot is a property of the direction rather than of what runs there, and Jellyfin's network-facing indexer will land on a host that was invisible before it arrived. **Collecting the logs costs more than the gap.** A `40 → 99:3100` rule would be the first upward path in the estate, and it would put an agent token, which is write access to the log store, on the one segment whose stated assumption is that everything on it is already compromised. The ingest ports are authenticated since #182; that makes the token the thing a compromise of `smaug` would take, and it does not make handing one over cheaper. What still watches the segment is the `igc0.40` tripwire and nothing else — Suricata runs on Skids and Degens, not on 40. **No new alert is warranted, and that is the second half of this row.** `TerminalSegmentReachedInternalNetwork` fires on a `pass` from a terminal VLAN toward `30`, `50`, `99` or the switch LAN, which catches a compromise *leaving*; anything that stays on CasaBonita or egresses to the internet leaves no record at all, and a rule cannot be written against a record that does not exist. Decided under [#255](https://github.com/Gerrrt/HomeLab/issues/255); the detail is in [`docs/security.md`](docs/security.md). | **The switch was the honest gap, and it closed on 2026-11-05** with the diff --git a/docs/changelog.md b/docs/changelog.md index bb0dc2d8..12daa623 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -37,9 +37,15 @@ docstring gives: it is a record, not a claim about now. - no NTP server, so the iLO's clock restarts at 1970 after each power removal, and the event log logs only every third failed login. - `docs/hardware.md`, `docs/security.md` and `SECURITY.md` now say so. The - fixes are on the iLO and on `morpheus`, by hand, and are not done in this - entry. + All but HTTP were fixed the same day, by hand. On the iLO, the second + account was deleted; the XML reply, SSDP and Federation's multicast + discovery were turned off; SNTP was set to `10.0.30.1` with DHCP's time + settings off; and every failed login is now logged. On `morpheus`, a block + on `igc0.30` from `10.0.30.10` to anything off the segment sits above + *Allow internet*. A second Redfish read confirmed the iLO side, and the + event log holds each change; the `morpheus` rule was not tested from the + BMC's side. `docs/hardware.md`, `docs/network.md`, `docs/security.md` and + `SECURITY.md` describe the result. - **`ragnarok`'s outbound scope is a filter on the VM, as ADR-0014 decided.** ADR-0014 says the attack VM's "own outbound scope is limited on the VM diff --git a/docs/hardware.md b/docs/hardware.md index e0c0453a..286e73a0 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -324,7 +324,7 @@ version component in either table. | Host | BMC | Address | Notes | | --- | --- | --- | --- | -| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. **Read again 2026-11-13** over Redfish: the 2026-09-09 settings hold, and these are on beside them, never recorded until then: HTTP on `80`, virtual media on `17988`, SSDP on `1900/udp` with Federation's multicast discovery, the unauthenticated XML reply (`/xmldata`, which serves the serial number and every NIC's MAC to the segment), and the serial CLI behind a login. It has **two** local accounts, both with every privilege: the one this row has meant, and a second one older than the retained event log (2025-10-12), which nothing here records. It takes its gateway, DNS and DDNS registration by DHCP and has no NTP server, so its clock restarts at 1970 after every power removal and many event-log entries carry no date. The certificate is HPE's factory self-signed one, to 2031-01-13 Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | +| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. **Read again and re-hardened 2026-11-13**: the 2026-09-09 settings had held, and the re-read found what they never covered. Since then, also off: the unauthenticated XML reply (`/xmldata`, which had served the serial number and every NIC's MAC to the segment), SSDP, and Federation's multicast discovery. A second local account with every privilege, older than the retained event log and recorded nowhere, was deleted, so `Administrator` is the only account. The clock comes from SNTP at `10.0.30.1` rather than DHCP, so it no longer restarts at 1970 after a power removal, and every failed login is logged. What stays on: HTTPS, HTTP on `80`, the remote console, virtual media on `17988`, the serial CLI behind a login, and SNMP. It takes its gateway and DNS by DHCP, and `morpheus` blocks it from anything off the segment ([`network.md`](network.md#imaginationlan--vlan-30--lab)). The certificate is HPE's factory self-signed one, to 2031-01-13. Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | The BMC and the host it manages carry different names and different addresses: `shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.20`. Earlier diff --git a/docs/network.md b/docs/network.md index 17989c9d..3335eb45 100644 --- a/docs/network.md +++ b/docs/network.md @@ -574,7 +574,13 @@ Where things get broken on purpose. ### Notes -- Reachable from Hicks only; outbound internet permitted. +- Reachable from Hicks only; outbound internet permitted, except for `shiva`. +- **`shiva` reaches nothing off the segment.** A block on `igc0.30` from + `10.0.30.10` to anything outside `10.0.30.0/24` sits above *Allow + internet*, since 2026-11-13. The iLO takes the segment's gateway by DHCP and + had the lab's outbound internet until then. Its SNMP replies to the + monitoring host still pass, because pf state carries them, as it did for the + return-path rule deleted on 2026-09-09. - **The block that stops it reaching CasaBonita is *Block access to CasaBonita*** on `igc0.30`, one of a run of per-segment blocks (Degens, Skids, CasaBonita, Hicks, Winterfell, the untagged LAN) that sit above the diff --git a/docs/security.md b/docs/security.md index 91d681b2..69f117ef 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it; the lab tripwire logs anything that gets past the blocks ([#234](https://github.com/Gerrrt/HomeLab/issues/234)) | | A lost or stolen tunnel peer | **Accepted.** A WireGuard peer is a device with a key and no second factor ([ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a `/32` pin on the server and a preshared key per peer. Revocation is a `wg0.conf` edit on `phoenix` ([runbook](runbooks/open-the-remote-path.md#rollback)), proportionate at two or three devices and recorded as the thing that stops being so | -| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its rule to the monitoring host deleted, so nothing on the lab interface lets it reach another segment. It still takes the segment's gateway, so it has the lab's outbound internet like every lab host, and the 2026-11-13 re-read found services and a second administrator account the hardening never recorded ([`hardware.md`](hardware.md#management)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | +| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its rule to the monitoring host deleted — and again on 2026-11-13, when a re-read found what that missed: a second administrator account, now deleted, the unauthenticated XML reply and SSDP, now off, and the lab's outbound internet through its DHCP gateway, now blocked on `morpheus`, so it reaches nothing off the segment ([`hardware.md`](hardware.md#management)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | | Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them | | Someone on the lab segment reading the domain's metrics | **Scoped, unauthenticated, and accepted.** `windows_exporter` listens on `9182` on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits `10.0.30.40` (`alexander`) alone ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) §7). That rule is the difference between an endpoint a `/24` sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns `alexander` reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as `node_exporter` on the NAS below |