diff --git a/SECURITY.md b/SECURITY.md index c91892de..28ff5425 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -51,7 +51,7 @@ is a very different thing from an overlooked one. Full detail in | The lab's Prometheus and Loki published to VLAN 30 with no authentication | **Open until the rollout lands.** The fix is authored ([#834](https://github.com/Gerrrt/HomeLab/issues/834)), and none of the four hosts converges, so it is live only once `stacks/lab/README.md`'s ordered rollout has run and its three `curl`s return 401. `alexander` published both to the segment that exists to hold attackers (#436, 2026-09). Anything on it could `POST /-/quit`, forge series and delete log ranges in the lab's evidence store. Once rolled out, they are on `127.0.0.1`, behind `stacks/lab/Caddyfile`: one token each for `odin`, `phoenix` and `fenrir`, and a reader token, all the lab's own and never the estate's. **Two residuals.** The tokens cross VLAN 30 as plain HTTP, where ARP spoofing is expected (#764's question). And nothing pages if the proxy is bypassed, because the lab has no Alertmanager (ADR-0020, [#858](https://github.com/Gerrrt/HomeLab/issues/858)). | | The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. | | Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. | -| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. | +| The hypervisor's BMC shares a broadcast domain with the attack VM | **Accepted residual, not a fix in progress.** `shiva`, the iLO 4 at `10.0.30.10` on firmware 2.82, sits on ImaginationLAN, the segment [ADR-0014](docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) gives to `gandalf`'s Kali VM, and a BMC does not get patched the way a guest does. [ADR-0033](docs/adr/0033-keep-the-ilo-on-the-lab-segment.md) keeps it there on purpose: the iLO is part of the estate under attack, and a BMC compromise in the lab costs the lab. Moving it to Winterfell would put an end-of-line BMC beside the firewall's admin UI and open three more ports on the Hicks list to reach its console. Controls: the BMC's own hardening (IPMI-over-LAN and unused services off, a credential shared with nothing in the house), and the lab tripwire — anything the BMC initiates toward another segment is a logged pass and a `LabSegmentReachedInternalNetwork` alert. Both done 2026-09-09: IPMI-over-LAN, SSH and iLO Federation off on the BMC, its account's credential shared with nothing in the house, its security log read for a baseline; and the `10.0.30.10 → 10.0.99.20/udp` "return path" rule deleted from `morpheus` — it had been evaluated 7.4 million times and matched zero packets, because pf state carries the scrape's replies, and it was the BMC's only path to the monitoring host's syslog listener. The scrape failed once at the reload and has been clean since; the lab interface now passes nothing across a segment except `Saruman`'s two agent ports. Re-read 2026-09-22, when [ADR-0042](docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s tunnel changed who can be on this segment: the residual was written against an attacker the operator starts on purpose, and it now also covers a remote peer — the operator's own devices, keyed and pinned to a `/32`, behind the same tripwire. It stands on the same reasoning; a peer that is not the operator's is ADR-0022's third trigger, not a widening of this row. Re-read 2026-11-13 over Redfish: the 2026-09-09 settings held, and the control had been weaker than this row described, because the tripwire watches the other segments and not the internet. A second local account with every privilege was deleted; the unauthenticated XML reply, SSDP and Federation's multicast discovery were turned off; the clock now comes from SNTP and every failed login is logged; and a block on `morpheus` takes away the outbound internet the BMC had through its DHCP gateway. Detail in [`docs/hardware.md`](docs/hardware.md#management). | | `smaug` ships no logs, and no absence rule can say so | **Accepted residual, not a fix in progress.** [ADR-0016](docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) reverses the monitoring direction for CasaBonita (40) so that nothing on that segment initiates upward: `prometheus` scrapes `10.0.40.30` over `99 → 40:9100`, and `smaug` runs no Alloy agent. Loki has no pull — ingest is a push or it is nothing — so the NAS ships no logs. Its SMART attributes went the same way until [ADR-0047](docs/adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) put them under the scrape — a root cron job on the host writes a textfile the exporter serves, nothing initiating from 40 — and its patch state is deliberately not collected, an appliance having no `apt` to ask ([#483](https://github.com/Gerrrt/HomeLab/issues/483)). It is the one host in the estate whose compromise is invisible: Prometheus shows it alive and busy, and nothing shows what it did — on the segment whose stated assumption is that everything on it is already compromised. That is already true with nothing deployed on it: the blind spot is a property of the direction rather than of what runs there, and Jellyfin's network-facing indexer will land on a host that was invisible before it arrived. **Collecting the logs costs more than the gap.** A `40 → 99:3100` rule would be the first upward path in the estate, and it would put an agent token, which is write access to the log store, on the one segment whose stated assumption is that everything on it is already compromised. The ingest ports are authenticated since #182; that makes the token the thing a compromise of `smaug` would take, and it does not make handing one over cheaper. What still watches the segment is the `igc0.40` tripwire and nothing else — Suricata runs on Skids and Degens, not on 40. **No new alert is warranted, and that is the second half of this row.** `TerminalSegmentReachedInternalNetwork` fires on a `pass` from a terminal VLAN toward `30`, `50`, `99` or the switch LAN, which catches a compromise *leaving*; anything that stays on CasaBonita or egresses to the internet leaves no record at all, and a rule cannot be written against a record that does not exist. Decided under [#255](https://github.com/Gerrrt/HomeLab/issues/255); the detail is in [`docs/security.md`](docs/security.md). | **The switch was the honest gap, and it closed on 2026-11-05** with the diff --git a/docs/changelog.md b/docs/changelog.md index 0da8fc73..b70b4929 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -59,6 +59,34 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-13 +- **`shiva` re-read over Redfish, from `Saruman`.** This was a read-only audit: + GET requests only, plus a handshake and a port check. Nothing was installed and + nothing was set. The 2026-09-09 hardening holds. SSH (`22`) and IPMI-over-LAN + (`623/udp`) are off on the iLO and closed on the wire. Federation management + is off. SNMP answers only the `prometheus` v3 user, with SHA and AES and no + community. Firmware 2.82 is still HPE's last iLO 4 release (2023-03-02), as + ADR-0033 assumed. Health is clean: fans, both supplies, all eight DIMMs, the + four drives and the Smart Storage Battery, which reads `Ok` after the + August–September failures in the IML. What the hardening never recorded: + - a second local account with every privilege, older than the retained event + log; + - the unauthenticated XML reply (`/xmldata`), SSDP with Federation's + multicast discovery, and HTTP on `80`; + - a DHCP gateway, so the BMC has the lab's internet egress, which + `docs/security.md`'s "its one path out of the segment deleted" overstated; + - no NTP server, so the iLO's clock restarts at 1970 after each power + removal, and the event log logs only every third failed login. + + All but HTTP were fixed the same day, by hand. On the iLO, the second + account was deleted; the XML reply, SSDP and Federation's multicast + discovery were turned off; SNTP was set to `10.0.30.1` with DHCP's time + settings off; and every failed login is now logged. On `morpheus`, a block + on `igc0.30` from `10.0.30.10` to anything off the segment sits above + *Allow internet*. A second Redfish read confirmed the iLO side, and the + event log holds each change; the `morpheus` rule was not tested from the + BMC's side. `docs/hardware.md`, `docs/network.md`, `docs/security.md` and + `SECURITY.md` describe the result. + - **An audit of `garuda` found SSH open to the lab domain, and owner GitHub credentials on it.** A read-only audit against this repository found the role intact: no Purple SOC daemons, GVM stopped, the fence files identical to diff --git a/docs/hardware.md b/docs/hardware.md index 9eade0ae..286e73a0 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -324,7 +324,7 @@ version component in either table. | Host | BMC | Address | Notes | | --- | --- | --- | --- | -| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | +| `Saruman` | `shiva` — HPE iLO 4, firmware 2.82 | `10.0.30.10` | iLO Advanced licensed. Dedicated network port. DHCP with a reservation. Hardened 2026-09-09 per [ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md): IPMI-over-LAN, SSH and iLO Federation off; HTTPS, the remote console and SNMP stay on; the account's credential is shared with nothing else; the security log was read for a baseline. **Read again and re-hardened 2026-11-13**: the 2026-09-09 settings had held, and the re-read found what they never covered. Since then, also off: the unauthenticated XML reply (`/xmldata`, which had served the serial number and every NIC's MAC to the segment), SSDP, and Federation's multicast discovery. A second local account with every privilege, older than the retained event log and recorded nowhere, was deleted, so `Administrator` is the only account. The clock comes from SNTP at `10.0.30.1` rather than DHCP, so it no longer restarts at 1970 after a power removal, and every failed login is logged. What stays on: HTTPS, HTTP on `80`, the remote console, virtual media on `17988`, the serial CLI behind a login, and SNMP. It takes its gateway and DNS by DHCP, and `morpheus` blocks it from anything off the segment ([`network.md`](network.md#imaginationlan--vlan-30--lab)). The certificate is HPE's factory self-signed one, to 2031-01-13. Polled over SNMPv3 authPriv (SHA, AES, user `prometheus`) since 2026-09-20 per [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md), with *SNMPv1 Request* off — it answers no community at all | The BMC and the host it manages carry different names and different addresses: `shiva` is the iLO, `Saruman` is the hypervisor at `10.0.30.20`. Earlier diff --git a/docs/network.md b/docs/network.md index d74bff88..fbcdcc58 100644 --- a/docs/network.md +++ b/docs/network.md @@ -578,7 +578,13 @@ Where things get broken on purpose. ### Notes -- Reachable from Hicks only; outbound internet permitted. +- Reachable from Hicks only; outbound internet permitted, except for `shiva`. +- **`shiva` reaches nothing off the segment.** A block on `igc0.30` from + `10.0.30.10` to anything outside `10.0.30.0/24` sits above *Allow + internet*, since 2026-11-13. The iLO takes the segment's gateway by DHCP and + had the lab's outbound internet until then. Its SNMP replies to the + monitoring host still pass, because pf state carries them, as it did for the + return-path rule deleted on 2026-09-09. - **The block that stops it reaching CasaBonita is *Block access to CasaBonita*** on `igc0.30`, one of a run of per-segment blocks (Degens, Skids, CasaBonita, Hicks, Winterfell, the untagged LAN) that sit above the diff --git a/docs/security.md b/docs/security.md index 0f619423..2fe1e0ad 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it; the lab tripwire logs anything that gets past the blocks ([#234](https://github.com/Gerrrt/HomeLab/issues/234)) | | A lost or stolen tunnel peer | **Accepted.** A WireGuard peer is a device with a key and no second factor ([ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a `/32` pin on the server and a preshared key per peer. Revocation is a `wg0.conf` edit on `phoenix` ([runbook](runbooks/open-the-remote-path.md#rollback)), proportionate at two or three devices and recorded as the thing that stops being so | -| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | +| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its rule to the monitoring host deleted — and again on 2026-11-13, when a re-read found what that missed: a second administrator account, now deleted, the unauthenticated XML reply and SSDP, now off, and the lab's outbound internet through its DHCP gateway, now blocked on `morpheus`, so it reaches nothing off the segment ([`hardware.md`](hardware.md#management)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | | Someone on the lab segment reaching the domain's configuration path | **Scoped, key-only, and accepted for `phoenix` itself.** Each of ADR-0029's six listens on `22` for `ansible/` ([ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md)). The rule admits `10.0.30.70` alone, `sshd` refuses passwords, and the only administrators' key is `phoenix`'s, installed by the template and never by hand on a clone. The residual: whoever owns `phoenix` owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them | | Someone on the lab segment logging in to the sensor | **Authored, and live only once applied.** `fenrir` listens on `22` to the whole segment and has no host firewall. [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md) §3 makes `sshd` key-only for one user, refuses root and X11 forwarding, and keeps `authorized_keys` to the machines that administer the guest. Until that step has run on `fenrir`, the installer's `sshd` defaults are the running state. The source address stays open: anyone on the segment can still offer a key |