From dfc0676cd662a77d3c67af3f4bbab458344c62a5 Mon Sep 17 00:00:00 2001 From: Gerrrt Date: Fri, 13 Nov 2026 16:19:33 -0800 Subject: [PATCH 1/2] docs(smaug): correct stale rule counts, golem's pass and runbook residue --- docs/architecture.md | 2 +- docs/runbooks/build-the-nas.md | 50 ++++++++++++++++++++-------------- docs/security.md | 15 +++++++++- stacks/media/README.md | 26 ++++++++++-------- stacks/media/compose.yaml | 5 ++-- 5 files changed, 62 insertions(+), 36 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 9856f4d3..7ac27892 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -205,7 +205,7 @@ hole from the monitoring VLAN into the monitored one. | `golem` (10.0.30.80) | ๐ŸŸข 30 | *(none โ€” an appliance, no Docker)* | Proxmox Backup Server 4.2, built 2026-10-03 by [`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) ([#485](https://github.com/Gerrrt/HomeLab/issues/485)): the lab's backups, placed by [ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md). A guest on `Saruman`, VMID 180, whose only datastore, `erebor`, is `smaug`'s `erebor/pbs` over NFSv4 through one `2049` pass, with `atime` on because garbage collection depends on it, and the mountpoint made immutable so an unmounted share cannot fill the OS disk. Every backup is encrypted on `Saruman` before it leaves, with a key made on `Saruman` and copied, on 2026-10-03, into `secrets/lab.sops.yaml` and onto paper (the runbook's ยง7), so a lost `Saruman` does not take the backups' readability with it; `Saruman` writes through a token that can back up and cannot prune. PBS prunes (7 daily, 4 weekly, 3 monthly), collects garbage on Saturdays and verifies on Sundays; TrueNAS keeps fourteen daily snapshots of the dataset as the copy PBS cannot prune. The nightly job takes `odin` and the domain's six; `alexander`, `phoenix`, `fenrir` and `golem` itself are rebuilt from this repository and stay out. `scripts/collect-pbs-task-state.py` publishes every job's last outcome and the snapshots' verify state hourly, and seven rules in `stacks/lab` read them; the first nightly run, 2026-10-03, was verified the next morning. Its Alloy pushes to `alexander`, never to VLAN 99, since 2026-10-03. Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | ๐Ÿ”ด 99 | [`stacks/wiki`](../stacks/wiki) | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)), hand-run containers until [#251](https://github.com/Gerrrt/HomeLab/issues/251) wrote them down as a stack deployed by hand from `main`, like `smaug`'s. **Cut over 2026-10-07** ([#847](https://github.com/Gerrrt/HomeLab/issues/847), [ADR-0082](adr/0082-serve-the-wiki-over-tls-and-deploy-it-from-a-verified-checkout.md)): deployed by hand from a checkout whose signature is verified first, and served over https from a Caddy on a leaf from the estate's CA, with port 80 only redirecting ([`stacks/wiki/README.md`](../stacks/wiki/README.md#tls)); its database is pulled to `prometheus` nightly as a `pg_dump` (`make backup-wiki`, [ADR-0065](adr/0065-pull-the-wikis-database-to-prometheus-as-a-dump.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs โ€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | ๐Ÿ”ด 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Holding real data since 2026-09-28, Immich's first 615 photographs, which passed the #404 step 10 gate while most of it was open. The library is copied nightly to `oracle`, off-host and not off-estate, until [#455](https://github.com/Gerrrt/HomeLab/issues/455) ([ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich โ€” four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) โ€” Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) run beside them, with the household's page of links to them at `home.matrix.elysium` and its live numbers on read-only tokens ([#137](https://github.com/Gerrrt/HomeLab/issues/137)) โ€” Homepage until [ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md) made it a page of Glance's. ntfy ([#136](https://github.com/Gerrrt/HomeLab/issues/136)) is the in-house receiver for Alertmanager's three real channels, **deployed and cut over 2026-09-29** by [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy), with `urgent` and `security` also sent to ntfy.sh for a phone away from home. Miniflux, a feed reader with a Postgres of its own, is the tier's first service beyond ADR-0008's nine, **deployed 2026-09-29** ([#147](https://github.com/Gerrrt/HomeLab/issues/147), [ADR-0057](adr/0057-add-miniflux-to-the-sensitive-tier-with-its-fetcher-kept-off-winterfell.md)): the one container here that reaches outward on a timer, with its fetcher refused every private address. Memos, the household's notes, is the second, decided by [ADR-0059](adr/0059-add-memos-to-the-sensitive-tier-for-notes-and-keep-documentation-in-docs.md) ([#145](https://github.com/Gerrrt/HomeLab/issues/145)) and deployed behind Caddy the same day, all sixteen containers healthy after that `make up`. Miniflux holds no real data; Memos holds the household's first real notes since 2026-10-22, ahead of ADR-0023's off-estate copy by the operator's accepted exception, with the nightly volume set on `oracle` its only copy off the host, since the household copy under [#455](https://github.com/Gerrrt/HomeLab/issues/455) does not carry `memos-data` yet. Mealie, the household's recipes at `recipes.matrix.elysium`, is the third, **deployed 2026-09-29** ([#146](https://github.com/Gerrrt/HomeLab/issues/146), [ADR-0060](adr/0060-add-mealie-to-the-sensitive-tier-as-recipes.md)), behind the same Caddy with no new rule, its URL import refused every private address. linkding, a bookmark manager, is the fourth, **deployed 2026-09-29** ([#144](https://github.com/Gerrrt/HomeLab/issues/144), [ADR-0061](adr/0061-add-linkding-to-the-sensitive-tier-behind-one-factor.md)). Actual ([#142](https://github.com/Gerrrt/HomeLab/issues/142)), the household's budget, is the fifth beyond ADR-0008's nine, **deployed 2026-09-29** ([ADR-0062](adr/0062-add-actual-to-the-sensitive-tier.md)), claimed from SOPS before its first start. It holds no real budget before #404 step 10. Stirling-PDF, the household's PDF editor at `pdf.matrix.elysium`, is the sixth, **deployed 2026-09-29** ([#143](https://github.com/Gerrrt/HomeLab/issues/143), [ADR-0063](adr/0063-add-stirling-pdf-to-the-sensitive-tier-and-keep-its-documents-in-memory.md)): its documents live on a tmpfs and never reach a disk. Glance, the operator's browser start page at `start.matrix.elysium`, is **deployed 2026-10-21** ([#1223](https://github.com/Gerrrt/HomeLab/issues/1223), [ADR-0098](adr/0098-add-glance-as-the-operators-start-page-beside-homepage.md)): the estate's alerts, probes and targets read from `prometheus` with the reader token, and feeds read from a Miniflux user of its own. Since [ADR-0100](adr/0100-grow-glance-into-a-homelab-page-and-keep-the-operators-lists-in-sops.md) it has a page per subject โ€” Home, Read, Watch, Sports & Markets, Homelab, Network & Storage and Maintenance โ€” the estate's read from `prometheus`, and the operator's own lists come from SOPS. It also serves the household's page at `home.matrix.elysium` ([ADR-0103](adr/0103-retire-homepage-and-serve-the-household-page-from-glance.md)), which took over from Homepage. Mosquitto and ring-mqtt bring the Ring Alarm into Home Assistant, the broker on an internal network that only those two join, **deployed 2026-10-22** ([ADR-0104](adr/0104-bring-the-ring-alarm-into-home-assistant-through-ring-mqtt.md)). Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | ๐ŸŸก 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 โ€” from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) ยง6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **deployed 2026-09-29**, whose clients are phones on Hicks and which therefore brought a fifth inbound rule by ADR-0050's count, the seventh to exist ([#140](https://github.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **deployed 2026-09-30** on the same terms, whose Hicks pass was created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy โ€” the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 โ†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. MakeMKV, the disc ripper, is in the file behind the compose profile `rip` and is **not yet deployed**: it reads this host's DVD-RW and writes only into `erebor/media/rips`, runs only while someone is ripping, and is reached from Hicks on `5800` through a pass ยง6.11 creates ([ADR-0114](adr/0114-rip-discs-on-smaug-with-makemkv-started-only-to-rip.md)). The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file โ€” still no agent, still nothing initiating from 40 โ€” and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | +| `smaug` (10.0.40.30) | ๐ŸŸก 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, its inbound rules verified in position (four that day; [`network.md`](network.md#casabonita--vlan-40--media) counts them now), the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 โ€” from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) ยง6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **deployed 2026-09-29**, whose clients are phones on Hicks and which therefore brought a fifth inbound rule by ADR-0050's count, the seventh to exist ([#140](https://github.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **deployed 2026-09-30** on the same terms, whose Hicks pass was created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy โ€” the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 โ†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. MakeMKV, the disc ripper, is in the file behind the compose profile `rip` and is **not yet deployed**: it reads this host's DVD-RW and writes only into `erebor/media/rips`, runs only while someone is ripping, and is reached from Hicks on `5800` through a pass ยง6.11 creates ([ADR-0114](adr/0114-rip-discs-on-smaug-with-makemkv-started-only-to-rip.md)). The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file โ€” still no agent, still nothing initiating from 40 โ€” and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | | `bahamut` (10.0.30.50) | ๐ŸŸข 30 | *(none โ€” Windows)* | Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` โ€” Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)) and promoted for `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | ๐ŸŸข 30 | *(none โ€” Windows)* | Windows Server 2025 second domain controller and DNS โ€” Tier 0. Static, for the same reason. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)) and promoted for `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | ๐ŸŸข 30 | *(none โ€” Windows)* | Windows Server 2025 file and member server โ€” the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound โ€” Tier 1. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)) and joined to `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 53ac3b96..7ebbecc7 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -251,6 +251,7 @@ first row here on ImaginationLAN. `golem`'s `2049` pass is ImaginationLAN's othe | Hicks (50) | `tcp` `vlan50 net` โ†’ `10.0.40.30` port `13378` | `Allow 13378 to smaug` | **above** *Block access to CasaBonita* โ€” ยง6.5 | | Hicks (50) | `tcp` `vlan50 net` โ†’ `10.0.40.30` port `445` | `Allow SMB to smaug` | **above** *Block access to CasaBonita* โ€” ยง5 | | ImaginationLAN (30) | `tcp` `10.0.30.20` โ†’ `10.0.40.30` port `2049` | `Allow NFS from Saruman to smaug` | **above** *Block access to CasaBonita* on `igc0.30` โ€” ยง5b | +| ImaginationLAN (30) | `tcp` `10.0.30.80` โ†’ `10.0.40.30` port `2049` | `Allow NFS from golem to smaug` | **above** *Block access to CasaBonita* on `igc0.30` โ€” [`build-the-backup-guest.md`](build-the-backup-guest.md) ยง4 | | Hicks (50) | `tcp` `vlan50 net` โ†’ `10.0.40.30` port `5800` | `Allow 5800 to smaug` | **above** *Block access to CasaBonita* โ€” ยง6.11, **specified, not yet created** | **Four, where [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) @@ -900,8 +901,8 @@ mkdir -p /mnt/erebor/apps/navidrome/data /mnt/erebor/media/music \ > ``` > > The mountpoint is asked for rather than written down, because where TrueNAS -> keeps its volumes is its business. Then re-fetch the two files (the `curl` -> lines above), `docker compose up -d`, and prove the state came across +> keeps its volumes is its business. Then re-fetch the two files (ยง6's +> `smaug-fetch.sh` block), `docker compose up -d`, and prove the state came across > before anything is removed: the users and the watch history are there in > the web UI, and `docker exec media-jellyfin ls /config/data` lists > `jellyfin.db`. Only then remove the orphan โ€” compose will warn about it on @@ -939,8 +940,8 @@ is why this stack still has no secrets file. `-it` because it prompts twice: docker exec -it media-navidrome navidrome user create --admin -u -n ``` -**Updating the stack is the same two `curl` lines and `docker compose up -d` -again.** Nothing on this host pulls from `main` on its own: there is no +**Updating the stack is the same `smaug-fetch.sh` block and +`docker compose up -d` again.** Nothing on this host pulls from `main` on its own: there is no converge timer here, so a Dependabot bump that merges is not deployed until someone does this. That is a residual of ADR-0040's shape and not a defect in it, and it wants a line in `stacks/media/README.md` rather than an issue until @@ -1287,13 +1288,16 @@ PATH=/usr/sbin:/usr/bin:/sbin:/bin timeout 600 /bin/bash /mnt/erebor/apps/stack/ ``` `--print` writes nothing. Read three things off it: `homelab_smart_devices` -is **3**; which `/dev/sdX` carries `model="INTEL SSDSC2BB240G7"` on its -`homelab_smart_healthy` line and `4` on its `homelab_smart_reallocated_sectors` -line โ€” **if it is not `sdc`, the baseline row is wrong and step 4 fixes -it**, because the S3520 sits on the chipset AHCI, the Exos pair on the -MegaRAID, and nothing but this run says which letter it drew; and that -`sdb`, if it is still the faulted disk, answered at all (the `timeout` is -there for the case it did not). Each piece of that command line is +is **3**; that the line carrying `model="INTEL SSDSC2BB240G7"` has +`slot="pci-0000:00:17.0-ata-6"` and `4` on its +`homelab_smart_reallocated_sectors` line โ€” **if the slot differs, the +baseline row is wrong and step 4 fixes it**. The row is keyed on the port +since [ADR-0066](../adr/0066-key-smart-series-on-the-port-not-the-letter.md), +because the letter a disk draws is not fixed: since +[ADR-0052](../adr/0052-cable-smaugs-pool-to-the-chipset-and-take-the-megaraid-out.md) +all three disks share the chipset AHCI, and the S3520 has been both `sdc` +and `sdb`. And that both Exos answered at all (the `timeout` is there for +the case one did not). Each piece of that command line is load-bearing: cron's default `PATH` has no `/usr/sbin`, which is where `smartctl` lives; `--host` pins the label to the `instance` ยง6.1's target gives the scrape, rather than to whatever `hostname` says. @@ -1352,7 +1356,7 @@ That re-renders `smart-baselines.prom`; confirm with before step 5. Nothing is silenced, here or later: a silence matches labels and not values, and the table is where a static count belongs. -**5. The redeploy.** From the console shell, the two `curl` lines from ยง6 for +**5. The redeploy.** From the console shell, ยง6's `smaug-fetch.sh` block for `compose.yaml` and `.env` โ€” the compose file now carries the mount and the `.env` the path โ€” then: @@ -1516,7 +1520,7 @@ service that was not there yet. Step 8 is where that stopped being allowed. && mkdir -p /mnt/erebor/media/audiobooks ``` -3. **Re-fetch the stack's two files and bring it up** โ€” ยง6's two `curl` lines +3. **Re-fetch the stack's two files and bring it up** โ€” ยง6's `smaug-fetch.sh` block for `compose.yaml` and `.env.example`, then: ```bash @@ -1613,7 +1617,7 @@ absent from the snapshot. Step 8 is where that stopped being allowed. ยง0.5 table gives it, and read its position from `morpheus` with the ยง0.6 `pfctl` line: it sits above *Block access to CasaBonita* on `igc0.50`. 2. **The directories**, as root on this host โ€” the Navidrome block in ยง6. -3. **Re-fetch the two files and bring it up** โ€” the `curl` lines in ยง6, +3. **Re-fetch the two files and bring it up** โ€” ยง6's `smaug-fetch.sh` block, then `docker compose up -d`. The services already running are unchanged and compose leaves them running. 4. **Create the admin, at once** โ€” the `docker exec -it โ€ฆ user create` @@ -1775,8 +1779,12 @@ it printed a SKIP naming this section. **After a TrueNAS upgrade** the next day's file carries the new version, and `check-versions` fails until `hardware.md` and `network.md` say it โ€” which is the check doing its job. -> **Not yet done.** The date and the two readings from step 4 go here, and -> [#616](https://github.com/Gerrrt/HomeLab/issues/616) closes on them. +> **Steps 1โ€“3 done, read on `smaug` 2026-11-13; step 4 not yet read.** The +> cron job `homelab truenas-version (#616)` exists at 08:40 daily, and +> `truenas-version.prom` carries `truenas_version_info` for `25.10.7`, with +> `node_textfile_scrape_error` 0 on the exporter. The `check-versions` +> reading from the monitoring host still goes here, and +> [#616](https://github.com/Gerrrt/HomeLab/issues/616) closes on it. ### ยง6.8 โ€” Turn leaf-state collection on @@ -2111,7 +2119,7 @@ measured. as 65534, write into `rips/`. `erebor/apps` is on the Apps preset and takes the mode, so the config directory uses `install`, as Jellyfin's did. -4. **Re-fetch the two files** with the `curl` lines in ยง6, then start a +4. **Re-fetch the two files** with ยง6's `smaug-fetch.sh` block, then start a session from the same folder. **Use the guarded start in *1. Start the ripper* in [`rip-a-disc.md`](rip-a-disc.md), never a bare `up`,** here and for every trial in step 5. It starts the container only if the sg node `.env` @@ -2234,7 +2242,7 @@ measured. - The `igc0.40` tripwire counter is **still zero** - `make backup-nas ARGS=--list` on the monitoring host shows a **complete** set, and the nightly `verify-backups` has passed at least once since (ยง6.2) -- Port 15 on `neo` reads PVID **40**, untagged, with `smaug`'s MAC learned on it +- `ether17` on the CRS326 reads PVID **40**, untagged, with `smaug`'s MAC learned on it in VLAN 40 โ€” read in the switch UI, and **not** inferred from the host having an address (ยง0.2b) - The `igc0.40` tripwire's **packet** count is **still zero** โ€” its evaluation @@ -2275,11 +2283,11 @@ measured. sends the newest of each beyond it, on the second age recipient's medium, by [`copy-the-backups-offsite.md`](copy-the-backups-offsite.md); what is left is the first visit, which is [#573](https://github.com/Gerrrt/HomeLab/issues/573)'s last box. -- **SMART is decided and not yet running** โ€” ยง6.4 waits on the console; and +- **SMART runs**, since ยง6.4 was done on 2026-09-21; and **patch state is decided and will not run**, by [ADR-0047](../adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md), which closed [#483](https://github.com/Gerrrt/HomeLab/issues/483). The vdev-state textfile [`replace-the-nas-disk.md`](replace-the-nas-disk.md) asked for rides the same mechanism. It is ยง6.8, built under - [#744](https://github.com/Gerrrt/HomeLab/issues/744), and runs once that - section has been done at the console. + [#744](https://github.com/Gerrrt/HomeLab/issues/744), and runs every five + minutes since that section was done on 2026-10-02. diff --git a/docs/security.md b/docs/security.md index 91674877..e100bb84 100644 --- a/docs/security.md +++ b/docs/security.md @@ -377,6 +377,19 @@ leaves is up to a day between a replacement and the page; a template built inside that day is the one to suspect. Root on `Saruman` maps to `pippin`, which owns that one directory, so the hypervisor is not root on the NAS. +**And one more from ImaginationLAN, for the backups, and it exists.** +`10.0.30.80 โ†’ 10.0.40.30:2049`, `Allow NFS from golem to smaug`, carries +`golem`'s PBS datastore on `erebor/pbs` +([ADR-0053](adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md)). +It was created on 2026-10-03 by +[`build-the-backup-guest.md`](runbooks/build-the-backup-guest.md) ยง4; +`golem` mounts the share through it, and `alexander` is refused. Its +residual is the same address trust as `Saruman`'s export, and two things +bound it: every backup is encrypted on `Saruman` before it leaves, so a host +that took `golem`'s address reads ciphertext, and what it could still do, +destroy the datastore, is what the fourteen daily snapshots TrueNAS keeps of +`erebor/pbs` are for. Root on `golem` maps to `backup`, uid 34, which owns +that one dataset. **One more from Hicks is specified, not created, and it writes.** `vlan50 net โ†’ 10.0.40.30:5800`, `Allow 5800 to smaug`, is for MakeMKV, the disc ripper @@ -442,7 +455,7 @@ them, so the residual is that `prometheus` learns whether each is up. The exporter that sends the echo keeps `cap_drop: [ALL]`. It pings over an unprivileged datagram socket, not a raw one. -One of CasaBonita's six is a monitoring pass, and it carries a residual worth +One of CasaBonita's passes is a monitoring pass, and it carries a residual worth naming. `10.0.99.20 โ†’ 10.0.40.30:9100` lets Prometheus scrape `node_exporter` on the NAS ([#256](https://github.com/Gerrrt/HomeLab/issues/256)), and that endpoint is **unauthenticated**. The firewall rule is what stops other segments diff --git a/stacks/media/README.md b/stacks/media/README.md index 7e4ee2e9..f333491b 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -15,9 +15,11 @@ and [ADR-0040] gave TrueNAS. **Deployed 2026-09-19**, from a copy of this directory's two files on the pool, under TrueNAS's own Docker; [`build-the-nas.md`] ยง6 is the procedure. `make up STACK=media` does **not** work on that host โ€” it renders secrets this stack does not have, on a box -without `make` or `sops` โ€” so a change here reaches `smaug` by re-fetching -`compose.yaml` and `.env.example` from `main` into `/mnt/erebor/apps/stack` -and running `docker compose up -d` there. Nothing pulls from `main` on that +without `make` or `sops` โ€” so a change here reaches `smaug` by fetching +`compose.yaml` and `.env.example` from one pinned commit into +`/mnt/erebor/apps/stack`, with the block +[`scripts/smaug-fetch.sh`](../../scripts/smaug-fetch.sh) prints, and running +`docker compose up -d` there. Nothing pulls from `main` on that host on its own: a Dependabot bump is merged here and deployed there by hand. A third file lives beside them since [ADR-0047]: `scripts/collect-smart-state.sh`, fetched the same way and run by TrueNAS's cron โ€” a change to it reaches @@ -108,17 +110,18 @@ every television would have to trust, and a second thing to be down. | A Hicks workstation | Two of the rules in [`build-the-nas.md`] ยง0.5 โ€” `50 โ†’ 10.0.40.30:443` and `50 โ†’ 10.0.40.30:8096`, one per port | | A Hicks phone, on `13378` | `50 โ†’ 10.0.40.30:13378`, `Allow 13378 to smaug` โ€” the fifth by [ADR-0050]'s count and the seventh to exist โ€” **created by [`build-the-nas.md`] ยง6.5, and reached from a Hicks workstation on 2026-09-29** ([ADR-0050]) | | Prometheus, on `9100` | A third โ€” `10.0.99.20 โ†’ 10.0.40.30:9100` | -| Prometheus, on `22` | The fourth rule โ€” `10.0.99.20 โ†’ 10.0.40.30:22`, inert until [`build-the-nas.md`] ยง6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | +| Prometheus, on `22` | The fourth rule โ€” `10.0.99.20 โ†’ 10.0.40.30:22`, live since [`build-the-nas.md`] ยง6.2 switched SSH on for the backup pull on 2026-09-19, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | | A Hicks phone, on `4533` | `50 โ†’ 10.0.40.30:4533`, `Allow 4533 to smaug`, for Navidrome โ€” **created 2026-09-22**, ahead of the service and of the 13378 pass, so it is the fifth that exists; 445 is the sixth, and 13378 the seventh (ยง6.6) | | A Hicks workstation, on `445` | `50 โ†’ 10.0.40.30:445`, `Allow SMB to smaug`, to mount the `media` share as `samwise` โ€” **created 2026-09-23** by [`build-the-nas.md`] ยง5's *Workstations* steps, and mounted from a Hicks workstation ([ADR-0051]) | | `Saruman`, on `2049` | `10.0.30.20 โ†’ 10.0.40.30:2049`, `Allow NFS from Saruman to smaug`, to mount `erebor/iso` as the ISO store. Not the media stack's, and listed because it is a way into this host. **Created 2026-10-01** by [`build-the-nas.md`] ยง5b, and mounted by `Saruman` ([ADR-0072]) | +| `golem`, on `2049` | `10.0.30.80 โ†’ 10.0.40.30:2049`, `Allow NFS from golem to smaug`, to mount `erebor/pbs` as the PBS datastore. Not the media stack's either, and listed for the same reason. **Created 2026-10-03** by [`build-the-backup-guest.md`](../../docs/runbooks/build-the-backup-guest.md) ยง4, and mounted by `golem` ([ADR-0053](../../docs/adr/0053-run-pbs-on-saruman-with-its-datastore-on-smaug-over-nfs.md)) | | A Hicks workstation, on `5800` | `50 โ†’ 10.0.40.30:5800`, `Allow 5800 to smaug`, for MakeMKV's web GUI while a disc is being ripped. **Specified, not yet created**: [`build-the-nas.md`] ยง6.11 creates it ([ADR-0114](../../docs/adr/0114-rip-discs-on-smaug-with-makemkv-started-only-to-rip.md)) | | Everything else on the estate | Not at all โ€” default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and -here there are five: every screen in the house, one workstation, the phones, -the monitoring host, and `Saruman`, which mounts the ISO store and is the -only consumer from ImaginationLAN ([ADR-0072]). The phones are the first consumers of a service on +here there are six: every screen in the house, one workstation, the phones, +the monitoring host, and from ImaginationLAN `Saruman`, which mounts the ISO +store ([ADR-0072]), and `golem`, which mounts the PBS datastore. The phones are the first consumers of a service on this host that are across a segment boundary rather than on it โ€” Jellyfin's case for publishing to the segment was that its clients live there, and Audiobookshelf's and Navidrome's do not. @@ -132,9 +135,10 @@ of this host's filesystems, uptime and load, by the televisions โ€” and `docs/security.md` records it rather than the firewall rule being mistaken for a boundary it is not. -The NFS export on `2049` is not one of those three media ports, and it -differs in kind. A device on CasaBonita can open the port, but the export -itself admits `10.0.30.20` alone, so the share answers only `Saruman`. The +The NFS exports on `2049` are not among those three media ports, and they +differ in kind. A device on CasaBonita can open the port, but each export +admits one address, `10.0.30.20` for the ISO store and `10.0.30.80` for +`erebor/pbs`, so each share answers only `Saruman` or only `golem`. The host scope is enforced twice, by the firewall rule and by the export, and only the export applies on the segment. @@ -246,7 +250,7 @@ operator's password manager, beside pfSense's and iLO's. `docs/security.md` class, and names what would retire it. What that costs is one command: `make render STACK=media` dies on the missing -file, and that is expected. This stack is deployed by the two `curl` lines in +file, and that is expected. This stack is deployed by the `smaug-fetch.sh` block in [`build-the-nas.md`] ยง6, never by `make up`. **Nobody needs the password to get back in.** From a workstation on Hicks, diff --git a/stacks/media/compose.yaml b/stacks/media/compose.yaml index 33bb4ab4..d0a841f6 100644 --- a/stacks/media/compose.yaml +++ b/stacks/media/compose.yaml @@ -12,8 +12,9 @@ # Deploy with: NOT `make up STACK=media`. That target renders a secrets file # this stack does not have โ€” by decision, #528 and # docs/security.md ยง Secrets โ€” on a box with neither make nor sops. -# A change here reaches smaug by re-fetching compose.yaml and -# .env.example from main into that folder and running +# A change here reaches smaug by fetching compose.yaml and +# .env.example from one pinned commit into that folder, with the +# block scripts/smaug-fetch.sh prints, and running # `docker compose up -d` there, by hand โ€” nothing on smaug pulls # from main on its own. README.md has the procedure. A third # file sits beside those two since ADR-0047: From 23b819c508413e0a3779d54281cf7e00f26bcddb Mon Sep 17 00:00:00 2001 From: Gerrrt Date: Fri, 13 Nov 2026 22:33:18 -0800 Subject: [PATCH 2/2] =?UTF-8?q?docs(smaug):=20record=20#616's=202026-09-23?= =?UTF-8?q?=20completion=20in=20=C2=A76.7,=20apart=20from=20the=202026-11-?= =?UTF-8?q?13=20re-read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/runbooks/build-the-nas.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 7ebbecc7..80c5ec18 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -1779,12 +1779,18 @@ it printed a SKIP naming this section. **After a TrueNAS upgrade** the next day's file carries the new version, and `check-versions` fails until `hardware.md` and `network.md` say it โ€” which is the check doing its job. -> **Steps 1โ€“3 done, read on `smaug` 2026-11-13; step 4 not yet read.** The -> cron job `homelab truenas-version (#616)` exists at 08:40 daily, and -> `truenas-version.prom` carries `truenas_version_info` for `25.10.7`, with -> `node_textfile_scrape_error` 0 on the exporter. The `check-versions` -> reading from the monitoring host still goes here, and -> [#616](https://github.com/Gerrrt/HomeLab/issues/616) closes on it. +> **Done 2026-09-23.** The collector was installed as the root cron job above, +> and Prometheus read `truenas_version_info{host="smaug", release="25.10", +> version="25.10.7"} 1`. `make check-versions`, run that day, printed both +> PASS lines for `smaug`, `docs/hardware.md` and `docs/network.md` each +> ending `truenas_version_info agrees`, where it had printed a SKIP. That is +> [#616's closing comment](https://github.com/Gerrrt/HomeLab/issues/616#issuecomment-5799057014), +> and the issue closed on it. +> +> **Re-read on `smaug` 2026-11-13**, locally, by a read-only audit: the cron +> job `homelab truenas-version (#616)` still exists at 08:40 daily, +> `truenas-version.prom` carries `25.10.7`, and `node_textfile_scrape_error` +> is 0 on the exporter. `check-versions` was not re-run then. ### ยง6.8 โ€” Turn leaf-state collection on