Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ hole from the monitoring VLAN into the monitored one.
| `fenrir` (10.0.30.90) | 🟢 30 | [`stacks/sensor`](../stacks/sensor) | Zeek on a `tc` mirror of `Saruman`'s lab bridge, so that the domain's east-west traffic, which crosses no router and so never reaches Suricata on `morpheus`, is observed at all ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)). A guest on `Saruman`, VMID 190, with a second NIC alone on the port-less `vmbr1` that receives only the mirror's copies. Its logs go to `alexander`, never to VLAN 99; whether the mirror is delivering is answered on the hypervisor by `homelab_zeek_mirror_active`. Built 2026-09-30 by [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md), and proved by rebooting `Saruman` with the mirror's timer disabled: the gauge fell to 0 and `ZeekMirrorInactive` fired. Alloy agent (Docker) |
| `eden` (10.0.30.41) | 🟢 30 | [`stacks/bloodhound`](../stacks/bloodhound) | Off between sessions: BloodHound CE over one Postgres that holds its graph too, for attack-path analysis of `ad.matrix.elysium` — which path to Domain Admin exists, so that `odin`, `fenrir` and Sysmon can be asked whether they saw it ([#451](https://github.com/Gerrrt/HomeLab/issues/451), [ADR-0081](adr/0081-run-bloodhound-ce-on-a-saruman-guest.md)). A guest on `Saruman`, VMID 141, tagged `on-demand` so that its being off is not an alert (ADR-0079), and in no backup job: the graph is one collection run from rebuilt. Its UI is TLS on a leaf from the estate's CA, reached from Hicks; the collector's upload is intra-segment, so it adds no firewall rule. The graph is a map of the lab's weaknesses and never leaves the segment; its Alloy pushes the guest's own telemetry to `alexander`. Built 2026-10-07 by [`build-the-bloodhound-guest.md`](runbooks/build-the-bloodhound-guest.md), a full clone of template 901. Alloy agent (Docker) |
| `diabolos` (10.0.30.61) | 🟢 30 | [`stacks/scratch`](../stacks/scratch) | **Not built yet**, and torn down between investigations: a DISPOSABLE copy of `stacks/soc` (Wazuh and Velociraptor on the same digests, soc's config mounted unchanged) for detonations and one-off questions, so that noise never spends `odin`'s shard budget or enters its record ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)). A guest on `Saruman`, VMID 161, created with the Proxmox tag `disposable`; the hypervisor's guest-state collector reports the tag and the guest's age, and `DisposableGuestOutlived` fires on the estate when it has existed for a fortnight, running or stopped. No Alloy, no scrape, no backup and no committed secrets: its age key and `secrets/scratch.sops.yaml` are made on the guest and destroyed with it. Built and destroyed by [`run-a-scratch-investigation.md`](runbooks/run-a-scratch-investigation.md) |
| `garuda` (10.0.30.62) | 🟢 30 | [`stacks/analyst`](../stacks/analyst) | Defense's analyst workstation: Kali Purple **without its SOC**, a seat for working the alerts `odin` raises, with CyberChef, Wireshark, and the [`dotfiles-Defense`](https://github.com/dotgibson/dotfiles-Defense) layer on `dotfiles-Debian` ([#921](https://github.com/Gerrrt/HomeLab/issues/921), [ADR-0091](adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)). Purple's SIEM and sensor daemons stay off, because Wazuh, Zeek and Suricata already run; Defense's own `siemup` lab comes up for an exercise only. A guest on `Saruman`, VMID 162, cloned by `tofu/` from template 910 into the `analyst` pool, **always on** and not tagged `on-demand`, so a stopped `garuda` is an alert (ADR-0079), and in `golem`'s backup job, because its case notes are in no repository. Standalone, not a domain member: whoever takes the domain must not also take the notes. Its stack is the collector alone, pushing to `alexander`. It is enrolled in `odin`'s Wazuh and Velociraptor from Debian packages staged on `odin` ([ADR-0092](adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)); OpenVAS runs on demand and scans the domain's six only, fenced by nftables on the scanner's cgroup ([ADR-0093](adr/0093-run-openvas-on-garuda-on-demand-scoped-to-the-domain-by-nftables.md), [`scan-the-lab-with-openvas.md`](runbooks/scan-the-lab-with-openvas.md)). Built 2026-10-10 by [`build-the-analyst-workstation.md`](runbooks/build-the-analyst-workstation.md), a full clone of template 910. Alloy agent (Docker) |
| `garuda` (10.0.30.62) | 🟢 30 | [`stacks/analyst`](../stacks/analyst) | Defense's analyst workstation: Kali Purple **without its SOC**, a seat for working the alerts `odin` raises, with CyberChef, Wireshark, and the [`dotfiles-Defense`](https://github.com/dotgibson/dotfiles-Defense) layer on `dotfiles-Debian` ([#921](https://github.com/Gerrrt/HomeLab/issues/921), [ADR-0091](adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)). Purple's SIEM and sensor daemons stay off, because Wazuh, Zeek and Suricata already run; Defense's own `siemup` lab comes up for an exercise only. A guest on `Saruman`, VMID 162, cloned by `tofu/` from template 910 into the `analyst` pool, **always on** and not tagged `on-demand`, so a stopped `garuda` is an alert (ADR-0079), and in `golem`'s backup job, because its case notes are in no repository. Standalone, not a domain member: whoever takes the domain must not also take the notes. Its build makes SSH key-only from Hicks and `phoenix`, makes `sudo` ask, and keeps GitHub write credentials off it (§4 and §7); until those steps have run on it, the stock defaults stand. Its stack is the collector alone, pushing to `alexander`. It is enrolled in `odin`'s Wazuh and Velociraptor from Debian packages staged on `odin` ([ADR-0092](adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)); OpenVAS runs on demand and scans the domain's six only, fenced at layer 2 by an nftables `netdev` egress hook on `scan0`, the scanner's own macvlan in its own network namespace ([ADR-0093](adr/0093-run-openvas-on-garuda-on-demand-scoped-to-the-domain-by-nftables.md), [`scan-the-lab-with-openvas.md`](runbooks/scan-the-lab-with-openvas.md)). Built 2026-10-10 by [`build-the-analyst-workstation.md`](runbooks/build-the-analyst-workstation.md), a full clone of template 910. Alloy agent (Docker) |
| `dot-debian` (10.0.30.91) | 🟢 30 | *(none — a test machine)* | Built 2026-10-08. Debian 13 (trixie), cloned from `tpl-debian-13` (903), where a release of the [dotgibson dotfiles](https://github.com/dotgibson/dotfiles-core)' Debian layer is bootstrapped on a real install. A guest on `Saruman`, VMID 191, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy |
| `dot-fedora` (10.0.30.92) | 🟢 30 | *(none — a test machine)* | Built 2026-10-08. Fedora Server 44, cloned from `tpl-fedora-server` (904), for the dotfiles' Fedora layer. A guest on `Saruman`, VMID 192, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy |
| `dot-opensuse` (10.0.30.93) | 🟢 30 | *(none — a test machine)* | Built 2026-10-09. openSUSE Tumbleweed, cloned from `tpl-opensuse-tw` (905), for the dotfiles' openSUSE layer. A guest on `Saruman`, VMID 193, tagged `on-demand` (ADR-0079) and off between runs, rolled back to its `clean` snapshot before each one ([#920](https://github.com/Gerrrt/HomeLab/issues/920), [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md), [`test-the-dotfiles-layers.md`](runbooks/test-the-dotfiles-layers.md)). Standalone, not a domain member. No scrape, no backup: it is rebuilt from its template, and runs no Alloy |
Expand Down
26 changes: 26 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,32 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-13

- **An audit of `garuda` found SSH open to the lab domain, and owner GitHub
credentials on it.** A read-only audit against this repository found the role
intact: no Purple SOC daemons, GVM stopped, the fence files identical to
`stacks/analyst/gvm/`, not a domain member, Wazuh and Velociraptor connected.
It also found Kali's stock `sshd` taking passwords on every address, with
cloud-init's passwordless `sudo` for `analyst`, on the segment the domain's
six share. And it found a `gh` login with the `repo` and `admin:public_key`
scopes, beside an SSH key for GitHub with no passphrase. This change writes
the fix into the build:
- `stacks/analyst/ssh/10-garuda.conf`, installed by
`build-the-analyst-workstation.md` §4, which makes SSH key-only from Hicks
and `phoenix` with forwarding off;
- the same step removes the passwordless `sudo`;
- §7 reads the dotfiles release tags without a GitHub login, and `garuda`
holds no GitHub write credential. It also has the steps that revoke and
remove the two the audit found.

The box itself changes when §4 is run on it, which this entry does not
record. Three smaller items rode along:
- `docs/architecture.md` still described the cgroup fence ADR-0093 replaced;
- the Linux agent pin in `stacks/soc/linux-agents.yaml` moves from 4.14.8 to
the manager's 4.14.10, with the hash checked against Wazuh's signed index;
- every Alloy runs with `--disable-reporting`, as Grafana and Loki already do:
the six stacks' Compose files, and both paths in `scripts/deploy-agent.sh`.
`garuda`'s attempts to report were failing at the resolver anyway.

- **`gandalf`'s SSH takes keys only.** A read-only audit of `gandalf` found
`sshd` taking root's password (`permitrootlogin yes`,
`passwordauthentication yes`), which is the Proxmox installer's default. Only
Expand Down
85 changes: 79 additions & 6 deletions docs/runbooks/build-the-analyst-workstation.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ Enrolling `garuda` in Wazuh and Velociraptor is §12, phase 2. OpenVAS, phase
| Domain | **Not joined** | The domain is the target and is reverted. A member would fall with it, notes and all (ADR-0091) |
| SIEM and sensors | **Off** | Wazuh, Zeek and Suricata already run. Their binaries stay, for offline work on a pcap; their daemons do not |
| Backup | **Yes**, `golem`'s nightly job | Its case notes are not in git |
| SSH and `sudo` | **Keys only, from Hicks and `phoenix`; `sudo` asks** (§4) | The domain's six share the segment, and no firewall stands between them and `garuda` |
| GitHub | **Read-only:** clones over https, no `gh` login, no GitHub key | The estate converges from `Gerrrt/HomeLab` ([ADR-0021](../adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)). Whoever takes `garuda` must not also be able to write to it |
| Login user | `analyst` | Not `operator`: Debian ships a system group of that name, which broke cloud-init on the dotfiles VMs |

## 1. Build template 910, from `phoenix`
Expand Down Expand Up @@ -159,6 +161,42 @@ manager, not in this repository or in OpenTofu's state:
ssh -t analyst@10.0.30.62 sudo passwd analyst
```

**Then close SSH to keys, and make sudo ask.** Kali's stock `sshd` takes
passwords on every address, and cloud-init gives `analyst` `sudo` with no
password. On a segment shared with the domain's six, which exist to be
compromised, those two together are a guessed password away from root on the
box that holds the case notes. So once the console password exists, SSH stops
accepting it, from anywhere but Hicks and `phoenix`, and `sudo` starts asking
for it. [`stacks/analyst/ssh/10-garuda.conf`](../../stacks/analyst/ssh/10-garuda.conf)
is the drop-in. From `phoenix`'s checkout:

```bash
scp stacks/analyst/ssh/10-garuda.conf analyst@10.0.30.62:
ssh -t analyst@10.0.30.62 '
sudo install -m 0644 -o root -g root 10-garuda.conf /etc/ssh/sshd_config.d/ && rm 10-garuda.conf &&
sudo sshd -t && sudo systemctl reload ssh'
```

**Before going on, open a second session** and check it from there, so a
mistake cannot lock out the one you are in:

```bash
ssh analyst@10.0.30.62 true && echo key-ok
ssh -o PubkeyAuthentication=no analyst@10.0.30.62 true # must say "Permission denied (publickey)"
```

Then take away the passwordless `sudo`. `analyst` keeps `sudo` through the
`sudo` group, which asks for the password. Check that it does before removing
the grant, so a wrong password is found while the grant still stands:

```bash
ssh -t analyst@10.0.30.62 'sudo -k && sudo -v && echo sudo-password-ok'
ssh -t analyst@10.0.30.62 'sudo rm /etc/sudoers.d/90-cloud-init-users && sudo -k && sudo -n true 2>&1 | grep -q "password is required" && echo nopasswd-gone'
```

Every later step that uses `sudo` asks for the password, so an `ssh` that
runs `sudo` takes `-t`.

## 5. Kali Purple, without its SOC

On `garuda`:
Expand Down Expand Up @@ -291,8 +329,9 @@ GitHub. Each bootstrap links from wherever it is cloned, so moving a clone
means re-running its bootstrap:

```bash
deb=$(gh api repos/dotgibson/dotfiles-Debian/releases/latest --jq .tag_name)
def=$(gh api repos/dotgibson/dotfiles-Defense/releases/latest --jq .tag_name)
latest() { curl -fsSL "https://api.github.com/repos/dotgibson/$1/releases/latest" | jq -r .tag_name; }
deb=$(latest dotfiles-Debian)
def=$(latest dotfiles-Defense)
echo "Debian $deb, Defense $def" # into §11

mkdir -p ~/code/dotgibson
Expand All @@ -306,7 +345,26 @@ exec zsh
core doctor; echo "doctor exit $?"
```

Without `gh` on the guest, read the two tags from the releases pages.
The release API needs no login, and `jq` is §5's. **No GitHub credential on
`garuda`:** no `gh auth login`, and no key added to the account. Every clone is
over https and read-only (§0). A change found here is made from Hicks or
`phoenix`, which can push.

**On a guest built before this rule**, take the credentials off before §9's
check. The audit of 2026-11-13 found a `gh` login and a GitHub SSH key on
`garuda`. Revoke them first, from a trusted host: on github.com, *Settings →
SSH and GPG keys* deletes the key, and *Settings → Applications → Authorized
OAuth Apps → GitHub CLI → Revoke* ends the token (and `gh` on every other
machine, which then logs in again). Then remove the local copies on `garuda`:

```bash
gh auth logout
git credential-cache exit
rm -f ~/.ssh/id_ed25519 ~/.ssh/id_ed25519.pub
```

`git credential-cache exit` drops an https login git may be holding in memory;
the clones keep pulling without one.

A good run is:

Expand Down Expand Up @@ -366,6 +424,16 @@ Each line is a pass or a stop:
- `docker ps --format '{{.Names}}'` prints exactly `analyst-alloy` and
`analyst-docker-socket-proxy`;
- `ss -lntup` shows no Suricata, Zeek or Elastic listener.
- **Access** (§4, §7). On `garuda`:
- `sudo sshd -T | grep -iE '^(passwordauthentication|permitrootlogin|allowusers|disableforwarding) '`
prints `passwordauthentication no`, `permitrootlogin no`,
`disableforwarding yes`, and `allowusers` for `analyst@10.0.50.0/24` and
`analyst@10.0.30.70`. `sshd -T` still lists the separate forwarding
options as `yes`; `DisableForwarding` overrides them;
- `sudo -k; sudo -n true` says a password is required;
- `gh auth status` says it is not logged in, or `gh` is absent;
- no key in `~/.ssh` is on the GitHub account, and any private key there has
a passphrase: `ssh-keygen -y -P '' -f <key>` fails.
- **dotfiles.** §7's two bootstrap exits and the `core doctor` exit.
- **Desktop.** From `Hicks`, `garuda`'s console in Proxmox shows the Purple
XFCE session. Wireshark opens a capture, and CyberChef opens.
Expand Down Expand Up @@ -623,9 +691,9 @@ the Wazuh agent and the Velociraptor client, as Debian packages staged on

```bash
d=$(mktemp -d) && cd "$d"
curl -fsSO http://10.0.30.60:8448/wazuh-agent_4.14.8_amd64.deb
curl -fsSO http://10.0.30.60:8448/wazuh-agent_4.14.10_amd64.deb
curl -fsSO http://10.0.30.60:8448/velociraptor-client_0.77.3_amd64.deb
echo "6ed202e211fd8c544d1906c9b94dcd4b503d1829eef7d0a82ce4bcf4a1688967 wazuh-agent_4.14.8_amd64.deb" | sha256sum -c -
echo "873cfec3720a7d7a344afc2c646cf5895a04caa3e4dd7b5124987264f711db94 wazuh-agent_4.14.10_amd64.deb" | sha256sum -c -
sha256sum velociraptor-client_0.77.3_amd64.deb # must equal staging's hash
```

Expand All @@ -637,7 +705,7 @@ the Wazuh agent and the Velociraptor client, as Debian packages staged on

```bash
sudo WAZUH_MANAGER=10.0.30.60 WAZUH_AGENT_NAME=garuda WAZUH_AGENT_GROUP=default \
dpkg -i wazuh-agent_4.14.8_amd64.deb
dpkg -i wazuh-agent_4.14.10_amd64.deb
sudo sh -c 'umask 027; cat > /var/ossec/etc/authd.pass && chgrp wazuh /var/ossec/etc/authd.pass' # paste, Enter, Ctrl-D
sudo systemctl enable --now wazuh-agent && sudo systemctl restart wazuh-agent
```
Expand All @@ -646,6 +714,11 @@ the Wazuh agent and the Velociraptor client, as Debian packages staged on
`sudo rm /var/ossec/etc/authd.pass`. The agent keeps its own key in
`client.keys`. A re-enrolment writes the file again.

**To upgrade an enrolled agent** after the pin moves, stage again on
`odin`, fetch and check the new `.deb` as in step 2, and
`sudo dpkg -i` it with no variables. The package keeps `ossec.conf` and
`client.keys`, so the agent comes back as itself, with no password.

4. **The Velociraptor client.** The package carries `odin`'s server URL, CA and
nonce, and installs and starts its own service:

Expand Down
Loading
Loading