From 140f8d333f2d4f7dcf0ca56ebbaa98235db11362 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 14 Nov 2026 00:29:44 +0000 Subject: [PATCH] fix(sensitive): declare trinity's age-key deadline as verify-key-backup-sensitive trinity's key-recipients.prom has recorded both recipients as never proved since 2026-10-03, and SecretsKeyBackupUnproven could not fire for either: it joins against a verify-key-backup deadline that SENSITIVE_JOBS never declared. Declaring that name on trinity too would break ScheduledJobStale and ScheduledJobStaleCritical for the whole estate (on(homelab_job) becomes many-to-many, proved under promtool), so trinity's row has its own name. The two key-proof rules match either name, the stale rules exclude it, `make secrets-verify-backup STACK=sensitive` records it, and Glance's maintenance page judges it per recipient. Co-Authored-By: Claude Opus 5.5 --- Makefile | 4 +- docs/changelog.md | 19 +++ docs/runbooks/back-up-the-age-key.md | 20 +++ docs/runbooks/schedule-maintenance.md | 11 +- scripts/install-timers.sh | 14 +- .../prometheus/rules/backup.rules.yaml | 33 +++-- .../prometheus/tests/backup.test.yaml | 123 ++++++++++++++++-- stacks/sensitive/glance/pages/maintenance.yml | 10 +- 8 files changed, 205 insertions(+), 29 deletions(-) diff --git a/Makefile b/Makefile index fe72e53c..22156a4d 100644 --- a/Makefile +++ b/Makefile @@ -744,7 +744,9 @@ secrets-verify-backup: ## Check a backup age key decrypts the secrets (KEY=/path @# enforced from the other end: a successful run records its timestamp, and @# SecretsKeyBackupUnproven fires when that proof passes ninety days old. @# Nagging is not as good as running it, but it beats remembering. - ./scripts/run-scheduled.sh --job verify-key-backup --lock keys \ + @# STACK=sensitive records verify-key-backup-sensitive, trinity's row in + @# install-timers.sh: a job name may be declared on one host only. + ./scripts/run-scheduled.sh --job verify-key-backup$(if $(filter sensitive,$(STACK)),-sensitive) --lock keys \ -- ./scripts/verify-key-backup.sh "$(KEY)" $(STACK) .PHONY: certs-verify-backup diff --git a/docs/changelog.md b/docs/changelog.md index 2f2fc079..8d7d2b53 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -17,6 +17,24 @@ roadmap as it read that day, and the *Done* entries keep the shape they had there. `check_docs.py` does not check this file, for the reason its module docstring gives: it is a record, not a claim about now. +## 2026-11-14 + +- **`trinity`'s age-key deadline is declared, as + `verify-key-backup-sensitive`.** A read-only audit of `trinity` on + 2026-11-13 found both recipients in its `key-recipients.prom` at `0`, never + proved, since 2026-10-03, and `SecretsKeyBackupUnproven` unable to fire for + either: the rule joins against a `verify-key-backup` deadline that + `SENSITIVE_JOBS` never declared. Declaring that name on `trinity` too was + tried under promtool first and would have broken `ScheduledJobStale` and + `ScheduledJobStaleCritical` for the whole estate, because they join + `on(homelab_job)` and one name on two instances is many-to-many. So + `trinity`'s row has its own name, the two key-proof rules match either name, + the stale rules exclude it, `make secrets-verify-backup STACK=sensitive` + records it, and Glance's maintenance page reads it per recipient. This + records the change, not the deploy: the row reaches `trinity` with + `make install-timers PROFILE=sensitive` there, and both recipients will then + alert until each offline copy is proved. + ## 2026-11-13 - **An audit of `golem`, read-only, found its last verify failed and a @@ -52,6 +70,7 @@ docstring gives: it is a record, not a claim about now. - **Corrected here.** `architecture.md`'s `golem` row said the nightly job takes `odin` and the domain's six. It takes `garuda` too, as §8 and `tofu/guests.tf` already said, and `eden` stays out. +||||||| parent of 8e49661 (fix(sensitive): declare trinity's age-key deadline as verify-key-backup-sensitive) ## 2026-11-12 diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index 0b12a0c7..d36b4da5 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -241,6 +241,26 @@ this is the same alert it always was. With two, one timestamp for both would mean that proving either copy vouched for the other — which is exactly backwards, because the point of the second copy is that it fails independently. +### On trinity + +`trinity` holds its own key, which opens only +[`secrets/sensitive.sops.yaml`](../../secrets/sensitive.sops.yaml), and the +same deadline applies to every recipient of that file. Its row is +`verify-key-backup-sensitive` in `SENSITIVE_JOBS`, not `verify-key-backup`, +because the alert rules join on the job name alone and a name declared on two +hosts breaks them. Prove each copy from the checkout on `trinity`, so the proof +lands in `trinity`'s own `key-recipients.prom`: + +```bash +make secrets-verify-backup KEY=/path/to/copy/keys.txt STACK=sensitive +``` + +`STACK=sensitive` is what records the run under `trinity`'s job name; without +it the make target checks the estate's file instead. `trinity` has no +`recipient-state` timer, so `make recipient-state STACK=sensitive` is what +rewrites its recipient list after a change that did not go through +`add-recipient.sh`. + Nagging is not as good as running it. It is a great deal better than remembering. The threshold lives in the `JOBS` table in [`install-timers.sh`](../../scripts/install-timers.sh); the mechanism is in diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index b4c7f3c9..eacbb9a7 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -739,6 +739,7 @@ checkout ([#404](https://github.com/Gerrrt/HomeLab/issues/404) step 9): | `verify-backups-sensitive` | `make verify-backups` with `STACK=sensitive` | daily 06:30 | 2 days | | `household-copy` | **you**, `make household-copy DEST=…` | no timer | 90 days | | `household-proof` | **you** and the holder, `make household-proof CODE=…` | no timer | 1 year | +| `verify-key-backup-sensitive` | **you**, `make secrets-verify-backup KEY=… STACK=sensitive` | no timer | 90 days | It runs daily, not weekly like the estate's backup, because it holds the password vault. Each run also copies the set to `oracle`, the same way the @@ -775,10 +776,14 @@ make install-timers PROFILE=sensitive ``` That writes a `homelab-jobs.prom` on `trinity` that declares only these -six. The last two are the household's copy +seven. `household-copy` and `household-proof` are the household's copy ([ADR-0073](../adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md), [`carry-the-household-copy.md`](carry-the-household-copy.md)). They have no -timer, and `HouseholdCopyStale` reads them. The alert rules join on the job name alone, so a name may appear in only +timer, and `HouseholdCopyStale` reads them. `verify-key-backup-sensitive` is +the estate's `verify-key-backup` for the tier's own age recipients: no timer, +the same ninety days, and `SecretsKeyBackupUnproven` holds every recipient in +`trinity`'s `key-recipients.prom` to it +([`back-up-the-age-key.md`](back-up-the-age-key.md#on-trinity)). The alert rules join on the job name alone, so a name may appear in only one table, and `--check` enforces that. The installer primes `backup-library`, `converge-sensitive` and `verify-backups-sensitive`, which stop nothing, and not `backup-sensitive`, which stops the tier. `make check-timers` checks both profiles, and `make validate` on `trinity` fails until the backup and converge timers are @@ -904,7 +909,7 @@ expected rather than a second fault. | Symptom | Cause | Fix | | --- | --- | --- | -| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key, for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-.service` and read the journal | +| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key (on `trinity`, `verify-key-backup-sensitive`), for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-.service` and read the journal | | `ScheduledJobStaleCritical` | A job has gone three of its own periods without a success — one and a half of the allowance in the `JOBS` table, which is twice the period; a weekly backup three weeks stale, a nightly one three nights, and never under a day for the hourly collectors — and the `ScheduledJobStale` warning for it has been repeating on the in-house channel unanswered | The same first look as the warning: `systemctl list-timers 'homelab-*'`, then `journalctl -u homelab-.service`. The rows below name the usual cause per job. The page holds the warning back while it is up, so a job that recovers clears both at once | | `OffsiteCopyStale` | Ninety days since the newest sets were last copied to the second recipient's medium and proved there | Mount it and run `make backup-offsite DEST=…` — [`copy-the-backups-offsite.md`](copy-the-backups-offsite.md). `--list`, `--verify-only` and `--prune` do not clear it, on purpose | | `CaKeyBackupUnproven` | No offline copy of `certificates/ca-key.pem` has been proved in ninety days — or ever, or not since the CA was re-minted, which the fingerprint label tells apart | Mount the medium and `make certs-verify-backup KEY=…` ([`back-up-the-ca-key.md`](back-up-the-ca-key.md)). After a re-mint, copy the new key there first; the old copy is refused | diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index be77b5ba..b13da745 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -184,7 +184,13 @@ JOBS=( # what keeps the alert joins one-to-one across hosts, so it must not reuse a # name from JOBS — check 8 below asserts that. And the nightly re-verification # of both kinds of set, here and on oracle (#856): verify-backups-sensitive, -# two days for the same reason as the backups it re-reads. +# two days for the same reason as the backups it re-reads. And +# verify-key-backup-sensitive, the estate's verify-key-backup for this tier's +# own recipients: the same ninety days and no unit, because it is the same +# human proof against a copy on removable media, run on trinity as +# `make secrets-verify-backup KEY=... STACK=sensitive`. SecretsKeyBackupUnproven +# holds every recipient in trinity's key-recipients.prom to it. It cannot be +# called verify-key-backup, for check 8's reason: that name is the estate's. # # The units under systemd/sensitive/ carry @DEPLOY_ROOT@, @RUN_USER@, # @RUN_GROUP@ and @RUN_HOME@ rather than a name, because the build runbook @@ -198,6 +204,7 @@ SENSITIVE_JOBS=( "verify-backups-sensitive homelab-verify-backups-sensitive 172800 verify-backups" "household-copy - 7776000 household-copy" "household-proof - 31536000 household-proof" + "verify-key-backup-sensitive - 7776000 secrets-verify-backup" ) die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } @@ -524,8 +531,9 @@ if [[ "${MODE}" == check ]]; then run_home="$(getent passwd "${run_user}" | cut -d: -f6)" for row in "${SENSITIVE_JOBS[@]}"; do unit="$(job_unit "${row}")" - # household-copy and household-proof are human jobs, "-" as on the - # estate's side: no timer to be enabled (ADR-0073). + # household-copy, household-proof and verify-key-backup-sensitive are + # human jobs, "-" as on the estate's side: no timer to be enabled + # (ADR-0073, ADR-0024). [[ "${unit}" == "-" ]] && continue state="$(systemctl is-enabled "${unit}.timer" 2>/dev/null || true)" case "${state}" in diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 37ac3d84..05a8cebf 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -82,7 +82,7 @@ groups: # shape, and HouseholdCopyStale reads them (ADR-0073). expr: | (time() - homelab_job_last_success_timestamp_seconds{ - homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) + homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) > on(homelab_job) group_left() homelab_job_max_age_seconds for: 30m labels: @@ -127,7 +127,7 @@ groups: # that rule exists to avoid. expr: | (time() - homelab_job_last_success_timestamp_seconds{ - homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) + homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) > on(homelab_job) group_left() clamp_min(1.5 * homelab_job_max_age_seconds, 86400) for: 30m labels: @@ -265,9 +265,19 @@ groups: # had no left side, and the deadline went unwatched from the day it was # deployed (#400). SecretsKeyRecipientsUnrecorded, below, is the rule # for that state, and the recipient-state timer is what ends it. + # + # Two declaration names, one per host. trinity proves the sensitive + # tier's recipients and declares the same ninety days as + # verify-key-backup-sensitive, because a name may be in only one of + # install-timers.sh's tables: ScheduledJobStale and the rules beside it + # join on(homelab_job) alone, and one name declared on two instances + # makes those joins many-to-many, so they error rather than fire. Here + # the join is on(instance), and each host declares exactly one of the + # two, so the regex still finds one row per host. expr: | (time() - homelab_key_recipient_last_proof_timestamp_seconds) - > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + > on(instance) group_left() + homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"} for: 1h labels: component: backup @@ -280,7 +290,7 @@ groups: recoverable from anywhere but this host — every other job runs on the same machine, with the same key, against the same disk. Proving a DIFFERENT recipient does not clear this one. Get that copy out and - run `make secrets-verify-backup KEY=`: + run `make secrets-verify-backup KEY= STACK={{ $labels.stack }}`: docs/runbooks/back-up-the-age-key.md. - alert: SecretsKeyRecipientsUnrecorded @@ -305,7 +315,7 @@ groups: # from the textfile directory, and for a recipient-state job that has # never run — the last of which ScheduledJobNeverRan also names. expr: | - homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"} unless on(instance) homelab_key_recipient_last_proof_timestamp_seconds for: 1h labels: @@ -316,12 +326,15 @@ groups: summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" description: >- scripts/install-timers.sh declared the ninety-day deadline for - verify-key-backup and nothing has written - homelab_key_recipient_last_proof_timestamp_seconds, so + {{ $labels.homelab_job }} on {{ $labels.instance }} and nothing has + written homelab_key_recipient_last_proof_timestamp_seconds there, so SecretsKeyBackupUnproven has nothing to read and cannot fire however - stale the proof is. `systemctl start homelab-recipient-state.service` - writes it; if that unit does not exist, the timers predate #400 and - `make install-timers` adds it. docs/runbooks/back-up-the-age-key.md. + stale the proof is. On the monitoring host + `systemctl start homelab-recipient-state.service` writes it; if that + unit does not exist, the timers predate #400 and + `make install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. - alert: CaKeyBackupUnproven # The estate CA's private key, held to the same ninety days as the age diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index 759c05f8..a9a8b4d8 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -221,13 +221,15 @@ tests: summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" description: >- scripts/install-timers.sh declared the ninety-day deadline for - verify-key-backup and nothing has written - homelab_key_recipient_last_proof_timestamp_seconds, so + verify-key-backup on prometheus and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds there, so SecretsKeyBackupUnproven has nothing to read and cannot fire - however stale the proof is. `systemctl start - homelab-recipient-state.service` writes it; if that unit does - not exist, the timers predate #400 and `make install-timers` - adds it. docs/runbooks/back-up-the-age-key.md. + however stale the proof is. On the monitoring host `systemctl + start homelab-recipient-state.service` writes it; if that unit + does not exist, the timers predate #400 and `make + install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- # @@ -267,8 +269,8 @@ tests: recoverable from anywhere but this host — every other job runs on the same machine, with the same key, against the same disk. Proving a DIFFERENT recipient does not clear this one. Get that - copy out and run `make secrets-verify-backup KEY=`: - docs/runbooks/back-up-the-age-key.md. + copy out and run `make secrets-verify-backup KEY= + STACK=observability`: docs/runbooks/back-up-the-age-key.md. # --- SecretsKeyRecipientsUnrecorded: silent once the file exists -------- # @@ -279,6 +281,111 @@ tests: alertname: SecretsKeyRecipientsUnrecorded exp_alerts: [] + # --- SecretsKeyBackupUnproven: trinity's recipients, trinity's row -------- + # + # trinity declares the age-key deadline as verify-key-backup-sensitive, not + # verify-key-backup, because one name on two instances makes every + # on(homelab_job) join many-to-many (install-timers.sh check 8). This pins + # the three things that name has to do: the stale-job rules still evaluate + # with both declarations present (a stale ordinary job still fires), the + # per-recipient rule holds trinity's never-proved recipient to trinity's + # row, and neither the job series nor the monitoring host's fresh proof + # leaks into the other host's answer. + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "3600+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup-sensitive", instance="trinity"}' + values: "3600+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1aaa", instance="prometheus"}' + values: "9000+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="sensitive", + recipient="age1ccc", instance="trinity"}' + values: "0+0x180" + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup-sensitive", + instance="trinity"}' + values: "0+0x180" + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="backup-volumes", instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="backup-volumes", instance="prometheus"}' + values: "3600+0x180" + alert_rule_test: + - eval_time: 2h30m + alertname: SecretsKeyBackupUnproven + exp_alerts: + - exp_labels: + alertname: SecretsKeyBackupUnproven + stack: sensitive + recipient: age1ccc + instance: trinity + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "No proven backup of age recipient age1ccc in 2h 30m 0s" + description: >- + Nothing else here proves the sensitive secrets are recoverable + from anywhere but this host — every other job runs on the same + machine, with the same key, against the same disk. Proving a + DIFFERENT recipient does not clear this one. Get that copy out + and run `make secrets-verify-backup KEY= + STACK=sensitive`: docs/runbooks/back-up-the-age-key.md. + + # The stale-job rule still evaluates with both declarations present, and + # the human proof's own job series is not its business on trinity either. + - eval_time: 2h30m + alertname: ScheduledJobStale + exp_alerts: + - exp_labels: + alertname: ScheduledJobStale + homelab_job: backup-volumes + instance: prometheus + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "backup-volumes has not succeeded in 2h 30m 0s" + description: >- + The timer has stopped running this, or every run since has + failed. Check `systemctl list-timers 'homelab-*'` and + `journalctl -u homelab-backup-volumes.service` on prometheus — + docs/runbooks/schedule-maintenance.md. + + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + + # --- SecretsKeyRecipientsUnrecorded: trinity's row with no file ---------- + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup-sensitive", instance="trinity"}' + values: "3600+0x180" + alert_rule_test: + - eval_time: 2h + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: + - exp_labels: + alertname: SecretsKeyRecipientsUnrecorded + homelab_job: verify-key-backup-sensitive + instance: trinity + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup-sensitive on trinity and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds there, so + SecretsKeyBackupUnproven has nothing to read and cannot fire + however stale the proof is. On the monitoring host `systemctl + start homelab-recipient-state.service` writes it; if that unit + does not exist, the timers predate #400 and `make + install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. + # --- CaKeyBackupUnproven: keyed on the key's fingerprint (#496) ------------ # # The re-mint shape: the old key's copy was proved 30 minutes ago and the diff --git a/stacks/sensitive/glance/pages/maintenance.yml b/stacks/sensitive/glance/pages/maintenance.yml index bbe9addc..6b656a9c 100644 --- a/stacks/sensitive/glance/pages/maintenance.yml +++ b/stacks/sensitive/glance/pages/maintenance.yml @@ -20,9 +20,10 @@ # that reads "never succeeded", not an age of decades. # - A job is overdue when its last success is older than its declared limit # (ScheduledJobStale), and flagged when its last run failed. - # - verify-key-backup and verify-ca-key-backup are not read from the generic - # success time. One proof refreshes that for every key, so it can be green - # while one key rots. They are judged per age recipient + # - verify-key-backup (verify-key-backup-sensitive on trinity) and + # verify-ca-key-backup are not read from the generic success time. One + # proof refreshes that for every key, so it can be green while one key + # rots. They are judged per age recipient # (homelab_key_recipient_last_proof_timestamp_seconds) and per current CA # key (homelab_ca_key_backup_last_proof_timestamp_seconds) against the same # limit, as SecretsKeyBackupUnproven and CaKeyBackupUnproven do. A proof @@ -77,7 +78,8 @@ {{ range $decl }} {{ $j := .String "metric.homelab_job" }}{{ $i := .String "metric.instance" }} {{ $limit := .Float "value.1" }} - {{ if or (eq $j "verify-key-backup") (eq $j "verify-ca-key-backup") }} + {{ if or (eq $j "verify-key-backup") (eq $j "verify-key-backup-sensitive") + (eq $j "verify-ca-key-backup") }} {{ $proofs := $keys }}{{ $label := "recipient" }} {{ if eq $j "verify-ca-key-backup" }}{{ $proofs = $ca }}{{ $label = "ca" }}{{ end }} {{ range $proofs }}