diff --git a/Makefile b/Makefile index fe72e53c..22156a4d 100644 --- a/Makefile +++ b/Makefile @@ -744,7 +744,9 @@ secrets-verify-backup: ## Check a backup age key decrypts the secrets (KEY=/path @# enforced from the other end: a successful run records its timestamp, and @# SecretsKeyBackupUnproven fires when that proof passes ninety days old. @# Nagging is not as good as running it, but it beats remembering. - ./scripts/run-scheduled.sh --job verify-key-backup --lock keys \ + @# STACK=sensitive records verify-key-backup-sensitive, trinity's row in + @# install-timers.sh: a job name may be declared on one host only. + ./scripts/run-scheduled.sh --job verify-key-backup$(if $(filter sensitive,$(STACK)),-sensitive) --lock keys \ -- ./scripts/verify-key-backup.sh "$(KEY)" $(STACK) .PHONY: certs-verify-backup diff --git a/docs/changelog.md b/docs/changelog.md index 6d09d2b1..1a1e0be5 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,25 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-14 +- **`trinity`'s age-key deadline is declared, as + `verify-key-backup-sensitive`, and a sensitive key copy can now be proved.** + A read-only audit of `trinity` on 2026-11-13 found both recipients in its + `key-recipients.prom` at `0`, never proved, since 2026-10-03. #1488 already + holds them to the estate's deadline through a fallback branch; this gives + `trinity` a declaration of its own. Declaring `verify-key-backup` there was + tried under promtool first and would have broken `ScheduledJobStale` and + `ScheduledJobStaleCritical` for the whole estate, because they join + `on(homelab_job)` and one name on two instances is many-to-many. So + `trinity`'s row has its own name, the two key-proof rules match either name + (the fallback stays for a host that declares neither), the stale rules + exclude it, `make secrets-verify-backup STACK=sensitive` records it, and + Glance's maintenance page reads it per recipient and shows a declaration + with no proofs as needing attention. `scripts/verify-key-backup.sh` held + every stack to the observability file's names (Grafana, Alertmanager, + SNMP), so a sensitive copy failed after it had decrypted and its proof was + never recorded; other stacks are now held to their compose guards, as + `render-config.sh` does. This records the change, not the deploy: the row + reaches `trinity` with `make install-timers PROFILE=sensitive` there. - **`oracle`'s sshd now takes keys only.** The audit of 2026-11-13 found it offering `publickey,password` on the host that holds every off-host copy ([#1500](https://github.com/Gerrrt/HomeLab/issues/1500)). The last diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index 0b12a0c7..d36b4da5 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -241,6 +241,26 @@ this is the same alert it always was. With two, one timestamp for both would mean that proving either copy vouched for the other — which is exactly backwards, because the point of the second copy is that it fails independently. +### On trinity + +`trinity` holds its own key, which opens only +[`secrets/sensitive.sops.yaml`](../../secrets/sensitive.sops.yaml), and the +same deadline applies to every recipient of that file. Its row is +`verify-key-backup-sensitive` in `SENSITIVE_JOBS`, not `verify-key-backup`, +because the alert rules join on the job name alone and a name declared on two +hosts breaks them. Prove each copy from the checkout on `trinity`, so the proof +lands in `trinity`'s own `key-recipients.prom`: + +```bash +make secrets-verify-backup KEY=/path/to/copy/keys.txt STACK=sensitive +``` + +`STACK=sensitive` is what records the run under `trinity`'s job name; without +it the make target checks the estate's file instead. `trinity` has no +`recipient-state` timer, so `make recipient-state STACK=sensitive` is what +rewrites its recipient list after a change that did not go through +`add-recipient.sh`. + Nagging is not as good as running it. It is a great deal better than remembering. The threshold lives in the `JOBS` table in [`install-timers.sh`](../../scripts/install-timers.sh); the mechanism is in diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index b4c7f3c9..eacbb9a7 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -739,6 +739,7 @@ checkout ([#404](https://github.com/Gerrrt/HomeLab/issues/404) step 9): | `verify-backups-sensitive` | `make verify-backups` with `STACK=sensitive` | daily 06:30 | 2 days | | `household-copy` | **you**, `make household-copy DEST=…` | no timer | 90 days | | `household-proof` | **you** and the holder, `make household-proof CODE=…` | no timer | 1 year | +| `verify-key-backup-sensitive` | **you**, `make secrets-verify-backup KEY=… STACK=sensitive` | no timer | 90 days | It runs daily, not weekly like the estate's backup, because it holds the password vault. Each run also copies the set to `oracle`, the same way the @@ -775,10 +776,14 @@ make install-timers PROFILE=sensitive ``` That writes a `homelab-jobs.prom` on `trinity` that declares only these -six. The last two are the household's copy +seven. `household-copy` and `household-proof` are the household's copy ([ADR-0073](../adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md), [`carry-the-household-copy.md`](carry-the-household-copy.md)). They have no -timer, and `HouseholdCopyStale` reads them. The alert rules join on the job name alone, so a name may appear in only +timer, and `HouseholdCopyStale` reads them. `verify-key-backup-sensitive` is +the estate's `verify-key-backup` for the tier's own age recipients: no timer, +the same ninety days, and `SecretsKeyBackupUnproven` holds every recipient in +`trinity`'s `key-recipients.prom` to it +([`back-up-the-age-key.md`](back-up-the-age-key.md#on-trinity)). The alert rules join on the job name alone, so a name may appear in only one table, and `--check` enforces that. The installer primes `backup-library`, `converge-sensitive` and `verify-backups-sensitive`, which stop nothing, and not `backup-sensitive`, which stops the tier. `make check-timers` checks both profiles, and `make validate` on `trinity` fails until the backup and converge timers are @@ -904,7 +909,7 @@ expected rather than a second fault. | Symptom | Cause | Fix | | --- | --- | --- | -| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key, for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-.service` and read the journal | +| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key (on `trinity`, `verify-key-backup-sensitive`), for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-.service` and read the journal | | `ScheduledJobStaleCritical` | A job has gone three of its own periods without a success — one and a half of the allowance in the `JOBS` table, which is twice the period; a weekly backup three weeks stale, a nightly one three nights, and never under a day for the hourly collectors — and the `ScheduledJobStale` warning for it has been repeating on the in-house channel unanswered | The same first look as the warning: `systemctl list-timers 'homelab-*'`, then `journalctl -u homelab-.service`. The rows below name the usual cause per job. The page holds the warning back while it is up, so a job that recovers clears both at once | | `OffsiteCopyStale` | Ninety days since the newest sets were last copied to the second recipient's medium and proved there | Mount it and run `make backup-offsite DEST=…` — [`copy-the-backups-offsite.md`](copy-the-backups-offsite.md). `--list`, `--verify-only` and `--prune` do not clear it, on purpose | | `CaKeyBackupUnproven` | No offline copy of `certificates/ca-key.pem` has been proved in ninety days — or ever, or not since the CA was re-minted, which the fingerprint label tells apart | Mount the medium and `make certs-verify-backup KEY=…` ([`back-up-the-ca-key.md`](back-up-the-ca-key.md)). After a re-mint, copy the new key there first; the old copy is refused | diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index be77b5ba..b13da745 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -184,7 +184,13 @@ JOBS=( # what keeps the alert joins one-to-one across hosts, so it must not reuse a # name from JOBS — check 8 below asserts that. And the nightly re-verification # of both kinds of set, here and on oracle (#856): verify-backups-sensitive, -# two days for the same reason as the backups it re-reads. +# two days for the same reason as the backups it re-reads. And +# verify-key-backup-sensitive, the estate's verify-key-backup for this tier's +# own recipients: the same ninety days and no unit, because it is the same +# human proof against a copy on removable media, run on trinity as +# `make secrets-verify-backup KEY=... STACK=sensitive`. SecretsKeyBackupUnproven +# holds every recipient in trinity's key-recipients.prom to it. It cannot be +# called verify-key-backup, for check 8's reason: that name is the estate's. # # The units under systemd/sensitive/ carry @DEPLOY_ROOT@, @RUN_USER@, # @RUN_GROUP@ and @RUN_HOME@ rather than a name, because the build runbook @@ -198,6 +204,7 @@ SENSITIVE_JOBS=( "verify-backups-sensitive homelab-verify-backups-sensitive 172800 verify-backups" "household-copy - 7776000 household-copy" "household-proof - 31536000 household-proof" + "verify-key-backup-sensitive - 7776000 secrets-verify-backup" ) die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } @@ -524,8 +531,9 @@ if [[ "${MODE}" == check ]]; then run_home="$(getent passwd "${run_user}" | cut -d: -f6)" for row in "${SENSITIVE_JOBS[@]}"; do unit="$(job_unit "${row}")" - # household-copy and household-proof are human jobs, "-" as on the - # estate's side: no timer to be enabled (ADR-0073). + # household-copy, household-proof and verify-key-backup-sensitive are + # human jobs, "-" as on the estate's side: no timer to be enabled + # (ADR-0073, ADR-0024). [[ "${unit}" == "-" ]] && continue state="$(systemctl is-enabled "${unit}.timer" 2>/dev/null || true)" case "${state}" in diff --git a/scripts/verify-key-backup.sh b/scripts/verify-key-backup.sh index 7d96d9bb..49269440 100755 --- a/scripts/verify-key-backup.sh +++ b/scripts/verify-key-backup.sh @@ -238,6 +238,27 @@ fi # two, SNMP_AUTHPASS_ and SNMP_PRIVPASS_. Keep the one-name-per-line # shape and the same order as render-config.sh, so the two read as the diff # they are. +# +# That list is the observability stack's alone. Any other stack — sensitive, +# on trinity — has no Grafana, no Alertmanager and no SNMP devices, so holding +# its file to those names failed every proof after the key had already +# decrypted it, and the per-recipient proof below was never recorded. For +# those stacks the names come the way render-config.sh's base list does: from +# the stack's compose guards, less the three it skips because they are not +# secrets. compose-guards.sh failing is fatal rather than an empty list, for +# the reason render-config.sh gives: an empty REQUIRED would pass anything. +REQUIRED=() +if [[ "${STACK}" != observability ]]; then + if ! GUARDS="$("${REPO_ROOT}/scripts/compose-guards.sh" "${REPO_ROOT}/stacks/${STACK}/compose.yaml")"; then + die "could not read the guards from stacks/${STACK}/compose.yaml — see above" + fi + while read -r var; do + [[ -n "${var}" ]] || continue + case "${var}" in RENDER_UID | RENDER_GID | LOG_READ_GID) continue ;; esac + REQUIRED+=("${var}") + done <<< "${GUARDS}" + ((${#REQUIRED[@]})) || die "stacks/${STACK}/compose.yaml guards no secrets, so there is nothing to hold the decrypted file to" +else REQUIRED=( GRAFANA_ADMIN_PASSWORD ALERTMANAGER_WEBHOOK_URL @@ -252,6 +273,7 @@ REQUIRED=( SNMP_AUTHPASS_NEO SNMP_PRIVPASS_NEO ) +fi found=0 missing=() for var in "${REQUIRED[@]}"; do diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index e1e3b141..2c098961 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -82,7 +82,7 @@ groups: # shape, and HouseholdCopyStale reads them (ADR-0073). expr: | (time() - homelab_job_last_success_timestamp_seconds{ - homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) + homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) > on(homelab_job) group_left() homelab_job_max_age_seconds for: 30m labels: @@ -127,7 +127,7 @@ groups: # that rule exists to avoid. expr: | (time() - homelab_job_last_success_timestamp_seconds{ - homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) + homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) > on(homelab_job) group_left() clamp_min(1.5 * homelab_job_max_age_seconds, 86400) for: 30m labels: @@ -266,23 +266,33 @@ groups: # deployed (#400). SecretsKeyRecipientsUnrecorded, below, is the rule # for that state, and the recipient-state timer is what ends it. # - # The second branch is for a host that records proofs but declares no - # deadline of its own: trinity. Its SENSITIVE_JOBS table cannot carry a - # verify-key-backup row (install-timers.sh check 8 forbids a job name in - # both tables), so the on(instance) join had no right side there and - # trinity's recipients sat at 0 — never proven — with nothing firing. - # Such a host is held to the estate's declared deadline instead, taken - # with max() so the ninety days is still written in one place only. + # Two declaration names, one per host. trinity proves the sensitive + # tier's recipients and declares the same ninety days as + # verify-key-backup-sensitive, because a name may be in only one of + # install-timers.sh's tables: ScheduledJobStale and the rules beside it + # join on(homelab_job) alone, and one name declared on two instances + # makes those joins many-to-many, so they error rather than fire. Here + # the join is on(instance), and each host declares exactly one of the + # two, so the regex still finds one row per host. + # + # The second branch is for a host that records proofs but declares + # neither name, as trinity did before its timers were reinstalled with + # that row: the on(instance) join has no right side there, and its + # recipients would sit at 0 — never proven — with nothing firing. Such + # a host is held to the estate's declared deadline instead, taken with + # max() so the ninety days is still written in one place only. expr: | ( (time() - homelab_key_recipient_last_proof_timestamp_seconds) - > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + > on(instance) group_left() + homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"} ) or ( ( (time() - homelab_key_recipient_last_proof_timestamp_seconds) - unless on(instance) homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + unless on(instance) + homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"} ) > on() group_left() max(homelab_job_max_age_seconds{homelab_job="verify-key-backup"}) ) @@ -298,7 +308,7 @@ groups: recoverable from anywhere but this host — every other job runs on the same machine, with the same key, against the same disk. Proving a DIFFERENT recipient does not clear this one. Get that copy out and - run `make secrets-verify-backup KEY=`: + run `make secrets-verify-backup KEY= STACK={{ $labels.stack }}`: docs/runbooks/back-up-the-age-key.md. - alert: SecretsKeyRecipientsUnrecorded @@ -323,7 +333,7 @@ groups: # from the textfile directory, and for a recipient-state job that has # never run — the last of which ScheduledJobNeverRan also names. expr: | - homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"} unless on(instance) homelab_key_recipient_last_proof_timestamp_seconds for: 1h labels: @@ -334,12 +344,15 @@ groups: summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" description: >- scripts/install-timers.sh declared the ninety-day deadline for - verify-key-backup and nothing has written - homelab_key_recipient_last_proof_timestamp_seconds, so + {{ $labels.homelab_job }} on {{ $labels.instance }} and nothing has + written homelab_key_recipient_last_proof_timestamp_seconds there, so SecretsKeyBackupUnproven has nothing to read and cannot fire however - stale the proof is. `systemctl start homelab-recipient-state.service` - writes it; if that unit does not exist, the timers predate #400 and - `make install-timers` adds it. docs/runbooks/back-up-the-age-key.md. + stale the proof is. On the monitoring host + `systemctl start homelab-recipient-state.service` writes it; if that + unit does not exist, the timers predate #400 and + `make install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. - alert: CaKeyBackupUnproven # The estate CA's private key, held to the same ninety days as the age diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index cbf925b2..7cfe7730 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -221,13 +221,15 @@ tests: summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" description: >- scripts/install-timers.sh declared the ninety-day deadline for - verify-key-backup and nothing has written - homelab_key_recipient_last_proof_timestamp_seconds, so + verify-key-backup on prometheus and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds there, so SecretsKeyBackupUnproven has nothing to read and cannot fire - however stale the proof is. `systemctl start - homelab-recipient-state.service` writes it; if that unit does - not exist, the timers predate #400 and `make install-timers` - adds it. docs/runbooks/back-up-the-age-key.md. + however stale the proof is. On the monitoring host `systemctl + start homelab-recipient-state.service` writes it; if that unit + does not exist, the timers predate #400 and `make + install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- # @@ -267,8 +269,8 @@ tests: recoverable from anywhere but this host — every other job runs on the same machine, with the same key, against the same disk. Proving a DIFFERENT recipient does not clear this one. Get that - copy out and run `make secrets-verify-backup KEY=`: - docs/runbooks/back-up-the-age-key.md. + copy out and run `make secrets-verify-backup KEY= + STACK=observability`: docs/runbooks/back-up-the-age-key.md. # --- SecretsKeyRecipientsUnrecorded: silent once the file exists -------- # @@ -279,8 +281,115 @@ tests: alertname: SecretsKeyRecipientsUnrecorded exp_alerts: [] + # --- SecretsKeyBackupUnproven: trinity's recipients, trinity's row -------- + # + # trinity declares the age-key deadline as verify-key-backup-sensitive, not + # verify-key-backup, because one name on two instances makes every + # on(homelab_job) join many-to-many (install-timers.sh check 8). This pins + # the three things that name has to do: the stale-job rules still evaluate + # with both declarations present (a stale ordinary job still fires), the + # per-recipient rule holds trinity's never-proved recipient to trinity's + # row, and neither the job series nor the monitoring host's fresh proof + # leaks into the other host's answer. + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "3600+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup-sensitive", instance="trinity"}' + values: "3600+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="observability", + recipient="age1aaa", instance="prometheus"}' + values: "9000+0x180" + - series: 'homelab_key_recipient_last_proof_timestamp_seconds{stack="sensitive", + recipient="age1ccc", instance="trinity"}' + values: "0+0x180" + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup-sensitive", + instance="trinity"}' + values: "0+0x180" + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="backup-volumes", instance="prometheus"}' + values: "0+0x180" + - series: 'homelab_job_max_age_seconds{homelab_job="backup-volumes", instance="prometheus"}' + values: "3600+0x180" + alert_rule_test: + - eval_time: 2h30m + alertname: SecretsKeyBackupUnproven + exp_alerts: + - exp_labels: + alertname: SecretsKeyBackupUnproven + stack: sensitive + recipient: age1ccc + instance: trinity + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "No proven backup of age recipient age1ccc in 2h 30m 0s" + description: >- + Nothing else here proves the sensitive secrets are recoverable + from anywhere but this host — every other job runs on the same + machine, with the same key, against the same disk. Proving a + DIFFERENT recipient does not clear this one. Get that copy out + and run `make secrets-verify-backup KEY= + STACK=sensitive`: docs/runbooks/back-up-the-age-key.md. + + # The stale-job rule still evaluates with both declarations present, and + # the human proof's own job series is not its business on trinity either. + - eval_time: 2h30m + alertname: ScheduledJobStale + exp_alerts: + - exp_labels: + alertname: ScheduledJobStale + homelab_job: backup-volumes + instance: prometheus + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "backup-volumes has not succeeded in 2h 30m 0s" + description: >- + The timer has stopped running this, or every run since has + failed. Check `systemctl list-timers 'homelab-*'` and + `journalctl -u homelab-backup-volumes.service` on prometheus — + docs/runbooks/schedule-maintenance.md. + + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + + # --- SecretsKeyRecipientsUnrecorded: trinity's row with no file ---------- + - interval: 1m + input_series: + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup-sensitive", instance="trinity"}' + values: "3600+0x180" + alert_rule_test: + - eval_time: 2h + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: + - exp_labels: + alertname: SecretsKeyRecipientsUnrecorded + homelab_job: verify-key-backup-sensitive + instance: trinity + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup-sensitive on trinity and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds there, so + SecretsKeyBackupUnproven has nothing to read and cannot fire + however stale the proof is. On the monitoring host `systemctl + start homelab-recipient-state.service` writes it; if that unit + does not exist, the timers predate #400 and `make + install-timers` adds it. On trinity, which has no such timer, + `make recipient-state STACK=sensitive` writes it. + docs/runbooks/back-up-the-age-key.md. + # --- SecretsKeyBackupUnproven: a host with no deadline row of its own ----- # + # (trinity before its timers carry verify-key-backup-sensitive.) + # # trinity records its recipients but cannot declare verify-key-backup (check # 8 in install-timers.sh), so the per-instance join used to find nothing and # its never-proven keys were silent. It is held to the estate's deadline. The @@ -315,7 +424,8 @@ tests: recoverable from anywhere but this host — every other job runs on the same machine, with the same key, against the same disk. Proving a DIFFERENT recipient does not clear this one. Get that - copy out and run `make secrets-verify-backup KEY=`: + copy out and run `make secrets-verify-backup KEY= + STACK=sensitive`: docs/runbooks/back-up-the-age-key.md. # --- CaKeyBackupUnproven: keyed on the key's fingerprint (#496) ------------ diff --git a/stacks/sensitive/glance/pages/maintenance.yml b/stacks/sensitive/glance/pages/maintenance.yml index bbe9addc..956dec20 100644 --- a/stacks/sensitive/glance/pages/maintenance.yml +++ b/stacks/sensitive/glance/pages/maintenance.yml @@ -20,13 +20,16 @@ # that reads "never succeeded", not an age of decades. # - A job is overdue when its last success is older than its declared limit # (ScheduledJobStale), and flagged when its last run failed. - # - verify-key-backup and verify-ca-key-backup are not read from the generic - # success time. One proof refreshes that for every key, so it can be green - # while one key rots. They are judged per age recipient + # - verify-key-backup (verify-key-backup-sensitive on trinity) and + # verify-ca-key-backup are not read from the generic success time. One + # proof refreshes that for every key, so it can be green while one key + # rots. They are judged per age recipient # (homelab_key_recipient_last_proof_timestamp_seconds) and per current CA # key (homelab_ca_key_backup_last_proof_timestamp_seconds) against the same # limit, as SecretsKeyBackupUnproven and CaKeyBackupUnproven do. A proof - # recorded as 0 reads "never proved". + # recorded as 0 reads "never proved", and a declaration with no proof + # series at all on its host reads "no proofs recorded" and counts as + # needing attention, as SecretsKeyRecipientsUnrecorded does. # # Everything is joined on `instance`, the machine label the rules join on. # Needing attention comes first; the rest follow. @@ -77,11 +80,14 @@ {{ range $decl }} {{ $j := .String "metric.homelab_job" }}{{ $i := .String "metric.instance" }} {{ $limit := .Float "value.1" }} - {{ if or (eq $j "verify-key-backup") (eq $j "verify-ca-key-backup") }} + {{ if or (eq $j "verify-key-backup") (eq $j "verify-key-backup-sensitive") + (eq $j "verify-ca-key-backup") }} {{ $proofs := $keys }}{{ $label := "recipient" }} {{ if eq $j "verify-ca-key-backup" }}{{ $proofs = $ca }}{{ $label = "ca" }}{{ end }} + {{ $n := 0 }} {{ range $proofs }} {{ if eq (.String "metric.instance") $i }} + {{ $n = add $n 1 }} {{ $age := .Float "value.1" }}{{ $never := gt $age 1e9 }} {{ $isBad := or $never (gt $age $limit) }} {{ if eq $pass 0 }} @@ -104,6 +110,17 @@ {{ end }} {{ end }} {{ end }} + {{/* Declared and nothing recorded for this host: the rule that + would nag has no left side (SecretsKeyRecipientsUnrecorded). */}} + {{ if eq $n 0 }} + {{ if eq $pass 0 }}{{ $all = add $all 1 }}{{ $bad = add $bad 1 }} + {{ else if eq $pass 1 }} +
  • + {{ $j }} on {{ $i }} + no proofs recorded +
    • + {{ end }} + {{ end }} {{ else }} {{ $age := -1.0 }}{{ $isFailed := false }} {{ range $success }}