fix(sensitive): declare trinity's age-key deadline as verify-key-backup-sensitive

Makefile

@@ -744,7 +744,9 @@ secrets-verify-backup: ## Check a backup age key decrypts the secrets (KEY=/path
 	@# enforced from the other end: a successful run records its timestamp, and
 	@# SecretsKeyBackupUnproven fires when that proof passes ninety days old.
 	@# Nagging is not as good as running it, but it beats remembering.
-	./scripts/run-scheduled.sh --job verify-key-backup --lock keys \
+	@# STACK=sensitive records verify-key-backup-sensitive, trinity's row in
+	@# install-timers.sh: a job name may be declared on one host only.
+	./scripts/run-scheduled.sh --job verify-key-backup$(if $(filter sensitive,$(STACK)),-sensitive) --lock keys \
 		-- ./scripts/verify-key-backup.sh "$(KEY)" $(STACK)
 
 .PHONY: certs-verify-backup

docs/changelog.md

@@ -19,6 +19,25 @@ docstring gives: it is a record, not a claim about now.
 
 ## 2026-11-14
 
+- **`trinity`'s age-key deadline is declared, as
+  `verify-key-backup-sensitive`, and a sensitive key copy can now be proved.**
+  A read-only audit of `trinity` on 2026-11-13 found both recipients in its
+  `key-recipients.prom` at `0`, never proved, since 2026-10-03. #1488 already
+  holds them to the estate's deadline through a fallback branch; this gives
+  `trinity` a declaration of its own. Declaring `verify-key-backup` there was
+  tried under promtool first and would have broken `ScheduledJobStale` and
+  `ScheduledJobStaleCritical` for the whole estate, because they join
+  `on(homelab_job)` and one name on two instances is many-to-many. So
+  `trinity`'s row has its own name, the two key-proof rules match either name
+  (the fallback stays for a host that declares neither), the stale rules
+  exclude it, `make secrets-verify-backup STACK=sensitive` records it, and
+  Glance's maintenance page reads it per recipient and shows a declaration
+  with no proofs as needing attention. `scripts/verify-key-backup.sh` held
+  every stack to the observability file's names (Grafana, Alertmanager,
+  SNMP), so a sensitive copy failed after it had decrypted and its proof was
+  never recorded; other stacks are now held to their compose guards, as
+  `render-config.sh` does. This records the change, not the deploy: the row
+  reaches `trinity` with `make install-timers PROFILE=sensitive` there.
 - **`oracle`'s sshd now takes keys only.** The audit of 2026-11-13 found it
   offering `publickey,password` on the host that holds every off-host copy
   ([#1500](https://github.com/Gerrrt/HomeLab/issues/1500)). The last

docs/runbooks/back-up-the-age-key.md

@@ -241,6 +241,26 @@ this is the same alert it always was. With two, one timestamp for both would
 mean that proving either copy vouched for the other — which is exactly backwards,
 because the point of the second copy is that it fails independently.
 
+### On trinity
+
+`trinity` holds its own key, which opens only
+[`secrets/sensitive.sops.yaml`](../../secrets/sensitive.sops.yaml), and the
+same deadline applies to every recipient of that file. Its row is
+`verify-key-backup-sensitive` in `SENSITIVE_JOBS`, not `verify-key-backup`,
+because the alert rules join on the job name alone and a name declared on two
+hosts breaks them. Prove each copy from the checkout on `trinity`, so the proof
+lands in `trinity`'s own `key-recipients.prom`:
+
+```bash
+make secrets-verify-backup KEY=/path/to/copy/keys.txt STACK=sensitive
+```
+
+`STACK=sensitive` is what records the run under `trinity`'s job name; without
+it the make target checks the estate's file instead. `trinity` has no
+`recipient-state` timer, so `make recipient-state STACK=sensitive` is what
+rewrites its recipient list after a change that did not go through
+`add-recipient.sh`.
+
 Nagging is not as good as running it. It is a great deal better than remembering.
 The threshold lives in the `JOBS` table in
 [`install-timers.sh`](../../scripts/install-timers.sh); the mechanism is in

docs/runbooks/schedule-maintenance.md

@@ -739,6 +739,7 @@ checkout ([#404](https://github.com/Gerrrt/HomeLab/issues/404) step 9):
 | `verify-backups-sensitive` | `make verify-backups` with `STACK=sensitive` | daily 06:30 | 2 days |
 | `household-copy` | **you**, `make household-copy DEST=…` | no timer | 90 days |
 | `household-proof` | **you** and the holder, `make household-proof CODE=…` | no timer | 1 year |
+| `verify-key-backup-sensitive` | **you**, `make secrets-verify-backup KEY=… STACK=sensitive` | no timer | 90 days |
 
 It runs daily, not weekly like the estate's backup, because it holds the
 password vault. Each run also copies the set to `oracle`, the same way the
@@ -775,10 +776,14 @@ make install-timers PROFILE=sensitive
 ```
 
 That writes a `homelab-jobs.prom` on `trinity` that declares only these
-six. The last two are the household's copy
+seven. `household-copy` and `household-proof` are the household's copy
 ([ADR-0073](../adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md),
 [`carry-the-household-copy.md`](carry-the-household-copy.md)). They have no
-timer, and `HouseholdCopyStale` reads them. The alert rules join on the job name alone, so a name may appear in only
+timer, and `HouseholdCopyStale` reads them. `verify-key-backup-sensitive` is
+the estate's `verify-key-backup` for the tier's own age recipients: no timer,
+the same ninety days, and `SecretsKeyBackupUnproven` holds every recipient in
+`trinity`'s `key-recipients.prom` to it
+([`back-up-the-age-key.md`](back-up-the-age-key.md#on-trinity)). The alert rules join on the job name alone, so a name may appear in only
 one table, and `--check` enforces that. The installer primes `backup-library`,
 `converge-sensitive` and `verify-backups-sensitive`, which stop nothing, and not `backup-sensitive`, which stops the tier. `make check-timers` checks both
 profiles, and `make validate` on `trinity` fails until the backup and converge timers are
@@ -904,7 +909,7 @@ expected rather than a second fault.
 
 | Symptom | Cause | Fix |
 | --- | --- | --- |
-| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key, for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-<job>.service` and read the journal |
+| `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key (on `trinity`, `verify-key-backup-sensitive`), for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-<job>.service` and read the journal |
 | `ScheduledJobStaleCritical` | A job has gone three of its own periods without a success — one and a half of the allowance in the `JOBS` table, which is twice the period; a weekly backup three weeks stale, a nightly one three nights, and never under a day for the hourly collectors — and the `ScheduledJobStale` warning for it has been repeating on the in-house channel unanswered | The same first look as the warning: `systemctl list-timers 'homelab-*'`, then `journalctl -u homelab-<job>.service`. The rows below name the usual cause per job. The page holds the warning back while it is up, so a job that recovers clears both at once |
 | `OffsiteCopyStale` | Ninety days since the newest sets were last copied to the second recipient's medium and proved there | Mount it and run `make backup-offsite DEST=…` — [`copy-the-backups-offsite.md`](copy-the-backups-offsite.md). `--list`, `--verify-only` and `--prune` do not clear it, on purpose |
 | `CaKeyBackupUnproven` | No offline copy of `certificates/ca-key.pem` has been proved in ninety days — or ever, or not since the CA was re-minted, which the fingerprint label tells apart | Mount the medium and `make certs-verify-backup KEY=…` ([`back-up-the-ca-key.md`](back-up-the-ca-key.md)). After a re-mint, copy the new key there first; the old copy is refused |

scripts/install-timers.sh

@@ -184,7 +184,13 @@ JOBS=(
 # what keeps the alert joins one-to-one across hosts, so it must not reuse a
 # name from JOBS — check 8 below asserts that. And the nightly re-verification
 # of both kinds of set, here and on oracle (#856): verify-backups-sensitive,
-# two days for the same reason as the backups it re-reads.
+# two days for the same reason as the backups it re-reads. And
+# verify-key-backup-sensitive, the estate's verify-key-backup for this tier's
+# own recipients: the same ninety days and no unit, because it is the same
+# human proof against a copy on removable media, run on trinity as
+# `make secrets-verify-backup KEY=... STACK=sensitive`. SecretsKeyBackupUnproven
+# holds every recipient in trinity's key-recipients.prom to it. It cannot be
+# called verify-key-backup, for check 8's reason: that name is the estate's.
 #
 # The units under systemd/sensitive/ carry @DEPLOY_ROOT@, @RUN_USER@,
 # @RUN_GROUP@ and @RUN_HOME@ rather than a name, because the build runbook
@@ -198,6 +204,7 @@ SENSITIVE_JOBS=(
   "verify-backups-sensitive homelab-verify-backups-sensitive 172800 verify-backups"
   "household-copy    -                          7776000  household-copy"
   "household-proof   -                         31536000  household-proof"
+  "verify-key-backup-sensitive -                7776000  secrets-verify-backup"
 )
 
 die()  { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; }
@@ -524,8 +531,9 @@ if [[ "${MODE}" == check ]]; then
     run_home="$(getent passwd "${run_user}" | cut -d: -f6)"
     for row in "${SENSITIVE_JOBS[@]}"; do
       unit="$(job_unit "${row}")"
-      # household-copy and household-proof are human jobs, "-" as on the
-      # estate's side: no timer to be enabled (ADR-0073).
+      # household-copy, household-proof and verify-key-backup-sensitive are
+      # human jobs, "-" as on the estate's side: no timer to be enabled
+      # (ADR-0073, ADR-0024).
       [[ "${unit}" == "-" ]] && continue
       state="$(systemctl is-enabled "${unit}.timer" 2>/dev/null || true)"
       case "${state}" in

scripts/verify-key-backup.sh

@@ -238,6 +238,27 @@ fi
 # two, SNMP_AUTHPASS_<X> and SNMP_PRIVPASS_<X>. Keep the one-name-per-line
 # shape and the same order as render-config.sh, so the two read as the diff
 # they are.
+#
+# That list is the observability stack's alone. Any other stack — sensitive,
+# on trinity — has no Grafana, no Alertmanager and no SNMP devices, so holding
+# its file to those names failed every proof after the key had already
+# decrypted it, and the per-recipient proof below was never recorded. For
+# those stacks the names come the way render-config.sh's base list does: from
+# the stack's compose guards, less the three it skips because they are not
+# secrets. compose-guards.sh failing is fatal rather than an empty list, for
+# the reason render-config.sh gives: an empty REQUIRED would pass anything.
+REQUIRED=()
+if [[ "${STACK}" != observability ]]; then
+  if ! GUARDS="$("${REPO_ROOT}/scripts/compose-guards.sh" "${REPO_ROOT}/stacks/${STACK}/compose.yaml")"; then
+    die "could not read the guards from stacks/${STACK}/compose.yaml — see above"
+  fi
+  while read -r var; do
+    [[ -n "${var}" ]] || continue
+    case "${var}" in RENDER_UID | RENDER_GID | LOG_READ_GID) continue ;; esac
+    REQUIRED+=("${var}")
+  done <<< "${GUARDS}"
+  ((${#REQUIRED[@]})) || die "stacks/${STACK}/compose.yaml guards no secrets, so there is nothing to hold the decrypted file to"
+else
 REQUIRED=(
   GRAFANA_ADMIN_PASSWORD
   ALERTMANAGER_WEBHOOK_URL
@@ -252,6 +273,7 @@ REQUIRED=(
   SNMP_AUTHPASS_NEO
   SNMP_PRIVPASS_NEO
 )
+fi
 found=0
 missing=()
 for var in "${REQUIRED[@]}"; do

stacks/observability/prometheus/rules/backup.rules.yaml

@@ -82,7 +82,7 @@ groups:
         # shape, and HouseholdCopyStale reads them (ADR-0073).
         expr: |
           (time() - homelab_job_last_success_timestamp_seconds{
-              homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"})
+              homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"})
             > on(homelab_job) group_left() homelab_job_max_age_seconds
         for: 30m
         labels:
@@ -127,7 +127,7 @@ groups:
         # that rule exists to avoid.
         expr: |
           (time() - homelab_job_last_success_timestamp_seconds{
-              homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"})
+              homelab_job!~"verify-key-backup|verify-key-backup-sensitive|verify-ca-key-backup|offsite-copy|household-copy|household-proof"})
             > on(homelab_job) group_left() clamp_min(1.5 * homelab_job_max_age_seconds, 86400)
         for: 30m
         labels:
@@ -266,23 +266,33 @@ groups:
         # deployed (#400). SecretsKeyRecipientsUnrecorded, below, is the rule
         # for that state, and the recipient-state timer is what ends it.
         #
-        # The second branch is for a host that records proofs but declares no
-        # deadline of its own: trinity. Its SENSITIVE_JOBS table cannot carry a
-        # verify-key-backup row (install-timers.sh check 8 forbids a job name in
-        # both tables), so the on(instance) join had no right side there and
-        # trinity's recipients sat at 0 — never proven — with nothing firing.
-        # Such a host is held to the estate's declared deadline instead, taken
-        # with max() so the ninety days is still written in one place only.
+        # Two declaration names, one per host. trinity proves the sensitive
+        # tier's recipients and declares the same ninety days as
+        # verify-key-backup-sensitive, because a name may be in only one of
+        # install-timers.sh's tables: ScheduledJobStale and the rules beside it
+        # join on(homelab_job) alone, and one name declared on two instances
+        # makes those joins many-to-many, so they error rather than fire. Here
+        # the join is on(instance), and each host declares exactly one of the
+        # two, so the regex still finds one row per host.
+        #
+        # The second branch is for a host that records proofs but declares
+        # neither name, as trinity did before its timers were reinstalled with
+        # that row: the on(instance) join has no right side there, and its
+        # recipients would sit at 0 — never proven — with nothing firing. Such
+        # a host is held to the estate's declared deadline instead, taken with
+        # max() so the ninety days is still written in one place only.
         expr: |
           (
             (time() - homelab_key_recipient_last_proof_timestamp_seconds)
-              > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"}
+              > on(instance) group_left()
+                homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"}
           )
           or
           (
             (
               (time() - homelab_key_recipient_last_proof_timestamp_seconds)
-                unless on(instance) homelab_job_max_age_seconds{homelab_job="verify-key-backup"}
+                unless on(instance)
+                  homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"}
             )
               > on() group_left() max(homelab_job_max_age_seconds{homelab_job="verify-key-backup"})
           )
@@ -298,7 +308,7 @@ groups:
             recoverable from anywhere but this host — every other job runs on the
             same machine, with the same key, against the same disk. Proving a
             DIFFERENT recipient does not clear this one. Get that copy out and
-            run `make secrets-verify-backup KEY=<path>`:
+            run `make secrets-verify-backup KEY=<path> STACK={{ $labels.stack }}`:
             docs/runbooks/back-up-the-age-key.md.
 
       - alert: SecretsKeyRecipientsUnrecorded
@@ -323,7 +333,7 @@ groups:
         # from the textfile directory, and for a recipient-state job that has
         # never run — the last of which ScheduledJobNeverRan also names.
         expr: |
-          homelab_job_max_age_seconds{homelab_job="verify-key-backup"}
+          homelab_job_max_age_seconds{homelab_job=~"verify-key-backup|verify-key-backup-sensitive"}
             unless on(instance) homelab_key_recipient_last_proof_timestamp_seconds
         for: 1h
         labels:
@@ -334,12 +344,15 @@ groups:
           summary: "The age-key deadline is declared but no recipient has a proof series to hold to it"
           description: >-
             scripts/install-timers.sh declared the ninety-day deadline for
-            verify-key-backup and nothing has written
-            homelab_key_recipient_last_proof_timestamp_seconds, so
+            {{ $labels.homelab_job }} on {{ $labels.instance }} and nothing has
+            written homelab_key_recipient_last_proof_timestamp_seconds there, so
             SecretsKeyBackupUnproven has nothing to read and cannot fire however
-            stale the proof is. `systemctl start homelab-recipient-state.service`
-            writes it; if that unit does not exist, the timers predate #400 and
-            `make install-timers` adds it. docs/runbooks/back-up-the-age-key.md.
+            stale the proof is. On the monitoring host
+            `systemctl start homelab-recipient-state.service` writes it; if that
+            unit does not exist, the timers predate #400 and
+            `make install-timers` adds it. On trinity, which has no such timer,
+            `make recipient-state STACK=sensitive` writes it.
+            docs/runbooks/back-up-the-age-key.md.
 
       - alert: CaKeyBackupUnproven
         # The estate CA's private key, held to the same ninety days as the age