[Gerrrt]

What changed

trinity now declares the ninety-day age-key deadline in SENSITIVE_JOBS, as verify-key-backup-sensitive. SecretsKeyBackupUnproven and SecretsKeyRecipientsUnrecorded match that name as well as verify-key-backup. ScheduledJobStale and ScheduledJobStaleCritical exclude it, like the other human proofs. make secrets-verify-backup STACK=sensitive records it, and Glance's maintenance page judges it per recipient and lists a declaration with no proof series as needing attention.

scripts/verify-key-backup.sh held every stack's decrypted file to the observability names (Grafana, Alertmanager, SNMP), so a sensitive copy failed after it had decrypted and its proof was never recorded. Other stacks are now held to their compose guards, as render-config.sh does. The observability list is unchanged.

Main landed #1488 meanwhile, which holds a host with no deadline row to the estate's deadline through a fallback branch. The merge keeps both: the primary branch matches either declaration name, and the fallback stays for a host that declares neither.

Why

A read-only audit of trinity on 2026-11-13 found both recipients in its key-recipients.prom at 0 (never proved) since 2026-10-03. SecretsKeyBackupUnproven couldn't fire for either of them: it joins against a homelab_job_max_age_seconds{homelab_job="verify-key-backup"} row that trinity never declared. A lost trinity key makes secrets/sensitive.sops.yaml and every sensitive backup set unreadable, and nothing was set up to warn.

The obvious fix was to add a verify-key-backup row to SENSITIVE_JOBS. I tested that under promtool first, and it breaks ScheduledJobStale and ScheduledJobStaleCritical for the whole estate: they join on(homelab_job), and one name declared on two instances is many-to-many (found duplicate series for the match group {homelab_job="verify-key-backup"}). That is why install-timers.sh check 8 forbids a name in both tables. So trinity's row gets its own name, following converge-sensitive.

Blast radius

The estate's backup alert rules on prometheus, Glance's maintenance page on trinity, and make secrets-verify-backup / scripts/verify-key-backup.sh. Without STACK=sensitive, the make target records verify-key-backup exactly as before.

With #1488 deployed, SecretsKeyBackupUnproven already fires for trinity's recipients. Once the timers are reinstalled on trinity, these fire until each offline copy is proved:

• SecretsKeyBackupUnproven, one alert per recipient (two today);
• ScheduledJobNeverRan for verify-key-backup-sensitive.

Both are correct: neither key has ever been proved.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

• promtool test rules backup.test.yaml (prom/prometheus v3.16.0, as pinned): SUCCESS. New cases:

  • both hosts declare their own row, and a stale ordinary job still fires ScheduledJobStale;
  • trinity's never-proved recipient fires SecretsKeyBackupUnproven with stack: sensitive and instance: trinity, while the monitoring host's fresh proof stays quiet;
  • trinity's row with no recipient file fires SecretsKeyRecipientsUnrecorded.
  • fix(alerts): hold trinity's age recipients to a deadline, and stop negative SSD wear #1488's fallback case (trinity with no row of its own) still passes.

• verify-key-backup.sh against a throwaway fixture (a fresh age key, stacks/sensitive/compose.yaml, a file holding its 29 guarded names with dummy values): the new script decrypts, passes and records the recipient's proof; main's script fails with the observability names missing. The names in the real secrets/sensitive.sops.yaml (key names only, not values) cover all 29.

• The same promtool run against a plain verify-key-backup row on two instances fails with the many-to-many error quoted above.

• scripts/install-timers.sh --check: check 8 passes, and the new row passes as "no timer, 7776000s deadline declared for the human". The installed-timer checks are skipped off-host, as designed.

• make -n secrets-verify-backup KEY=x STACK=sensitive runs --job verify-key-backup-sensitive. Without STACK, it runs --job verify-key-backup.

• make validate: exit 0, with 10 skips, none of them failures. yamllint, ruff (×2) and ansible-lint were unavailable locally. Four installed-timer and unit checks are off-host by design in a scratch clone. The /tmp tmpfs refusal case and the observability installed-copy comparison need their own host. I ran yamllint on the three changed YAML files directly in pipelinecomponents/yamllint, and it is clean. check_docs.py: docs OK. CI runs the full set.

• make validate passes (with the skips listed above)

• Deployed to the lab and confirmed working

• Docs updated (docs/, service README, or docs/roadmap.md)

After merge, on trinity (converge doesn't reinstall timers):

make install-timers PROFILE=sensitive

prometheus gets the rule change through its own converge. A changelog entry for the deploy comes when that has run.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The verifier rejects valid sensitive backups before recording recipient proofs, preventing alert clearance, while the maintenance page can hide missing proofs and incorrectly report all jobs on schedule.

2 open issues

• Make required-field validation stack-specific · New
• Report missing recipient proofs as unrecorded · New

What changed in this PR

This PR adds a distinct age-key backup deadline for trinity, allowing sensitive-stack recipients to be monitored without conflicting with estate-wide job-name joins.

Changes:

• Declares verify-key-backup-sensitive and routes sensitive proof runs to that job name.
• Updates alert rules, regression tests, and Glance’s per-recipient status handling.
• Documents verification and deployment steps, with a changelog entry distinguishing the change from deployment.

File	Description
stacks/​sensitive/​glance/​pages/​maintenance.yml	Handles the sensitive job as a per-recipient proof.
stacks/​observability/​prometheus/​tests/​backup.test.yaml	Adds sensitive-recipient and missing-metric alert cases.
stacks/​observability/​prometheus/​rules/​backup.rules.yaml	Recognizes the sensitive deadline and updates remediation text.
scripts/​install-timers.sh	Declares the sensitive stack’s ninety-day human-proof deadline.
Makefile	Selects the sensitive job name for STACK=sensitive.
docs/​runbooks/​schedule-maintenance.md	Documents the added deadline and expected alerts.
docs/​runbooks/​back-up-the-age-key.md	Adds verification instructions for trinity.
docs/​changelog.md	Records the monitoring change and deployment requirement.

_{Review effort: Balanced}