Repository navigation
Persistent/event-driven Reliability Governor follow-up #29
Description
Context
Production rollout of the timer-driven Reliability Governor is complete as of 2026-06-30.
Observed rollout state:
- Governor timer is enabled and running on the loop VM.
- Dry run and first live passes were conservative: no unexpected
loop:approvedlabels. - Missing Knowledge context routed work to
loop:knowledge-gap. - NOC LHP fetches worked for the observed handoffs and produced concrete CaseService payload hashes.
- Engineering daemon continued consuming only
loop:approved; after Governor routing, approved count was zero.
Remaining work
- Rollout hardening
- Make loop protocol label creation part of the managed rollout/runbook so first live pass cannot fail on missing
loop:knowledge-gap/ terminal labels. - Review idempotency around comment/CDR writes after partial failures. The first production attempt wrote an audit record/comment before label mutation failed because labels were missing.
- Decide whether duplicate GitHub comments should be de-duplicated or left as audit evidence.
- Knowledge-context recovery
- Debug why production Knowledge MCP/context returns
missingwith noincluded_refsfor current queue items. - Make the Governor's knowledge-gap output actionable enough for Knowledge Loop/humans to repair the missing context.
- Confirm stale/missing/contradictory Knowledge continues to deny conservatively.
- Re-run a production dry run after Knowledge context starts returning usable refs.
- Persistent/event-driven Governor design and implementation
- Add durable inbox/outbox storage for Governor work and side effects.
- Add GitHub webhook wakeups instead of relying only on timer polling.
- Add NOC, Knowledge, and Engineering callback ingestion.
- Move callback handling to reconciliation-based authorization instead of direct callback actions.
- Preserve human merge as mandatory and keep Engineering daemon consuming only
loop:approved. - Make GitHub comments, labels, and local CDR persistence idempotent and replay-safe.
- Add reconciliation for missed webhook/callback events.
- Add metrics/logging/alerts for queue depth, decision counts, callback failures, CDR writes, and label transition failures.
Acceptance criteria
- Governor can resume safely after process restart or partial GitHub/API failure.
- Every external mutation has a durable outbox record and can be retried without duplicate unsafe effects.
- Webhook/callback events wake the Governor, but authorization is still derived from reconciled GitHub/Knowledge/NOC state.
- No callback directly applies
loop:approvedor causes Engineering execution without Governor reconciliation. - CDRs remain the audit source for decisions and include enough identity material to detect stale LHP/Knowledge inputs.
- Engineering daemon remains label-gated on
loop:approved; human merge remains mandatory.
Non-goals
- Do not relax the conservative Knowledge-deny policy.
- Do not introduce auto-merge.
- Do not let NOC/Knowledge/Engineering callbacks directly authorize work.
Activity
hyrule-engineering-loop commentedon Jun 30, 2026
Reliability Governor Decision
staff_sre_autonomous_operationsknowledge_gapknowledgeknowledge_context_packknowledgelegal/ tier4nonemissing/unknown{ "affected_assets": [ "AS215932/engineering-loop" ], "affected_customers": [], "affected_services": [], "allowed_paths": [], "authority_text_hash": "d28df48831d667acfcf3a26481d9640ee1b61fff21f5c06f80be09c93ba158aa", "blast_radius": "legal/commercial", "controlled_loops": [ "engineering", "noc", "knowledge" ], "created_at": "2026-06-30T02:31:38.299132+00:00", "denial_reasons": [ "Knowledge policy result is deny", "Knowledge context returned no included_refs" ], "expected_paths": [ "legal/" ], "forbidden_paths": [], "governor_name": "Reliability Governor", "governor_role": "staff_sre_autonomous_operations", "handoff_contract": "knowledge_context_pack", "intent_type": "legal", "issue_id": "AS215932/engineering-loop#29", "issue_number": 29, "issue_text_hash": "65338112699440dac1802e2301e84b237074aedbea7bd7498725150a57e79672", "knowledge_authority_level": "unknown", "knowledge_context_pack_id": "ctx_f0760ef2d69f44faf2dba49ce90249f0", "knowledge_export_version": "local-20260623120725", "knowledge_status": "missing", "labels_to_add": [ "loop:knowledge-gap" ], "labels_to_remove": [ "loop:intake", "loop:candidate", "loop:approved", "loop:needs-context", "loop:needs-human" ], "lhp": null, "matched_capability": null, "next_loop": "knowledge", "policy_rules": [ "deny stale, contradictory, missing, or errored Knowledge context" ], "record_id": "2defd2ff41b649c8e445", "repo": "AS215932/engineering-loop", "required_checks": [], "risk_tier": 4, "rollback_plan": "", "routing_decision": "knowledge_gap", "schema_version": "reliability-governor.cdr.v1", "source": "knowledge", "storage_path": "/var/lib/engineering-loop/reliability-governor/as215932-engineering-loop-29-2defd2ff41b649c8e445.json", "verification_method": "" }Svaag commentedon Jul 21, 2026
A2A transport and task-server work is now tracked in #40, under the shared profile in AS215932/agent-core#14. This issue remains the owner of persistent wake/reconciliation behavior. A2A events may wake the Governor, but must never authorize execution or replace reconciliation against current CaseService, GitHub, and Knowledge state.