Skip to content

feat(settlement): log every auto-settle decision and audit them after the fact - #80

Draft
maschwenk wants to merge 1 commit into
mainfrom
feat/settlement-decision-log
Draft

maschwenk wants to merge 1 commit into
mainfrom
feat/settlement-decision-log

Conversation

@maschwenk

Copy link
Copy Markdown
Owner

i want to be able to study auto-settles after the fact and tune the rules. most of what that takes was already in the event log: since PR 47 every settle records its rule, and the PR states, runs and replies are all there. what got lost was everything only known at the moment of the decision

what changed

  • settlement_decisions table. one row per auto-settle, plus one per hold whenever its rule, reason or run changes, so a hold every sweep repeats is written once. each row has
    • what triggered the sweep (merge event, thread event, full sweep), the rule, and the hold reason
    • the run, and the reply check's record for it: model, who sent the request, verdict, attempts, error
    • the settlement candidate the rules read (inputs_json) and the settle settings, so the decision can be replayed
    • the build version
  • only holds a rule change could alter get rows: background work (watches, subagents, awaited tasks), a pending request, a waiting guest, and the reply check. pins, snoozes and running turns would add a row per turn and say nothing about the rules
  • the table is created at startup with CREATE TABLE IF NOT EXISTS, not as a migration, since fork migration ids collide with upstream's (that's what reconcileForkMigration.ts cleans up after). nothing else reads it. a failed write logs a warning and never blocks a settle
  • unsettleCause on the thread, set whenever a settled thread comes back: user or agent (explicit unsettle, MCP now sends reason: "agent"), pin, user_message, agent_message, scheduled_task, notification, subagent_result, guest. cleared on settle, same as settlementSource
  • node apps/server/scripts/settle-audit.ts reads the db read-only and labels every auto-settle in the window by what happened next: reopened by me within N hours (likely wrong), reopened later, reopened by a schedule or wake (fine), or never. it prints a table by rule, lists the ones i reopened, sums up the recorded holds, and --json writes one line per settle with the request and reply the check read. reopens from before this ships get their cause from the message that came with them
    • --replay reruns this checkout's rules on every recorded decision, with the recorded reply check verdict, and lists the ones that come out different. so a rule change gets checked against real history before it ships
  • resolveAutoSettlement now returns settle, hold with a reason, or null, instead of settle or null. settle results don't change: the existing 50 settlement tests pass untouched

first run on my data, last 7 days

rule settles reopened by me <24h reopened by a schedule stayed settled
PR merged 47 4 13 30
inactivity 551 0 0 551
PR closed / branch PR merged 6 0 0 6
no rule recorded (before 47) 163 0 0 163

all 4 i reopened were merge settles right after a PR-watch wake. in 3 of them the last reply says work is still in flight ("i'll pick up when the build finishes", "once the login goes through i'll apply the PR"), which the reply check prompt says should hold. there's no way to see today whether the check ran or what it answered. that's the first thing this log will tell us

tradeoffs

  • one more table in statev2. ~770 settles a week plus hold rows on change, a few KB each with the candidate snapshot, so tens of MB a year. no pruning yet
  • holds are deduped in memory, so each server restart writes every current hold once more. the audit counts threads next to rows for that reason
  • replay can't re-ask the model, it reuses the recorded verdict. for prompt changes --json has the request and reply pairs to run an eval over
  • unsettleCause only exists on new events. older reopens are inferred, and an MCP agent's unsettle from before this reads as user

proof

  • ThreadSettlementService.test.ts: "tells the orchestrator which rule settled each thread" also checks the logged rows, including a watched thread logged as held by background_work. "asks the model once per run" checks that 3 sweeps write one held row per thread with the model's verdict, then a settle row with the next run's scheduled verdict
  • Orchestrator.control-reads.test.ts: new case covering each cause
  • settle-audit.test.ts: replaying inactivity, a snooze, a watch and a reply check hold through JSON gives the same decision. all four fail if the date revival is broken
  • runtime: ran the dev server in this worktree on a pruned copy of my db (98 threads, migrate-dev-db) for ~2.5 min. the table got created at startup and the sweeps logged 3 held rows (2 merged, 1 closed, all needs_user with the model's verdict), with no repeats across later sweeps. a --watch restart wrote each hold once more, since the dedup map lives in memory. settle-audit --replay on that db: 6 decisions replayed, 0 changed

written with claude opus 5.5 in claude code

… the fact

Settle history was in the event log, but what was only known at decision
time was lost: the reply check's verdict lived in memory and a log line
that rolls over in minutes, holds were never recorded, an unsettle had no
cause, and nothing said which build or settings decided.

- settlement_decisions: one row per auto-settle and per hold change, with
  the trigger, rule, hold reason, run, the reply check's verdict, the
  inputs the rules read, the settings, and the build version. Created at
  startup instead of by a migration, since fork migration ids collide with
  upstream's.
- unsettleCause on the thread: who or what brought a settled thread back.
  An agent's MCP unsettle now sends reason "agent".
- apps/server/scripts/settle-audit.ts labels past auto-settles by what
  happened next and, with --replay, reruns this checkout's rules on every
  recorded decision.
@maschwenk maschwenk added the coder-task-generated Created from a coding task label Nov 14, 2026
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 14, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Baseline: ce933e8 · PR result: fdbc9c0 · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coder-task-generated Created from a coding task size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant