review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73

Problem and objective

The review graph currently relies on the coordinator for semantic routing. Evaluate whether TypeSafe's Jev decision model can improve applicability judgments and reduce coordinator work while preserving every required review, validation, and evidence guarantee.

The initial shadow implementation and bounded live experiments are complete. PR #94 merged the optional API client, credential handling, frozen-input comparison runner, replay, and accounting. The vendored typesafe-ai skill remains separate from that implementation. This issue stays open for the concrete follow-up and evaluation closeout below; it is not a request to restart the pilot.

Completed baseline and current decision

• Complete the initial graph correctness/recovery prerequisites: review-graph: validate coverage partitions before accepting worker payloads #68, review-graph: support recovery from validator launch failures #69, and review-graph: reduce orchestration overhead and handle blocked validation reliably #72 are closed; perf(skills): reduce review-graph orchestration overhead #80 is merged.
• Merge the opt-in shadow runner and accounting in feat(review-graph): add Jev routing comparisons and accounting #94. Ordinary production routing, mandatory reviews, validation, and correctness-evidence gates remain authoritative.
• Run bounded comparisons on feat: add folded and combined R-hat diagnostics markov-chain-monte-carlo#187 and refactor(changelog)!: adopt pinned shared changelog commands la-stack#261, freezing ordinary selector decisions before querying Jev. The la-stack experiment also ran paired downstream reviews on the same captured scope.
• Record the initial decision: retain shadow mode. The available measurements do not justify changing the default selector or claiming lower total review cost.

The la-stack pilot used an inclusion cutoff of 0.5. Jev selected the same four specialists as the ordinary selector plus build portability and CLI; neither extra specialist supplied a unique finding. Both experimental review graphs completed their proof checks while retaining a not-ready verdict for the reviewed code. The Jev request took 0.264 seconds and reported 15,127 input tokens and 836 output tokens, for an estimated API cost of $0.000635. Codex per-worker token and dollar costs were unavailable. Selector-workflow time and API latency have different measurement boundaries, so these observations do not establish a speedup ratio. PR #94 records the implementation and pilot summary.

Recommended sequence

1. Completed: preserve the implementation and initial Jev baseline from feat(review-graph): add Jev routing comparisons and accounting #94 and the captured pilot cases. Do not rerun the initial experiment merely to recreate work already done.
2. Next: address graph overhead in review-validator: make asynchronous timing wrappers shell-portable #89 and review-graph: default to compact runtime receipts and validation references #87. Alongside that work, fix the two small Jev reporting/error-handling gaps listed below. Completing every remaining acceptance item in review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73 is not a prerequisite for starting those overhead fixes.
3. Then: replay the saved responses to validate the reporting correction without new API calls, and repeat the existing captured comparison after the overhead fixes. Reconcile the remaining acceptance criteria, preserve the measurements and limitations, and record the final disposition of review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73.

The remaining repeat comparison follows #87 and #89; the initial pilot has already happened. A larger, previously unseen scope is optional follow-up, not a new condition for closing this bounded evaluation. #88, #90, and #91 are not added as prerequisites here. #85 is the separate Dependabot rollout. If production adoption would change dispatched work, #74 owns its additional evaluation before rollout.

Leave staging, commits, and pushes to the user unless explicitly authorized. Correctness and proof guarantees remain required independently of Jev's performance.

Concrete follow-up and closeout

• Correct the primary routing report to expose selected catalog IDs and disagreements at the chosen inclusion cutoff (0.5 for the current pilot). Its current primary report uses 0.2/0.8, while 0.5 appears only in aggregate sweep metrics. Keep incomplete/failed evidence unknown and distinguish inclusion decisions from probability calibration or correctness evidence.
• Catch malformed HTTP/protocol errors in scripts/typesafe_check.py without printing response text or exception details. Add a regression for the reproduced BadStatusLine path and preserve the credential-redaction contract.
• Replay saved experiment responses to validate the corrected selections and disagreements without new inference requests or overwriting the original evidence.
• After review-graph: default to compact runtime receipts and validation references #87 and review-validator: make asynchronous timing wrappers shell-portable #89, repeat the existing captured comparison and distinguish replay-only reporting changes from newly observed routing/review work. Preserve frozen baselines, source/catalog/model/policy identities, failures, overlap, and unavailable measurements.
• Complete the remaining independently reviewed applicability labels, specified failure fixtures, and configurable budget controls described below. Reconcile each acceptance item with retained evidence or an explicit conditional disposition; do not silently treat incomplete work as complete.
• Retain a non-secret, reproducible evidence summary and accessible supporting records for issue closeout. The local pilot artifacts currently reside in temporary storage; the public PR summary is not a replacement for those records.

The initial no-promotion decision is already made. Revisit it after the planned repeat; keeping shadow mode remains a valid outcome and does not require the production-adoption benchmark in #74 or integration work in #75.

Measurement scope transferred from #72

#72 supplied the correctness/recovery work and protocol-overhead baseline and is closed following #80. Its scripted reads, seeded findings, and projected concurrency are not observations of live review-agent behavior.

This issue owns the bounded live routing measurement work. On identical captured cases, record source/excerpt reads and bytes, coordinator operations and elapsed routing work, adapter latency/retries, observed context/token usage, and incremental cost. Separate coordinator and adapter time, account for overlap, and bind records to source, catalog, model, and policy identities. Leave unavailable observations explicitly unknown; do not infer billed tokens from byte counts.

Shadow suggestions must leave normal dispatches and coverage unchanged. Reuse the existing protocol benchmark and add only the instrumentation needed for this experiment. A separate full-graph semantic-recall comparison is not a prerequisite for shadow evaluation. If adoption would change dispatched work, #74 owns that comparison before rollout. A no-go result does not require the adoption-specific benchmark or #75.

Implementation contract and remaining requirements

1. Optional adapter at the existing routing interface

PR #94 implements review-graph/scripts/review_graph_routing_experiment.py, using the routing catalog/projection and bounded supplied source evidence. Retain this adapter boundary rather than moving TypeSafe transport into the deterministic planner.

Use the complete catalog candidate set, semantic triggers, explicit request scope, and relevant source excerpts. Ask independent Noul applicability questions per candidate, with enough evidence to identify the applicable paths. Several candidates may apply to the same change. Batch independent questions over shared state; make a later request only when earlier answers are needed to construct new evidence or options.

Keep TypeSafe-specific transport and credentials outside the deterministic planner. Preserve the upstream vendored skill.

2. Preserve graph authority and required coverage

Initially produce an advisory artifact only: shadow results must not alter routing_overrides, dispatches, selected paths, or completion.

If evaluation supports later adoption, start with coordinator-confirmed semantic additions and ambiguity handling through existing validated routing_overrides. Preserve classifier/projection requirements, shared ownership, required independent review, synthesis, and validation.

The planner and runtime continue to own catalog identity, paths, priorities, dependencies, exact reuse, fingerprints, accepted evidence, and proof completion. Model probability is neither applicability evidence by itself nor authorization to exclude work. Do not feed routing judgments to the conclusion-blind independent reviewer.

3. Explicit uncertainty and failure behavior

Represent insufficient evidence, ambiguous answers, malformed/incomplete responses, stale inputs, and service failures separately. Fall back to normal coordinator routing without losing required coverage.

The pilot's 0.5 inclusion cutoff is uncalibrated and governs experimental review selection only. Any tuning of question definitions or thresholds requires representative labeled development cases and a held-out evaluation; do not infer calibration from a single observed disagreement or copy cookbook thresholds. Preserve genuine no-match cases and multi-skill cases. Do not prune required reviewers or choose a single winner for the entire graph.

4. Reproducible records and bounded cost

Bind cached suggestions to captured source identity, full input/evidence content, catalog/projection identity, question definitions, adapter policy, and a pinned model version. Preserve the model actually returned, raw typed judgments, usage, latency, and failure reasons in external artifacts. Keep advisory decisions distinct from accepted review evidence.

Provide configurable token, request, retry, and cost limits, with a proposed $1 pilot budget and local accounting. Replays and ordinary offline tests must not call the service.

Credentials and API execution

• Live calls require a TypeSafe API key; offline development and recorded-response tests do not.
• Read TYPESAFE_API_KEY in the helper process.
• For local use, store the key in 1Password and launch only the helper with op run, using a secret reference supplied outside public repository defaults.
• Never store the key in tracked files, logs, prompts, or artifacts. Keep the actual account/vault reference local.
• Keep the SDK optional and manage it with the repository's uv workflow. Recheck current SDK/API contracts before implementation.

Evaluation and acceptance criteria

• Offline fixtures cover multi-skill applicability, no match, ambiguous ownership, missing evidence, stale captures, partial responses, timeouts, and rate limits.
• Shadow execution leaves the normal graph plan and coverage unchanged, including on every failure path.
• Extend existing routing fixtures with representative source evidence and independently reviewed expected applicability; path-only fixtures are not sufficient semantic evaluation.
• Compare normal coordinator routing and Jev shadow suggestions on the same captured cases. Report missed applicable skills, unnecessary selections, coverage by risk category, observed source/excerpt reads and bytes, coordinator operations and elapsed work, context/token use, adapter latency, retries, and incremental cost. Distinguish observations from review-graph: reduce orchestration overhead and handle blocked validation reliably #72's projections/scripted measurements, preserve unknowns, and retain reproducible measurement records.
• If question definitions or thresholds are tuned, use a held-out evaluation set; otherwise explicitly record that no tuning or calibration claim was made. A larger unseen experiment is not required merely to repeat the frozen pilot.
• Report the initial adoption decision based on measured benefit and limitations: retain shadow mode, as recorded above. Confirm or revise that disposition after the planned repeat. Ship automatic additions only in a subsequent scoped change if justified; coverage pruning is outside this proposal.
• Validate the merged implementation through repository recipes respecting the Git-mutation policy: feat(review-graph): add Jev routing comparisons and accounting #94 passed just security and just ci, including 805 tests; its PR CI passed. Follow-up changes still require their own appropriate validation.

Cost context and references

At planning time, the published Jev rate is $0.042 per million input tokens, with free output tokens. At an illustrative total of 25,000 input tokens per routing run, 1,000 runs cost about $1.05 in TypeSafe usage. This is not measured graph usage and excludes normal review-agent costs, account-specific credits, and purchase minimums. Recheck pricing before a live evaluation.

• TypeSafe skill suggestion cookbook: useful progressive-disclosure pattern, but its single-winner design must be adapted to overlapping graph coverage.
• Noul primitive
• Python SDK
• Current models and pricing
• 1Password process-scoped secret injection