Repository navigation
review-graph: adopt TypeSafe routing only after the shadow evaluation passes #74
Description
Objective
Turn a successful #73 shadow evaluation into an optional production routing provider. A documented no-go result from #73 should close this proposal as not planned rather than force adoption.
Scope
- Use review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73's measured applicability results and calibrated questions; start with coordinator-confirmed semantic additions and ambiguity handling.
- Preserve deterministic selection floors, shared ownership, independent review, proof identities, exact reuse, and all validation requirements.
- Keep normal coordinator routing available when disabled, uncertain, stale, or unavailable. Do not prune required reviews.
- Expose an explicit opt-in through the existing graph command surface. Keep the client dependency optional and support offline operation.
- Provide the process-scoped API launcher using local 1Password references and TYPESAFE_API_KEY. Do not publish keys or personal vault references.
- Bind model version, questions, evidence, decisions, latency, token usage, and configured cost limits to reproducible advisory records.
- Document the rollout and rollback path without changing the vendored upstream skill.
Deferred review-quality comparison from #72
If the proposed provider changes dispatched review work, evaluate that behavior before production adoption. Reuse #73's captured cases and measurement infrastructure; avoid building a separate full-graph live benchmark before the shadow experiment has justified adoption.
Run the normal and proposed workflows on identical captured source with the same model/settings and a declared four-slot agent budget. Execute all required specialist reviews and the conclusion-blind independent pass. Compare adjudicated findings and required coverage; investigate any loss of valid findings. Preserve exact publication/provenance records and observed source/context reads, coordinator operations, worker elapsed time, and orchestration time, with concurrency/overlap and unavailable measurements explicit. Exercise validation blockers and continuation through a truthful complete or partial/resumable result without discarding accepted audits or rewriting failed/blocked attempts.
#72's scripted protocol findings do not establish semantic recall. This live comparison is conditional on adoption changing dispatched work; it is not a new prerequisite for #73's unchanged-plan shadow experiment. If #73 records a no-go decision, close this proposal as not planned as already specified.
Acceptance criteria
- review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73 records a positive adoption decision with reproducible held-out results.
- Disabled/offline/failure paths retain the current graph's coverage and readiness behavior.
- Enabled routing can add justified review work without silently removing any selected path or owner.
- If adoption changes dispatched work, retain the controlled full-review comparison described above, with required coverage, adjudicated findings, measured costs, and accurate validation/continuation evidence. Resolve any loss of valid findings before rollout.
- Credentials remain local and live calls are explicitly enabled.
- Repository checks and focused integration tests respect the no-Git-mutation policy, including temporary fixtures.
- Document observed benefits, limitations, and rollback instructions.
This issue represents a later implementation decision, not authorization to integrate Jev while #68, #69, or #72 remain unresolved.
Activity
acgetchell commentedon Oct 4, 2026
Closing as not planned, following the disposition in #73 and merged PR #114.
The prospective eight-case evaluation did not meet its frozen success rule. On 354 resolved applicability labels, Jev at 0.50 missed 9 applicable concerns and selected 21 unnecessary concerns; Jev at 0.70 with blinded coordinator assessment missed 10 and selected 10 unnecessary concerns. The ordinary coordinator missed 1 and selected 3 unnecessary concerns. These are independent model labels on a small convenience sample, with differing instruction context, rather than human consensus or a general benchmark.
The prerequisite for this proposal—a positive adoption decision with reproducible held-out results—was not met. Production routing adoption and its conditional full-review benchmark will not proceed under this issue. The retained evaluation reports and replayable records document the outcome:
Semantic evidence acceptance was not evaluated by these routing experiments; no claim of improved evidence acceptance is being made.
remove(skills): drop vendored TypeSafe skill (#118)