Skip to content

review-graph: adopt TypeSafe routing only after the shadow evaluation passes #74

Closed as not planned

Description

@acgetchell

Objective

Turn a successful #73 shadow evaluation into an optional production routing provider. A documented no-go result from #73 should close this proposal as not planned rather than force adoption.

Scope

  • Use review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73's measured applicability results and calibrated questions; start with coordinator-confirmed semantic additions and ambiguity handling.
  • Preserve deterministic selection floors, shared ownership, independent review, proof identities, exact reuse, and all validation requirements.
  • Keep normal coordinator routing available when disabled, uncertain, stale, or unavailable. Do not prune required reviews.
  • Expose an explicit opt-in through the existing graph command surface. Keep the client dependency optional and support offline operation.
  • Provide the process-scoped API launcher using local 1Password references and TYPESAFE_API_KEY. Do not publish keys or personal vault references.
  • Bind model version, questions, evidence, decisions, latency, token usage, and configured cost limits to reproducible advisory records.
  • Document the rollout and rollback path without changing the vendored upstream skill.

Deferred review-quality comparison from #72

If the proposed provider changes dispatched review work, evaluate that behavior before production adoption. Reuse #73's captured cases and measurement infrastructure; avoid building a separate full-graph live benchmark before the shadow experiment has justified adoption.

Run the normal and proposed workflows on identical captured source with the same model/settings and a declared four-slot agent budget. Execute all required specialist reviews and the conclusion-blind independent pass. Compare adjudicated findings and required coverage; investigate any loss of valid findings. Preserve exact publication/provenance records and observed source/context reads, coordinator operations, worker elapsed time, and orchestration time, with concurrency/overlap and unavailable measurements explicit. Exercise validation blockers and continuation through a truthful complete or partial/resumable result without discarding accepted audits or rewriting failed/blocked attempts.

#72's scripted protocol findings do not establish semantic recall. This live comparison is conditional on adoption changing dispatched work; it is not a new prerequisite for #73's unchanged-plan shadow experiment. If #73 records a no-go decision, close this proposal as not planned as already specified.

Acceptance criteria

  • review-graph: evaluate TypeSafe/Jev routing in shadow mode before integration #73 records a positive adoption decision with reproducible held-out results.
  • Disabled/offline/failure paths retain the current graph's coverage and readiness behavior.
  • Enabled routing can add justified review work without silently removing any selected path or owner.
  • If adoption changes dispatched work, retain the controlled full-review comparison described above, with required coverage, adjudicated findings, measured costs, and accurate validation/continuation evidence. Resolve any loss of valid findings before rollout.
  • Credentials remain local and live calls are explicitly enabled.
  • Repository checks and focused integration tests respect the no-Git-mutation policy, including temporary fixtures.
  • Document observed benefits, limitations, and rollback instructions.

This issue represents a later implementation decision, not authorization to integrate Jev while #68, #69, or #72 remain unresolved.

Activity

  1. acgetchell commented on Oct 4, 2026

    @acgetchell
    OwnerAuthor

    Closing as not planned, following the disposition in #73 and merged PR #114.

    The prospective eight-case evaluation did not meet its frozen success rule. On 354 resolved applicability labels, Jev at 0.50 missed 9 applicable concerns and selected 21 unnecessary concerns; Jev at 0.70 with blinded coordinator assessment missed 10 and selected 10 unnecessary concerns. The ordinary coordinator missed 1 and selected 3 unnecessary concerns. These are independent model labels on a small convenience sample, with differing instruction context, rather than human consensus or a general benchmark.

    The prerequisite for this proposal—a positive adoption decision with reproducible held-out results—was not met. Production routing adoption and its conditional full-review benchmark will not proceed under this issue. The retained evaluation reports and replayable records document the outcome:

    Semantic evidence acceptance was not evaluated by these routing experiments; no claim of improved evidence acceptance is being made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions