feat(review-graph): add Jev routing comparisons and accounting

.github/DEPENDABOT.md

@@ -127,6 +127,15 @@ approval workflow was not reached. Request a fresh check from the repository's
 Dependabot updates page once the hosted image supports the required version.
 The failed dynamic run cannot be retried with `gh run rerun`.
 
+A [manual uv check](https://github.com/acgetchell/dotfiles/actions/runs/36526883205)
+on September 29 at 05:35 UTC (September 28 Pacific) succeeded against
+`abcb359bedc1769cd823e4414df23c6577316ca4`, but created no PR. Every direct
+dependency needed no update under the existing seven-day cooldown policy after
+the maintainer's dependency refresh. The log still reported hosted uv `0.12.18`;
+without an eligible dependency update, this run does not establish that the
+required-version mismatch is resolved. It also provides no approval or merge
+evidence. Repeat the manual check when an eligible update is available.
+
 Issue #85 remains open until a real dotfiles `uv` PR receives current-head bot
 approval and native auto-merge, followed by CI on its actual merge commit.
 Hosted evidence is also still needed for a preserving base merge, rejection of

README.md

@@ -328,6 +328,44 @@ Invoke it with `$typesafe-ai` in Codex, or ask an agent that has loaded it to
 requires a separately configured `TYPESAFE_API_KEY`; keep credentials out of
 this public repository. Installing the skill does not make API calls.
 
+Store the API key in a 1Password item. Enable the desktop app's CLI integration,
+then use a secret reference to inject the key for one command:
+
+```sh
+just typesafe-local 'op://<vault>/<item>/<field>' just typesafe-check
+```
+
+Replace the reference with your item's **Copy Secret Reference** value. If you
+use multiple 1Password accounts, prefix the command with
+`OP_ACCOUNT='<account sign-in address or ID>'`. The command may prompt for
+1Password authorization. The reference is configuration; never pass the actual
+key as a command argument or commit it in a file. No shell startup-file changes
+are required. `typesafe-local` can wrap other TypeSafe commands the same way.
+
+`just typesafe-check` reads the injected `TYPESAFE_API_KEY`, calls the TypeSafe
+model-list endpoint, and reports only authentication status and model count.
+It makes no inference request and sends no repository content. It does not log
+the key or raw response/error bodies. Keep 1Password's output masking enabled.
+When a cloud environment supplies `TYPESAFE_API_KEY` through its secret provider,
+run `just typesafe-check` directly. Credential provisioning is separate for each
+execution environment.
+
+For the configured Codex Cloud coding environment:
+
+1. Allow HTTPS access to `api.typesafe.ai` and have the environment request a
+   personal **Network secret** named `TYPESAFE_API_KEY` for that destination.
+2. In **Settings > Codex Cloud > Personal vault**, add a **Network secret** with
+   the matching key, enter the API key through the private settings form, and
+   apply it to the coding environment. Personal secrets do not automatically
+   add network destinations to the environment's allowlist.
+3. Save and publish the environment configuration. Once these repository changes
+   are available in its checkout, run `just typesafe-check` in a new cloud task.
+
+The cloud process receives a placeholder; its HTTPS proxy supplies the real
+credential for the allowed destination. The local 1Password session is not
+needed for cloud runs. See the official
+[cloud environment secret configuration](https://learn.chatgpt.com/docs/environments/cloud-environments#configure-environment-variables-and-network-secrets).
+
 To update, review and replace the complete upstream `skills/typesafe-ai/`
 directory, including any references and its license, while preserving the
 local `agents/openai.yaml`. Update the pinned revision above, then run

agents/.agents/skills/review-graph/SKILL.md

@@ -5,14 +5,12 @@ description: "Coordinate provenance-preserving mixed-surface repository reviews
 
 # Review Graph
 
-Deliver the coverage of every applicable focused review while keeping proof
-bookkeeping out of model-authored prose. Reviewers make semantic judgments;
-deterministic scripts own catalog identity, fingerprints, digests, canonical
-artifacts, evidence envelopes, and proof reconciliation.
+Cover every applicable focused review. Reviewers make semantic judgments;
+scripts own catalog identity, fingerprints, digests, canonical artifacts,
+evidence envelopes, and proof reconciliation.
 
-Read [the runtime contract](references/runtime-contract.md). Execute the helper
-scripts without reading their implementations unless a script fails or this
-task changes them.
+Read [the runtime contract](references/runtime-contract.md). Execute scripts;
+inspect implementations only for failures or requested changes.
 
 ## Profiles
 
@@ -38,9 +36,8 @@ contract with `worker_created: false`.
 
 ## Route Compactly And Exhaustively
 
-Use `references/routing-catalog.json` through
-`scripts/review_graph_plan.py`. The planner owns catalog IDs, router IDs, rule
-IDs, skill paths, priorities, and synthesis dependencies.
+Use `references/routing-catalog.json` through `scripts/review_graph_plan.py`,
+which owns catalog/router/rule IDs, skill paths, priorities, and synthesis dependencies.
 
 1. Apply the deterministic repository classifier to captured paths.
 2. Run `review_graph_runtime.py routing-projection` for the consulted routers.
@@ -58,6 +55,10 @@ IDs, skill paths, priorities, and synthesis dependencies.
 Every applicable leaf remains required. Resolve late handoffs before dependent
 validation or synthesis.
 
+For requested TypeSafe comparisons, follow
+[the shadow experiment](references/routing-experiment.md). Freeze ordinary routing
+first; preserve its decisions and proof gates.
+
 ## Execute Review Nodes
 
 Dispatch selected leaves with exact skills and owned paths. Workers stream
@@ -66,9 +67,8 @@ commands; the runtime validates before writing, binds approval retries, and
 atomically publishes. They return the publication receipt; `compile-node` reads
 the bound bytes without a second conversational copy. Workers do not author
 fingerprints, digests, evidence IDs, execution metadata, canonical Markdown,
-or machine-evidence JSON. Materialize exact dispatch bases from the accepted
-plan with `review_graph_runtime.py materialize-dispatches`; do not reconstruct
-planner-owned fields in prompts.
+or machine-evidence JSON. Materialize dispatch bases from the accepted plan
+with `review_graph_runtime.py materialize-dispatches`; never reconstruct planner-owned fields.
 
 The materialized command policy is authoritative. Review nodes attest to every
 command and do not execute validator-owned commands without an exact duplicate
@@ -78,7 +78,7 @@ execution identities.
 Identical skill/source/scope leaves execute once; each catalog requirement
 retains ownership of the coalesced judgment and evidence.
 
-Run the capture command before and after execution. Then invoke
+Capture before and after execution. Invoke
 `scripts/review_graph_runtime.py compile-node` with the node ID, materialized
 dispatch set, captures, and journal. It reads the bound payload, seals accepted
 bytes at a read-only content-addressed path, and records that copy in evidence.
@@ -124,9 +124,9 @@ Follow its returned continuation configuration; see
 
 ## Synthesize From A Compact Bundle
 
-Use `synthesis-bundle` rather than complete predecessor reports. Give synthesis
-workers its canonical hashed view, digest, accepted predecessor IDs, and
-exclusions. Keep raw artifacts in the proof store.
+Give synthesis workers the `synthesis-bundle` canonical hashed view, digest,
+accepted predecessor IDs, and exclusions. Keep complete predecessor reports and
+raw artifacts in the proof store.
 
 ## Complete And Report
 
@@ -146,8 +146,12 @@ It derives the mappings, manifest, and `RepositoryReviewProof`; report complete
 only when its verifier returns `complete`. Report repository readiness from
 typed synthesis separately from proof completeness and validation success.
 
-The default user report is compact: findings, changes, validation, blockers,
-selected skills, proof status, final repository state, and artifact-manifest
-location. Persist exhaustive lifecycle, routing, evidence, and resume views in
-the proof store; render them inline only when requested or needed to explain an
-incomplete result.
+Report findings, changes, validation, blockers, selected skills, proof status,
+final repository state, and artifact-manifest location compactly. Keep exhaustive
+lifecycle, routing, evidence, and resume views in the proof store; show them only
+when requested or needed to explain incompleteness.
+
+Report stage costs as measured, estimated, or unavailable using
+[graph accounting](references/routing-experiment.md#account-for-the-whole-graph).
+Set external `REVIEW_GRAPH_USAGE_LEDGER` for runtime timings; preserve unknown
+counts and failed/unfinished attempts.

agents/.agents/skills/review-graph/references/routing-experiment.md

@@ -0,0 +1,124 @@
+# TypeSafe Routing Experiment
+
+Use this workflow when a TypeSafe applicability comparison is requested. Jev
+runs in shadow mode: ordinary routing, mandatory nodes, validation, and proof
+acceptance continue to control execution. No experimental response is a routing
+override or proof of semantic correctness.
+
+## Run The Pilot
+
+From the dotfiles checkout, inspect the synthetic development inputs offline:
+
+```sh
+just review-routing-experiment
+```
+
+Make the three development requests with an injected credential:
+
+```sh
+just typesafe-local 'op://<vault>/<item>/<field>' \
+  just review-routing-experiment --live
+```
+
+In a cloud environment that already supplies `TYPESAFE_API_KEY`, use
+`just review-routing-experiment --live` directly. The client honors the HTTPS
+proxy. It neither follows redirects nor persists credentials or raw responses.
+Each invocation gets a new external artifact directory. There are no automatic
+retries; a failed attempt is recorded and ordinary graph routing continues.
+
+The portable skill entrypoint is
+`uv run python "$SKILLS_ROOT/review-graph/scripts/review_graph_routing_experiment.py"`.
+Only the Python standard library and sibling graph modules are required.
+
+The default model is pinned to `jev-1.13.0`. The dated rate card estimates input
+cost from provider-reported usage; output is currently free. A different returned
+model fails the comparison and leaves cost unavailable. Check [TypeSafe models](https://docs.typesafe.ai/models)
+before updating the model and rate card together.
+
+## Compare A Real Scope
+
+Use `--input <case.json>` with the shape in
+[`routing_pilot.json`](../scripts/fixtures/routing_pilot.json). Supply the actual
+review request, source text, relevant dependency context, and explicit completeness
+flags. Record the current selector's decisions before examining Jev's answers.
+The same frozen scope should be available to both methods; document extra context
+seen by either method. Never silently truncate or omit dependencies to fit a limit.
+The pilot rejects scopes above 32 KiB and requests above 96 KiB; these conservative
+byte limits are not token estimates.
+
+Use `--plan <plan.json>` to import the existing graph's routing decisions instead
+of manually entering a baseline. The input may be a plan or bootstrap bundle.
+The plan digest is retained, but importing a plan does not independently verify
+that its source capture matches the supplied text. Establish that correspondence
+before treating the comparison as controlled evidence. Unconsulted, blocked, and
+excluded baseline entries remain unknown, not negative labels.
+
+Every catalog leaf receives its own Noul question in one request, including leaves
+outside path-matched language surfaces. Source context is shared; questions contain
+the skill name and catalog semantic triggers. Syntheses, routers, and independent
+review remain graph-owned. The packet retains catalog and skill digests, prompt
+version, exact request, baseline, labels, and response model. Baseline judgments
+and expected answers are never sent to Jev.
+
+Probabilities at or below 0.2 are provisional negatives, at or above 0.8 provisional
+positives, and others uncertain. These thresholds are uncalibrated. Incomplete
+context makes every decision uncertain, regardless of its probability. Failures
+also remain uncertain. The comparison reports disagreements and a stable 20%
+agreement sample for adjudication. Omitted labels are unknown. The pilot's sparse
+labels and frozen coordinator baseline are provisional and share an author; they
+cannot establish independent recall or cost savings.
+
+Quality counts separately compare Jev, the supplied baseline, and deterministic
+path projection alone against available labels. Unresolved required labels reduce
+the reported recall lower bound. These are partial-label pilot metrics; the path
+projection is only one component of the ordinary selector, not its full judgment.
+
+The report also sweeps inclusion cutoffs of 0.1, 0.2, 0.35, 0.5, 0.65, and 0.8.
+Each cutoff includes every skill at or above that probability. Compare selection
+counts and labeled errors; a missing specialist generally costs more than an extra
+review, but extra reviews still contribute to the total budget. These alternatives
+do not alter the shadow execution policy. Reassess stored probabilities for free:
+
+```sh
+just review-routing-experiment --replay-case <experiment-case-directory>
+```
+
+Replay verifies the saved input digest, writes a new comparison, and records no
+new API usage. It does not modify prior reports or relabel the held-out split.
+
+`--split held-out --live` runs only the reserved split. Freeze prompts and
+thresholds before using it; inspecting or tuning against its outcomes consumes
+that holdout. Expand with independently labeled real cases before promoting Jev
+to routing control. This runner deliberately provides no promotion switch.
+
+## Account For The Whole Graph
+
+Set `REVIEW_GRAPH_USAGE_LEDGER` to `<proof-store>/usage.jsonl` for graph runtime
+commands. Each operation records a start and completion with measured wall time;
+failure and interruption do not erase earlier work. Its zero model cost applies
+only to the deterministic process and excludes coordinator inference and compute.
+
+For coordinator/model stages, use `review_graph_usage.py start` before work and
+`finish` afterward. Use stages such as `routing-baseline`, `specialist:<node-id>`,
+`validation:<node-id>`, `synthesis:<node-id>`, and `proof-coordination`. Keep the
+returned attempt ID. Supply actual provider/tool usage when exposed; otherwise
+use null counts and `measurement_source: "provider usage unavailable"`.
+Do not derive request tokens from account-wide allowance changes or byte counts.
+Each retry or fallback needs its own attempt; imported baseline measurements are
+reference data and must not be counted as new requests when replaying comparisons.
+
+The `--usage` JSON for `finish` accepts `input_tokens`, `output_tokens`,
+`cached_input_tokens`, `elapsed_seconds`, `cost_usd`, `cost_basis`
+(`measured`, `estimated`, or `unavailable`), and required `measurement_source`.
+Missing numeric fields remain null. Estimated cost must identify its rate source;
+subscription allowance is not an API invoice. Batched per-question costs are not
+measured and are not allocated by this runner.
+
+```sh
+just review-usage report <proof-store>/usage.jsonl <experiment-case>/usage.jsonl
+```
+
+Include the compact stage/provider totals, unknown measurements, and unfinished
+attempt count in the graph's final report, even for blocked or abandoned work.
+Compare total cost only when baseline, retries, fallback, and downstream review
+coverage are accounted for. A cheap routing call alone does not establish savings.