Skip to content

Disclose stub self-eval; add OpenAI-compatible upstream and real-model benchmark workflow - #18

Merged
patsypppe merged 4 commits into
mainfrom
docs/disclose-stub-provider
Oct 22, 2026
Merged

patsypppe merged 4 commits into
mainfrom
docs/disclose-stub-provider

Conversation

@patsypppe

Copy link
Copy Markdown
Owner

Summary

  • README now states that the self-eval numbers come from a deterministic stub provider, not a real LLM, and that the cost figure is notional.
  • The proxy can translate Anthropic messages to OpenAI-compatible chat completions (MERIDIAN_UPSTREAM_API=openai), so Groq, OpenAI, vLLM and Ollama work without changing the agent, the cassette format or the isolation model. The credential stays in the proxy container.
  • The operator picks the upstream model with MERIDIAN_UPSTREAM_MODEL; the agent cannot change it. Works on the Anthropic path too.
  • New manual workflow real-model-benchmark records and scores the checkout-agent suite against Groq or Anthropic under a hard budget cap. Only the chosen provider's key is exported.
  • GROQ_API_KEY is a recognised credential and gsk_ keys are redacted.

Test plan

  • make check (ruff, mypy --strict, 319 unit tests) green
  • Wire-level tests for the translation and model override against a mocked upstream
  • Docker integration tests not run in the authoring environment (no daemon)
  • real-model-benchmark workflow not yet run against a real provider

Generated by Claude Code

The headline self-eval runs against a deterministic stub, not a real LLM,
and the cost figure is notional. State both next to the numbers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
The proxy translates Anthropic messages to chat completions when
MERIDIAN_UPSTREAM_API=openai, so the agent, cassette format and isolation
model are unchanged and the credential stays in the proxy container.
Adds GROQ_API_KEY and gsk_ redaction.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
The proxy applies MERIDIAN_UPSTREAM_MODEL so the operator, not the agent,
picks the model under measurement. The manual workflow records the
checkout-agent suite against a real provider under a hard budget cap and
uploads pass^k, MDE and verdicts as an artifact.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
The proxy applies the operator-chosen model on the native Anthropic path too,
and the workflow exports only the chosen provider's key so one provider's
credential can never be sent to the other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
@github-actions

Copy link
Copy Markdown

Meridian gate

✅ PASS — suite pass^k moved -0.000 (0.486 → 0.486), within the 0.030 tolerance

Failing trials in this run
  • expired-coupon — SELECT status FROM orders WHERE id = 1042 returned [['pending']], expected [['invoiced']]
  • expired-on-bulk-order — SELECT status FROM orders WHERE id = 1045 returned [['pending']], expected [['invoiced']]
  • expiry-boundary — SELECT status FROM orders WHERE id = 1046 returned [['pending']], expected [['invoiced']]
  • happy-path — SELECT status FROM orders WHERE id = 1041 returned [['pending']], expected [['invoiced']]
  • stale-coupon — SELECT status FROM orders WHERE id = 1044 returned [['pending']], expected [['invoiced']]
  • unknown-coupon — SELECT status FROM orders WHERE id = 1047 returned [['pending']], expected [['invoiced']]
task baseline pass^3 head pass^3
expired-coupon 0.400 0.400
expired-on-bulk-order 0.400 0.400
expiry-boundary 0.400 0.400
happy-path 0.400 0.400
missing-field 1.000 1.000
stale-coupon 0.400 0.400
unknown-coupon 0.400 0.400
baseline head
suite pass^3 0.486 0.486

Sensitivity. Not estimated: all 7 tasks moved by the same amount, so there is no spread to estimate sensitivity from. Treat the verdict as directional.

n=5 · k=3 · status complete · harness errors 0.0% · cost 51c

head run run-1792688640-5228c8 · baseline run run-1792688611-a5370b · manifest sha256:1a6291c59a10ad9b… · reproduce with meridian replay run-1792688640-5228c8

@patsypppe
patsypppe merged commit f527afb into main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants