Skip to content

feat(audit): real-task harnesses — products-admin CRUD build + real-data modes - #620

Merged
LinuxDevil merged 1 commit into
mainfrom
audit/real-task-harnesses
Oct 12, 2026
Merged

LinuxDevil merged 1 commit into
mainfrom
audit/real-task-harnesses

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

Summary

Runs all six live harnesses against OpenRouter with a working key, adds a new real-task coding driver, and gives the canned harnesses real-data modes.

  • audit/products-admin/ (new): the coding-agent machinery — plan mode, approval-gated fs+shell, durable fileStore, loop guard — pointed at a genuine greenfield objective: scaffold a products admin dashboard CRUD (JSON store, node:http REST, single-page dashboard, node --test suite). The driver then boots the agent-built server and verifies the full CRUD over real HTTP.
  • REPO_FIXTURE=<dir> for repo-maintainer — reviews a real checkout + diff. Ships real-fixture/ built from merged PR fix(server): atomic session ownership claim - deny a lost claim instead of streaming in a foreign session (Eve2-F1) #616's actual diff + files at head.
  • CORPUS_DIR=<dir> for research-analyst — researches real documents. Verified against docs/ (64 real SDK docs) with a real question.
  • INCIDENT_TELEMETRY=<file.json> for incident-responder — real telemetry JSON (built from actual run logs + real git history) replaces canned fixtures.
  • hostinger-monitor/companion are inherently real already (live endpoints; live multi-turn conversation with durable memory).

Live results (OpenRouter, alpha.19 packed build @ 06aaff6e)

Harness Result
coding-agent all PASS
companion 13/13 PASS
incident-responder all PASS (canned + real telemetry)
hostinger-monitor all PASS
repo-maintainer 19/19 canned; 19/19 real-fixture
research-analyst all PASS (canned + real corpus)
products-admin all PASS on claude-haiku-4.5 — 15 agent-written tests + live CRUD verification

Harness fix (not an SDK defect)

research-analyst's cost check demanded sum(costUsd) === estimateCost(aggregate) exactly. Per-call billed costs (OpenRouter usage.cost applies prompt-cache discounts) can't bit-match a fresh registry estimate — the check now re-estimates with cachedInputTokens and allows a 15% band for billed runs. Real-corpus run: $0.018895 billed vs $0.018895 estimated.

Honest finding

gpt-4o-mini passes every bounded harness but cannot finish an open-ended greenfield build — wrong ESM specifiers, fetch shadowing, hallucinated node-fetch, and repeating an identical failing command until the loop-guard denied it. The SDK behaved correctly throughout; AUDIT_MODEL selects a stronger model (claude-haiku-4.5 verified green). Documented in audit/REPORT.md and products-admin/README.md.

Test plan

  • all 6 canned harnesses re-run live on OpenRouter — green
  • products-admin end-to-end: plan-mode deny → approval-gated build → node --test green → harness-verified real HTTP CRUD
  • repo-maintainer real fixture, research-analyst real corpus, incident-responder real telemetry — all pass
  • no SDK source changes; audit-only

Generated with Devin

…ata modes for the rest

- products-admin: new driver that points the coding-agent machinery (plan
  mode, approval-gated fs+shell, durable store, loop guard) at a real
  greenfield build — a products admin dashboard CRUD (JSON store, node:http
  REST, single-page dashboard, node --test suite). The harness boots the
  agent-built server and verifies the full CRUD over real HTTP.
- repo-maintainer: REPO_FIXTURE=<dir> reviews a real checkout + diff; ships
  real-fixture/ (the merged PR #616 diff + files at head). B5 adapts: a real
  diff has no planted bugs, so the check asserts a coherent typed review.
- research-analyst: CORPUS_DIR=<dir> researches real .md/.txt/.mdx docs
  (verified against ../docs). Cost check fixed: providers that bill per call
  (OpenRouter usage.cost applies cache discounts) can't bit-match a fresh
  registry estimate — re-estimate with cachedInputTokens and allow a 15%
  band; exact equality still required when nothing was billed.
- incident-responder: INCIDENT_TELEMETRY=<file.json> replaces canned
  telemetry; real-telemetry.json is built from actual run logs + git history.
- REPORT.md: real-task round results, including the honest finding that
  gpt-4o-mini cannot finish open-ended builds (use AUDIT_MODEL override).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@LinuxDevil
LinuxDevil merged commit c446647 into main Oct 12, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant