Skip to content

docs(benchmark): first run results, config and honest reading - #27

Merged
log0u7 merged 2 commits into
mainfrom
docs/benchmark-first-run
Oct 26, 2026
Merged

log0u7 merged 2 commits into
mainfrom
docs/benchmark-first-run

Conversation

@log0u7

@log0u7 log0u7 commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • First end-to-end benchmark run published in docs/explanation/benchmark.md: config (3 models, local llama.cpp teacher on the 3080 for batch 2, judge_independence warn documented), data (23 CTF cases, 15 accepted), training (QLoRA Qwen3-1.7B, 6 pre-cutoff records), eval on the 9-record post-cutoff holdout.
  • Numbers: base and student both 0.00 accuracy (prose output, no CtfVerdict JSON at the 6-record scale; the student degenerates into repetition loops). The honest reading: the pipeline is demonstrated end to end, the gain needs the 500-2000 record scale-up (ROADMAP phase 2).
  • Four defects the run surfaced, all fixed and referenced (PR 24/25/26 + the warmup-vs-steps pitfall documented).
  • README status row + ROADMAP benchmark item updated (ticked with the no-gain note), CHANGELOG entry.

Issue for this PR

N/A

Checklist

  • Conventional Commits
  • Docs only (benchmark.md, README, ROADMAP, CHANGELOG)

End-to-end run on real data: 23 CTF cases (15 accepted), QLoRA on the
3080, task-eval on the 9-record post-cutoff holdout. Numbers: base and
student both 0.00 accuracy (prose output, no CtfVerdict JSON at the
6-record scale). The reading, the config, the four defects the run
surfaced and the scale-up lever are documented.
@log0u7
log0u7 merged commit 26a5bf1 into main Oct 26, 2026
8 checks passed
@log0u7
log0u7 deleted the docs/benchmark-first-run branch October 26, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant