Run v2 on fresh Microcosm households in judged batches by MaxGhenis · Pull Request #240 · PolicyEngine/policybench

MaxGhenis · 2026-10-22T12:37:22Z

WIP. This builds PolicyBench v2's head-to-head on fresh households from the latest Microcosm build, run in small batches with AI judges between them. Max's direction (2026-10-22):

"if were doing v2 i wanted to use the latest microcosm"
"we should go slowly since the new microdata could reveal different pe bugs - we should do ai judges in small batches or real time"
Data: the d1811 Route A release. Engine: policyengine-us 2.58.0, served as one certified pair by a policyengine.py bundle certified on a branch.

In this PR so far

scripts/household_set.py builds a household set in the frozen board's layout:

Sampling uses the board's own sampler and eligibility rules, from the dataset the installed policyengine.py bundle certifies.
Households get fresh ids, mc_000 onward.
References come from the installed policyengine-us plus the reference conventions (latest_final.py, pins checked).
It writes the households in batches.

The set stays outside the repository until a release, so a web arm can't find its answers.

Check of the reference path. check-frozen runs the builder on the frozen board's own 100 households with policyengine-us 2.41.0. It reproduces all 1,935 scored references within 1e-6.

tool_pilot_argon.py --board DIR:

The engine pin and fix-module pins come from the board's last engine_upgrade revision.
On a household set, each arm is scored against the same model's no-tools run (--baseline-dir), which must use the same prompt version.
The frozen board's behaviour is unchanged.

Progress (2026-10-22)

Batch judge. The batch judge, item 1 of the plan below, is built:

scripts/batch_judge.py: payload and derivations.
The adversary takes a payload's outputDefinitions and answerCondition.
The Claude runner can bill a declared API key (JUDGE_API_KEY_SECRET), in a fresh config directory, refused alongside any login.

Dry run. On the frozen board's no-tools v2 answers it flagged 23 of 1,935 cells at 2 of 3 models.

Rehearsal (two frozen households, 2.41.0): one full batch ran end to end, covering 12 model-arm runs, scoring, the trigger, the blind two-stage judges (6 valid verdicts) and collection.

The Opus judge ruled scenario_031's CA SNAP ($0) reference_wrong under CalFresh modified categorical eligibility.
policyengine-us 2.49.13 fixes exactly that.

Engine upgrade, frozen board on 2.58.0 with latest_final: 6 of 1,984 outputs move.

The only scored move is 031 SNAP, $0 to $288: the CalFresh fix.
Four excluded outputs land exactly on their audited corrected values: 2.50.0, disability benefits in gross income.
One excluded SNAP output moves and stays excluded.

Judges (Max: "should we use gpt as judge for this?"):

GPT-6.1 Sol judges every case on a Codex lane.
Opus gives a second opinion, on the API key, only on cases Sol does not hold.

2026-10-23:

v2 waits for a clean 2.60.x bundle (Max). household_set.py build --purpose v2 refuses anything but a certified pair. The Microcosm recertification is tracked in Recertify a published US release for a newer policyengine-us without a rebuild microcosm#1366.
Test sets (--purpose test) may name a published dataset. Test batch 0 ran 10 households from the Route A build 0fb05b6 on 2.60.4, for $17.06.
Calculator tool version 2 (the pe2 arm, 6e1578f).
- It reads prompt labels and output keys as engine names.
- It places misplaced inputs.
- It reads realized contributions as _desired and flags inputs that replace a formula.
- It explains a variable with its parameters and cited sources.
Tool version 1 is unchanged.

Update (2026-10-23 evening). The interface tests run on cheap models (Haiku 5.5, GPT-6 Luna, Gemini 3.1 Flash-Lite) until v2 is ready. Cheap batch 0 is done; batch 1 waits for a host reservation. Tool v3 (independent calculator features for an ablation pilot, and a finding that the prompts label some facts in words that tool v2 cannot read) is in Tool v3: calculator features one at a time, for the ablation pilot #243, stacked on this branch. It merges here after cheap batch 1, which runs from this branch at 5bbd571.

Plan (to update as it lands)

The batch judge, still to build:
- Export each batch's no-tools and web results as a dashboard payload, with each model and arm counted as one "model".
- Run consensus-flags with a three-model threshold: 2 of 3 models agreeing on the same wrong answer.
- Run adversary-prepare with the v2-pilot prompt and definitions the models saw.
- Use engine traces from the reference system as derivations.
- Run the two-stage blind judge, then adversary-collect.
A policyengine-us 2.58.0 environment, plus checks of the reference conventions and v2 wording against it (fix modules, law-date fence).
Certify the d1811 build with policyengine-us 2.58.0 on a policyengine.py branch, then build the set (100 households, 10 batches).
Run batch by batch: 3 models × 4 arms, then the trigger, the judge, and triage, and only then the next batch.

Invariants

Determinism. references recomputes a set and must match it exactly.
Differential. The builder's reference path reproduces the frozen references on the frozen engine (check-frozen, and a slow test on three households).
Batches. They partition the draw in order: concatenation is identity, every batch is full except the last (property test).
Pins. A household set's fix modules equal the frozen 2.41.0 pins until an engine-upgrade audit changes them.

🤖 Generated with Claude Code

scripts/household_set.py samples households from the dataset the installed policyengine.py bundle certifies, renames them mc_000 onward, computes every output on the installed policyengine-us with the reference conventions, and writes a board in the frozen layout with batches. On the frozen board's own households it reproduces all 1,935 scored 2.41.0 references exactly. The tool pilot takes --board: the engine and fix pins come from the board's last engine_upgrade revision, and a household set's arms are scored against the same model's no-tools run (--baseline-dir). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-10-22T12:37:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 24, 2026 5:56am UTC

scripts/batch_judge.py writes one arm's answers on a batch as a dashboard payload (each model a board model), so consensus-flags and the two-stage adversary run unchanged: two of three models agreeing off the reference flag a cell. The payload carries what the models saw: the v2-pilot household prompt (checked against each record's sent prompt hash), the v2-pilot definitions, and the condition they answered under. Derivations are pruned engine traces on the household set's own reference system. The adversary takes a payload's outputDefinitions and answerCondition; the default renders both stage headers byte for byte as before. Dry run on today's frozen-board no-tools answers (v2 wording): 23 of 1,935 scored cells flagged, 12 of them among the 2026-10-05 adversary's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Max, 2026-10-22: "in case youre running any claude api stuff please use the key on my clipboard". With JUDGE_API_KEY_SECRET naming an agent-secret entry, the judges bill that key and nothing else: it refuses a lane config, a lane token, AUDIT_ACCOUNT or the desktop opt-in alongside it; every call runs in a fresh empty config directory removed at exit; the key joins only the claude calls' allowlisted environment; and check-login requires a first-party api_key login from ANTHROPIC_API_KEY in that directory. Sidecars record the mode and the secret's name, never the key. The lane and desktop paths are unchanged. Why now: the desktop login's CLI session had expired, and the Subfleet lanes were closed, so the batch judges had no login to run on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The adversary prompt tells a judge to exclude policybench.org, policyengine.org and GitHub on every search. Codex's search tool takes no blocked_domains, so GPT-6.1 Sol did it with -site: operators, and the audit rejected those stage-1 outputs as searches naming PolicyEngine (rehearsal, 2026-10-22: 2 of 3). searched_for() drops a -site: exclusion of a blocked domain before the contamination check, in both runners' audits. A query that still names the engine or the benchmark (beside exclusions, under site:, -inurl:, or a look-alike domain) is still rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Max, 2026-10-23: "we can still test before that but i'd like v2 to be clean". household_set.py build takes --purpose v2 (the default) or test. A v2 set needs the installed policyengine.py bundle to certify its own dataset for the installed engine; anything else is refused unless --purpose test. A test set may name a published dataset (--dataset-repo/--dataset-file/ --dataset-revision); it records the release manifest's build engine and compatibility claim, purpose test and certified_pair false. Engines after policyengine-us 2.2.1 ignore Populace's filing_status_input and derive filing status from the household; the build records each sampled household whose stated status differs from the engine's (its scenario and reference carry the engine's). The sampler takes the dataset as an option; the default call is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Max, 2026-10-23, on what test batch 0 showed about the calculator ("yes go"): - Names: a name written with spaces (as the prompt labels its facts) or a benchmark output key (free_school_meals_eligible, head_medicaid_eligible) is read as its engine variable, and the result's notes say how. Every model had a call refused for a guessed name in all 10 households. - Placement: an input set on another entity moves to the one unit that holds the people it was set for (a group left out becomes one unit of everyone, as the engine reads it); otherwise it stays an error. - Overrides: a contribution set as the realized amount is read as its _desired input (Sol and Opus put "traditional 401k contributions desired" into traditional_401k_contributions in 6 households, moving tax), and an input the engine computes and no stated fact sets is flagged as replacing its formula (Sol set rent, not pre_subsidy_rent). - Explanations: explain (up to 3 variables) returns the computation steps and the parameters read, with values and cited sources. Opus overrode a correct free-meals answer in Louisiana, which pays the reduced-price copay. Tool version 1 (pe, web+pe) is byte for byte as before. The notices and tool description now state the board's engine (a household set on 2.60.4 had been told 2.41.0); the frozen board's text is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Max, 2026-10-23: "thinking we might want to do all these tests with a cheap model until we're super ready for v2 with the web and tool experiments". batch_judge.py payload takes --models (the frontier three by default). The payload's household prompt is the first tool-contract model's, checked against that model's records, and the payload records which models it holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The calculator's six features (names, placement, overrides, explain, concise, flat) are each on or off, so a study can change one at a time. "pe2" keeps tool version 2 (the first four, its notice byte for byte), "pe3" offers all six, and any other set runs as "pe:" plus its features, e.g. "pe:names+flat". - concise: an explanation gives the value, its direct inputs and up to five parameters unless `detail` is "full". - flat: the household is described as a state, people with a role (head, spouse or dependent) and facts, and household facts; the tool builds the units the benchmark's own situations have and places each household fact on its entity. Tests: the v2 notice is reproduced from its features; each feature adds only its own behavior; flat builds the benchmark's units and refuses what it cannot place; and on the installed engine eight frozen-board households described flat compute every board output exactly as Scenario.to_pe_household does. Five mutants of the flat and arm logic are all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The prompts label some facts in words rather than as the variable name with spaces: employment_income is "gross wages and salaries", hours_worked_last_week "usual weekly hours worked", has_esi "has employer-sponsored insurance" (policybench.prompts.INPUT_LABEL_OVERRIDES). In the 20-household test set nearly every household has at least one; hours appear in 17, employer coverage and premiums in 11 each. Tool version 2's names feature cannot read these, and its notice says the facts carry the variable names. A Flash-Lite smoke test of the flat format met three such errors and fell back to setting employer_sponsored_insurance_premiums to 0 and weekly_hours_worked, which the SNAP work rules do not read. The new labels feature reads a fact under its prompt label (case and punctuation aside, renames applied) and, as the reference situations do, also sets stated usual weekly hours as weekly_hours_worked_before_lsr (PE_INPUT_ALIASES). pe3 now offers all seven features. pe2 is unchanged and keeps its notice as it ran; every other feature set's notice says only what the tool reads. Tests: label reading, the alias (and leaving a set value alone), pe2 unchanged, the variable-name claim only in pe2's notice across every feature set, every prompt label reading back to its engine name, and a differential: twelve frozen-board households described flat with each fact as its prompt writes it compute every board output as the benchmark does. Three mutants (no alias, no renames, strict label matching) are caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The v2-pilot prompt relabels two facts itself ("home energy assistance received" for spm_unit_energy_subsidy, "employer-paid health insurance premiums ..." for employer_sponsored_insurance_premiums); the labels feature now reads them, derived from V2_PILOT_CONVENTIONS so the two stay in step. In the flat format a person given no name is named by role (head, spouse, dependent_1, ...): Flash-Lite's first flat call in two smoke households left the names out. The label differential now writes each fact as the v2-pilot prompt does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Two of three Flash-Lite smoke households gave the head's rent as a household fact. pre_subsidy_rent is a person variable in policyengine-us, so the flat format refused it. Under the placement feature a person fact given for a household of one now goes to that person, as placement moves a person variable off a one-person unit; with two or more people it is still refused, now saying to give it under the person whose facts list it. The test registry's entities now match the engine's (pre_subsidy_rent is a person variable). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A differential against 5bbd571 (the commit the test batches ran) found version 2 requests carrying detail "full"; only execute_request reads it, through .get, so results were unchanged, but the request now matches exactly. 4,000 generated calls through versions 1 and 2, the notices, descriptions, schemas and 15 model profiles are identical to 5bbd571. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

GPT-6.1 Sol's review (REQUEST_CHANGES, 7 findings) at 9c43f4f: 1. A model's arguments could turn on features its arm does not offer: the worker read "__tool_features__" (and, as at 5bbd571, "__tool_version__") from the arguments dict. A pool now sends features in a tuple frame no JSON can be, the worker validates them (names, concise beside explain), and keys like "__tool_version__" in a model's arguments are refused as unknown arguments. No run record ever carried them. 2. A resume could mix treatments. Each household record now keeps its tool features and calculator schema hash; a resume compares features, notice and schema, and refuses a non-legacy record without them. 3. Labels missed what the prompt renders. The label index is now built from the prompt's own renderer (_format_input_line, then v2_pilot_text) for every promptable, labeled and reference-stated input: fallback labels, the Marketplace plan line, "state", partnership or S-corp income. Facts the reference drops (PE_DROPPED_INPUTS) are ignored with a note. A label that names two of the engine's inputs is not read. 4. Flat refuses person keys other than name, role and facts. 5. Flat stores household facts under the name written, so normalization compares them once normalized; roles set the relationship inputs and a stated one that contradicts its role is refused. 6. The flat example uses employment_income; placement replaces the wrong-entity error sentence; "full" detail is the trace, capped. 7. Diagnostics read flat people by their default names. Evidence: 210 passed, 4 frozen-engine skips. A labels-only differential builds 30 frozen households from their prompts' facts and matches the reference situation on every output, and so do all 100 frozen and all 20 test-set households in a one-off run, with or without take-up inputs. pe and pe2 are identical to 5bbd571 over 4,000 calls and 2,000 pool messages. 11 mutants of the fixes are caught. An end-to-end Flash-Lite pe3 smoke run had 0 tool errors on 3 households. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- flat: a household fact repeated with a malformed value was dropped without a word. _same_value now reports the repeat's validation problems, and only a repeat validated as equal is dropped quietly. A malformed first value is reported once, by normal validation, and never as a conflict. - The fallback record for a household the harness failed on carried no treatment, so one failure blocked resuming any custom arm. The record is built by harness_error_record, which now includes the notice hash and profile_treatment. resume_treatment_problems checks only the records a resume keeps, since a rerunnable record is replaced under the current profile. pe, web+pe and pe2 still behave exactly as at 5bbd571: 4,000 calls and 2,000 pool messages match. 5 mutants of these fixes were all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…core A rerun is not sure to replace a rerunnable record: a budget stop, lost workers or a narrower --scenarios leave it in place, and scoring read whatever was there. treatment_problems now checks every record in the folder, rerunnable ones included, and command_score refuses a folder that mixes treatments, rebuilding the notices on the board's engine. Runs written before 6e1578f (test batch 0's frontier calculator arms, b6d835a) were told policyengine-us 2.41.0 while computing on 2.60.4. A folder written wholly so is one treatment and scores with a note; a mix of the two texts is refused, and a resume into it stays refused, since its new records would state the board's engine. All 39 run folders score (628 records); the six frontier calculator folders take the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

) Reviewed head 6b00a6f (GPT-6.1 Sol r4 AGREE); full suite at that head: 218 passed, 4 skipped, ruff clean.

vercel Bot deployed to Preview October 22, 2026 12:39

View deployment

vercel Bot deployed to Preview October 22, 2026 12:47

View deployment

vercel Bot deployed to Preview October 22, 2026 13:19

View deployment

vercel Bot deployed to Preview October 22, 2026 18:29

View deployment

vercel Bot deployed to Preview October 23, 2026 11:14

View deployment

vercel Bot deployed to Preview October 23, 2026 16:43

View deployment

vercel Bot deployed to Preview October 23, 2026 18:20

View deployment

MaxGhenis and others added 5 commits October 23, 2026 17:11

MaxGhenis mentioned this pull request Oct 23, 2026

Tool v3: calculator features one at a time, for the ablation pilot #243

Merged

11 of 12 tasks

MaxGhenis and others added 5 commits October 23, 2026 18:35

Format the r2 fix

fb35d82

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tool v3: calculator features one at a time, for the ablation pilot (#243

5d807bb

) Reviewed head 6b00a6f (GPT-6.1 Sol r4 AGREE); full suite at that head: 218 passed, 4 skipped, ruff clean.

vercel Bot deployed to Preview October 24, 2026 05:56

View deployment

MaxGhenis mentioned this pull request Oct 25, 2026

Confirmatory runner: lanes, one engine pool per model, admission and the ceiling #244

Draft

5 of 6 tasks

Navigation Menu

Repository navigation

Run v2 on fresh Microcosm households in judged batches - #240

Run v2 on fresh Microcosm households in judged batches#240
MaxGhenis wants to merge 17 commits into
mainfrom
v2-microcosm-batches

MaxGhenis commented Oct 22, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MaxGhenis commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

In this PR so far

Progress (2026-10-22)

Plan (to update as it lands)

Invariants

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MaxGhenis commented Oct 22, 2026 •

edited

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading