[ZachFara]

What changed

tools/eval/github_tasks.py gets 1 more verified GitHub task, for 4 in all. That is short of the 10 asked for. I stopped adding tasks rather than lower the bar. The record follows the existing style: a source comment citing the issue, the fix and the upstream release notes; EXCLUDE_NEWER in the setup; and the same tests() helper.

Task	Repo (license)	Upstream release	upstream_change
gh-pydantic-settings-signature	pydantic/pydantic-settings (MIT)	pydantic 2.14.0, 2026-10-08	pydantic-2.14-field-name-for-signature

• Pairs: 0. The new key has one task. I found no second repo hit by the same change that met the bar (see the rejected candidates).
• Timing: the release came out after the June 2026 cutoff, and the upstream change (Use the field name in __signature__ when validate_by_alias isFalse pydantic/pydantic#13730) merged on 2026-08-29. Issue #1023 was opened 2026-10-09, and its fix #1029 was squash-merged 2026-10-13.
• Squash merge: base_commit is the parent of the squash commit (10dc635).
• Previous version: the base passes on pydantic 2.13.5 and fails on 2.14.0, so the release causes the break.
• Not a hard-set task under decision 0023. The TypeError names the missing validate_by_alias argument, which points at a fix, and the record's comment says so.
• Install size: about 16 MB.

Base failing output (containerized, -v) and cause-word check

• Test: tests/test_source_cli.py::test_cli_app_run_base_model_validate_by_name_only. The traceback ends in CliApp.run (pydantic_settings/main.py:883):
  E TypeError: _field_name_for_signature() missing 1 required positional argument: 'validate_by_alias'
• Cause words: 2.14, added a required, new required. Not private or signature: the traceback ends in pydantic/_internal/_signature.py, so a fix that only describes the symptom would say them (the same rule as tekore's _missing_).
• Checked, lowercased and as substrings, against:
  • the whole captured base output;
  • the whole laid-over test file at the fix sha (5,201 lines);
  • the symptom words, in both directions;
  • the function names run, _init_kwarg_name and _field_name_for_signature.
• No hits.
• The PR's other new test (test_cli_app_run_base_model_aliases) is left out: it fails at the base on any pydantic version, because a non-identifier alias never worked.

Leak review of the laid-over test

I read every comment, docstring and test name in tests/test_source_cli.py at the fix sha, plus the fix's whole diff. The new tests have no docstrings or comments. None of the file's 179 test names or its comments mention pydantic versions, signatures or private helpers.

Rejected candidates

• fix(orderbook): use pl.len() for trade counts (polars 2.0 compat, refs #162) TexasCoding/project-x-py#168 (polars 2.0 pl.count()): verified that it fails before and passes after, and that the base passes on polars 1.44.2. Rejected on review: the fix (pl.count() → pl.len()) has been a polars deprecation since 0.20 (2023), so it was widely answered before the cutoff, and the captured log names pl.len(). Only the hard error is new in 2.0.
• Test against polars 2.0; fix prefix_roots row order under streaming engine mmore500/phyloframe#114 (polars 2.0 join order): the fix changes no test; no license (NOASSERTION).
• Support polars 2.x (supersedes #415) mmcdermott/MEDS_transforms#418 (polars 2 join order): the fix changes no test.
• Support polars 2: lift the cap, lock 2.0, make row order explicit bourgeron-lab/genetics-viz#10 (polars 2 join order): the new test's docstring states the cause.
• release: 4.0.5 — polars 2.0 rolling warm-start fix irena-flextool/flextool#347 (polars 2 cross join order): the test module's docstring states the cause and the fix.
• feat(deps): specsolve runs on polars 2.0 and requires it fluxopt/specsolve#1857 (polars 2.0 join order): test docstrings state the cause.
• fix: merge で pandas と同じ行順を保証し、polars 2 に対応する ribuka/polars-helpers#11 (polars 2 merge order): no license.
• fix: validating a misaligned frame no longer panics on Polars 2.0.0 MaxwellB13/polspec#91 (polars 2.0 list.eval panic): the test docstring states the fix.
• fix: support polars 2.0; bump polars, ruff, urllib3 and virtualenv GraysonBellamy/pyngb#241 (polars 2 zero-width frame): the only test change is reformatting.
• feat(polars): support polars 2.0 null-equality rename capitalone/datacompy#593 (polars 2.0 removed join_nulls): the changed tests don't join, so they don't reach the code fix. The rename dates from before the cutoff.
• fix(deps): support polars 2.0 by replacing removed APIs PovertyAction/datasure#329 (polars 2.0 removed NUMERIC_DTYPES): removal announced at 1.0; Streamlit app.
• fix: keep clinical reports stable under polars 2.0 opentargets/mira#85 (polars 2.0 explode default): same cause as the hold-out real-polars-explode-empty.
• Fix incremental load failure in Polars 2.0 SneaksAndData/adapta#641: the tests need Trino and Iceberg services.
• Read CSV results through fsspec for Polars 2.0 pyathena-dev/PyAthena#1169, fix(cfb,codegen): declare CFB player loader columns; read release footers under polars 2.0 sportsdataverse/sportsdataverse-py#721, Polars 2.0.0 in the lock, and fits on Polars frames work under Polars 2 StrudelDoodleS/superglm#518, Migrate to Polars >= 2.0 and fix four tie-break bugs bkelly-lab/jkp-data#309, Follow the casts, the kernels and the API of polars 2.0 thomasblanchet/astrolabe#16: test comments or docstrings state the cause, the repo is too large, or the fix is too broad.
• Fix from_wcsfile reading an empty blocking field as NaN with pandas 3.1 spacetelescope/slitlessutils#167 (pandas 3.1 converter NA detection): the install is far over 200 MB.
• FIX EventsInput: keep 'n/a' labels in event_factors with pandas 3.1 Eelbrain/Eelbrain#261 (same converter change): the fix changes no test.
• Preserve typed table cells with Pandas 3.1 openvax/topiary#508, Fix predict-scan tests reading empty C-terminal flanks as NaN under pandas 3.1 (2.3.18) openvax/mhcflurry#517 (same converter change): dependencies too large.
• Keep literal words and blanks right under pandas 3.1 lacclab/scanpath-studio#524 (pandas 3.1 NA reading): 505 MB repo.
• merge: support pandas 3.1 (_get_combined_index returns a tuple) metagenopolis/meteor#136 (pandas 3.1 private _get_combined_index): the test comment states the cause, and the test fails at the base on any pandas.
• covariate_labels(): a sort key pandas 3.1 can sort itemresponsewarehouse/Python-pkg#128 (pandas 3.1 tuple sort key): the test comment states the cause and the fix.
• deps: pandas 3.1.0, with str-dtype fixes weirdapps/etorotrade#300: the test docstring names the cause (pandas 3 str dtype, from before the cutoff).
• fix: empty resultsets fail to expand on pandas 3.1 meridianlabs-ai/inspect_scout#747, pandas 3.1: read mapped link distances as floats ethanbuckley/swimsignal#354, fix: market-history collector fails on pandas 3.1 (bool * Timedelta) emarti13/Tennis_Quant_Lab_V5#75, Support latest pandas (3.1) and bump certifi, virtualenv past security advisories appeler/ethnicolr2#73: the fix changes no test.
• Parity replay: issued means no rejection reason; survive pandas 3.1 NaN vishalrana/stock-recommendation-engine#51, Polars 2.0 follow-ups: bin_intervals in the xG audit, test types HeikkiMatias/nhl-edge#242: no license.
• Fix stats collection crash with pytest 9.2 boxed/mutmut#616 (pytest 9.2 removed Node._nodeid): the fix changes no test.
• fix: impacted-tests reads configuration as pytest 9.2 does; pytest 9.2.0, mypy 2.4.1 promptromp/pytest-impacted#115, Support pytest 9.2 without losing xdist grouping nredd/pytest-airflow-in-a-box#385, test: restore xdist loadgroup grouping under pytest 9.2 (#6188) stranske/Trend_Model_Project#6192 (pytest 9.2): test docstrings and comments state the cause; Airflow; 247 MB repo.
• Ensure Quantities work with scipy >= 2.0 special functions. astropy/astropy#20771 (SciPy 2.0 special-function wrappers): a test comment states the cause. Fix CircularGaussianPRF and CircularGaussianSigmaPRF with Quantity inputs on SciPy 2.0 astropy/photutils#2533 (same change): the fix changes no test.
• Read field names when validate_by_alias=False on pydantic 2.14 pydantic/pydantic-settings#1022 (same upstream change as the accepted task): same repository, so not a two-repo pair.

Why

Ticket: MIT-618 (second batch), the GitHub-task eval. These are real post-cutoff tasks for the arms run.

How it was tested

• The task was checked with a helper that calls tools/eval/repos.py and tools/eval/container.py directly, the same steps agent_eval.py check runs:
  • it materialized the base plus the fix's test file, and the fix tree;
  • it ran setup, pins and tests in throwaway containers, one at a time;
  • it kept the verbose output.
• Result: fails before, passes after, and the base passes on pydantic 2.13.5. No --on-host was used, and no containers were left behind.
• server/tests/test_eval_repos.py and test_eval_grader.py pass.
• make check passes.

Learned

• Before verifying a candidate, ask whether its fix was answered before the cutoff, not only whether the release came after it. A deprecation that becomes a hard error (polars pl.count()) fails that test even when the release is new.
• Polars 2.0's _polars_runtime_32 is 169 MB installed, so every polars 2.0 task lands near or over the 200 MB install guideline.
• cause_words like "2.0" match ordinary float data ("21002.0") and timestamps. Check them against the full captured output.
• Most polars 2.0 join-order fixes (six seen) either add no test or have a test docstring that states the cause. Finding a pair there probably means reading the fix diffs, not searching titles.
• gh search prs with a bare version string such as "polars 2" matches unrelated PRs numbered 2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y