Skip to content

feat(catalogue): offer GSM8K-Platinum as a second dataset of GSM8K - #132

Merged
aitianshi merged 1 commit into
mainfrom
feat/gsm8k-platinum
Nov 16, 2026
Merged

aitianshi merged 1 commit into
mainfrom
feat/gsm8k-platinum

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #131

What

GSM8K gains a second dataset, GSM8K-Platinum: the test split with its errors cleaned (Vendrow et al., 2025; 110 problems removed, 10 answers corrected), beside the original split, so the same model can be run on each and the difference the benchmark's own errors make can be read.

  • Pin. The file is main/test-00000-of-00001.parquet of madrylab/gsm8k-platinum at revision e762492, checked by its sha256, downloaded by Nullius on the first run as any url dataset is.
  • Unit ids. A Platinum problem keeps the number of the test file's problem it is. All 1,209 Platinum questions are byte-identical to a question of the original test.jsonl, each once and in the same order, so the driver finds the number by the question. Platinum's ids are therefore a subset of the original's, with gaps where a problem was removed; unit 0289 is the same problem in both. A Platinum question the test file lacks is an error, never a new number.
  • Check. Unchanged: the upstream's own extract_answer and is_correct, applied to Platinum's answer field, which has the same #### <number> format.
  • Driver. The only code of ours is loading glue: read the parquet file, name each row by its number in the test file. Its environment gains pyarrow==26.0.1, because Platinum is published as parquet only.
  • Entry. task_count 1319 so the headline is the set of problems, not the sum of two overlapping datasets; the authors' citation beside GSM8K's, as they ask; one Good to know line on the shared numbering.

Touches

catalogue · the standard (GSM8K's definition and driver) · docs and site (the generated GSM8K page) · tests

Verified

  • python3 app/tests/parallel.py: 1988 tests in 116 modules, OK. Route suite (app/ui_tests): 422 tests in 40 modules, OK. catalogue_docs.py --write is clean.
  • nullius standard prove gsm8k --dataset platinum, all 1,209 problems, in a temporary profile: units stable and unique, no leak 1209/1209, nop 0/1209, oracle 1209/1209. That record is the proven.platinum block of this diff.
  • prove gsm8k --dataset test against the changed driver, all 1,319: same ids, nop 0/1319, oracle 1319/1319. Its recorded proof is not touched.
  • Both datasets loaded through the driver: 1,319 and 1,209 ids; every Platinum id is an original id (110 absent); the prompt under the same id is identical for all 1,209.
  • The Original setup run end to end on problem 0289 (an answer Platinum revised from 3 to 4) against a local stand-in model that always answers #### 4: reward 0.0 on test, 1.0 on platinum. A --units file naming a problem Platinum removed is refused on Platinum by name.
  • Browser, 1280 wide, light theme: the builder's Dataset step for GSM8K shows two cards, GSM8K test (selected) and GSM8K-Platinum, each with its four checks; the benchmark page lists both datasets, both proofs and both citations.
  • The new driver test (test_a_platinum_problem_keeps_its_number_in_the_test_file) passes where GSM8K's upstream and driver environment are installed; it is skipped elsewhere, like the driver tests beside it.
  • A real run of the Original setup on each whole dataset, meta/muse-spark-1.3-contributor through OpenRouter, provider default temperature: 1292/1319 (98.0%) on test, 1204/1209 (99.6%) on platinum. Of the 27 misses on test, 18 are problems Platinum removed, 7 are answers Platinum revised to what the model gave, and 2 passed on the Platinum run with an unchanged reference (the model answered differently). Of the 5 misses on platinum, 3 are revised answers where the model gave the original one, and 2 passed on the test run with an unchanged reference. The two jobs compared by unit id with no mapping.
  • Not verified: a run on a Settings host or in the container backend, the dark theme and narrow widths. No before and after images are attached.

Risk

GSM8K's driver environment changes (pyarrow added), so the first run after this rebuilds it. If the pinned Hugging Face file became unavailable, only the Platinum dataset would fail to start, at the download, with the original split unaffected. test_agents pinned GSM8K as the one-dataset example of the proof form; it now checks the selector on GSM8K and the fixed value on HumanEval. Undone by a revert; no stored data changes shape.

GSM8K-Platinum is the test split with its errors cleaned (Vendrow et al.,
2025): 110 problems removed, 10 answers corrected. It runs beside the
original split so the same model can be scored on each.

The file is pinned to a revision of madrylab/gsm8k-platinum and its sha256.
A Platinum problem keeps the number of the test file's problem it is, found
by its question, which is the same word for word in both, so the two
datasets compare problem by problem. The check is unchanged: the upstream's
extract_answer and is_correct. The driver's environment gains pyarrow, to
read Platinum's parquet file.
@aitianshi aitianshi self-assigned this Nov 15, 2026
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 7 of 7 yes · #132 ↔ #131

Merge risk 🔴 MAJOR the judge's reading of the diff
Priority 🟡 MEDIUM the deepest path is app/nullius/catalogue/benchmarks/gsm8k/benchmark.json
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown –
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 The GSM8K entry lists two datasets, the original test split and GSM8K-Platinum, and the builder offers both for GSM8K. ✅ met
2 The Platinum file is pinned: a fixed revision of madrylab/gsm8k-platinum in its address and the file's sha256. ✅ met
3 A Platinum unit carries the id of the same problem in the original test split (its line number there), and the choice i… ✅ met
4 The score of both datasets is the upstream's own extract_answer and is_correct, with no scoring code of Nullius's. ✅ met
5 The original test dataset's units, ids and proofs are unchanged. ✅ met
6 The Platinum dataset's controls are proven (units, leak, nop, oracle) or the entry says plainly that they are not. ✅ met
7 The Supported pages and the docs that describe GSM8K name the new dataset (catalogue_docs.py --write is clean). ✅ met
8 The backend and route suites pass. ✅ met
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

@aitianshi
aitianshi merged commit 06935f8 into main Nov 16, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/gsm8k-platinum branch November 16, 2026 10:00
aitianshi added a commit that referenced this pull request Nov 16, 2026
No linked issue: the owner asked for v0.1.1 as a release, so that the
Hugging Face endpoint setting (#129) reaches PyPI.

## What

Prepares v0.1.1. The version becomes `0.1.1`, and
`docs/releases/v0.1.1.md` gives the highlights since v0.1.0, a
Contributors section, the full changelog of every pull request in
`v0.1.0..main` (#114, #117, #120, #122, #125, #127, #129, #132, #137,
and #135 among the test and CI changes) and the installation notes, in
the shape of v0.1.0's. Install examples, the release guide, the docs'
release-notes link and menu, and the workflows' default tags name 0.1.1.
The release tooling already accepts final `vX.Y.Z` tags and does not
change.

## Touches

CI and skills (the release and publish workflows' default tags) · docs
and site

## Verified

- Backend suite (`python3 app/tests/parallel.py`): 1991 tests in 116
modules, OK. Route suite: 428 tests in 41 modules, OK.
- `catalogue_docs.py --write` and `diagrams.py --write` left no change.
- `python3 app/packaging/release.py build --tag v0.1.1` built and
verified `nullius_eval-0.1.1-py3-none-any.whl`,
`nullius_eval-0.1.1.tar.gz` and `SHA256SUMS`; `release.py verify --tag
v0.1.1` and `verify.py` on the wheel passed, including the
installed-tool and profile-preservation checks.
- The workflows have not run; they run only by hand on `main`.

## Risk

Version strings and docs only. A wrong default tag in a workflow is an
input the person running it can change, and `release.py` refuses a tag
that does not match `pyproject.toml` or has no notes. Nothing is tagged,
drafted or published by this change. Undo with a revert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offer GSM8K-Platinum as a second dataset of GSM8K

1 participant