benchmarking/swebench: fix the review findings from #94-#96 (dataset builder rejects all rows, delete-park leak, cluster script), and a flaky test

[Tomer Glottmann (tomergee)]

Fixes the findings from my reviews of #94, #95 and #96 that are on main after those merged, plus a flaky test.

The defects

#	Where	Defect	Fix
1	benchmarking/swebench/make-dataset.py	The eval-script check rejects each row of SWE-bench/SWE-bench_Verified with each swebench 5.x release. The script exits 2 at the first row and writes no manifest. swebench 5's make_test_spec adds two lines around the End marker (SWEBENCH_TEST_EXIT_CODE=$? and the Test Exit Code echo), so the row's script is never a substring of the harness script.	Compare line for line, without the shell header and those two lines.
2	same	Without --revision, the meta file records "revision": null.	Resolve the dataset to a commit first (or the sha256 of a local file), load that commit, and record it as resolved_revision.
3	benchmarking/swebench/run-in-cluster.sh	While waiting for the exit code, any kubectl error reads as a lost pod (|| echo gone). The script exits, and its trap deletes the running Job and its results.	Only NotFound means gone. Other errors are retried, up to MAX_API_ERRORS (30) times in a row.
4	same	A loader that exits before writing its events file makes kubectl cp fail without the loader's exit code. With no pod there is no diagnostic. A killed script leaves the Job for good.	Print the exit code first and exit with it. Print the Job events. Add activeDeadlineSeconds (48 h) and ttlSecondsAfterFinished (1 h).
5	internal/benchmarking/swebench/run.go	A failed delete park leaks the environment. The next step overwrites its only handle.	The environment leaves the run and keeps its handle for the final cleanup.
6	same	The 20-character run-id cap does not keep names within 63 characters. A run id, a 36-character instance part, -10 and -s10 make 64.	validate builds the longest name the run uses and refuses a config whose names are longer than 63.
7	internal/benchmarking/swebench/dataset.go	An https manifest URL can redirect to http and is followed.	CheckRedirect refuses each non-https hop.
8	trajectories/swebench-grade.py	The grader takes the first End marker, even one before Start, and voids a valid run.	Look for End after Start, as the harness does. The remaining, stricter differences are documented.
9	report.go, events.go, run.go, grader docstring	live: 0 is dropped from the JSON. Event.Env does not say that names change from step 2. isNotFound sits between isTransient and its doc. The grader docstring predates the success/failure/neither split.	Fixed.
10	guest/process/service_test.go	Three tests use the shared $TMPDIR/ate-jobs, and the helper's cleanup removes it. internal/apiservice's wake tests then fail intermittently with "creating stdout log: … no such file or directory".	Each test uses t.TempDir().

Why the earlier full runs worked

The full-shape runs on the cluster (posted on #94–#96 on 2026-10-16 at 02:39Z) predate items 1 and 2.
The eval-script check came in d00f349 at 19:48Z, in reply to a review, so those runs never executed it.
They parked with suspend, not delete, so item 5 never came up. SWE-bench ids have 32 characters or
less, so item 6 never came up. Item 3 needs a transient API error during the wait.

The check, per release, over all 500 rows of SWE-bench/SWE-bench_Verified (commit 78f471bf):

swebench	Check on main	Check in this PR
5.0.2 (the README's swebench>=5)	0 of 500; stops at the first row	500 of 500
4.1.0	480 of 500; stops at astropy__astropy-7336	480 of 500, the same 20 rows rejected

On 4.x, make_test_spec builds the script from its own tables, and 20 rows differ from the dataset.
Both checks rightly refuse them. So the change fixes 5.x and keeps the 4.x behavior row for row.

Verification

• Item 1, against the real dataset with swebench 5.0.2:
  • Old check: 0 of 500 rows pass. New check: 500 of 500.
  • A tampered test command is still rejected.
  • End to end with a stub crane, make-dataset.py --limit 5 writes 5 instances and records resolved_revision (78f471bf…).
  • Main's version exits 2 at astropy__astropy-12907.
  • test_make_dataset.py covers the check and the file hash offline, and CI now runs it.
• Items 3 and 4, with stub kubectl, ko and go:
  • Two transient "i/o timeout" errors are retried, and the run completes with exit 0.
  • NotFound gives "is gone" and exit 1.
  • An early loader exit gives "loader exit code 3" and exit 3.
  • No pod prints the Job events.
  • Three errors in a row with MAX_API_ERRORS=3 exit 1.
  • The applied manifest has both Job bounds. shellcheck is clean.
• Items 5, 6 and 7 have new Go tests. Each fails with its fix removed.
• Item 8 has a new grader test. The test that rejects fully reversed markers still passes.
• Whole repo: go build, go vet, go test -race ./... and staticcheck on the benchmarking packages pass. make verify-gofmt verify-boilerplate and zizmor on ci.yaml are clean.
• Item 10: the full Go suite passes, and guest, internal/apiservice and clients/go passed three runs in a row. Before the fix, the full suite failed in TestGuestCallWaitsForWake with the error above.