Repository navigation
heavy full-suite: 77 real failures once the runner no longer OOMs #19
Description
Summary
heavy / full-suite now runs to completion on blacksmith-16vcpu-ubuntu-2404 (PR #18, dispatch run 39420692371, 55 min 27 s) instead of being evicted at ~3 min. It fails on real tests: 77 failed, 2705 passed, 17 skipped, 17 errors in 3293 s.
Evidence
- Before the eviction era, run 38997624076 (2026-10-17, hosted, ran to the end) failed with 75 failed / 17 errors on the same groups. These are pre-existing.
- New since then:
tests/test_publish_exclusions.py::test_current_repo_tree_scan_cleanand::test_bare_selected_copy_has_publisher_runtime_controls. They also fail locally onmain4a7166d (2 failed, 53 passed):The scanner flags test fixtures as secrets, so it is a test/scanner issue, not a leaked secret (to be confirmed by reading those lines).SECRET tests/beacon/test_push.py:282 entropy H=4.42 SECRET tests/datasets/test_pin.py:19 credential value under TOKEN
Failure groups (count, test)
13 ERROR tests/challenge/test_authority_snapshot_v2.py::test_modified_package_rejects_before_restore
1 ERROR tests/challenge/test_authority_snapshot_v2.py::test_original_export_validates_and_bootstraps_actual_service
3 ERROR tests/challenge/test_carry_genesis_v2.py::test_trial_stages_pinned_complete_carry_genesis
3 FAILED tests/auditor/test_island_bisect_v2.py::test_honest_traces_when_real_island_hooks
3 FAILED tests/auditor/test_island_replay_v2.py::test_full_carry_and_window_equality
1 FAILED tests/auditor/test_island_replay_v2.py::test_verified_anchor_restart_corruption_and_od_warm_start
1 FAILED tests/challenge/test_capacity_outer_v2.py::test_actual_bounded_genesis_preparation_and_duplicate_debit
1 FAILED tests/challenge/test_capacity_outer_v2.py::test_actual_bounded_outer_child_matches_original_signed_replay
1 FAILED tests/challenge/test_service_capacity_v2.py::test_real_capacity_launch_failure_keeps_debit_and_prevents_restart_rerun
6 FAILED tests/layout/test_od_island_v2.py::test_od_bf16_same_layout_cpu
18 FAILED tests/layout/test_od_island_v2.py::test_od_objectives_layout_reference_and_hooks
4 FAILED tests/miner/test_capacity_runtime_v2.py::test_actual_small_kernel_attempt_and_cleanup
1 FAILED tests/miner/test_capacity_runtime_v2.py::test_real_runner_parent_death_retains_absolute_manager_timer
1 FAILED tests/miner/test_island_launch_v2.py::test_capacity_branch_uses_real_bounded_failure_before_publication
3 FAILED tests/miner/test_island_launch_v2.py::test_real_od_two_rank_worker_paths
1 FAILED tests/models/test_opendecision.py::test_check_manifest_param_count
1 FAILED tests/models/test_opendecision.py::test_eval_holdout_and_eval_cli
1 FAILED tests/models/test_opendecision.py::test_extend_params_and_warmstart_cli
1 FAILED tests/models/test_opendecision.py::test_stage_a_param_set_excludes_head
1 FAILED tests/protocol/test_network_v2.py::test_assign_unit_when_only_island_batch_is_divisible
24 FAILED tests/protocol/test_network_v2.py::test_od_general_n_when_v2
1 FAILED tests/relay/test_transport_v2.py::test_launch_safety_independent_systemd_units
1 FAILED tests/relay/test_transport_v2.py::test_provider_fix_pinned_kind_version_and_fail_closed
1 FAILED tests/test_launch_cuda_run.py::test_preflight_names_the_first_missing_input_and_spends_nothing
1 FAILED tests/test_publish_exclusions.py::test_bare_selected_copy_has_publisher_runtime_controls
1 FAILED tests/test_publish_exclusions.py::test_current_repo_tree_scan_clean
Likely causes per group, not verified one by one:
odfixtures (test_od_island_v2,test_network_v2::test_od_general_n_when_v2,test_opendecision,test_real_od_two_rank_worker_paths):od-tinydims were removed in feat(od): activate od-base, remove od-tiny (light CI) #7 and the fixtures still use them.- Native systemd/kernel cases (
test_actual_*,test_real_capacity_*,test_launch_safety_independent_systemd_units): they are deselected in per-PR CI, run inheavy/nativein an isolated root env, and fail when run as the plain runner user. test_provider_fix_pinned_kind_version_and_fail_closed: expects kindv0.31.0, findsv0.33.0on the runner image.
No continue-on-error was added. heavy stays red until these are fixed or the suite is scoped.
Activity
Merge pull request #20 from CortexLM/ci/heavy-green
Merge pull request #23 from CortexLM/ci/test-tmp-leak
echobt commentedon Oct 26, 2026
Closing: the heavy workflow (native, full-suite, canary, od-suite island/layout-n4/layout-rest) is green on main 679eb5b (run 2026-10-26 03:11Z, 1h09) and on the nightly runs of 24/10 and 25/10; the 23/10 failure was a separate nightly. The 77 real failures this issue tracked were fixed by the heavy-green work. Reopen with a run link if a regression returns.