Skip to content

test(judge): check the batch overlap with a barrier, not wall time - #135

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-batch-overlap-test
Nov 16, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:fix/judge-batch-overlap-test

Conversation

@harenome

Copy link
Copy Markdown
Collaborator

Closes #109

What

test_calls_overlap_and_the_count_and_order_hold in app/ui_tests/test_judge_batch.py asserted that 16 calls of 0.1 s end in less than 0.8 s. On a loaded machine the batch takes longer, and the test fails although the calls overlap. Now each call waits on a threading.Barrier of judge.CONCURRENCY parties. The barrier releases the calls only when that many are in flight at the same time. The batch has 2 * judge.CONCURRENCY items, so the barrier fills two times for every value of CONCURRENCY. If the calls run one at a time, the barrier breaks at its 10 s timeout and the test fails. The timeout only stops a hang, and the test measures no duration. The count and order check does not change. No production code changes.

Touches

Routes (tests only).

Verified

Base is c7af06d5. The load runs are in a rootless podman container, --cpus 1, with busy-loop shell processes in parallel. Each repetition runs python3 -m unittest ui_tests.test_judge_batch in a new interpreter.

  • With 64 busy loops, 20 repetitions: the test before this change passed 3 times and failed 17 times on the wall-time assertion. This branch passed 20 times.
  • With 32 busy loops, 10 repetitions: the base passed 8 times and failed 2 times. This branch passed 10 times.
  • CONCURRENCY set to 6 and to 3 in a scratch copy: RunBatch passes for both. With a fixed count of 16 items and CONCURRENCY 6, the test fails with threading.BrokenBarrierError.
  • Mutation: run_batch changed to use the built-in map in place of pool.map (one call at a time). The test fails with threading.BrokenBarrierError after 10 s.
  • python3 app/tests/parallel.py app/ui_tests: 322 tests in 40 modules, OK, on this branch and on the base.

Risk

Low. The change is in one test. Revert the commit to undo.

The test asserted that 16 calls of 0.1 s end in less than 0.8 s. On a
loaded machine the wall time grows, and the test failed although the
calls ran at the same time.

Each call now waits on a barrier of CONCURRENCY parties. The barrier
lets the calls go only when that many are in flight at once. Serial
calls break it at its timeout and the test fails. The batch has
2 * CONCURRENCY items, so the barrier fills twice for any value of
CONCURRENCY. The count and order checks are the same.
@harenome
harenome merged commit 4e55563 into ThinkFlowLab:main Nov 16, 2026
2 checks passed
@harenome
harenome deleted the fix/judge-batch-overlap-test branch November 16, 2026 10:13
aitianshi added a commit that referenced this pull request Nov 16, 2026
No linked issue: the owner asked for v0.1.1 as a release, so that the
Hugging Face endpoint setting (#129) reaches PyPI.

## What

Prepares v0.1.1. The version becomes `0.1.1`, and
`docs/releases/v0.1.1.md` gives the highlights since v0.1.0, a
Contributors section, the full changelog of every pull request in
`v0.1.0..main` (#114, #117, #120, #122, #125, #127, #129, #132, #137,
and #135 among the test and CI changes) and the installation notes, in
the shape of v0.1.0's. Install examples, the release guide, the docs'
release-notes link and menu, and the workflows' default tags name 0.1.1.
The release tooling already accepts final `vX.Y.Z` tags and does not
change.

## Touches

CI and skills (the release and publish workflows' default tags) · docs
and site

## Verified

- Backend suite (`python3 app/tests/parallel.py`): 1991 tests in 116
modules, OK. Route suite: 428 tests in 41 modules, OK.
- `catalogue_docs.py --write` and `diagrams.py --write` left no change.
- `python3 app/packaging/release.py build --tag v0.1.1` built and
verified `nullius_eval-0.1.1-py3-none-any.whl`,
`nullius_eval-0.1.1.tar.gz` and `SHA256SUMS`; `release.py verify --tag
v0.1.1` and `verify.py` on the wheel passed, including the
installed-tool and profile-preservation checks.
- The workflows have not run; they run only by hand on `main`.

## Risk

Version strings and docs only. A wrong default tag in a workflow is an
input the person running it can change, and `release.py` refuses a tag
that does not match `pyproject.toml` or has no notes. Nothing is tagged,
drafted or published by this change. Undo with a revert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests: the judge batch overlap test fails on wall time under load

1 participant