Skip to content

loopguard: pytest asked only to show fixtures, markers or its cache runs no tests - #302

Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix-loopguard-pytest-listing
Open

Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix-loopguard-pytest-listing

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

pytest's own help says --fixtures (--funcargs), --fixtures-per-test and --markers show what is available, --setup-plan shows what would run "but don't execute anything", --setup-only sets up fixtures and does "not execute tests", and --cache-show does not "perform collection or tests". Only --collect-only was read as no run (_NOT_A_RUN_RE). The others counted as verifications, so an agent that ran pytest --fixtures (exit 0) after a failing test run made the turn count as ended green, and the envelope learned from a turn that ended red.

Fix: they are read as no run, as --collect-only is. --setup-show, which shows the setup while it runs the tests, still counts, as do runs given a path such as tests/test_markers.py or -m slow, and an option that only starts with one of these names (--fixtures-dir). The change is the _NOT_A_RUN_RE line, apart from the lines #294 and #295 change; the branch merges with both.

Tests, in a new tests/test_loopguard_pytest_listing.py:

  • 9 commands, each not a verification (fail on main);
  • the same 9 after a failing test run leave the turn not ended green (fail on main);
  • 4 commands that execute tests still count.
python -m pytest tests -n 4 --dist worksteal -rfEs --tb=line
1423 passed, 41 skipped

The skips need Windows, a real agent CLI (Codex, Hermes, opencode) or aider, or a non-root user.

Fixes #298

…uns no tests

pytest's own help says `--fixtures` (`--funcargs`), `--fixtures-per-test` and `--markers` show
what is available, `--setup-plan` shows what would run "but don't execute anything",
`--setup-only` sets up fixtures and does "not execute tests", and `--cache-show` does not
"perform collection or tests". Only `--collect-only` was read as no run. The others counted as
verifications, so an agent that ran `pytest --fixtures` (exit 0) after a failing test run made
the turn count as ended green, and the envelope learned from a turn that ended red.

They are now read as no run, as `--collect-only` is. `--setup-show`, which shows the setup while
it runs the tests, and runs with a path or `-m` still count.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loopguard: pytest --fixtures after a failing run makes the turn count as ended green

1 participant