Skip to content

CI use benchopt info --version to gate tests on min_benchopt_version - #63

Merged
tomMoral merged 1 commit into
benchopt:mainfrom
tomMoral:use-benchopt-info-version
Sep 21, 2026
Merged

tomMoral merged 1 commit into
benchopt:mainfrom
tomMoral:use-benchopt-info-version

Conversation

@tomMoral

Copy link
Copy Markdown
Member

Problem

The compatibility gate compared versions with a bash lexicographic test:

if [[ "$benchopt_version" < "$min_version" ]]

< in [[ ]] is a string comparison, so once benchopt reaches 1.10.0 it is (wrongly) treated as older than 1.2–1.9. Because the Test step is guarded by if: compatible == 'true', every benchmark pinning one of those min_benchopt_version values would silently skip its tests (green, not red) under benchopt 1.10.0.

Change

Delegate the comparison to benchopt itself via the new benchopt info <benchmark> --version (benchopt/benchopt#1009), which exits non-zero when the installed benchopt is too old:

if [[ "$BENCHOPT_VERSION" == "git" ]] \
   || ! benchopt info --help | grep -q -- '--version' \
   || benchopt info ${{ inputs.benchmark_dir }} --version
then
  echo "compatible=true" >> $GITHUB_OUTPUT
else
  echo "compatible=false" >> $GITHUB_OUTPUT
fi
  • git (dev) builds: always compatible, as before.
  • The ! benchopt info --help | grep -q -- '--version' clause makes benchopt < 1.10.0 (which lacks the flag) fall through to compatible=true, so the test runs rather than being silently skipped.
  • Otherwise the exit code of benchopt info --version decides.

Merge after benchopt 1.10.0 is on PyPI — the latest job needs the new flag. The default matrix (latest + git) is covered; the fall-through handles any benchmark pinning an older benchopt.

Depends on benchopt/benchopt#1009.

@tomMoral
tomMoral merged commit 8f17128 into benchopt:main Sep 21, 2026
4 checks passed
@tomMoral
tomMoral deleted the use-benchopt-info-version branch September 21, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant