Skip to content

fix(submission): enforce leaderboard eligibility - #183

Merged
KartikP merged 2 commits into
unified-model-interface-v2from
fix/leaderboard-eligibility
Oct 13, 2026
Merged

KartikP merged 2 commits into
unified-model-interface-v2from
fix/leaderboard-eligibility

Conversation

@KartikP

@KartikP KartikP commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Change

Prevent local-only benchmarks and scores from entering the submission database or S3. Check benchmark eligibility before loading the model, preserve legacy behavior when the flag is absent, and report metadata eligibility errors directly.

Validation

  • Coordinated offline checks: 1,657 passed, 43 allowed skips, no failures.
  • Publication guards: 17 tests passed, using fake entries without live database or S3 access.
  • Prior independent base/head audit found no new broad-suite failures; full-suite qualification remains limited by existing environment and data dependencies.
  • Jenkins build 138 completed the full test run: 920 passed, 3 skipped. Its final job status is failure after log parsing. The log contains pip's existing version conflict: published brainscore-vision 2.4.1 requires core >=2.4, while v2 is 2.4.0rc1. No tests failed. Unified's coordinated CI uses pinned peers instead.

@KartikP
KartikP merged commit e9a627f into unified-model-interface-v2 Oct 13, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant