Skip to content

feat(evaluators): uptake scorer_invoke grant from orbit - #274

Open
namrataghadi-galileo wants to merge 2 commits into
mainfrom
feature/SAO-17620-acquire-scorer-grant
Open

namrataghadi-galileo wants to merge 2 commits into
mainfrom
feature/SAO-17620-acquire-scorer-grant

Conversation

@namrataghadi-galileo

@namrataghadi-galileo namrataghadi-galileo commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add run-scoped scorer-grant acquisition to the Galileo Luna evaluator.
  • When both GALILEO_API_KEY and GALILEO_API_URL are configured, exchange the application API key with Orbit for a signed scorer grant and send that grant to the Luna runner.
  • Cache grants per Orbit URL, target type, run ID, and caller context; reuse them across scorers and refresh expired grants.
  • Preserve legacy secret-only Luna authentication. When both legacy and grant credentials are present, grant mode takes precedence; incomplete grant configuration and grant errors fail clearly without falling back.
  • Pass opaque target type and ID from the evaluation request into evaluators so Galileo can use the target ID as the Orbit run ID.

Scope and compatibility

  • The application API key is sent only to Orbit's scorer-grant endpoint; it is never sent to Runners.
  • Orbit signs the scorer grant. Agent Control does not mint it locally, and the scorer grant remains separate from the normal Agent Control runtime grant.
  • Generic evaluator behavior remains compatible through the default request-context hook, which delegates to the existing evaluator context method.
  • Existing Luna request serialization, scorer IDs and versions, record serialization, and record factory behavior are unchanged.
  • Updated the Galileo README and example configuration for the new credentials and target ID.

Risk and rollout

  • Orbit's scorer-grant endpoint must be deployed first (Orbit Jira 1 dependency).
  • Grant mode is enabled by configuring both GALILEO_API_KEY and GALILEO_API_URL. Secret-only configuration continues to use legacy Luna authentication.

Testing

  • Galileo evaluator package tests: 166 passed.
  • Engine core tests: 58 passed.
  • Evaluator base tests: 13 passed.
  • Ruff, mypy on changed production source files, and git diff --check passed.
  • Make test targets could not resolve splunk-ao==0.4.0 because the configured package index returned HTTP 403; the test suites above were run directly without syncing dependencies.

@namrataghadi-galileo namrataghadi-galileo changed the title feat(evaluators):uptake scorer_invoke grant from orbit feat(evaluators): uptake scorer_invoke grant from orbit Sep 29, 2026
@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant