Release v0.16.2

.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.16.1",
+      "version": "0.16.2",
       "keywords": [
         "agentops",
         "evaluation",

.github/plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.16.1",
+      "version": "0.16.2",
       "keywords": [
         "agentops",
         "evaluation",

CHANGELOG.md

@@ -5,6 +5,24 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
 
 ## [Unreleased]
 
+## [0.16.2] - 2026-11-06
+
+### Changed
+- **Refresh the E2E sample memory dependency (#546).** Update `mem0ai` from
+  2.0.5 to 2.0.18 in the containerized sample's requirements.
+
+### Security
+- **Trusted PR evaluation (#499).** Generated GitHub PR gates execute a trusted
+  base checkout and import only immutable candidate dataset/prompt data. Azure
+  DevOps gates use a closed protected-ref `extends` template with PR-only
+  resources, fixed Git/API destinations and constrained logging commands.
+  Configuration/authentication changes are rejected in credentialed candidate
+  evaluation. Foundry detection now checks hostname boundaries, and credentialed
+  HTTP redirects cannot cross origins or downgrade HTTPS. Private/sovereign
+  hosts and explicit local configuration remain supported. Regenerate copied
+  workflows and configure the documented external resource/identity protections
+  and exclusive status provenance before relying on a secure merge gate.
+
 ## [0.16.1] - 2026-11-06
 
 ### Changed

docs/ci-github-actions.md

@@ -127,6 +127,131 @@ GitHub or Azure, and call out anything that needs owner/admin permission.
 
 ## Configuration walkthrough
 
+### PR gate trust boundary and migration
+
+Regenerate existing copied PR workflows after installing a release containing
+the #499 fix. Updating the package alone does not update copied YAML files.
+The authenticated evaluator must execute only administrator-reviewed workflow,
+toolkit, configuration, recipes and scripts. PR content is read as immutable
+Git blobs, never checked out, installed or imported in the credentialed job.
+Only JSONL rows at the configured dataset path and plain instructions at the
+configured prompt path are copied into that trusted checkout.
+
+Changes to `agentops.yaml`, including destinations, authentication, thresholds,
+recipes and prompt paths, fail the PR gate before evaluation. Review and land
+such configuration through a separate controlled change, then rerun candidate
+evaluation. Unrelated code changes do not run with evaluation credentials.
+Code tests remain a separate, unprivileged workflow. Remote Foundry datasets
+continue to come from trusted configuration. Local CLI configuration precedence,
+the flat schema and exit codes are unchanged.
+
+HTTPS is required for credential-bearing PR destinations. Automatic Foundry
+classification uses the hostname, not Azure-looking text in a path or query.
+Private/custom hosts can use explicit `responses` or `invocations` protocols,
+and supported sovereign Azure hostnames remain recognized. Credentialed HTTP
+redirects may remain within the same origin, never across hosts, ports or an
+HTTPS downgrade. Git input fetching does not follow redirects when using its
+read token.
+
+#### GitHub Actions
+
+PR templates use `pull_request_target`, check out the exact trusted base SHA
+and fetch candidate blobs without persisted checkout credentials. Do not add
+candidate checkout, package installation, imports, actions, recipes or shell
+execution to this workflow. The changed trigger deliberately permits evaluating
+fork PR data with the same sandbox configuration. An optional environment
+approval can remain in place, but a maintainer must not approve an environment
+for arbitrary PR-authored code.
+
+Restrict the sandbox Azure identity and any status-writer key to the protected
+workflow/ref. An environment name or default OIDC environment subject alone
+does not identify a workflow. Protect allowed deployment branches and use a
+workflow-bound custom OIDC subject where supported, or require an administrator
+to review the exact workflow/revision before granting environment access.
+Do not make the same credentials available to ordinary PR jobs or arbitrary
+branch `workflow_dispatch` runs.
+
+The status is posted on the actual PR head SHA. **That SHA and the status name
+alone are not trustworthy merge evidence.** Other writers and PR-authored
+workflows can publish a status using the shared GitHub Actions identity.
+For a required `agentops/pr-eval` gate, install a dedicated status-only GitHub
+App, put its client ID in `AGENTOPS_STATUS_APP_CLIENT_ID`, and keep
+`AGENTOPS_STATUS_APP_PRIVATE_KEY` in the protected evaluator environment.
+Select that App as the required status's **expected source** in the branch
+rule/ruleset. Do not select "any source" or the shared GitHub Actions App.
+Restrict access to this key as carefully as Azure credentials. Without the App,
+evaluation still runs and posts an informational status, but that shared status
+must not be used as proof that the trusted evaluator approved a merge.
+See [GitHub's expected-source rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/available-rules-for-rulesets#require-status-checks-to-pass-before-merging).
+
+#### Azure DevOps
+
+Generation writes two files: the PR entrypoint at
+`.azuredevops/pipelines/agentops-pr.yml` and a closed `extends` template at
+`.azuredevops/templates/agentops-pr-trusted.yml`. A separate repository is
+**not required**. Store the template on an administrator-protected ref in the
+same repository, or use an existing protected template repository.
+For Azure Repos Git, configure an automatic build-validation branch policy
+pointing at the entrypoint to trigger PR runs. Azure Repos does not use the
+YAML `pr` trigger. Keep the exclusive required status policy below as the
+authoritative proof, even when build validation also requires a passing build.
+Replace `YOUR_PROJECT/YOUR_REPOSITORY` in both the wrapper resource and every
+literal trusted checkout, and select the same protected ref throughout.
+The template checks out that literal protected revision, not a repository
+alias or target branch supplied by PR YAML. Its config and scripts govern
+evaluation even if the PR targets another branch.
+
+Create PR-only resources rather than reusing deployment identities: the
+`agentops-pr` variable group and the literal `agentops-pr-azure` service
+connection. The group contains the usual Foundry sandbox values plus
+`AGENTOPS_ADO_REPOSITORY_URL`, `AGENTOPS_ADO_COLLECTION_URI`,
+`AGENTOPS_ADO_PROJECT_ID` and `AGENTOPS_ADO_REPOSITORY_ID`. These are the approved
+HTTPS source Git URL, collection URL, project ID and repository ID. Git/API
+tokens are sent only to these protected destinations, not caller-overridden
+`Build.*` or `System.*` destinations. An optional
+`AGENTOPS_AGENT_PROMPT_FILE` must likewise be administrator-controlled.
+
+On the service connection, add a Required template check binding the exact
+repository, protected ref and `.azuredevops/templates/agentops-pr-trusted.yml`
+path. Restrict updates to the variable group to administrators. Nonsecret
+group values are not protected by access checks. If the group contains secrets,
+its **own** checks must independently enforce the trusted template before
+exposing them. A connection check alone cannot protect a secret group when a
+malicious root YAML removes the connection. Verify that the group's available
+checks reject that negative case, or keep its secrets unavailable to PR builds.
+See [variable group security](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops).
+Restrict pipeline
+permissions, protect updates to the ref/resources/checks, limit queue-time
+variables, and deny ordinary PR pipelines access to all other privileged
+connections, variable groups, secure files and agent pools. Do not add custom
+step/job/auth/config parameters to the closed template. It rejects free
+execution variables, isolates package installation, and restricts logging
+commands and variables that individual tasks may set. Use disposable hosted
+agents rather than a shared credential-bearing worktree.
+
+Register the **exclusive service connection principal** in the ADO organization
+and grant it the permission to publish PR statuses. Its ADO token publishes
+`agentops/pr-eval` for the verified source SHA/current PR iteration. Configure
+a required status policy with that principal as **Authorized identity**, apply
+the policy immediately and reset it on source changes. Give the Build Service
+only source-read and optional PR-comment permissions, not status authority for
+this policy. A PR can remove protected resources and fake a successful build
+without triggering resource checks, so a build-validation result or a shared
+Build Service status is not sufficient proof.
+
+These are administrator-managed prerequisites, not settings provisioned by
+the generator. Until enforced, do not grant credentials to PR execution or
+treat its result as a secure merge gate. The same-repository design depends
+on [Required template checks](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops#required-template),
+[closed security templates](https://learn.microsoft.com/en-us/azure/devops/pipelines/security/templates?view=azure-devops)
+and [identity-bound status policies](https://learn.microsoft.com/en-us/azure/devops/repos/git/pr-status-policy?view=azure-devops).
+
+Candidate prompts and dataset rows still influence the agent and its tools.
+This separation prevents CI code/configuration from selecting credential
+destinations, not general prompt injection. Keep the evaluation Foundry project
+and tools sandboxed with least privilege, no production data and no production
+deployment rights.
+
 ### 1. Repository variables (OIDC)
 
 In Settings → Secrets and variables → Actions → **Variables**, add:

infra/e2e/agent-app/requirements.txt

@@ -2,4 +2,4 @@ fastapi==0.136.3
 uvicorn[standard]==0.32.1
 agent-framework==1.8.1
 azure-identity==1.19.0
-mem0ai==2.0.5
+mem0ai==2.0.18

plugins/agentops/package.json

@@ -2,7 +2,7 @@
   "name": "agentops-accelerator",
   "displayName": "AgentOps Accelerator — Skills for GitHub Copilot",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.16.1",
+  "version": "0.16.2",
   "publisher": "AgentOpsAccelerator",
   "icon": "icon.png",
   "license": "MIT",

plugins/agentops/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentops-accelerator",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.16.1",
+  "version": "0.16.2",
   "author": {
     "name": "AgentOps Accelerator",
     "url": "https://github.com/Azure/agentops"

src/agentops/core/agentops_config.py

@@ -31,6 +31,7 @@
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any, Dict, List, Literal, Mapping, Optional
+from urllib.parse import urlsplit
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
@@ -1166,20 +1167,21 @@ class TargetResolution:
 def _looks_like_foundry_url(url: str) -> bool:
     """Return ``True`` when ``url`` matches a Foundry hosted endpoint pattern.
 
-    Heuristic - Foundry URLs include the segment ``/agents/`` and the host
-    ends in a Foundry-recognized domain. We err on the side of accepting more
-    URLs as Foundry hosted (the user can force ``http-json`` via ``protocol``).
+    This classifies a hostname, not a credential destination allowlist.
+    Explicit protocols support private/custom hosts without trusting text
+    in a URL's path, query, userinfo, or an attacker-controlled suffix.
     """
-    lowered = url.lower()
+    hostname = (urlsplit(url).hostname or "").lower().rstrip(".")
     foundry_domains = (
         ".azure.com",
+        ".azure.us",
+        ".azure.cn",
         ".azureml.ms",
-        ".cognitiveservices.azure.com",
-        ".services.ai.azure.com",
-        ".inference.ml.azure.com",
         ".azurewebsites.net",  # rare; users can override
+        ".azurewebsites.us",
+        ".chinacloudsites.cn",
     )
-    return any(domain in lowered for domain in foundry_domains)
+    return any(hostname.endswith(domain) for domain in foundry_domains)
 
 
 #: Matches the ``/agents/<name>/versions/<version>`` segment pair that Foundry
@@ -1358,7 +1360,9 @@ def classify_agent(
 
     lowered = raw.lower()
     if lowered.startswith(("http://", "https://")):
-        if _looks_like_foundry_url(raw):
+        if protocol in {"responses", "invocations"} or (
+            protocol != "http-json" and _looks_like_foundry_url(raw)
+        ):
             resolved_protocol: Protocol = protocol or "responses"
             if resolved_protocol not in {"responses", "invocations"}:
                 raise ValueError(

src/agentops/pipeline/commit_info.py

@@ -16,7 +16,7 @@
 
 from agentops.core.results import CommitInfo
 
-_CI_SHA_ENV_VARS = ("GITHUB_SHA", "BUILD_SOURCEVERSION")
+_CI_SHA_ENV_VARS = ("AGENTOPS_PR_SHA", "GITHUB_SHA", "BUILD_SOURCEVERSION")
 _FIELD_SEP = "\x1f"
 _GIT_SHOW_FORMAT = f"%H{_FIELD_SEP}%h{_FIELD_SEP}%s{_FIELD_SEP}%an{_FIELD_SEP}%aI"
 _GIT_TIMEOUT_SECONDS = 5
@@ -52,7 +52,7 @@ def _pull_request_head_sha() -> Optional[str]:
 
 
 def _ci_git_sha(*, workspace: Optional[Path] = None) -> Optional[str]:
-    if os.environ.get("GITHUB_EVENT_NAME") == "pull_request":
+    if os.environ.get("GITHUB_EVENT_NAME") in {"pull_request", "pull_request_target"}:
         head_sha = _pull_request_head_sha()
         # The generated PR workflow's default `actions/checkout` has no
         # `ref`/`fetch-depth` override, so it checks out only the

src/agentops/pipeline/invocations.py

@@ -24,7 +24,8 @@
 import urllib.parse
 import urllib.request
 from dataclasses import dataclass, field
-from typing import Any, Dict, List, Optional
+from http.client import HTTPMessage, HTTPResponse
+from typing import Any, Dict, IO, List, Optional
 
 from agentops.core.agentops_config import AgentOpsConfig, TargetResolution
 from agentops.pipeline.diagnostics import with_tenant_mismatch_guidance
@@ -201,6 +202,39 @@ def _row_input(row: Dict[str, Any]) -> str:
     return str(value)
 
 
+def _http_origin(url: str) -> tuple[str, str, int]:
+    parts = urllib.parse.urlsplit(url)
+    port = parts.port if parts.port is not None else (443 if parts.scheme == "https" else 80)
+    return parts.scheme.lower(), (parts.hostname or "").lower(), port
+
+
+class _SameOriginRedirectHandler(urllib.request.HTTPRedirectHandler):
+    def redirect_request(
+        self, req: urllib.request.Request, fp: IO[bytes], code: int,
+        msg: str, headers: HTTPMessage, newurl: str,
+    ) -> urllib.request.Request | None:
+        if _http_origin(req.full_url) != _http_origin(newurl):
+            raise RuntimeError(
+                "Refusing an authenticated HTTP redirect to another origin. "
+                "Configure the final trusted endpoint directly."
+            )
+        return super().redirect_request(req, fp, code, msg, headers, newurl)
+
+
+def _open_http(request: urllib.request.Request, *, timeout: float) -> HTTPResponse:
+    # Custom headers can carry credentials under arbitrary names.
+    public_headers = {"content-type": "application/json"}
+    authenticated = any(
+        public_headers.get(name.lower()) != value
+        for name, value in request.header_items()
+    )
+    if authenticated:
+        return urllib.request.build_opener(_SameOriginRedirectHandler()).open(
+            request, timeout=timeout
+        )
+    return urllib.request.urlopen(request, timeout=timeout)  # noqa: S310
+
+
 def _http_request_json(
     *,
     method: str,
@@ -216,7 +250,7 @@ def _http_request_json(
     last_exc: Optional[BaseException] = None
     for attempt in range(1, 4):
         try:
-            with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310
+            with _open_http(request, timeout=timeout) as response:
                 payload = response.read().decode("utf-8")
             break
         except urllib.error.HTTPError as exc:
@@ -263,7 +297,7 @@ def _http_request_stream(
     last_exc: Optional[BaseException] = None
     for attempt in range(1, 4):
         try:
-            with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310
+            with _open_http(request, timeout=timeout) as response:
                 # HTTPResponse is iterable line-by-line; joining restores the
                 # full body (including newlines) for both sse and text modes.
                 chunks = [line.decode("utf-8", errors="replace") for line in response]

src/agentops/pipeline/prompt_deploy.py

@@ -623,7 +623,9 @@ def _detect_pr_stage() -> Optional[str]:
     DevOps (``BUILD_REASON == 'PullRequest'``).
     """
 
-    if os.environ.get("GITHUB_EVENT_NAME") == "pull_request":
+    if os.environ.get("AGENTOPS_PR_NUMBER", "").isdigit():
+        return os.environ["AGENTOPS_PR_NUMBER"]
+    if os.environ.get("GITHUB_EVENT_NAME") in {"pull_request", "pull_request_target"}:
         ref = os.environ.get("GITHUB_REF", "")
         # ``refs/pull/<N>/merge`` or ``refs/pull/<N>/head``.
         if ref.startswith("refs/pull/"):
@@ -648,7 +650,8 @@ def _detect_pr_stage() -> Optional[str]:
 
 def _git_sha() -> str:
     return (
-        os.environ.get("GITHUB_SHA")
+        os.environ.get("AGENTOPS_PR_SHA")
+        or os.environ.get("GITHUB_SHA")
         or os.environ.get("BUILD_SOURCEVERSION")
         or os.environ.get("Build.SourceVersion")
         or ""
@@ -687,7 +690,10 @@ def _write_candidate_eval_config(
         dataset_path = dataset_reference.local_path or Path(str(config.dataset))
         if not dataset_path.is_absolute():
             dataset_path = (source_config_path.parent / dataset_path).resolve()
-        data["dataset"] = str(dataset_path)
+        try:
+            data["dataset"] = os.path.relpath(dataset_path, destination.parent.resolve())
+        except ValueError:  # Paths on different Windows drives cannot be relative.
+            data["dataset"] = str(dataset_path)
     destination.parent.mkdir(parents=True, exist_ok=True)
     save_yaml(destination, data)