From 2845ac91f438de96c339b90bc06790fdabe71246 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 09:04:54 +0200 Subject: [PATCH 01/29] Mark Phase 1 telemetry features as shipped (#470) Closes #455 --- website/content/telemetry.mdx | 22 +++++++++++++++------- website/public/llms-full.txt | 20 +++++++++++++------- website/public/telemetry.md | 20 +++++++++++++------- 3 files changed, 41 insertions(+), 21 deletions(-) diff --git a/website/content/telemetry.mdx b/website/content/telemetry.mdx index 3991ceb8..7e6253ae 100644 --- a/website/content/telemetry.mdx +++ b/website/content/telemetry.mdx @@ -7,9 +7,11 @@ import { Callout } from 'nextra/components' # Telemetry & cost - **Phase 1 — designed, not yet shipped.** Attribute and API shapes below are a - proposal; the committed part is the behaviour and the acceptance bar. See - [Roadmap](/roadmap) and [Verification policy](/concepts/verification). + **Shipped in Phase 1.** OpenTelemetry GenAI span export (#194), correlation + IDs (#195), cost-attribution tags (#196), and routing and resilience signals + (#309) are available. Platform-facing verification caveats are called out + below; see [Roadmap](/roadmap) and + [Verification policy](/concepts/verification). Two pieces of the six-piece minimum land here, because they answer the same @@ -136,6 +138,11 @@ path for message text to leak. ## Routing & resilience + + **Shipped in Phase 1** (#309). Routing, fallback, and served-model signals + are available through the shared transport and on OpenTelemetry spans. + + The gateway can fail over between providers when one degrades ("Enhanced Resilience for Intelligent Routing"). It reports what it *did* with each request on the response — which provider and model served it, how it routed, and whether @@ -194,9 +201,8 @@ donkey = Donkey.from_env(on_model_substitution="raise") ## Correlation IDs - **Shipped in Phase 1** (#195) — unlike the cost-attribution tags below, which - are still designed-not-shipped. See [Observability](/concepts/observability) - for the full detail. + **Shipped in Phase 1** (#195). See + [Observability](/concepts/observability) for the full detail. Set a per-**run** id once, and every call inside the block carries it — on the @@ -313,4 +319,6 @@ JSON blob into a header. --- -**Status: Phase 1 — not yet shipped.** +**Status: Phase 1 — OpenTelemetry GenAI spans and OTLP export, per-run +correlation IDs, cost-attribution tags, and routing and resilience signals are +shipped. Platform-facing verification caveats are called out above.** diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 1b3a428a..6d77e203 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -2597,9 +2597,11 @@ Source: https://donkey-development-kit.github.io/donkey-development-kit/telemetr # Telemetry & cost - **Phase 1 — designed, not yet shipped.** Attribute and API shapes below are a - proposal; the committed part is the behaviour and the acceptance bar. See - [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) and [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). + **Shipped in Phase 1.** OpenTelemetry GenAI span export (#194), correlation + IDs (#195), cost-attribution tags (#196), and routing and resilience signals + (#309) are available. Platform-facing verification caveats are called out + below; see [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) and + [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). Two pieces of the six-piece minimum land here, because they answer the same two questions: *what happened?* and *who pays for it?* @@ -2719,6 +2721,9 @@ path for message text to leak. ## Routing & resilience + **Shipped in Phase 1** (#309). Routing, fallback, and served-model signals + are available through the shared transport and on OpenTelemetry spans. + The gateway can fail over between providers when one degrades ("Enhanced Resilience for Intelligent Routing"). It reports what it *did* with each request on the response — which provider and model served it, how it routed, and whether @@ -2774,9 +2779,8 @@ donkey = Donkey.from_env(on_model_substitution="raise") ## Correlation IDs - **Shipped in Phase 1** (#195) — unlike the cost-attribution tags below, which - are still designed-not-shipped. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md) - for the full detail. + **Shipped in Phase 1** (#195). See + [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md) for the full detail. Set a per-**run** id once, and every call inside the block carries it — on the wire, on every span, and on every exception — with nothing threaded through your @@ -2886,7 +2890,9 @@ JSON blob into a header. --- -**Status: Phase 1 — not yet shipped.** +**Status: Phase 1 — OpenTelemetry GenAI spans and OTLP export, per-run +correlation IDs, cost-attribution tags, and routing and resilience signals are +shipped. Platform-facing verification caveats are called out above.** --- diff --git a/website/public/telemetry.md b/website/public/telemetry.md index 3d813903..5f1003d3 100644 --- a/website/public/telemetry.md +++ b/website/public/telemetry.md @@ -1,8 +1,10 @@ # Telemetry & cost - **Phase 1 — designed, not yet shipped.** Attribute and API shapes below are a - proposal; the committed part is the behaviour and the acceptance bar. See - [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) and [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). + **Shipped in Phase 1.** OpenTelemetry GenAI span export (#194), correlation + IDs (#195), cost-attribution tags (#196), and routing and resilience signals + (#309) are available. Platform-facing verification caveats are called out + below; see [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) and + [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). Two pieces of the six-piece minimum land here, because they answer the same two questions: *what happened?* and *who pays for it?* @@ -122,6 +124,9 @@ path for message text to leak. ## Routing & resilience + **Shipped in Phase 1** (#309). Routing, fallback, and served-model signals + are available through the shared transport and on OpenTelemetry spans. + The gateway can fail over between providers when one degrades ("Enhanced Resilience for Intelligent Routing"). It reports what it *did* with each request on the response — which provider and model served it, how it routed, and whether @@ -177,9 +182,8 @@ donkey = Donkey.from_env(on_model_substitution="raise") ## Correlation IDs - **Shipped in Phase 1** (#195) — unlike the cost-attribution tags below, which - are still designed-not-shipped. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md) - for the full detail. + **Shipped in Phase 1** (#195). See + [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md) for the full detail. Set a per-**run** id once, and every call inside the block carries it — on the wire, on every span, and on every exception — with nothing threaded through your @@ -289,4 +293,6 @@ JSON blob into a header. --- -**Status: Phase 1 — not yet shipped.** +**Status: Phase 1 — OpenTelemetry GenAI spans and OTLP export, per-run +correlation IDs, cost-attribution tags, and routing and resilience signals are +shipped. Platform-facing verification caveats are called out above.** From df5fc5a96ee8d1c490c7449e47823ab4e6eafa4f Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 09:20:42 +0200 Subject: [PATCH 02/29] Close Donkey-owned auth transport during teardown (#466) Closes #448 --- python/src/donkey_kit/donkey.py | 38 ++++++++++++++++------ python/tests/unit/test_donkey_surface.py | 40 ++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 9 deletions(-) diff --git a/python/src/donkey_kit/donkey.py b/python/src/donkey_kit/donkey.py index 3f663cb4..08569f7f 100644 --- a/python/src/donkey_kit/donkey.py +++ b/python/src/donkey_kit/donkey.py @@ -135,7 +135,11 @@ def __init__( # never an error) otherwise. This is the single funnel — from_env() # delegates here — and it is idempotent across many Donkey() instances. configure_otlp_export(self._cfg) - self._auth = auth if auth is not None else self._default_auth(self._cfg) + self._owned_auth_http: DonkeyAsyncClient | None = None + if auth is None: + self._auth, self._owned_auth_http = self._default_auth(self._cfg) + else: + self._auth = auth # One Budget per Donkey (never global, BG §1.3 / #185): both transports feed # it in-band from every response's x-token-* headers. self._budget = Budget() @@ -462,8 +466,16 @@ def _sync_http_client(self) -> DonkeyClient: return self._sync_http async def aclose(self) -> None: - await self._http.aclose() - self.close() + auth_http = self._owned_auth_http + self._owned_auth_http = None + try: + await self._http.aclose() + finally: + try: + if auth_http is not None: + await auth_http.aclose() + finally: + self.close() async def __aenter__(self) -> Donkey: return self @@ -472,8 +484,12 @@ async def __aexit__(self, *exc: object) -> None: await self.aclose() def close(self) -> None: - """Close the blocking transport. ``aclose()`` calls this too, so an async - caller who also used ``client(sync=True)`` still closes both.""" + """Close the blocking transport. + + This sync method cannot close either async transport. A sync-only caller + never opens them; mixed or async callers must use :meth:`aclose`, which + closes both async transports and calls this method for the blocking one. + """ if self._sync_http is not None: self._sync_http.close() self._sync_http = None @@ -508,14 +524,18 @@ def __getattr__(self, name: str) -> Adapter: return adapter @staticmethod - def _default_auth(cfg: DonkeyConfig) -> AuthProvider | None: + def _default_auth( + cfg: DonkeyConfig, + ) -> tuple[AuthProvider | None, DonkeyAsyncClient | None]: """Build control-plane auth when credentials are present. The LLM proxy credential is separate and handled by the OpenAI client (`BG §1.1`).""" if cfg.client_id and cfg.client_secret: - return AnypointConnectedApp( + http_client = build_http_client(cfg, None) # token fetches need no auth + auth = AnypointConnectedApp( client_id=cfg.client_id, client_secret=cfg.client_secret, control_plane_url=cfg.control_plane_url, - http_client=build_http_client(cfg, None), # token fetches need no auth + http_client=http_client, ) - return None + return auth, http_client + return None, None diff --git a/python/tests/unit/test_donkey_surface.py b/python/tests/unit/test_donkey_surface.py index 7f41262f..522208cf 100644 --- a/python/tests/unit/test_donkey_surface.py +++ b/python/tests/unit/test_donkey_surface.py @@ -6,6 +6,7 @@ import pytest from donkey_kit import Donkey, DonkeyConfig +from donkey_kit.core.auth import AnypointConnectedApp def _cfg() -> DonkeyConfig: @@ -275,3 +276,42 @@ async def test_aclose_also_closes_a_blocking_transport() -> None: assert transport is not None await fab.aclose() assert transport.is_closed + + +async def test_aclose_closes_default_auth_transport() -> None: + """The token-fetch client is owned by Donkey and shares its lifecycle (BG §1.1).""" + fab = Donkey(DonkeyConfig(client_id="control-id", client_secret="control-secret")) + auth_http = fab._owned_auth_http + assert auth_http is not None + + await fab.aclose() + + assert auth_http.is_closed + assert fab._owned_auth_http is None + + +async def test_aclose_leaves_injected_auth_provider_resources_open() -> None: + """A caller-supplied auth provider remains caller-owned.""" + import httpx + + async with httpx.AsyncClient() as auth_http: + auth = AnypointConnectedApp( + client_id="control-id", + client_secret="control-secret", + control_plane_url="https://anypoint.example", + http_client=auth_http, + token_path="/token", + ) + cfg = DonkeyConfig( + client_id="default-control-id", + client_secret="default-control-secret", + llm_proxy_url="https://proxy", + llm_proxy_client_id="cid", + llm_proxy_client_secret="csecret", + ) + fab = Donkey(cfg, auth=auth) + assert fab._owned_auth_http is None + + await fab.aclose() + + assert not auth_http.is_closed From cbdaac21b47545e4850b31f75b06630619e05342 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 09:21:38 +0200 Subject: [PATCH 03/29] fix(tools): return independent filtered views (#468) Closes #462 --- python/src/donkey_kit/tools/session.py | 23 +++++++++++++++-------- python/tests/unit/test_pure_logic.py | 20 ++++++++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/python/src/donkey_kit/tools/session.py b/python/src/donkey_kit/tools/session.py index 7504594d..4fcb7d39 100644 --- a/python/src/donkey_kit/tools/session.py +++ b/python/src/donkey_kit/tools/session.py @@ -20,9 +20,14 @@ class ToolSet: - def __init__(self, servers: list[McpServerHandle]) -> None: - self._servers = servers - self._filter = ToolFilter() + def __init__( + self, + servers: list[McpServerHandle], + *, + _filter: ToolFilter | None = None, + ) -> None: + self._servers = list(servers) + self._filter = _filter if _filter is not None else ToolFilter() # ---- filtering (pure, implemented) ------------------------------------ def filter( @@ -36,12 +41,14 @@ def filter( tools; handing 60 descriptors to a model degrades it and inflates token cost (BG §2.7).""" - self._filter = ToolFilter( - allow=frozenset(allow) if allow is not None else None, - deny=frozenset(deny or ()), - predicate=predicate, + return ToolSet( + self._servers, + _filter=ToolFilter( + allow=frozenset(allow) if allow is not None else None, + deny=frozenset(deny or ()), + predicate=predicate, + ), ) - return self @property def name_map(self) -> dict[str, str]: diff --git a/python/tests/unit/test_pure_logic.py b/python/tests/unit/test_pure_logic.py index 46f87986..72e579cb 100644 --- a/python/tests/unit/test_pure_logic.py +++ b/python/tests/unit/test_pure_logic.py @@ -6,8 +6,10 @@ from donkey_kit.core.cache import TTLCache from donkey_kit.provisioning.publish import content_digest from donkey_kit.registry.governance import Check, GovernanceCriteria, evaluate +from donkey_kit.registry.models import AssetRef, McpServerHandle from donkey_kit.registry.publication import check_description_quality from donkey_kit.tools.filter import ToolDescriptor, ToolFilter, resolve_collisions +from donkey_kit.tools.session import ToolSet def test_ttl_cache_expires() -> None: @@ -42,6 +44,24 @@ def test_tool_filter_allow_deny_predicate() -> None: assert not f2.accepts(ToolDescriptor("hr", "other")) +def test_tool_set_filter_returns_independent_views() -> None: + server = McpServerHandle( + ref=AssetRef("com.acme", "hr", "1.0.0"), + endpoint_url="https://example.com/mcp", + tool_descriptors=({"name": "read"}, {"name": "write"}), + ) + tools = ToolSet([server]) + + read_only = tools.filter(allow=["read"]) + write_only = tools.filter(allow=["write"]) + + assert read_only is not tools + assert read_only._servers is not tools._servers + assert read_only.name_map == {"read": "read"} + assert write_only.name_map == {"write": "write"} + assert tools.name_map == {"read": "read", "write": "write"} + + def test_description_quality_flags_tautological_and_missing() -> None: issues = check_description_quality( [ From 9ec944a0ddd3cab12d4da9f4cbb6f4c1fbcfe04c Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 09:21:43 +0200 Subject: [PATCH 04/29] test(adapters): cover Agent Framework middleware and guard paths (#469) Closes #457 --- python/tests/unit/test_adapter_ergonomics.py | 74 ++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/python/tests/unit/test_adapter_ergonomics.py b/python/tests/unit/test_adapter_ergonomics.py index 7935c9d9..2c4489b7 100644 --- a/python/tests/unit/test_adapter_ergonomics.py +++ b/python/tests/unit/test_adapter_ergonomics.py @@ -14,6 +14,7 @@ from __future__ import annotations +import builtins import importlib import sys import types @@ -210,6 +211,79 @@ def test_only_langgraph_is_conformance_tested() -> None: } +# --- Agent Framework behavior (BG §1.2) ------------------------------------- + + +async def test_agent_framework_policy_middleware_passes_through_result() -> None: + from donkey_kit.integrations.agent_framework import AgentFrameworkAdapter + + context = object() + result = object() + + async def next_(received: object) -> object: + assert received is context + return result + + middleware = AgentFrameworkAdapter(_cfg(), _http()).policy_middleware() + + assert await middleware(context, next_) is result + + +async def test_agent_framework_policy_middleware_preserves_policy_violation() -> None: + from donkey_kit.core.errors import PolicyViolation + from donkey_kit.integrations.agent_framework import AgentFrameworkAdapter + + violation = PolicyViolation("gateway refused the request") + + async def next_(_context: object) -> None: + raise violation + + middleware = AgentFrameworkAdapter(_cfg(), _http()).policy_middleware() + + with pytest.raises(PolicyViolation) as exc_info: + await middleware(object(), next_) + + assert exc_info.value is violation + + +async def test_agent_framework_policy_middleware_preserves_unrelated_error() -> None: + from donkey_kit.integrations.agent_framework import AgentFrameworkAdapter + + error = RuntimeError("agent failed") + + async def next_(_context: object) -> None: + raise error + + middleware = AgentFrameworkAdapter(_cfg(), _http()).policy_middleware() + + with pytest.raises(RuntimeError) as exc_info: + await middleware(object(), next_) + + assert exc_info.value is error + + +def test_agent_framework_chat_client_blocks_unverified_import( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from donkey_kit.integrations.agent_framework import AgentFrameworkAdapter + + import_error = ImportError("OpenAIChatClient is unavailable") + original_import = builtins.__import__ + + def fail_openai_import(name: str, *args: Any, **kwargs: Any) -> Any: + if name == "agent_framework.openai": + raise import_error + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", fail_openai_import) + adapter = AgentFrameworkAdapter(_cfg(), _http()) + + with pytest.raises(NotImplementedError, match="blocked on verification") as exc_info: + adapter.chat_client("gpt-4o") + + assert exc_info.value.__cause__ is import_error + + # --- The two paths cannot drift (issue #33 AC) ------------------------------ # # Each framework exposes the SAME governed native object two ways: (a) the From cd392570f902c89e7f4f8d8b3c30960740cd264b Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 10:21:43 +0200 Subject: [PATCH 05/29] Fix duplicate public AssetType contracts (#467) Closes #463 --- CONTRIBUTING.md | 31 +++++++++++++------ docs/verified-apis.md | 1 + python/pyproject.toml | 5 +-- python/src/donkey_kit/__init__.py | 2 ++ python/src/donkey_kit/registry/__init__.py | 5 +-- python/src/donkey_kit/registry/publication.py | 10 +++--- python/tests/typecheck/asset_type_contract.py | 25 +++++++++++++++ python/tests/unit/test_registry_shapes.py | 12 ++++++- website/content/concepts/verification.mdx | 2 +- website/content/publishing.mdx | 16 +++++----- website/content/tool-access/discovery.mdx | 2 +- website/public/concepts/verification.md | 2 +- website/public/llms-full.txt | 20 ++++++------ website/public/publishing.md | 16 +++++----- website/public/tool-access/discovery.md | 2 +- 15 files changed, 102 insertions(+), 49 deletions(-) create mode 100644 python/tests/typecheck/asset_type_contract.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fb90a818..79b6cbc8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -229,8 +229,8 @@ issues**; the release PR's title carries the milestone and version (e.g. ## 2. Testing strategy -The repo has **five distinct test surfaces**, each with its own job and its own -CI gate. Getting the surface wrong either weakens a real gate (a framework test +The repo has **six distinct test surfaces**, each mapped to a specific CI or +local gate. Getting the surface wrong either weakens a real gate (a framework test slipped into `tests/unit`) or produces a false negative (a skipped conformance scenario nobody reviews). Pick by what the change exercises: @@ -242,8 +242,9 @@ scenario nobody reviews). Pick by what the change exercises: | A running local Omni Gateway (docker) | `@pytest.mark.local_gateway` (off by default) | | A real Anypoint sandbox | `@pytest.mark.sandbox` (off by default, gated by `DONKEY_SANDBOX_TESTS=1`) | | A framework's constructor signature/kwargs | `scripts/verify_frameworks.py` (not pytest) | +| A downstream package-root import/call combination that source-only analysis cannot exercise | **`tests/typecheck/`** — checked by `mypy`, not pytest | -If a change fits none of these, stop and ask — don't invent a sixth surface. +If a change fits none of these, stop and ask — don't invent a seventh surface. ### `tests/unit/` — the framework-free gate @@ -257,6 +258,15 @@ provable: `PolicyViolation` stays distinct from the retryable `remediation` (assert it directly). See [`ARCHITECTURE.md`](ARCHITECTURE.md#error-taxonomy-design-bg-12) for why. +### `tests/typecheck/` — downstream static contracts + +`mypy` strict-checks these small, non-pytest modules alongside `src/donkey_kit`. +Use this surface when the contract depends on how a consumer combines public +package-root imports and annotated calls — a composition source-only analysis +cannot exercise. Keep a matching runtime assertion under `tests/unit/` when the +contract also has runtime behavior; a typecheck fixture is not a substitute for +a runtime test. + ### The conformance kit — "never a silent skip" One suite (`python/tests/conformance/suite.py`) runs identically against every @@ -345,13 +355,14 @@ python scripts/verify_frameworks.py [--live] [--only ] [--emit-verified] The full pre-write checklist lives in the `ddk-coding-conventions` skill and the build plan; the load-bearing rules: -- **`mypy --strict`, blocking.** The whole `src/donkey_kit` tree is - strict-checked. Annotate every public signature; no untyped defs, no implicit - `Any`. Prefer `X | None` over `Optional[X]` (ruff `UP` rewrites the old form), - and put `from __future__ import annotations` at the top of every module (house - style). Don't silence a real signature mismatch with an unexplained - `# type: ignore` — the single `[[tool.mypy.overrides]]` block already handles - optional/absent framework deps. +- **`mypy --strict`, blocking.** The whole `src/donkey_kit` tree and the + downstream public-API contracts under `tests/typecheck/` are strict-checked. + Annotate every public signature; no untyped defs, no implicit `Any`. Prefer + `X | None` over `Optional[X]` (ruff `UP` rewrites the old form), and put + `from __future__ import annotations` at the top of every module (house style). + Don't silence a real signature mismatch with an unexplained `# type: ignore` + — the single `[[tool.mypy.overrides]]` block already handles optional/absent + framework deps. - **Framework-free core & lazy imports.** `core/` depends on **httpx + pydantic only** — no agent framework, ever. Adapters import their framework **lazily, inside the method that uses it**, never at module top level; import the diff --git a/docs/verified-apis.md b/docs/verified-apis.md index b8455d76..bc5056ee 100644 --- a/docs/verified-apis.md +++ b/docs/verified-apis.md @@ -360,6 +360,7 @@ provisioning API. Exact REST calls behind the CLI are now recorded in §12 | Item | Gates | Status | Finding | Source | |---|---|---|---|---| | First-class Exchange asset types for MCP servers & AI agents? | BG §2.5 + `asset_types` filter (governed-only discovery) | VERIFIED (CLI) | Yes — Exchange assets named "… MCP Server" and "… Agent Network"/agent, managed as API Manager instances | 2026-08-28: `api-mgr:api:list` | +| Exact publication / discovery asset-type tokens (`mcp`, `a2a-agent`, `agent`, `api`) | `PublicationAssetType`, `AssetRef.type`, `ExchangeRegistry.search(asset_types=)` | UNVERIFIED | SDK-local assumption only: publication currently reuses the discovery `AssetType` Literal values; the direct publication and search contracts remain blocked pending capture | — | | Publication mechanism for non-Mule assets (REST / CLI / Maven)? JVM needed? | §7 CI story | VERIFIED (CLI) | Maven + CLI: `agent-network project publish` publishes the built project to Exchange; **JVM required** (`mvnw`) | 2026-08-28: `agent-network:project:publish --help` | | Publication uses Maven GAV coordinates | BG §2.5 | VERIFIED (CLI) | group-id/asset-id/asset-version; groupId defaults to org id | 2026-08-28: `:project:create --help`, `api-mgr:api:describe` | | Documentation pages publishable? | BG §2.5 | PARTIAL | `exchange asset page` + `exchange asset resource` topics exist | 2026-08-28: `exchange:asset --help` | diff --git a/python/pyproject.toml b/python/pyproject.toml index d1891311..c3f65ae0 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -136,10 +136,11 @@ packages = ["src/donkey_kit"] "tests/fixtures/anypoint/llm_proxy" = "donkey_kit/simulator/_fixtures/anypoint/llm_proxy" [tool.mypy] -# Working instruction #8: mypy --strict, blocking in CI. +# Working instruction #8: mypy --strict, blocking in CI. Typecheck fixtures pin +# downstream package-root usage that source-only analysis cannot exercise. strict = true python_version = "3.10" -files = ["src/donkey_kit"] +files = ["src/donkey_kit", "tests/typecheck"] plugins = ["pydantic.mypy"] [[tool.mypy.overrides]] diff --git a/python/src/donkey_kit/__init__.py b/python/src/donkey_kit/__init__.py index c37e8a41..94dda37c 100644 --- a/python/src/donkey_kit/__init__.py +++ b/python/src/donkey_kit/__init__.py @@ -55,6 +55,7 @@ Contact, GovernanceCriteria, Publication, + PublicationAssetType, ) __version__ = "0.1.0.dev2" @@ -85,6 +86,7 @@ "PolicyViolation", "PromptInjectionBlocked", "Publication", + "PublicationAssetType", "ProvisioningError", "PublicationDrift", "Region", diff --git a/python/src/donkey_kit/registry/__init__.py b/python/src/donkey_kit/registry/__init__.py index c6dc4295..f9f946d3 100644 --- a/python/src/donkey_kit/registry/__init__.py +++ b/python/src/donkey_kit/registry/__init__.py @@ -2,12 +2,12 @@ from .exchange import ExchangeRegistry from .governance import STRICT, Check, GovernanceCriteria, GovernanceReport, evaluate -from .models import AgentHandle, AssetRef, McpServerHandle +from .models import AgentHandle, AssetRef, AssetType, McpServerHandle from .publication import ( - AssetType, Contact, DescriptionIssue, Publication, + PublicationAssetType, VersionStrategy, check_description_quality, ) @@ -25,6 +25,7 @@ "GovernanceReport", "McpServerHandle", "Publication", + "PublicationAssetType", "VersionStrategy", "check_description_quality", "evaluate", diff --git a/python/src/donkey_kit/registry/publication.py b/python/src/donkey_kit/registry/publication.py index d37aeda2..da856a65 100644 --- a/python/src/donkey_kit/registry/publication.py +++ b/python/src/donkey_kit/registry/publication.py @@ -24,9 +24,11 @@ from ..core import _verify -class AssetType(Enum): - MCP_SERVER = "mcp_server" - A2A_AGENT = "a2a_agent" +class PublicationAssetType(Enum): + """SDK publication categories; exact Exchange token strings are UNVERIFIED.""" + + MCP_SERVER = "mcp" + A2A_AGENT = "a2a-agent" AGENT = "agent" API = "api" @@ -52,7 +54,7 @@ class DescriptionIssue: @dataclass(frozen=True) class Publication: - asset_type: AssetType + asset_type: PublicationAssetType group_id: str asset_id: str version: str diff --git a/python/tests/typecheck/asset_type_contract.py b/python/tests/typecheck/asset_type_contract.py new file mode 100644 index 00000000..1807cf12 --- /dev/null +++ b/python/tests/typecheck/asset_type_contract.py @@ -0,0 +1,25 @@ +"""Static contract for package-root asset-type imports (#463).""" + +from __future__ import annotations + +from donkey_kit import AssetRef, AssetType, Publication, PublicationAssetType +from donkey_kit.registry import ExchangeRegistry + +discovery_type: AssetType = "mcp" +publication_type: AssetType = PublicationAssetType.MCP_SERVER.value +asset_ref = AssetRef( + group_id="com.acme", asset_id="tools", version="1.0.0", type=discovery_type +) + +publication = Publication( + asset_type=PublicationAssetType.MCP_SERVER, + group_id="com.acme", + asset_id="tools", + version="1.0.0", + name="Tools", + description="Tools for static contract verification.", +) + + +async def search_by_public_asset_type(registry: ExchangeRegistry) -> None: + await registry.search(asset_types=[discovery_type]) diff --git a/python/tests/unit/test_registry_shapes.py b/python/tests/unit/test_registry_shapes.py index 98ad5a3b..10198a91 100644 --- a/python/tests/unit/test_registry_shapes.py +++ b/python/tests/unit/test_registry_shapes.py @@ -16,7 +16,8 @@ import pytest -from donkey_kit.registry.models import AssetRef, McpServerHandle +from donkey_kit import AssetRef, AssetType, PublicationAssetType +from donkey_kit.registry.models import McpServerHandle from donkey_kit.tools.filter import ToolDescriptor, resolve_collisions FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "a2d" @@ -25,6 +26,15 @@ _TRANSPORT_NORMALIZATION = {"streamableHttp": "streamable_http"} +def test_package_root_asset_types_share_discovery_literal_values() -> None: + discovery_type: AssetType = "mcp" + ref = AssetRef(group_id="com.acme", asset_id="tools", version="1.0.0", type=discovery_type) + + assert ref.type == "mcp" + assert PublicationAssetType.MCP_SERVER.value == ref.type + assert PublicationAssetType.A2A_AGENT.value == "a2a-agent" + + def _load(name: str) -> dict: return json.loads((FIXTURES / name).read_text()) diff --git a/website/content/concepts/verification.mdx b/website/content/concepts/verification.mdx index 821d9b0a..59bab7f2 100644 --- a/website/content/concepts/verification.mdx +++ b/website/content/concepts/verification.mdx @@ -33,7 +33,7 @@ how a blocked value flips to verified — is documented for contributors in |---|---|---| | Framework **constructor signatures** (docs/verified-apis.md §8) | Unverified per framework | Adapters target recorded class names; a mismatch surfaces via the verification harness / nightly matrix rather than silently misbehaving. Agent Framework's client raises `blocked on verification` today. | | MCP tool discovery + binding ([Phase 2](/tool-access)) | Unverified | `donkey.tools.discover` and the binding classes raise `blocked on verification`. | -| Exchange publish API ([Phase 2](/publishing)) | Unverified | `donkey publish` is blocked; whether Exchange exposes first-class MCP/agent asset types is unconfirmed. | +| Exchange publish API ([Phase 2](/publishing)) | Partially verified | First-class MCP/agent asset types exist, but the exact publication and discovery token vocabulary is unverified. `donkey publish` and Exchange search remain blocked rather than sending the SDK's assumed `mcp` / `a2a-agent` values. | | Token-exchange endpoint for [on-behalf-of](/identity) (Phase 2) | Unverified | The exact endpoint and header the gateway expects are not guessed. | | [Policy discovery](/policies) (Phase 3) | **Missing upstream** | No such endpoint exists on the gateway today. Filed as an upstream gap — the handshake cannot be built until it ships. | | Budget query endpoint ([Phase 1](/budget)) | **Missing upstream** | Budget is reported in-band on response headers only, so `remaining` is last-known-good; `observed_at` exposes the staleness. | diff --git a/website/content/publishing.mdx b/website/content/publishing.mdx index 0d77f4df..056b8fc7 100644 --- a/website/content/publishing.mdx +++ b/website/content/publishing.mdx @@ -59,10 +59,10 @@ explicit `--allow-duplicate`, which logs at `WARNING`. ## The `Publication` object ```python -from donkey_kit import Publication, AssetType, Contact +from donkey_kit import Contact, Publication, PublicationAssetType pub = Publication( - asset_type=AssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API + asset_type=PublicationAssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API group_id="${ANYPOINT_ORG_ID}", asset_id="hr-tools-mcp", version="1.3.0", # semver @@ -98,12 +98,12 @@ pub = Publication( | `API` | OpenAPI / AsyncAPI | user-supplied file; no generation | - **Verification gate.** Whether Exchange exposes first-class asset types for - MCP servers and AI agents is unverified. The Agent Registry is built on - Exchange, so it is likely — but if it does not, publication degrades to a - generic asset type carrying tags, which weakens discoverability and breaks - `asset_types=["mcp"]` filtering in [Tool access](/tool-access). The two - features share this dependency; it is verified once, recorded once. + **Verification gate.** Exchange exposes first-class MCP-server and AI-agent + asset categories, but the exact token vocabulary used by publication and + discovery filters is unverified. `PublicationAssetType` currently shares the + SDK's `mcp` / `a2a-agent` discovery values; no blocked Exchange call sends + those assumed values until the direct contract is captured. The two features + share this dependency, so it is verified once and recorded once. ## `descriptor="auto"` — deriving the spec from code diff --git a/website/content/tool-access/discovery.mdx b/website/content/tool-access/discovery.mdx index 6b30a910..9c4cdec2 100644 --- a/website/content/tool-access/discovery.mdx +++ b/website/content/tool-access/discovery.mdx @@ -41,7 +41,7 @@ tools = await donkey.tools.discover( | `governed_only` | `bool \| GovernanceCriteria \| None` | `True` applies the default criteria; pass a `GovernanceCriteria` (e.g. `STRICT`) for explicit rules; `None` = unfiltered. | | `domain` | `str \| None` | Catalog domain. | | `tags` | `list[str] \| None` | All listed tags must be present. | -| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`. | +| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`; exact Exchange filter tokens remain unverified. | | `environment` | `str \| None` | Which environment governance is computed against. | | `limit` | `int` | Max results (default 50). | diff --git a/website/public/concepts/verification.md b/website/public/concepts/verification.md index f2ca812b..06efac4a 100644 --- a/website/public/concepts/verification.md +++ b/website/public/concepts/verification.md @@ -29,7 +29,7 @@ how a blocked value flips to verified — is documented for contributors in |---|---|---| | Framework **constructor signatures** (docs/verified-apis.md §8) | Unverified per framework | Adapters target recorded class names; a mismatch surfaces via the verification harness / nightly matrix rather than silently misbehaving. Agent Framework's client raises `blocked on verification` today. | | MCP tool discovery + binding ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/tool-access.md)) | Unverified | `donkey.tools.discover` and the binding classes raise `blocked on verification`. | -| Exchange publish API ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/publishing.md)) | Unverified | `donkey publish` is blocked; whether Exchange exposes first-class MCP/agent asset types is unconfirmed. | +| Exchange publish API ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/publishing.md)) | Partially verified | First-class MCP/agent asset types exist, but the exact publication and discovery token vocabulary is unverified. `donkey publish` and Exchange search remain blocked rather than sending the SDK's assumed `mcp` / `a2a-agent` values. | | Token-exchange endpoint for [on-behalf-of](https://donkey-development-kit.github.io/donkey-development-kit/identity.md) (Phase 2) | Unverified | The exact endpoint and header the gateway expects are not guessed. | | [Policy discovery](https://donkey-development-kit.github.io/donkey-development-kit/policies.md) (Phase 3) | **Missing upstream** | No such endpoint exists on the gateway today. Filed as an upstream gap — the handshake cannot be built until it ships. | | Budget query endpoint ([Phase 1](https://donkey-development-kit.github.io/donkey-development-kit/budget.md)) | **Missing upstream** | Budget is reported in-band on response headers only, so `remaining` is last-known-good; `observed_at` exposes the staleness. | diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 6d77e203..3b2973e7 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -3533,7 +3533,7 @@ tools = await donkey.tools.discover( | `governed_only` | `bool \| GovernanceCriteria \| None` | `True` applies the default criteria; pass a `GovernanceCriteria` (e.g. `STRICT`) for explicit rules; `None` = unfiltered. | | `domain` | `str \| None` | Catalog domain. | | `tags` | `list[str] \| None` | All listed tags must be present. | -| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`. | +| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`; exact Exchange filter tokens remain unverified. | | `environment` | `str \| None` | Which environment governance is computed against. | | `limit` | `int` | Max results (default 50). | @@ -4105,10 +4105,10 @@ explicit `--allow-duplicate`, which logs at `WARNING`. ## The `Publication` object ```python -from donkey_kit import Publication, AssetType, Contact +from donkey_kit import Contact, Publication, PublicationAssetType pub = Publication( - asset_type=AssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API + asset_type=PublicationAssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API group_id="${ANYPOINT_ORG_ID}", asset_id="hr-tools-mcp", version="1.3.0", # semver @@ -4143,12 +4143,12 @@ pub = Publication( | `AGENT` | agent descriptor (no A2A surface) | framework introspection, best-effort | | `API` | OpenAPI / AsyncAPI | user-supplied file; no generation | - **Verification gate.** Whether Exchange exposes first-class asset types for - MCP servers and AI agents is unverified. The Agent Registry is built on - Exchange, so it is likely — but if it does not, publication degrades to a - generic asset type carrying tags, which weakens discoverability and breaks - `asset_types=["mcp"]` filtering in [Tool access](https://donkey-development-kit.github.io/donkey-development-kit/tool-access.md). The two - features share this dependency; it is verified once, recorded once. + **Verification gate.** Exchange exposes first-class MCP-server and AI-agent + asset categories, but the exact token vocabulary used by publication and + discovery filters is unverified. `PublicationAssetType` currently shares the + SDK's `mcp` / `a2a-agent` discovery values; no blocked Exchange call sends + those assumed values until the direct contract is captured. The two features + share this dependency, so it is verified once and recorded once. ## `descriptor="auto"` — deriving the spec from code @@ -4307,7 +4307,7 @@ how a blocked value flips to verified — is documented for contributors in |---|---|---| | Framework **constructor signatures** (docs/verified-apis.md §8) | Unverified per framework | Adapters target recorded class names; a mismatch surfaces via the verification harness / nightly matrix rather than silently misbehaving. Agent Framework's client raises `blocked on verification` today. | | MCP tool discovery + binding ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/tool-access.md)) | Unverified | `donkey.tools.discover` and the binding classes raise `blocked on verification`. | -| Exchange publish API ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/publishing.md)) | Unverified | `donkey publish` is blocked; whether Exchange exposes first-class MCP/agent asset types is unconfirmed. | +| Exchange publish API ([Phase 2](https://donkey-development-kit.github.io/donkey-development-kit/publishing.md)) | Partially verified | First-class MCP/agent asset types exist, but the exact publication and discovery token vocabulary is unverified. `donkey publish` and Exchange search remain blocked rather than sending the SDK's assumed `mcp` / `a2a-agent` values. | | Token-exchange endpoint for [on-behalf-of](https://donkey-development-kit.github.io/donkey-development-kit/identity.md) (Phase 2) | Unverified | The exact endpoint and header the gateway expects are not guessed. | | [Policy discovery](https://donkey-development-kit.github.io/donkey-development-kit/policies.md) (Phase 3) | **Missing upstream** | No such endpoint exists on the gateway today. Filed as an upstream gap — the handshake cannot be built until it ships. | | Budget query endpoint ([Phase 1](https://donkey-development-kit.github.io/donkey-development-kit/budget.md)) | **Missing upstream** | Budget is reported in-band on response headers only, so `remaining` is last-known-good; `observed_at` exposes the staleness. | diff --git a/website/public/publishing.md b/website/public/publishing.md index d964af8b..03ee3670 100644 --- a/website/public/publishing.md +++ b/website/public/publishing.md @@ -49,10 +49,10 @@ explicit `--allow-duplicate`, which logs at `WARNING`. ## The `Publication` object ```python -from donkey_kit import Publication, AssetType, Contact +from donkey_kit import Contact, Publication, PublicationAssetType pub = Publication( - asset_type=AssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API + asset_type=PublicationAssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API group_id="${ANYPOINT_ORG_ID}", asset_id="hr-tools-mcp", version="1.3.0", # semver @@ -87,12 +87,12 @@ pub = Publication( | `AGENT` | agent descriptor (no A2A surface) | framework introspection, best-effort | | `API` | OpenAPI / AsyncAPI | user-supplied file; no generation | - **Verification gate.** Whether Exchange exposes first-class asset types for - MCP servers and AI agents is unverified. The Agent Registry is built on - Exchange, so it is likely — but if it does not, publication degrades to a - generic asset type carrying tags, which weakens discoverability and breaks - `asset_types=["mcp"]` filtering in [Tool access](https://donkey-development-kit.github.io/donkey-development-kit/tool-access.md). The two - features share this dependency; it is verified once, recorded once. + **Verification gate.** Exchange exposes first-class MCP-server and AI-agent + asset categories, but the exact token vocabulary used by publication and + discovery filters is unverified. `PublicationAssetType` currently shares the + SDK's `mcp` / `a2a-agent` discovery values; no blocked Exchange call sends + those assumed values until the direct contract is captured. The two features + share this dependency, so it is verified once and recorded once. ## `descriptor="auto"` — deriving the spec from code diff --git a/website/public/tool-access/discovery.md b/website/public/tool-access/discovery.md index bbd6483d..33cc0ccc 100644 --- a/website/public/tool-access/discovery.md +++ b/website/public/tool-access/discovery.md @@ -39,7 +39,7 @@ tools = await donkey.tools.discover( | `governed_only` | `bool \| GovernanceCriteria \| None` | `True` applies the default criteria; pass a `GovernanceCriteria` (e.g. `STRICT`) for explicit rules; `None` = unfiltered. | | `domain` | `str \| None` | Catalog domain. | | `tags` | `list[str] \| None` | All listed tags must be present. | -| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`. | +| `asset_types` | `list[AssetType] \| None` | e.g. `["mcp"]`; exact Exchange filter tokens remain unverified. | | `environment` | `str \| None` | Which environment governance is computed against. | | `limit` | `int` | Max results (default 50). | From 742ebda81c87094fc29013725dbd90f76001448f Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 10:40:32 +0200 Subject: [PATCH 06/29] docs(adapters): correct framework verification comments (#483) Closes #456 --- docs/m1-completion-checklist.md | 3 ++- python/src/donkey_kit/integrations/adk.py | 2 +- python/src/donkey_kit/integrations/agent_framework.py | 4 ++-- python/src/donkey_kit/integrations/anthropic.py | 2 +- python/src/donkey_kit/integrations/crewai.py | 2 +- python/src/donkey_kit/integrations/langgraph.py | 2 +- python/src/donkey_kit/integrations/llamaindex.py | 4 +++- python/src/donkey_kit/integrations/openai_agents.py | 2 +- python/src/donkey_kit/integrations/strands.py | 2 +- 9 files changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/m1-completion-checklist.md b/docs/m1-completion-checklist.md index c16f2cae..64a8c76b 100644 --- a/docs/m1-completion-checklist.md +++ b/docs/m1-completion-checklist.md @@ -21,7 +21,8 @@ external verification (cannot be closed by writing code). exists.** Every adapter is wired to the *live-verified proxy contract* (the framework-free core), but `docs/verified-apis.md` §8 currently lists **all eight** framework constructor/kwarg signatures as `UNVERIFIED`. The adapters' inline -`# verified: docs §8` comments are pointers to that table, **not** proof. +`# VERIFY …: docs/verified-apis.md §8` markers identify facts that still need +verification; they are **not** proof. ### How to verify (the procedure, made executable) diff --git a/python/src/donkey_kit/integrations/adk.py b/python/src/donkey_kit/integrations/adk.py index b22c24c6..e25ae118 100644 --- a/python/src/donkey_kit/integrations/adk.py +++ b/python/src/donkey_kit/integrations/adk.py @@ -44,7 +44,7 @@ def connection_kwargs(self) -> dict[str, Any]: } def model(self, model: str, **kw: Any) -> LiteLlm: - from google.adk.models.lite_llm import LiteLlm # verified: docs/verified-apis.md §8 + from google.adk.models.lite_llm import LiteLlm # VERIFY name/path: docs/verified-apis.md §8 # LiteLLM's OpenAI-compatible route needs the ``openai/`` prefix. return LiteLlm(model=f"openai/{model}", **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/agent_framework.py b/python/src/donkey_kit/integrations/agent_framework.py index 6aac243a..376eb2af 100644 --- a/python/src/donkey_kit/integrations/agent_framework.py +++ b/python/src/donkey_kit/integrations/agent_framework.py @@ -42,7 +42,7 @@ def chat_client(self, model: str, **kw: Any) -> Any: self._require_proxy() try: from agent_framework.openai import ( - OpenAIChatClient, # VERIFY name/path docs/verified-apis.md §8 + OpenAIChatClient, # VERIFY name/path: docs/verified-apis.md §8 ) except ImportError as exc: raise _verify.blocked( @@ -53,7 +53,7 @@ def chat_client(self, model: str, **kw: Any) -> Any: ) from exc return OpenAIChatClient( - model_id=model, # VERIFY kwarg name docs/verified-apis.md §8 + model_id=model, # VERIFY kwarg name: docs/verified-apis.md §8 **self.connection_kwargs(), **kw, ) diff --git a/python/src/donkey_kit/integrations/anthropic.py b/python/src/donkey_kit/integrations/anthropic.py index 11006576..e769e36f 100644 --- a/python/src/donkey_kit/integrations/anthropic.py +++ b/python/src/donkey_kit/integrations/anthropic.py @@ -67,7 +67,7 @@ def client(self, **kw: Any) -> AsyncAnthropic: """Return a native ``anthropic.AsyncAnthropic`` pointed at the proxy. Pass the model id per call (``messages.create(model=..., ...)``), per the Anthropic SDK's own surface (BG §1.8).""" - from anthropic import AsyncAnthropic # verified: docs/verified-apis.md §8 + from anthropic import AsyncAnthropic # VERIFY name/path: docs/verified-apis.md §8 return AsyncAnthropic(**self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/crewai.py b/python/src/donkey_kit/integrations/crewai.py index e2c9fb7a..7a4ab7b6 100644 --- a/python/src/donkey_kit/integrations/crewai.py +++ b/python/src/donkey_kit/integrations/crewai.py @@ -44,7 +44,7 @@ def connection_kwargs(self) -> dict[str, Any]: def llm(self, model: str, **kw: Any) -> LLM: """Return a native ``crewai.LLM`` pointed at the proxy (BG §1.8).""" - from crewai import LLM # verified: docs/verified-apis.md §8 + from crewai import LLM # VERIFY name/path: docs/verified-apis.md §8 # LiteLLM's OpenAI-compatible route needs the ``openai/`` prefix. return LLM(model=f"openai/{model}", **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/langgraph.py b/python/src/donkey_kit/integrations/langgraph.py index f3ee4288..c3fbcc30 100644 --- a/python/src/donkey_kit/integrations/langgraph.py +++ b/python/src/donkey_kit/integrations/langgraph.py @@ -62,7 +62,7 @@ def connection_kwargs(self) -> dict[str, Any]: def chat_model(self, model: str, **kw: Any) -> ChatOpenAI: """Return a native ``ChatOpenAI`` pointed at the proxy (BG §1.8).""" - from langchain_openai import ChatOpenAI # verified name: docs/verified-apis.md §8 + from langchain_openai import ChatOpenAI # VERIFY name/path: docs/verified-apis.md §8 return ChatOpenAI(model=model, **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/llamaindex.py b/python/src/donkey_kit/integrations/llamaindex.py index eb469456..55a926c6 100644 --- a/python/src/donkey_kit/integrations/llamaindex.py +++ b/python/src/donkey_kit/integrations/llamaindex.py @@ -41,7 +41,9 @@ def connection_kwargs(self) -> dict[str, Any]: } def llm(self, model: str, **kw: Any) -> OpenAILike: - from llama_index.llms.openai_like import OpenAILike # verified: docs/verified-apis.md §8 + from llama_index.llms.openai_like import ( + OpenAILike, # VERIFY name/path: docs/verified-apis.md §8 + ) return OpenAILike(model=model, **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/openai_agents.py b/python/src/donkey_kit/integrations/openai_agents.py index e5c45a22..5fe6460a 100644 --- a/python/src/donkey_kit/integrations/openai_agents.py +++ b/python/src/donkey_kit/integrations/openai_agents.py @@ -58,7 +58,7 @@ def connection_kwargs(self) -> dict[str, Any]: def model(self, model: str, **kw: Any) -> OpenAIChatCompletionsModel: """Return a native ``OpenAIChatCompletionsModel`` pointed at the proxy, ready to pass into ``agents.Agent(model=...)`` (BG §1.8).""" - from agents import OpenAIChatCompletionsModel # verified: docs/verified-apis.md §8 + from agents import OpenAIChatCompletionsModel # VERIFY name/path: docs/verified-apis.md §8 return OpenAIChatCompletionsModel(model=model, **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/strands.py b/python/src/donkey_kit/integrations/strands.py index 78a50dd8..b91996b4 100644 --- a/python/src/donkey_kit/integrations/strands.py +++ b/python/src/donkey_kit/integrations/strands.py @@ -36,7 +36,7 @@ def connection_kwargs(self) -> dict[str, Any]: } def model(self, model: str, **kw: Any) -> OpenAIModel: - from strands.models.openai import OpenAIModel # verified: docs/verified-apis.md §8 + from strands.models.openai import OpenAIModel # VERIFY name/path: docs/verified-apis.md §8 return OpenAIModel(model_id=model, **self.connection_kwargs(), **kw) From 69e54fd918bfded3250be9a4bee4b4c6bdb78e54 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 10:47:32 +0200 Subject: [PATCH 07/29] docs(cli): replace stale provisioning citations (#482) Closes #460 --- python/src/donkey_kit/governance.py | 8 +++----- python/src/donkey_kit/provisioning/cli.py | 7 ++++--- python/src/donkey_kit/provisioning/lint.py | 4 ++-- python/src/donkey_kit/provisioning/planner.py | 6 +++--- 4 files changed, 12 insertions(+), 13 deletions(-) diff --git a/python/src/donkey_kit/governance.py b/python/src/donkey_kit/governance.py index beccdf1b..e99d7c80 100644 --- a/python/src/donkey_kit/governance.py +++ b/python/src/donkey_kit/governance.py @@ -113,11 +113,9 @@ def simulate(self) -> SimulationContext: # ---- verb 2: export (sandbox/prod, laptop) ---------------------------- def export(self, path: str | Path | None = None) -> str: - """Compile to a ``donkey.yaml`` fragment (provisioning-as-code). Lands with M4 (the build - plan phases).""" + """Compile to a ``donkey.yaml`` fragment (provisioning-as-code).""" raise _verify.blocked( - "Governance.export() emits the M4 donkey.yaml spec format; it ships " - "with provisioning (M4, the build plan phases). Schema is shared with " + "Governance.export() emits the donkey.yaml spec format. Schema is shared with " "provisioning-as-code — do not diverge it (provisioning-as-code)." ) @@ -175,7 +173,7 @@ async def __aenter__(self) -> Any: "local Omni Gateway docker harness + Local-Mode LLM-Proxy/MCP-Bridge " "availability (BG §1.4, the Verification milestone). The [local] extra and the loud " "skipped-policy report (skipped_policies()) are scaffolded; the docker " - "orchestration is M2.5 and gated on the M0 local-mode findings." + "orchestration is gated on the Verification milestone's local-mode findings." ) async def __aexit__(self, *exc: object) -> None: diff --git a/python/src/donkey_kit/provisioning/cli.py b/python/src/donkey_kit/provisioning/cli.py index 58c5c1a4..ac72d0e2 100644 --- a/python/src/donkey_kit/provisioning/cli.py +++ b/python/src/donkey_kit/provisioning/cli.py @@ -78,9 +78,10 @@ def validate(file: Path = typer.Option(..., "-f", "--file", help="donkey.yaml")) """Validate a donkey.yaml against the schema (needs no platform API). Hidden: it operates on the ``donkey.yaml`` provisioning spec, part of the - provisioning control-plane surface that is cut for Phase 1 (`BG §1.8`). The - command still works for anyone driving that YAML, but the four supported - ``donkey`` commands are ``init``/``doctor``/``mock``/``test``. + provisioning control-plane surface excluded by the build plan's "Do not build, + at any phase" boundary. The command still works for anyone driving that YAML, + but the four supported ``donkey`` commands are + ``init``/``doctor``/``mock``/``test``. """ spec = _load_spec(file) typer.secho( diff --git a/python/src/donkey_kit/provisioning/lint.py b/python/src/donkey_kit/provisioning/lint.py index 3e97386d..1931adc4 100644 --- a/python/src/donkey_kit/provisioning/lint.py +++ b/python/src/donkey_kit/provisioning/lint.py @@ -1,7 +1,7 @@ """Governance lint (provisioning-as-code). -Small, cheap, uncontroversial, and the fastest way to get a platform team to say -yes to the SDK — built in M1 even though the rest of provisioning is M3/M4. +Small, cheap, uncontroversial, and independent of the provisioning control plane +excluded by the build plan's "Do not build, at any phase" boundary. Validates API specs against project and centralized rulesets BEFORE anything is published, and fails the PR on ``error`` severity. Ruleset RESOLUTION against the diff --git a/python/src/donkey_kit/provisioning/planner.py b/python/src/donkey_kit/provisioning/planner.py index 0af49bbb..4f177c07 100644 --- a/python/src/donkey_kit/provisioning/planner.py +++ b/python/src/donkey_kit/provisioning/planner.py @@ -7,8 +7,8 @@ * ``--dry-run`` / ``--out plan.json`` for CI gating. GATE (provisioning-as-code, verification discipline): whether a usable MCP Bridge provisioning API -exists is an M0 finding. If UI-only, this whole module is cut and we emit Terraform instead -(provisioning-as-code). Until confirmed, planning against live state is blocked. +exists is an open Verification-milestone question. If UI-only, this whole module is cut and we emit +Terraform instead (provisioning-as-code). Until confirmed, planning against live state is blocked. """ from __future__ import annotations @@ -47,6 +47,6 @@ def render(self) -> str: async def build_plan(spec: DonkeySpec, donkey: object) -> Plan: raise _verify.blocked( "MCP Bridge provisioning read API for read-before-write planning " - "(provisioning-as-code). If M0 finds it UI-only, pivot to Terraform generation " + "(provisioning-as-code). If verification finds it UI-only, pivot to Terraform generation " "(provisioning-as-code) — do NOT reverse-engineer internal endpoints." ) From 204fc0dc30549081a461273a95865e70d360a997 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 11:08:41 +0200 Subject: [PATCH 08/29] Fix budget pacing recovery after window reset (#480) Closes #452 --- python/src/donkey_kit/core/budget.py | 27 ++++++++++++------- python/tests/unit/test_budget_pacing.py | 29 +++++++++++++++++++++ spec/donkey-development-kit-build-guide.md | 17 ++++++++---- website/content/budget.mdx | 23 +++++++++------- website/content/scenarios/nightly-batch.mdx | 4 ++- website/public/budget.md | 23 +++++++++------- website/public/llms-full.txt | 27 +++++++++++-------- website/public/scenarios/nightly-batch.md | 4 ++- 8 files changed, 107 insertions(+), 47 deletions(-) diff --git a/python/src/donkey_kit/core/budget.py b/python/src/donkey_kit/core/budget.py index 194eaa18..5af54b08 100644 --- a/python/src/donkey_kit/core/budget.py +++ b/python/src/donkey_kit/core/budget.py @@ -187,23 +187,32 @@ async def pace(self, *, reserve: float = 0.0) -> AsyncIterator[None]: full exhaustion. On entry, if the observed :attr:`fraction_used` has reached ``1.0 - reserve``, :class:`~donkey_kit.core.errors.BudgetReserveReached` is raised and the guarded block never runs — so the request that would cross - the reserve is never issued. Recover with :meth:`wait_for_reset` and retry:: - - try: - async with donkey.budget.pace(reserve=0.05): - await enrich(batch) - except BudgetReserveReached: - await donkey.budget.wait_for_reset() + the reserve is never issued. Recover with :meth:`wait_for_reset` and retry + the same work:: + + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached: + await donkey.budget.wait_for_reset() + continue + break An **unobserved** budget (no call has returned yet, so :attr:`fraction_used` is ``None``) lets the block through: with nothing observed there is no basis to refuse, and blocking forever on a cold start - would be worse than one request that discovers the real headroom. + would be worse than one request that discovers the real headroom. An + observation whose :attr:`reset_at` has elapsed is stale for the same reason, + so the retry passes through and its response refreshes the in-band budget. + Before ``reset_at`` — or when no reset time was observed — the reserve guard + remains active. """ if not 0.0 <= reserve <= 1.0: raise ValueError(f"reserve must be within [0.0, 1.0], got {reserve!r}") used = self.fraction_used - if used is not None and used >= 1.0 - reserve: + window_expired = self.reset_at is not None and _utcnow() >= self.reset_at + if used is not None and used >= 1.0 - reserve and not window_expired: raise BudgetReserveReached( f"Budget reserve reached: {used:.1%} of the window used, " f"reserve is {reserve:.1%} (trips at {1.0 - reserve:.1%}).", diff --git a/python/tests/unit/test_budget_pacing.py b/python/tests/unit/test_budget_pacing.py index 86b42852..99f86414 100644 --- a/python/tests/unit/test_budget_pacing.py +++ b/python/tests/unit/test_budget_pacing.py @@ -27,6 +27,12 @@ def _observe(b: Budget, *, limit: int, remaining: int, reset_ms: int | None = No b.observe(_resp(**headers), now=_FIXED_NOW) +@pytest.fixture(autouse=True) +def fixed_clock(monkeypatch: pytest.MonkeyPatch) -> None: + """Keep pace() on the same deterministic clock as the injected observations.""" + monkeypatch.setattr(budget_mod, "_utcnow", lambda: _FIXED_NOW) + + @pytest.fixture def recorded_sleeps(monkeypatch: pytest.MonkeyPatch) -> list[float]: """Replace ``asyncio.sleep`` inside budget.py with a recorder so tests never @@ -182,3 +188,26 @@ async def test_pace_then_wait_then_resume_cycle(recorded_sleeps: list[float]) -> async with b.pace(reserve=0.05): ran_after = True assert ran_after + + +async def test_pace_allows_probe_after_wait_without_manual_observe( + monkeypatch: pytest.MonkeyPatch, + recorded_sleeps: list[float], +) -> None: + """Regression #452: the documented wait-and-retry loop must make progress + without injecting the fresh observation that only the guarded request can produce.""" + b = Budget() + _observe(b, limit=20000, remaining=500, reset_ms=60000) # 97.5% used + + with pytest.raises(BudgetReserveReached): + async with b.pace(reserve=0.05): + raise AssertionError("body must not run before the window resets") + + await b.wait_for_reset(now=_FIXED_NOW) + assert recorded_sleeps == [pytest.approx(60.0)] + + monkeypatch.setattr(budget_mod, "_utcnow", lambda: _FIXED_NOW + timedelta(seconds=60)) + ran_probe = False + async with b.pace(reserve=0.05): + ran_probe = True + assert ran_probe diff --git a/spec/donkey-development-kit-build-guide.md b/spec/donkey-development-kit-build-guide.md index a176a37a..c2309678 100644 --- a/spec/donkey-development-kit-build-guide.md +++ b/spec/donkey-development-kit-build-guide.md @@ -143,14 +143,20 @@ async with donkey.budget.pace(reserve=0.10): # raises BudgetReserveReached at 9 ```python for batch in chunks(records, 200): - async with donkey.budget.pace(reserve=0.05): - await enrich(batch) + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached: + await donkey.budget.wait_for_reset() + continue + break checkpoint(batch) -# on BudgetReserveReached: -await donkey.budget.wait_for_reset(); continue ``` -The job finishes by itself, overnight, with no human. +After `reset_at`, the old observation is stale, so `pace()` lets the retry +through and its response refreshes the in-band budget. The job finishes by +itself, overnight, with no human. **Why it matters — Scenario A.** A dashboard shows `fraction_used` per agent. The support agent's owner sees it climbing at 14:00 and requests an increase before the 16:00 peak instead of after the outage. @@ -159,6 +165,7 @@ The job finishes by itself, overnight, with no human. **Acceptance.** - `x-token-reset` in milliseconds is converted correctly (fixture with a known value; assert `reset_at` to the second). - `pace()` raises before the request that would cross the reserve, not after a 429. +- After `reset_at`, `pace()` lets one retry through so its response can refresh the in-band budget; the documented `pace()` → `wait_for_reset()` → retry loop makes progress without a manual observation. - Budget object is per-`Donkey`, not global; two `Donkey` instances with different credentials do not share state. **Effort:** S. diff --git a/website/content/budget.mdx b/website/content/budget.mdx index 5167bc88..b046e8a0 100644 --- a/website/content/budget.mdx +++ b/website/content/budget.mdx @@ -57,19 +57,20 @@ spending the budget twice to do the same work. ```python for batch in chunks(records, 200): - async with donkey.budget.pace(reserve=0.05): - await enrich(batch) + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached: + await donkey.budget.wait_for_reset() + continue + break checkpoint(batch) ``` -On `BudgetReserveReached`, wait for the window and carry on: - -```python -await donkey.budget.wait_for_reset() -continue -``` - -The job finishes unattended. Nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the +retry through so its response can refresh the in-band budget. The job finishes +unattended; nobody re-runs anything. ## The dashboard that prevents the outage @@ -100,6 +101,8 @@ last-known-good. See [Roadmap](/roadmap). against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. +- After `reset_at`, `pace()` lets the retry through so the response can refresh + the in-band budget; no manual observation is required. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/content/scenarios/nightly-batch.mdx b/website/content/scenarios/nightly-batch.mdx index 2ac0d698..b6f408ba 100644 --- a/website/content/scenarios/nightly-batch.mdx +++ b/website/content/scenarios/nightly-batch.mdx @@ -95,7 +95,9 @@ async def enrich_all(records, enrich): request that would cross the last 5% of the window — so you never spend the request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header - (milliseconds, converted for you). See [Budget & pacing](/budget). + (milliseconds, converted for you). Once that time has elapsed, the old + observation is stale, so `pace()` lets the retry through and the response + refreshes the in-band budget. See [Budget & pacing](/budget). ## The honest limitation diff --git a/website/public/budget.md b/website/public/budget.md index 3144cf91..17e2d7ea 100644 --- a/website/public/budget.md +++ b/website/public/budget.md @@ -49,19 +49,20 @@ spending the budget twice to do the same work. ```python for batch in chunks(records, 200): - async with donkey.budget.pace(reserve=0.05): - await enrich(batch) + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached: + await donkey.budget.wait_for_reset() + continue + break checkpoint(batch) ``` -On `BudgetReserveReached`, wait for the window and carry on: - -```python -await donkey.budget.wait_for_reset() -continue -``` - -The job finishes unattended. Nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the +retry through so its response can refresh the in-band budget. The job finishes +unattended; nobody re-runs anything. ## The dashboard that prevents the outage @@ -90,6 +91,8 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. +- After `reset_at`, `pace()` lets the retry through so the response can refresh + the in-band budget; no manual observation is required. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 3b2973e7..b866a0a2 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -2152,19 +2152,20 @@ spending the budget twice to do the same work. ```python for batch in chunks(records, 200): - async with donkey.budget.pace(reserve=0.05): - await enrich(batch) + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached: + await donkey.budget.wait_for_reset() + continue + break checkpoint(batch) ``` -On `BudgetReserveReached`, wait for the window and carry on: - -```python -await donkey.budget.wait_for_reset() -continue -``` - -The job finishes unattended. Nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the +retry through so its response can refresh the in-band budget. The job finishes +unattended; nobody re-runs anything. ## The dashboard that prevents the outage @@ -2193,6 +2194,8 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. +- After `reset_at`, `pace()` lets the retry through so the response can refresh + the in-band budget; no manual observation is required. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. @@ -3273,7 +3276,9 @@ async def enrich_all(records, enrich): request that would cross the last 5% of the window — so you never spend the request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header - (milliseconds, converted for you). See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + (milliseconds, converted for you). Once that time has elapsed, the old + observation is stale, so `pace()` lets the retry through and the response + refreshes the in-band budget. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation diff --git a/website/public/scenarios/nightly-batch.md b/website/public/scenarios/nightly-batch.md index cbdf6fb0..dcf399f2 100644 --- a/website/public/scenarios/nightly-batch.md +++ b/website/public/scenarios/nightly-batch.md @@ -84,7 +84,9 @@ async def enrich_all(records, enrich): request that would cross the last 5% of the window — so you never spend the request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header - (milliseconds, converted for you). See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + (milliseconds, converted for you). Once that time has elapsed, the old + observation is stale, so `pace()` lets the retry through and the response + refreshes the in-band budget. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation From 1a3302d7747021ebb7072e490843bac0c64a925b Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 11:18:02 +0200 Subject: [PATCH 09/29] Fix AuthError consumer credential guidance (#478) Closes #451 --- python/src/donkey_kit/core/errors.py | 7 ++++--- python/tests/unit/test_llm_proxy_contract.py | 3 +++ 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/python/src/donkey_kit/core/errors.py b/python/src/donkey_kit/core/errors.py index 917beb7b..4e02de61 100644 --- a/python/src/donkey_kit/core/errors.py +++ b/python/src/donkey_kit/core/errors.py @@ -74,7 +74,7 @@ class ConfigError(DonkeyError): class AuthError(DonkeyError): - """401/403 on the control plane. + """Rejected credentials on the Anypoint control plane or LLM-proxy data plane. ``remediation`` is a class attribute (like :class:`GatewayUnavailable`, not the constructor-enforced :class:`PolicyViolation` contract) so ``donkey @@ -549,8 +549,9 @@ def classify( # wrong-credential 401 still lands here. if status == 401 or (status == 403 and "www-authenticate" in response.headers): return AuthError( - f"Authentication/authorization failed ({status}). Check the " - f"connected-app credentials and their scopes (see docs/verified-apis.md §1).", + f"Authentication/authorization failed ({status}). Check the consumer " + "client_id/client_secret pair and its API Manager authorization " + "for this LLM-proxy instance (see docs/verified-apis.md §2).", **kw, ) diff --git a/python/tests/unit/test_llm_proxy_contract.py b/python/tests/unit/test_llm_proxy_contract.py index 36f10005..a61bbb4c 100644 --- a/python/tests/unit/test_llm_proxy_contract.py +++ b/python/tests/unit/test_llm_proxy_contract.py @@ -74,6 +74,9 @@ def test_client_id_enforcement_rejection_classifies_as_auth() -> None: resp = httpx.Response(401, headers=h, json=body) err = classify(resp) assert isinstance(err, AuthError) + assert "consumer client_id/client_secret pair" in str(err) + assert "docs/verified-apis.md §2" in str(err) + assert "connected-app" not in str(err) def test_two_error_envelope_families_are_distinguishable() -> None: From 0d9a70f7d40bd670b70f72ebff5c40166bedcdc9 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 12:10:06 +0200 Subject: [PATCH 10/29] Distinguish shipped roadmap capabilities from proposals (#472) Closes #471 --- website/content/index.mdx | 11 +++++----- website/content/roadmap.mdx | 39 +++++++++++++++++++++++------------- website/public/index.md | 7 ++++--- website/public/llms-full.txt | 36 ++++++++++++++++++++++----------- website/public/roadmap.md | 31 +++++++++++++++++++--------- 5 files changed, 80 insertions(+), 44 deletions(-) diff --git a/website/content/index.mdx b/website/content/index.mdx index 9cbad254..220f8d84 100644 --- a/website/content/index.mdx +++ b/website/content/index.mdx @@ -154,8 +154,8 @@ simulation, and refusal handlers can all attach *without you wiring each one*. title="The six-piece minimum" href="/roadmap" icon="◈" - badge="Phase 1" - badgeTone="roadmap" + badge="Shipped" + badgeTone="accent" > Typed refusals, budget pacing, a local simulator, a conformance plugin, OTel GenAI spans, and correlation IDs — one release, because the skeleton @@ -184,9 +184,10 @@ simulation, and refusal handlers can all attach *without you wiring each one*. adapters are wired to that verified contract. -Everything else on this site carries a phase badge, and those pages say -plainly that their API shapes are proposals. Where a platform fact is not -confirmed, the SDK raises +Beyond the live-verified contract above, shipped SDK capability pages say what +is implemented and call out any remaining platform-verification caveats. +Planned Phase 2 and Phase 3 pages carry roadmap badges and describe proposal +API shapes. Where a platform fact is not confirmed, the SDK raises `NotImplementedError("blocked on verification: …")` rather than fabricate an endpoint, header, or class name — see [Verification policy](/concepts/verification) for the current ledger, and the [Roadmap](/roadmap) for what lands when. diff --git a/website/content/roadmap.mdx b/website/content/roadmap.mdx index e92db7a9..3bbc836a 100644 --- a/website/content/roadmap.mdx +++ b/website/content/roadmap.mdx @@ -7,17 +7,21 @@ import { Badge, Card, CardGrid } from '../components' # Roadmap -This page exists so you can tell, at a glance, what you can rely on **today** -from what is **designed but not yet shipped**. Every capability page on this -site carries the same badge you see here. +This page separates SDK behavior that is **shipped today**, platform contracts +that are **verified live**, and capabilities that are **designed but not yet +shipped**. - **How to read the badges.** Live means verified - against a real Anypoint sandbox and usable now. + **How to read the badges.** Live means the + platform-facing behavior is verified against a real Anypoint sandbox and + usable now. Shipped means the SDK capability is + implemented and available now; it does not imply that every platform-facing + contract it uses is live-verified. Those caveats remain explicit on the + capability page and in the [Verification policy](/concepts/verification). Phase 2 and friends mean *designed, with an agreed acceptance bar, but not yet shipped* — the API shapes on those pages are proposals and will move. Nothing here is described as working before it - does; see [Verification policy](/concepts/verification). + does. ## Why the SDK exists at all @@ -42,28 +46,34 @@ The bar for Phase 1 is deliberately concrete: a developer who tries the SDK for 15 minutes should find three things they cannot get from `base_url` plus headers, and one of them should save them from a production incident. +Four core rejection shapes are live-verified; the remaining typed refusal paths +are shipped with explicit verification caveats. Budget and pacing, the local +simulator, testing and conformance, telemetry and cost, and the CLI and +decorators are shipped SDK capabilities; their pages call out any remaining +platform-verification caveats. + A `403` from a PII policy is not an auth error, and a policy `429` must never be retried. Branch on governance outcomes instead of parsing bodies. - + Remaining token budget as a first-class object, so an overnight batch paces itself instead of dying at 2am. - + A local server that replays real rejection fixtures, so you can test your PII branch without a gateway. - + `simulate()` for in-process refusal injection, plus a pytest plugin you run against *your own* agent. - + OpenTelemetry GenAI spans, per-run correlation IDs, and validated cost-attribution tags. - + `donkey doctor` tells wrong credentials from wrong URL from model-not-allowed, instead of one opaque failure. @@ -143,7 +153,8 @@ platform's own capability reachable and typed, not to reproduce it: - A home-grown A2A protocol implementation — the official `a2a-sdk` is wrapped. - **API shapes on roadmap pages are proposals.** They exist so the design can - be argued about concretely, and they will change before they ship. What is - committed is the *behaviour* and the acceptance bar, not the signature. + **API shapes on planned Phase 2 and Phase 3 pages are proposals.** They exist + so the design can be argued about concretely, and they will change before + they ship. What is committed is the *behaviour* and the acceptance bar, not + the signature. diff --git a/website/public/index.md b/website/public/index.md index fe16393e..a7e05202 100644 --- a/website/public/index.md +++ b/website/public/index.md @@ -126,9 +126,10 @@ simulation, and refusal handlers can all attach *without you wiring each one*. token-budget `429`, upstream passthrough). The framework-free client and the adapters are wired to that verified contract. -Everything else on this site carries a phase badge, and those pages say -plainly that their API shapes are proposals. Where a platform fact is not -confirmed, the SDK raises +Beyond the live-verified contract above, shipped SDK capability pages say what +is implemented and call out any remaining platform-verification caveats. +Planned Phase 2 and Phase 3 pages carry roadmap badges and describe proposal +API shapes. Where a platform fact is not confirmed, the SDK raises `NotImplementedError("blocked on verification: …")` rather than fabricate an endpoint, header, or class name — see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) for the current ledger, and the [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) for what lands when. diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index b866a0a2..dbf067d6 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -136,9 +136,10 @@ simulation, and refusal handlers can all attach *without you wiring each one*. token-budget `429`, upstream passthrough). The framework-free client and the adapters are wired to that verified contract. -Everything else on this site carries a phase badge, and those pages say -plainly that their API shapes are proposals. Where a platform fact is not -confirmed, the SDK raises +Beyond the live-verified contract above, shipped SDK capability pages say what +is implemented and call out any remaining platform-verification caveats. +Planned Phase 2 and Phase 3 pages carry roadmap badges and describe proposal +API shapes. Where a platform fact is not confirmed, the SDK raises `NotImplementedError("blocked on verification: …")` rather than fabricate an endpoint, header, or class name — see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) for the current ledger, and the [Roadmap](https://donkey-development-kit.github.io/donkey-development-kit/roadmap.md) for what lands when. @@ -579,16 +580,20 @@ Source: https://donkey-development-kit.github.io/donkey-development-kit/roadmap. # Roadmap -This page exists so you can tell, at a glance, what you can rely on **today** -from what is **designed but not yet shipped**. Every capability page on this -site carries the same badge you see here. +This page separates SDK behavior that is **shipped today**, platform contracts +that are **verified live**, and capabilities that are **designed but not yet +shipped**. - **How to read the badges.** Live means verified - against a real Anypoint sandbox and usable now. + **How to read the badges.** Live means the + platform-facing behavior is verified against a real Anypoint sandbox and + usable now. Shipped means the SDK capability is + implemented and available now; it does not imply that every platform-facing + contract it uses is live-verified. Those caveats remain explicit on the + capability page and in the [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). Phase 2 and friends mean *designed, with an agreed acceptance bar, but not yet shipped* — the API shapes on those pages are proposals and will move. Nothing here is described as working before it - does; see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). + does. ## Why the SDK exists at all @@ -612,6 +617,12 @@ The bar for Phase 1 is deliberately concrete: a developer who tries the SDK for 15 minutes should find three things they cannot get from `base_url` plus headers, and one of them should save them from a production incident. +Four core rejection shapes are live-verified; the remaining typed refusal paths +are shipped with explicit verification caveats. Budget and pacing, the local +simulator, testing and conformance, telemetry and cost, and the CLI and +decorators are shipped SDK capabilities; their pages call out any remaining +platform-verification caveats. + A `403` from a PII policy is not an auth error, and a policy `429` must never be retried. Branch on governance outcomes instead of parsing bodies. @@ -706,9 +717,10 @@ platform's own capability reachable and typed, not to reproduce it: - The gateway inside your agent process. - A home-grown A2A protocol implementation — the official `a2a-sdk` is wrapped. - **API shapes on roadmap pages are proposals.** They exist so the design can - be argued about concretely, and they will change before they ship. What is - committed is the *behaviour* and the acceptance bar, not the signature. + **API shapes on planned Phase 2 and Phase 3 pages are proposals.** They exist + so the design can be argued about concretely, and they will change before + they ship. What is committed is the *behaviour* and the acceptance bar, not + the signature. --- diff --git a/website/public/roadmap.md b/website/public/roadmap.md index dc8a2f5a..47906920 100644 --- a/website/public/roadmap.md +++ b/website/public/roadmap.md @@ -1,15 +1,19 @@ # Roadmap -This page exists so you can tell, at a glance, what you can rely on **today** -from what is **designed but not yet shipped**. Every capability page on this -site carries the same badge you see here. - - **How to read the badges.** Live means verified - against a real Anypoint sandbox and usable now. +This page separates SDK behavior that is **shipped today**, platform contracts +that are **verified live**, and capabilities that are **designed but not yet +shipped**. + + **How to read the badges.** Live means the + platform-facing behavior is verified against a real Anypoint sandbox and + usable now. Shipped means the SDK capability is + implemented and available now; it does not imply that every platform-facing + contract it uses is live-verified. Those caveats remain explicit on the + capability page and in the [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). Phase 2 and friends mean *designed, with an agreed acceptance bar, but not yet shipped* — the API shapes on those pages are proposals and will move. Nothing here is described as working before it - does; see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). + does. ## Why the SDK exists at all @@ -33,6 +37,12 @@ The bar for Phase 1 is deliberately concrete: a developer who tries the SDK for 15 minutes should find three things they cannot get from `base_url` plus headers, and one of them should save them from a production incident. +Four core rejection shapes are live-verified; the remaining typed refusal paths +are shipped with explicit verification caveats. Budget and pacing, the local +simulator, testing and conformance, telemetry and cost, and the CLI and +decorators are shipped SDK capabilities; their pages call out any remaining +platform-verification caveats. + A `403` from a PII policy is not an auth error, and a policy `429` must never be retried. Branch on governance outcomes instead of parsing bodies. @@ -127,6 +137,7 @@ platform's own capability reachable and typed, not to reproduce it: - The gateway inside your agent process. - A home-grown A2A protocol implementation — the official `a2a-sdk` is wrapped. - **API shapes on roadmap pages are proposals.** They exist so the design can - be argued about concretely, and they will change before they ship. What is - committed is the *behaviour* and the acceptance bar, not the signature. + **API shapes on planned Phase 2 and Phase 3 pages are proposals.** They exist + so the design can be argued about concretely, and they will change before + they ship. What is committed is the *behaviour* and the acceptance bar, not + the signature. From b620fc3a5f2d2c3629c290f624ce4f205bbb40d3 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 13:06:02 +0200 Subject: [PATCH 11/29] Replace stale milestone citations outside provisioning (#490) Closes #486 --- python/src/donkey_kit/integrations/anthropic.py | 13 +++++++------ python/src/donkey_kit/registry/exchange.py | 4 ++-- python/src/donkey_kit/registry/governance.py | 2 +- python/src/donkey_kit/registry/publication.py | 6 +++--- python/src/donkey_kit/tools/session.py | 2 +- 5 files changed, 14 insertions(+), 13 deletions(-) diff --git a/python/src/donkey_kit/integrations/anthropic.py b/python/src/donkey_kit/integrations/anthropic.py index e769e36f..50d1bc60 100644 --- a/python/src/donkey_kit/integrations/anthropic.py +++ b/python/src/donkey_kit/integrations/anthropic.py @@ -13,10 +13,11 @@ UNVERIFIED DEPENDENCY (docs/verified-apis.md §8): the Omni Gateway LLM proxy is OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API -route** is an open M0 verification item (verification discipline). If it does not, this adapter's -requests will not reach a working upstream — override ``base_url`` via ``**kw`` -to point at a real Anthropic-native route once confirmed. The first ``client()`` -call emits a one-time :class:`~donkey_kit.core._verify.UnverifiedValueWarning`. +route** is an open Verification-milestone item (verification discipline). If it +does not, this adapter's requests will not reach a working upstream — override +``base_url`` via ``**kw`` to point at a real Anthropic-native route once confirmed. +The first ``client()`` call emits a one-time +:class:`~donkey_kit.core._verify.UnverifiedValueWarning`. Class names / kwargs UNVERIFIED — docs/verified-apis.md §8. """ @@ -48,8 +49,8 @@ def connection_kwargs(self) -> dict[str, Any]: warnings.warn( "The Omni Gateway LLM proxy is verified OpenAI-compatible; its " "Anthropic-native Messages API route is UNVERIFIED (docs/" - "verified-apis.md §8, an open M0 item). If the proxy does not " - "serve Anthropic's API, override base_url via **kw once a real " + "verified-apis.md §8, an open Verification-milestone item). If the proxy " + "does not serve Anthropic's API, override base_url via **kw once a real " "route is confirmed.", _verify.UnverifiedValueWarning, stacklevel=3, diff --git a/python/src/donkey_kit/registry/exchange.py b/python/src/donkey_kit/registry/exchange.py index 069bf348..2e1d62c5 100644 --- a/python/src/donkey_kit/registry/exchange.py +++ b/python/src/donkey_kit/registry/exchange.py @@ -4,7 +4,7 @@ Every method that requires a real Anypoint endpoint is gated behind a verification-blocked error (verification discipline, working instruction #2) rather than a fabricated request path. The N+1-avoiding index design is documented in -:meth:`warm` for the implementer who fills it in after M0. +:meth:`warm` for the implementer who fills it in after the Verification milestone. """ from __future__ import annotations @@ -74,7 +74,7 @@ async def explain( async def warm(self, *, environment: str | None = None) -> None: """Build the in-memory governance index at startup. - Design (fill in after M0): + Design (fill in after the Verification milestone): 1. ONE call to list all API Manager instances for (org, environment); index by (groupId, assetId, version) AND (groupId, assetId). 2. Bulk policies call if one exists (verify the Verification milestone), else diff --git a/python/src/donkey_kit/registry/governance.py b/python/src/donkey_kit/registry/governance.py index 77eea778..89d20b40 100644 --- a/python/src/donkey_kit/registry/governance.py +++ b/python/src/donkey_kit/registry/governance.py @@ -4,7 +4,7 @@ systems, environment-scoped. The criteria and report types here are pure and fully implemented. The JOIN that populates a report from live API Manager / ruleset state lives in :mod:`donkey_kit.registry.exchange` and is -gated on M0 verification (the Verification milestone). +gated on the Verification milestone. """ from __future__ import annotations diff --git a/python/src/donkey_kit/registry/publication.py b/python/src/donkey_kit/registry/publication.py index da856a65..83e3753b 100644 --- a/python/src/donkey_kit/registry/publication.py +++ b/python/src/donkey_kit/registry/publication.py @@ -81,10 +81,10 @@ async def preview(self, donkey: Any) -> str: # ---- verb: export (laptop) -------------------------------------------- def export(self, path: str | Path | None = None) -> str: - """Compile into the donkey.yaml spec (provisioning-as-code). Lands with M4 (the build plan - phases).""" + """Compile to a ``donkey.yaml`` fragment (provisioning-as-code).""" raise _verify.blocked( - "Publication.export() emits the M4 spec (provisioning-as-code, the build plan phases)." + "Publication.export() emits the donkey.yaml spec format. Schema is shared with " + "provisioning-as-code — do not diverge it (provisioning-as-code)." ) # ---- verb: verify (runtime, READ-ONLY) -------------------------------- diff --git a/python/src/donkey_kit/tools/session.py b/python/src/donkey_kit/tools/session.py index 4fcb7d39..53ab1c4d 100644 --- a/python/src/donkey_kit/tools/session.py +++ b/python/src/donkey_kit/tools/session.py @@ -2,7 +2,7 @@ ``ToolSet`` wraps N ``McpServerHandle``s. Filtering and collision resolution are pure and implemented here. The per-framework binding methods return each -framework's NATIVE tool type (BG §2.7) and are gated on M0 verification of the MCP +framework's NATIVE tool type (BG §2.7) and are gated on verification of the MCP binding class names (docs/verified-apis.md §9) — connections open on first tool use, never in ``discover()`` (BG §2.7). """ From c431b2e531772a40cb734a58375bbbd7df88d950 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 13:23:45 +0200 Subject: [PATCH 12/29] Remove provisioning-as-code from the README headline (#481) Closes #459 --- README.md | 5 +- python/README.md | 5 +- python/pyproject.toml | 2 +- python/tests/unit/test_product_description.py | 49 +++++++++++++++++++ 4 files changed, 54 insertions(+), 7 deletions(-) create mode 100644 python/tests/unit/test_product_description.py diff --git a/README.md b/README.md index 500ff4e2..a5f410f7 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,8 @@ # Donkey Development Kit -An SDK for consuming **Agent Fabric** capabilities — governed model access, -governed tool access, and provisioning-as-code — from your own agent framework, -in your own IDE, without adopting Mule. +An SDK for consuming **Agent Fabric** capabilities — governed model and tool +access — from your own agent framework, in your own IDE, without adopting Mule. > **Project status — alpha, pre-release.** This is `v0.1.0.dev2` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; diff --git a/python/README.md b/python/README.md index f99f6677..db4c42b3 100644 --- a/python/README.md +++ b/python/README.md @@ -4,9 +4,8 @@ # Donkey Development Kit -An SDK for consuming **Agent Fabric** capabilities — governed model access, -governed tool access, and provisioning-as-code — from your own agent framework, -in your own IDE, without adopting Mule. +An SDK for consuming **Agent Fabric** capabilities — governed model and tool +access — from your own agent framework, in your own IDE, without adopting Mule. > **Project status — alpha, pre-release.** This is `v0.1.0.dev2` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; diff --git a/python/pyproject.toml b/python/pyproject.toml index c3f65ae0..7eed1943 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "hatchling.build" [project] name = "donkey-kit" version = "0.1.0.dev2" -description = "An SDK for consuming Agent Fabric capabilities (governed model access, governed tools, provisioning-as-code) from your own agent framework." +description = "An SDK for consuming Agent Fabric capabilities (governed model and tool access) from your own agent framework." readme = "README.md" requires-python = ">=3.10" license = { text = "Apache-2.0" } diff --git a/python/tests/unit/test_product_description.py b/python/tests/unit/test_product_description.py new file mode 100644 index 00000000..33d70c83 --- /dev/null +++ b/python/tests/unit/test_product_description.py @@ -0,0 +1,49 @@ +"""Product descriptions stay aligned with the provisioning boundary (#459).""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +if sys.version_info >= (3, 11): + import tomllib +else: # pragma: no cover - 3.10 backfill + import tomli as tomllib + +_PYTHON_ROOT = Path(__file__).resolve().parents[2] +_REPOSITORY_README = _PYTHON_ROOT.parent / "README.md" + + +def _opening_description(readme: Path) -> str: + return readme.read_text().split("# Donkey Development Kit", maxsplit=1)[1].split( + "> **Project status", maxsplit=1 + )[0] + + +@pytest.mark.parametrize( + "readme", + [ + pytest.param(_PYTHON_ROOT / "README.md", id="package-readme"), + pytest.param( + _REPOSITORY_README, + id="repository-readme", + marks=pytest.mark.skipif( + _PYTHON_ROOT.name != "python" or not _REPOSITORY_README.is_file(), + reason="repository README is not included in the sdist", + ), + ), + ], +) +def test_readme_headline_does_not_promise_provisioning_as_code(readme: Path) -> None: + description = " ".join(_opening_description(readme).split()) + assert "governed model and tool access" in description + assert "governed tool access, and provisioning-as-code" not in description + + +def test_package_description_does_not_promise_provisioning_as_code() -> None: + metadata = tomllib.loads((_PYTHON_ROOT / "pyproject.toml").read_text()) + description = metadata["project"]["description"] + assert "governed model and tool access" in description + assert "governed tools, provisioning-as-code" not in description From bdd71d9b767ae81c104ebb1f29333fff571f5362 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 14:32:01 +0200 Subject: [PATCH 13/29] Populate the platform support boundary ledger (#473) Closes #465 --- docs/unsupported-boundary.md | 32 +++++++++++++- docs/verified-apis.md | 20 +++++---- .../reference/unsupported-boundary.mdx | 43 ++++++++++++++++--- website/public/llms-full.txt | 41 +++++++++++++++--- .../public/reference/unsupported-boundary.md | 41 +++++++++++++++--- 5 files changed, 149 insertions(+), 28 deletions(-) diff --git a/docs/unsupported-boundary.md b/docs/unsupported-boundary.md index f48de37f..dba39fb3 100644 --- a/docs/unsupported-boundary.md +++ b/docs/unsupported-boundary.md @@ -4,6 +4,13 @@ Every platform API the SDK calls, classified. Enterprise buyers will ask; having this pre-written converts a two-week procurement stall into a five-minute conversation. Linked from the README above the fold. +This is a **support-classification ledger**, not the verification ledger. +[`verified-apis.md`](verified-apis.md) records how each fact was established +(`VERIFIED (LIVE)`, `VERIFIED (plugin)`, and so on). This file answers a +different question: whether the shipping SDK depends on a public MuleSoft +contract, a live-observed extension with no third-party compatibility SLA, or +an undocumented surface. + Classification: - **Documented and public** — safe. @@ -13,7 +20,30 @@ Classification: | API / surface | Module | Classification | Justification / owner | |---|---|---|---| -| _(populate during M0/verification discipline as each surface is verified)_ | | | | +| Connected-app bearer-token request: `POST /accounts/api/v2/oauth2/token` | `core.auth.AnypointConnectedApp` | **Documented and public** | MuleSoft documents the client-credentials request and response in [Getting the Bearer Token for a Connected App](https://docs.mulesoft.com/access-management/connected-app-bearer-token-example). The SDK path is also `VERIFIED (plugin)` in `verified-apis.md` §1 and §12.1. | +| Model Proxy OpenAI-format data plane: caller-configured base path plus `POST /responses`, including streaming and the OpenAI `usage` object | `llm.client.LLMClient`; `core.lastcall.LastCall` | **Documented and public** | MuleSoft documents the endpoint, `client_id` / `client_secret` headers, request bodies, and streaming in [Sending Requests to Model Proxies](https://docs.mulesoft.com/general/model-proxy-request). The SDK's `/responses` path is `VERIFIED (LIVE)` in `verified-apis.md` §2. The raw `AsyncOpenAI` client can also call the documented `/chat/completions` route, but this repository has not live-probed that route. | +| Client ID Enforcement request headers and `401` rejection contract | `core.transport.proxy_auth_headers`; `core.errors.classify` | **Documented and public** | The [Client ID Enforcement policy](https://docs.mulesoft.com/gateway/latest/policies-included-client-id-enforcement) documents the default header expressions and `WWW-Authenticate: Client-ID-Enforcement`. The configured proxy contract is `VERIFIED (LIVE)` in `verified-apis.md` §2 and §4. | +| LLM Token Based Rate Limit `429` response: `x-token-limit`, `x-token-remaining`, and `x-token-reset` | `core.budget.Budget`; `core.errors.classify` | **Documented and public** | The [LLM Token Based Rate Limit policy](https://docs.mulesoft.com/gateway/latest/policies-included-llm-token-rate-limit) documents all three headers and the reset unit. The SDK's rejection fixture is `VERIFIED (LIVE)` in `verified-apis.md` §4. | +| LLM PII Detection `403` rejection body (`error.type = "pii_detected"`) | `core.errors.classify` | **Documented and public** | The [LLM PII Detection policy](https://docs.mulesoft.com/gateway/latest/policies-included-llm-pii-detection) documents the rejection body. The SDK also captured it live; see `verified-apis.md` §4. | +| Regex Prompt Guard body and Injection Protection response header | `core.errors.classify` | **Documented and public** | MuleSoft documents `matched_patterns` in [Regex Prompt Guard](https://docs.mulesoft.com/gateway/latest/policies-included-regex-prompt-guard) and `x-injection-protection: blocked` in [Injection Protection](https://docs.mulesoft.com/gateway/latest/policies-included-injection-protection). These discriminators are documented but still pending direct live capture; see `verified-apis.md` §4. | +| Azure Content Safety and Amazon Bedrock Guardrails rejection headers and bodies | `core.errors.classify` | **Documented and public** | The [Azure Content Safety](https://docs.mulesoft.com/gateway/latest/policies-included-azure-content-safety) and [Amazon Bedrock Guardrails](https://docs.mulesoft.com/gateway/latest/policies-included-bedrock-guardrails) policy pages document their action/reason headers and `403` bodies. These shapes are pending direct live capture; see `verified-apis.md` §4. | +| Upstream provider error pass-through: non-`429` `4xx` nested error envelope; generic `5xx` status family | `core.errors.classify` | **Documented but no SLA for third-party use** | A real Model Proxy passed an OpenAI `400 model_not_found` envelope through verbatim, including `code`, `type`, and `param` (`VERIFIED (LIVE)`, `verified-apis.md` §4). MuleSoft's public Model Proxy page does not state a compatibility contract for provider error envelopes, whose schema ultimately belongs to the upstream provider. The SDK maps any generic `5xx` to `UpstreamModelError` by status only and assumes no body shape. Owner: DDK maintainers; retain the captured provider fixture and classify changed envelopes conservatively. | +| Successful-response budget window in `x-llm-proxy-ratelimit` prose | `core.budget.Budget` | **Documented but no SLA for third-party use** | Captured from a real Model Proxy and parsed fail-open: an absent or changed sentence is ignored, never fatal. The exact prose contract is `VERIFIED (LIVE)` in `verified-apis.md` §4 but is not stated in the public policy page. Owner: DDK maintainers; re-capture on gateway-policy changes. | +| Gateway identity and routing response metadata: `x-request-id`, `x-envoy-decorator-operation`, and `x-llm-proxy-{routing-type,routing-fallback,llm-provider,llm-model}` | `core.lastcall.LastCall`; `core.transport.DonkeyAsyncClient` | **Documented but no SLA for third-party use** | Captured from a real Model Proxy (`verified-apis.md` §2 and §3). Parsers are fail-open: missing or unrecognised values become `None`, and only an explicit `routing-fallback: true` changes retry behavior. Owner: DDK maintainers; contract changes require new fixtures before parser changes. | + +## Deliberately outside the shipping boundary + +- Exchange search and asset resolution, API Manager governed-state reads, MCP + discovery/binding, and provisioning/publication raise + `NotImplementedError("blocked on verification: ...")` before network I/O. + They are therefore not called platform APIs and are not classified above. +- `LLMClient.list_models(live=True)` does not call `GET /models`; live capture + established that Model Proxy has no model-catalog endpoint. +- Correlation, per-call, application/business-group attribution, and cost-tag + request-header names remain warning-emitting, overridable `Unverified` + placeholders (`verified-apis.md` §3). The SDK makes no claim that the gateway + reads them and does not depend on it doing so, so they are not represented as + supported platform contracts here. ## Undocumented surfaces diff --git a/docs/verified-apis.md b/docs/verified-apis.md index bc5056ee..cd6dfc6b 100644 --- a/docs/verified-apis.md +++ b/docs/verified-apis.md @@ -220,7 +220,8 @@ message); a top-level `matched_patterns` list → `PromptInjectionBlocked` header (Azure Content Safety / Amazon Bedrock Guardrails) → `ContentSafetyBlocked` (parses `categories` from the sibling `…-reason` header) — both of these also checked *before* the 401/403→auth rule so a `403` moderation block is not -mis-typed as auth; header `x-injection-protection: blocked` → +mis-typed as auth; header `x-injection-protection: blocked` (documented by the +[Injection Protection policy](https://docs.mulesoft.com/gateway/latest/policies-included-injection-protection)) → `PromptInjectionBlocked` (the header, not the status, is the discriminator, so a bare `400` is unaffected); `429` → `TokenBudgetExceeded` with `retry_after` derived from `x-token-reset` (ms→s); non-auth 4xx with a nested `error` object → @@ -233,14 +234,15 @@ whose message names the observed status and any `x-llm-proxy-*` policy headers a states the **shape is unconfirmed** (#184), rather than being mis-typed. The full eight-shape taxonomy is indexed in `tests/fixtures/rejections/README.md`. -Two of those shapes are **DOCUMENTED, pending live capture (#253)** — typed from -the policy pages, not from a live sandbox round-trip, so **no `_verify.py` row -flips to `verified=True`** for them: +The Regex Prompt Guard and content-safety/guardrail shapes below are +**DOCUMENTED, pending live capture (#253)** — typed from the policy pages, not +from a live sandbox round-trip, so **no `_verify.py` row flips to +`verified=True`** for them: | Shape | HTTP | Discriminator | Maps to | Source | |---|---|---|---|---| -| Regex Prompt Guard | `403` | top-level `matched_patterns` list (flat-string `error`) | `PromptInjectionBlocked` (`policy="regex-prompt-guard"`) | DOCUMENTED, pending live capture (#253) | -| Content safety / guardrails | `403` | `x-llm-proxy-azure-content-safety-action` / `x-llm-proxy-bedrock-guardrail-action` == `reject`; reasons in the sibling `…-reason` header | `ContentSafetyBlocked` (parses `categories`) | DOCUMENTED, pending live capture (#253) | +| Regex Prompt Guard | `403` | top-level `matched_patterns` list (flat-string `error`) | `PromptInjectionBlocked` (`policy="regex-prompt-guard"`) | [Regex Prompt Guard policy](https://docs.mulesoft.com/gateway/latest/policies-included-regex-prompt-guard); DOCUMENTED, pending live capture (#253) | +| Content safety / guardrails | `403` | `x-llm-proxy-azure-content-safety-action` / `x-llm-proxy-bedrock-guardrail-action` == `reject`; reasons in the sibling `…-reason` header | `ContentSafetyBlocked` (parses `categories`) | [Azure Content Safety policy](https://docs.mulesoft.com/gateway/latest/policies-included-azure-content-safety); [Amazon Bedrock Guardrails policy](https://docs.mulesoft.com/gateway/latest/policies-included-bedrock-guardrails); DOCUMENTED, pending live capture (#253) | | Unrecognised refusal (fall-through) | any non-429 `4xx` | matches **none** of the discriminators above; no nested `error` envelope; no `www-authenticate` (e.g. an unrecognised `403`) | generic `PolicyViolation` (`policy="unknown"`), message says **shape unconfirmed** and names the observed status + `x-llm-proxy-*` headers | UNVERIFIED — no known contract; capture + type via #184/#253 | The **injection body** and any **other content-moderation / federated-guardrail** @@ -249,9 +251,9 @@ any such unrecognised refusal **honestly** — a `PolicyViolation` whose message states the shape is unconfirmed and whose remediation asks the operator to file the observed status/headers/body so the shape can be typed (#184). This is the last row above; it is deliberately **not** an `AuthError`, even for a `403`, once -the verified `www-authenticate` auth shape is excluded. Re-confirming all of these -against current docs and a sandbox is tracked in #253 (verification discipline: no invented docs URL -or version is recorded for them). +the verified `www-authenticate` auth shape is excluded. The official policy +pages above establish the documented contracts; direct sandbox capture remains +tracked in #253 and is still required before any verification status changes. **Budget window emission — two forms, keyed on outcome not status class (LIVE-VERIFIED).** The gateway signals its token-rate-limit window in two diff --git a/website/content/reference/unsupported-boundary.mdx b/website/content/reference/unsupported-boundary.mdx index 31f90d2d..c62ad338 100644 --- a/website/content/reference/unsupported-boundary.mdx +++ b/website/content/reference/unsupported-boundary.mdx @@ -8,7 +8,11 @@ stall into a short conversation. The authoritative, maintained list lives in the repository at [`docs/unsupported-boundary.md`](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/docs/unsupported-boundary.md). -Every platform API the SDK calls is classified: +It is separate from the [verification ledger](/concepts/verification): verification +records how a fact was established, while this page records whether MuleSoft +publishes the contract for third-party use. + +Every platform API the shipping SDK calls is classified: | Classification | Meaning | |---|---| @@ -16,14 +20,41 @@ Every platform API the SDK calls is classified: | **Documented, no SLA for third-party use** | May break; we'll fix. | | **Undocumented** | Should be empty. Anything here needs a written justification and an owner. | +## Current shipping boundary + +The SDK currently reaches two platform destinations. The rows below classify +the distinct contracts it consumes at those destinations. Everything else that +needs an unverified endpoint is blocked before network I/O. + +| Destination / contract | Classification | SDK use | +|---|---|---| +| Anypoint connected-app token endpoint | **Documented and public** | Retrieves an OAuth bearer token with client credentials. | +| Model Proxy OpenAI-format `/responses` endpoint | **Documented and public** | Sends buffered or streaming model requests with the documented `client_id` / `client_secret` headers and reads OpenAI-format usage. The raw client can also call documented OpenAI-native routes such as `/chat/completions`, but this repository has live-probed only `/responses`. | +| Live-captured Model Proxy policy refusals | **Documented and public** | Classifies Client ID Enforcement, token-rate-limit, and PII responses observed against a deployed proxy. | +| Documentation-derived Model Proxy policy refusals | **Documented and public** | Classifies Regex Prompt Guard, Injection Protection, Azure Content Safety, and Amazon Bedrock Guardrails contracts from their official policy pages. These shapes are pending direct live capture. | +| Upstream provider error pass-through | **Documented, no SLA for third-party use** | Classifies the live-captured nested non-`429` `4xx` provider envelope as `UpstreamRequestError`; generic `5xx` responses become `UpstreamModelError` by status only. The envelope schema belongs to the upstream provider, and MuleSoft's public Model Proxy page states no pass-through compatibility contract. | +| `x-llm-proxy-ratelimit` success-budget sentence | **Documented, no SLA for third-party use** | Updates `donkey.budget`; an absent or changed value is ignored. | +| Gateway identity and routing extension headers | **Documented, no SLA for third-party use** | Populates `donkey.last_call`; missing or unrecognised values become `None`. | + + + Exchange search and resolution, API Manager governed-state reads, MCP discovery + and binding, and provisioning/publication are not hidden dependencies. They + raise `NotImplementedError("blocked on verification: ...")` before making a + network request. The SDK also does not call a Model Proxy `/models` endpoint, + because live verification established that no such catalog endpoint exists. + + +The full ledger links each contract to its official documentation, SDK consumer, +verification evidence, and maintenance owner. Its **Undocumented surfaces** +section is empty. + ## Support statement (the trademark/support boundary) - **Before any public release**, the maintainer and support expectations must be - stated (README, the trademark/support boundary). If this project is **not** published with MuleSoft's - endorsement, it must ship under a distinct, org-scoped distribution name so it - doesn't read as a first-party, official-status SDK. "Agent Fabric", - "Anypoint", and "Omni Gateway" are Salesforce trademarks. + Donkey Development Kit is an independent, community-maintained project with + best-effort maintainer support and no SLA. It is not affiliated with, endorsed + by, or supported by Salesforce or MuleSoft. "Agent Fabric", "Anypoint", and + "Omni Gateway" are Salesforce trademarks. ## Why this matters (verification discipline) diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index dbf067d6..7e852769 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -4964,7 +4964,11 @@ stall into a short conversation. The authoritative, maintained list lives in the repository at [`docs/unsupported-boundary.md`](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/docs/unsupported-boundary.md). -Every platform API the SDK calls is classified: +It is separate from the [verification ledger](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md): verification +records how a fact was established, while this page records whether MuleSoft +publishes the contract for third-party use. + +Every platform API the shipping SDK calls is classified: | Classification | Meaning | |---|---| @@ -4972,13 +4976,38 @@ Every platform API the SDK calls is classified: | **Documented, no SLA for third-party use** | May break; we'll fix. | | **Undocumented** | Should be empty. Anything here needs a written justification and an owner. | +## Current shipping boundary + +The SDK currently reaches two platform destinations. The rows below classify +the distinct contracts it consumes at those destinations. Everything else that +needs an unverified endpoint is blocked before network I/O. + +| Destination / contract | Classification | SDK use | +|---|---|---| +| Anypoint connected-app token endpoint | **Documented and public** | Retrieves an OAuth bearer token with client credentials. | +| Model Proxy OpenAI-format `/responses` endpoint | **Documented and public** | Sends buffered or streaming model requests with the documented `client_id` / `client_secret` headers and reads OpenAI-format usage. The raw client can also call documented OpenAI-native routes such as `/chat/completions`, but this repository has live-probed only `/responses`. | +| Live-captured Model Proxy policy refusals | **Documented and public** | Classifies Client ID Enforcement, token-rate-limit, and PII responses observed against a deployed proxy. | +| Documentation-derived Model Proxy policy refusals | **Documented and public** | Classifies Regex Prompt Guard, Injection Protection, Azure Content Safety, and Amazon Bedrock Guardrails contracts from their official policy pages. These shapes are pending direct live capture. | +| Upstream provider error pass-through | **Documented, no SLA for third-party use** | Classifies the live-captured nested non-`429` `4xx` provider envelope as `UpstreamRequestError`; generic `5xx` responses become `UpstreamModelError` by status only. The envelope schema belongs to the upstream provider, and MuleSoft's public Model Proxy page states no pass-through compatibility contract. | +| `x-llm-proxy-ratelimit` success-budget sentence | **Documented, no SLA for third-party use** | Updates `donkey.budget`; an absent or changed value is ignored. | +| Gateway identity and routing extension headers | **Documented, no SLA for third-party use** | Populates `donkey.last_call`; missing or unrecognised values become `None`. | + + Exchange search and resolution, API Manager governed-state reads, MCP discovery + and binding, and provisioning/publication are not hidden dependencies. They + raise `NotImplementedError("blocked on verification: ...")` before making a + network request. The SDK also does not call a Model Proxy `/models` endpoint, + because live verification established that no such catalog endpoint exists. + +The full ledger links each contract to its official documentation, SDK consumer, +verification evidence, and maintenance owner. Its **Undocumented surfaces** +section is empty. + ## Support statement (the trademark/support boundary) - **Before any public release**, the maintainer and support expectations must be - stated (README, the trademark/support boundary). If this project is **not** published with MuleSoft's - endorsement, it must ship under a distinct, org-scoped distribution name so it - doesn't read as a first-party, official-status SDK. "Agent Fabric", - "Anypoint", and "Omni Gateway" are Salesforce trademarks. + Donkey Development Kit is an independent, community-maintained project with + best-effort maintainer support and no SLA. It is not affiliated with, endorsed + by, or supported by Salesforce or MuleSoft. "Agent Fabric", "Anypoint", and + "Omni Gateway" are Salesforce trademarks. ## Why this matters (verification discipline) diff --git a/website/public/reference/unsupported-boundary.md b/website/public/reference/unsupported-boundary.md index f48c8e7b..d08e0d64 100644 --- a/website/public/reference/unsupported-boundary.md +++ b/website/public/reference/unsupported-boundary.md @@ -6,7 +6,11 @@ stall into a short conversation. The authoritative, maintained list lives in the repository at [`docs/unsupported-boundary.md`](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/docs/unsupported-boundary.md). -Every platform API the SDK calls is classified: +It is separate from the [verification ledger](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md): verification +records how a fact was established, while this page records whether MuleSoft +publishes the contract for third-party use. + +Every platform API the shipping SDK calls is classified: | Classification | Meaning | |---|---| @@ -14,13 +18,38 @@ Every platform API the SDK calls is classified: | **Documented, no SLA for third-party use** | May break; we'll fix. | | **Undocumented** | Should be empty. Anything here needs a written justification and an owner. | +## Current shipping boundary + +The SDK currently reaches two platform destinations. The rows below classify +the distinct contracts it consumes at those destinations. Everything else that +needs an unverified endpoint is blocked before network I/O. + +| Destination / contract | Classification | SDK use | +|---|---|---| +| Anypoint connected-app token endpoint | **Documented and public** | Retrieves an OAuth bearer token with client credentials. | +| Model Proxy OpenAI-format `/responses` endpoint | **Documented and public** | Sends buffered or streaming model requests with the documented `client_id` / `client_secret` headers and reads OpenAI-format usage. The raw client can also call documented OpenAI-native routes such as `/chat/completions`, but this repository has live-probed only `/responses`. | +| Live-captured Model Proxy policy refusals | **Documented and public** | Classifies Client ID Enforcement, token-rate-limit, and PII responses observed against a deployed proxy. | +| Documentation-derived Model Proxy policy refusals | **Documented and public** | Classifies Regex Prompt Guard, Injection Protection, Azure Content Safety, and Amazon Bedrock Guardrails contracts from their official policy pages. These shapes are pending direct live capture. | +| Upstream provider error pass-through | **Documented, no SLA for third-party use** | Classifies the live-captured nested non-`429` `4xx` provider envelope as `UpstreamRequestError`; generic `5xx` responses become `UpstreamModelError` by status only. The envelope schema belongs to the upstream provider, and MuleSoft's public Model Proxy page states no pass-through compatibility contract. | +| `x-llm-proxy-ratelimit` success-budget sentence | **Documented, no SLA for third-party use** | Updates `donkey.budget`; an absent or changed value is ignored. | +| Gateway identity and routing extension headers | **Documented, no SLA for third-party use** | Populates `donkey.last_call`; missing or unrecognised values become `None`. | + + Exchange search and resolution, API Manager governed-state reads, MCP discovery + and binding, and provisioning/publication are not hidden dependencies. They + raise `NotImplementedError("blocked on verification: ...")` before making a + network request. The SDK also does not call a Model Proxy `/models` endpoint, + because live verification established that no such catalog endpoint exists. + +The full ledger links each contract to its official documentation, SDK consumer, +verification evidence, and maintenance owner. Its **Undocumented surfaces** +section is empty. + ## Support statement (the trademark/support boundary) - **Before any public release**, the maintainer and support expectations must be - stated (README, the trademark/support boundary). If this project is **not** published with MuleSoft's - endorsement, it must ship under a distinct, org-scoped distribution name so it - doesn't read as a first-party, official-status SDK. "Agent Fabric", - "Anypoint", and "Omni Gateway" are Salesforce trademarks. + Donkey Development Kit is an independent, community-maintained project with + best-effort maintainer support and no SLA. It is not affiliated with, endorsed + by, or supported by Salesforce or MuleSoft. "Agent Fabric", "Anypoint", and + "Omni Gateway" are Salesforce trademarks. ## Why this matters (verification discipline) From be4673766b25f0a81255dc560f7069f7cda8b4a3 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 14:36:14 +0200 Subject: [PATCH 14/29] Test connected-app OAuth token flow (#493) Closes #449 --- python/tests/unit/test_auth.py | 145 +++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 python/tests/unit/test_auth.py diff --git a/python/tests/unit/test_auth.py b/python/tests/unit/test_auth.py new file mode 100644 index 00000000..b1f7476f --- /dev/null +++ b/python/tests/unit/test_auth.py @@ -0,0 +1,145 @@ +"""Connected-app OAuth token acquisition and caching (BG §1.1).""" + +from __future__ import annotations + +from urllib.parse import parse_qs + +import httpx +import pytest + +from donkey_kit.core.auth import AnypointConnectedApp +from donkey_kit.core.errors import AuthError + + +async def test_connected_app_posts_credentials_and_caches_until_safety_margin() -> None: + now = [100.0] + requests: list[httpx.Request] = [] + + def handler(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, + json={"access_token": f"token-{len(requests)}", "expires_in": 120}, + ) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + auth = AnypointConnectedApp( + client_id="client-id", + client_secret="client-secret", + control_plane_url="https://anypoint.example/", + http_client=http_client, + token_path="/oauth/token", + clock=lambda: now[0], + ) + + assert await auth.token() == "token-1" + now[0] = 159.0 + assert await auth.token() == "token-1" + now[0] = 160.0 + assert await auth.token() == "token-2" + + assert len(requests) == 2 + request = requests[0] + assert request.method == "POST" + assert request.url == httpx.URL("https://anypoint.example/oauth/token") + assert request.headers["accept"] == "application/json" + assert request.headers["content-type"] == "application/x-www-form-urlencoded" + assert parse_qs(request.content.decode()) == { + "grant_type": ["client_credentials"], + "client_id": ["client-id"], + "client_secret": ["client-secret"], + } + + +async def test_connected_app_invalidate_forces_refetch() -> None: + calls = 0 + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal calls + calls += 1 + return httpx.Response(200, json={"access_token": f"token-{calls}"}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + auth = AnypointConnectedApp( + client_id="client-id", + client_secret="client-secret", + control_plane_url="https://anypoint.example", + http_client=http_client, + token_path="/oauth/token", + ) + + assert await auth.token() == "token-1" + await auth.invalidate() + assert await auth.token() == "token-2" + + assert calls == 2 + + +@pytest.mark.parametrize("status_code", [401, 403]) +async def test_connected_app_maps_rejected_credentials_to_auth_error(status_code: int) -> None: + response: httpx.Response | None = None + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal response + response = httpx.Response(status_code) + return response + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + auth = AnypointConnectedApp( + client_id="client-id", + client_secret="client-secret", + control_plane_url="https://anypoint.example", + http_client=http_client, + token_path="/oauth/token", + ) + + with pytest.raises(AuthError, match="token request rejected") as exc_info: + await auth.token() + + assert exc_info.value.response is response + + +async def test_connected_app_maps_other_http_errors_to_auth_error() -> None: + response: httpx.Response | None = None + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal response + response = httpx.Response(500) + return response + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + auth = AnypointConnectedApp( + client_id="client-id", + client_secret="client-secret", + control_plane_url="https://anypoint.example", + http_client=http_client, + token_path="/oauth/token", + ) + + with pytest.raises(AuthError, match="token endpoint returned 500") as exc_info: + await auth.token() + + assert exc_info.value.response is response + + +async def test_connected_app_rejects_success_response_without_access_token() -> None: + response: httpx.Response | None = None + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal response + response = httpx.Response(200, json={"expires_in": 3600}) + return response + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + auth = AnypointConnectedApp( + client_id="client-id", + client_secret="client-secret", + control_plane_url="https://anypoint.example", + http_client=http_client, + token_path="/oauth/token", + ) + + with pytest.raises(AuthError, match="returned no access_token") as exc_info: + await auth.token() + + assert exc_info.value.response is response From 07e888f62243e0c32c8b048a1bb5f252957b4cd6 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 14:37:25 +0200 Subject: [PATCH 15/29] Demote legacy provisioning scaffolding from public API (#474) Closes #458 --- ARCHITECTURE.md | 7 ++++--- python/src/donkey_kit/__init__.py | 10 ---------- python/src/donkey_kit/provisioning/cli.py | 2 +- python/tests/unit/test_cli_help.py | 17 +++++++++++++++++ python/tests/unit/test_donkey_surface.py | 16 ++++++++++++++++ website/content/concepts/environments.mdx | 2 +- website/public/concepts/environments.md | 2 +- website/public/llms-full.txt | 2 +- 8 files changed, 41 insertions(+), 17 deletions(-) create mode 100644 python/tests/unit/test_cli_help.py diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 23ecfe4c..d83d537c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -114,9 +114,10 @@ framework that may not be installed. — span consumers must close in a `finally`, never relying on the response hook. A hookless client behaves exactly as it did before the hooks were added. The full contracts live in the `core/transport.py` docstrings. -- **`Governance`** (`governance.py`) is a second top-level object alongside - `Donkey`, outside the linear import stack — it depends only on `core`. It is - ONE object behind three verbs: `simulate()` (an ephemeral local +- **`Governance`** (`governance.py`) is legacy scaffolding outside the linear + import stack — it depends only on `core` and remains reachable from its module, + but it is not exported as first-class `donkey_kit` API. It is ONE object behind + three verbs: `simulate()` (an ephemeral local gateway harness), `export()` (emit the governed-state manifest), and `resolve()` (reconcile a running `Donkey` against it, raising `GovernanceDrift` on mismatch); a separate platform-team-only `apply()` is the deliberate escape diff --git a/python/src/donkey_kit/__init__.py b/python/src/donkey_kit/__init__.py index 94dda37c..b28cc40e 100644 --- a/python/src/donkey_kit/__init__.py +++ b/python/src/donkey_kit/__init__.py @@ -42,12 +42,6 @@ ) from .core.toolspec import ToolSpec, registered_tools from .donkey import Donkey -from .governance import ( - GatewayTarget, - Governance, - PolicyBinding, - PolicyPortability, -) from .registry import ( STRICT, AssetRef, @@ -74,15 +68,11 @@ "Donkey", "DonkeyConfig", "DonkeyError", - "GatewayTarget", "GatewayUnavailable", - "Governance", "GovernanceCriteria", "GovernanceDrift", "ModelSubstituted", "PIIDetected", - "PolicyBinding", - "PolicyPortability", "PolicyViolation", "PromptInjectionBlocked", "Publication", diff --git a/python/src/donkey_kit/provisioning/cli.py b/python/src/donkey_kit/provisioning/cli.py index ac72d0e2..2b3668e7 100644 --- a/python/src/donkey_kit/provisioning/cli.py +++ b/python/src/donkey_kit/provisioning/cli.py @@ -33,7 +33,7 @@ app = typer.Typer( add_completion=False, - help="SDK for Agent Fabric — governed models, tools, provisioning-as-code.", + help="SDK for Agent Fabric — governed model access, refusals, budgets, and telemetry.", ) diff --git a/python/tests/unit/test_cli_help.py b/python/tests/unit/test_cli_help.py new file mode 100644 index 00000000..f8afd493 --- /dev/null +++ b/python/tests/unit/test_cli_help.py @@ -0,0 +1,17 @@ +"""The root CLI help advertises the supported SDK surface, not the refused +provisioning control plane from the build plan's do-not-build list (#458).""" + +from __future__ import annotations + +from typer.testing import CliRunner + +from donkey_kit.provisioning.cli import app + + +def test_root_help_does_not_market_provisioning_control_plane() -> None: + result = CliRunner().invoke(app, ["--help"]) + + assert result.exit_code == 0 + for supported_surface in ("governed model access", "refusals", "budgets", "telemetry"): + assert supported_surface in result.output + assert "provisioning-as-code" not in result.output diff --git a/python/tests/unit/test_donkey_surface.py b/python/tests/unit/test_donkey_surface.py index 522208cf..0a839eff 100644 --- a/python/tests/unit/test_donkey_surface.py +++ b/python/tests/unit/test_donkey_surface.py @@ -5,6 +5,7 @@ import pytest +import donkey_kit from donkey_kit import Donkey, DonkeyConfig from donkey_kit.core.auth import AnypointConnectedApp @@ -17,6 +18,21 @@ def _cfg() -> DonkeyConfig: ) +def test_legacy_governance_scaffolding_is_not_top_level_public_api() -> None: + """The refused provisioning control plane remains importable from its module, + but its structural types are not promoted as first-class package exports.""" + from donkey_kit.governance import Governance + + assert Governance.__module__ == "donkey_kit.governance" + assert { + "GatewayTarget", + "Governance", + "PolicyBinding", + "PolicyPortability", + }.isdisjoint(donkey_kit.__all__) + assert not hasattr(donkey_kit, "Governance") + + def test_uninstalled_adapter_raises_curated_import_error( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/website/content/concepts/environments.mdx b/website/content/concepts/environments.mdx index 65e05049..7280c036 100644 --- a/website/content/concepts/environments.mdx +++ b/website/content/concepts/environments.mdx @@ -14,7 +14,7 @@ actually enforce on your laptop versus a shared control plane. ## The `Governance` object ```python -from donkey_kit import Governance, GatewayTarget, PolicyBinding +from donkey_kit.governance import Governance, GatewayTarget, PolicyBinding gov = Governance( name="hr-agent", diff --git a/website/public/concepts/environments.md b/website/public/concepts/environments.md index 96e5d03b..9a61cca8 100644 --- a/website/public/concepts/environments.md +++ b/website/public/concepts/environments.md @@ -10,7 +10,7 @@ actually enforce on your laptop versus a shared control plane. ## The `Governance` object ```python -from donkey_kit import Governance, GatewayTarget, PolicyBinding +from donkey_kit.governance import Governance, GatewayTarget, PolicyBinding gov = Governance( name="hr-agent", diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 7e852769..dcc74fad 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -4405,7 +4405,7 @@ actually enforce on your laptop versus a shared control plane. ## The `Governance` object ```python -from donkey_kit import Governance, GatewayTarget, PolicyBinding +from donkey_kit.governance import Governance, GatewayTarget, PolicyBinding gov = Governance( name="hr-agent", From 6d61501befc266020bffc420f0b7d0f6f4d65232 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 14:44:48 +0200 Subject: [PATCH 16/29] Inject the clock into pace and guard the e2e retry (#492) Closes #489 --- python/src/donkey_kit/core/budget.py | 10 +++- .../conformance/test_budget_pacing_e2e.py | 17 +++--- python/tests/unit/test_budget_pacing.py | 57 ++++++++++++++----- 3 files changed, 61 insertions(+), 23 deletions(-) diff --git a/python/src/donkey_kit/core/budget.py b/python/src/donkey_kit/core/budget.py index 5af54b08..447cb36b 100644 --- a/python/src/donkey_kit/core/budget.py +++ b/python/src/donkey_kit/core/budget.py @@ -178,7 +178,9 @@ def observe(self, response: httpx.Response, *, now: datetime | None = None) -> N self.reset_at = ts + timedelta(milliseconds=reset_ms) @asynccontextmanager - async def pace(self, *, reserve: float = 0.0) -> AsyncIterator[None]: + async def pace( + self, *, reserve: float = 0.0, now: datetime | None = None + ) -> AsyncIterator[None]: """Guard a request so it is refused *before* it crosses your reserve, not after a 429 comes back (BG §1.3, #186). @@ -206,12 +208,14 @@ async def pace(self, *, reserve: float = 0.0) -> AsyncIterator[None]: observation whose :attr:`reset_at` has elapsed is stale for the same reason, so the retry passes through and its response refreshes the in-band budget. Before ``reset_at`` — or when no reset time was observed — the reserve guard - remains active. + remains active. ``now`` is injectable for tests; production passes nothing + and the wall clock (UTC) is used. """ if not 0.0 <= reserve <= 1.0: raise ValueError(f"reserve must be within [0.0, 1.0], got {reserve!r}") used = self.fraction_used - window_expired = self.reset_at is not None and _utcnow() >= self.reset_at + current = now if now is not None else _utcnow() + window_expired = self.reset_at is not None and current >= self.reset_at if used is not None and used >= 1.0 - reserve and not window_expired: raise BudgetReserveReached( f"Budget reserve reached: {used:.1%} of the window used, " diff --git a/python/tests/conformance/test_budget_pacing_e2e.py b/python/tests/conformance/test_budget_pacing_e2e.py index 64fb196d..a2913c7a 100644 --- a/python/tests/conformance/test_budget_pacing_e2e.py +++ b/python/tests/conformance/test_budget_pacing_e2e.py @@ -21,7 +21,7 @@ 2. **Pace trips.** With the window observed as fully used, :meth:`Budget.pace` refuses the guarded block *before* issuing a request (#186). 3. **Recover.** :meth:`Budget.wait_for_reset` sleeps until the window rolls over. -4. **Resume.** The next call succeeds and the budget replenishes. +4. **Resume.** The next guarded call succeeds and the budget replenishes. """ from __future__ import annotations @@ -133,17 +133,20 @@ async def test_pace_then_recover_against_a_real_budget_window( block_ran = True # pragma: no cover - must never execute assert block_ran is False - # 3 + 4. Recover, then resume — the arc the pace() docstring prescribes: - # wait_for_reset() then retry. Looped so the exact rollover instant is - # never a race; bounded by the wall-clock ceiling. + # 3 + 4. Recover, then resume through pace() — the arc its docstring + # prescribes. Looped so the exact rollover instant is never a race; + # bounded by the wall-clock ceiling. resumed = None while time.monotonic() - started < _DEADLINE_S: await donkey.budget.wait_for_reset() try: - resumed = await client.responses.create( - model=_HAPPY_MODEL, input="ping", max_output_tokens=16 - ) + async with donkey.budget.pace(reserve=0.0): + resumed = await client.responses.create( + model=_HAPPY_MODEL, input="ping", max_output_tokens=16 + ) break + except BudgetReserveReached: + continue # local clock has not quite reached reset_at; wait again except openai.APIStatusError as exc: assert exc.status_code == 429 # window not yet rolled; wait again await asyncio.sleep(0.1) diff --git a/python/tests/unit/test_budget_pacing.py b/python/tests/unit/test_budget_pacing.py index 99f86414..cfb8832a 100644 --- a/python/tests/unit/test_budget_pacing.py +++ b/python/tests/unit/test_budget_pacing.py @@ -27,12 +27,6 @@ def _observe(b: Budget, *, limit: int, remaining: int, reset_ms: int | None = No b.observe(_resp(**headers), now=_FIXED_NOW) -@pytest.fixture(autouse=True) -def fixed_clock(monkeypatch: pytest.MonkeyPatch) -> None: - """Keep pace() on the same deterministic clock as the injected observations.""" - monkeypatch.setattr(budget_mod, "_utcnow", lambda: _FIXED_NOW) - - @pytest.fixture def recorded_sleeps(monkeypatch: pytest.MonkeyPatch) -> list[float]: """Replace ``asyncio.sleep`` inside budget.py with a recorder so tests never @@ -84,7 +78,7 @@ async def test_pace_raises_before_the_body_when_reserve_crossed() -> None: _observe(b, limit=1000, remaining=50, reset_ms=60000) # 95% used ran = False with pytest.raises(BudgetReserveReached) as exc: - async with b.pace(reserve=0.10): # threshold 90% + async with b.pace(reserve=0.10, now=_FIXED_NOW): # threshold 90% ran = True assert ran is False # the body never executed → no request was issued assert exc.value.fraction_used == pytest.approx(0.95) @@ -130,6 +124,45 @@ async def test_pace_default_reserve_only_fires_at_full_exhaustion() -> None: pass +async def test_pace_defaults_to_wall_clock_after_reset() -> None: + """Omitting now= uses the real UTC clock, as production callers do.""" + b = Budget() + observed_at = datetime.now(timezone.utc) - timedelta(seconds=2) + b.observe( + _resp( + **{ + "x-token-limit": "1000", + "x-token-remaining": "0", + "x-token-reset": "1000", + } + ), + now=observed_at, + ) + + ran = False + async with b.pace(): + ran = True + assert ran + + +async def test_pace_defaults_to_wall_clock_before_reset() -> None: + """The real UTC clock keeps the reserve guard armed before reset_at.""" + b = Budget() + b.observe( + _resp( + **{ + "x-token-limit": "1000", + "x-token-remaining": "0", + "x-token-reset": "60000", + } + ) + ) + + with pytest.raises(BudgetReserveReached): + async with b.pace(): + raise AssertionError("body must not run before the window resets") + + # --- wait_for_reset(): the recovery sleep ----------------------------------- @@ -177,7 +210,7 @@ async def test_pace_then_wait_then_resume_cycle(recorded_sleeps: list[float]) -> # 2) window nearly exhausted → the next batch is refused before it runs _observe(b, limit=20000, remaining=500, reset_ms=60000) # 97.5% used with pytest.raises(BudgetReserveReached): - async with b.pace(reserve=0.05): # threshold 95% + async with b.pace(reserve=0.05, now=_FIXED_NOW): # threshold 95% raise AssertionError("body must not run once the reserve is reached") # 3) wait for the window, then a fresh window lets work resume @@ -185,13 +218,12 @@ async def test_pace_then_wait_then_resume_cycle(recorded_sleeps: list[float]) -> assert recorded_sleeps == [pytest.approx(60.0)] _observe(b, limit=20000, remaining=20000, reset_ms=60000) # reset happened ran_after = False - async with b.pace(reserve=0.05): + async with b.pace(reserve=0.05, now=_FIXED_NOW + timedelta(seconds=60)): ran_after = True assert ran_after async def test_pace_allows_probe_after_wait_without_manual_observe( - monkeypatch: pytest.MonkeyPatch, recorded_sleeps: list[float], ) -> None: """Regression #452: the documented wait-and-retry loop must make progress @@ -200,14 +232,13 @@ async def test_pace_allows_probe_after_wait_without_manual_observe( _observe(b, limit=20000, remaining=500, reset_ms=60000) # 97.5% used with pytest.raises(BudgetReserveReached): - async with b.pace(reserve=0.05): + async with b.pace(reserve=0.05, now=_FIXED_NOW): raise AssertionError("body must not run before the window resets") await b.wait_for_reset(now=_FIXED_NOW) assert recorded_sleeps == [pytest.approx(60.0)] - monkeypatch.setattr(budget_mod, "_utcnow", lambda: _FIXED_NOW + timedelta(seconds=60)) ran_probe = False - async with b.pace(reserve=0.05): + async with b.pace(reserve=0.05, now=_FIXED_NOW + timedelta(seconds=60)): ran_probe = True assert ran_probe From c86a49bce708668ae0f0794ed315b9dbe952f820 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 14:48:01 +0200 Subject: [PATCH 17/29] Clarify export verification boundary (#500) Closes #497 --- python/src/donkey_kit/governance.py | 8 ++++++-- python/src/donkey_kit/registry/publication.py | 8 ++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/python/src/donkey_kit/governance.py b/python/src/donkey_kit/governance.py index e99d7c80..48173d33 100644 --- a/python/src/donkey_kit/governance.py +++ b/python/src/donkey_kit/governance.py @@ -115,8 +115,12 @@ def simulate(self) -> SimulationContext: def export(self, path: str | Path | None = None) -> str: """Compile to a ``donkey.yaml`` fragment (provisioning-as-code).""" raise _verify.blocked( - "Governance.export() emits the donkey.yaml spec format. Schema is shared with " - "provisioning-as-code — do not diverge it (provisioning-as-code)." + "Governance.export() emits the shared donkey.yaml spec format; keep it identical " + "to donkey_kit.provisioning.spec and do not diverge it. The export mapping remains " + "unresolved: docs/verified-apis.md §5 confirms an Agent Network Maven-project + CLI " + "flow but not whether export() wraps that toolchain or emits its project layout " + "(the Verification milestone). This surface does not add a provisioning control " + "plane competing with API Manager or Terraform." ) # ---- verb 3: resolve (runtime, READ-ONLY) ----------------------------- diff --git a/python/src/donkey_kit/registry/publication.py b/python/src/donkey_kit/registry/publication.py index 83e3753b..10bf3a1e 100644 --- a/python/src/donkey_kit/registry/publication.py +++ b/python/src/donkey_kit/registry/publication.py @@ -83,8 +83,12 @@ async def preview(self, donkey: Any) -> str: def export(self, path: str | Path | None = None) -> str: """Compile to a ``donkey.yaml`` fragment (provisioning-as-code).""" raise _verify.blocked( - "Publication.export() emits the donkey.yaml spec format. Schema is shared with " - "provisioning-as-code — do not diverge it (provisioning-as-code)." + "Publication.export() emits the shared donkey.yaml spec format; keep it identical " + "to donkey_kit.provisioning.spec and do not diverge it. The export mapping remains " + "unresolved: docs/verified-apis.md §5 confirms an Agent Network Maven-project + CLI " + "flow but not whether export() wraps that toolchain or emits its project layout " + "(the Verification milestone). This surface does not add a provisioning control " + "plane competing with API Manager or Terraform." ) # ---- verb: verify (runtime, READ-ONLY) -------------------------------- From 290f4e3297317c18bcc83daf412846e28f3e1942 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:09:11 +0200 Subject: [PATCH 18/29] Retire obsolete milestone vocabulary outside Python source (#499) Closes #496 --- .github/workflows/ci.yml | 2 +- .github/workflows/nightly-matrix.yml | 7 +- CONTRIBUTING.md | 2 +- docs/README.md | 1 - docs/m1-completion-checklist.md | 141 ------------------ docs/verified-apis.md | 4 +- python/examples/anthropic/README.md | 10 +- python/examples/anthropic/main.py | 14 +- python/tests/conformance/suite.py | 8 +- python/tests/fixtures/a2d/README.md | 2 +- python/tests/fixtures/anypoint/README.md | 2 +- .../fixtures/anypoint/llm_proxy/README.md | 2 +- 12 files changed, 27 insertions(+), 168 deletions(-) delete mode 100644 docs/m1-completion-checklist.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9a0df3fa..f428bae9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -134,7 +134,7 @@ jobs: # Nightly framework matrix (§8.4) — runs the conformance kit against the # LATEST release of each framework and opens an issue on failure. Scheduled - # separately once adapters are conformance-wired (M1+). + # separately during Phase 1 and later. docs-llms-drift: # #205 / BG §1.10: llms.txt, llms-full.txt, and the per-page .md files are diff --git a/.github/workflows/nightly-matrix.yml b/.github/workflows/nightly-matrix.yml index fb37f50b..5ebea76a 100644 --- a/.github/workflows/nightly-matrix.yml +++ b/.github/workflows/nightly-matrix.yml @@ -22,10 +22,9 @@ # 3. pytest — the unit + ergonomics suites. # # NOTE (known constraint): the extras carry no `openai` upper bound ("floors, -# never ceilings"), so a fresh resolve may pull openai 3.x / vendored httpx2, -# which breaks the httpx transport injection the SDK is verified against (openai -# 1.x). If a framework job fails on that, it is EXPECTED signal, not a flake — -# see the pending openai-version decision in docs/m1-completion-checklist.md §3. +# never ceilings"). OpenAI 3.x retyped its injected client to vendored httpx2, +# but runtime injection is re-verified and test-pinned; see +# docs/verified-apis.md §8.1. # # Live verification (a real proxy round-trip) is intentionally NOT in the nightly # run: it needs sandbox credentials. Trigger it manually via workflow_dispatch diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79b6cbc8..191175da 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -223,7 +223,7 @@ don't rely on GitHub auto-close, which can silently miss. Closing the issue is what advances its milestone's completed count, which is how release readiness is tracked. A `develop → main` promotion happens when a milestone reaches **0 open issues**; the release PR's title carries the milestone and version (e.g. -`Release: M1 — Model access (0.1.0)`). +`Release: Phase 1 — Build the MVP (0.1.0)`). --- diff --git a/docs/README.md b/docs/README.md index f2e3c2fb..d4385c28 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,7 +15,6 @@ against a real Anypoint sandbox versus what is still assumed or blocked. | File | What it is | | --- | --- | | [`verified-apis.md`](verified-apis.md) | **The verification ledger** — the single source of truth for every endpoint, header, and class name the SDK touches and its status (`VERIFIED (LIVE)` / `VERIFIED (CLI)` / `VERIFIED (plugin)` / `UNVERIFIED` / blocked). When a value is confirmed against a sandbox, flip its row here **and** set `verified=True` in `core/_verify.py` — the two edits move together. | -| [`m1-completion-checklist.md`](m1-completion-checklist.md) | **The 0.1.0 milestone tracker** (build plan Phase 1, "M1 — Model access"). A status-keyed checklist toward a truthful, publishable release; a box flips to ✅ only when the fact is confirmed against the installed framework / real sandbox, never just because the code that assumes it was written. | | [`unsupported-boundary.md`](unsupported-boundary.md) | **The procurement doc** — classifies every platform API the SDK calls (public / no-SLA / undocumented), so enterprise-buyer questions get a five-minute answer instead of a two-week stall. Its "Undocumented surfaces" section is designed to stay empty. | | [`releasing.md`](releasing.md) | **How a release reaches PyPI** — the Trusted Publishing (OIDC) workflow wired as `.github/workflows/release.yml` (#206), the public API surface semver governs, and the one-time human step to register the trusted publisher. The version scheme and tagging live in the `ddk-release` skill; this is the publish half. | diff --git a/docs/m1-completion-checklist.md b/docs/m1-completion-checklist.md deleted file mode 100644 index 64a8c76b..00000000 --- a/docs/m1-completion-checklist.md +++ /dev/null @@ -1,141 +0,0 @@ -# M1 — Model access: completion checklist (→ 0.1.0) - -Tracks the work between today's state and a **truthful, publishable 0.1.0** -(build plan Phase 1 "M1 — Model access"). M1 ships Pillar 1 only — governed model -access through the Omni Gateway LLM proxy, plus the four Tier-1 adapters, the -conformance kit, the `lint` command, and a docs skeleton. - -> **Rule that governs every unchecked box (verification discipline):** never invent an endpoint, -> header, or class name. A box flips to ✅ only when the fact is confirmed -> against the *installed framework* / *real sandbox* — not when the code that -> assumes it is written. - -Status keys: ✅ done · 🟡 in progress / partial · ⬜ not started · 🔒 blocked on -external verification (cannot be closed by writing code). - ---- - -## 1. The critical path — verify the 8 constructor signatures (docs/verified-apis.md §8) - -**This is the only thing between us and a truthful M1 for the code that already -exists.** Every adapter is wired to the *live-verified proxy contract* (the framework-free core), but -`docs/verified-apis.md` §8 currently lists **all eight** framework -constructor/kwarg signatures as `UNVERIFIED`. The adapters' inline -`# VERIFY …: docs/verified-apis.md §8` markers identify facts that still need -verification; they are **not** proof. - -### How to verify (the procedure, made executable) - -Use the harness `python/scripts/verify_frameworks.py`. It runs two independent -checks per framework and never guesses a name: - -- **A — Signature (offline; needs only the framework `pip`-installed).** - It calls the adapter's factory, which imports the exact class we name and - constructs it with the exact kwargs we pass. A wrong class path → - `ImportError`/`AttributeError`; a wrong kwarg → `TypeError`. It then imports - the class from its **recorded docs/verified-apis.md §8 path** and checks the object `isinstance` of - it, so a silently-renamed or re-exported class is caught. **Construction - succeeding is the signature verification.** -- **B — Live round-trip (`--live`; needs the 3 `DONKEY_LLM_PROXY_*` env vars).** - Makes one real completion. Only LangGraph's runtime call (`ChatOpenAI.ainvoke`) - is exercised directly — the other seven frameworks' agent-loop APIs are - themselves unverified, so the harness constructs the object and relies on the - already-live-verified shared proxy path (the framework-free core) rather than guessing a method. - -```bash -cd python -pip install -e ".[dev,llm,langgraph]" # install the frameworks you're verifying -python scripts/verify_frameworks.py # signature check, all installed -python scripts/verify_frameworks.py --live # + one real proxy round-trip -python scripts/verify_frameworks.py --emit-verified # prints docs/verified-apis.md §8 markdown rows to paste -``` - -Per framework, "verified" means: **install the real package → harness green -(signature + live) → maintainer signs off on scope → flip the docs/verified-apis.md §8 row and paste -the confirmed class path → remove/relax the code's UNVERIFIED note.** - -### Per-framework signature status - -| Framework | Tier | docs/verified-apis.md §8 class we target | Signature (harness) | Live round-trip | docs/verified-apis.md §8 row flipped | -|---|---|---|---|---|---| -| LangGraph | 1 | `langchain_openai.ChatOpenAI` | ✅ confirmed by harness (installed locally; public path re-export verified) | 🟡 needs a `--live` run against the sandbox | ⬜ pending maintainer sign-off | -| Google ADK | 1 | `google.adk.models.lite_llm.LiteLlm` | ⬜ install `[adk]` + run harness | ⬜ | ⬜ | -| Strands | 1 | `strands.models.openai.OpenAIModel` | ⬜ install `[strands]` + run harness | ⬜ | ⬜ | -| MS Agent Framework | 1 | `agent_framework.openai.OpenAIChatClient` | 🔒 adapter raises `blocked on verification` — class path/kwarg unconfirmed; package renamed classes recently | ⬜ | ⬜ | -| OpenAI Agents SDK | 1 | `agents.OpenAIChatCompletionsModel` | ⬜ install `[openai-agents]` + run harness | ⬜ | ⬜ | -| Anthropic SDK | 1 | `anthropic.AsyncAnthropic` | ⬜ install `[anthropic]` + run harness | ⬜ | ⬜ | -| CrewAI | 1 | `crewai.LLM` | ⬜ install `[crewai]` + run harness | ⬜ | ⬜ | -| LlamaIndex | 2 | `llama_index.llms.openai_like.OpenAILike` | ⬜ install `[llamaindex]` + run harness | ⬜ | ⬜ | - -> M1's stated scope is the **four Tier-1** adapters (LangGraph, ADK, Strands, -> Agent Framework). OpenAI Agents SDK/Anthropic SDK/CrewAI/LlamaIndex exist but -> land officially in M2; they can be verified opportunistically. - ---- - -## 2. Definition of done, per adapter - -Applies to each Tier-1 adapter before it counts as shipped. - -| DoD item | LangGraph | ADK | Strands | Agent Framework | -|---|---|---|---|---| -| 1. Passes the conformance kit, or has an asserted documented exemption | ⬜ | 🟡 (correlation-id exemption recorded in `KNOWN_LIMITATIONS`) | ⬜ | ⬜ | -| 2. Runnable `examples//` (env-vars-only) | ✅ | ✅ | ✅ | ✅ (guards the blocked path) | -| 3. Docs page with the manual equivalent (users can eject) | ✅ (`website/`) | ✅ | ✅ | ✅ | -| 4. Listed in the nightly matrix | ✅ (`.github/workflows/nightly-matrix.yml`) | ✅ | ✅ | ✅ | -| 5. Version floor declared, no ceiling | ✅ (pyproject extras) | ✅ | ✅ | ✅ | - -> DoD #1 is the real gap: the conformance **scenario bodies** are not yet wired -> (`tests/conformance/suite.py` fixes the scenario list + exemption table only). -> Wiring them against the captured contract fixtures (BG §1.5) is the largest -> remaining M1 build task after signature verification. - ---- - -## 3. Remaining M1 build tasks (code, once Section 1 is unblocked) - -- ⬜ **Conformance kit bodies** — implement the M1-relevant scenarios - (`simple_completion`, `streaming_completion`, `attribution_headers_present`, - `correlation_id_propagated`, `policy_violation_terminal`) against captured - fixtures + the shared proxy path, executed identically per adapter. -- 🟡 **`lint` command** — provisioning `lint` exists; confirm it's exposed and - documented for the M1 surface, or descope to M2 explicitly. -- ✅ **Framework-free client** (`donkey.llm.client()`) — live-verified. -- ✅ **Error taxonomy + `classify()`** — live-verified against the 4 rejection shapes. -- ✅ **Model handles** (`resolve()`), honest `list_models(live=True)` ConfigError. -- ✅ **Two adapter ergonomics** — `connection_kwargs()` + module-level factories. -- ✅ **Docs site skeleton** (`website/`, Nextra) — deployable to Vercel/Railway. -- ⬜ **`openai` major-version decision** — extras have no ceiling, so a fresh - install pulls openai 3.x, which breaks the httpx transport injection the SDK - is verified against (openai 1.x). Decide: cap `openai<2` (documented exception - to "floors, never ceilings") **or** verify against openai 3.x / httpx2. See the - `openai-1x-http-client-constraint` memory. - ---- - -## 4. Gate status (must stay green) - -- ✅ `ruff check .` -- ✅ `mypy` (`--strict`, incl. `scripts/verify_frameworks.py`) -- ✅ `lint-imports` (2 contracts: core-is-framework-free, layered architecture) -- ✅ `pytest` (unit + ergonomics) -- ⬜ nightly framework matrix green on at least the Tier-1 four (needs their - extras installable in CI) - ---- - -## 5. Publish-readiness (the trademark/support boundary / docs/unsupported-boundary.md) - -- ⬜ Resolve the **maintainer / support statement** TODO in the README (the trademark/support boundary) — - blocks any public release; decide the distribution name if not MuleSoft-endorsed. -- ✅ `docs/unsupported-boundary.md` exists and is linked from the README. -- ⬜ Tag `0.1.0` only after the Tier-1 four are signature-verified **and** their - conformance bodies pass (or carry an asserted exemption). - ---- - -_Definition of "M1 done": the four Tier-1 adapters are signature-verified against -installed packages (Section 1), pass the wired conformance scenarios or carry an -asserted exemption (Section 2), all gates + the nightly matrix are green (Section 4), and the -trademark/support boundary support statement is resolved (Section 5). Everything else in the plan (Pillars 2–3, -Publication, TypeScript) is out of M1 scope._ diff --git a/docs/verified-apis.md b/docs/verified-apis.md index cd6dfc6b..d9ff2bbd 100644 --- a/docs/verified-apis.md +++ b/docs/verified-apis.md @@ -1,4 +1,4 @@ -# Verified APIs — output of the verification discipline (M0) +# Verified APIs — output of the verification discipline > **Status (2026-08-28): partially verified.** The **management/control-plane** > contract (OAuth token path, Exchange/API-Manager/gateway REST endpoints, @@ -30,7 +30,7 @@ > data shape confirmed from an Anypoint-adjacent source (A2D), not the direct > contract. `UNVERIFIED` = not yet confirmed; its code guard stays in place. > -> This is the deliverable of Milestone M0. For any row still `UNVERIFIED`, the +> This is the verification ledger. For any row still `UNVERIFIED`, the > implementing engineer must, against a **real Anypoint sandbox**, confirm the > signature, then change the row's status, fill in the **Verified value**, > **Date**, and **Source** columns. A code guard diff --git a/python/examples/anthropic/README.md b/python/examples/anthropic/README.md index f711c244..cb3f8a81 100644 --- a/python/examples/anthropic/README.md +++ b/python/examples/anthropic/README.md @@ -13,11 +13,13 @@ and the model id is a per-call argument (`c.messages.create(model=..., ...)`), not a constructor one. So this adapter exposes `client()` rather than the `model(...)` factory the OpenAI-compatible adapters use. -**Unverified dependency (docs/verified-apis.md §8).** The Omni Gateway LLM proxy is verified +**Unverified dependency (docs/verified-apis.md §8).** The Omni Gateway LLM proxy is OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API -route** is an open M0 verification item. Until confirmed, `client()` emits a -one-time warning and no live call is made. Once a real route is confirmed, -override `base_url` if needed. +route** is an open Verification-milestone item (verification discipline). If it +does not, this adapter's requests will not reach a working upstream — override +`base_url` via `**kw` to point at a real Anthropic-native route once confirmed. +The first `client()` call emits a one-time unverified-route warning; this example +does not make a live call. > 📖 **Prefer reading to running?** The canonical walkthrough — install, > configure, and the manual equivalent — is in the docs: diff --git a/python/examples/anthropic/main.py b/python/examples/anthropic/main.py index d1b0e0cd..29950609 100644 --- a/python/examples/anthropic/main.py +++ b/python/examples/anthropic/main.py @@ -8,13 +8,13 @@ from donkey_kit.integrations.anthropic import client c = client() # the model id is a per-call argument, not a constructor one -Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* over the -OpenAI-compatible route (base URL, client_id/secret auth, attribution headers) is live-verified. -What is UNVERIFIED is whether the proxy exposes an **Anthropic-native Messages API -route** at all (an open M0 item) — so this example constructs the client and -emits the SDK's one-time unverified-route warning, but does NOT make a live -call. Once a real Anthropic-native route is confirmed, override ``base_url`` and -drive ``c.messages.create(model=..., ...)`` per Anthropic's own docs. +UNVERIFIED DEPENDENCY (docs/verified-apis.md §8): the Omni Gateway LLM proxy is +OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API +route** is an open Verification-milestone item (verification discipline). If it +does not, this adapter's requests will not reach a working upstream — override +``base_url`` via ``**kw`` to point at a real Anthropic-native route once confirmed. +The first ``client()`` call emits a one-time unverified-route warning. This +example constructs the client but does NOT make a live call. """ from __future__ import annotations diff --git a/python/tests/conformance/suite.py b/python/tests/conformance/suite.py index af7ed9d1..a2771fe8 100644 --- a/python/tests/conformance/suite.py +++ b/python/tests/conformance/suite.py @@ -9,10 +9,10 @@ (#223/#244). The customer-facing pytest plugin (``donkey_kit.conformance``, #191) is the shipped deliverable this internal matrix backstops. -The scenario bodies are wired in M1+ against captured contract fixtures (BG §1.5) -and the local gateway (BG §1.4). This module fixes the scenario list and the -exemption table now so the kit exists before the second adapter is built -(working instruction #5). +The scenario bodies are wired against captured contract fixtures (BG §1.5) and +the local gateway (BG §1.4) as their gating features land (#444). This module +fixes the scenario list and the exemption table now so the kit exists before +the second adapter is built (working instruction #5). """ from __future__ import annotations diff --git a/python/tests/fixtures/a2d/README.md b/python/tests/fixtures/a2d/README.md index fb798c03..b0370e5e 100644 --- a/python/tests/fixtures/a2d/README.md +++ b/python/tests/fixtures/a2d/README.md @@ -1,4 +1,4 @@ -# Captured fixtures — A2D platform MCP tools (M0 shape verification) +# Captured fixtures — A2D platform MCP tools (shape verification) These JSON files are **faithful, trimmed captures** of real responses from the `mcp-a2d` MCP server (host `www.a2d-ai.com`), taken 2026-08-28. They exist so the diff --git a/python/tests/fixtures/anypoint/README.md b/python/tests/fixtures/anypoint/README.md index 806bc352..cca4f24b 100644 --- a/python/tests/fixtures/anypoint/README.md +++ b/python/tests/fixtures/anypoint/README.md @@ -1,4 +1,4 @@ -# Captured fixtures — real Anypoint sandbox via `anypoint-cli-v4` (M0) +# Captured fixtures — real Anypoint sandbox via `anypoint-cli-v4` Captured 2026-08-28 from the real sandbox org `82a0453b-22e6-430d-bbf4-35b989d043dc`, environment **Sandbox** diff --git a/python/tests/fixtures/anypoint/llm_proxy/README.md b/python/tests/fixtures/anypoint/llm_proxy/README.md index bc836ae7..a6305e13 100644 --- a/python/tests/fixtures/anypoint/llm_proxy/README.md +++ b/python/tests/fixtures/anypoint/llm_proxy/README.md @@ -1,4 +1,4 @@ -# LLM proxy — LIVE captures from a real governed ingress proxy (M0, docs/verified-apis.md §2/§3/§4) +# LLM proxy — LIVE captures from a real governed ingress proxy (docs/verified-apis.md §2/§3/§4) Captured 2026-08-28 from a real deployed Agent Network LLM proxy in the sandbox org `82a0453b-22e6-430d-bbf4-35b989d043dc`, env **Sandbox** From 52ca2598c7c30e354175dbe058835d598f79f56b Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:20:57 +0200 Subject: [PATCH 19/29] chore(release): bump version to 0.1.0.dev3 (#502) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Advance the pre-release counter 0.1.0.dev2 → 0.1.0.dev3 across pyproject.toml, __init__.py, and both README status lines, ahead of the next develop → main promotion. --- README.md | 2 +- python/README.md | 2 +- python/pyproject.toml | 2 +- python/src/donkey_kit/__init__.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a5f410f7..76684fda 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ An SDK for consuming **Agent Fabric** capabilities — governed model and tool access — from your own agent framework, in your own IDE, without adopting Mule. -> **Project status — alpha, pre-release.** This is `v0.1.0.dev2` +> **Project status — alpha, pre-release.** This is `v0.1.0.dev3` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; > most other surfaces are verification-gated (see > [What's verified](#whats-verified-verification-discipline) below). **Not yet published to PyPI** — diff --git a/python/README.md b/python/README.md index db4c42b3..bfbac14b 100644 --- a/python/README.md +++ b/python/README.md @@ -7,7 +7,7 @@ An SDK for consuming **Agent Fabric** capabilities — governed model and tool access — from your own agent framework, in your own IDE, without adopting Mule. -> **Project status — alpha, pre-release.** This is `v0.1.0.dev2` +> **Project status — alpha, pre-release.** This is `v0.1.0.dev3` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; > most other surfaces are verification-gated (see > [What's verified](#whats-verified-verification-discipline) below). **Not yet published to PyPI** — diff --git a/python/pyproject.toml b/python/pyproject.toml index 7eed1943..e5c4fa6d 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "donkey-kit" -version = "0.1.0.dev2" +version = "0.1.0.dev3" description = "An SDK for consuming Agent Fabric capabilities (governed model and tool access) from your own agent framework." readme = "README.md" requires-python = ">=3.10" diff --git a/python/src/donkey_kit/__init__.py b/python/src/donkey_kit/__init__.py index b28cc40e..b8717f7c 100644 --- a/python/src/donkey_kit/__init__.py +++ b/python/src/donkey_kit/__init__.py @@ -52,7 +52,7 @@ PublicationAssetType, ) -__version__ = "0.1.0.dev2" +__version__ = "0.1.0.dev3" __all__ = [ "STRICT", From 6afb93229e4d8764668e9730b444e2f446f2708d Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:23:07 +0200 Subject: [PATCH 20/29] test(cli): cover hidden provisioning command contracts (#476) Closes #461 --- python/tests/unit/test_cli_provisioning.py | 97 ++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 python/tests/unit/test_cli_provisioning.py diff --git a/python/tests/unit/test_cli_provisioning.py b/python/tests/unit/test_cli_provisioning.py new file mode 100644 index 00000000..f11f96b0 --- /dev/null +++ b/python/tests/unit/test_cli_provisioning.py @@ -0,0 +1,97 @@ +"""Smoke tests for the hidden provisioning CLI surface (#461). + +The provisioning control plane is cut for Phase 1, so these commands stay +hidden and verification-gated. The tests pin the honest §0.3 contract: local +spec validation works, while commands needing an unverified platform API exit +3 with a clear ``blocked on verification`` message. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import typer +from typer.testing import CliRunner + +from donkey_kit.provisioning.cli import app + +runner = CliRunner() + +_VALID_SPEC = """\ +apiVersion: donkey/v1 +kind: DonkeySpec +metadata: + name: orders-agent + environment: Sandbox +mcpBridges: + - name: orders + gateway: flex-sandbox +""" + + +def _write_spec(tmp_path: Path, content: str = _VALID_SPEC) -> Path: + path = tmp_path / "donkey.yaml" + path.write_text(content) + return path + + +def test_validate_accepts_a_valid_spec(tmp_path: Path) -> None: + spec = _write_spec(tmp_path) + + result = runner.invoke(app, ["validate", "--file", str(spec)]) + + assert result.exit_code == 0, result.output + assert "OK: orders-agent" in result.output + assert "1 MCP bridge(s)" in result.output + assert "env Sandbox" in result.output + + +@pytest.mark.parametrize( + "content", + [ + "metadata: [", + "apiVersion: donkey/v1\nkind: DonkeySpec\n", + ], +) +def test_validate_rejects_invalid_specs(content: str, tmp_path: Path) -> None: + spec = _write_spec(tmp_path, content) + + result = runner.invoke(app, ["validate", "--file", str(spec)]) + + assert result.exit_code == 2 + assert f"Invalid spec {spec}:" in result.output + assert "blocked on verification" not in result.output + + +def test_provisioning_commands_stay_hidden() -> None: + commands = typer.main.get_command(app).commands + visible = {name for name, command in commands.items() if not command.hidden} + + assert visible == {"init", "doctor", "mock", "test"} + + +@pytest.mark.parametrize( + "command", + ["plan", "apply", "drift", "lint", "generate"], +) +def test_spec_commands_are_honestly_blocked( + command: str, + tmp_path: Path, +) -> None: + spec = _write_spec(tmp_path) + + result = runner.invoke(app, [command, "--file", str(spec)]) + + assert result.exit_code == 3 + assert "blocked on verification:" in result.output + assert "See docs/verified-apis.md" in result.output + + +@pytest.mark.parametrize("command", ["status", "publish", "verify"]) +def test_platform_commands_are_honestly_blocked(command: str) -> None: + result = runner.invoke(app, [command]) + + assert result.exit_code == 3 + assert "blocked on verification:" in result.output + assert "See docs/verified-apis.md" in result.output From c62fafff1b1dfed0764118e8260d7a77bf2c0350 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:34:01 +0200 Subject: [PATCH 21/29] Correct OAuth token path verification wording (#477) Closes #450 --- python/src/donkey_kit/core/auth.py | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/python/src/donkey_kit/core/auth.py b/python/src/donkey_kit/core/auth.py index cb34a824..1af5f8ec 100644 --- a/python/src/donkey_kit/core/auth.py +++ b/python/src/donkey_kit/core/auth.py @@ -8,12 +8,12 @@ The control-plane credential and the LLM-proxy credential are SEPARATE and must not be conflated (BG §1.1). -VERIFICATION NOTE: the token endpoint PATH and the scopes each operation -needs are UNVERIFIED. The path is a loud, overridable placeholder from -``_verify.OAUTH_TOKEN_PATH``; the per-operation scope table must be recorded in -``docs/`` once confirmed. Some operations require an *admin* connected app with -user context rather than pure client credentials — that path is not yet -implemented and raises a verification-blocked error where it is needed. +VERIFICATION NOTE: the default token endpoint path from +``_verify.OAUTH_TOKEN_PATH`` is VERIFIED (plugin); see +``docs/verified-apis.md`` §1 and §12.1. The scopes each operation needs and the +operations that require an *admin* connected app with user context remain +UNVERIFIED. The latter path is not yet implemented and raises a +verification-blocked error where it is needed. """ from __future__ import annotations @@ -74,7 +74,7 @@ def __init__( self._client_secret = client_secret self._base = control_plane_url.rstrip("/") self._http = http_client - # UNVERIFIED placeholder unless the caller supplies a confirmed path. + # The default is VERIFIED (plugin); callers may override it for their environment. self._token_path = token_path or _verify.OAUTH_TOKEN_PATH.get() self._clock = clock self._cached: str | None = None @@ -109,17 +109,19 @@ async def _fetch(self) -> str: ) if resp.status_code >= 400: raise AuthError( - f"Anypoint token endpoint returned {resp.status_code}. The token " - f"path is UNVERIFIED ({self._token_path!r}) — confirm it against a " - f"sandbox and record it in docs/verified-apis.md §1.", + f"Anypoint token endpoint returned {resp.status_code} for " + f"{self._token_path!r}. Check that the control-plane host is reachable, " + "that any token_path override is correct, and that connected-app " + "requirements are met (see docs/verified-apis.md §1 and §12.1).", response=resp, ) body = resp.json() token: str | None = body.get("access_token") if not token: raise AuthError( - "Token endpoint returned no access_token. Response shape is " - "UNVERIFIED — capture it as a fixture (BG §1.5).", + "Token endpoint returned no access_token. The expected response shape " + "includes access_token and expires_in (see docs/verified-apis.md §1 and " + "§12.1); capture the unexpected response as a fixture (BG §1.5).", response=resp, ) expires_in = float(body.get("expires_in", 3600)) From 545d14889f0975cd26e88f071f17829ea1ab2adb Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:37:48 +0200 Subject: [PATCH 22/29] Correct budget reset pass-through wording (#495) Closes #488 --- python/src/donkey_kit/core/budget.py | 11 ++++-- python/tests/unit/test_budget_pacing.py | 41 +++++++++++++++++++++ spec/donkey-development-kit-build-guide.md | 10 +++-- website/content/budget.mdx | 14 ++++--- website/content/scenarios/nightly-batch.mdx | 6 ++- website/public/budget.md | 14 ++++--- website/public/llms-full.txt | 20 ++++++---- website/public/scenarios/nightly-batch.md | 6 ++- 8 files changed, 93 insertions(+), 29 deletions(-) diff --git a/python/src/donkey_kit/core/budget.py b/python/src/donkey_kit/core/budget.py index 447cb36b..49467812 100644 --- a/python/src/donkey_kit/core/budget.py +++ b/python/src/donkey_kit/core/budget.py @@ -206,10 +206,13 @@ async def pace( observed there is no basis to refuse, and blocking forever on a cold start would be worse than one request that discovers the real headroom. An observation whose :attr:`reset_at` has elapsed is stale for the same reason, - so the retry passes through and its response refreshes the in-band budget. - Before ``reset_at`` — or when no reset time was observed — the reserve guard - remains active. ``now`` is injectable for tests; production passes nothing - and the wall clock (UTC) is used. + so :meth:`pace` no longer refuses requests. A later response carrying a + recognised budget signal updates the observed fields; a refreshed + :attr:`reset_at` in the future makes the reserve guard active again. A + response that does not supply a refreshed future :attr:`reset_at` leaves the + stale pass-through open. Before ``reset_at`` — or when no reset time was + observed — the reserve guard remains active. ``now`` is injectable for + tests; production passes nothing and the wall clock (UTC) is used. """ if not 0.0 <= reserve <= 1.0: raise ValueError(f"reserve must be within [0.0, 1.0], got {reserve!r}") diff --git a/python/tests/unit/test_budget_pacing.py b/python/tests/unit/test_budget_pacing.py index cfb8832a..410a053b 100644 --- a/python/tests/unit/test_budget_pacing.py +++ b/python/tests/unit/test_budget_pacing.py @@ -242,3 +242,44 @@ async def test_pace_allows_probe_after_wait_without_manual_observe( async with b.pace(reserve=0.05, now=_FIXED_NOW + timedelta(seconds=60)): ran_probe = True assert ran_probe + + +async def test_signal_free_response_leaves_expired_window_open() -> None: + """BG §1.3: once reset_at has elapsed, a response with no budget signal + leaves the stale pass-through open rather than restoring the old refusal.""" + b = Budget() + _observe(b, limit=20000, remaining=500, reset_ms=1000) # 97.5% used + later = _FIXED_NOW + timedelta(seconds=2) + + b.observe(_resp(), now=later) + + ran = False + async with b.pace(reserve=0.05, now=later): + ran = True + assert ran + + +async def test_future_reset_at_rearms_expired_window() -> None: + """BG §1.3: a recognised signal with a future reset_at makes the reserve + guard active again after the previous observation expired.""" + b = Budget() + _observe(b, limit=20000, remaining=500, reset_ms=1000) # 97.5% used + later = _FIXED_NOW + timedelta(seconds=2) + + async with b.pace(reserve=0.05, now=later): + pass + + b.observe( + _resp( + **{ + "x-token-limit": "20000", + "x-token-remaining": "500", + "x-token-reset": "60000", + } + ), + now=later, + ) + + with pytest.raises(BudgetReserveReached): + async with b.pace(reserve=0.05, now=later): + pass diff --git a/spec/donkey-development-kit-build-guide.md b/spec/donkey-development-kit-build-guide.md index c2309678..8e973090 100644 --- a/spec/donkey-development-kit-build-guide.md +++ b/spec/donkey-development-kit-build-guide.md @@ -154,9 +154,11 @@ for batch in chunks(records, 200): checkpoint(batch) ``` -After `reset_at`, the old observation is stale, so `pace()` lets the retry -through and its response refreshes the in-band budget. The job finishes by -itself, overnight, with no human. +After `reset_at`, the old observation is stale, so `pace()` no longer refuses. +A response carrying a budget signal updates the observed fields; a fresh future +`reset_at` makes the guard active again. A response that does not supply a fresh +future `reset_at` leaves the stale pass-through open. The job can therefore +continue overnight without a manual budget observation. **Why it matters — Scenario A.** A dashboard shows `fraction_used` per agent. The support agent's owner sees it climbing at 14:00 and requests an increase before the 16:00 peak instead of after the outage. @@ -165,7 +167,7 @@ itself, overnight, with no human. **Acceptance.** - `x-token-reset` in milliseconds is converted correctly (fixture with a known value; assert `reset_at` to the second). - `pace()` raises before the request that would cross the reserve, not after a 429. -- After `reset_at`, `pace()` lets one retry through so its response can refresh the in-band budget; the documented `pace()` → `wait_for_reset()` → retry loop makes progress without a manual observation. +- After `reset_at`, `pace()` no longer refuses; the documented `pace()` → `wait_for_reset()` → retry loop makes progress without a manual budget observation. A later response updates the budget only when it carries a recognised signal, and the guard becomes active again when that update supplies a future `reset_at`. - Budget object is per-`Donkey`, not global; two `Donkey` instances with different credentials do not share state. **Effort:** S. diff --git a/website/content/budget.mdx b/website/content/budget.mdx index b046e8a0..b440daec 100644 --- a/website/content/budget.mdx +++ b/website/content/budget.mdx @@ -68,9 +68,11 @@ for batch in chunks(records, 200): checkpoint(batch) ``` -Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the -retry through so its response can refresh the in-band budget. The job finishes -unattended; nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer +refuses. A response carrying a budget signal updates the observed fields; a +fresh future `reset_at` makes the guard active again. A response that does not +supply a fresh future `reset_at` leaves the stale pass-through open. The job can +continue unattended without a manual budget observation. ## The dashboard that prevents the outage @@ -101,8 +103,10 @@ last-known-good. See [Roadmap](/roadmap). against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. -- After `reset_at`, `pace()` lets the retry through so the response can refresh - the in-band budget; no manual observation is required. +- After `reset_at`, `pace()` no longer refuses, so the wait-and-retry loop can + continue without a manual budget observation. A later response updates the + budget only when it carries a recognised signal, and the guard becomes active + again when that update supplies a future `reset_at`. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/content/scenarios/nightly-batch.mdx b/website/content/scenarios/nightly-batch.mdx index b6f408ba..bbf68a29 100644 --- a/website/content/scenarios/nightly-batch.mdx +++ b/website/content/scenarios/nightly-batch.mdx @@ -96,8 +96,10 @@ async def enrich_all(records, enrich): request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header (milliseconds, converted for you). Once that time has elapsed, the old - observation is stale, so `pace()` lets the retry through and the response - refreshes the in-band budget. See [Budget & pacing](/budget). + observation is stale, so `pace()` no longer refuses. A response carrying a + budget signal updates the observed fields; a fresh future `reset_at` makes + the guard active again. A response that does not supply a fresh future + `reset_at` leaves the stale pass-through open. See [Budget & pacing](/budget). ## The honest limitation diff --git a/website/public/budget.md b/website/public/budget.md index 17e2d7ea..a3a5e3c9 100644 --- a/website/public/budget.md +++ b/website/public/budget.md @@ -60,9 +60,11 @@ for batch in chunks(records, 200): checkpoint(batch) ``` -Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the -retry through so its response can refresh the in-band budget. The job finishes -unattended; nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer +refuses. A response carrying a budget signal updates the observed fields; a +fresh future `reset_at` makes the guard active again. A response that does not +supply a fresh future `reset_at` leaves the stale pass-through open. The job can +continue unattended without a manual budget observation. ## The dashboard that prevents the outage @@ -91,8 +93,10 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. -- After `reset_at`, `pace()` lets the retry through so the response can refresh - the in-band budget; no manual observation is required. +- After `reset_at`, `pace()` no longer refuses, so the wait-and-retry loop can + continue without a manual budget observation. A later response updates the + budget only when it carries a recognised signal, and the guard becomes active + again when that update supplies a future `reset_at`. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index dcc74fad..563006de 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -2175,9 +2175,11 @@ for batch in chunks(records, 200): checkpoint(batch) ``` -Once `reset_at` has elapsed, the old observation is stale. `pace()` lets the -retry through so its response can refresh the in-band budget. The job finishes -unattended; nobody re-runs anything. +Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer +refuses. A response carrying a budget signal updates the observed fields; a +fresh future `reset_at` makes the guard active again. A response that does not +supply a fresh future `reset_at` leaves the stale pass-through open. The job can +continue unattended without a manual budget observation. ## The dashboard that prevents the outage @@ -2206,8 +2208,10 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d against a fixture with a known value. - `pace()` raises before the request that would cross the reserve, never after a `429`. -- After `reset_at`, `pace()` lets the retry through so the response can refresh - the in-band budget; no manual observation is required. +- After `reset_at`, `pace()` no longer refuses, so the wait-and-retry loop can + continue without a manual budget observation. A later response updates the + budget only when it carries a recognised signal, and the guard becomes active + again when that update supplies a future `reset_at`. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. @@ -3289,8 +3293,10 @@ async def enrich_all(records, enrich): request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header (milliseconds, converted for you). Once that time has elapsed, the old - observation is stale, so `pace()` lets the retry through and the response - refreshes the in-band budget. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + observation is stale, so `pace()` no longer refuses. A response carrying a + budget signal updates the observed fields; a fresh future `reset_at` makes + the guard active again. A response that does not supply a fresh future + `reset_at` leaves the stale pass-through open. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation diff --git a/website/public/scenarios/nightly-batch.md b/website/public/scenarios/nightly-batch.md index dcf399f2..ac643908 100644 --- a/website/public/scenarios/nightly-batch.md +++ b/website/public/scenarios/nightly-batch.md @@ -85,8 +85,10 @@ async def enrich_all(records, enrich): request that earns the `429`. `wait_for_reset()` sleeps until `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header (milliseconds, converted for you). Once that time has elapsed, the old - observation is stale, so `pace()` lets the retry through and the response - refreshes the in-band budget. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + observation is stale, so `pace()` no longer refuses. A response carrying a + budget signal updates the observed fields; a fresh future `reset_at` makes + the guard active again. A response that does not supply a fresh future + `reset_at` leaves the stale pass-through open. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation From 36c8f69c644627f701e28c5e555b5063ece357fe Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:37:54 +0200 Subject: [PATCH 23/29] Protect verification-blocked registry and tool guards (#475) Closes #464 --- python/tests/unit/test_registry_shapes.py | 70 ++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/python/tests/unit/test_registry_shapes.py b/python/tests/unit/test_registry_shapes.py index 10198a91..04baa405 100644 --- a/python/tests/unit/test_registry_shapes.py +++ b/python/tests/unit/test_registry_shapes.py @@ -11,20 +11,67 @@ from __future__ import annotations +import inspect import json +from collections.abc import Callable from pathlib import Path +from typing import cast import pytest -from donkey_kit import AssetRef, AssetType, PublicationAssetType +from donkey_kit import AssetRef, AssetType, Publication, PublicationAssetType +from donkey_kit.core.config import DonkeyConfig +from donkey_kit.core.transport import DonkeyAsyncClient +from donkey_kit.registry.exchange import ExchangeRegistry +from donkey_kit.registry.governance import GovernanceCriteria +from donkey_kit.registry.introspect import derive_descriptor from donkey_kit.registry.models import McpServerHandle from donkey_kit.tools.filter import ToolDescriptor, resolve_collisions +from donkey_kit.tools.session import ToolSet FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "a2d" # Observed transport kinds → the SDK's normalized transport identifier. _TRANSPORT_NORMALIZATION = {"streamableHttp": "streamable_http"} +_REF = AssetRef(group_id="com.example", asset_id="blocked-asset", version="1.0.0") +_REGISTRY = ExchangeRegistry( + DonkeyConfig(client_id="client", client_secret="secret", org_id="org"), + cast(DonkeyAsyncClient, object()), +) +_TOOL_SET = ToolSet([McpServerHandle(ref=_REF, endpoint_url="https://example.test/mcp")]) +_PUBLICATION = Publication( + asset_type=PublicationAssetType.MCP_SERVER, + group_id=_REF.group_id, + asset_id=_REF.asset_id, + version=_REF.version, + name="Blocked asset", + description="An asset whose publication surfaces remain verification-blocked.", +) + +_BLOCKED_SURFACES: tuple[tuple[str, Callable[[], object]], ...] = ( + ("registry.search", lambda: _REGISTRY.search()), + ("registry.resolve_mcp", lambda: _REGISTRY.resolve_mcp(_REF)), + ("registry.resolve_agent", lambda: _REGISTRY.resolve_agent(_REF)), + ( + "registry.explain", + lambda: _REGISTRY.explain(_REF, criteria=GovernanceCriteria()), + ), + ("registry.warm", lambda: _REGISTRY.warm()), + ("tools.langgraph", _TOOL_SET.langgraph), + ("tools.adk", _TOOL_SET.adk), + ("tools.strands", _TOOL_SET.strands), + ("tools.llamaindex", _TOOL_SET.llamaindex), + ("tools.openai", _TOOL_SET.openai), + ("tools.anthropic", _TOOL_SET.anthropic), + ("tools.crewai", _TOOL_SET.crewai), + ("tools.agent_framework", _TOOL_SET.agent_framework), + ("publication.preview", lambda: _PUBLICATION.preview(object())), + ("publication.export", _PUBLICATION.export), + ("publication.verify", lambda: _PUBLICATION.verify(object())), + ("registry.derive_descriptor", lambda: derive_descriptor("example:asset")), +) + def test_package_root_asset_types_share_discovery_literal_values() -> None: discovery_type: AssetType = "mcp" @@ -50,6 +97,27 @@ def _endpoint_for(environments: list[dict], asset_id: str, env_type: str) -> str raise LookupError(f"no mcp_server env for {asset_id} / {env_type}") +@pytest.mark.parametrize( + ("surface", "invoke"), + _BLOCKED_SURFACES, + ids=[surface for surface, _ in _BLOCKED_SURFACES], +) +async def test_unverified_registry_and_tool_surfaces_remain_blocked( + surface: str, invoke: Callable[[], object] +) -> None: + """Protect verification guards from guessed implementations (§0.3). + + The Publication methods currently raise before inspecting their placeholder + ``donkey`` arguments; unlike the registry cases, those rows assert only the + guard. When a surface is verified, remove its row here in the same change + that records the VERIFIED date and source in ``docs/verified-apis.md``. + """ + with pytest.raises(NotImplementedError, match=r"^blocked on verification:"): + result = invoke() + if inspect.isawaitable(result): + await result + + def test_mcp_server_spec_maps_to_handle() -> None: spec = _load("mcp_server_spec.account_management.json")["spec"] environments = _load("environments.sample.json")["environments"] From 9197896e5075e269b0b8fd2a0cf344256810521a Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 15:41:52 +0200 Subject: [PATCH 24/29] Make fixture-lock helpers wheel-safe (#498) Closes #453 Closes #491 --- .github/workflows/ci.yml | 27 ++++++++++++- CONTRIBUTING.md | 5 +++ python/pyproject.toml | 1 + python/src/donkey_kit/simulator/fixtures.py | 42 +++++++++++++++++---- python/tests/unit/test_fixture_integrity.py | 37 ++++++++++++++---- website/content/simulator.mdx | 14 ++++--- website/public/llms-full.txt | 14 ++++--- website/public/simulator.md | 14 ++++--- 8 files changed, 121 insertions(+), 33 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f428bae9..0d4d062d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,7 +16,8 @@ defaults: jobs: base-only: # Working instruction #9: install ONLY the base package and run the base - # suite, to catch accidental top-level framework imports. + # suite to catch accidental top-level framework imports. The installed-wheel + # check runs afterward in its own environment. runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -27,6 +28,30 @@ jobs: - name: import donkey_kit with no framework extras installed run: python -c "import donkey_kit; from donkey_kit import Donkey; print(donkey_kit.__version__)" - run: pytest -q tests/unit + - name: installed wheel resolves packaged simulator fixtures (BG §1.4) + run: | + wheel_dir="$(mktemp -d)" + venv_dir="$(mktemp -d)" + trap 'rm -rf "$wheel_dir" "$venv_dir"' EXIT + pip install build + python -m build --wheel --outdir "$wheel_dir" + set -- "$wheel_dir"/*.whl + test "$#" -eq 1 + python -m venv "$venv_dir" + "$venv_dir/bin/pip" install "$1" + "$venv_dir/bin/python" - <<'PY' + from donkey_kit.simulator import fixtures + + # Prove load() cannot silently fall back to checkout-only resources. + assert not fixtures._SOURCE_ROOT.exists(), fixtures._SOURCE_ROOT + for shape in fixtures.SHAPES: + fixtures.load(shape) + assert fixtures.LOCK_PATH.exists(), fixtures.LOCK_PATH + assert fixtures.compute_manifest() == fixtures.read_lock() + before = fixtures.LOCK_PATH.read_bytes() + fixtures.write_lock() + assert fixtures.LOCK_PATH.read_bytes() == before + PY typecheck-and-lint: runs-on: ubuntu-latest diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 191175da..c2fe6137 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -258,6 +258,11 @@ provable: `PolicyViolation` stays distinct from the retryable `remediation` (assert it directly). See [`ARCHITECTURE.md`](ARCHITECTURE.md#error-taxonomy-design-bg-12) for why. +That job also builds and installs the wheel into an isolated environment, then +verifies the packaged simulator fixtures against their shipped integrity lock. +This is the packaging-path gate; source-checkout tests alone cannot prove those +resources landed in the wheel. + ### `tests/typecheck/` — downstream static contracts `mypy` strict-checks these small, non-pytest modules alongside `src/donkey_kit`. diff --git a/python/pyproject.toml b/python/pyproject.toml index e5c4fa6d..ced08528 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -134,6 +134,7 @@ packages = ["src/donkey_kit"] # packaged copy first, falling back to the source-checkout tests/fixtures/ tree. "tests/fixtures/rejections" = "donkey_kit/simulator/_fixtures/rejections" "tests/fixtures/anypoint/llm_proxy" = "donkey_kit/simulator/_fixtures/anypoint/llm_proxy" +"tests/fixtures/fixtures.lock" = "donkey_kit/simulator/_fixtures/fixtures.lock" [tool.mypy] # Working instruction #8: mypy --strict, blocking in CI. Typecheck fixtures pin diff --git a/python/src/donkey_kit/simulator/fixtures.py b/python/src/donkey_kit/simulator/fixtures.py index 751766bd..54b5fab9 100644 --- a/python/src/donkey_kit/simulator/fixtures.py +++ b/python/src/donkey_kit/simulator/fixtures.py @@ -17,6 +17,9 @@ tests read from, so in a source tree the simulator and the tests read the byte-identical files. +The fixture-integrity lock follows the same resolution order, so the public +lock helpers work from both a source checkout and an installed wheel. + Nothing here imports a web framework: this module is safe under the base-only CI job (``[dev]`` only, no ``[local]`` extra). It imports only the framework-free ``core`` layer for the verified header names (an allowed upward import). @@ -223,16 +226,17 @@ class _Spec: # Derived from this module's location: fixtures.py is at # python/src/donkey_kit/simulator/fixtures.py, so parents[3] is python/. _SOURCE_ROOT = Path(__file__).resolve().parents[3] / "tests" / "fixtures" +_PACKAGED_ROOT = importlib.resources.files("donkey_kit.simulator") / "_fixtures" def _packaged_bytes(directory: str, name: str) -> bytes | None: """Read ``_fixtures//`` from the installed package, or ``None`` if it is not present (e.g. an editable/source checkout that never ran the wheel force-include).""" - res = importlib.resources.files("donkey_kit.simulator") + res = _PACKAGED_ROOT # Chain single-segment ``/`` joins for 3.10 compatibility (multi-arg # joinpath() only landed in 3.11). - for part in ("_fixtures", directory, name): + for part in (directory, name): res = res / part if res.is_file(): return res.read_bytes() @@ -337,7 +341,15 @@ def replay_headers(fixture: Fixture) -> dict[str, str]: # ``python -m donkey_kit.simulator.fixtures --relock`` — the deliberate, # reviewable "I re-captured this, I meant it" step. This is an integrity # assertion, not fixture-capture tooling (which #189 puts out of scope). -LOCK_PATH = _SOURCE_ROOT / "fixtures.lock" +_SOURCE_LOCK_PATH = _SOURCE_ROOT / "fixtures.lock" +_PACKAGED_LOCK = _PACKAGED_ROOT / "fixtures.lock" +# ``LOCK_PATH`` is public and writable, so it must remain a concrete ``Path``. +# Normal wheel installs are unpacked to a pathlib.Path. Zip-imported packages +# can still read fixture bytes through Traversable, but cannot expose or rewrite +# the lock through this Path-based public API. +LOCK_PATH = _SOURCE_LOCK_PATH +if not LOCK_PATH.is_file() and isinstance(_PACKAGED_LOCK, Path): + LOCK_PATH = _PACKAGED_LOCK def _served_files() -> list[tuple[str, str]]: @@ -364,21 +376,28 @@ def compute_manifest() -> dict[str, str]: def read_lock() -> dict[str, str]: - """The committed manifest at :data:`LOCK_PATH`.""" + """The committed manifest at :data:`LOCK_PATH`, from either package layout.""" data: dict[str, str] = json.loads(LOCK_PATH.read_text(encoding="utf-8")) return data def write_lock() -> None: """Regenerate :data:`LOCK_PATH` from the current fixture bytes. Called by - ``python -m donkey_kit.simulator.fixtures --relock`` after a re-capture.""" + ``python -m donkey_kit.simulator.fixtures --relock`` after a re-capture. + + A source checkout always wins when present, keeping the committed lock + canonical. From an installed wheel this updates that environment's packaged + lock; the ``--relock`` command rejects that layout because it cannot produce + a file to commit. + """ LOCK_PATH.write_text( json.dumps(compute_manifest(), indent=2, sort_keys=True) + "\n", encoding="utf-8", ) -if __name__ == "__main__": # pragma: no cover - dev-only relock entrypoint +def _main() -> None: + """Run the dev-only relock command.""" import argparse parser = argparse.ArgumentParser(description="Simulator fixture integrity lock.") @@ -389,7 +408,16 @@ def write_lock() -> None: ) args = parser.parse_args() if args.relock: + if not _SOURCE_LOCK_PATH.is_file(): + parser.error( + "--relock must run from an editable source checkout so it updates " + "tests/fixtures/fixtures.lock" + ) write_lock() - print(f"wrote {LOCK_PATH} ({len(compute_manifest())} fixtures)") + print(f"wrote {_SOURCE_LOCK_PATH} ({len(compute_manifest())} fixtures)") else: parser.error("nothing to do; pass --relock to regenerate the lock") + + +if __name__ == "__main__": # pragma: no cover - dev-only relock entrypoint + _main() diff --git a/python/tests/unit/test_fixture_integrity.py b/python/tests/unit/test_fixture_integrity.py index 6decd899..1bee27bc 100644 --- a/python/tests/unit/test_fixture_integrity.py +++ b/python/tests/unit/test_fixture_integrity.py @@ -7,18 +7,22 @@ JSON, an added field, a whitespace tweak) leaves every assertion green while silently drifting the bytes the simulator replays away from the captured shape. -This lock closes that gap. ``tests/fixtures/fixtures.lock`` pins the sha256 of -every fixture file the simulator serves; this test fails loudly the moment any -of those bytes change. A legitimate re-capture is accepted by regenerating the -lock (``python -m donkey_kit.simulator.fixtures --relock``) and committing it — -the deliberate, reviewable "I meant this" step. It is an integrity assertion, -not fixture-capture tooling (which #189 puts out of scope). +This lock closes that gap. :data:`donkey_kit.simulator.fixtures.LOCK_PATH` pins +the sha256 of every fixture file the simulator serves; in a source checkout it +is ``tests/fixtures/fixtures.lock``, while a wheel resolves its packaged copy. +This test fails loudly the moment any of those bytes change. A legitimate +re-capture is accepted by regenerating the source lock +(``python -m donkey_kit.simulator.fixtures --relock``) and committing it — the +deliberate, reviewable "I meant this" step. It is an integrity assertion, not +fixture-capture tooling (which #189 puts out of scope). Base-only safe: imports only the framework-free ``simulator.fixtures`` module. """ from __future__ import annotations +import pytest + from donkey_kit.simulator import fixtures @@ -31,7 +35,7 @@ def test_served_fixtures_match_the_committed_lock() -> None: locked = fixtures.read_lock() assert current == locked, ( - "A simulator fixture's bytes changed but tests/fixtures/fixtures.lock " + f"A simulator fixture's bytes changed but {fixtures.LOCK_PATH} " "was not updated. If you re-captured a shape from a real gateway, run\n" " python -m donkey_kit.simulator.fixtures --relock\n" "and commit the updated lock. If you did NOT intend to change a " @@ -42,6 +46,25 @@ def test_served_fixtures_match_the_committed_lock() -> None: ) +def test_source_checkout_uses_the_committed_lock() -> None: + """Repository tests must compare the source fixtures with the source lock.""" + assert fixtures.LOCK_PATH == fixtures._SOURCE_ROOT / "fixtures.lock" + + +def test_relock_cli_requires_a_source_checkout( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """The dev-only command cannot silently rewrite an installed wheel.""" + monkeypatch.setattr(fixtures, "_SOURCE_LOCK_PATH", fixtures._SOURCE_ROOT / "missing.lock") + monkeypatch.setattr("sys.argv", ["fixtures", "--relock"]) + + with pytest.raises(SystemExit, match="2"): + fixtures._main() + + assert "--relock must run from an editable source checkout" in capsys.readouterr().err + + def test_lock_covers_exactly_the_served_files() -> None: """The lock has no stale rows and no gaps — its keys are exactly the files the shape table resolves to, so a newly added shape cannot ship unlocked.""" diff --git a/website/content/simulator.mdx b/website/content/simulator.mdx index c46d5e2a..ae750471 100644 --- a/website/content/simulator.mdx +++ b/website/content/simulator.mdx @@ -175,11 +175,13 @@ taxonomy fail together, rather than the simulator quietly teaching you a contract that no longer exists. Because "same files, both fail together" only catches a fixture edit that -flips something a test asserts on, a committed **integrity lock** -(`tests/fixtures/fixtures.lock`) pins the sha256 of every fixture the -simulator serves. A *benign* byte edit — reformatted JSON, an added field — -fails the lock loudly instead of silently drifting the replayed bytes. A -genuine re-capture is accepted by regenerating the lock +flips something a test asserts on, a committed **integrity manifest** pins the +sha256 of every fixture the simulator serves. Its source copy lives at +`tests/fixtures/fixtures.lock`; wheel builds copy it beside the packaged +fixtures, so an installed wheel can verify that its replayed bytes still match +the committed manifest. A *benign* byte edit — reformatted JSON, an added field +— fails the lock loudly instead of silently drifting the replayed bytes. A +genuine re-capture is accepted by regenerating the source lock (`python -m donkey_kit.simulator.fixtures --relock`) and committing it. ### It replays; it does not evaluate @@ -207,7 +209,7 @@ genuine re-capture is accepted by regenerating the lock ([#189](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/189)). ✅ - The fixtures are the *same files* the `classify()` contract tests load, so the two fail together if the gateway contract ever drifts — and a committed - integrity lock fails loudly on any byte edit that the "both fail together" + integrity manifest fails loudly on any byte edit that the "both fail together" rule would otherwise miss. ✅ - The simulator replays captured shapes; it does **not** evaluate policy (real policy-configuration testing needs gateway diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 563006de..7c9333e7 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -2386,11 +2386,13 @@ taxonomy fail together, rather than the simulator quietly teaching you a contract that no longer exists. Because "same files, both fail together" only catches a fixture edit that -flips something a test asserts on, a committed **integrity lock** -(`tests/fixtures/fixtures.lock`) pins the sha256 of every fixture the -simulator serves. A *benign* byte edit — reformatted JSON, an added field — -fails the lock loudly instead of silently drifting the replayed bytes. A -genuine re-capture is accepted by regenerating the lock +flips something a test asserts on, a committed **integrity manifest** pins the +sha256 of every fixture the simulator serves. Its source copy lives at +`tests/fixtures/fixtures.lock`; wheel builds copy it beside the packaged +fixtures, so an installed wheel can verify that its replayed bytes still match +the committed manifest. A *benign* byte edit — reformatted JSON, an added field +— fails the lock loudly instead of silently drifting the replayed bytes. A +genuine re-capture is accepted by regenerating the source lock (`python -m donkey_kit.simulator.fixtures --relock`) and committing it. ### It replays; it does not evaluate @@ -2416,7 +2418,7 @@ genuine re-capture is accepted by regenerating the lock ([#189](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/189)). ✅ - The fixtures are the *same files* the `classify()` contract tests load, so the two fail together if the gateway contract ever drifts — and a committed - integrity lock fails loudly on any byte edit that the "both fail together" + integrity manifest fails loudly on any byte edit that the "both fail together" rule would otherwise miss. ✅ - The simulator replays captured shapes; it does **not** evaluate policy (real policy-configuration testing needs gateway diff --git a/website/public/simulator.md b/website/public/simulator.md index 8f75379f..c88b494a 100644 --- a/website/public/simulator.md +++ b/website/public/simulator.md @@ -161,11 +161,13 @@ taxonomy fail together, rather than the simulator quietly teaching you a contract that no longer exists. Because "same files, both fail together" only catches a fixture edit that -flips something a test asserts on, a committed **integrity lock** -(`tests/fixtures/fixtures.lock`) pins the sha256 of every fixture the -simulator serves. A *benign* byte edit — reformatted JSON, an added field — -fails the lock loudly instead of silently drifting the replayed bytes. A -genuine re-capture is accepted by regenerating the lock +flips something a test asserts on, a committed **integrity manifest** pins the +sha256 of every fixture the simulator serves. Its source copy lives at +`tests/fixtures/fixtures.lock`; wheel builds copy it beside the packaged +fixtures, so an installed wheel can verify that its replayed bytes still match +the committed manifest. A *benign* byte edit — reformatted JSON, an added field +— fails the lock loudly instead of silently drifting the replayed bytes. A +genuine re-capture is accepted by regenerating the source lock (`python -m donkey_kit.simulator.fixtures --relock`) and committing it. ### It replays; it does not evaluate @@ -191,7 +193,7 @@ genuine re-capture is accepted by regenerating the lock ([#189](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/189)). ✅ - The fixtures are the *same files* the `classify()` contract tests load, so the two fail together if the gateway contract ever drifts — and a committed - integrity lock fails loudly on any byte edit that the "both fail together" + integrity manifest fails loudly on any byte edit that the "both fail together" rule would otherwise miss. ✅ - The simulator replays captured shapes; it does **not** evaluate policy (real policy-configuration testing needs gateway From f5d97097ad5596af896f080bc6089c680ad1d594 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 16:06:14 +0200 Subject: [PATCH 25/29] Stop budget retry loops when reset time is unknown (#494) Closes #487 --- python/src/donkey_kit/core/budget.py | 20 ++++++++---- python/src/donkey_kit/core/errors.py | 19 ++++++++--- python/tests/unit/test_budget_pacing.py | 28 +++++++++++++++++ spec/donkey-development-kit-build-guide.md | 29 ++++++++++++----- website/content/budget.mdx | 12 +++++-- website/content/errors.mdx | 12 ++++--- website/content/scenarios/nightly-batch.mdx | 11 +++++-- website/public/budget.md | 12 +++++-- website/public/errors.md | 12 ++++--- website/public/llms-full.txt | 35 +++++++++++++++------ website/public/scenarios/nightly-batch.md | 11 +++++-- 11 files changed, 155 insertions(+), 46 deletions(-) diff --git a/python/src/donkey_kit/core/budget.py b/python/src/donkey_kit/core/budget.py index 49467812..f14bbd04 100644 --- a/python/src/donkey_kit/core/budget.py +++ b/python/src/donkey_kit/core/budget.py @@ -196,7 +196,9 @@ async def pace( try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise # no reset time means wait_for_reset() cannot make progress await donkey.budget.wait_for_reset() continue break @@ -211,8 +213,11 @@ async def pace( :attr:`reset_at` in the future makes the reserve guard active again. A response that does not supply a refreshed future :attr:`reset_at` leaves the stale pass-through open. Before ``reset_at`` — or when no reset time was - observed — the reserve guard remains active. ``now`` is injectable for - tests; production passes nothing and the wall clock (UTC) is used. + observed — the reserve guard remains active. The no-reset state does not age + out on its own: there is no known safe time to release the guard, so the + caller must handle or propagate :class:`BudgetReserveReached` rather than + retrying through :meth:`pace`. ``now`` is injectable for tests; production + passes nothing and the wall clock (UTC) is used. """ if not 0.0 <= reserve <= 1.0: raise ValueError(f"reserve must be within [0.0, 1.0], got {reserve!r}") @@ -233,10 +238,13 @@ async def wait_for_reset(self, *, now: datetime | None = None) -> None: """Sleep until :attr:`reset_at`, then return — the recovery half of pacing (BG §1.3, #186). - A single sleep, never a spin loop. If the window is unobserved + A single sleep, never a spin loop. If the reset time is unobserved (:attr:`reset_at` is ``None``) or already past, this returns immediately — - there is nothing to wait for. ``now`` is injectable for tests; production - passes nothing and the wall clock (UTC) is used. + there is nothing to wait for. Callers retrying guarded work must propagate + :class:`BudgetReserveReached` when its ``reset_at`` is ``None``; otherwise + an unconditional wait-and-retry loop cannot make progress. ``now`` is + injectable for tests; production passes nothing and the wall clock (UTC) + is used. """ if self.reset_at is None: return diff --git a/python/src/donkey_kit/core/errors.py b/python/src/donkey_kit/core/errors.py index 4e02de61..9706a2fd 100644 --- a/python/src/donkey_kit/core/errors.py +++ b/python/src/donkey_kit/core/errors.py @@ -162,14 +162,23 @@ class BudgetReserveReached(DonkeyError): Deliberately NOT a :class:`PolicyViolation`: that base marks a gateway-enforced refusal that is terminal and never retried. This is the client-side opposite — a pre-emptive signal, raised locally from observed budget headers before any - request is sent, that the caller is *expected* to recover from (typically - ``await budget.wait_for_reset()`` then continue). Classifying it as a refusal - would misrepresent it at the framework boundary. + request is sent. When :attr:`reset_at` is known, the caller can recover with + ``await budget.wait_for_reset()`` then continue. When it is ``None``, waiting + cannot make progress, so the caller must handle or propagate the signal instead + of retrying immediately. Classifying it as a refusal would misrepresent it at + the framework boundary. Carries the observed budget state at the moment pacing tripped so a handler can decide without re-reading the object: ``fraction_used`` (0.0-1.0), the - ``reserve`` that was requested, and ``reset_at`` (``None`` if the window has not - been observed yet).""" + ``reserve`` that was requested, and ``reset_at`` (``None`` if the reset time has + not been observed).""" + + remediation: str = ( + "When reset_at is known, call await budget.wait_for_reset() before retrying. " + "When reset_at is None, waiting cannot make progress; preserve the last " + "checkpoint and handle or propagate this exception instead of retrying " + "immediately." + ) def __init__( self, diff --git a/python/tests/unit/test_budget_pacing.py b/python/tests/unit/test_budget_pacing.py index 410a053b..20d51430 100644 --- a/python/tests/unit/test_budget_pacing.py +++ b/python/tests/unit/test_budget_pacing.py @@ -244,6 +244,34 @@ async def test_pace_allows_probe_after_wait_without_manual_observe( assert ran_probe +async def test_documented_retry_loop_raises_when_reset_time_is_unknown( + recorded_sleeps: list[float], +) -> None: + """Regression #487: partial budget state cannot hot-spin when there is no + reset time for ``wait_for_reset()`` to wait on.""" + b = Budget() + _observe(b, limit=20000, remaining=500) # 97.5% used, reset_at unknown + attempts = 0 + + with pytest.raises(BudgetReserveReached) as exc: + while True: + try: + attempts += 1 + async with b.pace(reserve=0.05, now=_FIXED_NOW): + raise AssertionError("body must not run once the reserve is reached") + except BudgetReserveReached as error: + if error.reset_at is None: + raise + await b.wait_for_reset() + continue + break + + assert exc.value.reset_at is None + assert exc.value.remediation is BudgetReserveReached.remediation + assert attempts == 1 + assert recorded_sleeps == [] + + async def test_signal_free_response_leaves_expired_window_open() -> None: """BG §1.3: once reset_at has elapsed, a response with no budget signal leaves the stale pass-through open rather than restoring the old refusal.""" diff --git a/spec/donkey-development-kit-build-guide.md b/spec/donkey-development-kit-build-guide.md index 8e973090..5a6370a3 100644 --- a/spec/donkey-development-kit-build-guide.md +++ b/spec/donkey-development-kit-build-guide.md @@ -147,7 +147,9 @@ for batch in chunks(records, 200): try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise # no reset time means waiting cannot make progress await donkey.budget.wait_for_reset() continue break @@ -158,7 +160,11 @@ After `reset_at`, the old observation is stale, so `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future `reset_at` leaves the stale pass-through open. The job can therefore -continue overnight without a manual budget observation. +continue overnight without a manual budget observation. If a partial observation +reaches the reserve without a `reset_at`, the loop re-raises +`BudgetReserveReached` instead of calling `wait_for_reset()` and spinning at zero +delay. The caller can preserve its last checkpoint and escalate the incomplete +budget signal without crossing its reserve. **Why it matters — Scenario A.** A dashboard shows `fraction_used` per agent. The support agent's owner sees it climbing at 14:00 and requests an increase before the 16:00 peak instead of after the outage. @@ -168,6 +174,7 @@ continue overnight without a manual budget observation. - `x-token-reset` in milliseconds is converted correctly (fixture with a known value; assert `reset_at` to the second). - `pace()` raises before the request that would cross the reserve, not after a 429. - After `reset_at`, `pace()` no longer refuses; the documented `pace()` → `wait_for_reset()` → retry loop makes progress without a manual budget observation. A later response updates the budget only when it carries a recognised signal, and the guard becomes active again when that update supplies a future `reset_at`. +- When the reserve is reached but `reset_at` is unknown, the documented loop re-raises `BudgetReserveReached` after one attempt instead of retrying at zero delay. - Budget object is per-`Donkey`, not global; two `Donkey` instances with different credentials do not share state. **Effort:** S. @@ -390,11 +397,17 @@ donkey.on(PolicyViolation).escalate(to=hitl_queue) # catch-all ```python # BEFORE — 1.3: recovery written inside every loop that touches the model for batch in chunks(records, 200): - try: - async with donkey.budget.pace(reserve=0.05): - await enrich(batch); checkpoint(batch) - except BudgetReserveReached: - await donkey.budget.wait_for_reset(); continue + while True: + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(batch) + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise + await donkey.budget.wait_for_reset() + continue + break + checkpoint(batch) # AFTER — 2.1: recovery declared once, runs at the transport donkey.on(TokenBudgetExceeded).wait_for_reset() @@ -408,7 +421,7 @@ for batch in chunks(records, 200): **Why it matters — Scenario A.** Twelve LangGraph nodes call the model. Without handlers, each node needs the same PII try/except. With handlers, it is defined once and applied at the transport. -**Rule to enforce.** Handlers *react*; they never *decide policy*. A handler cannot un-refuse a request. If someone proposes `donkey.on(PIIDetected).ignore()`, that is client-side enforcement by another name — reject it in review. +**Rule to enforce.** Handlers *react*; they never *decide policy*. A handler cannot un-refuse a request. If someone proposes `donkey.on(PIIDetected).ignore()`, that is client-side enforcement by another name — reject it in review. Every wait-and-retry handler must also prove it can wait: if the refusal carries no reset time, re-raise it rather than calling a no-op wait and retrying at zero delay. `max_times` and `max_wait` remain mandatory outer bounds even when a reset time is present. **Effort:** M. diff --git a/website/content/budget.mdx b/website/content/budget.mdx index b440daec..1c35100f 100644 --- a/website/content/budget.mdx +++ b/website/content/budget.mdx @@ -61,7 +61,9 @@ for batch in chunks(records, 200): try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() continue break @@ -72,7 +74,11 @@ Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future `reset_at` leaves the stale pass-through open. The job can -continue unattended without a manual budget observation. +continue unattended without a manual budget observation. A partial observation +can report usage without reporting `reset_at`; in that defensive case the loop +re-raises `BudgetReserveReached` after one attempt instead of calling +`wait_for_reset()` and spinning at zero delay. Preserve the last checkpoint and +escalate the incomplete signal rather than crossing the reserve. ## The dashboard that prevents the outage @@ -107,6 +113,8 @@ last-known-good. See [Roadmap](/roadmap). continue without a manual budget observation. A later response updates the budget only when it carries a recognised signal, and the guard becomes active again when that update supplies a future `reset_at`. +- If the reserve is reached without a known `reset_at`, the retry loop raises + once instead of spinning at zero delay. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/content/errors.mdx b/website/content/errors.mdx index 382a0add..0d315443 100644 --- a/website/content/errors.mdx +++ b/website/content/errors.mdx @@ -90,10 +90,14 @@ never burn an exhausted budget or replay a blocked prompt. Two more `DonkeyError`s are **client-side signals**, not gateway refusals, so they sit outside the retry question: `BudgetReserveReached` is raised *before* a call by [`donkey.budget.pace()`](/budget) and is meant to be recovered from -(`await donkey.budget.wait_for_reset()`, then continue), and `ModelSubstituted` -reports that a call *succeeded* against a different model than requested (opt-in -via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, -and reports every missing field at once — fix the config and re-run. +(`await donkey.budget.wait_for_reset()`, then continue) when its `.reset_at` is +known. If `.reset_at` is `None`, propagate or handle it instead — waiting returns +immediately and an unconditional retry would spin. Its `.remediation` carries +that branch as an inspectable next step, rather than requiring you to parse the +exception message. `ModelSubstituted` reports that a call *succeeded* against a +different model than requested (opt-in via `on_model_substitution="raise"`). +`ConfigError` is raised locally, pre-flight, and reports every missing field at +once — fix the config and re-run. [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 diff --git a/website/content/scenarios/nightly-batch.mdx b/website/content/scenarios/nightly-batch.mdx index bbf68a29..51cec0b4 100644 --- a/website/content/scenarios/nightly-batch.mdx +++ b/website/content/scenarios/nightly-batch.mdx @@ -68,7 +68,8 @@ python enrich.py The pacing and resume logic is a handful of lines. `pace()` guards each batch; on `BudgetReserveReached` you wait for the window and continue from where you -left off: +left off when `.reset_at` is known. If it is unknown, propagate the signal +instead of retrying at zero delay: ```python import asyncio @@ -82,8 +83,10 @@ async def enrich_all(records, enrich): try: async with donkey.budget.pace(reserve=0.05): await enrich(donkey, batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: # We're within 5% of the window's limit — don't take the 429. + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() # sleeps until reset_at continue # retry the same batch checkpoint(batch) # only advance on success @@ -99,7 +102,9 @@ async def enrich_all(records, enrich): observation is stale, so `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future - `reset_at` leaves the stale pass-through open. See [Budget & pacing](/budget). + `reset_at` leaves the stale pass-through open. If a partial observation reaches + the reserve without a `reset_at`, the loop re-raises after one attempt instead + of spinning at zero delay. See [Budget & pacing](/budget). ## The honest limitation diff --git a/website/public/budget.md b/website/public/budget.md index a3a5e3c9..820d4a2c 100644 --- a/website/public/budget.md +++ b/website/public/budget.md @@ -53,7 +53,9 @@ for batch in chunks(records, 200): try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() continue break @@ -64,7 +66,11 @@ Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future `reset_at` leaves the stale pass-through open. The job can -continue unattended without a manual budget observation. +continue unattended without a manual budget observation. A partial observation +can report usage without reporting `reset_at`; in that defensive case the loop +re-raises `BudgetReserveReached` after one attempt instead of calling +`wait_for_reset()` and spinning at zero delay. Preserve the last checkpoint and +escalate the incomplete signal rather than crossing the reserve. ## The dashboard that prevents the outage @@ -97,6 +103,8 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d continue without a manual budget observation. A later response updates the budget only when it carries a recognised signal, and the guard becomes active again when that update supplies a future `reset_at`. +- If the reserve is reached without a known `reset_at`, the retry loop raises + once instead of spinning at zero delay. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. diff --git a/website/public/errors.md b/website/public/errors.md index 9202a82f..354f7876 100644 --- a/website/public/errors.md +++ b/website/public/errors.md @@ -88,10 +88,14 @@ never burn an exhausted budget or replay a blocked prompt. Two more `DonkeyError`s are **client-side signals**, not gateway refusals, so they sit outside the retry question: `BudgetReserveReached` is raised *before* a call by [`donkey.budget.pace()`](https://donkey-development-kit.github.io/donkey-development-kit/budget.md) and is meant to be recovered from -(`await donkey.budget.wait_for_reset()`, then continue), and `ModelSubstituted` -reports that a call *succeeded* against a different model than requested (opt-in -via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, -and reports every missing field at once — fix the config and re-run. +(`await donkey.budget.wait_for_reset()`, then continue) when its `.reset_at` is +known. If `.reset_at` is `None`, propagate or handle it instead — waiting returns +immediately and an unconditional retry would spin. Its `.remediation` carries +that branch as an inspectable next step, rather than requiring you to parse the +exception message. `ModelSubstituted` reports that a call *succeeded* against a +different model than requested (opt-in via `on_model_substitution="raise"`). +`ConfigError` is raised locally, pre-flight, and reports every missing field at +once — fix the config and re-run. [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 7c9333e7..e8f703d6 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -1985,10 +1985,14 @@ never burn an exhausted budget or replay a blocked prompt. Two more `DonkeyError`s are **client-side signals**, not gateway refusals, so they sit outside the retry question: `BudgetReserveReached` is raised *before* a call by [`donkey.budget.pace()`](https://donkey-development-kit.github.io/donkey-development-kit/budget.md) and is meant to be recovered from -(`await donkey.budget.wait_for_reset()`, then continue), and `ModelSubstituted` -reports that a call *succeeded* against a different model than requested (opt-in -via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, -and reports every missing field at once — fix the config and re-run. +(`await donkey.budget.wait_for_reset()`, then continue) when its `.reset_at` is +known. If `.reset_at` is `None`, propagate or handle it instead — waiting returns +immediately and an unconditional retry would spin. Its `.remediation` carries +that branch as an inspectable next step, rather than requiring you to parse the +exception message. `ModelSubstituted` reports that a call *succeeded* against a +different model than requested (opt-in via `on_model_substitution="raise"`). +`ConfigError` is raised locally, pre-flight, and reports every missing field at +once — fix the config and re-run. [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 @@ -2168,7 +2172,9 @@ for batch in chunks(records, 200): try: async with donkey.budget.pace(reserve=0.05): await enrich(batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() continue break @@ -2179,7 +2185,11 @@ Once `reset_at` has elapsed, the old observation is stale and `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future `reset_at` leaves the stale pass-through open. The job can -continue unattended without a manual budget observation. +continue unattended without a manual budget observation. A partial observation +can report usage without reporting `reset_at`; in that defensive case the loop +re-raises `BudgetReserveReached` after one attempt instead of calling +`wait_for_reset()` and spinning at zero delay. Preserve the last checkpoint and +escalate the incomplete signal rather than crossing the reserve. ## The dashboard that prevents the outage @@ -2212,6 +2222,8 @@ last-known-good. See [Roadmap](https://donkey-development-kit.github.io/donkey-d continue without a manual budget observation. A later response updates the budget only when it carries a recognised signal, and the guard becomes active again when that update supplies a future `reset_at`. +- If the reserve is reached without a known `reset_at`, the retry loop raises + once instead of spinning at zero delay. - The budget object is per-`Donkey`, not global: two instances with different credentials do not share state. @@ -3268,7 +3280,8 @@ python enrich.py The pacing and resume logic is a handful of lines. `pace()` guards each batch; on `BudgetReserveReached` you wait for the window and continue from where you -left off: +left off when `.reset_at` is known. If it is unknown, propagate the signal +instead of retrying at zero delay: ```python import asyncio @@ -3282,8 +3295,10 @@ async def enrich_all(records, enrich): try: async with donkey.budget.pace(reserve=0.05): await enrich(donkey, batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: # We're within 5% of the window's limit — don't take the 429. + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() # sleeps until reset_at continue # retry the same batch checkpoint(batch) # only advance on success @@ -3298,7 +3313,9 @@ async def enrich_all(records, enrich): observation is stale, so `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future - `reset_at` leaves the stale pass-through open. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + `reset_at` leaves the stale pass-through open. If a partial observation reaches + the reserve without a `reset_at`, the loop re-raises after one attempt instead + of spinning at zero delay. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation diff --git a/website/public/scenarios/nightly-batch.md b/website/public/scenarios/nightly-batch.md index ac643908..0b248516 100644 --- a/website/public/scenarios/nightly-batch.md +++ b/website/public/scenarios/nightly-batch.md @@ -58,7 +58,8 @@ python enrich.py The pacing and resume logic is a handful of lines. `pace()` guards each batch; on `BudgetReserveReached` you wait for the window and continue from where you -left off: +left off when `.reset_at` is known. If it is unknown, propagate the signal +instead of retrying at zero delay: ```python import asyncio @@ -72,8 +73,10 @@ async def enrich_all(records, enrich): try: async with donkey.budget.pace(reserve=0.05): await enrich(donkey, batch) - except BudgetReserveReached: + except BudgetReserveReached as exc: # We're within 5% of the window's limit — don't take the 429. + if exc.reset_at is None: + raise # waiting cannot make progress without a reset time await donkey.budget.wait_for_reset() # sleeps until reset_at continue # retry the same batch checkpoint(batch) # only advance on success @@ -88,7 +91,9 @@ async def enrich_all(records, enrich): observation is stale, so `pace()` no longer refuses. A response carrying a budget signal updates the observed fields; a fresh future `reset_at` makes the guard active again. A response that does not supply a fresh future - `reset_at` leaves the stale pass-through open. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). + `reset_at` leaves the stale pass-through open. If a partial observation reaches + the reserve without a `reset_at`, the loop re-raises after one attempt instead + of spinning at zero delay. See [Budget & pacing](https://donkey-development-kit.github.io/donkey-development-kit/budget.md). ## The honest limitation From 5d461ba003151a8e15b7519e19b585da1663b43d Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 17:31:56 +0200 Subject: [PATCH 26/29] docs(testing): publish gateway identity exemptions (#479) Closes #454 --- ARCHITECTURE.md | 16 +-- README.md | 2 + python/README.md | 2 + website/content/concepts/attribution.mdx | 11 +- website/content/concepts/observability.mdx | 4 +- website/content/frameworks/adk.mdx | 13 +- .../content/frameworks/agent-framework.mdx | 19 ++- website/content/frameworks/crewai.mdx | 15 ++- website/content/frameworks/index.mdx | 8 +- website/content/frameworks/llamaindex.mdx | 14 ++- website/content/telemetry.mdx | 31 ++++- website/public/concepts/attribution.md | 11 +- website/public/concepts/observability.md | 4 +- website/public/frameworks.md | 8 +- website/public/frameworks/adk.md | 13 +- website/public/frameworks/agent-framework.md | 19 ++- website/public/frameworks/crewai.md | 15 ++- website/public/frameworks/llamaindex.md | 14 ++- website/public/llms-full.txt | 115 ++++++++++++++---- website/public/telemetry.md | 31 ++++- 20 files changed, 286 insertions(+), 79 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d83d537c..88d7a37a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -241,18 +241,20 @@ up front. The demotion landed in #197; deepening LangGraph is tracked in #198. (`python/tests/conformance/suite.py`) runs identically against every adapter. A framework is "supported" only when it passes every scenario **or** records an *asserted exemption* in `KNOWN_LIMITATIONS` — never a silent skip. Those -exemptions are published in the README as credibility (e.g. adapters that reach -models through LiteLLM cannot propagate a per-run correlation ID, because LiteLLM -owns the transport). +exemptions are published in the README as credibility. ADK and CrewAI cannot +propagate a per-run correlation ID or populate `donkey.last_call`, because +LiteLLM owns the transport. LlamaIndex and Microsoft Agent Framework cannot +populate `donkey.last_call`, because they receive only `default_headers`, not +the SDK's shared HTTP client. The centre of gravity moves with the roster cut (`BG §1.5`): the internal matrix shrinks to LangGraph, and the deliverable becomes the **customer-facing pytest plugin** users run against their own agent (#191). -Two adapters carry documented conformance exemptions: CrewAI (per-run -correlation degrades to per-client because it reaches models through LiteLLM, -like ADK) and the Anthropic SDK (depends on the proxy exposing an -Anthropic-native Messages API route, an open verification item). +Four adapters carry documented conformance exemptions: ADK and CrewAI for +per-run correlation and gateway-identity observation, plus LlamaIndex and +Microsoft Agent Framework for gateway-identity observation. The conformance +suite pins those exemptions to each adapter's actual transport behavior. --- diff --git a/README.md b/README.md index 76684fda..648721f2 100644 --- a/README.md +++ b/README.md @@ -124,3 +124,5 @@ legitimately cannot satisfy a scenario, the reason is asserted in code | Framework | Scenario | Why it's exempt | | --- | --- | --- | | ADK, CrewAI | correlation ID propagated | LiteLLM owns the transport, so the SDK's `httpx` client cannot be injected — the correlation ID ends up per-client, not per-run. A LiteLLM logger callback may recover trace correlation later. | +| ADK, CrewAI | gateway identity observed | LiteLLM owns the transport, so no response reaches the SDK's `_on_response` hook. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | +| LlamaIndex, Microsoft Agent Framework | gateway identity observed | These adapters receive `default_headers`, not the SDK's `httpx` client, so no response reaches `_on_response`. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | diff --git a/python/README.md b/python/README.md index bfbac14b..60605d3c 100644 --- a/python/README.md +++ b/python/README.md @@ -124,3 +124,5 @@ legitimately cannot satisfy a scenario, the reason is asserted in code | Framework | Scenario | Why it's exempt | | --- | --- | --- | | ADK, CrewAI | correlation ID propagated | LiteLLM owns the transport, so the SDK's `httpx` client cannot be injected — the correlation ID ends up per-client, not per-run. A LiteLLM logger callback may recover trace correlation later. | +| ADK, CrewAI | gateway identity observed | LiteLLM owns the transport, so no response reaches the SDK's `_on_response` hook. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | +| LlamaIndex, Microsoft Agent Framework | gateway identity observed | These adapters receive `default_headers`, not the SDK's `httpx` client, so no response reaches `_on_response`. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | diff --git a/website/content/concepts/attribution.mdx b/website/content/concepts/attribution.mdx index fb25cc4f..c68d6c1e 100644 --- a/website/content/concepts/attribution.mdx +++ b/website/content/concepts/attribution.mdx @@ -21,7 +21,9 @@ bespoke header the SDK invents. - **Token usage** for cost attribution comes from the response `usage` block (`input_tokens`, `output_tokens`, `total_tokens`, plus the detail counts `cached_tokens` / `cache_write_tokens` / `reasoning_tokens`), passed through - verbatim from the provider and surfaced per-call on `donkey.last_call`. + verbatim from the provider and surfaced per-call on `donkey.last_call` when + the response passes through the SDK's shared HTTP client. See + [when `last_call` is unavailable](/telemetry#when-last_call-is-unavailable). ## Optional attribution metadata @@ -49,10 +51,9 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on for the full two-id model. - **Google ADK caveat.** LiteLLM owns ADK's transport, so the SDK can't inject - its HTTP client there — the correlation id is per-client, not per-run. This is - a documented, asserted conformance exemption, not a silent gap. Every other - framework gets per-run correlation. + **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK + can't inject its HTTP client — the correlation id is per-client, not per-run. + These are documented, asserted conformance exemptions, not silent gaps. diff --git a/website/content/concepts/observability.mdx b/website/content/concepts/observability.mdx index a24c3fd5..2efffc01 100644 --- a/website/content/concepts/observability.mdx +++ b/website/content/concepts/observability.mdx @@ -141,7 +141,9 @@ takes, it produces **exactly one span**, and that span closes: span stays open until the stream finishes and is closed **exactly once**, whether you drain it fully, abandon it mid-iteration, or it raises partway through. `gen_ai.usage.*` and the `donkey.usage.*` detail counts are populated - from that terminal event (and merged onto `donkey.last_call` at the same time). + from that terminal event (and merged onto `donkey.last_call` at the same time + when the response passes through the SDK's shared HTTP client; see + [when `last_call` is unavailable](/telemetry#when-last_call-is-unavailable)). Streaming token counts appear **only if the stream actually carries a usage diff --git a/website/content/frameworks/adk.mdx b/website/content/frameworks/adk.mdx index 8b68d483..2ae14bc0 100644 --- a/website/content/frameworks/adk.mdx +++ b/website/content/frameworks/adk.mdx @@ -7,9 +7,10 @@ Google's Agent Development Kit (ADK) is Gemini-first; it reaches the Agent Fabric LLM proxy through ADK's `LiteLlm` model wrapper, which speaks LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below). +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below). ## Install @@ -102,6 +103,14 @@ you. > ADK is the one adapter in the set where header injection is full but > transport injection is not possible. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + > `google-adk` requires `litellm>=1.84` as a floor (not a ceiling) — pin your > own upper bound if you need one. diff --git a/website/content/frameworks/agent-framework.mdx b/website/content/frameworks/agent-framework.mdx index a9d065a1..81b272a9 100644 --- a/website/content/frameworks/agent-framework.mdx +++ b/website/content/frameworks/agent-framework.mdx @@ -7,10 +7,12 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter also ships the strongest -> policy-integration story of the eight frameworks (`policy_middleware()`), -> but its native client class is young and **UNVERIFIED** — read the callout -> below before depending on it. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented +> exemption: it receives `default_headers`, not the SDK's shared httpx client, +> so `donkey.last_call` response observation is unavailable. It also ships the +> strongest policy-integration story of the eight frameworks +> (`policy_middleware()`), but its native client class is young and +> **UNVERIFIED** — read the callouts below before depending on it. ## Install @@ -111,6 +113,15 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. +> **This adapter cannot populate `donkey.last_call`.** Agent Framework +> receives the governed `default_headers`, but not the SDK's httpx client, so no +> response reaches `_on_response` and gateway identity, routing, and usage +> fields cannot be observed. When every adapter resolved on a `Donkey` is +> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, +> `available == False`, and names the resolved adapters in `surface`. +> This is a documented, asserted `gateway_identity_observed` conformance +> exemption. + See the [error taxonomy](/errors) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the [verification policy](/concepts/verification) page for the current diff --git a/website/content/frameworks/crewai.mdx b/website/content/frameworks/crewai.mdx index d7f49db1..b64f425b 100644 --- a/website/content/frameworks/crewai.mdx +++ b/website/content/frameworks/crewai.mdx @@ -7,10 +7,11 @@ CrewAI gets a governed `LLM`, backed by LiteLLM under the hood, pointed at the Agent Fabric LLM proxy through LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below) — the same exemption as the [Google ADK](/frameworks/adk) -> adapter. +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below) — the same exemptions as the +> [Google ADK](/frameworks/adk) adapter. ## Install @@ -102,6 +103,14 @@ you. > header injection is full, but transport injection is not possible, exactly > as with Google ADK. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + See the [error taxonomy](/errors) for how proxy rejections surface through LiteLLM's error path, and the [verification policy](/concepts/verification) page for the current status of every constructor signature this adapter diff --git a/website/content/frameworks/index.mdx b/website/content/frameworks/index.mdx index 20addf7d..216c2584 100644 --- a/website/content/frameworks/index.mdx +++ b/website/content/frameworks/index.mdx @@ -99,12 +99,12 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ✅ | `is_chat_model=True` forced (avoids the completions-endpoint gotcha). | +| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](/frameworks/anthropic). | -| MS Agent Framework | ✅ | ✅ | Class name/kwarg **unverified**. | -| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport — correlation-ID is per-client, a documented exemption. | -| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM-backed `LLM`; same correlation-ID exemption as Google ADK. | +| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | +| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | The proxy contract is verified; the exact **constructor signatures** are still diff --git a/website/content/frameworks/llamaindex.mdx b/website/content/frameworks/llamaindex.mdx index 5b9710b6..0a1f97bb 100644 --- a/website/content/frameworks/llamaindex.mdx +++ b/website/content/frameworks/llamaindex.mdx @@ -8,8 +8,10 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection; watch the -> `is_chat_model` gotcha below if you ever build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one +> documented exemption: LlamaIndex receives `default_headers`, not the SDK's +> shared httpx client, so `donkey.last_call` response observation is unavailable +> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. ## Install @@ -102,4 +104,12 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. +> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives +> the governed `default_headers`, but not the SDK's httpx client, so no response +> reaches `_on_response` and gateway identity, routing, and usage fields cannot +> be observed. When every adapter resolved on a `Donkey` is non-observing, the +> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. This is a documented, asserted +> `gateway_identity_observed` conformance exemption. + diff --git a/website/content/telemetry.mdx b/website/content/telemetry.mdx index 7e6253ae..0391a120 100644 --- a/website/content/telemetry.mdx +++ b/website/content/telemetry.mdx @@ -45,7 +45,8 @@ The three `donkey.usage.*` counts carry the cost-relevant detail tokens the semconv has no pinned key for — cached / cache-write prompt tokens and reasoning-model thinking tokens. They are read from the response `usage` block's detail sub-objects and are **omitted, never `0`,** when the provider reports no -detail counts. The same counts are exposed per-call on `donkey.last_call`. +detail counts. When the response passes through the SDK's shared HTTP client, +the same counts are exposed per-call on `donkey.last_call`. Export goes over OTLP to wherever you already send spans. **Nothing in the emit path is Anypoint-specific.** @@ -150,8 +151,8 @@ that was a **fallback**. Donkey reads those signals off the shared transport, so you get them with **no framework required** — the raw `donkey.llm.client()` path benefits just as the deep adapters do. -Every governed call exposes them on `donkey.last_call`, beside the usage and -identity fields: +Every governed call through the SDK's shared HTTP client exposes them on +`donkey.last_call`, beside the usage and identity fields: ```python donkey = Donkey.from_env() @@ -171,6 +172,30 @@ They also land on the span (`gen_ai.response.model`, `donkey.routing.type`, latency spikes: it tells an operator whether a slow call was routed normally or recovered from a degraded provider. +### When `last_call` is unavailable + +`donkey.last_call` is populated only when the governed response passes through +the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route +outside that response path: ADK and CrewAI use LiteLLM's transport, while +LlamaIndex and Microsoft Agent Framework receive only `default_headers`. + +When every adapter resolved on a `Donkey` is one of those four, a cold read +reports the limitation explicitly. For a `Donkey` that resolved only ADK: + +```python +r = donkey.last_call +r.status # LastCallStatus.UNAVAILABLE +r.available # False +r.surface # "adk" +``` + +This is different from `UNOBSERVED`, which means the current context has not +yet received a governed response. On an unavailable surface the SDK cannot +observe any response-derived `last_call` field, including gateway identity, +routing, fallback, and usage. If multiple non-observing adapters were resolved, +`surface` lists their names. Each limitation is asserted as the +`gateway_identity_observed` conformance exemption rather than silently skipped. + ### Two behaviours worth knowing **The SDK never double-retries a fallback.** Donkey retries `502/503/504` with diff --git a/website/public/concepts/attribution.md b/website/public/concepts/attribution.md index a0705b57..9ba90496 100644 --- a/website/public/concepts/attribution.md +++ b/website/public/concepts/attribution.md @@ -19,7 +19,9 @@ bespoke header the SDK invents. - **Token usage** for cost attribution comes from the response `usage` block (`input_tokens`, `output_tokens`, `total_tokens`, plus the detail counts `cached_tokens` / `cache_write_tokens` / `reasoning_tokens`), passed through - verbatim from the provider and surfaced per-call on `donkey.last_call`. + verbatim from the provider and surfaced per-call on `donkey.last_call` when + the response passes through the SDK's shared HTTP client. See + [when `last_call` is unavailable](https://donkey-development-kit.github.io/donkey-development-kit/telemetry.md#when-last_call-is-unavailable). ## Optional attribution metadata @@ -46,10 +48,9 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on `DonkeyError.call_id`. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md#grouping-a-run-the-run-id-on-the-span-the-call-id-on-the-error) for the full two-id model. - **Google ADK caveat.** LiteLLM owns ADK's transport, so the SDK can't inject - its HTTP client there — the correlation id is per-client, not per-run. This is - a documented, asserted conformance exemption, not a silent gap. Every other - framework gets per-run correlation. + **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK + can't inject its HTTP client — the correlation id is per-client, not per-run. + These are documented, asserted conformance exemptions, not silent gaps. The **agent→agent egress** attribution header (`x-anypoint-api-instance-id`) is a separate telemetry path for component-to-component traffic, not needed for diff --git a/website/public/concepts/observability.md b/website/public/concepts/observability.md index dd4d5c0e..a5fa1f04 100644 --- a/website/public/concepts/observability.md +++ b/website/public/concepts/observability.md @@ -129,7 +129,9 @@ takes, it produces **exactly one span**, and that span closes: span stays open until the stream finishes and is closed **exactly once**, whether you drain it fully, abandon it mid-iteration, or it raises partway through. `gen_ai.usage.*` and the `donkey.usage.*` detail counts are populated - from that terminal event (and merged onto `donkey.last_call` at the same time). + from that terminal event (and merged onto `donkey.last_call` at the same time + when the response passes through the SDK's shared HTTP client; see + [when `last_call` is unavailable](https://donkey-development-kit.github.io/donkey-development-kit/telemetry.md#when-last_call-is-unavailable)). Streaming token counts appear **only if the stream actually carries a usage event.** For OpenAI-style Chat Completions that means requesting it with diff --git a/website/public/frameworks.md b/website/public/frameworks.md index 046a7f61..4551df50 100644 --- a/website/public/frameworks.md +++ b/website/public/frameworks.md @@ -86,12 +86,12 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ✅ | `is_chat_model=True` forced (avoids the completions-endpoint gotcha). | +| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/anthropic.md). | -| MS Agent Framework | ✅ | ✅ | Class name/kwarg **unverified**. | -| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport — correlation-ID is per-client, a documented exemption. | -| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM-backed `LLM`; same correlation-ID exemption as Google ADK. | +| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | +| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | The proxy contract is verified; the exact **constructor signatures** are still being confirmed per framework — see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). diff --git a/website/public/frameworks/adk.md b/website/public/frameworks/adk.md index d1c02aeb..82e6fd1e 100644 --- a/website/public/frameworks/adk.md +++ b/website/public/frameworks/adk.md @@ -4,9 +4,10 @@ Google's Agent Development Kit (ADK) is Gemini-first; it reaches the Agent Fabric LLM proxy through ADK's `LiteLlm` model wrapper, which speaks LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below). +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below). ## Install @@ -107,6 +108,14 @@ you. > ADK is the one adapter in the set where header injection is full but > transport injection is not possible. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + > `google-adk` requires `litellm>=1.84` as a floor (not a ceiling) — pin your > own upper bound if you need one. diff --git a/website/public/frameworks/agent-framework.md b/website/public/frameworks/agent-framework.md index 130951a6..7b0c731d 100644 --- a/website/public/frameworks/agent-framework.md +++ b/website/public/frameworks/agent-framework.md @@ -4,10 +4,12 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter also ships the strongest -> policy-integration story of the eight frameworks (`policy_middleware()`), -> but its native client class is young and **UNVERIFIED** — read the callout -> below before depending on it. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented +> exemption: it receives `default_headers`, not the SDK's shared httpx client, +> so `donkey.last_call` response observation is unavailable. It also ships the +> strongest policy-integration story of the eight frameworks +> (`policy_middleware()`), but its native client class is young and +> **UNVERIFIED** — read the callouts below before depending on it. ## Install @@ -116,6 +118,15 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. +> **This adapter cannot populate `donkey.last_call`.** Agent Framework +> receives the governed `default_headers`, but not the SDK's httpx client, so no +> response reaches `_on_response` and gateway identity, routing, and usage +> fields cannot be observed. When every adapter resolved on a `Donkey` is +> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, +> `available == False`, and names the resolved adapters in `surface`. +> This is a documented, asserted `gateway_identity_observed` conformance +> exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current diff --git a/website/public/frameworks/crewai.md b/website/public/frameworks/crewai.md index 32a5a388..6cf896b6 100644 --- a/website/public/frameworks/crewai.md +++ b/website/public/frameworks/crewai.md @@ -4,10 +4,11 @@ CrewAI gets a governed `LLM`, backed by LiteLLM under the hood, pointed at the Agent Fabric LLM proxy through LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below) — the same exemption as the [Google ADK](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/adk.md) -> adapter. +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below) — the same exemptions as the +> [Google ADK](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/adk.md) adapter. ## Install @@ -107,6 +108,14 @@ you. > header injection is full, but transport injection is not possible, exactly > as with Google ADK. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface through LiteLLM's error path, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current status of every constructor signature this adapter diff --git a/website/public/frameworks/llamaindex.md b/website/public/frameworks/llamaindex.md index e7442db4..228fbfef 100644 --- a/website/public/frameworks/llamaindex.md +++ b/website/public/frameworks/llamaindex.md @@ -4,8 +4,10 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection; watch the -> `is_chat_model` gotcha below if you ever build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one +> documented exemption: LlamaIndex receives `default_headers`, not the SDK's +> shared httpx client, so `donkey.last_call` response observation is unavailable +> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. ## Install @@ -106,6 +108,14 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. +> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives +> the governed `default_headers`, but not the SDK's httpx client, so no response +> reaches `_on_response` and gateway identity, routing, and usage fields cannot +> be observed. When every adapter resolved on a `Donkey` is non-observing, the +> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. This is a documented, asserted +> `gateway_identity_observed` conformance exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface as typed exceptions, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current status of every constructor signature this adapter depends on. diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index e8f703d6..038758d5 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -814,12 +814,12 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ✅ | `is_chat_model=True` forced (avoids the completions-endpoint gotcha). | +| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/anthropic.md). | -| MS Agent Framework | ✅ | ✅ | Class name/kwarg **unverified**. | -| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport — correlation-ID is per-client, a documented exemption. | -| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM-backed `LLM`; same correlation-ID exemption as Google ADK. | +| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | +| CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | The proxy contract is verified; the exact **constructor signatures** are still being confirmed per framework — see [Verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md). @@ -1032,9 +1032,10 @@ Google's Agent Development Kit (ADK) is Gemini-first; it reaches the Agent Fabric LLM proxy through ADK's `LiteLlm` model wrapper, which speaks LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below). +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below). ## Install @@ -1135,6 +1136,14 @@ you. > ADK is the one adapter in the set where header injection is full but > transport injection is not possible. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + > `google-adk` requires `litellm>=1.84` as a floor (not a ceiling) — pin your > own upper bound if you need one. @@ -1272,10 +1281,12 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter also ships the strongest -> policy-integration story of the eight frameworks (`policy_middleware()`), -> but its native client class is young and **UNVERIFIED** — read the callout -> below before depending on it. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented +> exemption: it receives `default_headers`, not the SDK's shared httpx client, +> so `donkey.last_call` response observation is unavailable. It also ships the +> strongest policy-integration story of the eight frameworks +> (`policy_middleware()`), but its native client class is young and +> **UNVERIFIED** — read the callouts below before depending on it. ## Install @@ -1384,6 +1395,15 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. +> **This adapter cannot populate `donkey.last_call`.** Agent Framework +> receives the governed `default_headers`, but not the SDK's httpx client, so no +> response reaches `_on_response` and gateway identity, routing, and usage +> fields cannot be observed. When every adapter resolved on a `Donkey` is +> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, +> `available == False`, and names the resolved adapters in `surface`. +> This is a documented, asserted `gateway_identity_observed` conformance +> exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current @@ -1667,10 +1687,11 @@ CrewAI gets a governed `LLM`, backed by LiteLLM under the hood, pointed at the Agent Fabric LLM proxy through LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with two documented exemptions: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected -> here (see below) — the same exemption as the [Google ADK](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/adk.md) -> adapter. +> here. Per-run correlation and `donkey.last_call` response observation are +> unavailable as a result (see below) — the same exemptions as the +> [Google ADK](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/adk.md) adapter. ## Install @@ -1770,6 +1791,14 @@ you. > header injection is full, but transport injection is not possible, exactly > as with Google ADK. +> **This adapter cannot populate `donkey.last_call`.** The same LiteLLM-owned +> transport means no response reaches the SDK's `_on_response` hook, so gateway +> identity, routing, and usage fields cannot be observed. When every adapter +> resolved on a `Donkey` is non-observing, the record reports +> `status == LastCallStatus.UNAVAILABLE`, `available == False`, and names the +> resolved adapters in `surface`. This is a +> documented, asserted `gateway_identity_observed` conformance exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface through LiteLLM's error path, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current status of every constructor signature this adapter @@ -1785,8 +1814,10 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection; watch the -> `is_chat_model` gotcha below if you ever build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one +> documented exemption: LlamaIndex receives `default_headers`, not the SDK's +> shared httpx client, so `donkey.last_call` response observation is unavailable +> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. ## Install @@ -1887,6 +1918,14 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. +> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives +> the governed `default_headers`, but not the SDK's httpx client, so no response +> reaches `_on_response` and gateway identity, routing, and usage fields cannot +> be observed. When every adapter resolved on a `Donkey` is non-observing, the +> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. This is a documented, asserted +> `gateway_identity_observed` conformance exemption. + See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface as typed exceptions, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for the current status of every constructor signature this adapter depends on. @@ -2667,7 +2706,8 @@ The three `donkey.usage.*` counts carry the cost-relevant detail tokens the semconv has no pinned key for — cached / cache-write prompt tokens and reasoning-model thinking tokens. They are read from the response `usage` block's detail sub-objects and are **omitted, never `0`,** when the provider reports no -detail counts. The same counts are exposed per-call on `donkey.last_call`. +detail counts. When the response passes through the SDK's shared HTTP client, +the same counts are exposed per-call on `donkey.last_call`. Export goes over OTLP to wherever you already send spans. **Nothing in the emit path is Anypoint-specific.** @@ -2764,8 +2804,8 @@ that was a **fallback**. Donkey reads those signals off the shared transport, so you get them with **no framework required** — the raw `donkey.llm.client()` path benefits just as the deep adapters do. -Every governed call exposes them on `donkey.last_call`, beside the usage and -identity fields: +Every governed call through the SDK's shared HTTP client exposes them on +`donkey.last_call`, beside the usage and identity fields: ```python donkey = Donkey.from_env() @@ -2785,6 +2825,30 @@ They also land on the span (`gen_ai.response.model`, `donkey.routing.type`, latency spikes: it tells an operator whether a slow call was routed normally or recovered from a degraded provider. +### When `last_call` is unavailable + +`donkey.last_call` is populated only when the governed response passes through +the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route +outside that response path: ADK and CrewAI use LiteLLM's transport, while +LlamaIndex and Microsoft Agent Framework receive only `default_headers`. + +When every adapter resolved on a `Donkey` is one of those four, a cold read +reports the limitation explicitly. For a `Donkey` that resolved only ADK: + +```python +r = donkey.last_call +r.status # LastCallStatus.UNAVAILABLE +r.available # False +r.surface # "adk" +``` + +This is different from `UNOBSERVED`, which means the current context has not +yet received a governed response. On an unavailable surface the SDK cannot +observe any response-derived `last_call` field, including gateway identity, +routing, fallback, and usage. If multiple non-observing adapters were resolved, +`surface` lists their names. Each limitation is asserted as the +`gateway_identity_observed` conformance exemption rather than silently skipped. + ### Two behaviours worth knowing **The SDK never double-retries a fallback.** Donkey retries `502/503/504` with @@ -4663,7 +4727,9 @@ bespoke header the SDK invents. - **Token usage** for cost attribution comes from the response `usage` block (`input_tokens`, `output_tokens`, `total_tokens`, plus the detail counts `cached_tokens` / `cache_write_tokens` / `reasoning_tokens`), passed through - verbatim from the provider and surfaced per-call on `donkey.last_call`. + verbatim from the provider and surfaced per-call on `donkey.last_call` when + the response passes through the SDK's shared HTTP client. See + [when `last_call` is unavailable](https://donkey-development-kit.github.io/donkey-development-kit/telemetry.md#when-last_call-is-unavailable). ## Optional attribution metadata @@ -4690,10 +4756,9 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on `DonkeyError.call_id`. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md#grouping-a-run-the-run-id-on-the-span-the-call-id-on-the-error) for the full two-id model. - **Google ADK caveat.** LiteLLM owns ADK's transport, so the SDK can't inject - its HTTP client there — the correlation id is per-client, not per-run. This is - a documented, asserted conformance exemption, not a silent gap. Every other - framework gets per-run correlation. + **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK + can't inject its HTTP client — the correlation id is per-client, not per-run. + These are documented, asserted conformance exemptions, not silent gaps. The **agent→agent egress** attribution header (`x-anypoint-api-instance-id`) is a separate telemetry path for component-to-component traffic, not needed for @@ -4835,7 +4900,9 @@ takes, it produces **exactly one span**, and that span closes: span stays open until the stream finishes and is closed **exactly once**, whether you drain it fully, abandon it mid-iteration, or it raises partway through. `gen_ai.usage.*` and the `donkey.usage.*` detail counts are populated - from that terminal event (and merged onto `donkey.last_call` at the same time). + from that terminal event (and merged onto `donkey.last_call` at the same time + when the response passes through the SDK's shared HTTP client; see + [when `last_call` is unavailable](https://donkey-development-kit.github.io/donkey-development-kit/telemetry.md#when-last_call-is-unavailable)). Streaming token counts appear **only if the stream actually carries a usage event.** For OpenAI-style Chat Completions that means requesting it with diff --git a/website/public/telemetry.md b/website/public/telemetry.md index 5f1003d3..a8856d7a 100644 --- a/website/public/telemetry.md +++ b/website/public/telemetry.md @@ -37,7 +37,8 @@ The three `donkey.usage.*` counts carry the cost-relevant detail tokens the semconv has no pinned key for — cached / cache-write prompt tokens and reasoning-model thinking tokens. They are read from the response `usage` block's detail sub-objects and are **omitted, never `0`,** when the provider reports no -detail counts. The same counts are exposed per-call on `donkey.last_call`. +detail counts. When the response passes through the SDK's shared HTTP client, +the same counts are exposed per-call on `donkey.last_call`. Export goes over OTLP to wherever you already send spans. **Nothing in the emit path is Anypoint-specific.** @@ -134,8 +135,8 @@ that was a **fallback**. Donkey reads those signals off the shared transport, so you get them with **no framework required** — the raw `donkey.llm.client()` path benefits just as the deep adapters do. -Every governed call exposes them on `donkey.last_call`, beside the usage and -identity fields: +Every governed call through the SDK's shared HTTP client exposes them on +`donkey.last_call`, beside the usage and identity fields: ```python donkey = Donkey.from_env() @@ -155,6 +156,30 @@ They also land on the span (`gen_ai.response.model`, `donkey.routing.type`, latency spikes: it tells an operator whether a slow call was routed normally or recovered from a degraded provider. +### When `last_call` is unavailable + +`donkey.last_call` is populated only when the governed response passes through +the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route +outside that response path: ADK and CrewAI use LiteLLM's transport, while +LlamaIndex and Microsoft Agent Framework receive only `default_headers`. + +When every adapter resolved on a `Donkey` is one of those four, a cold read +reports the limitation explicitly. For a `Donkey` that resolved only ADK: + +```python +r = donkey.last_call +r.status # LastCallStatus.UNAVAILABLE +r.available # False +r.surface # "adk" +``` + +This is different from `UNOBSERVED`, which means the current context has not +yet received a governed response. On an unavailable surface the SDK cannot +observe any response-derived `last_call` field, including gateway identity, +routing, fallback, and usage. If multiple non-observing adapters were resolved, +`surface` lists their names. Each limitation is asserted as the +`gateway_identity_observed` conformance exemption rather than silently skipped. + ### Two behaviours worth knowing **The SDK never double-retries a fallback.** Donkey retries `502/503/504` with From bb5483b09dc43b9fff107cd4a476db00eb47c5fe Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 18:11:23 +0200 Subject: [PATCH 27/29] Test remaining verification-blocked surfaces (#505) Closes #501 --- python/tests/unit/test_adapter_ergonomics.py | 2 +- .../test_verification_blocked_surfaces.py | 105 ++++++++++++++++++ 2 files changed, 106 insertions(+), 1 deletion(-) create mode 100644 python/tests/unit/test_verification_blocked_surfaces.py diff --git a/python/tests/unit/test_adapter_ergonomics.py b/python/tests/unit/test_adapter_ergonomics.py index 2c4489b7..44da72a4 100644 --- a/python/tests/unit/test_adapter_ergonomics.py +++ b/python/tests/unit/test_adapter_ergonomics.py @@ -278,7 +278,7 @@ def fail_openai_import(name: str, *args: Any, **kwargs: Any) -> Any: monkeypatch.setattr(builtins, "__import__", fail_openai_import) adapter = AgentFrameworkAdapter(_cfg(), _http()) - with pytest.raises(NotImplementedError, match="blocked on verification") as exc_info: + with pytest.raises(NotImplementedError, match=r"^blocked on verification:") as exc_info: adapter.chat_client("gpt-4o") assert exc_info.value.__cause__ is import_error diff --git a/python/tests/unit/test_verification_blocked_surfaces.py b/python/tests/unit/test_verification_blocked_surfaces.py new file mode 100644 index 00000000..7d593bd8 --- /dev/null +++ b/python/tests/unit/test_verification_blocked_surfaces.py @@ -0,0 +1,105 @@ +"""Regression coverage for verification-blocked SDK surfaces (§0.3). + +CLI ``_blocked(...)`` paths are intentionally not duplicated here: all eight +are covered by ``test_cli_provisioning.py`` from PR #476. When a surface is +verified, remove its row here in the same change that records the VERIFIED date +and source in ``docs/verified-apis.md``. +""" + +from __future__ import annotations + +from collections.abc import Awaitable, Callable + +import pytest + +from donkey_kit import Donkey, DonkeyConfig, Publication, PublicationAssetType +from donkey_kit.core.auth import StaticToken +from donkey_kit.governance import GatewayTarget, Governance +from donkey_kit.provisioning.applier import apply as apply_plan +from donkey_kit.provisioning.planner import Plan, build_plan +from donkey_kit.provisioning.publish import publish_if_changed +from donkey_kit.provisioning.spec import DonkeySpec, SpecMetadata + +_TARGET = GatewayTarget(mode="local", base_url="http://localhost", connected=False) +_GOVERNANCE = Governance(name="blocked-governance", gateway=_TARGET) +_SPEC = DonkeySpec( + metadata=SpecMetadata(name="blocked-spec", environment="Sandbox"), +) +_PLAN = Plan() +_PUBLICATION = Publication( + asset_type=PublicationAssetType.MCP_SERVER, + group_id="com.example", + asset_id="blocked-asset", + version="1.0.0", + name="Blocked asset", + description="An asset whose publication surface remains verification-blocked.", +) +_CONFIG = DonkeyConfig( + telemetry=False, + correlation_header="X-Test-Correlation-Id", + call_id_header="X-Test-Call-Id", +) + + +async def _enter_simulation(_donkey: Donkey) -> None: + async with _GOVERNANCE.simulate(): + pass + + +_SYNC_BLOCKED_SURFACES: tuple[ + tuple[str, Callable[[Donkey], object]], ... +] = ( + ("Donkey.tools.lock", lambda donkey: donkey.tools.lock()), + ("Governance.export", lambda _donkey: _GOVERNANCE.export()), +) + +_ASYNC_BLOCKED_SURFACES: tuple[ + tuple[str, Callable[[Donkey], Awaitable[object]]], ... +] = ( + ("Donkey.tools.discover", lambda donkey: donkey.tools.discover()), + ("Governance.resolve", lambda donkey: _GOVERNANCE.resolve(donkey)), + ( + "Governance.apply", + lambda _donkey: _GOVERNANCE.apply(_TARGET, i_am_the_platform_team=True), + ), + ("SimulationContext.__aenter__", _enter_simulation), + ( + "publish_if_changed", + lambda donkey: publish_if_changed(_PUBLICATION, donkey), + ), + ("build_plan", lambda donkey: build_plan(_SPEC, donkey)), + ("provisioning.apply", lambda donkey: apply_plan(_PLAN, donkey)), +) + + +@pytest.mark.parametrize( + ("_surface", "invoke"), + _SYNC_BLOCKED_SURFACES, + ids=[surface for surface, _ in _SYNC_BLOCKED_SURFACES], +) +async def test_sync_verification_blocked_surfaces_remain_blocked( + _surface: str, + invoke: Callable[[Donkey], object], +) -> None: + async with Donkey(_CONFIG, auth=StaticToken("unused")) as donkey: + with pytest.raises(NotImplementedError, match=r"^blocked on verification:"): + invoke(donkey) + + +@pytest.mark.parametrize( + ("_surface", "invoke"), + _ASYNC_BLOCKED_SURFACES, + ids=[surface for surface, _ in _ASYNC_BLOCKED_SURFACES], +) +async def test_async_verification_blocked_surfaces_remain_blocked( + _surface: str, + invoke: Callable[[Donkey], Awaitable[object]], +) -> None: + async with Donkey(_CONFIG, auth=StaticToken("unused")) as donkey: + with pytest.raises(NotImplementedError, match=r"^blocked on verification:"): + await invoke(donkey) + + +async def test_governance_apply_requires_platform_team_authorization() -> None: + with pytest.raises(PermissionError): + await _GOVERNANCE.apply(_TARGET) From 19189de1ae27bc4f54ea87405cf656b9326c4443 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 18:13:18 +0200 Subject: [PATCH 28/29] Fix AuthError remediation for control-plane failures (#507) Closes #484 --- ARCHITECTURE.md | 5 ++++- python/src/donkey_kit/core/auth.py | 8 ++++++- python/src/donkey_kit/core/errors.py | 30 +++++++++++++++++++++---- python/tests/unit/test_auth.py | 33 +++++++++++++++++++++++++++- python/tests/unit/test_errors.py | 11 ++++++++++ website/content/errors.mdx | 11 ++++++++-- website/public/errors.md | 11 ++++++++-- website/public/llms-full.txt | 11 ++++++++-- 8 files changed, 107 insertions(+), 13 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 88d7a37a..44543b72 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -183,7 +183,10 @@ the SDK's clearest value over raw HTTP. Two invariants govern the taxonomy in 2. **Every error carries a `remediation`.** On `PolicyViolation` the human-readable next step is a *required* field — the concrete action to take (e.g. "the budget window resets in 42m; request an increase in API Manager") is worth more than a - stack trace. + stack trace. `AuthError` uses its class-level LLM-proxy guidance by default and + canonical class-level overrides for connected-app and provider-chain failures, + so its message and remediation always name the same credential plane and auth + provider. **`classify()` is fixture-driven, not guessed.** The HTTP-response → exception mapping in `classify()` is populated from real rejection captures taken against a diff --git a/python/src/donkey_kit/core/auth.py b/python/src/donkey_kit/core/auth.py index 1af5f8ec..ea29ac4e 100644 --- a/python/src/donkey_kit/core/auth.py +++ b/python/src/donkey_kit/core/auth.py @@ -105,6 +105,7 @@ async def _fetch(self) -> str: "Anypoint token request rejected. Verify the connected-app " "client_id/secret and that the app has the scopes the operation " "needs (see docs/verified-apis.md §1).", + remediation=AuthError.connected_app_remediation, response=resp, ) if resp.status_code >= 400: @@ -113,6 +114,7 @@ async def _fetch(self) -> str: f"{self._token_path!r}. Check that the control-plane host is reachable, " "that any token_path override is correct, and that connected-app " "requirements are met (see docs/verified-apis.md §1 and §12.1).", + remediation=AuthError.connected_app_remediation, response=resp, ) body = resp.json() @@ -122,6 +124,7 @@ async def _fetch(self) -> str: "Token endpoint returned no access_token. The expected response shape " "includes access_token and expires_in (see docs/verified-apis.md §1 and " "§12.1); capture the unexpected response as a fixture (BG §1.5).", + remediation=AuthError.connected_app_remediation, response=resp, ) expires_in = float(body.get("expires_in", 3600)) @@ -145,7 +148,10 @@ async def token(self) -> str: return await provider.token() except Exception as exc: # noqa: BLE001 - fall through to next provider last = exc - raise AuthError(f"No auth provider yielded a token. Last error: {last}") + raise AuthError( + f"No auth provider yielded a token. Last error: {last}", + remediation=AuthError.provider_chain_remediation, + ) async def invalidate(self) -> None: for provider in self._providers: diff --git a/python/src/donkey_kit/core/errors.py b/python/src/donkey_kit/core/errors.py index 9706a2fd..657ef874 100644 --- a/python/src/donkey_kit/core/errors.py +++ b/python/src/donkey_kit/core/errors.py @@ -76,10 +76,11 @@ class ConfigError(DonkeyError): class AuthError(DonkeyError): """Rejected credentials on the Anypoint control plane or LLM-proxy data plane. - ``remediation`` is a class attribute (like :class:`GatewayUnavailable`, not - the constructor-enforced :class:`PolicyViolation` contract) so ``donkey - doctor`` (#202) has one canonical wording to print for a credentials - rejection rather than a second copy.""" + The class-level ``remediation`` is the LLM-proxy data-plane default that + ``donkey doctor`` (#202) reuses. Control-plane callers override it with one + of the canonical class values below, so the next step matches the auth + provider that failed (#484). This is not the constructor-enforced + :class:`PolicyViolation` contract.""" #: Single source of next-step wording for a rejected-credentials diagnosis. remediation: str = ( @@ -89,6 +90,27 @@ class AuthError(DonkeyError): "Anypoint control-plane credential and not a bearer token — and that the " "consumer is authorized on the instance in API Manager (docs/verified-apis.md §2)." ) + connected_app_remediation: str = ( + "Check the configured Anypoint control-plane auth provider. For a connected app, " + "verify ANYPOINT_CLIENT_ID / ANYPOINT_CLIENT_SECRET and that the app has the scopes " + "the operation needs (docs/verified-apis.md §1)." + ) + provider_chain_remediation: str = ( + "Every configured Anypoint control-plane auth provider failed. Inspect the last " + "error and verify each provider's credentials or token source. If the chain uses " + "a connected app, also verify its required scopes (docs/verified-apis.md §1)." + ) + + def __init__( + self, + message: str, + *, + remediation: str | None = None, + **kw: Any, + ) -> None: + super().__init__(message, **kw) + if remediation is not None: + self.remediation = remediation class PolicyViolation(DonkeyError): diff --git a/python/tests/unit/test_auth.py b/python/tests/unit/test_auth.py index b1f7476f..dd107f2e 100644 --- a/python/tests/unit/test_auth.py +++ b/python/tests/unit/test_auth.py @@ -7,10 +7,27 @@ import httpx import pytest -from donkey_kit.core.auth import AnypointConnectedApp +from donkey_kit.core.auth import AnypointConnectedApp, ChainedAuth from donkey_kit.core.errors import AuthError +def _assert_control_plane_remediation(error: AuthError) -> None: + assert error.remediation is AuthError.connected_app_remediation + assert "ANYPOINT_CLIENT_ID" in error.remediation + assert "ANYPOINT_CLIENT_SECRET" in error.remediation + assert "scopes" in error.remediation + assert "docs/verified-apis.md §1" in error.remediation + assert "DONKEY_LLM_PROXY_CLIENT_ID" not in error.remediation + + +class _FailingAuth: + async def token(self) -> str: + raise RuntimeError("provider failed") + + async def invalidate(self) -> None: + return None + + async def test_connected_app_posts_credentials_and_caches_until_safety_margin() -> None: now = [100.0] requests: list[httpx.Request] = [] @@ -97,6 +114,7 @@ def handler(request: httpx.Request) -> httpx.Response: await auth.token() assert exc_info.value.response is response + _assert_control_plane_remediation(exc_info.value) async def test_connected_app_maps_other_http_errors_to_auth_error() -> None: @@ -120,6 +138,7 @@ def handler(request: httpx.Request) -> httpx.Response: await auth.token() assert exc_info.value.response is response + _assert_control_plane_remediation(exc_info.value) async def test_connected_app_rejects_success_response_without_access_token() -> None: @@ -143,3 +162,15 @@ def handler(request: httpx.Request) -> httpx.Response: await auth.token() assert exc_info.value.response is response + _assert_control_plane_remediation(exc_info.value) + + +async def test_chained_auth_failure_uses_provider_neutral_remediation() -> None: + auth = ChainedAuth(_FailingAuth()) + + with pytest.raises(AuthError, match="No auth provider yielded a token") as exc_info: + await auth.token() + + assert exc_info.value.remediation is AuthError.provider_chain_remediation + assert "each provider's credentials or token source" in exc_info.value.remediation + assert "ANYPOINT_CLIENT_ID" not in exc_info.value.remediation diff --git a/python/tests/unit/test_errors.py b/python/tests/unit/test_errors.py index b27d48e8..bd2c61f9 100644 --- a/python/tests/unit/test_errors.py +++ b/python/tests/unit/test_errors.py @@ -29,6 +29,17 @@ def test_401_is_auth_error() -> None: assert isinstance(classify(_resp(401)), AuthError) +def test_auth_error_uses_data_plane_default_unless_overridden() -> None: + default = AuthError("boom") + control_plane = AuthError("boom", remediation="check the connected app") + + assert default.remediation is AuthError.remediation + assert "DONKEY_LLM_PROXY_CLIENT_ID" in default.remediation + assert control_plane.remediation == "check the connected app" + assert AuthError.connected_app_remediation.strip() + assert AuthError.provider_chain_remediation.strip() + + def test_generic_4xx_is_terminal_policy_violation() -> None: err = classify(_resp(400)) assert isinstance(err, PolicyViolation) diff --git a/website/content/errors.mdx b/website/content/errors.mdx index 0d315443..ff65c289 100644 --- a/website/content/errors.mdx +++ b/website/content/errors.mdx @@ -50,7 +50,7 @@ All importable from `donkey_kit`: ``` DonkeyError # base of the whole tree ├─ ConfigError # misconfiguration (raised locally, pre-flight) -├─ AuthError # 401 — bad/missing consumer credentials +├─ AuthError # rejected data-plane credentials or control-plane auth ├─ PolicyViolation # base for every governance rejection │ ├─ PIIDetected # 403, type=pii_detected; .entities │ ├─ TokenBudgetExceeded # 429; .retry_after (seconds) @@ -77,7 +77,7 @@ never burn an exhausted budget or replay a blocked prompt. | Exception | Discriminator | Retryable? | Next step (`.remediation`) | |---|---|---|---| -| `AuthError` | `401`, or `403` + `www-authenticate` | **No** — terminal. The async client refreshes its token and retries **once** on a `401`, then surfaces it. | Check the consumer `client_id` / `client_secret` pair and that the consumer is authorized in API Manager. | +| `AuthError` | Data plane: `401`, or `403` + `www-authenticate`. Control plane: Anypoint auth-provider or connected-app token acquisition fails. | **No** — terminal. The async data-plane client refreshes its token and retries **once** on a `401`, then surfaces it; control-plane token acquisition surfaces immediately. | Data plane: check the consumer `client_id` / `client_secret` pair and API Manager authorization. Control plane: check the configured auth provider; for a connected app, verify `ANYPOINT_CLIENT_ID` / `ANYPOINT_CLIENT_SECRET` and the required scopes. | | `PIIDetected` | `403`, nested `type: "pii_detected"`, **no** `www-authenticate` | **No** — a `PolicyViolation`, never retried. | Remove or redact the flagged values (`.entities`), or relax the policy's entity list in API Manager. | | `TokenBudgetExceeded` | `429`, empty body, `x-token-*` headers | **Not immediately** — never auto-retried; only worth retrying *after* the window resets. | Wait for `.retry_after` (seconds) / the reset, then retry — or request an increase in API Manager. | | `PromptInjectionBlocked` | header `x-injection-protection: blocked`, **or** a top-level `matched_patterns` list (regex prompt guard) | **No** — a `PolicyViolation`, never retried. | Review and sanitise the untrusted input, or adjust the policy's sensitivity / deny-list in API Manager. | @@ -99,6 +99,13 @@ different model than requested (opt-in via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, and reports every missing field at once — fix the config and re-run. +`AuthError.remediation` follows the plane that failed. Errors classified from +an LLM-proxy response use the canonical consumer-credential guidance that +[`donkey doctor`](/cli) also prints. Control-plane token failures override that +default with canonical guidance for the provider that failed: connected-app +errors point to the Anypoint credentials and scopes, while an exhausted +`ChainedAuth` points to each configured provider's credential or token source. + [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 ## When the gateway can't be reached at all diff --git a/website/public/errors.md b/website/public/errors.md index 354f7876..72e01fe2 100644 --- a/website/public/errors.md +++ b/website/public/errors.md @@ -48,7 +48,7 @@ All importable from `donkey_kit`: ``` DonkeyError # base of the whole tree ├─ ConfigError # misconfiguration (raised locally, pre-flight) -├─ AuthError # 401 — bad/missing consumer credentials +├─ AuthError # rejected data-plane credentials or control-plane auth ├─ PolicyViolation # base for every governance rejection │ ├─ PIIDetected # 403, type=pii_detected; .entities │ ├─ TokenBudgetExceeded # 429; .retry_after (seconds) @@ -75,7 +75,7 @@ never burn an exhausted budget or replay a blocked prompt. | Exception | Discriminator | Retryable? | Next step (`.remediation`) | |---|---|---|---| -| `AuthError` | `401`, or `403` + `www-authenticate` | **No** — terminal. The async client refreshes its token and retries **once** on a `401`, then surfaces it. | Check the consumer `client_id` / `client_secret` pair and that the consumer is authorized in API Manager. | +| `AuthError` | Data plane: `401`, or `403` + `www-authenticate`. Control plane: Anypoint auth-provider or connected-app token acquisition fails. | **No** — terminal. The async data-plane client refreshes its token and retries **once** on a `401`, then surfaces it; control-plane token acquisition surfaces immediately. | Data plane: check the consumer `client_id` / `client_secret` pair and API Manager authorization. Control plane: check the configured auth provider; for a connected app, verify `ANYPOINT_CLIENT_ID` / `ANYPOINT_CLIENT_SECRET` and the required scopes. | | `PIIDetected` | `403`, nested `type: "pii_detected"`, **no** `www-authenticate` | **No** — a `PolicyViolation`, never retried. | Remove or redact the flagged values (`.entities`), or relax the policy's entity list in API Manager. | | `TokenBudgetExceeded` | `429`, empty body, `x-token-*` headers | **Not immediately** — never auto-retried; only worth retrying *after* the window resets. | Wait for `.retry_after` (seconds) / the reset, then retry — or request an increase in API Manager. | | `PromptInjectionBlocked` | header `x-injection-protection: blocked`, **or** a top-level `matched_patterns` list (regex prompt guard) | **No** — a `PolicyViolation`, never retried. | Review and sanitise the untrusted input, or adjust the policy's sensitivity / deny-list in API Manager. | @@ -97,6 +97,13 @@ different model than requested (opt-in via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, and reports every missing field at once — fix the config and re-run. +`AuthError.remediation` follows the plane that failed. Errors classified from +an LLM-proxy response use the canonical consumer-credential guidance that +[`donkey doctor`](https://donkey-development-kit.github.io/donkey-development-kit/cli.md) also prints. Control-plane token failures override that +default with canonical guidance for the provider that failed: connected-app +errors point to the Anypoint credentials and scopes, while an exhausted +`ChainedAuth` points to each configured provider's credential or token source. + [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 ## When the gateway can't be reached at all diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index 038758d5..df4a5d88 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -1984,7 +1984,7 @@ All importable from `donkey_kit`: ``` DonkeyError # base of the whole tree ├─ ConfigError # misconfiguration (raised locally, pre-flight) -├─ AuthError # 401 — bad/missing consumer credentials +├─ AuthError # rejected data-plane credentials or control-plane auth ├─ PolicyViolation # base for every governance rejection │ ├─ PIIDetected # 403, type=pii_detected; .entities │ ├─ TokenBudgetExceeded # 429; .retry_after (seconds) @@ -2011,7 +2011,7 @@ never burn an exhausted budget or replay a blocked prompt. | Exception | Discriminator | Retryable? | Next step (`.remediation`) | |---|---|---|---| -| `AuthError` | `401`, or `403` + `www-authenticate` | **No** — terminal. The async client refreshes its token and retries **once** on a `401`, then surfaces it. | Check the consumer `client_id` / `client_secret` pair and that the consumer is authorized in API Manager. | +| `AuthError` | Data plane: `401`, or `403` + `www-authenticate`. Control plane: Anypoint auth-provider or connected-app token acquisition fails. | **No** — terminal. The async data-plane client refreshes its token and retries **once** on a `401`, then surfaces it; control-plane token acquisition surfaces immediately. | Data plane: check the consumer `client_id` / `client_secret` pair and API Manager authorization. Control plane: check the configured auth provider; for a connected app, verify `ANYPOINT_CLIENT_ID` / `ANYPOINT_CLIENT_SECRET` and the required scopes. | | `PIIDetected` | `403`, nested `type: "pii_detected"`, **no** `www-authenticate` | **No** — a `PolicyViolation`, never retried. | Remove or redact the flagged values (`.entities`), or relax the policy's entity list in API Manager. | | `TokenBudgetExceeded` | `429`, empty body, `x-token-*` headers | **Not immediately** — never auto-retried; only worth retrying *after* the window resets. | Wait for `.retry_after` (seconds) / the reset, then retry — or request an increase in API Manager. | | `PromptInjectionBlocked` | header `x-injection-protection: blocked`, **or** a top-level `matched_patterns` list (regex prompt guard) | **No** — a `PolicyViolation`, never retried. | Review and sanitise the untrusted input, or adjust the policy's sensitivity / deny-list in API Manager. | @@ -2033,6 +2033,13 @@ different model than requested (opt-in via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, and reports every missing field at once — fix the config and re-run. +`AuthError.remediation` follows the plane that failed. Errors classified from +an LLM-proxy response use the canonical consumer-credential guidance that +[`donkey doctor`](https://donkey-development-kit.github.io/donkey-development-kit/cli.md) also prints. Control-plane token failures override that +default with canonical guidance for the provider that failed: connected-app +errors point to the Anypoint credentials and scopes, while an exhausted +`ChainedAuth` points to each configured provider's credential or token source. + [#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 ## When the gateway can't be reached at all From 176503be100dc8891b7ac57a758603995d4e2850 Mon Sep 17 00:00:00 2001 From: Tommaso Bolis Date: Mon, 21 Sep 2026 18:14:04 +0200 Subject: [PATCH 29/29] Assert header-only adapter correlation exemptions (#506) Closes #504 --- ARCHITECTURE.md | 14 ++-- README.md | 1 + python/README.md | 1 + python/tests/conformance/suite.py | 19 ++++-- ...mption.py => test_transport_exemptions.py} | 49 ++++++++++++-- website/content/concepts/attribution.mdx | 6 +- .../content/frameworks/agent-framework.mdx | 27 ++++---- website/content/frameworks/index.mdx | 4 +- website/content/frameworks/llamaindex.mdx | 26 ++++---- website/content/telemetry.mdx | 3 + website/public/concepts/attribution.md | 6 +- website/public/frameworks.md | 4 +- website/public/frameworks/agent-framework.md | 27 ++++---- website/public/frameworks/llamaindex.md | 26 ++++---- website/public/llms-full.txt | 66 +++++++++++-------- website/public/telemetry.md | 3 + 16 files changed, 185 insertions(+), 97 deletions(-) rename python/tests/conformance/{test_gateway_identity_exemption.py => test_transport_exemptions.py} (64%) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 44543b72..2080019a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -246,18 +246,20 @@ framework is "supported" only when it passes every scenario **or** records an *asserted exemption* in `KNOWN_LIMITATIONS` — never a silent skip. Those exemptions are published in the README as credibility. ADK and CrewAI cannot propagate a per-run correlation ID or populate `donkey.last_call`, because -LiteLLM owns the transport. LlamaIndex and Microsoft Agent Framework cannot -populate `donkey.last_call`, because they receive only `default_headers`, not +LiteLLM owns the transport. LlamaIndex and Microsoft Agent Framework have the +same two exemptions because they receive only a static `default_headers` +snapshot, which deliberately excludes the per-run correlation ID, rather than the SDK's shared HTTP client. The centre of gravity moves with the roster cut (`BG §1.5`): the internal matrix shrinks to LangGraph, and the deliverable becomes the **customer-facing pytest plugin** users run against their own agent (#191). -Four adapters carry documented conformance exemptions: ADK and CrewAI for -per-run correlation and gateway-identity observation, plus LlamaIndex and -Microsoft Agent Framework for gateway-identity observation. The conformance -suite pins those exemptions to each adapter's actual transport behavior. +Four adapters carry documented conformance exemptions for per-run correlation +and gateway-identity observation: ADK and CrewAI because LiteLLM owns their +transport, plus LlamaIndex and Microsoft Agent Framework because they receive +only static headers. The conformance suite pins those exemptions to each +adapter's actual transport behavior. --- diff --git a/README.md b/README.md index 648721f2..ca7fbbfb 100644 --- a/README.md +++ b/README.md @@ -124,5 +124,6 @@ legitimately cannot satisfy a scenario, the reason is asserted in code | Framework | Scenario | Why it's exempt | | --- | --- | --- | | ADK, CrewAI | correlation ID propagated | LiteLLM owns the transport, so the SDK's `httpx` client cannot be injected — the correlation ID ends up per-client, not per-run. A LiteLLM logger callback may recover trace correlation later. | +| LlamaIndex, Microsoft Agent Framework | correlation ID propagated | These adapters receive a static `default_headers` snapshot, which deliberately excludes the per-run correlation ID. Without the SDK's `httpx` client, `donkey.run(id=...)` cannot update their request headers. | | ADK, CrewAI | gateway identity observed | LiteLLM owns the transport, so no response reaches the SDK's `_on_response` hook. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | | LlamaIndex, Microsoft Agent Framework | gateway identity observed | These adapters receive `default_headers`, not the SDK's `httpx` client, so no response reaches `_on_response`. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | diff --git a/python/README.md b/python/README.md index 60605d3c..e0016bda 100644 --- a/python/README.md +++ b/python/README.md @@ -124,5 +124,6 @@ legitimately cannot satisfy a scenario, the reason is asserted in code | Framework | Scenario | Why it's exempt | | --- | --- | --- | | ADK, CrewAI | correlation ID propagated | LiteLLM owns the transport, so the SDK's `httpx` client cannot be injected — the correlation ID ends up per-client, not per-run. A LiteLLM logger callback may recover trace correlation later. | +| LlamaIndex, Microsoft Agent Framework | correlation ID propagated | These adapters receive a static `default_headers` snapshot, which deliberately excludes the per-run correlation ID. Without the SDK's `httpx` client, `donkey.run(id=...)` cannot update their request headers. | | ADK, CrewAI | gateway identity observed | LiteLLM owns the transport, so no response reaches the SDK's `_on_response` hook. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | | LlamaIndex, Microsoft Agent Framework | gateway identity observed | These adapters receive `default_headers`, not the SDK's `httpx` client, so no response reaches `_on_response`. When every resolved adapter is non-observing, `donkey.last_call` reports `UNAVAILABLE` and names them in `surface`. | diff --git a/python/tests/conformance/suite.py b/python/tests/conformance/suite.py index a2771fe8..2f7d5e19 100644 --- a/python/tests/conformance/suite.py +++ b/python/tests/conformance/suite.py @@ -51,6 +51,11 @@ "correlation ID is per-client, not per-run (BG §1.8). A LiteLLM custom " "logger callback may later recover trace correlation." ) +_DEFAULT_HEADERS_CORRELATION_EXEMPTION = ( + "The adapter is handed only a static default_headers snapshot, which " + "deliberately excludes the per-run correlation ID; without our httpx " + "client, donkey.run(id=...) cannot update the request headers (BG §1.8)." +) # gateway_identity_observed has the SAME structural cause as the correlation-id # exemption, stated for last_call (#362): the record is populated by our @@ -79,11 +84,15 @@ "correlation_id_propagated": _LITELLM_TRANSPORT_EXEMPTION, "gateway_identity_observed": _LITELLM_LAST_CALL_EXEMPTION, }, - # LlamaIndex and MS Agent Framework get only default_headers, no httpx client - # (BG §1.8), so they cannot observe last_call either — but they CAN propagate the - # correlation id through those headers, so that scenario is not exempt for them. - "llamaindex": {"gateway_identity_observed": _DEFAULT_HEADERS_LAST_CALL_EXEMPTION}, + # LlamaIndex and MS Agent Framework get only a static default_headers snapshot, + # no httpx client (BG §1.8). The snapshot deliberately excludes the per-run + # correlation ID and cannot observe last_call responses. + "llamaindex": { + "correlation_id_propagated": _DEFAULT_HEADERS_CORRELATION_EXEMPTION, + "gateway_identity_observed": _DEFAULT_HEADERS_LAST_CALL_EXEMPTION, + }, "agent_framework": { - "gateway_identity_observed": _DEFAULT_HEADERS_LAST_CALL_EXEMPTION + "correlation_id_propagated": _DEFAULT_HEADERS_CORRELATION_EXEMPTION, + "gateway_identity_observed": _DEFAULT_HEADERS_LAST_CALL_EXEMPTION, }, } diff --git a/python/tests/conformance/test_gateway_identity_exemption.py b/python/tests/conformance/test_transport_exemptions.py similarity index 64% rename from python/tests/conformance/test_gateway_identity_exemption.py rename to python/tests/conformance/test_transport_exemptions.py index 466696d3..7d7e4289 100644 --- a/python/tests/conformance/test_gateway_identity_exemption.py +++ b/python/tests/conformance/test_transport_exemptions.py @@ -1,5 +1,5 @@ -"""The gateway-identity conformance exemption is ASSERTED, never skipped (#362, the -conformance kit). +"""Transport-owned conformance exemptions are ASSERTED, never skipped (#362, #504, +the conformance kit). ``donkey.last_call`` is populated by the shared transport's ``_on_response``, so an adapter that does not route through our httpx client structurally cannot observe @@ -13,6 +13,9 @@ set whose :attr:`Adapter.observes_last_call` is ``False``. The exemption table and the code fact it documents cannot drift apart: add a non-observing adapter without recording the exemption (or vice-versa) and this fails. +3. Header-only adapters do not receive the active run correlation ID in their + static ``default_headers`` snapshot, and both record the corresponding + ``correlation_id_propagated`` exemption. Reading ``observes_last_call`` off each adapter class imports only the adapter modules, which import their framework lazily inside methods — so this needs no @@ -28,10 +31,15 @@ # import mode puts this directory on sys.path and ``suite`` resolves to it. from suite import CONFORMANCE_SCENARIOS, KNOWN_LIMITATIONS +from donkey_kit.core.config import DonkeyConfig +from donkey_kit.core.telemetry import run_context +from donkey_kit.core.transport import build_http_client from donkey_kit.integrations import ADAPTERS from donkey_kit.integrations._base import Adapter -_SCENARIO = "gateway_identity_observed" +_GATEWAY_SCENARIO = "gateway_identity_observed" +_CORRELATION_SCENARIO = "correlation_id_propagated" +_HEADER_ONLY_ADAPTERS = {"llamaindex", "agent_framework"} def _adapter_class(attr: str) -> type[Adapter]: @@ -42,7 +50,7 @@ def _adapter_class(attr: str) -> type[Adapter]: def test_scenario_is_registered() -> None: - assert _SCENARIO in CONFORMANCE_SCENARIOS + assert {_GATEWAY_SCENARIO, _CORRELATION_SCENARIO} <= set(CONFORMANCE_SCENARIOS) def test_every_known_limitation_names_a_real_scenario() -> None: @@ -63,7 +71,7 @@ def test_exemption_matches_observes_last_call_flag() -> None: exempted = { adapter for adapter, limits in KNOWN_LIMITATIONS.items() - if _SCENARIO in limits + if _GATEWAY_SCENARIO in limits } non_observing = { attr for attr in ADAPTERS if not _adapter_class(attr).observes_last_call @@ -75,3 +83,34 @@ def test_exemption_matches_observes_last_call_flag() -> None: # And it must be a non-empty set — a conformance surface with zero recorded # exemptions here would mean every adapter observes, which is not true. assert non_observing == {"adk", "crewai", "llamaindex", "agent_framework"} + + +async def test_header_only_correlation_exemptions_match_connection_kwargs() -> None: + cfg = DonkeyConfig( + llm_proxy_url="https://proxy", + llm_proxy_client_id="cid", + llm_proxy_client_secret="secret", + correlation_header="x-test-correlation-id", + call_id_header="x-test-call-id", + ) + + for attr in _HEADER_ONLY_ADAPTERS: + client = build_http_client(cfg, None) + try: + with run_context("run-123"): + headers = _adapter_class(attr)(cfg, client).connection_kwargs()[ + "default_headers" + ] + finally: + await client.aclose() + + assert cfg.correlation_header not in headers + assert "run-123" not in headers.values() + assert _CORRELATION_SCENARIO in KNOWN_LIMITATIONS[attr] + + exempted = { + adapter + for adapter, limits in KNOWN_LIMITATIONS.items() + if _CORRELATION_SCENARIO in limits + } + assert exempted == {"adk", "crewai", *_HEADER_ONLY_ADAPTERS} diff --git a/website/content/concepts/attribution.mdx b/website/content/concepts/attribution.mdx index c68d6c1e..85aa451a 100644 --- a/website/content/concepts/attribution.mdx +++ b/website/content/concepts/attribution.mdx @@ -51,8 +51,10 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on for the full two-id model. - **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK - can't inject its HTTP client — the correlation id is per-client, not per-run. + **Header-only adapter caveat.** Google ADK and CrewAI route through LiteLLM, + while LlamaIndex and Microsoft Agent Framework receive only a static + `default_headers` snapshot. None receives the SDK's HTTP client, so they + cannot propagate the correlation id bound by `donkey.run(id=...)` per run. These are documented, asserted conformance exemptions, not silent gaps. diff --git a/website/content/frameworks/agent-framework.mdx b/website/content/frameworks/agent-framework.mdx index 81b272a9..10e47ba5 100644 --- a/website/content/frameworks/agent-framework.mdx +++ b/website/content/frameworks/agent-framework.mdx @@ -7,10 +7,11 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented -> exemption: it receives `default_headers`, not the SDK's shared httpx client, -> so `donkey.last_call` response observation is unavailable. It also ships the -> strongest policy-integration story of the eight frameworks +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> This adapter receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions. It also ships the strongest +> policy-integration story of the eight frameworks > (`policy_middleware()`), but its native client class is young and > **UNVERIFIED** — read the callouts below before depending on it. @@ -113,14 +114,16 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. -> **This adapter cannot populate `donkey.last_call`.** Agent Framework -> receives the governed `default_headers`, but not the SDK's httpx client, so no -> response reaches `_on_response` and gateway identity, routing, and usage -> fields cannot be observed. When every adapter resolved on a `Donkey` is -> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, -> `available == False`, and names the resolved adapters in `surface`. -> This is a documented, asserted `gateway_identity_observed` conformance -> exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** Agent Framework receives a static `default_headers` +> snapshot, which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. See the [error taxonomy](/errors) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the diff --git a/website/content/frameworks/index.mdx b/website/content/frameworks/index.mdx index 216c2584..6dd7054d 100644 --- a/website/content/frameworks/index.mdx +++ b/website/content/frameworks/index.mdx @@ -99,10 +99,10 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | +| LlamaIndex | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](/frameworks/anthropic). | -| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| MS Agent Framework | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. Class name/kwarg **unverified**. | | Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | | CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | diff --git a/website/content/frameworks/llamaindex.mdx b/website/content/frameworks/llamaindex.mdx index 0a1f97bb..f07e9582 100644 --- a/website/content/frameworks/llamaindex.mdx +++ b/website/content/frameworks/llamaindex.mdx @@ -8,10 +8,11 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one -> documented exemption: LlamaIndex receives `default_headers`, not the SDK's -> shared httpx client, so `donkey.last_call` response observation is unavailable -> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> LlamaIndex receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions (see below). Watch the `is_chat_model` +> gotcha if you build the client by hand. ## Install @@ -104,12 +105,15 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. -> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives -> the governed `default_headers`, but not the SDK's httpx client, so no response -> reaches `_on_response` and gateway identity, routing, and usage fields cannot -> be observed. When every adapter resolved on a `Donkey` is non-observing, the -> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, -> and names the resolved adapters in `surface`. This is a documented, asserted -> `gateway_identity_observed` conformance exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** LlamaIndex receives a static `default_headers` snapshot, +> which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. diff --git a/website/content/telemetry.mdx b/website/content/telemetry.mdx index 0391a120..420f2376 100644 --- a/website/content/telemetry.mdx +++ b/website/content/telemetry.mdx @@ -178,6 +178,9 @@ recovered from a degraded provider. the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route outside that response path: ADK and CrewAI use LiteLLM's transport, while LlamaIndex and Microsoft Agent Framework receive only `default_headers`. +That static snapshot deliberately excludes the correlation ID bound later by +`donkey.run(id=...)`, so those two adapters also carry the asserted +`correlation_id_propagated` exemption. When every adapter resolved on a `Donkey` is one of those four, a cold read reports the limitation explicitly. For a `Donkey` that resolved only ADK: diff --git a/website/public/concepts/attribution.md b/website/public/concepts/attribution.md index 9ba90496..da343b9b 100644 --- a/website/public/concepts/attribution.md +++ b/website/public/concepts/attribution.md @@ -48,8 +48,10 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on `DonkeyError.call_id`. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md#grouping-a-run-the-run-id-on-the-span-the-call-id-on-the-error) for the full two-id model. - **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK - can't inject its HTTP client — the correlation id is per-client, not per-run. + **Header-only adapter caveat.** Google ADK and CrewAI route through LiteLLM, + while LlamaIndex and Microsoft Agent Framework receive only a static + `default_headers` snapshot. None receives the SDK's HTTP client, so they + cannot propagate the correlation id bound by `donkey.run(id=...)` per run. These are documented, asserted conformance exemptions, not silent gaps. The **agent→agent egress** attribution header (`x-anypoint-api-instance-id`) is diff --git a/website/public/frameworks.md b/website/public/frameworks.md index 4551df50..f5c816a2 100644 --- a/website/public/frameworks.md +++ b/website/public/frameworks.md @@ -86,10 +86,10 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | +| LlamaIndex | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/anthropic.md). | -| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| MS Agent Framework | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. Class name/kwarg **unverified**. | | Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | | CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | diff --git a/website/public/frameworks/agent-framework.md b/website/public/frameworks/agent-framework.md index 7b0c731d..dba8b4df 100644 --- a/website/public/frameworks/agent-framework.md +++ b/website/public/frameworks/agent-framework.md @@ -4,10 +4,11 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented -> exemption: it receives `default_headers`, not the SDK's shared httpx client, -> so `donkey.last_call` response observation is unavailable. It also ships the -> strongest policy-integration story of the eight frameworks +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> This adapter receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions. It also ships the strongest +> policy-integration story of the eight frameworks > (`policy_middleware()`), but its native client class is young and > **UNVERIFIED** — read the callouts below before depending on it. @@ -118,14 +119,16 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. -> **This adapter cannot populate `donkey.last_call`.** Agent Framework -> receives the governed `default_headers`, but not the SDK's httpx client, so no -> response reaches `_on_response` and gateway identity, routing, and usage -> fields cannot be observed. When every adapter resolved on a `Donkey` is -> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, -> `available == False`, and names the resolved adapters in `surface`. -> This is a documented, asserted `gateway_identity_observed` conformance -> exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** Agent Framework receives a static `default_headers` +> snapshot, which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the diff --git a/website/public/frameworks/llamaindex.md b/website/public/frameworks/llamaindex.md index 228fbfef..33285dab 100644 --- a/website/public/frameworks/llamaindex.md +++ b/website/public/frameworks/llamaindex.md @@ -4,10 +4,11 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one -> documented exemption: LlamaIndex receives `default_headers`, not the SDK's -> shared httpx client, so `donkey.last_call` response observation is unavailable -> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> LlamaIndex receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions (see below). Watch the `is_chat_model` +> gotcha if you build the client by hand. ## Install @@ -108,13 +109,16 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. -> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives -> the governed `default_headers`, but not the SDK's httpx client, so no response -> reaches `_on_response` and gateway identity, routing, and usage fields cannot -> be observed. When every adapter resolved on a `Donkey` is non-observing, the -> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, -> and names the resolved adapters in `surface`. This is a documented, asserted -> `gateway_identity_observed` conformance exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** LlamaIndex receives a static `default_headers` snapshot, +> which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface as typed exceptions, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt index df4a5d88..eb941814 100644 --- a/website/public/llms-full.txt +++ b/website/public/llms-full.txt @@ -814,10 +814,10 @@ each framework exposes: |---|---|---|---| | LangGraph | ✅ | ✅ | Best case — `default_headers` + custom async client. | | Strands | ✅ | ✅ | Via `client_args`. | -| LlamaIndex | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. `is_chat_model=True` is forced. | +| LlamaIndex | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. `is_chat_model=True` is forced. | | OpenAI Agents SDK | ✅ | ✅ | We build the `AsyncOpenAI` client ourselves. | | Anthropic SDK | ✅ | ✅ | Returns a bare `client()`, not a model-bound object — see the [divergence note](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/anthropic.md). | -| MS Agent Framework | ✅ | ❌ | Receives `default_headers`, not the SDK's HTTP client, so it cannot populate `donkey.last_call`. Class name/kwarg **unverified**. | +| MS Agent Framework | ✅ | ❌ | Receives a static `default_headers` snapshot, so per-run correlation and `donkey.last_call` are unavailable. Class name/kwarg **unverified**. | | Google ADK | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; correlation is per-client and the adapter cannot populate `donkey.last_call`. | | CrewAI | ✅ (`extra_headers`) | ❌ | LiteLLM owns the transport; same correlation and `donkey.last_call` exemptions as Google ADK. | @@ -1281,10 +1281,11 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter has one documented -> exemption: it receives `default_headers`, not the SDK's shared httpx client, -> so `donkey.last_call` response observation is unavailable. It also ships the -> strongest policy-integration story of the eight frameworks +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> This adapter receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions. It also ships the strongest +> policy-integration story of the eight frameworks > (`policy_middleware()`), but its native client class is young and > **UNVERIFIED** — read the callouts below before depending on it. @@ -1395,14 +1396,16 @@ llm = OpenAIChatClient( > **UNVERIFIED**; the shipped middleware is a plain async wrapper pending > confirmation of the framework's middleware protocol. -> **This adapter cannot populate `donkey.last_call`.** Agent Framework -> receives the governed `default_headers`, but not the SDK's httpx client, so no -> response reaches `_on_response` and gateway identity, routing, and usage -> fields cannot be observed. When every adapter resolved on a `Donkey` is -> non-observing, the record reports `status == LastCallStatus.UNAVAILABLE`, -> `available == False`, and names the resolved adapters in `surface`. -> This is a documented, asserted `gateway_identity_observed` conformance -> exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** Agent Framework receives a static `default_headers` +> snapshot, which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for the full `PolicyViolation` hierarchy that `policy_middleware()` catches, and the @@ -1814,10 +1817,11 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection, with one -> documented exemption: LlamaIndex receives `default_headers`, not the SDK's -> shared httpx client, so `donkey.last_call` response observation is unavailable -> (see below). Watch the `is_chat_model` gotcha if you build the client by hand. +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** +> LlamaIndex receives a static `default_headers` snapshot, not the SDK's shared +> httpx client, so per-run correlation and `donkey.last_call` response +> observation are documented exemptions (see below). Watch the `is_chat_model` +> gotcha if you build the client by hand. ## Install @@ -1918,13 +1922,16 @@ model = OpenAILike( > LlamaIndex uses `api_base` rather than `base_url` for its endpoint kwarg; > `connection_kwargs()` already translates for you. -> **This adapter cannot populate `donkey.last_call`.** LlamaIndex receives -> the governed `default_headers`, but not the SDK's httpx client, so no response -> reaches `_on_response` and gateway identity, routing, and usage fields cannot -> be observed. When every adapter resolved on a `Donkey` is non-observing, the -> record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, -> and names the resolved adapters in `surface`. This is a documented, asserted -> `gateway_identity_observed` conformance exemption. +> **This adapter cannot propagate per-run correlation or populate +> `donkey.last_call`.** LlamaIndex receives a static `default_headers` snapshot, +> which deliberately excludes the correlation ID bound later by +> `donkey.run(id=...)`, and not the SDK's httpx client. No response reaches +> `_on_response`, so gateway identity, routing, and usage fields cannot be +> observed either. When every adapter resolved on a `Donkey` is non-observing, +> the record reports `status == LastCallStatus.UNAVAILABLE`, `available == False`, +> and names the resolved adapters in `surface`. These are documented, asserted +> `correlation_id_propagated` and `gateway_identity_observed` conformance +> exemptions. See the [error taxonomy](https://donkey-development-kit.github.io/donkey-development-kit/errors.md) for how proxy rejections surface as typed exceptions, and the [verification policy](https://donkey-development-kit.github.io/donkey-development-kit/concepts/verification.md) page for @@ -2838,6 +2845,9 @@ recovered from a degraded provider. the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route outside that response path: ADK and CrewAI use LiteLLM's transport, while LlamaIndex and Microsoft Agent Framework receive only `default_headers`. +That static snapshot deliberately excludes the correlation ID bound later by +`donkey.run(id=...)`, so those two adapters also carry the asserted +`correlation_id_propagated` exemption. When every adapter resolved on a `Donkey` is one of those four, a cold read reports the limitation explicitly. For a `Donkey` that resolved only ADK: @@ -4763,8 +4773,10 @@ request additionally carries a per-call `X-Donkey-Request-Id` that surfaces on `DonkeyError.call_id`. See [Observability](https://donkey-development-kit.github.io/donkey-development-kit/concepts/observability.md#grouping-a-run-the-run-id-on-the-span-the-call-id-on-the-error) for the full two-id model. - **Google ADK and CrewAI caveat.** LiteLLM owns their transports, so the SDK - can't inject its HTTP client — the correlation id is per-client, not per-run. + **Header-only adapter caveat.** Google ADK and CrewAI route through LiteLLM, + while LlamaIndex and Microsoft Agent Framework receive only a static + `default_headers` snapshot. None receives the SDK's HTTP client, so they + cannot propagate the correlation id bound by `donkey.run(id=...)` per run. These are documented, asserted conformance exemptions, not silent gaps. The **agent→agent egress** attribution header (`x-anypoint-api-instance-id`) is diff --git a/website/public/telemetry.md b/website/public/telemetry.md index a8856d7a..e62b0ad9 100644 --- a/website/public/telemetry.md +++ b/website/public/telemetry.md @@ -162,6 +162,9 @@ recovered from a degraded provider. the SDK's shared httpx client. Four `connection_kwargs()`-only adapters route outside that response path: ADK and CrewAI use LiteLLM's transport, while LlamaIndex and Microsoft Agent Framework receive only `default_headers`. +That static snapshot deliberately excludes the correlation ID bound later by +`donkey.run(id=...)`, so those two adapters also carry the asserted +`correlation_id_propagated` exemption. When every adapter resolved on a `Donkey` is one of those four, a cold read reports the limitation explicitly. For a `Donkey` that resolved only ADK: