diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7f8de791..9a0df3fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,6 +5,10 @@ on: branches: [main] pull_request: +# Least privilege: this workflow only reads the checkout, never writes back. +permissions: + contents: read + defaults: run: working-directory: python diff --git a/.github/workflows/nightly-matrix.yml b/.github/workflows/nightly-matrix.yml index 34272464..fb37f50b 100644 --- a/.github/workflows/nightly-matrix.yml +++ b/.github/workflows/nightly-matrix.yml @@ -47,6 +47,10 @@ on: type: boolean default: false +# Least privilege: the matrix only reads the checkout, never writes back. +permissions: + contents: read + defaults: run: working-directory: python diff --git a/.github/workflows/rebrand-completeness.yml b/.github/workflows/rebrand-completeness.yml index 5a3cf601..e2e0e96a 100644 --- a/.github/workflows/rebrand-completeness.yml +++ b/.github/workflows/rebrand-completeness.yml @@ -10,6 +10,10 @@ on: branches: [main] pull_request: +# Least privilege: the guard only reads the checkout, never writes back. +permissions: + contents: read + jobs: rebrand-completeness: runs-on: ubuntu-latest diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index acc49315..23ecfe4c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -6,11 +6,11 @@ trustworthy. It is a distillation, not the spec. The authoritative specs are [`spec/donkey-development-kit-build-plan.md`](spec/donkey-development-kit-build-plan.md) (phases, milestones, standing invariants) and [`spec/donkey-development-kit-build-guide.md`](spec/donkey-development-kit-build-guide.md) -(feature scope, cited as `BG §N.N`). A **bare** `§N.N` reference below points -into the archived v1 plan at -[`spec/archive/donkey-development-kit-build-plan-v1.md`](spec/archive/donkey-development-kit-build-plan-v1.md), -which is where most existing citations in the tree still resolve. When a rule -here feels arbitrary, read the cited section — the constraints are deliberate. +(feature scope, cited as `BG §N.N`). The five **standing invariants** in the +build plan keep a bare `§` label — `§0.3`, `§1.1`, `§2.1`, `§8.1`, `§8.4` — as +their stable historical name; every other citation is a `BG §N.N`, a named +invariant, or a `Phase N`. When a rule here feels arbitrary, read the cited +section — the constraints are deliberate. For *using* the SDK, see the consumer docs site (`website/`). For *working in* the repo — branch/PR flow, testing surfaces, coding conventions — see @@ -38,7 +38,7 @@ owns. The gateway — not the SDK — enforces policy. --- -## Layered architecture (§1.1) +## Layered architecture The package is a strict stack. Higher layers depend on lower layers; **no lower layer may import a higher one.** @@ -61,13 +61,17 @@ core/ config · auth · transport · errors · budget · telemetry · endpoints are `_verify.blocked`. Treat it as legacy scaffolding: reachable, but do not deepen it (see "Still blocked", below). -**The hard rule (§1.1):** `core/` has no dependency on any agent framework. +**The hard rule (the layered architecture):** `core/` has no dependency on any agent framework. Each `integrations/*` adapter may depend on exactly one framework, and nothing in `integrations/` may be imported by `core`, `llm`, `registry`, or `tools`. This is enforced in CI by `import-linter` (`lint-imports`); a violating import -fails the build. The `base-only` CI job additionally installs *only* the base -package and imports `donkey_kit` to catch an accidental top-level framework -import leaking into a lower layer. +fails the build. A companion `independence` contract additionally forbids one +adapter from importing another (directly or indirectly), so a convenient +cross-import can never drag a second framework's optional dependency onto an +install that asked only for the first — the shared `_base` sibling is exempt, +since every adapter importing it downward is expected. The `base-only` CI job +additionally installs *only* the base package and imports `donkey_kit` to catch +an accidental top-level framework import leaking into a lower layer. Because of that rule, adapters import their framework **lazily, inside methods** — never at module top level — so importing the base package never drags in a @@ -88,7 +92,7 @@ framework that may not be installed. object (e.g. a real `langchain_openai.ChatOpenAI`), so there is nothing to unlearn and a three-line escape hatch (`connection_kwargs()`) out of the SDK. - **Configuration** resolves in a fixed precedence — constructor kwargs → env - vars → `.donkey-kit.toml` → default (§2.1) — and reports every missing field + vars → `.donkey-kit.toml` → default — and reports every missing field at once rather than one failure per run. `Donkey.from_env()` is the entry point. - **The transport is the attachment point.** `DonkeyAsyncClient` exposes four internal lifecycle hooks — no-op by default, **not** public API, mirrored on the @@ -112,12 +116,12 @@ framework that may not be installed. full contracts live in the `core/transport.py` docstrings. - **`Governance`** (`governance.py`) is a second top-level object alongside `Donkey`, outside the linear import stack — it depends only on `core`. It is - ONE object behind three verbs (§6.2–§6.4): `simulate()` (an ephemeral local + ONE object behind three verbs: `simulate()` (an ephemeral local gateway harness), `export()` (emit the governed-state manifest), and `resolve()` (reconcile a running `Donkey` against it, raising `GovernanceDrift` on mismatch); a separate platform-team-only `apply()` is the deliberate escape hatch. **All of these are currently `_verify.blocked`** — the `simulate()` - harness included — pending the §6 verification items, so today the object is the + harness included — pending the Verification milestone, so today the object is the shape, not yet the behaviour. Do not confuse it with `registry/governance.py`, which types governed-state *assets* one layer down. @@ -127,7 +131,7 @@ module-level factory. --- -## Verification discipline (§0.3) +## Verification discipline This is the SDK's most distinctive principle and its strongest trust guarantee. @@ -135,7 +139,7 @@ This is the SDK's most distinctive principle and its strongest trust guarantee. A fabricated endpoint that 404s in a customer sandbox destroys confidence in the whole package, so the codebase makes fabrication structurally hard. `core/_verify.py` -is the single home for every value that §0.3 says must be confirmed against a real +is the single home for every value that the verification discipline says must be confirmed against a real Anypoint sandbox before it can be trusted, and it offers exactly two mechanisms: - **`blocked("…")`** returns a `NotImplementedError("blocked on verification: …")`. @@ -160,11 +164,11 @@ is **no `/v1`** — the `client_id`/`client_secret` request-header pair, streami and the live rejection shapes), the OAuth2 control-plane token path, and the CLI-plugin REST contract (from static analysis). Still blocked: Exchange→MCP tool discovery, the provisioning control plane, and the exact framework-adapter class -names/kwargs (§8–§10). +names/kwargs (the conformance kit and the build plan phases). --- -## Error-taxonomy design (§2.4) +## Error-taxonomy design (BG §1.2) Turning the gateway's policy rejections into catchable, actionable exceptions is the SDK's clearest value over raw HTTP. Two invariants govern the taxonomy in @@ -182,7 +186,7 @@ the SDK's clearest value over raw HTTP. Two invariants govern the taxonomy in **`classify()` is fixture-driven, not guessed.** The HTTP-response → exception mapping in `classify()` is populated from real rejection captures taken against a -live governed proxy (§8.2), not hand-written assumptions. The authoritative +live governed proxy (BG §1.5), not hand-written assumptions. The authoritative discriminator is the error **`type`** plus specific headers — **not the status code alone.** The captures established, for example, that: @@ -204,7 +208,7 @@ A **prompt-injection** block is typed on its own signal: the is still pending live capture (#253). Only **content-moderation / federated-guardrail** shapes remain under-documented, and those deliberately fall through to a generic `PolicyViolation` whose message *says so* rather than -pretending to a precision the captures don't yet support — the same §0.3 honesty +pretending to a precision the captures don't yet support — the same verification-discipline honesty as the verification ledger. All errors subclass `DonkeyError`, which carries the correlation/request IDs and the raw response for inspection. @@ -247,7 +251,7 @@ pytest plugin** users run against their own agent (#191). Two adapters carry documented conformance exemptions: CrewAI (per-run correlation degrades to per-client because it reaches models through LiteLLM, like ADK) and the Anthropic SDK (depends on the proxy exposing an -Anthropic-native Messages API route, an open verification item, §0.3). +Anthropic-native Messages API route, an open verification item). --- @@ -257,9 +261,7 @@ Anthropic-native Messages API route, an open verification item, §0.3). authoritative plan: phases, milestones, label taxonomy, standing invariants. - [`spec/donkey-development-kit-build-guide.md`](spec/donkey-development-kit-build-guide.md) — feature-by-feature scope and acceptance bars; cited as `BG §N.N`. -- [`spec/archive/donkey-development-kit-build-plan-v1.md`](spec/archive/donkey-development-kit-build-plan-v1.md) — - archived v1 plan, not authoritative; a bare `§N.N` resolves here. -- [`docs/verified-apis.md`](docs/verified-apis.md) — the §0.3 verification ledger +- [`docs/verified-apis.md`](docs/verified-apis.md) — the verification ledger (source of truth for what is verified vs. blocked). - [`CONTRIBUTING.md`](CONTRIBUTING.md) — branch/PR/release flow, testing surfaces, coding conventions. @@ -274,4 +276,4 @@ Anthropic-native Messages API route, an open verification item, §0.3). *"Agent Fabric", "Anypoint", and "Omni Gateway" are Salesforce trademarks; this project is a descriptive, non-first-party SDK for consuming those capabilities -(§0.4).* +(the trademark/support boundary).* diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6d681f39..fb90a818 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,10 @@ Thanks for working on the Donkey Development Kit. This guide is the contributor-facing runbook for the repo: how a change moves from an issue to `main`, how the code is -tested and linted, and the conventions that keep the package trustworthy. +tested and linted, and the conventions that keep the package trustworthy. It +serves both **internal** contributors (org members with push access) and +**external** ones (contributing via a fork + PR); where the two diverge is +spelled out in [§1](#who-can-contribute-internal-vs-external). For *how the SDK is built* — the layer boundaries, the verification discipline, the error taxonomy, adapter support depth — read [`ARCHITECTURE.md`](ARCHITECTURE.md) @@ -26,6 +29,60 @@ noted. There is no Makefile — every command runs directly. ## 1. Branch, PR & release workflow +### Who can contribute: internal vs external + +This repo is **public** and takes contributions two ways. There is only one +workflow; what differs is *where your branch lives and who can triage and merge +it* — not the discipline. + +- **Internal contributors** — members of the `Donkey-Development-Kit` org with push + access. You branch directly in this repo, push your `/-` + branch to `origin`, set your own milestone/labels/assignee, and (after review) + merge. The rest of this document is written from this seat. +- **External contributors** — anyone without push access. You work from a + **fork** and open a PR from your fork into this repo's `develop`. Everything + substantive is identical — issue-first, the same branch naming, the same + pre-PR gate, the same docs-sync rule — with the exceptions called out as + **(fork)** notes throughout this section. + +**The fork flow, end to end.** Fork `Donkey-Development-Kit/donkey-development-kit` on +GitHub, then clone your fork and add this repo as `upstream`: + +```bash +git clone https://github.com//donkey-development-kit.git +cd donkey-development-kit +git remote add upstream https://github.com/Donkey-Development-Kit/donkey-development-kit.git +``` + +Branch from `upstream/develop` (not your fork's possibly-stale copy), keep the +`/-` name, and push the branch to your fork (`origin`): + +```bash +git fetch upstream +git checkout -b docs/13-verified-apis-update upstream/develop +git push -u origin docs/13-verified-apis-update +``` + +Open the PR from `:docs/13-verified-apis-update` → +`Donkey-Development-Kit:develop`, and **tick "Allow edits by maintainers"** so a +maintainer can rebase or push a small fix without a round-trip. Keep your branch +current by rebasing on `upstream/develop` (`git fetch upstream && git rebase +upstream/develop`), the same default as an internal branch. + +What a fork contributor **can't** do — a maintainer does each instead, so don't +block waiting on it: + +- **Triage fields.** You can file an issue, but setting its milestone, labels, + and assignee needs write access. File it, then say in the issue that you plan + to work it, so it isn't picked up by someone else. +- **The `.claude/skills/**` shortcut** (below) — committing straight to + `develop` — needs push access. From a fork, even a skills-only change goes + through a PR. +- **Merging and promotion** — squashing a branch into `develop` and promoting + `develop` → `main` are maintainer-only. +- **Secret-gated CI and live/sandbox verification** — see + [the pre-PR gate](#the-pre-pr-gate) and [§2](#2-testing-strategy). + ### The issue is the plan **No code change lands without a GitHub issue and a branch named after it.** @@ -37,7 +94,8 @@ scratch files. The one exception: edits scoped **entirely** to `.claude/skills/**` may be committed directly to `develop` after a recap, since skills are agent tooling rather than SDK code. Anything touching `python/`, `website/`, CI, the build -plan, or repo policy follows the full flow below. +plan, or repo policy follows the full flow below. **(fork)** This shortcut needs +push access — from a fork, a skills-only change still goes through a PR. ### Branch model @@ -57,7 +115,9 @@ find yourself on `main` about to start work, `git checkout develop` first. Donkey-Development-Kit/donkey-development-kit --search ""`); file one if none matches. Every issue carries exactly one **milestone** — that milestone is the release the branch targets. Triage is by milestone + labels; there is no - Projects board. + Projects board. **(fork)** File the issue, but leave milestone/labels/assignee + to a maintainer — setting them needs write access; note in the issue that you + plan to work it. 2. **Cut the branch from `develop`:** ```bash git fetch origin @@ -68,19 +128,21 @@ find yourself on `main` about to start work, `git checkout develop` first. mandatory. `` is one of `feat` (new capability), `fix` (something that should have worked), `docs` (README, website, comments), or `chore` (tooling, deps, refactors). The slug is 2–5 kebab-case words describing *what* - changes, e.g. `fix/42-proxy-url-trailing-slash`. + changes, e.g. `fix/42-proxy-url-trailing-slash`. **(fork)** Branch from + `upstream/develop` and push to your fork (`origin`) instead — see the fork + flow above. 3. **Re-confirm the boundaries before writing code** (see [`ARCHITECTURE.md`](ARCHITECTURE.md)): the change stays within the layering (`integrations → tools → registry → llm → core`, lower never imports higher); `core/` stays framework-free; and any endpoint/header/class name it depends on is already `VERIFIED` in [`docs/verified-apis.md`](docs/verified-apis.md) — if not, that is a verification question (`_verify.blocked(...)` or an - `Unverified(...)` placeholder), not a place to guess (§0.3). + `Unverified(...)` placeholder), not a place to guess (verification discipline). 4. **Commit, push, open a PR into `develop`.** Once you're on a correctly-named branch, commit and push autonomously — the branch is the isolation boundary. Commit messages cite `§N.N` when the change implements or modifies spec-governed behavior, e.g. `fix(llm): correct proxy base URL handling - (§2.1)`. + (config resolution)`. **Use a worktree when there's any chance of a parallel session** (another editor window, a running dev server, a `pytest --looponfail` holding files): one issue = @@ -102,11 +164,11 @@ cd python pytest -q # the `test` matrix job (3.10/3.11/3.12 in CI) mypy # mypy --strict, BLOCKING ruff check . # E,F,I,UP,B; line-length 100 -lint-imports # the §1.1 layered, framework-free-core contract +lint-imports # the layered, framework-free-core contract ``` If the diff touches an adapter or framework wiring, also run the signature check -(the executable form of the §8 verification step, and the nightly-matrix gate): +(the executable form of the `docs/verified-apis.md §8` verification step, and the nightly-matrix gate): ```bash python scripts/verify_frameworks.py @@ -116,6 +178,18 @@ If you added or touched an adapter, sanity-check that a bare `pip install -e ".[dev]"` + `python -c "import donkey_kit"` still succeeds — that's the `base-only` CI job catching a framework import that leaked into a lower layer. +**(fork)** GitHub withholds repository secrets from pull requests opened from a +fork, so the secret-gated jobs (the `--live` framework round-trip and anything +reading the `DONKEY_LLM_PROXY_*` env vars) do **not** run on your PR — a +maintainer runs them before merge. Run everything that needs no secrets locally +so the gate is green on what CI *can* check on a fork: `pytest -q tests/unit`, +`mypy`, `ruff check .`, `lint-imports`, and offline `python +scripts/verify_frameworks.py` (no `--live`). The `sandbox` and `local_gateway` +suites need creds/docker you likely don't have; they clean-skip, which is +correct (Section 2). And **never flip a `docs/verified-apis.md` row or `verified=True` +from a fork** — that requires a real sandbox round-trip only a maintainer can +run (verification discipline); raise it in the issue instead. + ### The PR The PR targets `develop` and its body includes `Closes #`, a `## Summary` @@ -141,7 +215,8 @@ Merge method is fixed by direction — don't pick per PR: Never rebase-merge into `develop`, never squash or fast-forward `develop` into `main`, and never merge `main` back into `develop` (cherry-pick a hotfix onto -`develop` instead). +`develop` instead). **(fork)** You can't run this step — a maintainer +squash-merges your PR and closes the linked issue with the merge SHA. After a PR merges, **close the linked issue explicitly** with the merge SHA — don't rely on GitHub auto-close, which can silently miss. Closing the issue is @@ -179,7 +254,7 @@ import to a file under `tests/unit/`** — that's exactly the drift this job exists to catch. Error-classification changes must keep the taxonomy invariants provable: `PolicyViolation` stays distinct from the retryable `UpstreamModelError`, and every `PolicyViolation` carries a non-empty -`remediation` (assert it directly). See [`ARCHITECTURE.md`](ARCHITECTURE.md#error-taxonomy-design-24) +`remediation` (assert it directly). See [`ARCHITECTURE.md`](ARCHITECTURE.md#error-taxonomy-design-bg-12) for why. ### The conformance kit — "never a silent skip" @@ -198,7 +273,7 @@ frameworks or it doesn't belong there. ### Fixture-driven tests — captures, not conveniences `tests/fixtures/anypoint/` holds **real captures** from a sandbox org, not -hand-written JSON. The error taxonomy is fixture-derived (§8.2), not +hand-written JSON. The error taxonomy is fixture-derived (BG §1.5), not assumption-derived. If you need a new response shape, capture it for real and document its provenance in the relevant `README` (org id, environment, CLI/proxy version, what produced the rejection, confirmation nothing sensitive @@ -213,7 +288,7 @@ job via `pytest -q -m local_gateway`); `sandbox` is exercised by `tests/sandbox/` (#400), which calls the real provisioned proxies and is **never** run in CI — it is opt-in, local only. -- **`@pytest.mark.local_gateway`** needs a local Omni Gateway via docker (§6.5). +- **`@pytest.mark.local_gateway`** needs a local Omni Gateway via docker (the Verification milestone). Docker is a hard dependency **for this feature only** — it ships behind the optional `[local]` extra; the rest of the SDK works without it. The local harness (`Governance.simulate()`) is designed so that **port allocation is @@ -242,13 +317,13 @@ with `pytest -q -m local_gateway` / `-m sandbox` (with the service/env in place) ### `scripts/verify_frameworks.py` — signatures, outside pytest -This is the executable §0.3 verification step for adapters' native constructor +This is the executable verification-discipline step for adapters' native constructor signatures (`docs/verified-apis.md` §8): does the exact class we name exist and accept the exact kwargs we pass, against the framework as actually installed. `--live` adds one real completion round-trip (needs the three `DONKEY_LLM_PROXY_*` env vars); `--only ` restricts scope; -`--emit-verified` prints §8 markdown rows after maintainer sign-off. A -`_verify.blocked(...)`-guarded adapter correctly shows as `BLOCKED (§0.3)`, not a +`--emit-verified` prints `docs/verified-apis.md §8` markdown rows after maintainer sign-off. A +`_verify.blocked(...)`-guarded adapter correctly shows as `BLOCKED (verification discipline)`, not a failure — don't "fix" the script to make a genuinely-blocked adapter pass. ### Commands @@ -284,14 +359,14 @@ build plan; the load-bearing rules: `import donkey_kit` succeed with no framework installed, and the `base-only` job enforces it. The layering (`integrations → tools → registry → llm → core`, lower never imports higher) is enforced by `lint-imports`. See - [`ARCHITECTURE.md`](ARCHITECTURE.md#layered-architecture-11). -- **Verification guards (§0.3).** Never invent an endpoint, header, or class + [`ARCHITECTURE.md`](ARCHITECTURE.md#layered-architecture). +- **Verification guards.** Never invent an endpoint, header, or class name. Use `core/_verify.py`: `blocked("…")` where there's no defensible placeholder, `Unverified(...)` for an overridable best-guess that warns once. Flip `verified=True` **and** the row in `docs/verified-apis.md` together, never one without the other. Details in - [`ARCHITECTURE.md`](ARCHITECTURE.md#verification-discipline-03). -- **Extras are floors, never ceilings (§8.4).** No upper version pins in + [`ARCHITECTURE.md`](ARCHITECTURE.md#verification-discipline). +- **Extras are floors, never ceilings.** No upper version pins in `pyproject.toml` — add `foo>=X`, never `foo **Project status — alpha, pre-release.** This is `v0.1.0.dev1` +> **Project status — alpha, pre-release.** This is `v0.1.0.dev2` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; > most other surfaces are verification-gated (see -> [What's verified](#whats-verified-03) below). **Not yet published to PyPI** — +> [What's verified](#whats-verified-verification-discipline) below). **Not yet published to PyPI** — > [install from source](#install). **Unofficial:** an independent project, > **not** affiliated with or endorsed by Salesforce or MuleSoft. @@ -21,7 +21,7 @@ in your own IDE, without adopting Mule. > maps every renamed import, class, CLI, config key, and environment variable, > and calls out the breaking OpenTelemetry attribute-namespace change. -> ### Support & trademark statement (please read — §0.4) +> ### Support & trademark statement (please read) > > **"Agent Fabric" is a MuleSoft (Salesforce) product name, not a generic > term.** `MuleSoft`, `Anypoint`, `Omni Gateway`, and `Agent Fabric` are @@ -98,7 +98,7 @@ demand evidence, one at a time (#223/#244) — never guessed up front. See the [framework pages](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/) for each. -## What's verified (§0.3) +## What's verified (verification discipline) The **LLM data plane** — governed model access through the Omni Gateway proxy — is live-verified against a real Anypoint sandbox, and both the framework-free @@ -111,7 +111,7 @@ currently includes Exchange→MCP tool discovery, the provisioning control-plane and the exact framework adapter class names/kwargs. The discipline behind this is documented in -[`ARCHITECTURE.md` → Verification discipline](ARCHITECTURE.md#verification-discipline-03); +[`ARCHITECTURE.md` → Verification discipline](ARCHITECTURE.md#verification-discipline); the row-by-row worklist is [`docs/verified-apis.md`](docs/verified-apis.md). ## Conformance exemptions @@ -120,7 +120,7 @@ The [conformance plugin](https://donkey-development-kit.github.io/donkey-develop holds the SDK to the same bar it asks of your agent. Where a framework legitimately cannot satisfy a scenario, the reason is asserted in code (`KNOWN_LIMITATIONS`) and published here as credibility — never a silent skip -(§8.1): +(the conformance kit): | Framework | Scenario | Why it's exempt | | --- | --- | --- | diff --git a/docs/README.md b/docs/README.md index 3f334ccb..f2e3c2fb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,17 +8,17 @@ consumer docs: for *using* the SDK see the Nextra site under `website/`; for [`../spec/`](../spec/) — the build plan (phases, invariants) and the build guide (feature scope, cited `BG §N.N`). -Everything here serves the **verification discipline** (§0.3): *never invent an +Everything here serves the **verification discipline**: *never invent an endpoint, header name, or class name.* The files track what has been proven against a real Anypoint sandbox versus what is still assumed or blocked. | File | What it is | | --- | --- | | [`verified-apis.md`](verified-apis.md) | **The verification ledger** — the single source of truth for every endpoint, header, and class name the SDK touches and its status (`VERIFIED (LIVE)` / `VERIFIED (CLI)` / `VERIFIED (plugin)` / `UNVERIFIED` / blocked). When a value is confirmed against a sandbox, flip its row here **and** set `verified=True` in `core/_verify.py` — the two edits move together. | -| [`m1-completion-checklist.md`](m1-completion-checklist.md) | **The 0.1.0 milestone tracker** (build plan §9.1, "M1 — Model access"). A status-keyed checklist toward a truthful, publishable release; a box flips to ✅ only when the fact is confirmed against the installed framework / real sandbox, never just because the code that assumes it was written. | -| [`unsupported-boundary.md`](unsupported-boundary.md) | **The §9.3 procurement doc** — classifies every platform API the SDK calls (public / no-SLA / undocumented), so enterprise-buyer questions get a five-minute answer instead of a two-week stall. Its "Undocumented surfaces" section is designed to stay empty. | +| [`m1-completion-checklist.md`](m1-completion-checklist.md) | **The 0.1.0 milestone tracker** (build plan Phase 1, "M1 — Model access"). A status-keyed checklist toward a truthful, publishable release; a box flips to ✅ only when the fact is confirmed against the installed framework / real sandbox, never just because the code that assumes it was written. | +| [`unsupported-boundary.md`](unsupported-boundary.md) | **The procurement doc** — classifies every platform API the SDK calls (public / no-SLA / undocumented), so enterprise-buyer questions get a five-minute answer instead of a two-week stall. Its "Undocumented surfaces" section is designed to stay empty. | | [`releasing.md`](releasing.md) | **How a release reaches PyPI** — the Trusted Publishing (OIDC) workflow wired as `.github/workflows/release.yml` (#206), the public API surface semver governs, and the one-time human step to register the trusted publisher. The version scheme and tagging live in the `ddk-release` skill; this is the publish half. | *"Agent Fabric", "Anypoint", and "Omni Gateway" are Salesforce trademarks; this project is a descriptive, non-first-party SDK for consuming those capabilities -(§0.4).* +(the trademark/support boundary).* diff --git a/docs/m1-completion-checklist.md b/docs/m1-completion-checklist.md index c7038534..c16f2cae 100644 --- a/docs/m1-completion-checklist.md +++ b/docs/m1-completion-checklist.md @@ -1,11 +1,11 @@ # M1 — Model access: completion checklist (→ 0.1.0) Tracks the work between today's state and a **truthful, publishable 0.1.0** -(build plan §9.1 "M1 — Model access"). M1 ships Pillar 1 only — governed model +(build plan Phase 1 "M1 — Model access"). M1 ships Pillar 1 only — governed model access through the Omni Gateway LLM proxy, plus the four Tier-1 adapters, the conformance kit, the `lint` command, and a docs skeleton. -> **Rule that governs every unchecked box (§0.3):** never invent an endpoint, +> **Rule that governs every unchecked box (verification discipline):** never invent an endpoint, > header, or class name. A box flips to ✅ only when the fact is confirmed > against the *installed framework* / *real sandbox* — not when the code that > assumes it is written. @@ -15,10 +15,10 @@ external verification (cannot be closed by writing code). --- -## 1. The critical path — verify the 8 constructor signatures (§8) +## 1. The critical path — verify the 8 constructor signatures (docs/verified-apis.md §8) **This is the only thing between us and a truthful M1 for the code that already -exists.** Every adapter is wired to the *live-verified proxy contract* (§2), but +exists.** Every adapter is wired to the *live-verified proxy contract* (the framework-free core), but `docs/verified-apis.md` §8 currently lists **all eight** framework constructor/kwarg signatures as `UNVERIFIED`. The adapters' inline `# verified: docs §8` comments are pointers to that table, **not** proof. @@ -32,30 +32,30 @@ checks per framework and never guesses a name: It calls the adapter's factory, which imports the exact class we name and constructs it with the exact kwargs we pass. A wrong class path → `ImportError`/`AttributeError`; a wrong kwarg → `TypeError`. It then imports - the class from its **recorded §8 path** and checks the object `isinstance` of + the class from its **recorded docs/verified-apis.md §8 path** and checks the object `isinstance` of it, so a silently-renamed or re-exported class is caught. **Construction succeeding is the signature verification.** - **B — Live round-trip (`--live`; needs the 3 `DONKEY_LLM_PROXY_*` env vars).** Makes one real completion. Only LangGraph's runtime call (`ChatOpenAI.ainvoke`) is exercised directly — the other seven frameworks' agent-loop APIs are themselves unverified, so the harness constructs the object and relies on the - already-live-verified shared proxy path (§2) rather than guessing a method. + already-live-verified shared proxy path (the framework-free core) rather than guessing a method. ```bash cd python pip install -e ".[dev,llm,langgraph]" # install the frameworks you're verifying python scripts/verify_frameworks.py # signature check, all installed python scripts/verify_frameworks.py --live # + one real proxy round-trip -python scripts/verify_frameworks.py --emit-verified # prints §8 markdown rows to paste +python scripts/verify_frameworks.py --emit-verified # prints docs/verified-apis.md §8 markdown rows to paste ``` Per framework, "verified" means: **install the real package → harness green -(signature + live) → maintainer signs off on scope → flip the §8 row and paste +(signature + live) → maintainer signs off on scope → flip the docs/verified-apis.md §8 row and paste the confirmed class path → remove/relax the code's UNVERIFIED note.** ### Per-framework signature status -| Framework | Tier | §8 class we target | Signature (harness) | Live round-trip | §8 row flipped | +| Framework | Tier | docs/verified-apis.md §8 class we target | Signature (harness) | Live round-trip | docs/verified-apis.md §8 row flipped | |---|---|---|---|---|---| | LangGraph | 1 | `langchain_openai.ChatOpenAI` | ✅ confirmed by harness (installed locally; public path re-export verified) | 🟡 needs a `--live` run against the sandbox | ⬜ pending maintainer sign-off | | Google ADK | 1 | `google.adk.models.lite_llm.LiteLlm` | ⬜ install `[adk]` + run harness | ⬜ | ⬜ | @@ -72,11 +72,11 @@ the confirmed class path → remove/relax the code's UNVERIFIED note.** --- -## 2. Definition of done, per adapter (§9.2) +## 2. Definition of done, per adapter Applies to each Tier-1 adapter before it counts as shipped. -| DoD item (§9.2) | LangGraph | ADK | Strands | Agent Framework | +| DoD item | LangGraph | ADK | Strands | Agent Framework | |---|---|---|---|---| | 1. Passes the conformance kit, or has an asserted documented exemption | ⬜ | 🟡 (correlation-id exemption recorded in `KNOWN_LIMITATIONS`) | ⬜ | ⬜ | | 2. Runnable `examples//` (env-vars-only) | ✅ | ✅ | ✅ | ✅ (guards the blocked path) | @@ -86,12 +86,12 @@ Applies to each Tier-1 adapter before it counts as shipped. > DoD #1 is the real gap: the conformance **scenario bodies** are not yet wired > (`tests/conformance/suite.py` fixes the scenario list + exemption table only). -> Wiring them against the captured contract fixtures (§8.2) is the largest +> Wiring them against the captured contract fixtures (BG §1.5) is the largest > remaining M1 build task after signature verification. --- -## 3. Remaining M1 build tasks (code, once §1 is unblocked) +## 3. Remaining M1 build tasks (code, once Section 1 is unblocked) - ⬜ **Conformance kit bodies** — implement the M1-relevant scenarios (`simple_completion`, `streaming_completion`, `attribution_headers_present`, @@ -116,16 +116,16 @@ Applies to each Tier-1 adapter before it counts as shipped. - ✅ `ruff check .` - ✅ `mypy` (`--strict`, incl. `scripts/verify_frameworks.py`) -- ✅ `lint-imports` (2 contracts: core-is-framework-free, layered §1.1) +- ✅ `lint-imports` (2 contracts: core-is-framework-free, layered architecture) - ✅ `pytest` (unit + ergonomics) - ⬜ nightly framework matrix green on at least the Tier-1 four (needs their extras installable in CI) --- -## 5. Publish-readiness (§0.4 / §9.3) +## 5. Publish-readiness (the trademark/support boundary / docs/unsupported-boundary.md) -- ⬜ Resolve the **maintainer / support statement** TODO in the README (§0.4) — +- ⬜ Resolve the **maintainer / support statement** TODO in the README (the trademark/support boundary) — blocks any public release; decide the distribution name if not MuleSoft-endorsed. - ✅ `docs/unsupported-boundary.md` exists and is linked from the README. - ⬜ Tag `0.1.0` only after the Tier-1 four are signature-verified **and** their @@ -134,7 +134,7 @@ Applies to each Tier-1 adapter before it counts as shipped. --- _Definition of "M1 done": the four Tier-1 adapters are signature-verified against -installed packages (§1), pass the wired conformance scenarios or carry an -asserted exemption (§2), all gates + the nightly matrix are green (§4), and the -§0.4 support statement is resolved (§5). Everything else in the plan (Pillars 2–3, +installed packages (Section 1), pass the wired conformance scenarios or carry an +asserted exemption (Section 2), all gates + the nightly matrix are green (Section 4), and the +trademark/support boundary support statement is resolved (Section 5). Everything else in the plan (Pillars 2–3, Publication, TypeScript) is out of M1 scope._ diff --git a/docs/releasing.md b/docs/releasing.md index f2d1305a..a0e2bbc4 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -27,7 +27,7 @@ of the **milestone being shipped**. The milestone titles are the ground truth A milestone ships its **final** version only when it reaches **0 open issues**. The `Verification` and `Upstream gaps` milestones have no version and never ship -— they are standing §0.3 gates, not releases. +— they are standing verification discipline gates, not releases. ### The pre-release ladder @@ -98,7 +98,7 @@ There are **two paths, one per destination** (#410), and they never overlap: Either trigger runs the same `build` job first — sdist + wheel, `twine check`, and the assertion that the built metadata carries only `>=` floors -(floors-never-ceilings, §8.4) — then uploads via `pypa/gh-action-pypi-publish`. +(floors-never-ceilings) — then uploads via `pypa/gh-action-pypi-publish`. A manual dispatch is **structurally incapable** of reaching prod: the `publish-pypi` job gates on `github.event_name == 'release'`, so only a published Release can trigger it. The workflow never creates tags or releases — it only @@ -126,8 +126,8 @@ conformance harness) are **not** part of the contract. There is intentionally **no `CHANGELOG.md`**. The **GitHub Release is the changelog** (`[project.urls].Changelog` points at the Releases feed); release -notes call out breaking changes, §0.3 verification-status flips, and §8.4 extras -changes. For a pre-release, the notes also state plainly what is *not* yet real +notes call out breaking changes, verification-status flips, and extras +changes (the floors-never-ceilings rule). For a pre-release, the notes also state plainly what is *not* yet real (e.g. "pre-MVP: docs and scaffolding only") so a `.devN` build never reads like a usable SDK. diff --git a/docs/unsupported-boundary.md b/docs/unsupported-boundary.md index 4cd8af8a..f48de37f 100644 --- a/docs/unsupported-boundary.md +++ b/docs/unsupported-boundary.md @@ -1,4 +1,4 @@ -# Unsupported boundary (§9.3) +# Unsupported boundary Every platform API the SDK calls, classified. Enterprise buyers will ask; having this pre-written converts a two-week procurement stall into a five-minute @@ -13,7 +13,7 @@ Classification: | API / surface | Module | Classification | Justification / owner | |---|---|---|---| -| _(populate during M0/§0.3 as each surface is verified)_ | | | | +| _(populate during M0/verification discipline as each surface is verified)_ | | | | ## Undocumented surfaces diff --git a/docs/verified-apis.md b/docs/verified-apis.md index 1e8d38f1..b8455d76 100644 --- a/docs/verified-apis.md +++ b/docs/verified-apis.md @@ -1,4 +1,4 @@ -# Verified APIs — output of §0.3 (M0) +# Verified APIs — output of the verification discipline (M0) > **Status (2026-08-28): partially verified.** The **management/control-plane** > contract (OAuth token path, Exchange/API-Manager/gateway REST endpoints, @@ -41,7 +41,7 @@ > > Working instruction #2: *never invent an endpoint, header name, or class name.* > A fabricated endpoint that 404s in a customer sandbox destroys trust in the -> whole package (§0.3). +> whole package (verification discipline). Placeholder constants that gate behaviour live in `src/donkey_kit/core/_verify.py`. Each is emitted with a runtime warning until @@ -64,7 +64,7 @@ VERIFIED with a concrete path. | Auth env var names | `core/config.py` | VERIFIED (CLI) | `ANYPOINT_CLIENT_ID`, `ANYPOINT_CLIENT_SECRET`, `ANYPOINT_ORG`, `ANYPOINT_ENV`, `ANYPOINT_BEARER` — match DonkeyConfig | 2026-08-28 | CLI flag defaults (`agent-network:*`) | | Auth model | `core/auth.py` | VERIFIED (CLI) | connected-app `client_id`/`client_secret` OR direct `bearer` token both accepted | 2026-08-28 | CLI flags | | Org / business-group as attribution unit | `core/config.py` | VERIFIED (CLI) | root BG `anypoint-cbp-1780648272` = org id (UUID); assets publish under org id as Maven groupId | 2026-08-28 | `account:business-group:list`, `api-mgr:api:describe` | -| Environments | `core/config.py` §6.2 | VERIFIED (CLI) | `{Name, Id (UUID), Sandbox: Y/N}`; sandbox has `Design` + `Sandbox` | 2026-08-28 | `account:environment:list` | +| Environments | `core/config.py` (environment targeting) | VERIFIED (CLI) | `{Name, Id (UUID), Sandbox: Y/N}`; sandbox has `Design` + `Sandbox` | 2026-08-28 | `account:environment:list` | | OAuth2 token endpoint path | `core/auth.py` | VERIFIED (plugin) | `POST /accounts/api/v2/oauth2/token`, body `{client_id, client_secret, grant_type}` → `{access_token, expires_in}` | 2026-08-28 | §12.1 (`anypoint-cli-command/lib/login.js`) | | Region host variants (US/EU/CA/JP Hyperforce) | `core/config.py` | UNVERIFIED | US = `anypoint.mulesoft.com` confirmed; others pending | — | — | | Connected-app scopes: Exchange read | `core/auth.py`, docs | UNVERIFIED | — | — | — | @@ -89,14 +89,14 @@ request against the deployed gateway. | Endpoint/API surface | `llm/client.py` | VERIFIED (LIVE) | OpenAI **Responses API** works (`POST /openai-sdk/responses`, body `{model, input}`). Upstream registered as `https://api.openai.com/v1/`; `proxyUri http://0.0.0.0:8081/openai-sdk` | 2026-08-28 | `api:describe`, live probe | | Auth: header name / model | `core/transport.py`, `core/config.py` | VERIFIED (LIVE) | **`client_id` + `client_secret` request headers** (NOT bearer) — enforced by `client-id-enforcement` 1.3.3. A consumer **credential pair**, mapped to an Anypoint client application | 2026-08-28 | live probe + `policy:list` | | Auth: model-wallet ingress (alt to `client_id`/`client_secret`, #372) | `core/transport.py`, `core/config.py` — UNVERIFIED, not wired | UNVERIFIED | Wallet-backed model proxies auth via an **IdP-issued JWT + a client ID**, with **no `client_secret`**: the client ID travels as the `X-Client-Id` request header (selects the wallet) *and* as a `client_id` **JWT claim** (`#[authentication.properties.claims.client_id]`, read after the **JWT Validation** policy publishes verified claims); requires an org **IdP** (JWKS URL / signing key — "orgs without an IdP can't use model wallets"); the default **DataWeave Headers Transformation** + **Client ID Enforcement** policies are disabled. A **parallel** ingress model, NOT a replacement of the LIVE `client_id`/`client_secret` pair above. **Live-capture worklist:** confirm the exact `X-Client-Id` header name/casing, the JWT claim path, and the request auth shape against a wallet-enabled sandbox with an IdP configured — until then no `_verify.py` value is marked verified and no guard comes off (§12.8; a doc read is not a live capture). | — | `docs.mulesoft.com/general/exp-model-wallets-manage` (doc read, not a live probe) | -| Model routing | `llm/client.py`; consumed by `core/lastcall.py` + `core/transport.py` (#309) | VERIFIED (LIVE) | `model-based-routing` 1.0.3 reads `model` from body → provider. Response headers `x-llm-proxy-routing-type: ModelBased`, `x-llm-proxy-routing-fallback: false`, `x-llm-proxy-llm-provider: openai`, `x-llm-proxy-llm-model: gpt-5.1`, `x-llm-proxy-model-based-routing-success: Request successfully matched. …`. As of #309 the SDK surfaces these at `donkey.last_call` (`served_provider`/`served_model`/`routing_type`/`fallback`, `substituted` when served≠requested), emits them as `gen_ai.response.model` / `donkey.routing.type` / `donkey.routing.fallback` span attrs, and **does not retry** a `503` a fallback header already marked failed-over (§309, must not fight #183) | 2026-08-28 | live probe (`responses.success.headers.txt`) | +| Model routing | `llm/client.py`; consumed by `core/lastcall.py` + `core/transport.py` (#309) | VERIFIED (LIVE) | `model-based-routing` 1.0.3 reads `model` from body → provider. Response headers `x-llm-proxy-routing-type: ModelBased`, `x-llm-proxy-routing-fallback: false`, `x-llm-proxy-llm-provider: openai`, `x-llm-proxy-llm-model: gpt-5.1`, `x-llm-proxy-model-based-routing-success: Request successfully matched. …`. As of #309 the SDK surfaces these at `donkey.last_call` (`served_provider`/`served_model`/`routing_type`/`fallback`, `substituted` when served≠requested), emits them as `gen_ai.response.model` / `donkey.routing.type` / `donkey.routing.fallback` span attrs, and **does not retry** a `503` a fallback header already marked failed-over (#309, must not fight #183) | 2026-08-28 | live probe (`responses.success.headers.txt`) | | Token accounting (cost attribution) | `llm/*`, telemetry | VERIFIED (LIVE) | response `usage: {input_tokens, output_tokens, total_tokens, input_tokens_details, output_tokens_details}` — the detail sub-objects carry `cached_tokens` / `cache_write_tokens` (under `input_tokens_details`) and `reasoning_tokens` (under `output_tokens_details`), surfaced per-call at `donkey.last_call` and on the span (#307, see §3); also upstream `x-ratelimit-*` headers passed through (Go-style **duration-string** resets, e.g. `0s`/`12ms` — distinct from the gateway's own `x-llm-proxy-ratelimit`/`x-token-*` window, see §4) | 2026-08-28 | `responses.success.body.json` | | Request fields passed through | `llm/client.py` | VERIFIED (LIVE) | OpenAI body passed through verbatim; response returned verbatim (model `gpt-5.1`→`gpt-5.1-2025-11-13`, full Responses object) | 2026-08-28 | live probe | | Streaming support | `llm/client.py` | VERIFIED (LIVE) | `"stream": true` → `200`, `content-type: text/event-stream`, chunked SSE (`event: response.created` / `response.in_progress` / …); same `x-llm-proxy-*` headers | 2026-08-28 | live probe (`responses.stream.*`) | | `/models` endpoint | `llm/catalog.py` | VERIFIED (LIVE) | **Does not exist** — `GET /openai-sdk/models` → `404`, `x-llm-proxy-model-based-routing-success: Request passed through without model-based routing`. The proxy only routes requests carrying `model` in the body; no catalog endpoint. `llm/catalog.py` must source models elsewhere | 2026-08-28 | live probe (`models.notfound.headers.txt`) | | Supported providers | `llm/catalog.py`, `governance.py` | VERIFIED (CLI) | openai, azureopenai, gemini, **bedrock**, **anthropic** (each a `*-llm-provider-policy-flex` in org `68ef9520…`) | 2026-08-28 | `exchange:asset:list llm` | -## 3. Token attribution headers (**highest-priority unknown**, §0.3) +## 3. Token attribution headers (**highest-priority unknown**, verification discipline) Without these the core value proposition (per-agent cost attribution) does not work. @@ -135,7 +135,7 @@ happy path and surfaces them at `donkey.last_call`: `x-request-id` becomes `LastCall.api_instance_id` (`21133858`) and `LastCall.environment_id` (`3e6ce455-…`, a UUID — dashes but no dots, so a three-way split on `.` is unambiguous). An absent or unrecognised shape leaves both `None` and never -raises (§0.3). The response `x-correlation-id` is **not** parsed into the record +raises (verification discipline). The response `x-correlation-id` is **not** parsed into the record yet — whether it echoes the client-sent id is unconfirmed (#300). As of #307, `core/lastcall.py` also parses the per-call **usage token counts** @@ -173,7 +173,7 @@ The two #195 rows are **request** headers the SDK *sends* (the client→gateway join keys behind `donkey.run()` and `DonkeyError.correlation_id`/`.call_id`). The verified `x-correlation-id` above is the gateway's **response** echo — a different direction. Until an inbound-read name is confirmed against a sandbox, -both request-header names stay `Unverified(...)` placeholders and emit the §0.3 +both request-header names stay `Unverified(...)` placeholders and emit the verification-discipline one-time warning; a customer whose gateway reads different names points the SDK at them via config rather than the SDK guessing. @@ -187,7 +187,7 @@ value regardless of the header question — so cost attribution works in tracing today, and flips to on-the-wire headers the moment a customer sets the `cost_*_header` overrides or the gateway-side names are verified here. -## 4. Policy rejection response shapes (capture as fixtures, §8.2) +## 4. Policy rejection response shapes (capture as fixtures, BG §1.5) **Four rejection shapes** are LIVE-VERIFIED (2026-08-28) from the `openai-sdk` proxy (client-id-enforcement, upstream passthrough, PII, token-rate-limit). @@ -250,7 +250,7 @@ states the shape is unconfirmed and whose remediation asks the operator to file the observed status/headers/body so the shape can be typed (#184). This is the last row above; it is deliberately **not** an `AuthError`, even for a `403`, once the verified `www-authenticate` auth shape is excluded. Re-confirming all of these -against current docs and a sandbox is tracked in #253 (§0.3: no invented docs URL +against current docs and a sandbox is tracked in #253 (verification discipline: no invented docs URL or version is recorded for them). **Budget window emission — two forms, keyed on outcome not status class @@ -339,35 +339,35 @@ provisioning API. Exact REST calls behind the CLI are now recorded in §12 | Exact REST endpoints behind the CLI | VERIFIED (plugin) | gatewaymanager / runtimefabric / apimanager / amc / proxies / exchange paths + bodies | 2026-08-28 | §12.3–§12.5 | | `mulesoft/anypoint` Terraform provider coverage | UNVERIFIED | not needed if CLI/Maven path adopted | — | — | -## 6. Governance / local-mode (§6.7) +## 6. Governance / local-mode (the Verification milestone) | Item | Gates | Status | Finding | Source | |---|---|---|---|---| -| Can Local Mode run the LLM Proxy? | §6.4/§6.5 | UNVERIFIED | — | — | -| Can Local Mode run MCP Bridge? | §6.4/§6.5 | UNVERIFIED | — | — | +| Can Local Mode run the LLM Proxy? | the Verification milestone | UNVERIFIED | — | — | +| Can Local Mode run MCP Bridge? | the Verification milestone | UNVERIFIED | — | — | | Does Local Mode need a control-plane licence/registration artifact? | OSS/CI viability | UNVERIFIED | — | — | -| Which policies are Connected-Mode-only? (portability table) | §6.4 | UNVERIFIED | — | — | +| Which policies are Connected-Mode-only? (portability table) | the Verification milestone | UNVERIFIED | — | — | | Is "deployed to gateway" readable per API instance? | `require_deployed` | VERIFIED (CLI) | Yes — `api-mgr:api:list` (per env) + `:api:describe ` returns Endpoint URI, gateway, deployment target | 2026-08-28 | | Are applied policies readable per API instance? | governed-state join | VERIFIED (CLI) | Yes — `api-mgr:policy:list ` returns `{ID, Template ID, Asset ID, Asset Version, Label, Status, Configuration}` | 2026-08-28 | | Are governance ruleset results exposed via API? | `require_governance_pass` | PARTIAL | `governance:api` evaluates rulesets; `governance:profile:*` manages profiles (ruleset refs = Maven GAV). Result-read shape pending | 2026-08-28 | -| Can applied policies be fetched in bulk per environment? | §6.1.3 perf | UNVERIFIED | per-instance confirmed; bulk endpoint pending plugin analysis | — | -| MCP-specific + enforcement policies observed | §6.4 portability | VERIFIED (CLI) | `mcp-support` (`injectMcpNameHeaders`), `client-id-enforcement` (client_id/client_secret headers), `header-injection` (`x-gateway-token`) | 2026-08-28 | +| Can applied policies be fetched in bulk per environment? | governed-only discovery perf | UNVERIFIED | per-instance confirmed; bulk endpoint pending plugin analysis | — | +| MCP-specific + enforcement policies observed | the Verification milestone (portability) | VERIFIED (CLI) | `mcp-support` (`injectMcpNameHeaders`), `client-id-enforcement` (client_id/client_secret headers), `header-injection` (`x-gateway-token`) | 2026-08-28 | | Governed MCP endpoint URL shape | `McpServerHandle.endpoint_url` | VERIFIED (CLI) | ingress gw: `https://agent-network-ingress-gw-..cloudhub.io/mcp//` | 2026-08-28 | -| Governed **LLM proxy** policy stack (live) | governed-state join, §6.4 | VERIFIED (LIVE) | on `openai-sdk`: `cors 1.3.2`, `dataweave-headers-transformation 1.0.0`, `client-id-enforcement 1.3.3`, `llm-proxy-core 1.0.5`, `model-based-routing 1.0.3`, `openai-transcoding-policy 1.0.3` — all Enabled | 2026-08-28 | +| Governed **LLM proxy** policy stack (live) | governed-state join, the Verification milestone | VERIFIED (LIVE) | on `openai-sdk`: `cors 1.3.2`, `dataweave-headers-transformation 1.0.0`, `client-id-enforcement 1.3.3`, `llm-proxy-core 1.0.5`, `model-based-routing 1.0.3`, `openai-transcoding-policy 1.0.3` — all Enabled | 2026-08-28 | -## 7. Publication / Exchange (§7.9) +## 7. Publication / Exchange (BG §2.5) | Item | Gates | Status | Finding | Source | |---|---|---|---|---| -| First-class Exchange asset types for MCP servers & AI agents? | §7.2 + `asset_types` filter §6.1 | VERIFIED (CLI) | Yes — Exchange assets named "… MCP Server" and "… Agent Network"/agent, managed as API Manager instances | 2026-08-28: `api-mgr:api:list` | +| First-class Exchange asset types for MCP servers & AI agents? | BG §2.5 + `asset_types` filter (governed-only discovery) | VERIFIED (CLI) | Yes — Exchange assets named "… MCP Server" and "… Agent Network"/agent, managed as API Manager instances | 2026-08-28: `api-mgr:api:list` | | Publication mechanism for non-Mule assets (REST / CLI / Maven)? JVM needed? | §7 CI story | VERIFIED (CLI) | Maven + CLI: `agent-network project publish` publishes the built project to Exchange; **JVM required** (`mvnw`) | 2026-08-28: `agent-network:project:publish --help` | -| Publication uses Maven GAV coordinates | §7.5 | VERIFIED (CLI) | group-id/asset-id/asset-version; groupId defaults to org id | 2026-08-28: `:project:create --help`, `api-mgr:api:describe` | -| Documentation pages publishable? | §7.2 | PARTIAL | `exchange asset page` + `exchange asset resource` topics exist | 2026-08-28: `exchange:asset --help` | -| Can arbitrary metadata/tags be attached (content digest)? | §7.5 | UNVERIFIED | Tags field exists on instances (empty here); attach mechanism pending | — | -| Asset lifecycle states (draft/published/deprecated) via API? | `require_lifecycle`, §7.5 | PARTIAL | `Deprecated` + `Public` flags on instances; A2D showed `status: draft\|published` | 2026-08-28 | -| Native descriptor formats (MCP manifest, A2A card) vs file attach? | §7.3 output | VERIFIED (plugin) | typed Exchange files w/ fixed classifiers (`agent-metadata`, `mcp-metadata`, `llm-metadata`, `a2a-card`, `schema`, `mule-application`) attached to `agent-network` root asset | 2026-08-28 | §12.7 | +| Publication uses Maven GAV coordinates | BG §2.5 | VERIFIED (CLI) | group-id/asset-id/asset-version; groupId defaults to org id | 2026-08-28: `:project:create --help`, `api-mgr:api:describe` | +| Documentation pages publishable? | BG §2.5 | PARTIAL | `exchange asset page` + `exchange asset resource` topics exist | 2026-08-28: `exchange:asset --help` | +| Can arbitrary metadata/tags be attached (content digest)? | BG §2.5 | UNVERIFIED | Tags field exists on instances (empty here); attach mechanism pending | — | +| Asset lifecycle states (draft/published/deprecated) via API? | `require_lifecycle`, BG §2.5 | PARTIAL | `Deprecated` + `Public` flags on instances; A2D showed `status: draft\|published` | 2026-08-28 | +| Native descriptor formats (MCP manifest, A2A card) vs file attach? | BG §2.5 output | VERIFIED (plugin) | typed Exchange files w/ fixed classifiers (`agent-metadata`, `mcp-metadata`, `llm-metadata`, `a2a-card`, `schema`, `mule-application`) attached to `agent-network` root asset | 2026-08-28 | §12.7 | -## 8. Framework APIs (§3.3) — re-verify every constructor +## 8. Framework APIs (BG §1.8) — re-verify every constructor | Framework | Symbol / kwarg | Status | Verified value | Date | Source | |---|---|---|---|---|---| @@ -380,26 +380,29 @@ provisioning API. Exact REST calls behind the CLI are now recorded in §12 | LlamaIndex | `llama_index.llms.openai_like.OpenAILike(is_chat_model=True)` | UNVERIFIED | — | — | — | | Strands | `strands.models.openai.OpenAIModel(client_args={...})` | UNVERIFIED | — | — | — | -### 8.1 Known upstream incompatibilities (floors, not ceilings — §8.4) +### 8.1 Known upstream incompatibilities (floors, not ceilings) -Per §8.4 the extras declare **floors and no ceilings**, so a fresh resolve always +Per the floors-never-ceilings rule the extras declare **floors and no ceilings**, so a fresh resolve always takes the newest release. That is deliberate: the nightly framework matrix exists to find breakage early rather than to pin it away. Recorded here so a red matrix run is diagnosable instead of surprising. | Dependency | Breaks at | Symptom | Status | Date | |---|---|---|---|---| -| `openai` | `>=3.0` | The SDK passes its shared `DonkeyAsyncClient` (an `httpx.AsyncClient` subclass) into `AsyncOpenAI(http_client=…)`. openai 3.x retyped that parameter to `httpx2.AsyncClient`, a distinct class from a separate distribution, so `mypy --strict` flagged `llm/client.py` and `integrations/openai_agents.py`. Calls still succeed at runtime — openai duck-types the client — so this was a typecheck failure, not a test failure. **Mitigated** with a targeted `# type: ignore[arg-type]` at the three call sites (`llm/client.py` ×2, `integrations/openai_agents.py`); `mypy --strict` passes under openai 3.x (#137). | MITIGATED | 2026-09-02 | +| `openai` | `>=3.0` | The SDK passes its shared `DonkeyAsyncClient` (an `httpx.AsyncClient` subclass) into `AsyncOpenAI(http_client=…)`. openai 3.x retyped that parameter to `httpx2.AsyncClient`, a distinct class from a separate distribution, so `mypy --strict` flagged `llm/client.py` and `integrations/openai_agents.py`. **Type-annotation-only** — when an `http_client` is injected, openai builds and sends every request *through that client*, so `httpx2` never touches our path. **Mitigated** with a targeted `# type: ignore[arg-type]` at the three call sites (`llm/client.py` ×2, `integrations/openai_agents.py`); `mypy --strict` passes under openai 3.x (#137). **Runtime re-verified and test-pinned** against openai 3.x (async + sync): `test_llm_client_openai3_injection.py` drives `donkey.llm.client()` through a mock transport and asserts the `client_id`/`client_secret` pair is injected and the base URL carries no `/v1` (#18). | MITIGATED | 2026-09-20 | The targeted ignores keep `mypy --strict` green against the newest `openai` a fresh -resolve pulls (§8.4). A full fix — migrating `core/transport.py` off `httpx` onto +resolve pulls (the floors-never-ceilings rule). A full fix — migrating `core/transport.py` off `httpx` onto `httpx2` — also touches every adapter that accepts `http_async_client` / -`client_args`, so it is a core-layer change (§1.1) and out of scope for the mitigation. +`client_args`, so it is a core-layer change (the layered architecture) and out of scope for the mitigation. +The re-verification (#18) confirms it is unnecessary: injection works end to end under openai 3.x, and +`test_llm_client_openai3_injection.py` is the standing gate that would fail loudly if a future openai +release actually broke it — the point of keeping the extras floor-only rather than capping `openai<3`. Because the ignores are code-specific and `--strict` enables `warn_unused_ignores`, a pinned `openai<3` (where the mismatch does not occur) would now flag them as *unused* — a non-issue while the extras stay floor-only and every resolve takes openai 3.x. -## 9. MCP tool binding classes (§4.4) — verify each name +## 9. MCP tool binding classes (BG §2.7) — verify each name | Framework | Binding class | Status | Source | |---|---|---|---| @@ -412,7 +415,7 @@ a non-issue while the extras stay floor-only and every resolve takes openai 3.x. | LlamaIndex | `llama_index.tools.mcp.BasicMCPClient` + `McpToolSpec` | UNVERIFIED | — | | Strands | `MCPClient(lambda: streamablehttp_client(...))` | UNVERIFIED | — | -## 10. Descriptor-derivation attributes (§7.3.1) — semi-public, put in nightly matrix +## 10. Descriptor-derivation attributes (BG §2.5) — semi-public, put in nightly matrix | Framework | Attributes read | Status | Source | |---|---|---|---| @@ -435,7 +438,7 @@ agent/API **design + mocking + Exchange-publishing** tool. Server specs carry is Anypoint-adjacent but wraps an **unknown backend REST contract**. These are therefore recorded as `VERIFIED-SHAPE-ONLY`: good enough to validate the SDK's value types against real data, **not** a license to point `ExchangeRegistry` at -`www.a2d-ai.com` as if it were Anypoint Exchange (that stays blocked, §0.3). +`www.a2d-ai.com` as if it were Anypoint Exchange (that stays blocked, verification discipline). | Item | Where used | Status | Verified value | |---|---|---|---| @@ -444,7 +447,7 @@ value types against real data, **not** a license to point `ExchangeRegistry` at | MCP protocol version | registry | VERIFIED-SHAPE-ONLY | `2025-06-18` | | Tool-descriptor shape | `tools/filter.py` `ToolDescriptor` | VERIFIED-SHAPE-ONLY | `{name, description, inputSchema (JSON Schema)}` per tool | | MCP server list record | registry search | VERIFIED-SHAPE-ONLY | `{id (uuid), name, type (openapi\|mock), description, status (published\|draft), enabled, organization_id, protocol_version, created_at, updated_at}` | -| Environment record | env targeting §6.2 | VERIFIED-SHAPE-ONLY | `{id, organization_id, asset_type (mcp_server\|agent_card\|rest_api), asset_id, name, base_url, environment_type (mocked\|pre_prod\|prod), auth_type, auth_config, extra_headers, timestamps}` | +| Environment record | environment targeting | VERIFIED-SHAPE-ONLY | `{id, organization_id, asset_type (mcp_server\|agent_card\|rest_api), asset_id, name, base_url, environment_type (mocked\|pre_prod\|prod), auth_type, auth_config, extra_headers, timestamps}` | ### Open design questions surfaced by the probe (need a platform-team decision) @@ -461,7 +464,7 @@ value types against real data, **not** a license to point `ExchangeRegistry` at still unconfirmed — `McpServerHandle.auth_required` default stays `True`. 4. **Environment→version semantics.** A2D environments are `mocked|pre_prod|prod` per asset, orthogonal to Exchange asset versions; - reconcile with the plan's `environment` targeting (§6.2). + reconcile with the plan's `environment` targeting. ## 12. Agent-donkey CLI plugin — direct REST contract (static analysis, 2026-08-28) @@ -473,7 +476,7 @@ official client that issues the real calls against the sandbox — so paths, header names, and bodies here are **read from the shipping client, not invented**. Status label **`VERIFIED (plugin)`** = the exact signature is known from authoritative client code; a live request has not additionally been -replayed. Per §0.3, code guards are only removed after the owning row is +replayed. Per the verification discipline, code guards are only removed after the owning row is confirmed and the maintainer signs off on scope (see "Unblocking" note below). ### 12.1 Auth (transport dependency `anypoint-cli-command/lib/`) @@ -632,7 +635,7 @@ This resolves §7's "native descriptor formats" question: descriptors are **typed Exchange files with fixed classifiers**, attached to the agent-network root asset — not a single bespoke manifest. -### 12.8 Unblocking guidance (§0.3) +### 12.8 Unblocking guidance (verification discipline) These rows are strong enough to *design against* but a live request should confirm each before its `NotImplementedError("blocked on verification: …")` diff --git a/python/README.md b/python/README.md index a2913df3..f99f6677 100644 --- a/python/README.md +++ b/python/README.md @@ -8,10 +8,10 @@ An SDK for consuming **Agent Fabric** capabilities — governed model access, governed tool access, and provisioning-as-code — from your own agent framework, in your own IDE, without adopting Mule. -> **Project status — alpha, pre-release.** This is `v0.1.0.dev1` +> **Project status — alpha, pre-release.** This is `v0.1.0.dev2` > (`Development Status :: 3 - Alpha`). The **LLM data plane is live-verified**; > most other surfaces are verification-gated (see -> [What's verified](#whats-verified-03) below). **Not yet published to PyPI** — +> [What's verified](#whats-verified-verification-discipline) below). **Not yet published to PyPI** — > [install from source](#install). **Unofficial:** an independent project, > **not** affiliated with or endorsed by Salesforce or MuleSoft. @@ -21,7 +21,7 @@ in your own IDE, without adopting Mule. > maps every renamed import, class, CLI, config key, and environment variable, > and calls out the breaking OpenTelemetry attribute-namespace change. -> ### Support & trademark statement (please read — §0.4) +> ### Support & trademark statement (please read) > > **"Agent Fabric" is a MuleSoft (Salesforce) product name, not a generic > term.** `MuleSoft`, `Anypoint`, `Omni Gateway`, and `Agent Fabric` are @@ -98,7 +98,7 @@ demand evidence, one at a time (#223/#244) — never guessed up front. See the [framework pages](https://donkey-development-kit.github.io/donkey-development-kit/frameworks/) for each. -## What's verified (§0.3) +## What's verified (verification discipline) The **LLM data plane** — governed model access through the Omni Gateway proxy — is live-verified against a real Anypoint sandbox, and both the framework-free @@ -111,7 +111,7 @@ currently includes Exchange→MCP tool discovery, the provisioning control-plane and the exact framework adapter class names/kwargs. The discipline behind this is documented in -[`ARCHITECTURE.md` → Verification discipline](ARCHITECTURE.md#verification-discipline-03); +[`ARCHITECTURE.md` → Verification discipline](ARCHITECTURE.md#verification-discipline); the row-by-row worklist is [`docs/verified-apis.md`](docs/verified-apis.md). ## Conformance exemptions @@ -120,7 +120,7 @@ The [conformance plugin](https://donkey-development-kit.github.io/donkey-develop holds the SDK to the same bar it asks of your agent. Where a framework legitimately cannot satisfy a scenario, the reason is asserted in code (`KNOWN_LIMITATIONS`) and published here as credibility — never a silent skip -(§8.1): +(the conformance kit): | Framework | Scenario | Why it's exempt | | --- | --- | --- | diff --git a/python/examples/adk/README.md b/python/examples/adk/README.md index eb2a5d10..b508668c 100644 --- a/python/examples/adk/README.md +++ b/python/examples/adk/README.md @@ -35,7 +35,7 @@ python examples/adk/main.py ## The manual equivalent The factory call is equivalent to building `LiteLlm` yourself with the -governed connection values (§3.3): +governed connection values (BG §1.8): ```python from google.adk.models.lite_llm import LiteLlm diff --git a/python/examples/adk/main.py b/python/examples/adk/main.py index f1d7aaff..f4fb0a76 100644 --- a/python/examples/adk/main.py +++ b/python/examples/adk/main.py @@ -1,4 +1,4 @@ -"""Google ADK adapter example (§3.3). +"""Google ADK adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,12 +8,12 @@ from donkey_kit.integrations.adk import model m = model("gpt-4o") # sent to LiteLLM as "openai/gpt-4o" -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified, and ``LiteLlm``/its kwargs are +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified, and ``LiteLlm``/its kwargs are verified per the FACTS table. What is NOT attempted here is a live inference call: ADK drives models through its own ``Runner``/``Agent`` session machinery, not a simple one-line method on the model object, and -guessing that call risks inventing an API (§0.3). Construction is this +guessing that call risks inventing an API (verification discipline). Construction is this example's verified surface — once you have ``m``, wire it into your own ADK ``Agent``/``Runner`` per ADK's own docs. """ @@ -60,7 +60,7 @@ def main() -> None: print( "Construction is the SDK's verified surface; drive this object with " "ADK's own Agent/Runner API (see this example's README) — that " - "runtime call is UNVERIFIED here and deliberately not guessed (§0.3)." + "runtime call is UNVERIFIED here and deliberately not guessed (verification discipline)." ) diff --git a/python/examples/agent_framework/README.md b/python/examples/agent_framework/README.md index f54691b7..50de7dfe 100644 --- a/python/examples/agent_framework/README.md +++ b/python/examples/agent_framework/README.md @@ -7,7 +7,7 @@ Agent Framework OpenAI-compatible chat client pointed at the governed Agent Fabric LLM proxy. The proxy *contract* (base URL, `client_id`/ `client_secret` header auth, attribution headers) is live-verified. The chat-client class name/path itself — `agent_framework.openai.OpenAIChatClient` -— and its base-URL kwarg (`model_id`) are **UNVERIFIED** (§8): Agent +— and its base-URL kwarg (`model_id`) are **UNVERIFIED** (docs/verified-apis.md §8): Agent Framework is a young package that has renamed classes recently. If the import fails, the factory raises a `NotImplementedError` ("blocked on verification") rather than guessing further; this example catches that and @@ -37,11 +37,11 @@ python examples/agent_framework/main.py ## The manual equivalent The factory call is equivalent to attempting to build the chat client -yourself with the governed connection values (§3.3) — **class name and -kwarg UNVERIFIED (§8), confirm against your installed version**: +yourself with the governed connection values (BG §1.8) — **class name and +kwarg UNVERIFIED (docs/verified-apis.md §8), confirm against your installed version**: ```python -# CLASS NAME/PATH AND KWARG NAMES UNVERIFIED (§8) — confirm before relying on this +# CLASS NAME/PATH AND KWARG NAMES UNVERIFIED (docs/verified-apis.md §8) — confirm before relying on this from agent_framework.openai import OpenAIChatClient client = OpenAIChatClient( diff --git a/python/examples/agent_framework/main.py b/python/examples/agent_framework/main.py index de64f83f..005c414d 100644 --- a/python/examples/agent_framework/main.py +++ b/python/examples/agent_framework/main.py @@ -1,4 +1,4 @@ -"""Microsoft Agent Framework adapter example (§3.3). +"""Microsoft Agent Framework adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -9,8 +9,8 @@ from donkey_kit.integrations.agent_framework import chat_client client = chat_client("gpt-4o") -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified, but the exact chat-client class +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified, but the exact chat-client class name/path (``agent_framework.openai.OpenAIChatClient``) and its base-URL kwarg (``model_id``) are UNVERIFIED — Agent Framework is young and has renamed classes recently. The factory raises ``NotImplementedError`` with a @@ -65,7 +65,7 @@ def main() -> None: print( "Construction is the SDK's verified surface; drive this object with " "Agent Framework's own Agent API (see this example's README) — that " - "runtime call is UNVERIFIED here and deliberately not guessed (§0.3)." + "runtime call is UNVERIFIED here and deliberately not guessed (verification discipline)." ) diff --git a/python/examples/anthropic/README.md b/python/examples/anthropic/README.md index 81851cf7..f711c244 100644 --- a/python/examples/anthropic/README.md +++ b/python/examples/anthropic/README.md @@ -8,12 +8,12 @@ proxy — `client_id`/`client_secret` header auth (not bearer), attribution headers, and the SDK's shared transport (retry/telemetry hooks). The returned object is Anthropic's own client, not a wrapper. -**Divergence, by design (§11.10).** Anthropic's native surface is a *client*, +**Divergence, by design — the framework wins (BG §1.8).** Anthropic's native surface is a *client*, and the model id is a per-call argument (`c.messages.create(model=..., ...)`), not a constructor one. So this adapter exposes `client()` rather than the `model(...)` factory the OpenAI-compatible adapters use. -**Unverified dependency (§8).** The Omni Gateway LLM proxy is verified +**Unverified dependency (docs/verified-apis.md §8).** The Omni Gateway LLM proxy is verified OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API route** is an open M0 verification item. Until confirmed, `client()` emits a one-time warning and no live call is made. Once a real route is confirmed, @@ -40,7 +40,7 @@ python examples/anthropic/main.py ## The manual equivalent The factory call is equivalent to building `AsyncAnthropic` yourself with the -governed connection values (§3.1): +governed connection values (BG §1.8): ```python import httpx diff --git a/python/examples/anthropic/main.py b/python/examples/anthropic/main.py index b5dc5c70..d1b0e0cd 100644 --- a/python/examples/anthropic/main.py +++ b/python/examples/anthropic/main.py @@ -1,4 +1,4 @@ -"""Anthropic SDK adapter example (§3.3). +"""Anthropic SDK adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,9 +8,9 @@ from donkey_kit.integrations.anthropic import client c = client() # the model id is a per-call argument, not a constructor one -Honest status (§0.3/§8): the proxy *contract* over the OpenAI-compatible route -(base URL, client_id/secret auth, attribution headers) is live-verified. What -is UNVERIFIED is whether the proxy exposes an **Anthropic-native Messages API +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* over the +OpenAI-compatible route (base URL, client_id/secret auth, attribution headers) is live-verified. +What is UNVERIFIED is whether the proxy exposes an **Anthropic-native Messages API route** at all (an open M0 item) — so this example constructs the client and emits the SDK's one-time unverified-route warning, but does NOT make a live call. Once a real Anthropic-native route is confirmed, override ``base_url`` and @@ -56,7 +56,8 @@ def main() -> None: print(f"Constructed native object: {type(c).__module__}.{type(c).__name__}") print( "Construction is the SDK's verified surface. The proxy's Anthropic-native " - "route is UNVERIFIED (§8) — confirm it, override base_url if needed, then " + "route is UNVERIFIED (docs/verified-apis.md §8) — confirm it, " + "override base_url if needed, then " "call c.messages.create(model=..., ...) per Anthropic's own docs." ) diff --git a/python/examples/crewai/README.md b/python/examples/crewai/README.md index 82befe6d..a11c2678 100644 --- a/python/examples/crewai/README.md +++ b/python/examples/crewai/README.md @@ -11,7 +11,7 @@ into a `crewai` `Agent`/`Crew`. `crewai.LLM` wraps LiteLLM, so the OpenAI-compatible route uses the `openai/` model prefix and headers go via `extra_headers`. LiteLLM owns the transport, so the SDK's per-run correlation ID degrades to per-client — a documented -conformance exemption (§8.1), the same one ADK has. +conformance exemption, the same one ADK has. > 📖 **Prefer reading to running?** The canonical walkthrough — install, > configure, and the manual equivalent — is in the docs: @@ -34,7 +34,7 @@ python examples/crewai/main.py ## The manual equivalent The factory call is equivalent to building `crewai.LLM` yourself with the -governed connection values (§3.1): +governed connection values (BG §1.8): ```python from crewai import LLM diff --git a/python/examples/crewai/main.py b/python/examples/crewai/main.py index 8d053af0..ccba1fbf 100644 --- a/python/examples/crewai/main.py +++ b/python/examples/crewai/main.py @@ -1,4 +1,4 @@ -"""CrewAI adapter example (§3.3). +"""CrewAI adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,13 +8,13 @@ from donkey_kit.integrations.crewai import llm model = llm("gpt-4o") -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified. ``crewai.LLM`` wraps LiteLLM, so +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified. ``crewai.LLM`` wraps LiteLLM, so the OpenAI-compatible route uses the ``openai/`` model prefix and header injection via ``extra_headers``; LiteLLM owns the transport, so per-run -correlation degrades (a documented conformance exemption, §8.1). No live +correlation degrades (a documented conformance exemption). No live inference call is attempted here: CrewAI LLMs are driven through a ``Crew`` / -``Agent``, and guessing that runtime call risks inventing an API (§0.3). +``Agent``, and guessing that runtime call risks inventing an API (verification discipline). Construction is this example's verified surface. """ @@ -60,7 +60,7 @@ def main() -> None: print( "Construction is the SDK's verified surface; pass this object into a " "crewai Agent/Crew per CrewAI's own docs — that runtime call is " - "UNVERIFIED here and deliberately not guessed (§0.3)." + "UNVERIFIED here and deliberately not guessed (verification discipline)." ) diff --git a/python/examples/langgraph/README.md b/python/examples/langgraph/README.md index 7836885c..51939132 100644 --- a/python/examples/langgraph/README.md +++ b/python/examples/langgraph/README.md @@ -30,7 +30,7 @@ against). `main()` then runs the full Scenario A: It is timed in CI (the `langgraph-demo` job) so this first-run experience can never silently rot. On startup you'll see two `UnverifiedValueWarning` lines -about the correlation header names — that is the SDK's honest §0.3 signal that +about the correlation header names — that is the SDK's honest verification discipline signal that those header names aren't yet confirmed against a live proxy, not an error. The model itself is a *native* `langchain_openai.ChatOpenAI` pointed at the @@ -88,7 +88,7 @@ pytest --donkey-conformance --agent=examples.langgraph.main:build ## The manual equivalent The factory call is equivalent to building `ChatOpenAI` yourself with the -governed connection values (§3.1): +governed connection values (BG §1.8): ```python import httpx diff --git a/python/examples/langgraph/main.py b/python/examples/langgraph/main.py index 51b7c986..282825d9 100644 --- a/python/examples/langgraph/main.py +++ b/python/examples/langgraph/main.py @@ -42,8 +42,8 @@ state; the run id bound with ``donkey.run(id=…)`` shows up there for free because LangGraph runs nodes on context-copying ``asyncio`` tasks (#195). -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret auth, -attribution headers) is live-verified. ``ChatOpenAI``/``StateGraph`` are the +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified. ``ChatOpenAI``/``StateGraph`` are the frameworks' own classes and ``.ainvoke`` is their documented API — construction via the SDK factory is the verified surface; everything after is the frameworks' own runtime. diff --git a/python/examples/llamaindex/README.md b/python/examples/llamaindex/README.md index 1edb3cd0..f4e13c65 100644 --- a/python/examples/llamaindex/README.md +++ b/python/examples/llamaindex/README.md @@ -36,7 +36,7 @@ python examples/llamaindex/main.py ## The manual equivalent The factory call is equivalent to building `OpenAILike` yourself with the -governed connection values (§3.3): +governed connection values (BG §1.8): ```python from llama_index.llms.openai_like import OpenAILike diff --git a/python/examples/llamaindex/main.py b/python/examples/llamaindex/main.py index 9a4b7c85..28eccc5a 100644 --- a/python/examples/llamaindex/main.py +++ b/python/examples/llamaindex/main.py @@ -1,4 +1,4 @@ -"""LlamaIndex adapter example (§3.3). +"""LlamaIndex adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -9,14 +9,14 @@ from donkey_kit.integrations.llamaindex import llm m = llm("gpt-4o") -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified, and ``OpenAILike``/its kwargs +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified, and ``OpenAILike``/its kwargs are verified per the FACTS table — including ``is_chat_model=True``, which the factory always sets (``OpenAILike`` defaults it to ``False``, which silently routes to the completions endpoint against a chat-only proxy; the single most common LlamaIndex-with-a-gateway bug). What is NOT attempted here is a live inference call: guessing the right one-line LlamaIndex call -(``.chat``, ``.achat``, ``.complete``, ...) risks inventing an API (§0.3). +(``.chat``, ``.achat``, ``.complete``, ...) risks inventing an API (verification discipline). Construction is this example's verified surface — once you have ``m``, use it with LlamaIndex's own query/chat engines per its own docs. """ @@ -63,7 +63,8 @@ def main() -> None: print( "Construction is the SDK's verified surface; drive this object with " "LlamaIndex's own query/chat engine API (see this example's README) " - "— that runtime call is UNVERIFIED here and deliberately not guessed (§0.3)." + "— that runtime call is UNVERIFIED here and deliberately not guessed " + "(verification discipline)." ) diff --git a/python/examples/openai_agents/README.md b/python/examples/openai_agents/README.md index 7fac9484..6d913514 100644 --- a/python/examples/openai_agents/README.md +++ b/python/examples/openai_agents/README.md @@ -32,7 +32,7 @@ python examples/openai_agents/main.py ## The manual equivalent The factory call is equivalent to building the model yourself with the governed -connection values (§3.1): +connection values (BG §1.8): ```python import httpx diff --git a/python/examples/openai_agents/main.py b/python/examples/openai_agents/main.py index 5709e460..a4835f5b 100644 --- a/python/examples/openai_agents/main.py +++ b/python/examples/openai_agents/main.py @@ -1,4 +1,4 @@ -"""OpenAI Agents SDK adapter example (§3.3). +"""OpenAI Agents SDK adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,12 +8,12 @@ from donkey_kit.integrations.openai_agents import model m = model("gpt-4o") -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified. Because the adapter builds the +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified. Because the adapter builds the underlying ``AsyncOpenAI`` client itself, header AND transport injection are both available (full injection). No live inference call is attempted here: the Agents SDK runs models through an ``agents.Agent`` + ``Runner``, and guessing -that runtime call risks inventing an API (§0.3). Construction is this example's +that runtime call risks inventing an API (verification discipline). Construction is this example's verified surface — pass ``m`` into ``agents.Agent(model=m)`` per the SDK's docs. """ @@ -60,7 +60,7 @@ def main() -> None: "Construction is the SDK's verified surface; pass this object into " "agents.Agent(model=...) and drive it with Runner per the SDK's own " "docs — that runtime call is UNVERIFIED here and deliberately not " - "guessed (§0.3)." + "guessed (verification discipline)." ) diff --git a/python/examples/quickstart/main.py b/python/examples/quickstart/main.py index 89a1c91f..6d0ba359 100644 --- a/python/examples/quickstart/main.py +++ b/python/examples/quickstart/main.py @@ -20,7 +20,7 @@ never real credentials. This is a *fixture replay*, not a real gateway: every response it serves carries ``x-donkey-simulator: true`` (BG §1.4). -The same code lives in ``website/pages/quickstart.mdx`` — the two are kept in +The same code lives in ``website/content/quickstart.mdx`` — the two are kept in lockstep so the documented snippet is exactly the executed one. """ diff --git a/python/examples/strands/README.md b/python/examples/strands/README.md index 16c6b599..0ef08592 100644 --- a/python/examples/strands/README.md +++ b/python/examples/strands/README.md @@ -35,7 +35,7 @@ python examples/strands/main.py ## The manual equivalent The factory call is equivalent to building `OpenAIModel` yourself with the -governed connection values (§3.3): +governed connection values (BG §1.8): ```python import httpx diff --git a/python/examples/strands/main.py b/python/examples/strands/main.py index 9d7b1ff0..7208b8ed 100644 --- a/python/examples/strands/main.py +++ b/python/examples/strands/main.py @@ -1,4 +1,4 @@ -"""Strands Agents adapter example (§3.3). +"""Strands Agents adapter example (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,12 +8,12 @@ from donkey_kit.integrations.strands import model m = model("gpt-4o") -Honest status (§0.3/§8): the proxy *contract* (base URL, client_id/secret -auth, attribution headers) is live-verified, and ``OpenAIModel``/its kwargs +Honest status (verification discipline / docs/verified-apis.md §8): the proxy *contract* (base URL, +client_id/secret auth, attribution headers) is live-verified, and ``OpenAIModel``/its kwargs are verified per the FACTS table. What is NOT attempted here is a live inference call: Strands models are normally driven through a ``strands.Agent`` session, not a simple one-line method on the model object, -and guessing that call risks inventing an API (§0.3). Construction is this +and guessing that call risks inventing an API (verification discipline). Construction is this example's verified surface — once you have ``m``, wrap it in your own ``strands.Agent`` per Strands' own docs. """ @@ -60,7 +60,7 @@ def main() -> None: print( "Construction is the SDK's verified surface; drive this object with " "Strands' own Agent API (see this example's README) — that runtime " - "call is UNVERIFIED here and deliberately not guessed (§0.3)." + "call is UNVERIFIED here and deliberately not guessed (verification discipline)." ) diff --git a/python/pyproject.toml b/python/pyproject.toml index 9b562803..d1891311 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "donkey-kit" -version = "0.1.0.dev1" +version = "0.1.0.dev2" description = "An SDK for consuming Agent Fabric capabilities (governed model access, governed tools, provisioning-as-code) from your own agent framework." readme = "README.md" requires-python = ">=3.10" @@ -17,7 +17,7 @@ classifiers = [ "Typing :: Typed", ] -# core/ has ZERO agent-framework dependencies. httpx only. (§1.1 hard rule) +# core/ has ZERO agent-framework dependencies. httpx only. (the layered architecture hard rule) dependencies = [ "httpx>=0.27", "pydantic>=2.6", @@ -42,8 +42,8 @@ Changelog = "https://github.com/Donkey-Development-Kit/donkey-development-kit/re llm = ["openai>=1.30"] # --- Per-framework integrations (one extra each) --- -# Floors, never ceilings (working instruction, §8.4). Class names / kwargs -# inside the adapters are marked UNVERIFIED until §0.3 confirms them. +# Floors, never ceilings (working instruction). Class names / kwargs +# inside the adapters are marked UNVERIFIED until the verification discipline confirms them. # The deep, conformance-gated adapter (BG §1.8, #198): its extra ships the graph # runtime so `interrupt()` composition and the conformance harness run against a # real LangGraph app, not just a bare ChatOpenAI. @@ -63,10 +63,10 @@ a2a = ["a2a-sdk>=0.1"] # --- Local gateway simulator (BG §1.4): `donkey mock` --- # Pure-Python ASGI (starlette) served by uvicorn. Plain `uvicorn`, NOT # `uvicorn[standard]`: uvloop/httptools are C extensions and would break BG §1.4's -# pure-Python promise. Floors only, no upper pins (§8.4). +# pure-Python promise. Floors only, no upper pins. local = ["starlette>=0.37", "uvicorn>=0.30"] -# --- Telemetry (§2.5, BG §1.6) --- +# --- Telemetry (BG §1.6) --- # The OTLP exporter is what makes zero-config export real (#194): with it and # `OTEL_EXPORTER_OTLP_ENDPOINT` set, spans flow to the customer's own sink with no # SDK-specific env var. We ship the http/protobuf exporter (the OTel-recommended @@ -74,14 +74,14 @@ local = ["starlette>=0.37", "uvicorn>=0.30"] # would also drag in `grpcio` (a heavy C-extension wheel). grpc stays opt-in: # `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` is honoured only if the caller also installs # `opentelemetry-exporter-otlp-proto-grpc` (a one-time warning says so otherwise). -# Floors only, no upper pins (§8.4). +# Floors only, no upper pins. otel = [ "opentelemetry-api>=1.24", "opentelemetry-sdk>=1.24", "opentelemetry-exporter-otlp-proto-http>=1.24", ] -# --- CLI + provisioning (§5) --- +# --- CLI + provisioning --- cli = ["typer>=0.12", "pyyaml>=6.0", "rich>=13.0"] # --- Conformance pytest plugin (BG §1.5): `donkey-kit[test]` --- @@ -91,7 +91,7 @@ cli = ["typer>=0.12", "pyyaml>=6.0", "rich>=13.0"] # tests — and the `donkey` fixture used with `simulate()` — run under the # repo-standard asyncio_mode="auto". The plugin itself needs no framework and no # `[llm]`; it drives the agent through the customer's own Donkey. Floors only, -# no upper pins (§8.4). +# no upper pins. test = ["pytest>=8.0", "pytest-asyncio>=0.23"] # --- Everything, for convenience --- @@ -196,7 +196,7 @@ select = ["E", "F", "I", "UP", "B"] # typer's API requires calling typer.Option(...) in argument defaults (B008). "src/donkey_kit/provisioning/cli.py" = ["B008"] -# --- import-linter: enforce the §1.1 layering rule in CI, from day one --- +# --- import-linter: enforce the layered architecture rule in CI, from day one --- [tool.importlinter] root_package = "donkey_kit" @@ -212,7 +212,7 @@ source_modules = [ forbidden_modules = ["donkey_kit.integrations"] [[tool.importlinter.contracts]] -name = "layered architecture (§1.1)" +name = "layered architecture" type = "layers" # integrations (top) may use everything below; tools resolves registry handles; # registry/llm sit above the framework-free core (bottom). Lower never imports @@ -225,6 +225,27 @@ layers = [ "donkey_kit.core", ] +[[tool.importlinter.contracts]] +name = "integrations are mutually independent (no cross-framework leakage, §1.1)" +type = "independence" +# Each adapter depends on exactly ONE agent framework. No adapter may import +# another, directly or indirectly — a convenient cross-import would drag a +# second framework's optional dependency onto an install that only asked for +# the first, quietly breaking the "extras are genuinely optional" guarantee +# (working instruction #6). The shared `_base` sibling (DonkeyConfig + +# transport wiring) is deliberately NOT listed: every adapter importing it +# downward is allowed and expected, and `_base` imports no adapter. +modules = [ + "donkey_kit.integrations.langgraph", + "donkey_kit.integrations.adk", + "donkey_kit.integrations.strands", + "donkey_kit.integrations.agent_framework", + "donkey_kit.integrations.openai_agents", + "donkey_kit.integrations.anthropic", + "donkey_kit.integrations.crewai", + "donkey_kit.integrations.llamaindex", +] + [[tool.importlinter.contracts]] name = "simulator is a dev-only sibling (BG §1.4)" type = "forbidden" diff --git a/python/scripts/verify_frameworks.py b/python/scripts/verify_frameworks.py index 7658a11d..cf5af824 100644 --- a/python/scripts/verify_frameworks.py +++ b/python/scripts/verify_frameworks.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Live-verify the framework constructor signatures (docs/verified-apis.md §8). -This is the executable form of the §0.3 verification step for Pillar 1's adapters. +This is the executable form of the verification discipline step for Pillar 1's adapters. It answers one question per framework: **does the exact class we name actually exist, and does it accept the exact kwargs we pass?** — without inventing anything. Nothing here is asserted as fact until this script confirms it against @@ -14,7 +14,7 @@ class path is wrong -> ImportError/AttributeError. If a kwarg name is wrong -> TypeError. Construction succeeding *is* the signature verification. The script then confirms the object's real ``module.ClassName`` matches the - value recorded in §8, so a silently-renamed class is caught too. + value recorded in docs/verified-apis.md §8, so a silently-renamed class is caught too. B. LIVE ROUND-TRIP (``--live``, needs the 3 DONKEY_LLM_PROXY_* env vars): make one real completion through the framework's *own* native call and @@ -30,7 +30,8 @@ class path is wrong -> ImportError/AttributeError. If a kwarg name is wrong python scripts/verify_frameworks.py # signature check, all installed python scripts/verify_frameworks.py --live # + one real proxy round-trip python scripts/verify_frameworks.py --only langgraph strands - python scripts/verify_frameworks.py --emit-verified # print §8 markdown rows to paste + python scripts/verify_frameworks.py --emit-verified + # print docs/verified-apis.md §8 markdown rows to paste python scripts/verify_frameworks.py --json Exit code is non-zero if any *installed* framework fails its signature check, so @@ -47,7 +48,7 @@ class path is wrong -> ImportError/AttributeError. If a kwarg name is wrong import traceback from dataclasses import asdict, dataclass, field -# --- ground truth: the exact §8 rows this script confirms -------------------- +# --- ground truth: the exact docs/verified-apis.md §8 rows this script confirms --- # (framework key, factory import path, factory fn, expected native module.Class) # This stays the FULL eight-framework roster so the offline signature check can be # run on demand for any of them. Only LangGraph is conformance-tested and carried @@ -98,7 +99,7 @@ def verdict(self) -> str: if not self.installed: return "NOT INSTALLED" if self.blocked: - return "BLOCKED (§0.3)" + return "BLOCKED (verification discipline)" if self.signature_ok and self.class_matches: return "VERIFIED" if self.live in ("ok", "not run") else "SIGNATURE OK / LIVE FAIL" if self.signature_ok and self.class_matches is False: @@ -125,7 +126,7 @@ def _actual_class(obj: object) -> str: def _import_expected(path: str) -> type | None: """Import the class named at ``path`` (e.g. ``langchain_openai.ChatOpenAI``) from its *public* recorded location. Returns None if the path does not - resolve — which is itself a signature failure (the §8 name is wrong).""" + resolve — which is itself a signature failure (the docs/verified-apis.md §8 name is wrong).""" import importlib module_path, _, attr = path.rpartition(".") @@ -150,9 +151,9 @@ def check_signature(res: Result, import_path: str, factory: str) -> object | Non fn = getattr(mod, factory) try: # Anthropic's native surface is a client; the model id is a per-call - # argument, so its factory takes no positional model (§3.3 divergence). + # argument, so its factory takes no positional model (BG §1.8 divergence). obj: object = fn() if res.framework == "anthropic" else fn(MODEL) - except NotImplementedError as exc: # blocked on verification (§0.3) + except NotImplementedError as exc: # blocked on verification res.installed = True res.blocked = True res.detail = str(exc).splitlines()[0] @@ -175,20 +176,24 @@ def check_signature(res: Result, import_path: str, factory: str) -> object | Non res.installed = True res.signature_ok = True res.actual_class = _actual_class(obj) - # Verify the §8 name by importing the class from its RECORDED public path and - # checking the constructed object is an instance of it. This is robust to + # Verify the docs/verified-apis.md §8 name by importing the class from its RECORDED public + # path and checking the constructed object is an instance of it. This is robust to # re-exports (``langchain_openai.ChatOpenAI`` is defined in an internal # submodule but re-exported at the package top level, which is the path users - # import and §8 records). + # import and docs/verified-apis.md §8 records). expected_cls = _import_expected(res.expected_class) if expected_cls is None: res.class_matches = False - res.notes.append(f"§8 path {res.expected_class!r} does not resolve to a class — fix §8") + res.notes.append( + f"docs/verified-apis.md §8 path {res.expected_class!r} does not resolve to a class " + "— fix docs/verified-apis.md §8" + ) else: res.class_matches = isinstance(obj, expected_cls) if not res.class_matches: res.notes.append( - f"object is {res.actual_class!r}, not a {res.expected_class!r} — update §8" + f"object is {res.actual_class!r}, not a {res.expected_class!r} " + "— update docs/verified-apis.md §8" ) return obj @@ -197,8 +202,8 @@ async def check_live(res: Result, obj: object) -> None: """Check B — one real completion through the framework's native call. Only LangGraph's runtime call is verified here. For the rest, the framework's - agent-loop API is itself unverified (§0.3), so we do not guess a method — the - proxy path they share is already live-verified via the raw client (docs §2). + agent-loop API is itself unverified (verification discipline), so we do not guess a method — the + proxy path they share is already live-verified via the raw client (docs/verified-apis.md §2). """ if res.framework == "langgraph": try: @@ -210,8 +215,8 @@ async def check_live(res: Result, obj: object) -> None: res.live = f"fail: {type(exc).__name__}: {exc}" return res.live = ( - "skipped: framework runtime call API not verified (§8/§9); " - "shared proxy path verified via raw client (§2)" + "skipped: framework runtime call API not verified (docs/verified-apis.md §8/§9); " + "shared proxy path verified via raw client (docs/verified-apis.md §2)" ) @@ -238,7 +243,7 @@ def print_table(results: list[Result]) -> None: w = max((len(r.framework) for r in results), default=9) header = ( f"\n{'framework':<{w}} {'installed':<9} {'signature':<9} " - f"{'class §8':<8} {'live':<6} verdict" + f"{'expected':<8} {'live':<6} verdict" ) print(header) print("-" * (w + 55)) @@ -257,8 +262,8 @@ def print_table(results: list[Result]) -> None: def emit_verified_rows(results: list[Result]) -> None: - """Print §8 markdown rows for frameworks whose signature is now confirmed, so - a maintainer can paste them into docs/verified-apis.md after sign-off.""" + """Print docs/verified-apis.md §8 markdown rows for frameworks whose signature is now confirmed, + so a maintainer can paste them into docs/verified-apis.md after sign-off.""" from datetime import date # local import; only used for this opt-in report today = date.today().isoformat() @@ -279,7 +284,7 @@ def main() -> int: ap.add_argument("--json", action="store_true", help="emit JSON instead of a table") ap.add_argument( "--emit-verified", action="store_true", - help="print §8 markdown rows for confirmed frameworks", + help="print docs/verified-apis.md §8 markdown rows for confirmed frameworks", ) args = ap.parse_args() diff --git a/python/src/donkey_kit/__init__.py b/python/src/donkey_kit/__init__.py index 0f41aa23..c37e8a41 100644 --- a/python/src/donkey_kit/__init__.py +++ b/python/src/donkey_kit/__init__.py @@ -1,16 +1,16 @@ """donkey-kit — an SDK for consuming Agent Fabric capabilities from your own agent framework. -See the README for the maintainer + support statement and the trademark note -(§0.4). "Agent Fabric" is a MuleSoft product name; this package is descriptive. +See the README for the maintainer + support statement and the trademark note. +"Agent Fabric" is a MuleSoft product name; this package is descriptive. -Public surface (§3.2): +Public surface (BG §1.8): from donkey_kit import Donkey donkey = Donkey.from_env() donkey.langgraph.chat_model("gpt-4o") # native ChatOpenAI -Working-instruction reminder (§0.3, #2): many platform endpoints/headers/class +Working-instruction reminder (verification discipline, #2): many platform endpoints/headers/class names are UNVERIFIED. Those code paths raise ``NotImplementedError("blocked on verification: …")`` rather than guessing. See docs/verified-apis.md. @@ -57,7 +57,7 @@ Publication, ) -__version__ = "0.1.0.dev1" +__version__ = "0.1.0.dev2" __all__ = [ "STRICT", diff --git a/python/src/donkey_kit/conformance/__init__.py b/python/src/donkey_kit/conformance/__init__.py index c0ca7e61..a0da2dcc 100644 --- a/python/src/donkey_kit/conformance/__init__.py +++ b/python/src/donkey_kit/conformance/__init__.py @@ -11,7 +11,7 @@ - :mod:`~donkey_kit.conformance.report` — the dependency-free status table. - :mod:`~donkey_kit.conformance.plugin` — the pytest11 plugin (auto-loaded). -**Dev-only sibling of the five production layers** (§1.1): an import-linter +**Dev-only sibling of the five production layers** (the layered architecture): an import-linter contract forbids ``core``/``llm``/``registry``/``tools``/``integrations`` from importing it. Because the pytest11 entry point auto-loads :mod:`~donkey_kit.conformance.plugin` on *every* pytest run where this package diff --git a/python/src/donkey_kit/conformance/gateway.py b/python/src/donkey_kit/conformance/gateway.py new file mode 100644 index 00000000..d890a47e --- /dev/null +++ b/python/src/donkey_kit/conformance/gateway.py @@ -0,0 +1,217 @@ +"""An out-of-process local gateway simulator a test can assert against (#278, `BG §1.5`). + +``simulate()`` (#190) and the in-process ``donkey`` fixture (#191) cover anything +that calls the SDK in-process. They cannot cover the cases the vision deck names +explicitly — a containerised agent, a Node service, an A2A client, a manual +``curl`` — none of which import ``donkey_kit``. Those need a **real listener on a +real port**, plus a way to ask the gateway what it *received* ("did my agent stop +after the refusal, or retry four more times?"). + +This module boots the same local gateway simulator +(:func:`donkey_kit.simulator.build_app`, BG §1.4) that ``simulate()`` and +``donkey mock`` replay — so a stock client sees byte-identical rejection shapes — +on an **ephemeral port** (bind ``0``) in a daemon thread, and wraps it in a +request-recording ASGI spy the test can assert on. The public entry point is the +``gateway`` pytest fixture in :mod:`~donkey_kit.conformance.plugin`; this module +is imported lazily from there so the auto-loaded plugin never drags ``uvicorn`` +onto every pytest run. + +Framework isolation (the layered architecture): ``uvicorn`` is imported lazily +inside :func:`start_gateway`, and ``starlette`` only inside the simulator's own +``build_app`` — so ``import donkey_kit.conformance.gateway`` stays green under the +base-only CI job (``[dev]`` only, no ``[local]`` extra). Taking the fixture +without that extra raises an :class:`ImportError` naming the exact +``pip install`` — never a bare ``ModuleNotFoundError`` and never a silent skip. +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Awaitable, Callable, Mapping, MutableMapping +from dataclasses import dataclass +from typing import Any + +from ..simulator.app import ASGIApp, SimulatorConfig, build_app +from ..simulator.scenarios import Scenario, parse_scenario + +__all__ = ["Gateway", "RecordedRequest", "start_gateway"] + +# The one extra that makes the out-of-process path real: starlette (the ASGI app) +# + uvicorn (the server). Both live in `[local]`, so the hint names that extra — +# not `[test]`, which ships the fixture itself but not the server it drives. +_LOCAL_EXTRA_HINT = ( + "The `gateway` fixture needs the local simulator server extra " + "(starlette + uvicorn). Install it with:\n" + ' pip install "donkey-kit[local]"' +) + +# Request headers whose values are redacted in the spy: the consumer-auth secret +# (BG §1.1 — the `client_secret` request header the transport sends) and any +# bearer token. Matched case-insensitively; ASGI already lowercases header names. +_REDACTED_HEADERS = frozenset({"client_secret", "authorization"}) +_REDACTED = "***" + +# How long to wait for uvicorn to report `started` before giving up. +_STARTUP_TIMEOUT_S = 10.0 + + +@dataclass(frozen=True) +class RecordedRequest: + """One request the gateway received, as seen from the gateway's side. + + ``headers`` has the consumer-auth secret redacted (see :data:`_REDACTED_HEADERS`), + so a captured request is safe to print in an assertion failure or a fixture + dump. It does not carry the body — the simulator replays captured shapes and + does not evaluate request content (#189/#250).""" + + method: str + path: str + headers: Mapping[str, str] + + +def _record(scope: MutableMapping[str, Any]) -> RecordedRequest: + """Build a :class:`RecordedRequest` from an ASGI ``http`` scope, redacting + the consumer-auth secret and any bearer header.""" + raw: Any = scope.get("headers") or [] + headers: dict[str, str] = {} + for name, value in raw: + key = name.decode("latin-1") + headers[key] = _REDACTED if key.lower() in _REDACTED_HEADERS else value.decode("latin-1") + return RecordedRequest( + method=str(scope.get("method", "")), + path=str(scope.get("path", "")), + headers=headers, + ) + + +class _RecordingApp: + """An ASGI app that records every HTTP request, then delegates to a swappable + inner app. The inner app is swapped (never the outer one uvicorn holds) so + :meth:`Gateway.set_scenarios` can reconfigure the simulator live, without + restarting the server or losing the request log.""" + + def __init__(self, app: ASGIApp) -> None: + self.app = app + self.requests: list[RecordedRequest] = [] + + async def __call__( + self, + scope: MutableMapping[str, Any], + receive: Callable[[], Awaitable[MutableMapping[str, Any]]], + send: Callable[[MutableMapping[str, Any]], Awaitable[None]], + ) -> None: + if scope.get("type") == "http": + self.requests.append(_record(scope)) + await self.app(scope, receive, send) + + +class Gateway: + """A running out-of-process simulator plus the spy over what it received. + + Yielded by the ``gateway`` pytest fixture. ``.url`` is a real + ``http://127.0.0.1:`` any process can point ``DONKEY_LLM_PROXY_URL`` at; + the simulator replays the same captured fixtures the SDK's typed refusals are + tested against, and every response carries ``x-donkey-simulator: true`` (#189). + """ + + def __init__( + self, *, url: str, recorder: _RecordingApp, shutdown: Callable[[], None] + ) -> None: + self._url = url + self._recorder = recorder + self._shutdown = shutdown + + @property + def url(self) -> str: + """The base URL of the running simulator, e.g. ``http://127.0.0.1:54123``.""" + return self._url + + @property + def requests_received(self) -> int: + """How many requests the gateway has received since start (or the last + :meth:`reset`).""" + return len(self._recorder.requests) + + @property + def requests(self) -> list[RecordedRequest]: + """A snapshot of the recorded requests, oldest first (secret redacted).""" + return list(self._recorder.requests) + + def set_scenarios(self, *scenarios: str | Scenario) -> None: + """Reconfigure the simulator's #188 fault-injection scenarios for this + test, live (no restart). Each argument is either a parsed + :class:`~donkey_kit.simulator.scenarios.Scenario` or a ``--scenario`` spec + string (e.g. ``"pii_block:every=1"``). Scenarios are stateful and + single-use, so this builds a fresh simulator each call; the request log is + left untouched (use :meth:`reset` to clear it).""" + parsed = tuple( + parse_scenario(s) if isinstance(s, str) else s for s in scenarios + ) + self._recorder.app = build_app(SimulatorConfig(scenarios=parsed)) + + def reset(self) -> None: + """Clear the recorded-request log (the running server is left alone).""" + self._recorder.requests.clear() + + def close(self) -> None: + """Stop the server and join its thread. Idempotent; the fixture calls it + on teardown, including when the test failed.""" + self._shutdown() + + +def _bound_port(server: Any) -> int: + """Read the ephemeral port uvicorn actually bound. Valid only once + ``server.started`` is true (that is when ``server.servers`` is populated), + which is what makes ``port=0`` race-free — we never guess a port.""" + for srv in getattr(server, "servers", None) or []: + for sock in getattr(srv, "sockets", None) or []: + return int(sock.getsockname()[1]) + raise RuntimeError("could not determine the local gateway simulator's bound port") + + +def start_gateway( + config: SimulatorConfig | None = None, + *, + host: str = "127.0.0.1", + startup_timeout: float = _STARTUP_TIMEOUT_S, +) -> Gateway: + """Boot the local gateway simulator on an ephemeral port and return a + :class:`Gateway` handle. The caller owns teardown via :meth:`Gateway.close`. + + Raises :class:`ImportError` (naming ``pip install "donkey-kit[local]"``) when + the ``[local]`` extra — ``starlette`` + ``uvicorn`` — is not installed. + """ + try: + import uvicorn # lazy: provided by the [local] extra + except ImportError as exc: + raise ImportError(_LOCAL_EXTRA_HINT) from exc + + try: + # build_app imports starlette lazily; a missing [local] surfaces here too. + recorder = _RecordingApp(build_app(config)) + except ImportError as exc: + raise ImportError(_LOCAL_EXTRA_HINT) from exc + + uv_config = uvicorn.Config(recorder, host=host, port=0, log_level="warning") + server = uvicorn.Server(uv_config) + thread = threading.Thread(target=server.run, daemon=True) + thread.start() + + deadline = time.time() + startup_timeout + while not server.started and time.time() < deadline: + time.sleep(0.02) + if not server.started: + server.should_exit = True + thread.join(timeout=5.0) + raise RuntimeError( + f"local gateway simulator did not start within {startup_timeout}s" + ) + + port = _bound_port(server) + + def _shutdown() -> None: + server.should_exit = True + thread.join(timeout=10.0) + + return Gateway(url=f"http://{host}:{port}", recorder=recorder, shutdown=_shutdown) diff --git a/python/src/donkey_kit/conformance/harness.py b/python/src/donkey_kit/conformance/harness.py index c0fb8b29..a70b2101 100644 --- a/python/src/donkey_kit/conformance/harness.py +++ b/python/src/donkey_kit/conformance/harness.py @@ -15,7 +15,7 @@ rather than deduping retries, because counting re-sends is exactly how the retry scenario is observed. -Framework isolation (§1.1): imports only ``httpx``, the framework-free +Framework isolation (the layered architecture): imports only ``httpx``, the framework-free ``core``/``donkey`` surface, and the sibling simulator loader — never a web framework and never ``openai`` at module top. It is a dev-only simulator-style sibling, kept out of the five production layers by an import-linter contract. diff --git a/python/src/donkey_kit/conformance/plugin.py b/python/src/donkey_kit/conformance/plugin.py index 71f5cb0e..db6df945 100644 --- a/python/src/donkey_kit/conformance/plugin.py +++ b/python/src/donkey_kit/conformance/plugin.py @@ -109,7 +109,7 @@ def _import_target(spec: str, *, what: str) -> tuple[Any, Any]: def _resolve_agent(config: pytest.Config) -> tuple[AgentFactory, dict[str, str]]: """Resolve the ``--agent`` factory and its known-limitations mapping, and validate the exemptions **now** (collection time) so a bad key or an empty - reason fails loudly before any scenario runs (§8.1: asserted, never silent).""" + reason fails loudly before any scenario runs (the conformance kit: asserted, never silent).""" agent_spec = config.getoption("donkey_agent") if not agent_spec: raise pytest.UsageError( @@ -151,7 +151,7 @@ def runtest(self) -> None: raise _ConformanceFailure(f"{self._scenario.title}: {result.detail}") # pass and exempt both leave the item green; the asserted exemption is # validated at collection and shown in the summary table, so it is - # recorded, not silently skipped (§8.1). + # recorded, not silently skipped (the conformance kit). if result.status == "exempt": self.add_report_section("call", "exempt", result.detail) @@ -242,3 +242,33 @@ def donkey() -> Any: yield fab finally: fab.close() + + +@pytest.fixture +def gateway() -> Any: + """A running local gateway simulator on an ephemeral port, for tests whose + subject does **not** import ``donkey_kit`` — a containerised agent, a Node + service, an A2A client, a manual ``curl`` (#278):: + + async def test_agent_stops_after_refusal(gateway): + gateway.set_scenarios("pii_block:every=1") + app = deploy(env={"DONKEY_LLM_PROXY_URL": gateway.url}) + await app.run(ticket) + assert gateway.requests_received == 1 # stopped, did not retry + + It serves the same captured rejection fixtures ``simulate()`` and ``donkey + mock`` replay, on a real port bound to ``0`` (so parallel ``pytest -n`` runs + never collide), and records every request in a spy the test can assert on + (:attr:`~.gateway.Gateway.requests_received` plus per-request method, path and + headers with ``client_secret`` redacted). Torn down when the test exits, + including on failure. + + Needs the ``[local]`` extra (``starlette`` + ``uvicorn``); taking the fixture + without it raises an :class:`ImportError` naming the exact ``pip install``.""" + from .gateway import start_gateway + + gw = start_gateway() + try: + yield gw + finally: + gw.close() diff --git a/python/src/donkey_kit/conformance/suite.py b/python/src/donkey_kit/conformance/suite.py index 04259ca9..84019907 100644 --- a/python/src/donkey_kit/conformance/suite.py +++ b/python/src/donkey_kit/conformance/suite.py @@ -20,7 +20,7 @@ - **works without budget headers** — serving a success with no ``x-token-*`` and asserting the run still completes. -Framework isolation (§1.1): stdlib only at module top, plus the framework-free +Framework isolation (the layered architecture): stdlib only at module top, plus the framework-free ``core.errors`` taxonomy. The :class:`ScenarioContext` protocol the checks call is defined *here*, so ``suite.py`` never imports the harness — the harness imports the suite, not the other way round. No web framework and no ``openai`` @@ -228,7 +228,7 @@ async def _works_without_budget_headers(ctx: ScenarioContext) -> Outcome: def validate_known_limitations(mapping: object) -> dict[str, str]: """Validate an agent's ``KNOWN_LIMITATIONS`` and return it as a plain dict. - An exemption is an *asserted* claim, never a silent skip (§8.1): every key + An exemption is an *asserted* claim, never a silent skip (the conformance kit): every key must name a real scenario and every reason must be a non-empty string. A bad key or an empty reason raises — the plugin lets that surface at collection time so the run fails loudly rather than quietly excusing a scenario. diff --git a/python/src/donkey_kit/core/__init__.py b/python/src/donkey_kit/core/__init__.py index 98eadfdd..c11e2fe2 100644 --- a/python/src/donkey_kit/core/__init__.py +++ b/python/src/donkey_kit/core/__init__.py @@ -1,6 +1,6 @@ -"""core/ — the framework-free foundation (§2). +"""core/ — the framework-free foundation. -HARD RULE (§1.1): nothing in this package may import from +HARD RULE (the layered architecture): nothing in this package may import from ``donkey_kit.integrations``. Enforced by import-linter in CI. """ diff --git a/python/src/donkey_kit/core/_verify.py b/python/src/donkey_kit/core/_verify.py index e75f046e..eec21c2d 100644 --- a/python/src/donkey_kit/core/_verify.py +++ b/python/src/donkey_kit/core/_verify.py @@ -1,5 +1,5 @@ -"""Centralized home for every value that §0.3 says must be verified against a -real Anypoint sandbox before it can be trusted. +"""Centralized home for every value that the verification discipline says must +be verified against a real Anypoint sandbox before it can be trusted. Working instruction #2: *never invent an endpoint, header name, or class name.* @@ -62,13 +62,13 @@ def blocked(what: str) -> NotImplementedError: """Construct the standard verification-blocked error. Use for surfaces where we have no defensible placeholder at all (e.g. the - MCP Bridge provisioning endpoint, §0.3 / §5). + MCP Bridge provisioning endpoint, verification discipline / provisioning-as-code). """ return NotImplementedError(f"blocked on verification: {what}") -# --- Attribution headers (§3, the single most important unknown) ----------- +# --- Attribution headers (docs/verified-apis.md §3, the single most important unknown) --- # These names are GUESSES. The gateway may read entirely different header names. ATTRIBUTION_APP_HEADER = Unverified( key="attribution.application_header", @@ -81,10 +81,11 @@ def blocked(what: str) -> NotImplementedError: doc_ref="docs/verified-apis.md §3", ) -# --- Cost-attribution tag headers (§3, the HIGHEST-priority unknown, #196) --- +# --- Cost-attribution tag headers (docs/verified-apis.md §3, the HIGHEST-priority +# unknown, #196) --- # The fixed cost dimensions (team / project / env / enduser.id) are emitted as # request headers so the gateway can group spend per dimension. The gateway-side -# header NAMES are the single most important unverified value (docs §3): the +# header NAMES are the single most important unverified value (docs/verified-apis.md §3): the # direct-proxy path did NOT surface them, so these are loud, overridable # placeholders (config: ``cost_*_header``), and the value ALWAYS lands on the # ``donkey.cost.*`` span regardless (the SDK controls the span end to end, #196 @@ -111,10 +112,10 @@ def blocked(what: str) -> NotImplementedError: doc_ref="docs/verified-apis.md §3", ) -# --- Correlation / call-id request headers (§2.3, #195) --------------------- +# --- Correlation / call-id request headers (BG §1.1, #195) ------------------ # The gateway ECHOES `x-correlation-id` on RESPONSES (VERIFIED LIVE 2026-08-28, -# docs §3). Whether it READS an INBOUND correlation header — and under what name -# — is UNVERIFIED, as is any per-call request-id header. Both request-header +# docs/verified-apis.md §3). Whether it READS an INBOUND correlation header — +# and under what name — is UNVERIFIED, as is any per-call request-id header. Both request-header # names are therefore placeholders, overridable per-Donkey via config # (``DonkeyConfig.correlation_header`` / ``.call_id_header``) so a customer can # point them at the real names without waiting for us. ``X-Correlation-Id`` is @@ -130,9 +131,9 @@ def blocked(what: str) -> NotImplementedError: doc_ref="docs/verified-apis.md §3", ) -# --- Control-plane token endpoint (§1) -------------------------------------- -# Path is appended to the region base URL. VERIFIED (§12.1) from static analysis -# of the shipping `mulesoft-anypoint-cli-agent-fabric-plugin` (+ `anypoint-cli- +# --- Control-plane token endpoint (docs/verified-apis.md §1) ---------------- +# Path is appended to the region base URL. VERIFIED (docs/verified-apis.md §12.1) from +# static analysis of the shipping `mulesoft-anypoint-cli-agent-fabric-plugin` (+ `anypoint-cli- # command`): OAuth2 client_credentials → `POST /accounts/api/v2/oauth2/token`. OAUTH_TOKEN_PATH = Unverified( key="anypoint.oauth_token_path", @@ -141,7 +142,8 @@ def blocked(what: str) -> NotImplementedError: verified=True, ) -# --- LLM proxy consumer auth (§2/§3) — VERIFIED (LIVE 2026-08-28) ------------ +# --- LLM proxy consumer auth (docs/verified-apis.md §2/§3) — VERIFIED (LIVE +# 2026-08-28) ------------ # The directly-called ingress LLM proxy authenticates the caller with a # `client_id` + `client_secret` REQUEST-header pair (client-id-enforcement # 1.3.3), NOT a bearer token. This pair IS the per-agent attribution unit. These @@ -149,7 +151,7 @@ def blocked(what: str) -> NotImplementedError: LLM_PROXY_CLIENT_ID_HEADER = "client_id" LLM_PROXY_CLIENT_SECRET_HEADER = "client_secret" -# --- Region host map (§1) ---------------------------------------------------- +# --- Region host map (docs/verified-apis.md §1) ------------------------------ # UNVERIFIED — Hyperforce region hosts in particular need confirmation. REGION_HOSTS: dict[str, str] = { "us": "https://anypoint.mulesoft.com", diff --git a/python/src/donkey_kit/core/auth.py b/python/src/donkey_kit/core/auth.py index b50f56ac..cb34a824 100644 --- a/python/src/donkey_kit/core/auth.py +++ b/python/src/donkey_kit/core/auth.py @@ -1,4 +1,4 @@ -"""Auth (§2.2). +"""Auth (BG §1.1). ``AuthProvider`` is a tiny protocol so a customer can plug in their own vault. We ship three implementations: :class:`AnypointConnectedApp` (OAuth2 @@ -6,9 +6,9 @@ :class:`ChainedAuth`. The control-plane credential and the LLM-proxy credential are SEPARATE and must -not be conflated (§2.2). +not be conflated (BG §1.1). -VERIFICATION NOTE (§0.3): the token endpoint PATH and the scopes each operation +VERIFICATION NOTE: the token endpoint PATH and the scopes each operation needs are UNVERIFIED. The path is a loud, overridable placeholder from ``_verify.OAUTH_TOKEN_PATH``; the per-operation scope table must be recorded in ``docs/`` once confirmed. Some operations require an *admin* connected app with @@ -57,7 +57,7 @@ class AnypointConnectedApp(AuthProvider): Caches the token in memory with a 60s safety margin before expiry. On a 401 from any downstream call, ``invalidate()`` then retry exactly once (the - retry is performed by the transport layer, §2.3). + retry is performed by the transport layer, BG §1.1). """ def __init__( @@ -119,7 +119,7 @@ async def _fetch(self) -> str: if not token: raise AuthError( "Token endpoint returned no access_token. Response shape is " - "UNVERIFIED — capture it as a fixture (§8.2).", + "UNVERIFIED — capture it as a fixture (BG §1.5).", response=resp, ) expires_in = float(body.get("expires_in", 3600)) diff --git a/python/src/donkey_kit/core/budget.py b/python/src/donkey_kit/core/budget.py index 48bb50dd..194eaa18 100644 --- a/python/src/donkey_kit/core/budget.py +++ b/python/src/donkey_kit/core/budget.py @@ -1,5 +1,5 @@ """Budget — the token-budget window, parsed from the proxy's budget headers -(§1.3, BG §1.3, piece 2 of the six-piece minimum). +(BG §1.3, piece 2 of the six-piece minimum). The developer never parses a header:: @@ -63,7 +63,7 @@ # Matches `… 10000 tokens remaining of 10000 limit. Reset in 56711ms.`. Requires # all three values: a partial or reworded sentence fails to match and is treated -# as no signal (§0.3 — never guess at an unexpected wire shape). `search`, not +# as no signal (verification discipline — never guess at an unexpected wire shape). `search`, not # `match`, so the leading `Token rate limit:` label is not load-bearing. _RATELIMIT_RE = re.compile( r"(?P\d+)\s+tokens?\s+remaining\s+of\s+(?P\d+)\s+limit\b" @@ -78,7 +78,7 @@ def _utcnow() -> datetime: def _parse_int(raw: str | None) -> int | None: """A budget header as an int, or ``None`` if absent or non-numeric. Garbage is - ignored rather than fatal (§0.3: never let an unexpected wire value crash the + ignored rather than fatal (verification discipline: never let an unexpected wire value crash the caller's request path).""" if raw is None: return None @@ -92,7 +92,7 @@ def _parse_ratelimit_prose(raw: str | None) -> tuple[int | None, int | None, int """The prose ``x-llm-proxy-ratelimit`` header as ``(limit, remaining, reset_ms)``, or an all-``None`` tuple when the header is absent, partial, or reworded. - Like :func:`_parse_int` this never raises on unexpected input (§0.3): an + Like :func:`_parse_int` this never raises on unexpected input (verification discipline): an unparseable sentence is simply "no signal", never a crash on the caller's request path. All three values must be present in the recognised shape or the whole header is discarded — a half-parsed budget is worse than none.""" @@ -180,7 +180,7 @@ def observe(self, response: httpx.Response, *, now: datetime | None = None) -> N @asynccontextmanager async def pace(self, *, reserve: float = 0.0) -> AsyncIterator[None]: """Guard a request so it is refused *before* it crosses your reserve, not - after a 429 comes back (§1.3, #186). + after a 429 comes back (BG §1.3, #186). ``reserve`` is the fraction of the window to keep in hand (``0.0``-``1.0``): ``reserve=0.10`` trips at 90% used, ``reserve=0.0`` (the default) only at @@ -215,7 +215,7 @@ async def pace(self, *, reserve: float = 0.0) -> AsyncIterator[None]: async def wait_for_reset(self, *, now: datetime | None = None) -> None: """Sleep until :attr:`reset_at`, then return — the recovery half of pacing - (§1.3, #186). + (BG §1.3, #186). A single sleep, never a spin loop. If the window is unobserved (:attr:`reset_at` is ``None``) or already past, this returns immediately — diff --git a/python/src/donkey_kit/core/cache.py b/python/src/donkey_kit/core/cache.py index 00925da6..549b781b 100644 --- a/python/src/donkey_kit/core/cache.py +++ b/python/src/donkey_kit/core/cache.py @@ -1,7 +1,8 @@ -"""In-memory TTL cache for tokens and registry lookups (§2, cache.py). +"""In-memory TTL cache for tokens and registry lookups (the framework-free +core, cache.py). Deliberately tiny and dependency-free. Honours the ``DONKEY_NO_CACHE=1`` escape -hatch (§4.2). Not thread-safe across processes — it is a per-process cache for a +hatch (BG §2.7). Not thread-safe across processes — it is a per-process cache for a single agent run, which is all the SDK needs. Time is injected (``clock``) so tests do not sleep and so the module stays pure. diff --git a/python/src/donkey_kit/core/config.py b/python/src/donkey_kit/core/config.py index f18ffb5c..58457f05 100644 --- a/python/src/donkey_kit/core/config.py +++ b/python/src/donkey_kit/core/config.py @@ -1,4 +1,4 @@ -"""Configuration (§2.1). +"""Configuration. Resolution order: explicit kwarg → env var → ``.donkey-kit.toml`` (cwd or ``$XDG_CONFIG_HOME``) → default. We never read ``.env`` implicitly — the user @@ -6,7 +6,7 @@ ``validated()`` reports ALL missing fields in one error, not one per run — the one-missing-variable-per-run loop is the most common first-five-minutes -abandonment (§2.1). +abandonment (config resolution). """ from __future__ import annotations @@ -29,8 +29,9 @@ Region = Literal["us", "eu", "ca", "jp"] # What to do when the gateway serves a different model than the one requested — -# a routing fallback substituted the model (§3, #309). ``"off"`` (default): the -# substitution is surfaced passively on ``donkey.last_call.substituted`` and the +# a routing fallback substituted the model (docs/verified-apis.md §3, #309). +# ``"off"`` (default): the substitution is surfaced passively on +# ``donkey.last_call.substituted`` and the # span, but the call succeeds. ``"raise"``: the transport raises # :class:`~donkey_kit.core.errors.ModelSubstituted`, for callers who need model # determinism (e.g. an evaluation whose results are only comparable per-model). @@ -64,17 +65,17 @@ class DonkeyConfig: application_name: str | None = None # env: DONKEY_APP_NAME business_group: str | None = None # env: DONKEY_BUSINESS_GROUP - # --- Correlation request-header NAME overrides (§2.3, #195) --- + # --- Correlation request-header NAME overrides (BG §1.1, #195) --- # The gateway's inbound correlation/call-id header names are UNVERIFIED - # (docs §3); these let a customer point them at the real names without a + # (docs/verified-apis.md §3); these let a customer point them at the real names without a # release. Unset → the loud ``Unverified`` placeholders in ``core/_verify``. correlation_header: str | None = None # env: DONKEY_CORRELATION_HEADER call_id_header: str | None = None # env: DONKEY_CALL_ID_HEADER - # --- Cost-attribution tags + request-header NAME overrides (§3, #196) --- + # --- Cost-attribution tags + request-header NAME overrides (docs/verified-apis.md §3, #196) --- # The fixed dimensions (team/project/env/enduser.id), set once and emitted on # every call. The gateway-side header names are the highest-priority unknown - # (docs §3); the ``cost_*_header`` overrides let a customer point them at the + # (docs/verified-apis.md §3); the ``cost_*_header`` overrides let a customer point them at the # real names — unset → the loud ``Unverified`` placeholders in ``core/_verify``. cost: CostTags = CostTags() # env: DONKEY_COST_{TEAM,PROJECT,ENV,ENDUSER_ID} cost_team_header: str | None = None # env: DONKEY_COST_TEAM_HEADER @@ -93,7 +94,7 @@ class DonkeyConfig: # very content the platform just masked to whatever OTLP collector is wired # up (#306, BG §1.6). Opting in is the developer assuming that obligation. telemetry_capture_content: bool = False - # What to do when the gateway serves a different model than requested (§3, + # What to do when the gateway serves a different model than requested (docs/verified-apis.md §3, # #309). Default "off" — the substitution is surfaced passively on # ``donkey.last_call``; "raise" opts into a hard ``ModelSubstituted`` error. on_model_substitution: OnModelSubstitution = "off" @@ -180,7 +181,7 @@ def validated(self, *, need: str = "control_plane") -> DonkeyConfig: :class:`ConfigError` listing EVERY missing field at once. ``need`` is one of ``"control_plane"`` (registry/provisioning) or - ``"llm"`` (proxy). The two credentials are independent (§2.2): a user + ``"llm"`` (proxy). The two credentials are independent (BG §1.1): a user may legitimately have proxy access and no Exchange access. """ @@ -229,7 +230,7 @@ def _opt(v: object) -> str | None: def _resolve_cost_tags(toml_cost: object) -> CostTags: """Resolve the cost tags along the fixed precedence: ``[donkey.cost]`` toml table is the base (its keys validated against the fixed set — an unknown - dimension raises :class:`ConfigError`, never a silent drop, §196 AC #1), then + dimension raises :class:`ConfigError`, never a silent drop, #196 AC #1), then ``DONKEY_COST_*`` env vars override per dimension. Absent both → empty tags.""" if toml_cost is None: base = CostTags() @@ -258,8 +259,9 @@ def _as_bool(v: object) -> bool: def _as_substitution(v: object) -> OnModelSubstitution: - """Coerce and validate ``on_model_substitution`` (§3, #309). An unknown value - is a config mistake worth reporting up front — a silent fall-back to ``"off"`` + """Coerce and validate ``on_model_substitution`` (docs/verified-apis.md §3, #309). + An unknown value is a config mistake worth reporting up front — a silent + fall-back to ``"off"`` would leave a caller who typed ``"error"`` believing they had opted into strictness. Validated at resolve time, like ``region``.""" token = str(v).strip().lower() diff --git a/python/src/donkey_kit/core/cost.py b/python/src/donkey_kit/core/cost.py index 20350345..9369b198 100644 --- a/python/src/donkey_kit/core/cost.py +++ b/python/src/donkey_kit/core/cost.py @@ -1,11 +1,11 @@ -"""Cost-attribution tags (§3, BG §1.7, #196). +"""Cost-attribution tags (docs/verified-apis.md §3, BG §1.7, #196). A small, FIXED set of dimensions — ``team``, ``project``, ``env``, ``enduser.id`` — set once on the :class:`~donkey_kit.core.config.DonkeyConfig` (and optionally overridden per run via ``donkey.run(...)``), then emitted on every governed model call two ways: - * as request **headers**, under UNVERIFIED placeholder names (§0.3 — the + * as request **headers**, under UNVERIFIED placeholder names (verification discipline — the gateway-side cost-attribution header name is the single highest-priority unknown, ``docs/verified-apis.md`` §3); and * as ``donkey.cost.*`` **span attributes**, which carry the full value @@ -66,7 +66,7 @@ def _validate(field: str, value: str) -> None: @dataclass(frozen=True) class CostTags: - """The fixed four-dimension cost-attribution set (§3, #196). + """The fixed four-dimension cost-attribution set (docs/verified-apis.md §3, #196). Every field is optional and defaults to ``None`` (unset — omitted from both headers and span attributes). Construction validates every set value, so an diff --git a/python/src/donkey_kit/core/errors.py b/python/src/donkey_kit/core/errors.py index 5a7a59bd..917beb7b 100644 --- a/python/src/donkey_kit/core/errors.py +++ b/python/src/donkey_kit/core/errors.py @@ -1,4 +1,4 @@ -"""Exception taxonomy (§2.4). +"""Exception taxonomy (BG §1.2). Mapping gateway policy rejections to catchable, actionable exceptions is the SDK's clearest value over raw HTTP. @@ -11,12 +11,12 @@ 2. ``remediation`` is a structurally-guaranteed, human-readable next step — worth more than a stack trace. Every :class:`PolicyViolation` carries one: the constructor refuses to build an instance whose remediation is empty or - whitespace, and every concrete subclass ships a canonical default (§2.4, + whitespace, and every concrete subclass ships a canonical default (BG §1.2, #182). That default is the single source ``donkey doctor`` (#202) reuses, so a diagnosis and the exception it stands for can never disagree. The concrete HTTP-response → exception mapping lives in :func:`classify`, which -is driven by a table that MUST be populated from real captured fixtures (§8.2), +is driven by a table that MUST be populated from real captured fixtures (BG §1.5), not hand-written guesses. Until fixtures exist, :func:`classify` maps only the status-code families it can defensibly infer and otherwise returns a generic :class:`DonkeyError`. @@ -39,7 +39,7 @@ class DonkeyError(Exception): """Base for all SDK errors. Carries correlation/call/request IDs and the raw response so callers can inspect what actually happened. - Three ids, three provenances (§2.3, #195): + Three ids, three provenances (BG §1.1, #195): * ``correlation_id`` — the RUN id the client sent (``X-Correlation-Id``), shared by every request in a ``donkey.run()`` block. This is the @@ -70,7 +70,7 @@ def __init__( class ConfigError(DonkeyError): - """Configuration is missing or invalid. Reports ALL problems at once (§2.1).""" + """Configuration is missing or invalid. Reports ALL problems at once (config resolution).""" class AuthError(DonkeyError): @@ -87,7 +87,7 @@ class AuthError(DonkeyError): "DONKEY_LLM_PROXY_CLIENT_ID / DONKEY_LLM_PROXY_CLIENT_SECRET are the " "consumer client_id/secret pair for this LLM-proxy instance — not an " "Anypoint control-plane credential and not a bearer token — and that the " - "consumer is authorized on the instance in API Manager (§2)." + "consumer is authorized on the instance in API Manager (docs/verified-apis.md §2)." ) @@ -135,7 +135,7 @@ def __init__( if not resolved.strip(): raise ValueError( f"{type(self).__name__} requires a non-empty remediation naming the " - "caller's next step (§2.4, #182)." + "caller's next step (BG §1.2, #182)." ) self.remediation = resolved if policy is not None: @@ -157,7 +157,7 @@ def __init__(self, message: str, *, retry_after: float | None = None, **kw: Any) class BudgetReserveReached(DonkeyError): """Raised by :meth:`Budget.pace` *before* it lets a request through that would - cross the caller's reserve (§1.3, #186). + cross the caller's reserve (BG §1.3, #186). Deliberately NOT a :class:`PolicyViolation`: that base marks a gateway-enforced refusal that is terminal and never retried. This is the client-side opposite — @@ -189,7 +189,7 @@ def __init__( class ModelSubstituted(DonkeyError): """The gateway served a *different* model than the one requested — a routing fallback substituted the model, and ``on_model_substitution="raise"`` opted - the caller into treating that as an error (§3, #309). + the caller into treating that as an error (docs/verified-apis.md §3, #309). Deliberately NOT a :class:`PolicyViolation`: the request was neither refused nor failed — it succeeded, just against a model the developer did not choose. @@ -390,11 +390,11 @@ class ProvisioningError(DonkeyError): class GovernanceDrift(DonkeyError): - """resolve(): a declared policy is not actually applied on the gateway (§6.3).""" + """resolve(): a declared policy is not actually applied on the gateway.""" class PublicationDrift(DonkeyError): - """verify(): the live server no longer matches the Exchange descriptor (§7.4).""" + """verify(): the live server no longer matches the Exchange descriptor (BG §2.5).""" def classify( @@ -406,15 +406,15 @@ def classify( """Map an HTTP error response to a specific exception. ``correlation_id`` (the run id) and ``call_id`` (the per-request id) are the - two ids the client sent (§2.3, #195). When not passed explicitly they are + two ids the client sent (BG §1.1, #195). When not passed explicitly they are read back from the response's own request headers, so a caller bridging an ``openai.APIStatusError`` — ``classify(err.response)`` — gets a :class:`DonkeyError` whose ``correlation_id`` equals the header that was sent with no extra wiring. An explicit argument (e.g. when the header name was overridden via config) always wins over the auto-derived value. - The precise policy discrimination (§2.4, working instruction #4) is driven by - real rejection captures from a live governed LLM proxy (docs §4, + The precise policy discrimination (BG §1.2, working instruction #4) is driven by + real rejection captures from a live governed LLM proxy (docs/verified-apis.md §4, ``tests/fixtures/anypoint/llm_proxy/reject.*``), NOT hand-written guesses. What the captures established: @@ -443,13 +443,13 @@ def classify( checked before the auth rule. These two shapes are **documented, not yet live-captured** (same posture as - #181's header-based injection typing; docs §4). Exact strings are pinned from + #181's header-based injection typing; docs/verified-apis.md §4). Exact strings are pinned from the policy pages; no ``_verify.py`` row flips to ``verified=True`` until a live sandbox round-trip confirms them (#253). Any other content-moderation / federated-guardrail shape still falls through to a generic :class:`PolicyViolation` whose message names what was observed and says the shape is unconfirmed (#184). Because auth is the verified 401 / ``www-authenticate`` - shape (docs §4), a **403 carrying no ``www-authenticate`` header** and matching + shape (docs/verified-apis.md §4), a **403 carrying no ``www-authenticate`` header** and matching none of the policy discriminators above is treated as one of these unconfirmed refusals, not mis-typed as an :class:`AuthError`. """ @@ -459,7 +459,7 @@ def classify( status = response.status_code kw: dict[str, Any] = { # Explicit arg wins (e.g. a config-overridden header name); else the id - # the client actually sent, read back from the request (§2.3, #195). + # the client actually sent, read back from the request (BG §1.1, #195). "correlation_id": correlation_id if correlation_id is not None else sent_correlation, "call_id": call_id if call_id is not None else sent_call, "request_id": request_id, @@ -469,7 +469,7 @@ def classify( # A nested ``{"error": {...}}`` object is emitted by BOTH the upstream # provider AND some gateway LLM policies (e.g. PII). The ``type`` field — # not the status code or the mere presence of a nested object — is the - # authoritative discriminator (docs §4). ``body`` is the top-level JSON + # authoritative discriminator (docs/verified-apis.md §4). ``body`` is the top-level JSON # object (used also to spot the Regex-Prompt-Guard ``matched_patterns`` key); # ``error_obj`` is its nested ``error`` object iff it is itself an object. body = _json_body(response) @@ -490,7 +490,7 @@ def classify( ) # Content-safety / guardrails policy: 403 + a vendor `...-action: reject` - # header (Azure Content Safety / Amazon Bedrock Guardrails, docs §4). + # header (Azure Content Safety / Amazon Bedrock Guardrails, docs/verified-apis.md §4). # Documented, pending live capture (#253). Checked before the 401/403 → auth # rule because a moderation block is not an auth failure. Keyed on the # header, not the body, so the body-less Bedrock reject is caught too. @@ -510,7 +510,7 @@ def classify( ) # Regex Prompt Guard policy: 403 + a top-level `matched_patterns` list - # (flat-string `error`, so NOT the nested upstream envelope; docs §4). + # (flat-string `error`, so NOT the nested upstream envelope; docs/verified-apis.md §4). # Documented, pending live capture (#253). Checked before the 401/403 → auth # rule so a deny-list block is not mis-typed as an auth failure. matched = body.get("matched_patterns") if body is not None else None @@ -529,7 +529,7 @@ def classify( ) # Injection-protection policy: discriminated by the ``x-injection-protection: - # blocked`` header, NOT the status code (#181, docs §4). Checked before the + # blocked`` header, NOT the status code (#181, docs/verified-apis.md §4). Checked before the # generic 4xx / nested-error branch so an injection block wins even if its # body happens to be shaped like an upstream error envelope. A 400 WITHOUT # this header is an ordinary refusal, never PromptInjectionBlocked (AC (a)). @@ -540,7 +540,7 @@ def classify( **kw, ) - # Auth is discriminated by the verified client-id-enforcement shape (docs §4): + # Auth is discriminated by the verified client-id-enforcement shape (docs/verified-apis.md §4): # a 401, or a 403 carrying a ``www-authenticate`` challenge. A 403 WITHOUT that # header matched none of the policy discriminators above, so it is an # unrecognised gateway policy refusal — not an auth failure — and falls through @@ -616,17 +616,17 @@ def classify( def _sent_ids(response: httpx.Response) -> tuple[str | None, str | None]: """The ``(correlation_id, call_id)`` the client sent, read back from the response's own request headers under the names the transport actually used - (§2.3, #195, #363). + (BG §1.1, #195, #363). The transport stamps the resolved header names onto ``request.extensions`` at injection time (``donkey_correlation_header`` / ``donkey_call_id_header``), so a per-``Donkey`` header-name override (``DonkeyConfig.correlation_header`` - / ``.call_id_header``, docs §3) is honoured here without ``core/errors`` + / ``.call_id_header``, docs/verified-apis.md §3) is honoured here without ``core/errors`` importing ``DonkeyConfig`` — it only reads a plain dict carried on the same object ``response.request`` returns. When the stamp is absent (a response not produced by our transport — e.g. a hand-built stock-client response), the placeholder names are used directly, **not** ``Unverified.get()``, so reading - an id back never emits the §0.3 unverified warning; that warning belongs at + an id back never emits the verification discipline's unverified warning; that warning belongs at injection time, in the transport. Returns ``(None, None)`` when the request is unavailable (httpx raises if it @@ -650,13 +650,13 @@ def _sent_ids(response: httpx.Response) -> tuple[str | None, str | None]: def _retry_after(response: httpx.Response) -> float | None: """Seconds until the caller may retry. Prefers the standard ``retry-after`` (delta-seconds) header; falls back to the LLM token-rate-limit policy's - ``x-token-reset`` header, which is captured in **milliseconds** (docs §4).""" + ``x-token-reset`` header, which is captured in **milliseconds** (docs/verified-apis.md §4).""" raw = response.headers.get("retry-after") if raw is not None: try: return float(raw) except ValueError: - pass # HTTP-date form; left for the fixture-driven parser (§8.2) + pass # HTTP-date form; left for the fixture-driven parser (BG §1.5) reset_ms = response.headers.get("x-token-reset") if reset_ms is not None: try: @@ -672,7 +672,7 @@ def _retry_after(response: httpx.Response) -> float | None: def _pii_entities(message: str | None) -> list[str]: """Best-effort extraction of the flagged PII entity types from the PII policy's rejection message (a JSON-ish list of ``{"pii_type": "...", ...}`` - objects; docs §4). Returns an empty list if none can be parsed. + objects; docs/verified-apis.md §4). Returns an empty list if none can be parsed. Deliberately best-effort (#289): the LLM PII Detection policy documents only the ``{"error":{"message","type":"pii_detected"}}`` envelope, not a structured @@ -687,7 +687,8 @@ def _pii_entities(message: str | None) -> list[str]: def _json_body(response: httpx.Response) -> dict[str, Any] | None: """The response's top-level JSON object, or ``None`` when the body is absent, - not JSON, or not an object. Never raises on the caller's request path (§0.3).""" + not JSON, or not an object. Never raises on the caller's request path + (verification discipline).""" try: body = response.json() except (ValueError, UnicodeDecodeError): @@ -697,7 +698,7 @@ def _json_body(response: httpx.Response) -> dict[str, Any] | None: # Content-safety / guardrails policies report their verdict in a pair of vendor # headers — an ``...-action`` (``allow``|``reject``) and a comma-separated -# ``...-reason``. Both are ``x-llm-proxy--...`` (docs §4). Pinned from the +# ``...-reason``. Both are ``x-llm-proxy--...`` (docs/verified-apis.md §4). Pinned from the # Azure Content Safety and Amazon Bedrock Guardrails policy pages; documented, # pending live capture (#253). _CONTENT_SAFETY_VENDORS: tuple[tuple[str, str, str], ...] = ( diff --git a/python/src/donkey_kit/core/lastcall.py b/python/src/donkey_kit/core/lastcall.py index f7406cf3..ab95d9e3 100644 --- a/python/src/donkey_kit/core/lastcall.py +++ b/python/src/donkey_kit/core/lastcall.py @@ -1,5 +1,5 @@ """LastCall — the gateway's own metadata about the most recent governed model -call, read from the response (§2.3, #362, BG §1.1). +call, read from the response (#362, BG §1.1). On a refusal, :class:`~donkey_kit.core.errors.DonkeyError` already hands the developer the gateway's ids (``request_id`` and the two client-sent @@ -9,7 +9,7 @@ "which gateway instance served this, and what id do I quote in a ticket?" was answerable after a failure and unanswerable after a success. This record closes that asymmetry — one named container, populated on every governed model call -from the LIVE-VERIFIED §3 response headers, reachable at ``donkey.last_call``. +from the LIVE-VERIFIED docs/verified-apis.md §3 response headers, reachable at ``donkey.last_call``. **One container, defined once (#362).** #307 (cached/reasoning token counts) and #309 (gateway routing + fallback) each sketched a *different* accessor for the @@ -40,7 +40,7 @@ :class:`LastCallStatus`: ``OBSERVED`` (a governed response populated it), ``UNOBSERVED`` (no governed model call has returned in this context yet), and ``UNAVAILABLE`` (this surface structurally cannot be observed). The conformance -suite asserts the exemption rather than skipping it (§8.1). +suite asserts the exemption rather than skipping it. **The gateway correlation id is deliberately absent (hazard #1).** The response ``x-correlation-id`` is the gateway's own value; per #300 it is NOT confirmed to @@ -50,11 +50,11 @@ bake that ambiguity into the public API, so the field is not added until #300 lands and settles the semantics under an unambiguous name. -Every field traces to the LIVE-VERIFIED §3 "Gateway identity on response" row +Every field traces to the LIVE-VERIFIED docs/verified-apis.md §3 "Gateway identity on response" row (``responses.success.headers.txt``, 2026-08-28), so this is a consumption task, not a verification one: no ``_verify.Unverified(...)`` guard, no ``UnverifiedValueWarning``. Unparseable or absent headers leave a field ``None`` -and never raise on the caller's request path (§0.3). +and never raise on the caller's request path (verification discipline). """ from __future__ import annotations @@ -84,7 +84,7 @@ # and the routing strategy. All four are consumed here on the success path # (#309) — a substitution the developer did not choose is otherwise invisible. # ``LLM_PROVIDER_HEADER`` is also the SOLE source of ``gen_ai.system`` on the -# span, so ``core/transport.py`` imports it from here (one definition, §0.3). +# span, so ``core/transport.py`` imports it from here (one definition, verification discipline). ROUTING_TYPE_HEADER = "x-llm-proxy-routing-type" ROUTING_FALLBACK_HEADER = "x-llm-proxy-routing-fallback" LLM_PROVIDER_HEADER = "x-llm-proxy-llm-provider" @@ -93,7 +93,7 @@ # ``api-instance-21133858.3e6ce455-e3e8-4402-b830-9fcf07d9207b.svc`` → instance # ``21133858`` + environment ``3e6ce455-…`` (a UUID; it carries dashes but no # dots, so a plain three-way split on ``.`` is unambiguous). An unrecognised -# shape matches nothing and yields ``(None, None)`` — never a guess (§0.3). +# shape matches nothing and yields ``(None, None)`` — never a guess (verification discipline). _DECORATOR_RE = re.compile(r"^api-instance-(?P[^.]+)\.(?P[^.]+)\.svc$") @@ -119,7 +119,7 @@ class LastCallStatus(str, Enum): def _parse_decorator_operation(raw: str | None) -> tuple[str | None, str | None]: """``(api_instance_id, environment_id)`` from ``x-envoy-decorator-operation``, or ``(None, None)`` when the header is absent or not in the verified - ``api-instance-..svc`` shape. Never raises (§0.3).""" + ``api-instance-..svc`` shape. Never raises (verification discipline).""" if raw is None: return None, None m = _DECORATOR_RE.match(raw.strip()) @@ -134,7 +134,7 @@ def _parse_fallback(raw: str | None) -> bool | None: unrecognised value. VERIFIED values are the literal strings ``"true"`` / ``"false"``; anything else is treated as unknown (``None``) rather than guessed, so a bare ``None`` never masquerades as a definitive ``False`` - (§0.3).""" + (verification discipline).""" if raw is None: return None token = raw.strip().lower() @@ -193,7 +193,7 @@ def is_fallback(response: httpx.Response) -> bool: def _first_int(mapping: dict[str, object], *keys: str) -> int | None: """The first key present as an ``int`` (``bool`` excluded — it subclasses - ``int``), else ``None``. Absent/garbage is ``None``, never ``0`` (§0.3).""" + ``int``), else ``None``. Absent/garbage is ``None``, never ``0`` (verification discipline).""" for key in keys: value = mapping.get(key) if isinstance(value, int) and not isinstance(value, bool): @@ -221,7 +221,7 @@ def parse_usage(usage: object) -> dict[str, int | None]: ``prompt_tokens_details`` / ``completion_tokens_details``) shapes. A non-dict ``usage`` (absent, ``None``, wrong type) yields all-``None`` — an absent count is ``None``, never ``0`` (the same honesty rule ``Budget`` applies to an - unobserved window). Never raises (§0.3).""" + unobserved window). Never raises (verification discipline).""" if not isinstance(usage, dict): return { "input_tokens": None, @@ -261,7 +261,7 @@ def usage_mapping(obj: object) -> dict[str, object] | None: def usage_from_response(response: httpx.Response) -> dict[str, int | None]: """The six usage counts from a buffered 2xx JSON body's ``usage`` object, or all-``None`` when the response is non-2xx, streaming/unread, non-JSON, or - carries no ``usage``. Never raises on the caller's request path (§0.3) — a + carries no ``usage``. Never raises on the caller's request path (verification discipline) — a streaming body's ``.json()`` raises ``ResponseNotRead``, which is caught and treated as "no usage here" (it arrives later via :func:`observe_usage`).""" if response.status_code // 100 != 2: @@ -277,7 +277,8 @@ def usage_from_response(response: httpx.Response) -> dict[str, int | None]: @dataclass(frozen=True) class LastCall: - """What the gateway said about the most recent governed model call (§3, #362). + """What the gateway said about the most recent governed model call + (docs/verified-apis.md §3, #362). Reached at ``donkey.last_call``. Immutable: each governed response replaces the context's record wholesale rather than mutating in place, so a reader @@ -298,7 +299,7 @@ class LastCall: observed_at: datetime | None = None #: For UNAVAILABLE, the adapter surface(s) that cannot observe; else ``None``. surface: str | None = None - # --- gateway routing & fallback (§3, #309) ----------------------------- + # --- gateway routing & fallback (docs/verified-apis.md §3, #309) ------- #: The model the caller ASKED for (from the request body). The reference #: point for :attr:`substituted`; ``None`` when the request carried none. requested_model: str | None = None @@ -363,7 +364,7 @@ def from_response( requested_model: str | None = None, now: datetime | None = None, ) -> LastCall: - """Build an ``OBSERVED`` record from a governed response's §3 headers. + """Build an ``OBSERVED`` record from a governed response's docs/verified-apis.md §3 headers. Always ``OBSERVED`` — the SDK saw a response — even if the gateway carried no identity headers (each field then stays ``None``): "we saw the @@ -429,7 +430,7 @@ def observe_last_call( now: datetime | None = None, ) -> LastCall: """Record the gateway identity, routing and fallback from a governed model - response into this context (§2.3, #362/#309). Called from the transport's + response into this context (BG §1.1, #362/#309). Called from the transport's ``_on_response`` beside :meth:`Budget.observe`, which passes the requested model it already parsed from the request body. Returns the record it set, for tests. ``now`` is injectable; production uses the wall clock (UTC).""" diff --git a/python/src/donkey_kit/core/telemetry.py b/python/src/donkey_kit/core/telemetry.py index 62366f2a..81840d3b 100644 --- a/python/src/donkey_kit/core/telemetry.py +++ b/python/src/donkey_kit/core/telemetry.py @@ -1,4 +1,4 @@ -"""Telemetry (§2.5, BG §1.6) and the run-scoped correlation ID (§2.3). +"""Telemetry (BG §1.6) and the run-scoped correlation ID (BG §1.1). OpenTelemetry is an optional dependency (the ``[otel]`` extra). Telemetry is **on by default** (``DonkeyConfig.telemetry``), but the export pipeline is @@ -13,7 +13,7 @@ The correlation ID lives in a ``contextvar`` so a single agent run's fan-out of model calls and tool calls shares one trace ID end to end — letting a developer correlate their local trace with what the platform team sees in Omni Gateway's -observability view (a headline feature, §2.5). +observability view (a headline feature, BG §1.6). """ from __future__ import annotations @@ -46,7 +46,7 @@ # which copy the current context — with no threading through framework state. _cost_tags: ContextVar[CostTags | None] = ContextVar("donkey_cost_tags", default=None) -# Span name constants (§2.5). +# Span name constants (BG §1.6). SPAN_LLM_CHAT = "donkey.llm.chat" SPAN_REGISTRY_RESOLVE = "donkey.registry.resolve" SPAN_TOOL_CALL = "donkey.tool.call" @@ -103,15 +103,17 @@ DONKEY_POLICY_DECISION = "donkey.policy.decision" DONKEY_POLICY_TYPE = "donkey.policy.type" DONKEY_BUDGET_REMAINING = "donkey.budget.remaining" -# Cost-attribution dimensions (§3, BG §1.7, #196). One donkey.cost.* attribute -# per fixed dimension; ``enduser.id`` keeps its dotted external name. These -# carry the full value on the span even while the request-header names are -# UNVERIFIED (docs §3), so per-dimension spend attribution works end to end. +# Cost-attribution dimensions (docs/verified-apis.md §3, BG §1.7, #196). One +# donkey.cost.* attribute per fixed dimension; ``enduser.id`` keeps its dotted +# external name. These carry the full value on the span even while the +# request-header names are UNVERIFIED (docs/verified-apis.md §3), so +# per-dimension spend attribution works end to end. DONKEY_COST_TEAM = "donkey.cost.team" DONKEY_COST_PROJECT = "donkey.cost.project" DONKEY_COST_ENV = "donkey.cost.env" DONKEY_COST_ENDUSER = "donkey.cost.enduser.id" -# Gateway routing & resilience (§3, #309). The served provider already lands on +# Gateway routing & resilience (docs/verified-apis.md §3, #309). The served +# provider already lands on # ``gen_ai.system`` and the served model on ``gen_ai.response.model``; these two # carry the gateway-specific routing facts the semconv has no key for. Emitted # even when ``fallback`` is ``False`` — "we routed normally" is a signal an @@ -171,7 +173,7 @@ def new_correlation_id() -> str: def new_call_id() -> str: - """A fresh per-request **call id** (§2.3, #195). + """A fresh per-request **call id** (BG §1.1, #195). Unlike the run/correlation id — which is contextvar-bound and shared across every request in a :func:`run_context` / ``donkey.run()`` block — this is @@ -211,7 +213,7 @@ def run_context(run_id: str | None = None) -> Iterator[str]: class RunScope: """A **dual sync/async** context manager that binds the run correlation id - to :data:`_correlation_id` for the block (§2.3, #195). + to :data:`_correlation_id` for the block (BG §1.1, #195). This is what ``donkey.run(id=...)`` returns, so the same object works under both ``with donkey.run(...)`` and ``async with donkey.run(...)`` — binding a @@ -274,7 +276,7 @@ async def __aexit__(self, *exc: Any) -> None: def run_scope(run_id: str | None = None, cost: CostTags | None = None) -> RunScope: """Build a :class:`RunScope` — the dual sync/async run correlation binding - behind ``donkey.run(id=...)`` (§2.3, #195), optionally carrying per-run + behind ``donkey.run(id=...)`` (BG §1.1, #195), optionally carrying per-run cost-tag overrides (#196).""" return RunScope(run_id, cost) diff --git a/python/src/donkey_kit/core/toolspec.py b/python/src/donkey_kit/core/toolspec.py index 6262b221..fa0c8cf6 100644 --- a/python/src/donkey_kit/core/toolspec.py +++ b/python/src/donkey_kit/core/toolspec.py @@ -5,11 +5,11 @@ :data:`TOOL_MARKER` (``__donkey_tool__``) to it, and appends a :class:`ToolSpec` to a process-global registry (:func:`registered_tools`). -That one marker is the single thing the Phase 2 scanner (§2.5) and the A2A -agent-card generator (§2.9) both read — so the annotation pays off three times +That one marker is the single thing the Phase 2 scanner (BG §2.5) and the A2A +agent-card generator (BG §2.9) both read — so the annotation pays off three times (#200). Neither consumer exists yet; this introduces the marker they will read. -Framework-free (core, §1.1): stdlib ``inspect`` + ``dataclasses`` only — no +Framework-free (core, the layered architecture): stdlib ``inspect`` + ``dataclasses`` only — no httpx, no pydantic, no agent framework. """ diff --git a/python/src/donkey_kit/core/transport.py b/python/src/donkey_kit/core/transport.py index 619e1e3d..1e82477a 100644 --- a/python/src/donkey_kit/core/transport.py +++ b/python/src/donkey_kit/core/transport.py @@ -1,4 +1,4 @@ -"""Transport — the single place headers get injected (§2.3). +"""Transport — the single place headers get injected (BG §1.1). This is the most important piece of engineering in the SDK. Every framework has a different mechanism for setting request headers, and several have none. The @@ -7,27 +7,27 @@ The client: * injects, via a request event hook, on every outbound request: - the run correlation ID (uuid4 per logical agent run, from a contextvar, - §2.5) — shared by every request in a ``donkey.run()`` block, the + BG §1.6) — shared by every request in a ``donkey.run()`` block, the client↔gateway join key - attribution headers (application, business group) — header NAMES are UNVERIFIED (docs/verified-apis.md §3), emitted via loud placeholders - bearer token, refreshed lazily * pins a per-call ID ONCE, before the retry loop, so it is unique per logical - request yet stable across that request's retries and 401 refresh (§2.3, + request yet stable across that request's retries and 401 refresh (BG §1.1, #195). Two ids, two headers: the run id (``X-Correlation-Id``) groups a run; the call id (``X-Donkey-Request-Id``) pinpoints one request within it. Both - header NAMES are UNVERIFIED placeholders (docs §3), overridable via config. + header NAMES are UNVERIFIED placeholders (docs/verified-apis.md §3), overridable via config. * retries transient upstream/gateway failures (502/503/504) with exponential backoff + jitter, honouring Retry-After - * does NOT retry 4xx — gateway policy rejections are terminal (§2.4). This + * does NOT retry 4xx — gateway policy rejections are terminal (BG §1.2). This includes 429: on this proxy a 429 is a token-budget refusal (TokenBudgetExceeded), and retrying it only burns the same exhausted window - (§2.4, #183). retry_after is still surfaced for wait_for_reset() (#186). - * refreshes the token and retries exactly once on 401 (§2.2) + (BG §1.2, #183). retry_after is still surfaced for wait_for_reset() (#186). + * refreshes the token and retries exactly once on 401 (BG §1.1) For frameworks that only accept a ``default_headers`` dict (not a client), pass :func:`attribution_headers` — a snapshot — and accept that the correlation ID is -per-client rather than per-run. Document that degradation per adapter (§3.3). +per-client rather than per-run. Document that degradation per adapter (BG §1.8). """ from __future__ import annotations @@ -72,7 +72,7 @@ start_genai_span, ) -# Default request-header NAMES for the two correlation ids (§2.3, #195). Both are +# Default request-header NAMES for the two correlation ids (BG §1.1, #195). Both are # UNVERIFIED placeholders: the gateway ECHOES ``x-correlation-id`` on RESPONSES # (verified), but whether it READS an inbound correlation/call-id header — and # under what name — is not. A customer overrides them per-Donkey via config @@ -85,20 +85,20 @@ CALL_ID_HEADER = _verify.CALL_ID_HEADER.placeholder # The OpenAI-compatible SDKs reject an empty ``api_key``. The governed proxy # authenticates on the client_id/client_secret headers (client-id-enforcement, -# §2/§3) and ignores the bearer, so we fill the slot with a harmless sentinel +# docs/verified-apis.md §2/§3) and ignores the bearer, so we fill the slot with a harmless sentinel # whenever no explicit key is configured. PROXY_API_KEY_SENTINEL = "client-id-enforced" # 429 is deliberately NOT here: on this proxy every 429 is a token-budget # refusal that classify() maps to TokenBudgetExceeded (a PolicyViolation), and a # PolicyViolation is terminal — retrying it only burns the same exhausted budget -# window (§2.4, #183). Only genuinely transient upstream/gateway failures retry. +# window (BG §1.2, #183). Only genuinely transient upstream/gateway failures retry. _RETRYABLE_STATUS = frozenset({502, 503, 504}) _BACKOFF_BASE_S = 0.5 _BACKOFF_CAP_S = 30.0 # The four cost dimensions → the config field that overrides that header name → -# the UNVERIFIED placeholder used when there is no override (§3, #196). The +# the UNVERIFIED placeholder used when there is no override (docs/verified-apis.md §3, #196). The # gateway-side names are the highest-priority unknown, so each is a loud, # overridable placeholder resolved here. _COST_HEADER_SOURCES: tuple[tuple[str, str, _verify.Unverified], ...] = ( @@ -110,12 +110,14 @@ def cost_headers(cfg: DonkeyConfig, tags: CostTags) -> dict[str, str]: - """The request headers for the SET dimensions of ``tags`` (§3, BG §1.7, #196). + """The request headers for the SET dimensions of ``tags`` + (docs/verified-apis.md §3, BG §1.7, #196). Each header NAME is the config override (``cost_*_header``) if set, else the UNVERIFIED placeholder from ``core/_verify`` — the gateway-side cost header - name is the single highest-priority unknown (docs §3), so an un-overridden - name emits the one-time §0.3 warning. Values are pre-validated by + name is the single highest-priority unknown (docs/verified-apis.md §3), so + an un-overridden name emits the one-time verification-discipline warning. + Values are pre-validated by :class:`CostTags`, so they are always header-safe.""" override = { field: getattr(cfg, attr) for field, attr, _placeholder in _COST_HEADER_SOURCES @@ -141,16 +143,18 @@ def attribution_headers(cfg: DonkeyConfig) -> dict[str, str]: ``default_headers`` dict. Does NOT include the correlation ID (which must be per-run) or the bearer token (which must be refreshed lazily). - Header NAMES are UNVERIFIED (§0.3 / §3): the live direct-proxy path did NOT - surface application/business-group as request headers (docs §3), so these + Header NAMES are UNVERIFIED (verification discipline / docs/verified-apis.md §3): + the live direct-proxy path did NOT + surface application/business-group as request headers (docs/verified-apis.md §3), so these remain loud, overridable placeholders. The verified per-agent attribution unit is the ``client_id`` credential — see :func:`proxy_auth_headers`. - Includes the CONFIG-LEVEL cost tags (§3, #196). A static ``default_headers`` + Includes the CONFIG-LEVEL cost tags (docs/verified-apis.md §3, #196). A static + ``default_headers`` snapshot cannot see a later ``donkey.run(...)`` override — that binding is a contextvar the live client reads per send — so the snapshot path carries the set-once tags only, a documented degradation (like the per-run correlation - ID, §3.3).""" + ID, BG §1.8).""" headers: dict[str, str] = {} if cfg.application_name: @@ -162,7 +166,7 @@ def attribution_headers(cfg: DonkeyConfig) -> dict[str, str]: def proxy_auth_headers(cfg: DonkeyConfig) -> dict[str, str]: - """The LLM-proxy consumer-auth request headers, LIVE-VERIFIED (docs §2/§3): + """The LLM-proxy consumer-auth request headers, LIVE-VERIFIED (docs/verified-apis.md §2/§3): a ``client_id`` + ``client_secret`` pair enforced by ``client-id-enforcement``. This pair IS the per-agent attribution identity, NOT a bearer token. @@ -188,7 +192,7 @@ def proxy_api_key(cfg: DonkeyConfig) -> str: def _resolve_header_names(cfg: DonkeyConfig) -> tuple[str, str]: - """The ``(correlation, call_id)`` request-header NAMES for this config (§2.3, + """The ``(correlation, call_id)`` request-header NAMES for this config (BG §1.1, #195): each is the config override if set, else the UNVERIFIED placeholder from ``core/_verify``. Called ONCE per client at construction, so the one-time unverified warning for an un-overridden name fires there, not on @@ -232,7 +236,7 @@ def _apply_base_headers( def _apply_call_id_header(request: httpx.Request, call_id_header: str) -> None: """Pin a FRESH per-call ID on the request, ONCE, before the retry loop - (§2.3, #195). + (BG §1.1, #195). The run/correlation id is deterministic per run (a contextvar), so the per-send event hook can safely re-set it on every retry. The call id is @@ -297,7 +301,7 @@ def _gateway_unavailable( # two clients. The response header carrying the resolved upstream provider is # VERIFIED (LIVE) — docs/verified-apis.md §2 "Model routing" and §3 "Gateway # identity on response" — and is the SOLE source of gen_ai.system; absent → the -# attribute is omitted, never guessed (§0.3), because the proxy routes to +# attribute is omitted, never guessed (verification discipline), because the proxy routes to # several providers and defaulting one would misattribute the call. The header # NAME is defined once in ``lastcall`` (which also parses it onto ``last_call``) # and imported here as ``LLM_PROVIDER_HEADER`` so there is a single source (#309). @@ -359,13 +363,14 @@ def _record_response( wire — the sync and async transports source that id differently, and reading the header makes the recorded value correct for both. The per-call id is intentionally not a span attribute: the span already correlates one call, and - the run id is the cross-call join key (§2.3, #195).""" + the run id is the cross-call join key (BG §1.1, #195).""" try: decision, policy_type = _span_decision(response) usage = usage_from_response(response) gspan.record( system=response.headers.get(LLM_PROVIDER_HEADER), - # Served model + routing facts (§3, #309): request≠response model on + # Served model + routing facts (docs/verified-apis.md §3, #309): + # request≠response model on # the span is the fastest read that a gateway failover happened, and # the fallback flag is emitted even when False. response_model=response.headers.get(LLM_MODEL_HEADER), @@ -399,13 +404,14 @@ def _substitution_error( ) -> ModelSubstituted | None: """The :class:`ModelSubstituted` to raise for this response, or ``None``. - Off unless ``on_model_substitution="raise"`` (§3, #309): the default surfaces + Off unless ``on_model_substitution="raise"`` (docs/verified-apis.md §3, #309): + the default surfaces a substitution passively on ``donkey.last_call.substituted`` and the span. Only a **2xx** is checked — a refusal or upstream error is classified on its own terms elsewhere and is not a "silent substitution". A substitution requires both the requested model (from the body) and the served model (the gateway header) to be known and to differ; a missing either side is never a - guess (§0.3). Never raises here — it returns the error for the caller path to + guess (verification discipline). Never raises here — it returns the error for the caller path to raise once telemetry has been recorded.""" if cfg.on_model_substitution != "raise": return None @@ -583,13 +589,14 @@ def __init__( ) -> None: self._cfg = cfg # The two correlation request-header NAMES, resolved once (config override - # → UNVERIFIED placeholder). The one-time §0.3 warning for an un-overridden - # name fires here, at construction, not per request (§2.3, #195). + # → UNVERIFIED placeholder). The one-time verification-discipline warning + # for an un-overridden + # name fires here, at construction, not per request (BG §1.1, #195). self._correlation_header, self._call_id_header = _resolve_header_names(cfg) # NB: httpx.AsyncClient uses ``self._auth`` internally, so we must NOT # store our token provider there — super().__init__() would clobber it. self._token_provider = auth - # Optional in-band budget collaborator (§1.3, #185). When attached, the + # Optional in-band budget collaborator (BG §1.3, #185). When attached, the # response hook feeds it; when None the hook stays a byte-identical no-op, # so the control-plane token-fetch client tracks no budget. self._budget = budget @@ -609,7 +616,8 @@ async def _inject_headers(self, request: httpx.Request) -> None: if self._token_provider is not None: token = await self._token_provider.token() # Control plane uses OAuth2 client_credentials → ``Authorization: - # Bearer`` (VERIFIED §12.1). The LLM proxy (data plane) instead uses + # Bearer`` (VERIFIED docs/verified-apis.md §12.1). The LLM proxy (data + # plane) instead uses # client_id/client_secret headers and gets NO token provider, so it # never reaches here; ``setdefault`` also yields to the OpenAI SDK's # own Authorization if one was set at the call site. @@ -712,7 +720,7 @@ async def _send_with_retries( 2xx gets its span-closing stream wrapper while a buffered response keeps the context-manager lifecycle.""" # Pin the per-call id ONCE, before the loop, so it is stable across - # retries and the 401 refresh (§2.3, #195). The run correlation id is set + # retries and the 401 refresh (BG §1.1, #195). The run correlation id is set # per-send by the event hook (deterministic, so idempotent). _apply_call_id_header(request, self._call_id_header) await self._on_request(request) @@ -749,7 +757,7 @@ async def _send_with_retries( await response.aclose() await provider.invalidate() # A 401 refresh is an auth re-send, not a rate-limit backoff, so it - # does NOT consume the retry budget (§2.2: "retry exactly once on + # does NOT consume the retry budget (BG §1.1: "retry exactly once on # 401"): re-send once with the fresh token regardless of `attempt`, # so the retry still happens on the final attempt / max_retries=0. # Event hooks re-run on send() → fresh token. @@ -759,7 +767,8 @@ async def _send_with_retries( # retried, even on a retryable status: the gateway's Enhanced # Resilience routing is the first recovery layer, and an SDK retry on # top multiplies latency against an outage the gateway is already - # handling (§3, #309, #183). ``is_fallback`` is definitive-True-only, + # handling (docs/verified-apis.md §3, #309, #183). ``is_fallback`` is + # definitive-True-only, # so a non-proxy 5xx with no routing header still retries as before. if ( response.status_code in _RETRYABLE_STATUS @@ -822,7 +831,8 @@ async def _finish( ) # After telemetry (so the substituted call is still on the span), a # caller who opted into ``on_model_substitution="raise"`` gets a hard - # error instead of the response (§3, #309). Raising here propagates out + # error instead of the response (docs/verified-apis.md §3, #309). Raising + # here propagates out # through the span context manager (buffered) or the detached-span guard # in ``send()`` (streaming), so the span still closes. The response is # closed first so an aborted stream leaks no connection. @@ -852,17 +862,18 @@ class DonkeyClient(httpx.Client): It takes **no** :class:`AuthProvider`: that protocol is async-only (``async def token()``), and there is no correct way to await it from here. That costs nothing on the LLM data plane, which authenticates with the - ``client_id``/``client_secret`` header pair (LIVE-VERIFIED §2/§3) rather than + ``client_id``/``client_secret`` header pair (LIVE-VERIFIED + docs/verified-apis.md §2/§3) rather than a fetched token. It does mean the control-plane surfaces — ``registry`` and - ``tools`` — stay async-only; see §2.2 for why the two credentials are + ``tools`` — stay async-only; see BG §1.1 for why the two credentials are deliberately not conflated. """ def __init__(self, cfg: DonkeyConfig, *, budget: Budget | None = None, **kw: object) -> None: self._cfg = cfg - # Resolved once; see DonkeyAsyncClient.__init__ (§2.3, #195). + # Resolved once; see DonkeyAsyncClient.__init__ (BG §1.1, #195). self._correlation_header, self._call_id_header = _resolve_header_names(cfg) - self._budget = budget # see DonkeyAsyncClient.__init__ (§1.3, #185) + self._budget = budget # see DonkeyAsyncClient.__init__ (BG §1.3, #185) super().__init__( timeout=cfg.timeout_s, event_hooks={"request": [self._inject_headers]}, @@ -961,7 +972,7 @@ def _send_with_retries( # No 401-refresh branch: with no token provider there is nothing to # refresh, so a 401 here is a real credential failure and terminal. # A routing-fallback response is not double-retried — see the async - # twin (§3, #309, #183). + # twin (docs/verified-apis.md §3, #309, #183). if ( response.status_code in _RETRYABLE_STATUS and not is_fallback(response) @@ -1000,7 +1011,7 @@ def _finish( cost_tags=effective_cost_tags(self._cfg), ) # After telemetry, raise for an opted-in substitution — see the async - # twin (§3, #309). + # twin (docs/verified-apis.md §3, #309). substitution = _substitution_error(self._cfg, request, response) if substitution is not None: response.close() @@ -1021,14 +1032,14 @@ def build_http_client( *, budget: Budget | None = None, ) -> DonkeyAsyncClient: - """Factory for the shared client (§2.3). Pass ``budget`` to track the in-band - token window on every response (§1.3, #185); omit it for the control-plane + """Factory for the shared client (BG §1.1). Pass ``budget`` to track the in-band + token window on every response (BG §1.3, #185); omit it for the control-plane token-fetch client, which observes no budget.""" return DonkeyAsyncClient(cfg, auth, budget=budget) def build_sync_http_client(cfg: DonkeyConfig, *, budget: Budget | None = None) -> DonkeyClient: - """Factory for the shared blocking client (§2.3). See :class:`DonkeyClient` + """Factory for the shared blocking client (BG §1.1). See :class:`DonkeyClient` for why it takes no :class:`AuthProvider`. Pass ``budget`` to share one budget - object with the async client (§1.3, #185).""" + object with the async client (BG §1.3, #185).""" return DonkeyClient(cfg, budget=budget) diff --git a/python/src/donkey_kit/donkey.py b/python/src/donkey_kit/donkey.py index 1fdbb612..3f663cb4 100644 --- a/python/src/donkey_kit/donkey.py +++ b/python/src/donkey_kit/donkey.py @@ -1,4 +1,4 @@ -"""The ``Donkey`` public surface (§3.2). +"""The ``Donkey`` public surface (`BG §1.1`). from donkey_kit import Donkey donkey = Donkey.from_env() @@ -8,7 +8,7 @@ Per-framework adapters are lazy attributes. Accessing one whose extra is not installed raises :class:`ImportError` with the exact install command — never a -bare ``ModuleNotFoundError`` (§3.2). +bare ``ModuleNotFoundError`` (`BG §1.1`). """ from __future__ import annotations @@ -69,7 +69,7 @@ def _framework_installed(probe: str) -> bool: class _ToolsFacade: - """``donkey.tools`` — discovery + lock (§4.1, §4.2).""" + """``donkey.tools`` — discovery + lock (BG §2.7).""" def __init__(self, registry: ExchangeRegistry) -> None: self._registry = registry @@ -86,22 +86,24 @@ async def discover( """Discover a governed tool catalog and return a bindable ``ToolSet``. ``governed`` defaults to ``None`` (no filtering) in v1, with a startup - log line stating discovery is unfiltered (§6.1.2). Flipping the default + log line stating discovery is unfiltered (governed-only discovery). Flipping the default to ``True`` is a breaking change reserved for a later major version. Blocked until Exchange search + the governed-state join are verified - (§0.3 / §6.7). + (verification discipline / the Verification milestone). """ raise _verify.blocked( - "Exchange search + governed-state join (§6.1, §6.7). ToolSet filtering " - "and binding scaffolding exist; wire discover() once the APIs are " - "confirmed and use registry.explain() for the empty-result reason." + "Exchange search + governed-state join (governed-only discovery, the Verification " + "milestone). ToolSet filtering and binding scaffolding exist; wire discover() once the " + "APIs are confirmed and use registry.explain() for the empty-result reason." ) def lock(self) -> None: - """Write a ``donkey.lock`` of resolved versions + digests (§4.2).""" - raise _verify.blocked("resolution API needed for the lockfile (§4.2, §6.7).") + """Write a ``donkey.lock`` of resolved versions + digests (BG §2.7).""" + raise _verify.blocked( + "resolution API needed for the lockfile (BG §2.7, the Verification milestone)." + ) class Donkey: @@ -134,7 +136,7 @@ def __init__( # delegates here — and it is idempotent across many Donkey() instances. configure_otlp_export(self._cfg) self._auth = auth if auth is not None else self._default_auth(self._cfg) - # One Budget per Donkey (never global, §1.3 / #185): both transports feed + # One Budget per Donkey (never global, BG §1.3 / #185): both transports feed # it in-band from every response's x-token-* headers. self._budget = Budget() self._http: DonkeyAsyncClient = build_http_client( @@ -159,7 +161,7 @@ def from_env( on_model_substitution: OnModelSubstitution | None = None, ) -> Donkey: """Build from the environment (`BG §1.1`), optionally setting the fixed - cost-attribution tags once for every call (§3, BG §1.7, #196):: + cost-attribution tags once for every call (BG §1.7, #196):: donkey = Donkey.from_env(team="support", project="triage-v2", env="prod") @@ -170,7 +172,7 @@ def from_env( :class:`~donkey_kit.core.cost.CostTags`. Per-call overrides layer on via ``donkey.run(...)``. - ``on_model_substitution`` opts into model determinism (§3, #309): pass + ``on_model_substitution`` opts into model determinism (BG §1.7, #309): pass ``"raise"`` to have a call raise :class:`~donkey_kit.core.errors.ModelSubstituted` when the gateway serves a different model than requested (a routing fallback). Defaults to the @@ -198,7 +200,7 @@ def llm(self) -> LLMClient: @property def budget(self) -> Budget: """The token-budget window for this Donkey, updated in-band from every - response's ``x-token-*`` headers (§1.3, #185). Unobserved (all fields + response's ``x-token-*`` headers (BG §1.3, #185). Unobserved (all fields ``None``) until the first call returns; there is no budget-query endpoint, so it is only as fresh as ``budget.observed_at`` (upstream gap #2).""" return self._budget @@ -207,10 +209,10 @@ def budget(self) -> Budget: def last_call(self) -> LastCall: """The gateway's own metadata about the most recent governed model call in this context — its ``request_id``, ``api_instance_id`` and - ``environment_id`` (§3, #362); what the gateway *did* with the request — + ``environment_id`` (BG §1.7, #362); what the gateway *did* with the request — ``served_provider`` / ``served_model`` / ``routing_type`` and the ``fallback`` flag, with ``substituted`` true when the served model differs - from ``requested_model`` (§3, #309); and the per-call usage token counts + from ``requested_model`` (BG §1.7, #309); and the per-call usage token counts (``input_tokens`` / ``output_tokens`` / ``total_tokens`` and the cost-relevant ``cached_tokens`` / ``cache_write_tokens`` / ``reasoning_tokens``, #307). The success-path counterpart to the ids @@ -237,7 +239,7 @@ def last_call(self) -> LastCall: LlamaIndex / MS Agent Framework), so a response can never reach the record. :attr:`LastCall.surface` names which. This is derived from the adapters actually resolved, and the conformance suite asserts the - exemption rather than skipping it (§8.1). + exemption rather than skipping it (the conformance kit). """ observed = current_last_call() if observed is not None: @@ -267,7 +269,7 @@ def openai(self, *, sync: bool = False, **kw: Any) -> AsyncOpenAI | OpenAI: OpenAI Agents SDK. It delegates to :meth:`LLMClient.client`, returning a real ``openai.AsyncOpenAI`` (or ``OpenAI`` with ``sync=True``), governed on identical terms. The Agents SDK adapter now lives at - ``donkey.openai_agents`` (§3.3). + ``donkey.openai_agents`` (BG §1.8). """ if sync: return self._llm.client(sync=True, **kw) @@ -290,7 +292,7 @@ def run( env: str | None = None, enduser_id: str | None = None, ) -> RunScope: - """Group one logical agent run under a shared correlation id (§2.3, #195). + """Group one logical agent run under a shared correlation id (BG §1.7, #195). The headline ergonomic — bind a run id once, and every governed model call inside the block carries it, with no threading through framework @@ -316,7 +318,7 @@ def run( Works with or without OpenTelemetry installed — correlation is pure contextvar + headers; spans only decorate when OTel is present. - Cost-attribution overrides (§3, BG §1.7, #196) bind for the block on top + Cost-attribution overrides (BG §1.7, #196) bind for the block on top of the tags set once on the Donkey: ``donkey.run(team=..., project=..., env=..., enduser_id=...)`` wins per field for every call inside, and the rest fall back to the configured tags. Like the run id, the binding rides @@ -326,7 +328,7 @@ def run( return run_scope(id, override if not override.is_empty else None) def run_context(self, run_id: str | None = None) -> RunScope: - """Back-compat alias for :meth:`run` (§2.3). Prefer ``donkey.run(id=…)``.""" + """Back-compat alias for :meth:`run` (BG §1.7). Prefer ``donkey.run(id=…)``.""" return self.run(run_id) # --- one-line on-ramps: decorators (#200) ------------------------------ @@ -344,7 +346,7 @@ def governed( The one-line on-ramp to governed execution: every governed model call made inside the decorated function carries a fresh run/correlation id (a "run of one"), the optional per-run cost tags, the OTel span and typed - refusals — exactly the scope :meth:`run` establishes (§2.3, #195), with + refusals — exactly the scope :meth:`run` establishes (BG §1.7, #195), with nothing to thread through framework state:: @donkey.governed(team="support") @@ -405,8 +407,8 @@ def tool(func: _Callable) -> _Callable: @donkey.tool async def lookup_crm(customer_id: str) -> dict: ... - The same marker is what the Phase 2 scanner (§2.5) and the A2A agent-card - generator (§2.9) both read, so the one annotation pays off three times + The same marker is what the Phase 2 scanner (BG §2.5) and the A2A agent-card + generator (BG §2.9) both read, so the one annotation pays off three times (#200). Raises :class:`ValueError` when the callable has no docstring — an @@ -455,7 +457,7 @@ def simulate( def _sync_http_client(self) -> DonkeyClient: if self._sync_http is None: # Same Budget object as the async client, so a blocking caller updates - # donkey.budget on identical terms (§1.3, #185). + # donkey.budget on identical terms (BG §1.3, #185). self._sync_http = build_sync_http_client(self._cfg, budget=self._budget) return self._sync_http @@ -508,7 +510,7 @@ def __getattr__(self, name: str) -> Adapter: @staticmethod def _default_auth(cfg: DonkeyConfig) -> AuthProvider | None: """Build control-plane auth when credentials are present. The LLM proxy - credential is separate and handled by the OpenAI client (§2.2).""" + credential is separate and handled by the OpenAI client (`BG §1.1`).""" if cfg.client_id and cfg.client_secret: return AnypointConnectedApp( client_id=cfg.client_id, diff --git a/python/src/donkey_kit/governance.py b/python/src/donkey_kit/governance.py index 92b4c550..beccdf1b 100644 --- a/python/src/donkey_kit/governance.py +++ b/python/src/donkey_kit/governance.py @@ -1,21 +1,21 @@ -"""The Governance object and its three-verb lifecycle (§6.2–§6.4). +"""The Governance object and its three-verb lifecycle. -ONE object, THREE verbs, THREE trust levels (§6.3): +ONE object, THREE verbs, THREE trust levels: * ``simulate()`` — local, developer laptop. Ephemeral local gateway; touches nothing shared. - * ``export()`` — compiles to a ``donkey.yaml`` fragment (§5.1). Writes a + * ``export()`` — compiles to a ``donkey.yaml`` fragment (provisioning-as-code). Writes a file; touches nothing. * ``resolve()`` — runtime, READ-ONLY. Looks up the already-provisioned route, verifies the declared policies are actually applied, returns the base_url. Raises :class:`GovernanceDrift` on mismatch. -There is deliberately NO ``apply()`` on the runtime object (§6.3): applying to +There is deliberately NO ``apply()`` on the runtime object: applying to sandbox/prod goes through ``donkey apply`` in CI, against a reviewed spec, under platform-controlled credentials. An escape hatch exists for platform teams (:meth:`apply`) with an explicit, hard-to-miss kwarg. -Local is NOT sandbox-with-a-different-URL (§6.4): feature sets differ, policies +Local is NOT sandbox-with-a-different-URL: feature sets differ, policies are not portable across modes, identity/secrets differ. ``simulate()`` must loudly report which declared policies are skipped locally and why. """ @@ -41,10 +41,10 @@ class PolicyPortability(Enum): - """Whether a policy works in Local Mode, Connected Mode, or both (§6.4). + """Whether a policy works in Local Mode, Connected Mode, or both. - The concrete classification per policy MUST come from real data captured in - M0 (§6.7), not inference — until then policies default to ``UNKNOWN``. + The concrete classification per policy MUST come from real data captured by + the Verification milestone, not inference — until then policies default to ``UNKNOWN``. """ BOTH = "both" @@ -58,7 +58,7 @@ class PolicyBinding: asset_id: str version: str config: dict[str, Any] = field(default_factory=dict) - #: Populated from the M0 portability table (§6.4/§6.7); UNKNOWN until then. + #: Populated from the portability table (the Verification milestone); UNKNOWN until then. portability: PolicyPortability = PolicyPortability.UNKNOWN @@ -73,7 +73,7 @@ class GatewayTarget: @classmethod def from_env(cls) -> GatewayTarget: """Select a profile from ``.donkey-kit.toml`` via ``DONKEY_TARGET`` - (``local|sandbox|production``) (§6.2).""" + (``local|sandbox|production``) (environment targeting).""" target = os.environ.get("DONKEY_TARGET", "local") profiles = _load_targets() @@ -101,68 +101,71 @@ class Governance: # ---- verb 1: simulate (local) ----------------------------------------- def simulate(self) -> SimulationContext: - """Start an ephemeral local Omni Gateway harness (§6.5). + """Start an ephemeral local Omni Gateway harness (BG §1.4). Requires the ``[local]`` extra (docker). Whether Local Mode can run the - LLM Proxy / MCP Bridge at all is an M0 gate (§6.7); if not, LLM traffic + LLM Proxy / MCP Bridge at all is a gate in the Verification milestone; if not, LLM traffic is served by a clearly-labelled local mock proxy. Either way, skipped - connected-only policies are reported loudly and non-suppressibly (§6.4). + connected-only policies are reported loudly and non-suppressibly. """ return SimulationContext(self) # ---- verb 2: export (sandbox/prod, laptop) ---------------------------- def export(self, path: str | Path | None = None) -> str: - """Compile to a ``donkey.yaml`` fragment (§5.1). Lands with M4 (§9.1).""" + """Compile to a ``donkey.yaml`` fragment (provisioning-as-code). Lands with M4 (the build + plan phases).""" raise _verify.blocked( "Governance.export() emits the M4 donkey.yaml spec format; it ships " - "with provisioning (M4, §9.1). Schema is shared with §5.1 — do not " - "diverge it (§5.1)." + "with provisioning (M4, the build plan phases). Schema is shared with " + "provisioning-as-code — do not diverge it (provisioning-as-code)." ) # ---- verb 3: resolve (runtime, READ-ONLY) ----------------------------- async def resolve(self, donkey: Any) -> GatewayTarget: """READ-ONLY: verify the declared policies are actually applied on the provisioned gateway and return its route. Raises - :class:`~donkey_kit.core.errors.GovernanceDrift` on mismatch (§6.3). + :class:`~donkey_kit.core.errors.GovernanceDrift` on mismatch. - Blocked until the API Manager read APIs are verified (§6.7).""" + Blocked until the API Manager read APIs are verified (the Verification milestone).""" raise _verify.blocked( "runtime governed-route lookup + applied-policy read for resolve() " - "(§6.3, §6.7)." + "(the Verification milestone)." ) # ---- escape hatch (platform teams only) ------------------------------- async def apply(self, target: GatewayTarget, *, i_am_the_platform_team: bool = False) -> None: - """Platform-team-only direct apply (§6.3). Requires write scopes the + """Platform-team-only direct apply. Requires write scopes the default connected app will not hold; every use is logged at WARNING.""" if not i_am_the_platform_team: raise PermissionError( "Governance.apply() inverts the platform-team ownership model " - "(§5.4). Runtime code should use resolve() (read-only). If you are " + "(provisioning-as-code). Runtime code should use resolve() (read-only). If you are " "the platform team automating your own gateway, pass " "i_am_the_platform_team=True and ensure the connected app holds " "write scopes." ) - raise _verify.blocked("direct policy apply endpoint (§5, §6.3, §6.7).") + raise _verify.blocked( + "direct policy apply endpoint (provisioning-as-code, the Verification milestone)." + ) class SimulationContext: - """Async context manager for ``gov.simulate()`` (§6.5).""" + """Async context manager for ``gov.simulate()`` (BG §1.4).""" def __init__(self, gov: Governance) -> None: self._gov = gov def skipped_policies(self) -> list[tuple[PolicyBinding, str]]: """Declared policies that will NOT be exercised locally, with reasons — - printed loudly before start and repeated at teardown (§6.4).""" + printed loudly before start and repeated at teardown.""" skipped: list[tuple[PolicyBinding, str]] = [] for p in self._gov.policies: if p.portability in (PolicyPortability.CONNECTED_ONLY, PolicyPortability.UNKNOWN): reason = ( "connected-mode only (needs API Manager client apps)" if p.portability is PolicyPortability.CONNECTED_ONLY - else "portability UNKNOWN pending M0 verification (§6.7)" + else "portability UNKNOWN pending Verification-milestone verification" ) skipped.append((p, reason)) return skipped @@ -170,7 +173,7 @@ def skipped_policies(self) -> list[tuple[PolicyBinding, str]]: async def __aenter__(self) -> Any: raise _verify.blocked( "local Omni Gateway docker harness + Local-Mode LLM-Proxy/MCP-Bridge " - "availability (§6.5, §6.7). The [local] extra and the loud " + "availability (BG §1.4, the Verification milestone). The [local] extra and the loud " "skipped-policy report (skipped_policies()) are scaffolded; the docker " "orchestration is M2.5 and gated on the M0 local-mode findings." ) diff --git a/python/src/donkey_kit/integrations/__init__.py b/python/src/donkey_kit/integrations/__init__.py index e43e1cb8..05e4c303 100644 --- a/python/src/donkey_kit/integrations/__init__.py +++ b/python/src/donkey_kit/integrations/__init__.py @@ -1,12 +1,12 @@ -"""integrations/ — one optional extra per framework (§1.1). +"""integrations/ — one optional extra per framework (the layered architecture). -Each adapter returns NATIVE framework objects (§3.1). Modules are imported +Each adapter returns NATIVE framework objects (BG §1.8). Modules are imported lazily by :class:`donkey_kit.donkey.Donkey` so an uninstalled framework never breaks ``import donkey_kit``. The registry below maps the attribute name used on ``Donkey`` to the adapter's module + class + pip extra, so ``Donkey.__getattr__`` can raise a curated -ImportError with the exact install command (§3.2). +ImportError with the exact install command (BG §1.8). """ from __future__ import annotations @@ -29,7 +29,7 @@ class AdapterSpec: conformance_tested: bool #: A representative top-level module of the framework, probed with #: ``importlib.util.find_spec`` so ``Donkey.__getattr__`` can raise the - #: curated ImportError at ACCESS time (§3.2). The adapters import their + #: curated ImportError at ACCESS time (BG §1.8). The adapters import their #: framework lazily inside methods, so importing the adapter module alone #: never fails — this probe is what makes access-time detection work. probe: str diff --git a/python/src/donkey_kit/integrations/_base.py b/python/src/donkey_kit/integrations/_base.py index 3ea0fb4c..b1b81b59 100644 --- a/python/src/donkey_kit/integrations/_base.py +++ b/python/src/donkey_kit/integrations/_base.py @@ -1,8 +1,8 @@ """Shared base for framework adapters. -Design rule (§3.1): adapters return NATIVE framework objects, never wrappers. +Design rule (BG §1.8): adapters return NATIVE framework objects, never wrappers. Each adapter depends on exactly one framework. Nothing here may be imported by -``core``/``llm``/``registry``/``tools`` (§1.1, enforced by import-linter). +``core``/``llm``/``registry``/``tools`` (the layered architecture, enforced by import-linter). """ from __future__ import annotations @@ -23,7 +23,7 @@ class Adapter: """Base holding the config and the shared HTTP client every adapter needs.""" #: pip extra that provides this adapter's framework, for the curated - #: ImportError raised on access when it is not installed (§3.2). + #: ImportError raised on access when it is not installed (BG §1.8). extra: str = "" #: Whether a governed model call through this adapter reaches ``donkey.last_call`` @@ -33,7 +33,7 @@ class Adapter: #: (ADK, CrewAI) or adapters given only ``default_headers`` (LlamaIndex, MS #: Agent Framework). A ``False`` here is why ``donkey.last_call`` reports #: "not available on this surface" rather than a bare ``None`` (hazard #3), - #: and it is the fact the conformance suite asserts as an exemption (§8.1). + #: and it is the fact the conformance suite asserts as an exemption (the conformance kit). observes_last_call: bool = True def __init__(self, cfg: DonkeyConfig, http_client: DonkeyAsyncClient) -> None: @@ -46,7 +46,7 @@ def _attribution_headers(self) -> dict[str, str]: def _proxy_headers(self) -> dict[str, str]: """Default headers for a native OpenAI-compatible client pointed at the proxy: the LIVE-VERIFIED client_id/client_secret consumer-auth pair plus - any attribution headers (docs §2/§3).""" + any attribution headers (docs/verified-apis.md §2/§3).""" return proxy_auth_headers(self._cfg) def _proxy_api_key(self) -> str: diff --git a/python/src/donkey_kit/integrations/adk.py b/python/src/donkey_kit/integrations/adk.py index ef98983e..b22c24c6 100644 --- a/python/src/donkey_kit/integrations/adk.py +++ b/python/src/donkey_kit/integrations/adk.py @@ -1,4 +1,4 @@ -"""Google ADK adapter (§3.3). +"""Google ADK adapter (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -8,7 +8,7 @@ Header injection: via LiteLLM's ``extra_headers``. We CANNOT inject our httpx client — LiteLLM owns the transport. Consequence: transport retries and correlation-ID-per-run degrade to per-client. This is a documented, asserted -conformance exemption (§8.1 ``correlation_id_propagated``). A LiteLLM custom +conformance exemption (the conformance kit's ``correlation_id_propagated``). A LiteLLM custom logger callback may later recover trace correlation. Class names / kwargs UNVERIFIED — docs/verified-apis.md §8. ADK requires @@ -28,14 +28,14 @@ class ADKAdapter(Adapter): extra = "adk" # LiteLLM owns the transport, so no response reaches donkey.last_call (#362, - # the same reason as the §8.1 correlation_id_propagated exemption). + # the same reason as the conformance kit's correlation_id_propagated exemption). observes_last_call = False def connection_kwargs(self) -> dict[str, Any]: """Governed kwargs for a ``LiteLlm(model="openai/", **kwargs)`` you build yourself. LiteLLM uses ``api_base``/``extra_headers`` (not ``base_url``/``default_headers``) and owns its own transport, so the - shared http client is not injected here (§3.3 exemption §8.1).""" + shared http client is not injected here (BG §1.8 exemption; the conformance kit).""" conn = self._openai_connection() return { "api_base": conn["base_url"], @@ -44,7 +44,7 @@ def connection_kwargs(self) -> dict[str, Any]: } def model(self, model: str, **kw: Any) -> LiteLlm: - from google.adk.models.lite_llm import LiteLlm # verified: docs §8 + from google.adk.models.lite_llm import LiteLlm # verified: docs/verified-apis.md §8 # LiteLLM's OpenAI-compatible route needs the ``openai/`` prefix. return LiteLlm(model=f"openai/{model}", **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/agent_framework.py b/python/src/donkey_kit/integrations/agent_framework.py index 9175bbd2..6aac243a 100644 --- a/python/src/donkey_kit/integrations/agent_framework.py +++ b/python/src/donkey_kit/integrations/agent_framework.py @@ -1,4 +1,4 @@ -"""Microsoft Agent Framework adapter (§3.3). +"""Microsoft Agent Framework adapter (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -10,7 +10,7 @@ Agent Framework has first-class middleware for intercepting agent actions. We ship :meth:`policy_middleware` that catches :class:`PolicyViolation` and terminates the run cleanly rather than letting the agent loop retry — the best -policy-integration story of any of the seven (§3.3), and the flagship example. +policy-integration story of any of the seven (BG §1.8), and the flagship example. """ from __future__ import annotations @@ -33,15 +33,17 @@ class AgentFrameworkAdapter(Adapter): def connection_kwargs(self) -> dict[str, Any]: """Governed kwargs for an ``OpenAIChatClient(model_id=…, **kwargs)`` you - build yourself. NOTE (§8): the class name/path and its base-URL kwarg are - UNVERIFIED — confirm against the installed version before relying on - constructing the client by hand.""" + build yourself. NOTE (docs/verified-apis.md §8): the class name/path and its base-URL kwarg + are UNVERIFIED — confirm against the installed version before relying on constructing the + client by hand.""" return self._openai_connection() # base_url, api_key, default_headers def chat_client(self, model: str, **kw: Any) -> Any: self._require_proxy() try: - from agent_framework.openai import OpenAIChatClient # VERIFY name/path §8 + from agent_framework.openai import ( + OpenAIChatClient, # VERIFY name/path docs/verified-apis.md §8 + ) except ImportError as exc: raise _verify.blocked( "agent_framework OpenAI chat client class name/path " @@ -51,17 +53,17 @@ def chat_client(self, model: str, **kw: Any) -> Any: ) from exc return OpenAIChatClient( - model_id=model, # VERIFY kwarg name §8 + model_id=model, # VERIFY kwarg name docs/verified-apis.md §8 **self.connection_kwargs(), **kw, ) def policy_middleware(self) -> Callable[..., Any]: """Middleware that converts a :class:`PolicyViolation` into a clean, - terminal agent state instead of letting the loop retry (§2.4). + terminal agent state instead of letting the loop retry (BG §1.2). The exact middleware signature Agent Framework expects is UNVERIFIED - (§0.3). We return a plain async wrapper and mark the shape for + (verification discipline). We return a plain async wrapper and mark the shape for verification rather than guessing the framework's middleware protocol. """ @@ -69,7 +71,7 @@ async def middleware(context: Any, next_: Callable[[Any], Any]) -> Any: try: return await next_(context) except PolicyViolation: - # Terminal: re-raise so the host does not silently retry (§2.4). + # Terminal: re-raise so the host does not silently retry (BG §1.2). # Once the middleware protocol is verified, set the framework's # explicit "terminate run" signal here instead of re-raising. raise diff --git a/python/src/donkey_kit/integrations/anthropic.py b/python/src/donkey_kit/integrations/anthropic.py index b8e332ba..11006576 100644 --- a/python/src/donkey_kit/integrations/anthropic.py +++ b/python/src/donkey_kit/integrations/anthropic.py @@ -1,4 +1,4 @@ -"""Anthropic SDK adapter (``donkey.anthropic``) (§3.3). +"""Anthropic SDK adapter (``donkey.anthropic``) (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -6,14 +6,14 @@ we construct the client ourselves and hand it our shared http client, header AND transport injection are both available (full injection). -Divergence, by design (§11.10 — the framework wins): Anthropic's native surface +Divergence, by design (BG §1.8 — the framework wins): Anthropic's native surface is a *client*, and the model id is a per-call argument, not a constructor one. So this adapter exposes ``client()`` rather than the ``model(...)`` factory the OpenAI-compatible adapters use. UNVERIFIED DEPENDENCY (docs/verified-apis.md §8): the Omni Gateway LLM proxy is OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API -route** is an open M0 verification item (§0.3). If it does not, this adapter's +route** is an open M0 verification item (verification discipline). If it does not, this adapter's requests will not reach a working upstream — override ``base_url`` via ``**kw`` to point at a real Anthropic-native route once confirmed. The first ``client()`` call emits a one-time :class:`~donkey_kit.core._verify.UnverifiedValueWarning`. @@ -60,14 +60,14 @@ def connection_kwargs(self) -> dict[str, Any]: "api_key": conn["api_key"], "default_headers": conn["default_headers"], "http_client": self._http_client(), - "max_retries": 0, # we retry in transport (§2.3) + "max_retries": 0, # we retry in transport (BG §1.1) } def client(self, **kw: Any) -> AsyncAnthropic: """Return a native ``anthropic.AsyncAnthropic`` pointed at the proxy. Pass the model id per call (``messages.create(model=..., ...)``), per the - Anthropic SDK's own surface (§3.1).""" - from anthropic import AsyncAnthropic # verified: docs §8 + Anthropic SDK's own surface (BG §1.8).""" + from anthropic import AsyncAnthropic # verified: docs/verified-apis.md §8 return AsyncAnthropic(**self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/crewai.py b/python/src/donkey_kit/integrations/crewai.py index 00da64e7..e2c9fb7a 100644 --- a/python/src/donkey_kit/integrations/crewai.py +++ b/python/src/donkey_kit/integrations/crewai.py @@ -1,4 +1,4 @@ -"""CrewAI adapter (§3.3). +"""CrewAI adapter (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -9,7 +9,7 @@ Header injection: via LiteLLM's ``extra_headers``. We CANNOT inject our httpx client — LiteLLM owns the transport. Consequence: transport retries and correlation-ID-per-run degrade to per-client, the same documented, asserted -conformance exemption as ADK (§8.1 ``correlation_id_propagated``). +conformance exemption as ADK (the conformance kit's ``correlation_id_propagated``). Class names / kwargs UNVERIFIED — docs/verified-apis.md §8. """ @@ -27,14 +27,14 @@ class CrewAIAdapter(Adapter): extra = "crewai" # LiteLLM owns the transport, so no response reaches donkey.last_call (#362, - # the same reason as the §8.1 correlation_id_propagated exemption). + # the same reason as the conformance kit's correlation_id_propagated exemption). observes_last_call = False def connection_kwargs(self) -> dict[str, Any]: """Governed kwargs for a ``crewai.LLM(model="openai/", **kwargs)`` you build yourself. ``crewai.LLM`` forwards to LiteLLM, which uses ``base_url``/``extra_headers`` and owns its own transport, so the shared - http client is not injected here (§3.3 exemption §8.1).""" + http client is not injected here (BG §1.8 exemption; the conformance kit).""" conn = self._openai_connection() return { "base_url": conn["base_url"], @@ -43,8 +43,8 @@ def connection_kwargs(self) -> dict[str, Any]: } def llm(self, model: str, **kw: Any) -> LLM: - """Return a native ``crewai.LLM`` pointed at the proxy (§3.1).""" - from crewai import LLM # verified: docs §8 + """Return a native ``crewai.LLM`` pointed at the proxy (BG §1.8).""" + from crewai import LLM # verified: docs/verified-apis.md §8 # LiteLLM's OpenAI-compatible route needs the ``openai/`` prefix. return LLM(model=f"openai/{model}", **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/langgraph.py b/python/src/donkey_kit/integrations/langgraph.py index e6056391..f3ee4288 100644 --- a/python/src/donkey_kit/integrations/langgraph.py +++ b/python/src/donkey_kit/integrations/langgraph.py @@ -1,4 +1,4 @@ -"""LangGraph / LangChain adapter (§3.3). +"""LangGraph / LangChain adapter (BG §1.8). The one deep, conformance-gated adapter (BG §1.8, #198). Header injection is FULL (``default_headers`` + our custom async http client, so every governed @@ -9,7 +9,7 @@ * :meth:`LangGraphAdapter.chat_model` / :func:`chat_model` / calling the adapter (``donkey.langgraph("gpt-4o")``) — three ways to get the **native** - ``langchain_openai.ChatOpenAI`` pointed at the proxy (§3.1, README §2). + ``langchain_openai.ChatOpenAI`` pointed at the proxy (BG §1.8, README §2). * :meth:`LangGraphAdapter.connection_kwargs` — the governed kwargs to spread into a ``ChatOpenAI`` you build yourself. * :func:`typed_refusals` — a context manager that turns the proxy refusal a node @@ -20,7 +20,8 @@ ``donkey.run(id=…)`` is visible via ``current_correlation_id()`` inside each node with nothing threaded through graph state. -All class names / kwargs are UNVERIFIED until §0.3 — see docs/verified-apis.md §8. +All class names / kwargs are UNVERIFIED until verification discipline — see +docs/verified-apis.md §8. """ from __future__ import annotations @@ -41,26 +42,27 @@ class LangGraphAdapter(Adapter): def connection_kwargs(self) -> dict[str, Any]: """Governed kwargs to spread into a ``ChatOpenAI(model=…, **kwargs)`` you - build yourself (§3.1). Same values the factory uses — one source of + build yourself (BG §1.8). Same values the factory uses — one source of truth for the proxy connection.""" conn = self._openai_connection() return { **conn, # base_url, api_key, default_headers "http_async_client": self._http_client(), # our client, our hooks - "max_retries": 0, # we retry in transport (§2.3) + "max_retries": 0, # we retry in transport (BG §1.1) # Target the proxy's LIVE-VERIFIED endpoint: the data plane is the - # OpenAI Responses API (``/responses``, docs §4) — the same route the + # OpenAI Responses API (``/responses``, docs/verified-apis.md §4) — the same route the # raw ``donkey.llm`` client uses. Left at ChatOpenAI's chat-completions # default, ``donkey.langgraph(...)`` would call an UNVERIFIED - # ``/chat/completions`` route and risk a 404 in a real sandbox (§0.3). + # ``/chat/completions`` route and risk a 404 in a real sandbox (verification + # discipline). # Override per call (``use_responses_api=False``) if a deployment # exposes chat-completions instead. "use_responses_api": True, } def chat_model(self, model: str, **kw: Any) -> ChatOpenAI: - """Return a native ``ChatOpenAI`` pointed at the proxy (§3.1).""" - from langchain_openai import ChatOpenAI # verified name: docs §8 + """Return a native ``ChatOpenAI`` pointed at the proxy (BG §1.8).""" + from langchain_openai import ChatOpenAI # verified name: docs/verified-apis.md §8 return ChatOpenAI(model=model, **self.connection_kwargs(), **kw) @@ -117,7 +119,7 @@ async def call_model(state: State) -> State: ``APIStatusError``) are transport failures, not gateway refusals, and pass through untouched. """ - import openai # lazy: only the framework path needs it (§1.1) + import openai # lazy: only the framework path needs it (the layered architecture) from ..core.errors import classify diff --git a/python/src/donkey_kit/integrations/llamaindex.py b/python/src/donkey_kit/integrations/llamaindex.py index ae1ba2f1..eb469456 100644 --- a/python/src/donkey_kit/integrations/llamaindex.py +++ b/python/src/donkey_kit/integrations/llamaindex.py @@ -1,8 +1,8 @@ -"""LlamaIndex adapter (§3.3). +"""LlamaIndex adapter (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). -GOTCHA (§3.3): ``OpenAILike`` defaults ``is_chat_model=False``, which silently +GOTCHA (BG §1.8): ``OpenAILike`` defaults ``is_chat_model=False``, which silently routes to the completions endpoint and fails against a chat-only proxy. We always set it True. This is the single most common LlamaIndex-with-a-gateway bug. @@ -41,7 +41,7 @@ def connection_kwargs(self) -> dict[str, Any]: } def llm(self, model: str, **kw: Any) -> OpenAILike: - from llama_index.llms.openai_like import OpenAILike # verified: docs §8 + from llama_index.llms.openai_like import OpenAILike # verified: docs/verified-apis.md §8 return OpenAILike(model=model, **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/integrations/openai_agents.py b/python/src/donkey_kit/integrations/openai_agents.py index e16d4117..e5c45a22 100644 --- a/python/src/donkey_kit/integrations/openai_agents.py +++ b/python/src/donkey_kit/integrations/openai_agents.py @@ -1,4 +1,4 @@ -"""OpenAI Agents SDK adapter (``donkey.openai_agents``) (§3.3). +"""OpenAI Agents SDK adapter (``donkey.openai_agents``) (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). @@ -6,7 +6,7 @@ provider as an ``OpenAIChatCompletionsModel`` wrapping an ``AsyncOpenAI`` client. Because we construct that client ourselves, header AND transport injection are both available (full injection) — the preferred pattern anywhere a framework -accepts a pre-built OpenAI client (§3.3). +accepts a pre-built OpenAI client (BG §1.8). Point the SDK's *model* at the proxy per-agent rather than mutating the global default client, so one process can mix governed and ungoverned models. @@ -30,7 +30,7 @@ class OpenAIAgentsAdapter(Adapter): def _proxy_openai_client(self) -> Any: """A native ``AsyncOpenAI`` client bound to the proxy: our shared http client + the verified consumer-auth headers. One source of truth for the - governed connection (§3.1).""" + governed connection (BG §1.8).""" conn = self._openai_connection() from openai import AsyncOpenAI @@ -42,19 +42,25 @@ def _proxy_openai_client(self) -> Any: api_key=conn["api_key"], default_headers=conn["default_headers"], http_client=self._http_client(), # type: ignore[arg-type] - max_retries=0, # we retry in transport (§2.3) + max_retries=0, # we retry in transport (BG §1.1) ) + def connection_kwargs(self) -> dict[str, Any]: + """Governed kwargs for an ``OpenAIChatCompletionsModel(model=…, **kwargs)`` + you build yourself. Unlike the OpenAI-compatible adapters this returns a + single ``openai_client`` key holding a pre-built native ``AsyncOpenAI`` + bound to the proxy — the Agents SDK takes a ready-made client, not loose + connection kwargs, so header AND transport injection travel as one object + (BG §1.8). Same client the factory uses — one source of truth for the + proxy connection.""" + return {"openai_client": self._proxy_openai_client()} + def model(self, model: str, **kw: Any) -> OpenAIChatCompletionsModel: """Return a native ``OpenAIChatCompletionsModel`` pointed at the proxy, - ready to pass into ``agents.Agent(model=...)`` (§3.1).""" - from agents import OpenAIChatCompletionsModel # verified: docs §8 + ready to pass into ``agents.Agent(model=...)`` (BG §1.8).""" + from agents import OpenAIChatCompletionsModel # verified: docs/verified-apis.md §8 - return OpenAIChatCompletionsModel( - model=model, - openai_client=self._proxy_openai_client(), - **kw, - ) + return OpenAIChatCompletionsModel(model=model, **self.connection_kwargs(), **kw) def model(model: str, **kw: Any) -> OpenAIChatCompletionsModel: diff --git a/python/src/donkey_kit/integrations/strands.py b/python/src/donkey_kit/integrations/strands.py index c31a2ea5..78a50dd8 100644 --- a/python/src/donkey_kit/integrations/strands.py +++ b/python/src/donkey_kit/integrations/strands.py @@ -1,11 +1,11 @@ -"""Strands Agents adapter (§3.3). +"""Strands Agents adapter (BG §1.8). Supported at connection_kwargs() — not conformance-tested (BG §1.8). ``client_args`` is forwarded to the underlying OpenAI client, so header AND transport injection are both available (full injection). Strands also has lifecycle hooks (``BeforeToolCallEvent`` and friends) — used elsewhere for the -policy-termination pattern (§2.4, §3.3). +policy-termination pattern (BG §1.2, BG §1.8). Class names / kwargs UNVERIFIED — docs/verified-apis.md §8. """ @@ -36,7 +36,7 @@ def connection_kwargs(self) -> dict[str, Any]: } def model(self, model: str, **kw: Any) -> OpenAIModel: - from strands.models.openai import OpenAIModel # verified: docs §8 + from strands.models.openai import OpenAIModel # verified: docs/verified-apis.md §8 return OpenAIModel(model_id=model, **self.connection_kwargs(), **kw) diff --git a/python/src/donkey_kit/llm/__init__.py b/python/src/donkey_kit/llm/__init__.py index ef36c992..282f3743 100644 --- a/python/src/donkey_kit/llm/__init__.py +++ b/python/src/donkey_kit/llm/__init__.py @@ -1,4 +1,4 @@ -"""llm/ — governed model access, framework-free surface (§3.2, §3.4).""" +"""llm/ — governed model access, framework-free surface (BG §1.8, BG §1.1).""" from .catalog import ModelCapabilities, ModelHandle, heuristic_capabilities from .client import LLMClient diff --git a/python/src/donkey_kit/llm/catalog.py b/python/src/donkey_kit/llm/catalog.py index 8290a4b1..ba5aa757 100644 --- a/python/src/donkey_kit/llm/catalog.py +++ b/python/src/donkey_kit/llm/catalog.py @@ -1,4 +1,4 @@ -"""Model catalog (§3.4). +"""Model catalog (BG §1.1). ``list_models()`` returns the logical model names the proxy exposes, not raw provider names. Prefer the proxy's own ``/models`` endpoint if it has one; fall @@ -6,7 +6,7 @@ ``ModelHandle`` carries enough capability metadata to feed any framework that requires explicit capability flags and to let a developer branch on -function-calling support (§3.4). +function-calling support (BG §1.1). """ from __future__ import annotations @@ -16,7 +16,7 @@ @dataclass(frozen=True) class ModelCapabilities: - """Capability flags. Conservative defaults where unknown (§3.4).""" + """Capability flags. Conservative defaults where unknown (BG §1.1).""" function_calling: bool = True vision: bool = False @@ -33,7 +33,7 @@ class ModelHandle: capabilities: ModelCapabilities = field(default_factory=ModelCapabilities) -# Bundled heuristic capability table (§3.4), keyed by well-known model IDs. +# Bundled heuristic capability table (BG §1.1), keyed by well-known model IDs. # Clearly a heuristic; users may override. NOT authoritative. _HEURISTIC_CAPS: dict[str, ModelCapabilities] = { "gpt-4o": ModelCapabilities(function_calling=True, vision=True, json_output=True), @@ -49,7 +49,7 @@ def heuristic_capabilities(model_id: str) -> ModelCapabilities: Conservative fallback for unknown models: function calling on, vision/json off. Frameworks that demand explicit capability flags can seed them from - this (§3.4). + this (BG §1.1). """ return _HEURISTIC_CAPS.get(model_id, ModelCapabilities()) diff --git a/python/src/donkey_kit/llm/client.py b/python/src/donkey_kit/llm/client.py index 03cbdeae..95d17c89 100644 --- a/python/src/donkey_kit/llm/client.py +++ b/python/src/donkey_kit/llm/client.py @@ -1,4 +1,4 @@ -"""Raw, framework-free LLM client + model listing (§3.2, §3.4). +"""Raw, framework-free LLM client + model listing (BG §1.8, BG §1.1). ``donkey.llm.client()`` returns an ``AsyncOpenAI`` pointed at the proxy, sharing the SDK's shared httpx client so attribution/correlation/auth headers and the @@ -90,16 +90,22 @@ def client(self, *, sync: bool = False, **kw: Any) -> AsyncOpenAI | OpenAI: assert self._cfg.llm_proxy_url is not None # validated() guarantees this shared: dict[str, Any] = { - "base_url": self._cfg.llm_proxy_url, # no /v1 at ingress (§2) — verbatim + # no /v1 at ingress (docs/verified-apis.md §2) — verbatim + "base_url": self._cfg.llm_proxy_url, "api_key": proxy_api_key(self._cfg), - "default_headers": proxy_auth_headers(self._cfg), # client_id/secret (§2/§3) - "max_retries": 0, # we retry in transport (§2.3) + # client_id/secret (docs/verified-apis.md §2/§3) + "default_headers": proxy_auth_headers(self._cfg), + "max_retries": 0, # we retry in transport (BG §1.1) **kw, } # openai 3.x retyped http_client to httpx2.AsyncClient (a distinct class from # a separate distribution); our DonkeyClient/DonkeyAsyncClient are httpx - # subclasses, duck-typed fine at runtime. Typecheck-only mismatch — see - # docs/verified-apis.md (openai >=3.0 row); no upper pin, by design (§8.4). + # subclasses. Typecheck-only mismatch: when an http_client is injected, + # openai sends every request THROUGH it, so httpx2 never touches this path. + # Runtime-verified end to end (async + sync) against openai 3.x by + # tests/unit/test_llm_client_openai3_injection.py (#18); see + # docs/verified-apis.md (openai >=3.0 row). No upper pin, by design (the + # floors-never-ceilings rule). if sync: return OpenAI(http_client=self._sync_http(), **shared) # type: ignore[arg-type] return AsyncOpenAI(http_client=self._http, **shared) # type: ignore[arg-type] @@ -108,28 +114,28 @@ async def list_models(self, *, live: bool = False) -> list[ModelHandle]: """List logical models the proxy exposes. The governed proxy has **no** catalog endpoint — ``GET /models`` returns - ``404`` (LIVE-VERIFIED, §2): model-based-routing only routes requests + ``404`` (LIVE-VERIFIED, docs/verified-apis.md §2): model-based-routing only routes requests that carry ``model`` in the body. So ``live=True`` cannot be satisfied, and we say so plainly rather than guess a path. Use :meth:`resolve` for a heuristic :class:`ModelHandle` from a known model id, or source the - catalog from Exchange / provider config (§3.4). + catalog from Exchange / provider config (BG §1.1). """ if live: raise ConfigError( - "The governed LLM proxy exposes no /models endpoint (GET /models " - "→ 404, verified §2): it only routes requests carrying `model` in " - "the body. Live model listing is not available from the proxy. Use " - "resolve(model_id) or source the catalog from Exchange/provider config." + "The governed LLM proxy exposes no /models endpoint (GET /models → 404, verified " + "docs/verified-apis.md §2): it only routes requests carrying `model` in the body. " + "Live model listing is not available from the proxy. Use resolve(model_id) or " + "source the catalog from Exchange/provider config." ) raise ConfigError( - "list_models() has no offline source of truth yet, and the proxy has " - "no /models endpoint to enumerate (verified §2). Use resolve(model_id) " - "to get a heuristic ModelHandle, or source models from Exchange (§3.4)." + "list_models() has no offline source of truth yet, and the proxy has no /models " + "endpoint to enumerate (verified docs/verified-apis.md §2). Use resolve(model_id) to " + "get a heuristic ModelHandle, or source models from Exchange (BG §1.1)." ) def resolve(self, model_id: str, *, provider: str | None = None) -> ModelHandle: - """A heuristic :class:`ModelHandle` for a known model id (§3.4).""" + """A heuristic :class:`ModelHandle` for a known model id (BG §1.1).""" return ModelHandle( id=model_id, provider=provider, diff --git a/python/src/donkey_kit/provisioning/__init__.py b/python/src/donkey_kit/provisioning/__init__.py index 53212afb..98d91fff 100644 --- a/python/src/donkey_kit/provisioning/__init__.py +++ b/python/src/donkey_kit/provisioning/__init__.py @@ -1,4 +1,4 @@ -"""provisioning/ — CI-oriented declarative provisioning (§5). +"""provisioning/ — CI-oriented declarative provisioning-as-code. Separate entry point from the runtime SDK. Nothing in the runtime path mutates shared state (working instruction #11); every mutation lives here and runs in CI diff --git a/python/src/donkey_kit/provisioning/applier.py b/python/src/donkey_kit/provisioning/applier.py index 622619a5..177531b0 100644 --- a/python/src/donkey_kit/provisioning/applier.py +++ b/python/src/donkey_kit/provisioning/applier.py @@ -1,11 +1,11 @@ -"""Apply (§5.2). +"""Apply (provisioning-as-code). -Partial-failure handling (§5.2): apply resources in dependency order, stop on +Partial-failure handling (provisioning-as-code): apply resources in dependency order, stop on first failure, report exactly what was applied and what was not. Do NOT attempt automatic rollback — report state and let the operator re-plan. Silent partial rollback in a control plane is worse than a clear stop. -Policy ownership (§5.4): apply runs in CI under a platform-controlled connected +Policy ownership (provisioning-as-code): apply runs in CI under a platform-controlled connected app, and may only reference policy assets enumerated in a platform-owned allow-list (``policy-catalog.yaml``). The allow-list mechanism ships in v1 even if nobody asks — its absence gets the SDK banned in security review. @@ -22,7 +22,7 @@ @dataclass(frozen=True) class PolicyAllowList: - """Platform-owned allow-list of referenceable policy assets (§5.4).""" + """Platform-owned allow-list of referenceable policy assets (provisioning-as-code).""" allowed_asset_ids: frozenset[str] @@ -50,7 +50,7 @@ async def apply( plan: Plan, donkey: object, *, allow_list: PolicyAllowList | None = None ) -> ApplyResult: raise _verify.blocked( - "MCP Bridge provisioning write API for apply (§5.2, §5, §0.3). " + "MCP Bridge provisioning write API for apply (provisioning-as-code). " "Read-before-write + stop-on-first-failure semantics are specified; wire " - "them once the API is confirmed, or emit Terraform (§5.5)." + "them once the API is confirmed, or emit Terraform (provisioning-as-code)." ) diff --git a/python/src/donkey_kit/provisioning/cli.py b/python/src/donkey_kit/provisioning/cli.py index e733c27d..58c5c1a4 100644 --- a/python/src/donkey_kit/provisioning/cli.py +++ b/python/src/donkey_kit/provisioning/cli.py @@ -1,6 +1,6 @@ -"""The ``donkey`` CLI (§5.2, §7). +"""The ``donkey`` CLI (provisioning-as-code, BG §2.5). -Telemetry is on by default (§2.5, BG §1.6), but export stays inert unless an +Telemetry is on by default (BG §1.6), but export stays inert unless an OTLP endpoint is configured: set ``OTEL_EXPORTER_OTLP_ENDPOINT`` and spans flow to your own sink with no SDK-specific env var; opt out entirely with ``DONKEY_TELEMETRY=false``. Commands that need a verified @@ -11,6 +11,9 @@ from __future__ import annotations +import importlib.util +import json +import subprocess import sys from pathlib import Path @@ -24,7 +27,8 @@ ) raise SystemExit(1) from None -from ..core.errors import DonkeyError +from ..core.config import _TOML_NAME, DonkeyConfig +from ..core.errors import ConfigError, DonkeyError from .spec import DonkeySpec app = typer.Typer( @@ -33,6 +37,24 @@ ) +@app.callback() +def _global( + ctx: typer.Context, + config: Path | None = typer.Option( + None, "--config", metavar="PATH", help=f"Path to {_TOML_NAME} (default: cwd)." + ), + env: str | None = typer.Option( + None, "--env", metavar="NAME", help="Anypoint environment override (e.g. Sandbox)." + ), + as_json: bool = typer.Option( + False, "--json", help="Emit machine-readable JSON where the command supports it." + ), +) -> None: + """Global flags shared by every command (provisioning-as-code). Precede the subcommand: + ``donkey --json init``, ``donkey --config ./cfg.toml doctor``.""" + ctx.obj = {"config": config, "env": env, "json": as_json} + + def _load_spec(file: Path) -> DonkeySpec: try: return DonkeySpec.from_yaml(file.read_text()) @@ -44,16 +66,22 @@ def _load_spec(file: Path) -> DonkeySpec: def _blocked(what: str) -> None: typer.secho(f"blocked on verification: {what}", fg="yellow", err=True) typer.secho( - "See docs/verified-apis.md — this command is scaffolded but not wired " - "until the underlying platform API is confirmed against a sandbox (§0.3).", + "See docs/verified-apis.md — this command is scaffolded but not wired until the underlying " + "platform API is confirmed against a sandbox (verification discipline).", err=True, ) raise typer.Exit(3) -@app.command() +@app.command(hidden=True) def validate(file: Path = typer.Option(..., "-f", "--file", help="donkey.yaml")) -> None: - """Validate a donkey.yaml against the schema (needs no platform API).""" + """Validate a donkey.yaml against the schema (needs no platform API). + + Hidden: it operates on the ``donkey.yaml`` provisioning spec, part of the + provisioning control-plane surface that is cut for Phase 1 (`BG §1.8`). The + command still works for anyone driving that YAML, but the four supported + ``donkey`` commands are ``init``/``doctor``/``mock``/``test``. + """ spec = _load_spec(file) typer.secho( f"OK: {spec.metadata.name} — {len(spec.mcpBridges)} MCP bridge(s), " @@ -62,68 +90,241 @@ def validate(file: Path = typer.Option(..., "-f", "--file", help="donkey.yaml")) ) -@app.command() +@app.command(hidden=True) def plan(file: Path = typer.Option(..., "-f", "--file"), dry_run: bool = typer.Option(False, "--dry-run"), out: Path | None = typer.Option(None, "--out", help="write plan.json for CI")) -> None: - """Show the create/update/remove plan (read-before-write, §5.2).""" + """Show the create/update/remove plan (read-before-write, provisioning-as-code).""" _load_spec(file) - _blocked("MCP Bridge provisioning read API (§5.2, §5)") + _blocked("MCP Bridge provisioning read API (provisioning-as-code)") -@app.command() +@app.command(hidden=True) def apply(file: Path = typer.Option(..., "-f", "--file"), auto_approve: bool = typer.Option(False, "--auto-approve")) -> None: - """Apply the plan (CI-only, platform-controlled creds, §5.4).""" + """Apply the plan (CI-only, platform-controlled creds, provisioning-as-code).""" _load_spec(file) - _blocked("MCP Bridge provisioning write API (§5.2, §5.4)") + _blocked("MCP Bridge provisioning write API (provisioning-as-code)") -@app.command() +@app.command(hidden=True) def drift(file: Path = typer.Option(..., "-f", "--file")) -> None: - """Compare live state against the spec; exit non-zero on drift (§5.2).""" + """Compare live state against the spec; exit non-zero on drift (provisioning-as-code).""" _load_spec(file) - _blocked("MCP Bridge provisioning read API (§5.2)") + _blocked("MCP Bridge provisioning read API (provisioning-as-code)") -@app.command() +@app.command(hidden=True) def lint(file: Path = typer.Option(..., "-f", "--file")) -> None: - """Governance lint (§5.3). Local spec-shape checks run now; ruleset - resolution is gated (§0.3).""" + """Governance lint (provisioning-as-code). Local spec-shape checks run now; ruleset + resolution is gated (verification discipline).""" _load_spec(file) - _blocked("governance rulesets resolution API (§5.3, §0.3)") + _blocked("governance rulesets resolution API (provisioning-as-code)") -@app.command() +@app.command(hidden=True) def generate(file: Path = typer.Option(..., "-f", "--file"), target: str = typer.Option("terraform", "--target")) -> None: - """Emit Terraform from the spec — the §5.5 pivot if provisioning is UI-only.""" + """Emit Terraform from the spec — the provisioning-as-code pivot if provisioning is UI-only.""" _load_spec(file) - _blocked("Terraform provider coverage enumeration (§5.5, §0.3)") + _blocked("Terraform provider coverage enumeration (provisioning-as-code)") -@app.command() +@app.command(hidden=True) def status() -> None: - """Render published / reachable / governed per asset (§7.6).""" - _blocked("Exchange + API Manager read APIs (§7.6)") + """Render published / reachable / governed per asset (BG §2.5).""" + _blocked("Exchange + API Manager read APIs (BG §2.5)") -@app.command() -def init() -> None: - """Scan the project, propose publishable assets, write .donkey-kit.toml (§7.8).""" - _blocked("per-framework asset detection + descriptor derivation (§7.8, §7.3)") +# Config fields written to .donkey-kit.toml, grouped with a header comment. +# SECRETS ARE DELIBERATELY ABSENT — see _SECRET_FIELDS. This list is +# non-secret connection config only. +_INIT_GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = ( + ( + "Anypoint control plane (registry + provisioning)", + ("client_id", "org_id", "environment", "region", "base_url"), + ), + ( + "LLM proxy (data plane) — a SEPARATE credential from the control plane", + ("llm_proxy_url", "llm_proxy_client_id"), + ), + ( + "Attribution", + ("application_name", "business_group"), + ), +) + +# Written NEVER — as commented placeholders only. A committed config file is +# the wrong home for a secret (config resolution); these belong in env vars or a gitignored +# .donkey-kit.local.toml. +_SECRET_FIELDS: tuple[tuple[str, str], ...] = ( + ("client_secret", "ANYPOINT_CLIENT_SECRET"), + ("llm_proxy_client_secret", "DONKEY_LLM_PROXY_CLIENT_SECRET"), + ("llm_proxy_key", "DONKEY_LLM_PROXY_KEY"), +) + + +def _toml_str(value: str) -> str: + """Render a TOML basic string, escaping the two characters that matter.""" + escaped = value.replace("\\", "\\\\").replace('"', '\\"') + return f'"{escaped}"' + + +def _collect_missing(config: DonkeyConfig) -> list[str]: + """Every missing required field across BOTH capabilities, in one list — + reusing ``DonkeyConfig.validated`` as the single source of truth for what + is required (config resolution), rather than duplicating the field set here.""" + missing: list[str] = [] + for need in ("control_plane", "llm"): + try: + config.validated(need=need) + except ConfigError as exc: + for line in str(exc).splitlines(): + stripped = line.strip() + if stripped.startswith("- "): + missing.append(stripped[2:]) + return missing + + +def _render_toml(config: DonkeyConfig, missing: list[str]) -> str: + """Hand-render a commented ``.donkey-kit.toml`` from resolved, non-secret + values. No dependency on a TOML *writer*; the reader (`tomllib`) is enough.""" + lines: list[str] = [ + f"# {_TOML_NAME} — generated by `donkey init` from your current environment.", + "# Review before committing. Resolution order at runtime: kwargs → env →", + "# this file → defaults (config resolution).", + "#", + "# SECRETS ARE NEVER WRITTEN HERE. Provide them via environment variables", + "# (or a gitignored .donkey-kit.local.toml):", + ] + for field, envvar in _SECRET_FIELDS: + lines.append(f"# - {field:<24} → env {envvar}") + if missing: + lines += [ + "#", + "# Still missing (set these before the SDK can reach the platform):", + ] + lines += [f"# - {item}" for item in missing] + lines += ["", "[donkey]"] + for header, group_fields in _INIT_GROUPS: + emitted = [ + (name, getattr(config, name)) + for name in group_fields + if getattr(config, name) is not None + ] + if not emitted: + continue + lines.append(f"# {header}") + for name, value in emitted: + lines.append(f"{name} = {_toml_str(str(value))}") + lines.append("") + return "\n".join(lines).rstrip() + "\n" @app.command() +def init( + ctx: typer.Context, + force: bool = typer.Option( + False, "--force", help="Regenerate even if the file already exists." + ), +) -> None: + """Bootstrap a commented ``.donkey-kit.toml`` from the resolved config (config resolution). + + Writes every non-secret value already visible via kwargs/env/toml, and names + EVERY missing required field at once — reusing the same + ``DonkeyConfig.validated`` report the SDK uses at runtime, so ``init`` never + disagrees with a live call about what is required. Secrets are never written + (they surface as commented ``env`` pointers). Idempotent: an existing file is + left untouched unless ``--force`` is passed. Honors the global ``--config`` + (write target), ``--env`` (environment override), and ``--json`` flags. + """ + opts = ctx.obj or {} + target: Path = opts.get("config") or (Path.cwd() / _TOML_NAME) + as_json: bool = bool(opts.get("json")) + + if target.exists() and not force: + msg = f"{target} already exists; leaving it untouched. Re-run with --force to regenerate." + if as_json: + typer.echo(json.dumps({"path": str(target), "written": False, "missing": []})) + else: + typer.secho(msg, fg="yellow") + return + + config = DonkeyConfig.from_env() + env_override = opts.get("env") + if env_override: + config = config.with_overrides(environment=env_override) + + missing = _collect_missing(config) + content = _render_toml(config, missing) + + try: + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(content) + except OSError as exc: + typer.secho(f"Could not write {target}: {exc}", fg="red", err=True) + raise typer.Exit(1) from exc + + if as_json: + typer.echo(json.dumps({"path": str(target), "written": True, "missing": missing})) + return + + typer.secho(f"Wrote {target}.", fg="green") + if missing: + typer.echo("\nStill missing (set before the SDK can reach the platform):") + for item in missing: + typer.echo(f" - {item}") + else: + typer.echo("All required fields resolved.") + + +@app.command(context_settings={"allow_extra_args": True, "ignore_unknown_options": True}) +def test( + ctx: typer.Context, + agent: str | None = typer.Option( + None, + "--agent", + metavar="MODULE:FACTORY", + help="Import path to your agent factory, e.g. my.pkg:make_agent.", + ), +) -> None: + """Run the conformance suite against your agent (BG §1.5). + + A thin front end to ``pytest --donkey-conformance`` — it does not + re-implement the runner. Your ``--agent MODULE:FACTORY`` and any trailing + pytest arguments pass straight through, and pytest's exit code becomes + ``donkey test``'s own, so it drops into CI unchanged. Needs the ``[test]`` + extra (a missing pytest is an install prompt, exit 1, not a ``blocked on + verification`` message). + """ + if importlib.util.find_spec("pytest") is None: + typer.secho( + 'donkey test needs the [test] extra. Install it with:\n' + ' pip install "donkey-kit[test]"', + fg="yellow", + err=True, + ) + raise typer.Exit(1) + + argv = [sys.executable, "-m", "pytest", "--donkey-conformance"] + if agent: + argv += ["--agent", agent] + argv += list(ctx.args) + + completed = subprocess.run(argv, check=False) + raise typer.Exit(completed.returncode) + + +@app.command(hidden=True) def publish(if_changed: bool = typer.Option(True, "--if-changed/--always")) -> None: - """Publish code-first assets to Exchange (CI-only, §7.5/§7.7).""" - _blocked("Exchange publication mechanism + digest metadata (§7.5, §7.9)") + """Publish code-first assets to Exchange (CI-only, BG §2.5).""" + _blocked("Exchange publication mechanism + digest metadata (BG §2.5)") -@app.command() +@app.command(hidden=True) def verify() -> None: - """Check the live server against the Exchange descriptor (§7.4).""" - _blocked("Exchange descriptor read + live introspection (§7.4, §7.9)") + """Check the live server against the Exchange descriptor (BG §2.5).""" + _blocked("Exchange descriptor read + live introspection (BG §2.5)") @app.command() @@ -185,6 +386,7 @@ def mock( @app.command() def doctor( + ctx: typer.Context, model: str = typer.Option( "gpt-4o", "--model", help="model id to test against the proxy allow-list" ), @@ -192,7 +394,7 @@ def doctor( ) -> None: """Diagnose governed access: config, credentials, gateway, model, budget (#202). - Makes ONE real governed call and reads the result through the §2.4 error + Makes ONE real governed call and reads the result through the BG §1.2 error taxonomy to tell the three look-alike failures apart — wrong URL (``GatewayUnavailable``), wrong credentials (``AuthError``), and credentials-fine-but-model-rejected (the verified ``model_not_found`` @@ -208,6 +410,10 @@ def doctor( from ..core.errors import DonkeyError from . import doctor as _doctor + # The global --json (before the subcommand) and the local --json (after it) + # are equivalent — either turns on machine-readable output. + as_json = as_json or bool((ctx.obj or {}).get("json")) + try: checks = _doctor.run_diagnostics(model) except ImportError as exc: # openai (the [llm] extra) not installed @@ -225,8 +431,6 @@ def doctor( raise typer.Exit(1) from exc if as_json: - import json - typer.echo( json.dumps( [ diff --git a/python/src/donkey_kit/provisioning/doctor.py b/python/src/donkey_kit/provisioning/doctor.py index 3ee4ee4e..49c8a06e 100644 --- a/python/src/donkey_kit/provisioning/doctor.py +++ b/python/src/donkey_kit/provisioning/doctor.py @@ -1,6 +1,6 @@ """``donkey doctor`` — turn "why doesn't this work" into a thirty-second answer (#202). -One governed probe call, read through the §2.4 error taxonomy, tells the three +One governed probe call, read through the BG §1.2 error taxonomy, tells the three failures that look identical from the outside apart: * **wrong URL / unreachable gateway** — a transport-level failure with no HTTP @@ -17,7 +17,7 @@ has no budget-query endpoint (upstream gap #2), so a budget is only ever as fresh as the last response, and doctor never implies otherwise (AC3). -Honest scope (§0.3): the gateway's *allow-list* rejection (a model refused by +Honest scope (verification discipline): the gateway's *allow-list* rejection (a model refused by API Manager policy rather than missing at the provider) has no captured 403 shape yet, and enumerating the allowed alternatives needs the discovery endpoint (upstream gap #3, out of scope per #202). So the model line diagnoses @@ -155,7 +155,7 @@ def _probe_checks(result: ProbeResult) -> list[Check]: # The LIVE-VERIFIED model rejection is the provider passthrough (400 # model_not_found), which classify() maps to UpstreamRequestError carrying - # code/param. A 403 allow-list rejection has no captured shape yet (§0.3) and + # code/param. A 403 allow-list rejection has no captured shape yet (verification discipline) and # would surface as AuthError above — a known, documented limitation. if isinstance(err, UpstreamRequestError) and ( err.code == "model_not_found" or err.param == "model" diff --git a/python/src/donkey_kit/provisioning/lint.py b/python/src/donkey_kit/provisioning/lint.py index 97cee9cf..3e97386d 100644 --- a/python/src/donkey_kit/provisioning/lint.py +++ b/python/src/donkey_kit/provisioning/lint.py @@ -1,11 +1,11 @@ -"""Governance lint (§5.3). +"""Governance lint (provisioning-as-code). Small, cheap, uncontroversial, and the fastest way to get a platform team to say yes to the SDK — built in M1 even though the rest of provisioning is M3/M4. Validates API specs against project and centralized rulesets BEFORE anything is published, and fails the PR on ``error`` severity. Ruleset RESOLUTION against the -platform is gated (§0.3 — is a rulesets API exposed?), but local spec-shape +platform is gated (verification discipline — is a rulesets API exposed?), but local spec-shape validation and per-rule severity handling are implemented here. """ @@ -35,7 +35,7 @@ class LintResult: @property def failed(self) -> bool: - """True if any ``error``-severity finding exists — fail the PR (§5.3).""" + """True if any ``error``-severity finding exists — fail the PR (provisioning-as-code).""" return any(f.severity is Severity.ERROR for f in self.findings) def render(self) -> str: diff --git a/python/src/donkey_kit/provisioning/planner.py b/python/src/donkey_kit/provisioning/planner.py index 4baf6077..0af49bbb 100644 --- a/python/src/donkey_kit/provisioning/planner.py +++ b/python/src/donkey_kit/provisioning/planner.py @@ -1,4 +1,4 @@ -"""Plan / diff (§5.2). +"""Plan / diff (provisioning-as-code). Requirements the implementer must honour when this is wired to a real API: * Read-before-write, always — fetch current state, diff, render, then apply. @@ -6,9 +6,9 @@ * Idempotent — re-running apply with no change makes zero mutating calls. * ``--dry-run`` / ``--out plan.json`` for CI gating. -GATE (§5, §0.3): whether a usable MCP Bridge provisioning API exists is an M0 -finding. If UI-only, this whole module is cut and we emit Terraform instead -(§5.5). Until confirmed, planning against live state is blocked. +GATE (provisioning-as-code, verification discipline): whether a usable MCP Bridge provisioning API +exists is an M0 finding. If UI-only, this whole module is cut and we emit Terraform instead +(provisioning-as-code). Until confirmed, planning against live state is blocked. """ from __future__ import annotations @@ -47,6 +47,6 @@ def render(self) -> str: async def build_plan(spec: DonkeySpec, donkey: object) -> Plan: raise _verify.blocked( "MCP Bridge provisioning read API for read-before-write planning " - "(§5.2, §5, §0.3). If M0 finds it UI-only, pivot to Terraform generation " - "(§5.5) — do NOT reverse-engineer internal endpoints." + "(provisioning-as-code). If M0 finds it UI-only, pivot to Terraform generation " + "(provisioning-as-code) — do NOT reverse-engineer internal endpoints." ) diff --git a/python/src/donkey_kit/provisioning/publish.py b/python/src/donkey_kit/provisioning/publish.py index 60d9dbbd..7bc6278b 100644 --- a/python/src/donkey_kit/provisioning/publish.py +++ b/python/src/donkey_kit/provisioning/publish.py @@ -1,9 +1,9 @@ -"""Exchange publication, content digest, ``--if-changed`` (§7.5). +"""Exchange publication, content digest, ``--if-changed`` (BG §2.5). Prevents catalog spam (a new version on every merge). The content-digest computation is pure and implemented; the publish call and the "compare against latest published digest" read are gated on the verified Exchange mechanism -(§7.9). +(BG §2.5). """ from __future__ import annotations @@ -16,7 +16,7 @@ def content_digest(descriptor: dict[str, Any], metadata: dict[str, Any]) -> str: - """Stable hash over the canonical descriptor + metadata (§7.5). + """Stable hash over the canonical descriptor + metadata (BG §2.5). Canonicalised with sorted keys so semantically-identical inputs hash equal, which is what makes ``--if-changed`` reliable. @@ -31,11 +31,11 @@ def content_digest(descriptor: dict[str, Any], metadata: dict[str, Any]) -> str: async def publish_if_changed(publication: object, donkey: object) -> str: - """CI default (§7.5): compare digest against the latest published version; + """CI default (BG §2.5): compare digest against the latest published version; if identical, skip and exit zero. Blocked on the verified publication - mechanism + digest-metadata support (§7.9).""" + mechanism + digest-metadata support (BG §2.5).""" raise _verify.blocked( "Exchange publication mechanism (REST/CLI/Maven) + digest metadata support " - "(§7.5, §7.9). content_digest() is implemented; wire publish once the " - "mechanism is confirmed. Never delete/overwrite; deprecate via metadata (§7.5)." + "(BG §2.5). content_digest() is implemented; wire publish once the " + "mechanism is confirmed. Never delete/overwrite; deprecate via metadata (BG §2.5)." ) diff --git a/python/src/donkey_kit/provisioning/spec.py b/python/src/donkey_kit/provisioning/spec.py index 6667b61d..af769757 100644 --- a/python/src/donkey_kit/provisioning/spec.py +++ b/python/src/donkey_kit/provisioning/spec.py @@ -1,16 +1,16 @@ -"""Declarative spec models (§5.1). +"""Declarative spec models (provisioning-as-code). ``Governance.export()`` and ``Publication.export()`` emit fragments of exactly this format — the features share ONE schema, deliberately. Do not let them -diverge (§5.1). +diverge (provisioning-as-code). Design boundaries baked in here: * ``inputSchema: "auto"`` — derive from the API's published spec in Exchange - (§5.1). The OAS/RAML→JSON-Schema transform runs offline in CI (planner). - * DataWeave is a HARD boundary (§5.1): a raw ``httpMapping.dataweave`` string + (provisioning-as-code). The OAS/RAML→JSON-Schema transform runs offline in CI (planner). + * DataWeave is a HARD boundary (provisioning-as-code): a raw ``httpMapping.dataweave`` string passes through untouched; we never generate or parse DataWeave. * No secrets in the spec — reference them (``${secret:...}``), resolved at - apply time (§5.1). + apply time (provisioning-as-code). """ from __future__ import annotations @@ -25,13 +25,13 @@ class ToolSpec(BaseModel): method: str resource: str description: str - #: "auto" derives JSON Schema from the OAS/RAML spec in Exchange (§5.1), + #: "auto" derives JSON Schema from the OAS/RAML spec in Exchange (provisioning-as-code), #: or an explicit JSON Schema object. inputSchema: Literal["auto"] | dict[str, Any] = "auto" class HttpMapping(BaseModel): - """DataWeave passthrough only — never generated or parsed (§5.1).""" + """DataWeave passthrough only — never generated or parsed (provisioning-as-code).""" dataweave: str | None = None diff --git a/python/src/donkey_kit/registry/__init__.py b/python/src/donkey_kit/registry/__init__.py index 988ec2ee..c6dc4295 100644 --- a/python/src/donkey_kit/registry/__init__.py +++ b/python/src/donkey_kit/registry/__init__.py @@ -1,4 +1,4 @@ -"""registry/ — Exchange discovery + governed-only predicate (§4.2, §6.1).""" +"""registry/ — Exchange discovery + governed-only predicate (BG §2.7).""" from .exchange import ExchangeRegistry from .governance import STRICT, Check, GovernanceCriteria, GovernanceReport, evaluate diff --git a/python/src/donkey_kit/registry/exchange.py b/python/src/donkey_kit/registry/exchange.py index ada93f70..069bf348 100644 --- a/python/src/donkey_kit/registry/exchange.py +++ b/python/src/donkey_kit/registry/exchange.py @@ -1,9 +1,9 @@ -"""Exchange discovery + the governed-only join (§4.2, §6.1). +"""Exchange discovery + the governed-only join (BG §2.7). The public signatures are fixed here so callers and tests can be written now. Every method that requires a real Anypoint endpoint is gated behind a -verification-blocked error (§0.3, working instruction #2) rather than a -fabricated request path. The N+1-avoiding index design (§6.1.3) is documented in +verification-blocked error (verification discipline, working instruction #2) rather than a +fabricated request path. The N+1-avoiding index design is documented in :meth:`warm` for the implementer who fills it in after M0. """ @@ -38,7 +38,7 @@ async def search( raise _verify.blocked( "Exchange search API (endpoint, query params, response shape — " "docs/verified-apis.md §1/§7). Confirm against a sandbox and capture " - "a fixture (§8.2) before implementing search()." + "a fixture (BG §1.5) before implementing search()." ) async def resolve_mcp(self, ref: AssetRef | str) -> McpServerHandle: @@ -56,28 +56,29 @@ async def resolve_agent(self, ref: AssetRef | str) -> AgentHandle: async def explain( self, ref: AssetRef | str, *, criteria: GovernanceCriteria ) -> GovernanceReport: - """Explain why an asset is (not) governed (§6.1.2). + """Explain why an asset is (not) governed. First-class, documented, referenced in the empty-result warning — without it, ``governed=True`` returning empty is indistinguishable from a broken credential. Blocked until the API Manager / ruleset read APIs are - verified (§6.7). + verified (the Verification milestone). """ AssetRef.parse(ref) raise _verify.blocked( - "governed-state join: per-instance 'deployed' readability and ruleset " - "results API (§6.7). Until verified, explain() cannot produce real " - "Check rows; see registry/governance.py for the pure evaluation logic." + "governed-state join: per-instance 'deployed' readability and ruleset results API (the " + "Verification milestone). Until verified, explain() cannot produce real Check rows; " + "see registry/governance.py for the pure evaluation logic." ) async def warm(self, *, environment: str | None = None) -> None: - """Build the in-memory governance index at startup (§6.1.3). + """Build the in-memory governance index at startup. Design (fill in after M0): 1. ONE call to list all API Manager instances for (org, environment); index by (groupId, assetId, version) AND (groupId, assetId). - 2. Bulk policies call if one exists (verify §6.7), else per-candidate. + 2. Bulk policies call if one exists (verify the Verification milestone), else + per-candidate. 3. Apply Exchange-side filters (tags/lifecycle/type) BEFORE any API Manager calls to shrink the candidate set. 4. Cache the whole index under registry_cache_ttl_s, keyed by env. @@ -85,7 +86,7 @@ async def warm(self, *, environment: str | None = None) -> None: raise _verify.blocked( "API Manager instance-list + bulk-policy APIs for the governed-state " - "index (§6.1.3, §6.7)." + "index (the Verification milestone)." ) def refresh(self) -> None: diff --git a/python/src/donkey_kit/registry/governance.py b/python/src/donkey_kit/registry/governance.py index 407d6bc4..77eea778 100644 --- a/python/src/donkey_kit/registry/governance.py +++ b/python/src/donkey_kit/registry/governance.py @@ -1,10 +1,10 @@ -"""Governed-only discovery (§6.1). +"""Governed-only discovery. "Governed" is NOT a flag — it is a computed predicate joining state across three -systems, environment-scoped (§6.1.1). The criteria and report types here are +systems, environment-scoped. The criteria and report types here are pure and fully implemented. The JOIN that populates a report from live API Manager / ruleset state lives in :mod:`donkey_kit.registry.exchange` and is -gated on M0 verification (§6.7). +gated on M0 verification (the Verification milestone). """ from __future__ import annotations @@ -28,7 +28,7 @@ class GovernanceCriteria: require_lifecycle: list[str] = field(default_factory=list) #: If a check cannot be evaluated, does the asset pass? Matters a lot: with #: allow_unknown=False an unavailable API silently filters the catalog to - #: zero — which is why every excluded asset MUST carry a reason (§6.1.1). + #: zero — which is why every excluded asset MUST carry a reason (governed-only discovery). allow_unknown: bool = False @@ -42,7 +42,7 @@ class GovernanceCriteria: @dataclass(frozen=True) class Check: """One governance condition's outcome. ``passed=None`` means UNKNOWN / - could-not-evaluate — surfaced, never silently dropped (§6.1.2).""" + could-not-evaluate — surfaced, never silently dropped (governed-only discovery).""" name: str passed: bool | None @@ -51,7 +51,7 @@ class Check: @dataclass(frozen=True) class GovernanceReport: - """Result of :meth:`explain` — why an asset is (not) governed (§6.1.2).""" + """Result of :meth:`explain` — why an asset is (not) governed.""" governed: bool checks: list[Check] @@ -65,7 +65,7 @@ def evaluate(checks: list[Check], criteria: GovernanceCriteria) -> GovernanceRep A ``passed=None`` (UNKNOWN) check passes only when ``allow_unknown`` is True; otherwise it fails the asset — but the reason is always retained so a - developer can tell "filtered out" from "broken credential" (§6.1.2). + developer can tell "filtered out" from "broken credential" (governed-only discovery). """ governed = True diff --git a/python/src/donkey_kit/registry/introspect.py b/python/src/donkey_kit/registry/introspect.py index ef862fee..a129ff10 100644 --- a/python/src/donkey_kit/registry/introspect.py +++ b/python/src/donkey_kit/registry/introspect.py @@ -1,11 +1,11 @@ -"""descriptor='auto' derivation (§7.3). +"""descriptor='auto' derivation (BG §2.5). -KEY INSIGHT (§7.3.1): do NOT write a type-hint→JSON-Schema converter. Every +KEY INSIGHT (BG §2.5): do NOT write a type-hint→JSON-Schema converter. Every framework already derived the schema from signatures/type-hints/docstrings; ask the framework for the schema it already computed. Re-deriving would produce a second, subtly-different schema from the one the model actually sees. -Modes, by fidelity (§7.3.2): +Modes, by fidelity (BG §2.5): * ``auto`` — object introspection (default): import the entrypoint, read already-computed schemas. Importing user code EXECUTES it → require an explicit entrypoint; tool definitions must be import-safe. @@ -17,8 +17,8 @@ mismatch. The CI-friendly mode. Every per-framework attribute read here is semi-public and WILL break on -upstream releases → all go in the conformance kit + nightly matrix (§7.3.1, -§8.4). Names are UNVERIFIED (docs/verified-apis.md §10). +upstream releases → all go in the conformance kit + nightly matrix (BG §2.5). +Names are UNVERIFIED (docs/verified-apis.md §10). """ from __future__ import annotations @@ -32,9 +32,9 @@ async def derive_descriptor( entrypoint: str, *, mode: DerivationMode = "auto" ) -> dict[str, object]: """Derive an asset descriptor from code. Blocked until the per-framework - attribute contracts are verified (§7.3.1 / §10).""" + attribute contracts are verified (BG §2.5 / docs/verified-apis.md §10).""" raise _verify.blocked( f"per-framework tool-object attribute contracts for descriptor derivation " f"mode {mode!r} (docs/verified-apis.md §10). Read the already-computed " - f"schema; never re-derive it (§7.3.1)." + f"schema; never re-derive it (BG §2.5)." ) diff --git a/python/src/donkey_kit/registry/models.py b/python/src/donkey_kit/registry/models.py index 05ea460c..178e753d 100644 --- a/python/src/donkey_kit/registry/models.py +++ b/python/src/donkey_kit/registry/models.py @@ -1,4 +1,4 @@ -"""Registry value types (§4.2).""" +"""Registry value types (BG §2.7).""" from __future__ import annotations @@ -10,7 +10,7 @@ @dataclass(frozen=True) class AssetRef: - """``group_id/asset_id/version`` plus discovery metadata (§4.2). + """``group_id/asset_id/version`` plus discovery metadata (BG §2.7). Accept a shorthand string form (``"com.acme/vendor-shipment-mcp/1.0.0"``) everywhere a ref is taken, via :meth:`parse`. @@ -46,7 +46,7 @@ def coordinates(self) -> str: @dataclass(frozen=True) class McpServerHandle: """A resolved MCP server. Carries connection info; does NOT open a - connection (§4.3 — connect on first tool use).""" + connection (BG §2.7 — connect on first tool use).""" ref: AssetRef endpoint_url: str @@ -58,7 +58,7 @@ class McpServerHandle: @dataclass(frozen=True) class AgentHandle: - """A resolved A2A-compliant agent (§4.5).""" + """A resolved A2A-compliant agent (BG §2.9).""" ref: AssetRef endpoint_url: str diff --git a/python/src/donkey_kit/registry/publication.py b/python/src/donkey_kit/registry/publication.py index ebb186ca..d37aeda2 100644 --- a/python/src/donkey_kit/registry/publication.py +++ b/python/src/donkey_kit/registry/publication.py @@ -1,14 +1,14 @@ -"""Publication — registering code-first assets into Exchange (§7). +"""Publication — registering code-first assets into Exchange (BG §2.5). -Symmetric with §6: one declarative object, three verbs, "declare in code, apply -in CI, verify at runtime". The symmetry breaks at runtime (§7.4): there is NO +Symmetric with governed-only discovery: one declarative object, three verbs, "declare in code, apply +in CI, verify at runtime". The symmetry breaks at runtime (BG §2.5): there is NO runtime ``publish()`` — it would be actively harmful (immutable versions, catalog reflecting process starts, privilege escalation, no review). The runtime verb is :meth:`verify` (read-only drift check), the genuine mirror of ``resolve()``. -Verbs (§7.4): +Verbs (BG §2.5): * ``preview()`` — laptop. Renders the entry as it would appear. Writes nothing. - * ``export()`` — laptop. Compiles into donkey.yaml (§5.1). CI publishes on merge. + * ``export()`` — laptop. Compiles into donkey.yaml (provisioning-as-code). CI publishes on merge. * ``verify()`` — runtime, READ-ONLY. Fetches the published descriptor, introspects the live server, compares. Raises PublicationDrift on mismatch. """ @@ -69,28 +69,31 @@ class Publication: # ---- verb: preview (laptop) ------------------------------------------- async def preview(self, donkey: Any) -> str: - """Render the Exchange entry as it would appear (§7.4). Blocked until - descriptor derivation + Exchange render shape are verified (§7.9).""" + """Render the Exchange entry as it would appear (BG §2.5). Blocked until + descriptor derivation + Exchange render shape are verified (BG §2.5).""" raise _verify.blocked( - "descriptor derivation (§7.3) + Exchange entry render (§7.9). The " + "descriptor derivation (BG §2.5) + Exchange entry render (BG §2.5). The " "description-quality report (check_description_quality) is implemented " - "and should run here (§7.3.3)." + "and should run here (BG §2.5)." ) # ---- verb: export (laptop) -------------------------------------------- def export(self, path: str | Path | None = None) -> str: - """Compile into the donkey.yaml spec (§5.1). Lands with M4 (§9.1).""" - raise _verify.blocked("Publication.export() emits the M4 spec (§5.1, §9.1).") + """Compile into the donkey.yaml spec (provisioning-as-code). Lands with M4 (the build plan + phases).""" + raise _verify.blocked( + "Publication.export() emits the M4 spec (provisioning-as-code, the build plan phases)." + ) # ---- verb: verify (runtime, READ-ONLY) -------------------------------- async def verify(self, donkey: Any, *, raise_on_drift: bool = False) -> None: """Compare the live server against the published descriptor; raise - :class:`~donkey_kit.core.errors.PublicationDrift` on mismatch (§7.4). + :class:`~donkey_kit.core.errors.PublicationDrift` on mismatch (BG §2.5). Defaults to warn-and-continue — a drifted catalog must be loud but must not take down production traffic. Blocked until Exchange read + - introspection are verified (§7.9).""" + introspection are verified (BG §2.5).""" raise _verify.blocked( - "Exchange descriptor read + live introspection for verify() (§7.4, §7.9)." + "Exchange descriptor read + live introspection for verify() (BG §2.5)." ) @@ -100,7 +103,7 @@ async def verify(self, donkey: Any, *, raise_on_drift: bool = False) -> None: def check_description_quality( tools: list[tuple[str, str | None]], *, min_len: int = 12 ) -> list[DescriptionIssue]: - """Fail-worthy description problems (§7.3.3). Pure, implemented now. + """Fail-worthy description problems (BG §2.5). Pure, implemented now. A description missing, or equal to the identifier (after normalising underscores/case), or shorter than ``min_len`` is a FAILURE, not a warning — diff --git a/python/src/donkey_kit/simulator/app.py b/python/src/donkey_kit/simulator/app.py index ad583c61..b20a06fb 100644 --- a/python/src/donkey_kit/simulator/app.py +++ b/python/src/donkey_kit/simulator/app.py @@ -12,7 +12,7 @@ carries ``x-donkey-simulator: true`` so it can never be mistaken for a real gateway (BG §1.4, non-negotiable). -Framework isolation (§1.1): ``starlette`` is imported **lazily inside** +Framework isolation (the layered architecture): ``starlette`` is imported **lazily inside** :func:`build_app`, never at module top, so ``import donkey_kit.simulator`` stays green under the base-only CI job (``[dev]`` only, no ``[local]`` extra). The public return type is a framework-free ASGI ``Protocol`` so no ``starlette`` @@ -53,7 +53,7 @@ # The ONE fixed, documented, simulator-owned rejection-selection trigger (the # configurable --scenario engine is a linked follow-up, not #187). A request # whose model id is "donkey-sim/" is served that rejection shape. This is -# a simulator control surface only — NEVER a real Omni Gateway behaviour (§0.3). +# a simulator control surface only — NEVER a real Omni Gateway behaviour (verification discipline). SIM_MODEL_PREFIX = "donkey-sim/" # The budget window the live proxy emits on a happy-path `200` (with the @@ -168,7 +168,7 @@ async def dispatch(self, request: Any) -> Any: return await self._responses(request) if request.method == "GET" and path.endswith("/models"): # No catalog endpoint is verified; mirror the captured 404 rather - # than fabricate a model list (§0.3). + # than fabricate a model list (verification discipline). return self._response(load("models-notfound")) # Unknown route: an honest, honesty-stamped 404. return self._response(load("models-notfound")) @@ -213,7 +213,7 @@ def _happy(self, payload: Any, ratelimit_header: dict[str, str]) -> Any: # The captured stream sample is a single, truncated `response.created` # event — a real capture, NOT a complete SSE stream ending in # `data: [DONE]`. It is replayed verbatim rather than fabricating the - # remaining events (§0.3: never invent gateway output); a complete + # remaining events (verification discipline: never invent gateway output); a complete # SSE capture is a follow-up. It still carries the budget window prose # header, like any happy path. return self._response(load("stream"), extra=ratelimit_header) diff --git a/python/src/donkey_kit/simulator/fixtures.py b/python/src/donkey_kit/simulator/fixtures.py index e8ebf855..751766bd 100644 --- a/python/src/donkey_kit/simulator/fixtures.py +++ b/python/src/donkey_kit/simulator/fixtures.py @@ -31,7 +31,8 @@ from pathlib import Path # Verified header names, imported from the framework-free core so the simulator -# and the client parse the identical strings (upward import, allowed under §1.1). +# and the client parse the identical strings (upward import, allowed under the layered +# architecture). # RATELIMIT_HEADER is the prose budget header the live 200/403 carries (#352/#353). from ..core.budget import ( LIMIT_HEADER, @@ -248,7 +249,7 @@ def _source_bytes(directory: str, name: str) -> bytes | None: def fixture_bytes(directory: str, name: str) -> bytes: """Return the raw bytes of one fixture file, packaged copy first then the source-checkout fallback. Raises a clear, actionable error if neither is - found rather than serving a fabricated body (§0.3).""" + found rather than serving a fabricated body (verification discipline).""" data = _packaged_bytes(directory, name) if data is not None: return data diff --git a/python/src/donkey_kit/simulator/inject.py b/python/src/donkey_kit/simulator/inject.py index bf83d274..0cc2c8f8 100644 --- a/python/src/donkey_kit/simulator/inject.py +++ b/python/src/donkey_kit/simulator/inject.py @@ -6,9 +6,9 @@ branch of an agent with **no network and no server**. It replays the **same captured fixtures** ``core.errors.classify()`` and the simulator app are tested against, so the injected refusal is the real shape — and it asserts that -round-trip at enter time ("same files, both fail together", BG §1.4/§1.5). +round-trip at enter time ("same files, both fail together", BG §1.4 / BG §1.5). -Framework isolation (§1.1): this module imports only ``httpx``, the +Framework isolation (the layered architecture): this module imports only ``httpx``, the framework-free ``core.errors`` taxonomy, and the sibling fixtures loader — never a web framework — so it is safe on the base import path. It is a dev-only simulator sibling, kept out of the five production layers by the import-linter @@ -64,7 +64,7 @@ class _SwappableClient(Protocol): - """The transport-swap seam both Donkey HTTP clients expose (§2.3, BG §1.1). + """The transport-swap seam both Donkey HTTP clients expose (BG §1.1). Typed loosely on purpose — the sync and async clients carry different ``httpx`` transport types, and :class:`_FixtureTransport` satisfies both.""" @@ -148,7 +148,7 @@ async def handle_async_request(self, request: httpx.Request) -> httpx.Response: def _resolve(error: type[DonkeyError]) -> Fixture: """Resolve the requested exception type to the fixture that classify() maps back to it, asserting that round-trip. Raises ``TypeError`` for a non-error - type and ``ValueError`` for an unmapped one (§0.3: no silent miss).""" + type and ``ValueError`` for an unmapped one (verification discipline: no silent miss).""" if not (isinstance(error, type) and issubclass(error, DonkeyError)): raise TypeError( f"simulate() expects a DonkeyError subclass, got {error!r}. " diff --git a/python/src/donkey_kit/simulator/scenarios.py b/python/src/donkey_kit/simulator/scenarios.py index ca8a0e5e..27c83aff 100644 --- a/python/src/donkey_kit/simulator/scenarios.py +++ b/python/src/donkey_kit/simulator/scenarios.py @@ -15,14 +15,14 @@ live counter and the real milliseconds left in the window — until the window rolls over and the budget resets. -**Verification discipline (§0.3, #352/#353/#354).** The live happy-path ``200`` +**Verification discipline (#352/#353/#354).** The live happy-path ``200`` carries the budget window as the prose ``x-llm-proxy-ratelimit`` header, *not* the numeric ``x-token-*`` trio (those appear only on the ``429``). So the ``budget`` scenario emits the prose header on the ``200`` (rendered from its live counter) and the numeric ``x-token-*`` only on the ``429`` boundary. It never fabricates a header shape the gateway does not emit. -Framework isolation (§1.1): this module imports only the stdlib and the +Framework isolation (the layered architecture): this module imports only the stdlib and the framework-free sibling :mod:`~donkey_kit.simulator.fixtures` — never a web framework — so ``import donkey_kit.simulator.scenarios`` stays green under the base-only CI job. @@ -171,7 +171,7 @@ def on_call(self, text: str) -> ScenarioHit | None: def _default_cost() -> int: """Per-call token cost, defaulting to the happy-path fixture's own reported - ``total_tokens`` — an honest number, not a fabricated one (§0.3).""" + ``total_tokens`` — an honest number, not a fabricated one (verification discipline).""" try: body = json.loads(load("success").body) total = int(body.get("usage", {}).get("total_tokens", 0)) diff --git a/python/src/donkey_kit/tools/__init__.py b/python/src/donkey_kit/tools/__init__.py index 44590da5..9fea073b 100644 --- a/python/src/donkey_kit/tools/__init__.py +++ b/python/src/donkey_kit/tools/__init__.py @@ -1,4 +1,4 @@ -"""tools/ — governed tool access: MCP session mgmt + tool filtering (§4).""" +"""tools/ — governed tool access: MCP session mgmt + tool filtering (BG §2.7).""" from .filter import ToolDescriptor, ToolFilter, resolve_collisions from .session import ToolSet diff --git a/python/src/donkey_kit/tools/filter.py b/python/src/donkey_kit/tools/filter.py index da85a14d..565ae27c 100644 --- a/python/src/donkey_kit/tools/filter.py +++ b/python/src/donkey_kit/tools/filter.py @@ -1,4 +1,4 @@ -"""Tool filtering + name-collision resolution (§4.3). +"""Tool filtering + name-collision resolution (BG §2.7). Pure logic, no framework or network dependency, so it is fully implemented and unit-testable now. @@ -13,7 +13,7 @@ @dataclass(frozen=True) class ToolDescriptor: """A tool as seen before binding: which server it came from, its name, and - metadata used for filtering and token-cost logging (§4.3).""" + metadata used for filtering and token-cost logging (BG §2.7).""" server: str # short server name name: str @@ -41,11 +41,11 @@ def resolve_collisions( tools: list[ToolDescriptor], ) -> tuple[dict[str, ToolDescriptor], dict[str, str]]: """Resolve name collisions across servers by prefixing with the server's - short name (§4.3): ``hr__get_employee``. + short name (BG §2.7): ``hr__get_employee``. Returns ``(exposed_name -> descriptor, exposed_name -> original_name)`` so a developer can debug why the model called ``hr__get_employee`` via - ``ToolSet.name_map`` (§4.3). + ``ToolSet.name_map`` (BG §2.7). """ counts: dict[str, int] = {} diff --git a/python/src/donkey_kit/tools/session.py b/python/src/donkey_kit/tools/session.py index 4c569e71..7504594d 100644 --- a/python/src/donkey_kit/tools/session.py +++ b/python/src/donkey_kit/tools/session.py @@ -1,10 +1,10 @@ -"""MCP session management + the ``ToolSet`` facade (§4.3, §4.4). +"""MCP session management + the ``ToolSet`` facade (BG §2.7). ``ToolSet`` wraps N ``McpServerHandle``s. Filtering and collision resolution are pure and implemented here. The per-framework binding methods return each -framework's NATIVE tool type (§4.4) and are gated on M0 verification of the MCP +framework's NATIVE tool type (BG §2.7) and are gated on M0 verification of the MCP binding class names (docs/verified-apis.md §9) — connections open on first tool -use, never in ``discover()`` (§4.3). +use, never in ``discover()`` (BG §2.7). """ from __future__ import annotations @@ -34,7 +34,7 @@ def filter( ) -> ToolSet: """Return a filtered view. Enterprise MCP servers can expose dozens of tools; handing 60 descriptors to a model degrades it and inflates token - cost (§4.3).""" + cost (BG §2.7).""" self._filter = ToolFilter( allow=frozenset(allow) if allow is not None else None, @@ -46,7 +46,7 @@ def filter( @property def name_map(self) -> dict[str, str]: """``exposed_name -> original_name`` so a developer can debug why the - model called ``hr__get_employee`` (§4.3).""" + model called ``hr__get_employee`` (BG §2.7).""" exposed, name_map = resolve_collisions(self._descriptors()) return name_map @@ -71,7 +71,8 @@ def _descriptors(self) -> list[ToolDescriptor]: _log.debug("ToolSet exposes %d of %d tool descriptors", len(kept), len(descs)) return kept - # ---- per-framework binding (native types §4.4, gated on §0.3 §9) ------ + # ---- per-framework binding (native types BG §2.7, gated on verification discipline, + # docs/verified-apis.md §9) ------ def langgraph(self) -> list[object]: # -> list[BaseTool] raise _verify.blocked( "langchain_mcp_adapters.client.MultiServerMCPClient binding " @@ -79,22 +80,32 @@ def langgraph(self) -> list[object]: # -> list[BaseTool] ) def adk(self) -> list[object]: # -> list[McpToolset] - raise _verify.blocked("ADK McpToolset / StreamableHTTPConnectionParams (§9).") + raise _verify.blocked( + "ADK McpToolset / StreamableHTTPConnectionParams (docs/verified-apis.md §9)." + ) def strands(self) -> list[object]: # -> list[MCPClient] - raise _verify.blocked("Strands MCPClient(streamablehttp_client(...)) (§9).") + raise _verify.blocked( + "Strands MCPClient(streamablehttp_client(...)) (docs/verified-apis.md §9)." + ) def llamaindex(self) -> list[object]: # -> list[FunctionTool] - raise _verify.blocked("LlamaIndex BasicMCPClient + McpToolSpec (§9).") + raise _verify.blocked("LlamaIndex BasicMCPClient + McpToolSpec (docs/verified-apis.md §9).") def openai(self) -> list[object]: # -> list[agents.mcp.MCPServer] - raise _verify.blocked("OpenAI Agents SDK agents.mcp.MCPServerStreamableHttp (§9).") + raise _verify.blocked( + "OpenAI Agents SDK agents.mcp.MCPServerStreamableHttp (docs/verified-apis.md §9)." + ) def anthropic(self) -> list[object]: # -> list[MCP tool defs] - raise _verify.blocked("Anthropic SDK MCP tool / mcp_servers binding (§9).") + raise _verify.blocked( + "Anthropic SDK MCP tool / mcp_servers binding (docs/verified-apis.md §9)." + ) def crewai(self) -> list[object]: # -> list[crewai BaseTool] - raise _verify.blocked("CrewAI crewai_tools.MCPServerAdapter binding (§9).") + raise _verify.blocked( + "CrewAI crewai_tools.MCPServerAdapter binding (docs/verified-apis.md §9)." + ) def agent_framework(self) -> list[object]: - raise _verify.blocked("Agent Framework MCP client/tool class (§9).") + raise _verify.blocked("Agent Framework MCP client/tool class (docs/verified-apis.md §9).") diff --git a/python/tests/conformance/suite.py b/python/tests/conformance/suite.py index 3093caac..af7ed9d1 100644 --- a/python/tests/conformance/suite.py +++ b/python/tests/conformance/suite.py @@ -1,16 +1,16 @@ -"""The adapter conformance kit — the most important test asset (§8.1). +"""The adapter conformance kit — the most important test asset. ONE suite, defined once. Its blocking scope is the conformance-tested roster (BG §1.8, #197): LangGraph and the raw client. A framework is "supported" only when it passes all of it, or records a documented, asserted exemption in -``KNOWN_LIMITATIONS`` (§8.1) — never a silent skip. The other seven frameworks +``KNOWN_LIMITATIONS`` (the conformance kit) — never a silent skip. The other seven frameworks are supported at ``connection_kwargs()`` only and are not run here; a demoted framework rejoins with its own conformance run when demand promotes it (#223/#244). The customer-facing pytest plugin (``donkey_kit.conformance``, #191) is the shipped deliverable this internal matrix backstops. -The scenario bodies are wired in M1+ against captured contract fixtures (§8.2) -and the local gateway (§8.3). This module fixes the scenario list and the +The scenario bodies are wired in M1+ against captured contract fixtures (BG §1.5) +and the local gateway (BG §1.4). This module fixes the scenario list and the exemption table now so the kit exists before the second adapter is built (working instruction #5). """ @@ -38,17 +38,17 @@ "asset_type_detection", # After a governed 200, donkey.last_call carries the gateway's own identity # (request_id / api_instance_id / environment_id) for the call just made - # (§3, #362). Mirrors correlation_id_propagated: the adapters that route + # (BG §1.1, #362). Mirrors correlation_id_propagated: the adapters that route # outside our transport cannot observe it and record an asserted exemption # below rather than a bare None. "gateway_identity_observed", ] -# Documented, ASSERTED exemptions — published in the README (§8.1). A framework +# Documented, ASSERTED exemptions — published in the README (the conformance kit). A framework # that cannot satisfy a scenario records WHY here rather than skipping silently. _LITELLM_TRANSPORT_EXEMPTION = ( "LiteLLM owns the transport; we cannot inject our httpx client, so the " - "correlation ID is per-client, not per-run (§3.3). A LiteLLM custom " + "correlation ID is per-client, not per-run (BG §1.8). A LiteLLM custom " "logger callback may later recover trace correlation." ) @@ -61,7 +61,7 @@ _LITELLM_LAST_CALL_EXEMPTION = ( "LiteLLM owns the transport; no response reaches our _on_response, so " "donkey.last_call cannot observe the gateway identity of the call and " - "reports UNAVAILABLE (#362, same cause as correlation_id_propagated §3.3)." + "reports UNAVAILABLE (#362, same cause as correlation_id_propagated BG §1.8)." ) _DEFAULT_HEADERS_LAST_CALL_EXEMPTION = ( "The adapter is handed only default_headers, never our httpx client, so no " @@ -70,7 +70,7 @@ ) KNOWN_LIMITATIONS: dict[str, dict[str, str]] = { - # ADK and CrewAI both reach models through LiteLLM (§3.3). + # ADK and CrewAI both reach models through LiteLLM (BG §1.8). "adk": { "correlation_id_propagated": _LITELLM_TRANSPORT_EXEMPTION, "gateway_identity_observed": _LITELLM_LAST_CALL_EXEMPTION, @@ -80,7 +80,7 @@ "gateway_identity_observed": _LITELLM_LAST_CALL_EXEMPTION, }, # LlamaIndex and MS Agent Framework get only default_headers, no httpx client - # (§3.3), so they cannot observe last_call either — but they CAN propagate the + # (BG §1.8), so they cannot observe last_call either — but they CAN propagate the # correlation id through those headers, so that scenario is not exempt for them. "llamaindex": {"gateway_identity_observed": _DEFAULT_HEADERS_LAST_CALL_EXEMPTION}, "agent_framework": { diff --git a/python/tests/conformance/test_budget_pacing_e2e.py b/python/tests/conformance/test_budget_pacing_e2e.py index d995ebcc..64fb196d 100644 --- a/python/tests/conformance/test_budget_pacing_e2e.py +++ b/python/tests/conformance/test_budget_pacing_e2e.py @@ -1,4 +1,4 @@ -"""End-to-end budget pacing against the local gateway simulator (§1.3, #295). +"""End-to-end budget pacing against the local gateway simulator (BG §1.3, #295). ``local_gateway``-marked, so deselected by default and run only via ``pytest -m local_gateway`` (with the ``[local]`` + ``[llm]`` extras installed). diff --git a/python/tests/conformance/test_gateway_fixture.py b/python/tests/conformance/test_gateway_fixture.py new file mode 100644 index 00000000..de536e6d --- /dev/null +++ b/python/tests/conformance/test_gateway_fixture.py @@ -0,0 +1,92 @@ +"""End-to-end tests for the out-of-process ``gateway`` fixture (#278, `BG §1.5`). + +These boot the real simulator on an ephemeral port and drive it with a **stock +``httpx`` client** — deliberately not through ``donkey_kit`` — because the whole +point of the fixture is testing a subject that does not import the SDK. They are +marked ``local_gateway`` (deselected by default) and skip cleanly without the +``[local]`` extra. +""" + +from __future__ import annotations + +import httpx +import pytest + +from donkey_kit.conformance.gateway import start_gateway +from donkey_kit.simulator.app import SIM_MODEL_PREFIX, SIMULATOR_HEADER + +pytestmark = pytest.mark.local_gateway + +# Skip the whole module (rather than error) when the server extra is absent, even +# under `pytest -m local_gateway`. +pytest.importorskip("uvicorn") +pytest.importorskip("starlette") + + +def test_url_is_reachable_by_a_stock_client_and_is_honesty_stamped(gateway: object) -> None: + # A process that never imports donkey_kit points a plain httpx client at the + # URL and hits the simulator exactly like the real proxy. + url = gateway.url # type: ignore[attr-defined] + assert url.startswith("http://127.0.0.1:") + resp = httpx.post( + f"{url}/responses", + json={"model": "gpt-5.1", "input": "hello"}, + headers={"client_id": "pub", "client_secret": "shh"}, + ) + assert resp.status_code == 200 + assert resp.headers[SIMULATOR_HEADER] == "true" + + +def test_requests_received_counts_and_redacts_the_secret(gateway: object) -> None: + url = gateway.url # type: ignore[attr-defined] + for _ in range(3): + httpx.post( + f"{url}/responses", + json={"model": "gpt-5.1", "input": "hi"}, + headers={"client_id": "pub", "client_secret": "super-secret"}, + ) + assert gateway.requests_received == 3 # type: ignore[attr-defined] + recorded = gateway.requests # type: ignore[attr-defined] + assert all(r.method == "POST" and r.path == "/responses" for r in recorded) + # The secret never survives into the spy, even over the wire. + assert recorded[0].headers["client_id"] == "pub" + assert recorded[0].headers["client_secret"] == "***" + + +def test_scenarios_are_configurable_per_test(gateway: object) -> None: + # Per-test #188 config: arm a scenario live, no restart, no per-process flag. + gateway.set_scenarios("pii_block:every=1") # type: ignore[attr-defined] + url = gateway.url # type: ignore[attr-defined] + resp = httpx.post(f"{url}/responses", json={"model": "gpt-5.1", "input": "hi"}) + assert resp.status_code == 403 # pii-detected fixture + assert resp.headers[SIMULATOR_HEADER] == "true" + # The request is still recorded even though it was rejected. + assert gateway.requests_received == 1 # type: ignore[attr-defined] + + +def test_reset_clears_the_request_log(gateway: object) -> None: + url = gateway.url # type: ignore[attr-defined] + httpx.post(f"{url}/responses", json={"model": "gpt-5.1", "input": "hi"}) + assert gateway.requests_received == 1 # type: ignore[attr-defined] + gateway.reset() # type: ignore[attr-defined] + assert gateway.requests_received == 0 # type: ignore[attr-defined] + + +def test_model_id_sentinel_forces_a_rejection_shape(gateway: object) -> None: + # The simulator's fixed sentinel works over the real port too (no scenario). + url = gateway.url # type: ignore[attr-defined] + resp = httpx.post( + f"{url}/responses", json={"model": f"{SIM_MODEL_PREFIX}pii-detected", "input": "hi"} + ) + assert resp.status_code == 403 + + +def test_two_gateways_bind_distinct_ephemeral_ports() -> None: + # Ephemeral binding is what lets parallel `pytest -n` runs never collide. + a = start_gateway() + b = start_gateway() + try: + assert a.url != b.url + finally: + a.close() + b.close() diff --git a/python/tests/conformance/test_gateway_identity_exemption.py b/python/tests/conformance/test_gateway_identity_exemption.py index c1bc52c5..466696d3 100644 --- a/python/tests/conformance/test_gateway_identity_exemption.py +++ b/python/tests/conformance/test_gateway_identity_exemption.py @@ -1,4 +1,5 @@ -"""The gateway-identity conformance exemption is ASSERTED, never skipped (#362, §8.1). +"""The gateway-identity conformance exemption is ASSERTED, never skipped (#362, the +conformance kit). ``donkey.last_call`` is populated by the shared transport's ``_on_response``, so an adapter that does not route through our httpx client structurally cannot observe diff --git a/python/tests/fixtures/a2d/README.md b/python/tests/fixtures/a2d/README.md index c537a283..fb798c03 100644 --- a/python/tests/fixtures/a2d/README.md +++ b/python/tests/fixtures/a2d/README.md @@ -3,7 +3,7 @@ These JSON files are **faithful, trimmed captures** of real responses from the `mcp-a2d` MCP server (host `www.a2d-ai.com`), taken 2026-08-28. They exist so the registry value types (`McpServerHandle`, `ToolDescriptor`, `AssetRef`) can be -validated against real-world shapes without a live call (§8.2, fixture-driven). +validated against real-world shapes without a live call (BG §1.5, fixture-driven). ## What these are — and are NOT diff --git a/python/tests/fixtures/anypoint/README.md b/python/tests/fixtures/anypoint/README.md index 0b2de916..806bc352 100644 --- a/python/tests/fixtures/anypoint/README.md +++ b/python/tests/fixtures/anypoint/README.md @@ -26,7 +26,7 @@ contracts the SDK targets. ## What these verify (see docs/verified-apis.md §5/§6/§7) -- Governed-state join (§6.1.3): API Manager instances are listable per +- Governed-state join: API Manager instances are listable per environment and carry deployment + policy state — no reverse-engineering needed. `test_governed_state_shapes.py` proves the domain models represent this real data. diff --git a/python/tests/fixtures/anypoint/llm_proxy/README.md b/python/tests/fixtures/anypoint/llm_proxy/README.md index 4ac2d99a..bc836ae7 100644 --- a/python/tests/fixtures/anypoint/llm_proxy/README.md +++ b/python/tests/fixtures/anypoint/llm_proxy/README.md @@ -1,4 +1,4 @@ -# LLM proxy — LIVE captures from a real governed ingress proxy (M0, §2/§3/§4) +# LLM proxy — LIVE captures from a real governed ingress proxy (M0, docs/verified-apis.md §2/§3/§4) Captured 2026-08-28 from a real deployed Agent Network LLM proxy in the sandbox org `82a0453b-22e6-430d-bbf4-35b989d043dc`, env **Sandbox** @@ -7,7 +7,7 @@ API Manager instance `21133858`, deployed to the `agent-network-ingress-gw` Flex Gateway (v1.13.2), upstream `https://api.openai.com/v1/`. These are the **direct data-plane contract** the SDK's Pillar-1 model access -targets — the first live confirmation of §§2–4 (previously UNVERIFIED). See +targets — the first live confirmation of docs/verified-apis.md §§2–4 (previously UNVERIFIED). See `docs/verified-apis.md` §2/§3/§4/§6. ## Files @@ -58,7 +58,7 @@ Both policies above were applied to instance `21133858` only to capture these rejection contracts and were removed afterward. - `responses.stream.sample.sse` / `responses.stream.headers.txt` — `stream:true` → HTTP 200, `content-type: text/event-stream`, chunked SSE (`event: - response.created`, `response.in_progress`, …). Confirms §2 streaming. + response.created`, `response.in_progress`, …). Confirms docs/verified-apis.md §2 streaming. - `models.notfound.headers.txt` — `GET /openai-sdk/models` → HTTP 404 (`Request passed through without model-based routing`). The proxy exposes **no catalog endpoint**; it only routes requests carrying a `model` in the body. diff --git a/python/tests/fixtures/rejections/README.md b/python/tests/fixtures/rejections/README.md index 55d6f3fa..23072a49 100644 --- a/python/tests/fixtures/rejections/README.md +++ b/python/tests/fixtures/rejections/README.md @@ -1,4 +1,4 @@ -# Rejection contracts — the eight documented shapes (#181, #289, docs §4) +# Rejection contracts — the eight documented shapes (#181, #289, docs/verified-apis.md §4) The canonical index of the gateway rejection shapes `core/errors.classify()` discriminates, one row per shape. These fixtures are **shared with the local @@ -7,14 +7,14 @@ files, so contract drift fails `classify()` and the simulator at once. Keep them byte-faithful and parser-compatible (see `../anypoint/llm_proxy/README.md` for the `.headers.txt` / `.body.json` / `.body.empty` convention). -## Provenance & verification status (§0.3) +## Provenance & verification status (verification discipline) The build guide describes these contracts as "public in Omni Gateway v1.11–v1.13", but **no public docs URL for the rejection contracts is recorded anywhere in this repo**, and `v1.11–v1.13` has no concrete in-repo source (the -only live `v1.13` is the Flex Gateway *runtime* v1.13.2; docs §4 cites policy +only live `v1.13` is the Flex Gateway *runtime* v1.13.2; docs/verified-apis.md §4 cites policy *asset* versions). Writing a docs URL or a "docs version" into these fixtures -would be inventing an endpoint, which §0.3 forbids. So each row below records +would be inventing an endpoint, which the verification discipline forbids. So each row below records only what is **defensible**, and the "cite the public docs URL + version" portion of the #181 acceptance criteria is left open under **#253** (`verify: re-confirm the v1.11-v1.13 rejection contracts against current docs and a diff --git a/python/tests/sandbox/README.md b/python/tests/sandbox/README.md index ca0e6a54..ba358b5d 100644 --- a/python/tests/sandbox/README.md +++ b/python/tests/sandbox/README.md @@ -35,7 +35,7 @@ minted for it. The manifest holds **no secrets** — only a `base_url`, the rout ```toml [[proxy]] key = "openai-model-routing" -base_url = "https://:8081/" # ingress WITHOUT /v1 (§2) +base_url = "https://:8081/" # ingress WITHOUT /v1 (docs/verified-apis.md §2) model = "gpt-5-mini" client_id_env = "DDK_SANDBOX_OPENAI_ROUTING_CLIENT_ID" client_secret_env = "DDK_SANDBOX_OPENAI_ROUTING_CLIENT_SECRET" diff --git a/python/tests/sandbox/proxies.toml.example b/python/tests/sandbox/proxies.toml.example index 2ac21375..b3ab83cd 100644 --- a/python/tests/sandbox/proxies.toml.example +++ b/python/tests/sandbox/proxies.toml.example @@ -8,7 +8,7 @@ # pair per proxy with the `ddk-request-llm-proxy-access` skill, then export the # two env vars each entry names. # -# `base_url` is the proxy ingress WITHOUT a trailing `/v1` (§2 — the ingress is +# `base_url` is the proxy ingress WITHOUT a trailing `/v1` (docs/verified-apis.md §2 — the ingress is # verbatim; the OpenAI SDK appends `/responses` itself). Fill it from the real # host + port 8081 + base path shown for that proxy in the provisioning repo. # Values here are PLACEHOLDERS — replace every `REPLACE-ME` before running. diff --git a/python/tests/sandbox/test_proxy_contracts.py b/python/tests/sandbox/test_proxy_contracts.py index 00cab94d..78381b1f 100644 --- a/python/tests/sandbox/test_proxy_contracts.py +++ b/python/tests/sandbox/test_proxy_contracts.py @@ -26,7 +26,7 @@ def test_openai_routing_happy_path_matches_fixture( open_proxy: Callable[[str], Donkey], model_for: Callable[[str], str], ) -> None: - """§2/§3: the live model-based-routing proxy returns the OpenAI Responses + """docs/verified-apis.md §2/§3: the live model-based-routing proxy returns the OpenAI Responses object verbatim, with the ``usage`` token block cost attribution reads — the same shape ``responses.success.body.json`` pins offline.""" donkey = open_proxy("openai-model-routing") @@ -44,8 +44,8 @@ def test_openai_routing_happy_path_matches_fixture( assert usage is not None assert usage.total_tokens == usage.input_tokens + usage.output_tokens - # §3: the gateway records what it did on this call — the success-path twin of - # the ids classify() reads off a rejection. + # docs/verified-apis.md §3: the gateway records what it did on this call — the success-path twin + # of the ids classify() reads off a rejection. assert donkey.last_call.served_model assert donkey.last_call.total_tokens == usage.total_tokens @@ -58,8 +58,8 @@ def test_injection_guard_rejection_classifies_and_captures( """#253: the Regex Prompt Guard proxy is the live source for the injection rejection body ``classify()`` currently types on assumption. - The exact regex that trips the guard is UNVERIFIED (§0.3), so if the prompt - does not trip it the call succeeds and we skip with a pointer rather than + The exact regex that trips the guard is UNVERIFIED (verification discipline), so if the + prompt does not trip it the call succeeds and we skip with a pointer rather than assert an invented trigger. When it *does* reject, we assert the contract (a typed ``DonkeyError`` with non-empty remediation) and print the raw body + headers so a maintainer can capture them into @@ -76,7 +76,7 @@ def test_injection_guard_rejection_classifies_and_captures( except openai.APIStatusError as exc: err = classify(exc.response) assert isinstance(err, DonkeyError) - assert err.remediation, "every PolicyViolation must carry a remediation (§2.4)" + assert err.remediation, "every PolicyViolation must carry a remediation (BG §1.2)" # Capture aid for #253 — surfaced with `pytest -m sandbox -s`. with capsys.disabled(): @@ -87,6 +87,7 @@ def test_injection_guard_rejection_classifies_and_captures( pytest.skip( "injection prompt did not trip the Regex Prompt Guard — the trigger regex " - "is unverified (§0.3, #253). Refine the prompt against the deployed policy, " + "is unverified (verification discipline, #253). Refine the prompt against the deployed " + "policy, " "then capture the rejection body into tests/fixtures/anypoint/llm_proxy/." ) diff --git a/python/tests/unit/test_adapter_ergonomics.py b/python/tests/unit/test_adapter_ergonomics.py index a0f5ecbd..7935c9d9 100644 --- a/python/tests/unit/test_adapter_ergonomics.py +++ b/python/tests/unit/test_adapter_ergonomics.py @@ -1,4 +1,4 @@ -"""The two additive adapter ergonomics (§3.1/§3.3), alongside the existing +"""The two additive adapter ergonomics (BG §1.8), alongside the existing ``donkey..()`` methods: 1. ``connection_kwargs()`` — governed kwargs you spread into the framework's @@ -14,13 +14,19 @@ from __future__ import annotations +import importlib +import sys +import types +import warnings +from typing import Any, NamedTuple + import pytest from donkey_kit.core.config import DonkeyConfig from donkey_kit.core.errors import ConfigError from donkey_kit.core.transport import DonkeyAsyncClient, build_http_client from donkey_kit.integrations import _base -from donkey_kit.integrations._base import default_adapter +from donkey_kit.integrations._base import Adapter, default_adapter from donkey_kit.integrations.langgraph import LangGraphAdapter @@ -44,7 +50,8 @@ def test_connection_kwargs_carry_governed_values() -> None: assert "client_secret" in kw["default_headers"] assert kw["max_retries"] == 0 # we retry in transport, not the framework assert kw["http_async_client"] is not None # our shared, hooked client - assert kw["use_responses_api"] is True # verified /responses endpoint (docs §4) + # verified /responses endpoint (docs/verified-apis.md §4) + assert kw["use_responses_api"] is True # No model id — the caller supplies that: ChatOpenAI(model=…, **kw) assert "model" not in kw @@ -133,21 +140,21 @@ def test_anthropic_connection_kwargs_carry_proxy_and_shared_client() -> None: from donkey_kit.integrations.anthropic import AnthropicAdapter with warnings.catch_warnings(): - # The Anthropic-native proxy route is an open verification item (§0.3); + # The Anthropic-native proxy route is an open verification item (verification discipline); # connection_kwargs() warns once about it. Not what this test asserts. warnings.simplefilter("ignore") kw = AnthropicAdapter(_cfg(), _http()).connection_kwargs() assert kw["base_url"] == "https://proxy" assert "client_id" in kw["default_headers"] assert kw["http_client"] is not None - assert kw["max_retries"] == 0 # we retry in transport (§2.3) + assert kw["max_retries"] == 0 # we retry in transport (BG §1.1) def test_crewai_connection_kwargs_use_litellm_extra_headers() -> None: from donkey_kit.integrations.crewai import CrewAIAdapter # crewai.LLM forwards to LiteLLM, which uses extra_headers and owns its own - # transport, so no shared http client is injected (§3.3 exemption §8.1). + # transport, so no shared http client is injected (BG §1.8 exemption; the conformance kit). kw = CrewAIAdapter(_cfg(), _http()).connection_kwargs() assert kw["base_url"] == "https://proxy" assert "client_id" in kw["extra_headers"] @@ -166,19 +173,22 @@ def test_llamaindex_connection_kwargs_use_api_base_and_chat_flags() -> None: assert kw["is_function_calling_model"] is True -def test_openai_agents_governed_client_carries_proxy_config() -> None: - """The OpenAI Agents SDK wants a *pre-built* client, so this adapter has no - connection_kwargs() — its governed surface is a native AsyncOpenAI bound to - the proxy. Still supported at the connection level after the roster cut.""" +def test_openai_agents_connection_kwargs_carry_governed_client() -> None: + """The OpenAI Agents SDK wants a *pre-built* client, so unlike the + OpenAI-compatible adapters this one's connection_kwargs() returns a single + ``openai_client`` key holding a native AsyncOpenAI bound to the proxy — header + AND transport injection travel as one object (BG §1.8).""" openai = pytest.importorskip("openai") from donkey_kit.integrations.openai_agents import OpenAIAgentsAdapter - client = OpenAIAgentsAdapter(_cfg(), _http())._proxy_openai_client() + kw = OpenAIAgentsAdapter(_cfg(), _http()).connection_kwargs() + assert set(kw) == {"openai_client"} # not loose base_url/default_headers + client = kw["openai_client"] assert isinstance(client, openai.AsyncOpenAI) assert str(client.base_url) == "https://proxy" assert client.default_headers["client_id"] == "cid" - assert client.max_retries == 0 # we retry in transport (§2.3) + assert client.max_retries == 0 # we retry in transport (BG §1.1) def test_only_langgraph_is_conformance_tested() -> None: @@ -198,3 +208,143 @@ def test_only_langgraph_is_conformance_tested() -> None: "crewai", "llamaindex", } + + +# --- The two paths cannot drift (issue #33 AC) ------------------------------ +# +# Each framework exposes the SAME governed native object two ways: (a) the +# documented "eject" path — spread ``connection_kwargs()`` into the framework's +# own constructor yourself; (b) the module-level factory, which builds it for +# you. If the factory ever passed the native constructor different connection +# values than ``connection_kwargs()`` advertises, the docs' manual-equivalent +# block would silently lie. These tests pin the two paths together per framework. +# +# The check is offline for all eight: we replace each framework's native class +# with a spy that records the kwargs it is constructed with, then assert those +# kwargs carry exactly the governed values ``connection_kwargs()`` returns — no +# real framework install and no per-framework attribute introspection needed. + + +class _F(NamedTuple): + module: str # submodule under donkey_kit.integrations + factory: str # module-level factory function name + adapter_cls: str # adapter class name (for the shared default-adapter cache) + native_module: str # dotted module the factory imports its native class from + native_attr: str # native class attribute name to spy on + args: tuple[str, ...] # positional args the factory takes (model id, if any) + + +# Every adapter whose connection_kwargs() is a plain value dict spread straight +# into the native constructor. openai_agents is exercised separately below +# because its governed value is a freshly-built client object (BG §1.8). +_FACTORIES = [ + _F( + "langgraph", "chat_model", "LangGraphAdapter", "langchain_openai", "ChatOpenAI", ("gpt-4o",) + ), + _F("adk", "model", "ADKAdapter", "google.adk.models.lite_llm", "LiteLlm", ("gpt-4o",)), + _F("strands", "model", "StrandsAdapter", "strands.models.openai", "OpenAIModel", ("gpt-4o",)), + _F( + "agent_framework", + "chat_client", + "AgentFrameworkAdapter", + "agent_framework.openai", + "OpenAIChatClient", + ("gpt-4o",), + ), + _F("anthropic", "client", "AnthropicAdapter", "anthropic", "AsyncAnthropic", ()), + _F("crewai", "llm", "CrewAIAdapter", "crewai", "LLM", ("gpt-4o",)), + _F( + "llamaindex", + "llm", + "LlamaIndexAdapter", + "llama_index.llms.openai_like", + "OpenAILike", + ("gpt-4o",), + ), +] + + +def _install_native_stub( + monkeypatch: pytest.MonkeyPatch, dotted: str, attr: str +) -> dict[str, Any]: + """Replace ``.`` (the native class a factory imports lazily) + with a spy that records its constructor kwargs, registering stub modules for + any part of ``dotted`` that is not installed so the lazy ``from`` import + resolves offline. Returns the dict the spy populates.""" + captured: dict[str, Any] = {} + + class _Spy: + def __init__(self, **kwargs: Any) -> None: + captured.update(kwargs) + + parts = dotted.split(".") + for i in range(1, len(parts) + 1): + name = ".".join(parts[:i]) + if name not in sys.modules: + monkeypatch.setitem(sys.modules, name, types.ModuleType(name)) + if i > 1: # link submodule onto its parent so `from a.b import c` resolves + parent = sys.modules[".".join(parts[: i - 1])] + monkeypatch.setattr(parent, parts[i - 1], sys.modules[name], raising=False) + monkeypatch.setattr(sys.modules[dotted], attr, _Spy, raising=False) + return captured + + +def _set_proxy_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("DONKEY_LLM_PROXY_URL", "https://proxy") + monkeypatch.setenv("DONKEY_LLM_PROXY_CLIENT_ID", "cid") + monkeypatch.setenv("DONKEY_LLM_PROXY_CLIENT_SECRET", "csecret") + _base._DEFAULT_ADAPTERS.clear() + + +@pytest.mark.parametrize("f", _FACTORIES, ids=lambda f: f.module) +def test_factory_and_connection_kwargs_do_not_drift( + f: _F, monkeypatch: pytest.MonkeyPatch +) -> None: + _set_proxy_env(monkeypatch) + mod = importlib.import_module(f"donkey_kit.integrations.{f.module}") + factory = getattr(mod, f.factory) + adapter_cls: type[Adapter] = getattr(mod, f.adapter_cls) + + captured = _install_native_stub(monkeypatch, f.native_module, f.native_attr) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") # anthropic warns once about its unverified route + factory(*f.args) + + # The factory built the native object through the process-wide cached default + # adapter; read connection_kwargs() off that same instance, so the shared http + # client compares by identity and the header dicts by value. + adapter = default_adapter(adapter_cls) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + expected = adapter.connection_kwargs() + + # Every governed kwarg the eject path documents reached the native constructor + # with an identical value. (captured also holds model/model_id — not governed.) + assert {k: captured[k] for k in expected} == expected + + +def test_openai_agents_factory_and_connection_kwargs_do_not_drift( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """openai_agents' governed value is a pre-built AsyncOpenAI, freshly made on + each call, so the two paths yield *distinct* client objects — assert they + carry identical governed values rather than object identity (BG §1.8).""" + pytest.importorskip("openai") # _proxy_openai_client builds a real AsyncOpenAI + _set_proxy_env(monkeypatch) + + captured = _install_native_stub(monkeypatch, "agents", "OpenAIChatCompletionsModel") + from donkey_kit.integrations.openai_agents import OpenAIAgentsAdapter, model + + model("gpt-4o") + factory_client = captured["openai_client"] + accessor_client = default_adapter(OpenAIAgentsAdapter).connection_kwargs()["openai_client"] + + def _governed(c: Any) -> tuple[str, str, str, int]: + return ( + str(c.base_url), + c.default_headers["client_id"], + c.default_headers["client_secret"], + c.max_retries, + ) + + assert _governed(factory_client) == _governed(accessor_client) diff --git a/python/tests/unit/test_budget.py b/python/tests/unit/test_budget.py index 47128613..c6be5481 100644 --- a/python/tests/unit/test_budget.py +++ b/python/tests/unit/test_budget.py @@ -1,5 +1,5 @@ """Budget: the first-class object parsed from the LLM proxy's budget headers — -the numeric ``x-token-*`` trio (§1.3 / BG §1.3, #185) and the prose +the numeric ``x-token-*`` trio (BG §1.3, #185) and the prose ``x-llm-proxy-ratelimit`` fallback that is the only budget signal on a live 200 (#352). No network is touched — headers are attached to constructed ``httpx.Response`` objects, and the transport wiring is exercised via @@ -45,7 +45,7 @@ def test_budget_starts_unobserved() -> None: def test_reset_ms_converts_to_reset_at_to_the_second() -> None: - """AC: x-token-reset is milliseconds *to* reset (a delta, docs §4), so + """AC: x-token-reset is milliseconds *to* reset (a delta, docs/verified-apis.md §4), so reset_at = observed_at + that delta. Injected clock makes it exact.""" b = Budget() b.observe( @@ -170,7 +170,7 @@ def test_prose_fills_only_fields_the_numeric_trio_leaves_unset() -> None: def test_unparseable_prose_is_a_noop_and_never_raises() -> None: - """AC: an unparseable sentence is a no-op, per §0.3 — never a crash, never a + """AC: an unparseable sentence is a no-op, per verification discipline — never a crash, never a half-populated budget.""" b = Budget() b.observe(_resp(200, **{"x-llm-proxy-ratelimit": "rate limited, try later"}), now=_FIXED_NOW) diff --git a/python/tests/unit/test_budget_pacing.py b/python/tests/unit/test_budget_pacing.py index 99724c1c..86b42852 100644 --- a/python/tests/unit/test_budget_pacing.py +++ b/python/tests/unit/test_budget_pacing.py @@ -1,5 +1,5 @@ """Budget pacing: ``pace(reserve=)`` and ``wait_for_reset()`` — the recovery -half of piece 2 (§1.3 / BG §1.3, #186). No wall-clock sleeping: ``asyncio.sleep`` +half of piece 2 (BG §1.3, #186). No wall-clock sleeping: ``asyncio.sleep`` is monkeypatched to record its delay, and the clock is injected via ``now=`` the same way :meth:`Budget.observe` accepts it, so every assertion is deterministic.""" diff --git a/python/tests/unit/test_cli_doctor.py b/python/tests/unit/test_cli_doctor.py index d677dae7..3f3b6156 100644 --- a/python/tests/unit/test_cli_doctor.py +++ b/python/tests/unit/test_cli_doctor.py @@ -128,4 +128,4 @@ def _raise(_model: str, **_: object) -> object: assert result.exit_code == 1 out = _combined(result) assert 'pip install "donkey-kit[llm]"' in out - assert "blocked on verification" not in out # not a §0.3 gate + assert "blocked on verification" not in out # not a verification-discipline gate diff --git a/python/tests/unit/test_cli_init.py b/python/tests/unit/test_cli_init.py new file mode 100644 index 00000000..ed1a4b7b --- /dev/null +++ b/python/tests/unit/test_cli_init.py @@ -0,0 +1,197 @@ +"""``donkey init`` (#201): writes a commented ``.donkey-kit.toml`` from the +resolved config, names EVERY missing required field at once (reusing the +config-resolution all-at-once ``DonkeyConfig.validated`` report), never writes a +secret into the +committed file, and is idempotent — a bare re-run does not clobber an existing +file (``--force`` regenerates). + +Framework-free: needs only ``[dev]`` (typer). Runs in the base-only CI job. +""" + +from __future__ import annotations + +import sys + +import pytest +from typer.testing import CliRunner + +from donkey_kit.provisioning.cli import app + +if sys.version_info >= (3, 11): + import tomllib +else: # pragma: no cover - 3.10 backfill + import tomli as tomllib + +runner = CliRunner() + +# The three LLM-proxy secrets + the control-plane secret that must NEVER be +# written into the committed .donkey-kit.toml (they belong in env or the +# gitignored .donkey-kit.local.toml). +_SECRET_VALUES = { + "ANYPOINT_CLIENT_SECRET": "cp-super-secret", + "DONKEY_LLM_PROXY_CLIENT_SECRET": "proxy-super-secret", + "DONKEY_LLM_PROXY_KEY": "bearer-super-secret", +} + +_REQUIRED_ENV = ( + "ANYPOINT_CLIENT_ID", + "ANYPOINT_CLIENT_SECRET", + "ANYPOINT_ORG_ID", + "DONKEY_LLM_PROXY_URL", + "DONKEY_LLM_PROXY_CLIENT_ID", + "DONKEY_LLM_PROXY_CLIENT_SECRET", + "DONKEY_LLM_PROXY_KEY", + "DONKEY_APP_NAME", + "DONKEY_BUSINESS_GROUP", +) + + +def _combined(result: object) -> str: + text = getattr(result, "stdout", "") or "" + try: + text += result.stderr # type: ignore[attr-defined] + except (ValueError, AttributeError): + pass + return text + + +@pytest.fixture +def clean_env(monkeypatch: pytest.MonkeyPatch, tmp_path: object) -> None: + """A pristine cwd with none of the donkey env vars set, so each test opts + into exactly the values it wants.""" + monkeypatch.chdir(tmp_path) # type: ignore[arg-type] + for var in _REQUIRED_ENV: + monkeypatch.delenv(var, raising=False) + + +def test_init_writes_commented_toml_with_resolved_values( + monkeypatch: pytest.MonkeyPatch, clean_env: None, tmp_path: object +) -> None: + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "cp-client-id") + monkeypatch.setenv("ANYPOINT_ORG_ID", "org-123") + monkeypatch.setenv("ANYPOINT_ENV", "Production") + monkeypatch.setenv("DONKEY_LLM_PROXY_URL", "https://gw.example.internal") + monkeypatch.setenv("DONKEY_LLM_PROXY_CLIENT_ID", "proxy-client-id") + + result = runner.invoke(app, ["init"]) + assert result.exit_code == 0, _combined(result) + + from pathlib import Path + + written = Path(str(tmp_path)) / ".donkey-kit.toml" + assert written.is_file() + text = written.read_text() + # It is a *commented* file, not a bare dump. + assert text.lstrip().startswith("#") + # It parses as valid TOML with a [donkey] table carrying the resolved values. + table = tomllib.loads(text)["donkey"] + assert table["client_id"] == "cp-client-id" + assert table["org_id"] == "org-123" + assert table["environment"] == "Production" + assert table["llm_proxy_url"] == "https://gw.example.internal" + assert table["llm_proxy_client_id"] == "proxy-client-id" + + +def test_init_never_writes_secrets( + monkeypatch: pytest.MonkeyPatch, clean_env: None, tmp_path: object +) -> None: + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "cp-client-id") + monkeypatch.setenv("ANYPOINT_ORG_ID", "org-123") + monkeypatch.setenv("DONKEY_LLM_PROXY_URL", "https://gw.example.internal") + monkeypatch.setenv("DONKEY_LLM_PROXY_CLIENT_ID", "proxy-client-id") + for var, val in _SECRET_VALUES.items(): + monkeypatch.setenv(var, val) + + result = runner.invoke(app, ["init"]) + assert result.exit_code == 0, _combined(result) + + from pathlib import Path + + text = (Path(str(tmp_path)) / ".donkey-kit.toml").read_text() + # No secret VALUE ever lands in the committed file. + for val in _SECRET_VALUES.values(): + assert val not in text + # And the secret KEYS are not active keys in the parsed table. + table = tomllib.loads(text).get("donkey", {}) + assert "client_secret" not in table + assert "llm_proxy_client_secret" not in table + assert "llm_proxy_key" not in table + + +def test_init_lists_all_missing_required_fields_at_once( + monkeypatch: pytest.MonkeyPatch, clean_env: None +) -> None: + # Nothing configured: init still bootstraps a file (exit 0) but names every + # missing required field, not just the first (config-resolution all-at-once report). + result = runner.invoke(app, ["init"]) + assert result.exit_code == 0, _combined(result) + + out = _combined(result) + # Control-plane AND llm required fields both surfaced in one run. + assert "ANYPOINT_CLIENT_ID" in out + assert "ANYPOINT_CLIENT_SECRET" in out + assert "ANYPOINT_ORG_ID" in out + assert "DONKEY_LLM_PROXY_URL" in out + assert "DONKEY_LLM_PROXY_CLIENT_ID" in out + assert "DONKEY_LLM_PROXY_CLIENT_SECRET" in out + + +def test_init_is_idempotent_and_does_not_clobber( + monkeypatch: pytest.MonkeyPatch, clean_env: None, tmp_path: object +) -> None: + from pathlib import Path + + existing = Path(str(tmp_path)) / ".donkey-kit.toml" + existing.write_text('[donkey]\nclient_id = "hand-edited"\n') + + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "from-env") + + result = runner.invoke(app, ["init"]) + assert result.exit_code == 0, _combined(result) + # The hand-edited file is preserved untouched. + assert existing.read_text() == '[donkey]\nclient_id = "hand-edited"\n' + assert "exists" in _combined(result).lower() + + +def test_init_force_regenerates( + monkeypatch: pytest.MonkeyPatch, clean_env: None, tmp_path: object +) -> None: + from pathlib import Path + + existing = Path(str(tmp_path)) / ".donkey-kit.toml" + existing.write_text('[donkey]\nclient_id = "hand-edited"\n') + + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "from-env") + + result = runner.invoke(app, ["init", "--force"]) + assert result.exit_code == 0, _combined(result) + table = tomllib.loads(existing.read_text())["donkey"] + assert table["client_id"] == "from-env" + + +def test_init_writes_to_custom_config_path( + monkeypatch: pytest.MonkeyPatch, clean_env: None, tmp_path: object +) -> None: + from pathlib import Path + + target = Path(str(tmp_path)) / "nested" / "custom.toml" + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "cp-client-id") + + result = runner.invoke(app, ["--config", str(target), "init"]) + assert result.exit_code == 0, _combined(result) + assert target.is_file() + assert tomllib.loads(target.read_text())["donkey"]["client_id"] == "cp-client-id" + + +def test_init_json_output_is_machine_readable( + monkeypatch: pytest.MonkeyPatch, clean_env: None +) -> None: + import json + + monkeypatch.setenv("ANYPOINT_CLIENT_ID", "cp-client-id") + + result = runner.invoke(app, ["--json", "init"]) + assert result.exit_code == 0, _combined(result) + payload = json.loads(result.stdout) + assert payload["path"].endswith(".donkey-kit.toml") + assert "missing" in payload # the all-at-once list, machine-readable diff --git a/python/tests/unit/test_cli_mock.py b/python/tests/unit/test_cli_mock.py index 1698f0c4..8e0ff979 100644 --- a/python/tests/unit/test_cli_mock.py +++ b/python/tests/unit/test_cli_mock.py @@ -39,7 +39,7 @@ def _raise(**_: object) -> None: assert result.exit_code == 1 # install prompt, NOT the exit-3 verification block out = _combined(result) assert 'pip install "donkey-kit[local]"' in out - assert "blocked on verification" not in out # this is not a §0.3 gate + assert "blocked on verification" not in out # this is not a verification-discipline gate def test_mock_wires_host_and_port_through_to_serve(monkeypatch: pytest.MonkeyPatch) -> None: diff --git a/python/tests/unit/test_cli_test.py b/python/tests/unit/test_cli_test.py new file mode 100644 index 00000000..42c049ac --- /dev/null +++ b/python/tests/unit/test_cli_test.py @@ -0,0 +1,121 @@ +"""``donkey test`` (#201): a thin front end to ``pytest --donkey-conformance``. + +It is deliberately *not* a re-implementation of the conformance runner — it +shells out to pytest, forwards the customer's ``--agent MODULE:FACTORY`` plus +any trailing pytest args, and propagates pytest's exit code as its own. A +missing pytest (the ``[test]`` extra not installed) is a typed install prompt +with exit 1, never a stack trace and never a "blocked on verification". + +Framework-free (Surface 1): monkeypatches ``subprocess.run`` so no real pytest +runs. Base-only-safe. +""" + +from __future__ import annotations + +from typing import Any + +import pytest +from typer.testing import CliRunner + +from donkey_kit.provisioning import cli +from donkey_kit.provisioning.cli import app + +runner = CliRunner() + + +def _combined(result: object) -> str: + text = getattr(result, "stdout", "") or "" + try: + text += result.stderr # type: ignore[attr-defined] + except (ValueError, AttributeError): + pass + return text + + +class _FakeCompleted: + def __init__(self, returncode: int) -> None: + self.returncode = returncode + + +@pytest.fixture +def captured_run(monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]: + """Capture the argv passed to subprocess.run and return a controllable + exit code, without ever launching pytest.""" + captured: dict[str, Any] = {"argv": None, "returncode": 0} + + def fake_run(argv: list[str], *args: Any, **kwargs: Any) -> _FakeCompleted: + captured["argv"] = argv + return _FakeCompleted(captured["returncode"]) + + monkeypatch.setattr(cli.subprocess, "run", fake_run) + return captured + + +def test_test_invokes_pytest_with_conformance_flag(captured_run: dict[str, Any]) -> None: + result = runner.invoke(app, ["test"]) + assert result.exit_code == 0, _combined(result) + + argv = captured_run["argv"] + assert argv is not None + # It shells out to *this* interpreter's pytest, and passes the plugin flag. + assert argv[:3] == [__import__("sys").executable, "-m", "pytest"] + assert "--donkey-conformance" in argv + + +def test_test_forwards_agent_and_trailing_pytest_args(captured_run: dict[str, Any]) -> None: + result = runner.invoke( + app, + ["test", "--agent", "my.pkg:make_agent", "-k", "governance", "-x"], + ) + assert result.exit_code == 0, _combined(result) + + argv = captured_run["argv"] + assert "--donkey-conformance" in argv + # The agent factory is forwarded to the plugin. + assert "--agent" in argv + assert "my.pkg:make_agent" in argv + # Trailing pytest args pass straight through. + assert "-k" in argv and "governance" in argv + assert "-x" in argv + + +def test_test_propagates_nonzero_exit_code(captured_run: dict[str, Any]) -> None: + captured_run["returncode"] = 5 # pytest: no tests collected + result = runner.invoke(app, ["test"]) + assert result.exit_code == 5, _combined(result) + + +def test_test_propagates_failure_exit_code(captured_run: dict[str, Any]) -> None: + captured_run["returncode"] = 1 # pytest: tests failed + result = runner.invoke(app, ["test"]) + assert result.exit_code == 1, _combined(result) + + +def test_test_missing_pytest_is_typed_install_prompt( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # Simulate the [test] extra not being installed. + import importlib.util + + real_find_spec = importlib.util.find_spec + + def fake_find_spec(name: str, *args: Any, **kwargs: Any) -> Any: + if name == "pytest": + return None + return real_find_spec(name, *args, **kwargs) + + monkeypatch.setattr(cli.importlib.util, "find_spec", fake_find_spec) + + # subprocess.run must never be reached in this path. + def exploding_run(*args: Any, **kwargs: Any) -> None: + raise AssertionError("subprocess.run should not run when pytest is absent") + + monkeypatch.setattr(cli.subprocess, "run", exploding_run) + + result = runner.invoke(app, ["test"]) + assert result.exit_code == 1 + out = _combined(result) + assert "pip install" in out + assert "test" in out # names the [test] extra + # An install prompt, not a verification block. + assert "blocked on verification" not in out diff --git a/python/tests/unit/test_config.py b/python/tests/unit/test_config.py index a8b950e4..ef685630 100644 --- a/python/tests/unit/test_config.py +++ b/python/tests/unit/test_config.py @@ -1,4 +1,4 @@ -"""Config resolution + all-at-once validation (§2.1).""" +"""Config resolution + all-at-once validation.""" from __future__ import annotations @@ -14,7 +14,7 @@ def test_validated_reports_all_missing_control_plane_fields_at_once() -> None: with pytest.raises(ConfigError) as exc: cfg.validated(need="control_plane") msg = str(exc.value) - # All three must appear in ONE error (§2.1), not one-per-run. + # All three must appear in ONE error (config resolution), not one-per-run. assert "client_id" in msg assert "client_secret" in msg assert "org_id" in msg @@ -26,15 +26,15 @@ def test_validated_llm_is_independent_of_control_plane() -> None: llm_proxy_client_id="cid", llm_proxy_client_secret="csecret", ) - # LLM creds present, control-plane absent: llm validation passes (§2.2). + # LLM creds present, control-plane absent: llm validation passes (BG §1.1). assert cfg.validated(need="llm") is cfg with pytest.raises(ConfigError): cfg.validated(need="control_plane") def test_validated_llm_requires_client_id_and_secret_not_bearer() -> None: - # Verified auth (§2/§3) is a client_id/secret pair — a bare url + api-key is - # NOT sufficient, and the error names BOTH missing header credentials at once. + # Verified auth (docs/verified-apis.md §2/§3) is a client_id/secret pair — a bare url + api-key + # is NOT sufficient, and the error names BOTH missing header credentials at once. cfg = DonkeyConfig(llm_proxy_url="https://proxy", llm_proxy_key="k") with pytest.raises(ConfigError) as exc: cfg.validated(need="llm") @@ -84,7 +84,7 @@ def test_capture_content_env_overrides_toml(tmp_path, monkeypatch: pytest.Monkey assert DonkeyConfig.from_env().telemetry_capture_content is True # env wins -# --- on_model_substitution resolution + validation (§3, #309) --------------- +# --- on_model_substitution resolution + validation (BG §1.1, #309) --------------- # Off by default: a routing substitution is surfaced passively on last_call # unless the developer opts into a hard error, through the normal precedence. @@ -124,7 +124,7 @@ def test_unknown_on_model_substitution_is_a_config_error( tmp_path, monkeypatch: pytest.MonkeyPatch ) -> None: # A typo like "error" must fail loudly at resolve time — a silent fall-back to - # "off" would leave a caller believing they had opted into strictness (§309). + # "off" would leave a caller believing they had opted into strictness (#309). _isolate_toml(tmp_path, monkeypatch) monkeypatch.setenv("DONKEY_ON_MODEL_SUBSTITUTION", "error") with pytest.raises(ConfigError) as exc: @@ -132,7 +132,7 @@ def test_unknown_on_model_substitution_is_a_config_error( assert "error" in str(exc.value) -# --- cost-attribution tag resolution (§3, BG §1.7, #196) -------------------- +# --- cost-attribution tag resolution (docs/verified-apis.md §3, BG §1.7, #196) -------------------- def _isolate_toml(tmp_path, monkeypatch: pytest.MonkeyPatch, body: str | None = None) -> None: diff --git a/python/tests/unit/test_conformance_base_only.py b/python/tests/unit/test_conformance_base_only.py index 07dde5d4..34e328f0 100644 --- a/python/tests/unit/test_conformance_base_only.py +++ b/python/tests/unit/test_conformance_base_only.py @@ -1,4 +1,4 @@ -"""The base-only guarantee for the conformance kit (§1.1, BG §1.5). +"""The base-only guarantee for the conformance kit (the layered architecture, BG §1.5). The pytest11 entry point auto-loads ``donkey_kit.conformance.plugin`` on EVERY pytest run once ``donkey-kit`` is installed — including this base-only job, diff --git a/python/tests/unit/test_conformance_plugin.py b/python/tests/unit/test_conformance_plugin.py index c7ab7446..bd8d7aaf 100644 --- a/python/tests/unit/test_conformance_plugin.py +++ b/python/tests/unit/test_conformance_plugin.py @@ -8,7 +8,7 @@ options + an unused fixture); - a clean ``UsageError`` (exit code 4, no traceback) when ``--agent`` is missing or a ``KNOWN_LIMITATIONS`` entry is invalid — validated at collection, never a - silent skip (§8.1); + silent skip (the conformance kit); - a green run + printed scenario table for a well-behaved agent, and a red run naming the tripped scenario for a broken one. @@ -128,6 +128,15 @@ def test_donkey_fixture_is_available(pytester: pytest.Pytester) -> None: result.assert_outcomes(passed=1) +def test_gateway_fixture_is_registered(pytester: pytest.Pytester) -> None: + # The out-of-process `gateway` fixture (#278) is offered on every run, beside + # `donkey`. Asserted via --fixtures so it proves registration without booting + # a server (which would need the [local] extra); its docstring names its shape. + result = pytester.runpytest_subprocess("--fixtures", "-v") + result.stdout.fnmatch_lines(["*gateway*"]) + result.stdout.fnmatch_lines(["*ephemeral port*"]) + + def test_conformance_without_agent_is_a_usage_error(pytester: pytest.Pytester) -> None: # --donkey-conformance with no --agent is a misuse: a clean UsageError # (exit 4), not a traceback and not a silently-empty run. @@ -138,7 +147,7 @@ def test_conformance_without_agent_is_a_usage_error(pytester: pytest.Pytester) - def test_invalid_known_limitations_is_a_usage_error(pytester: pytest.Pytester) -> None: # A KNOWN_LIMITATIONS key that names no scenario is rejected at collection - # time (§8.1: asserted, never silent) — a UsageError before any scenario runs. + # time (the conformance kit: asserted, never silent) — a UsageError before any scenario runs. pytester.makepyfile(brokenagent=_BAD_KNOWN_LIMITATIONS_AGENT) result = pytester.runpytest_subprocess( "--donkey-conformance", "--agent=brokenagent:build" diff --git a/python/tests/unit/test_cost.py b/python/tests/unit/test_cost.py index 9c925ec2..e6bbfc96 100644 --- a/python/tests/unit/test_cost.py +++ b/python/tests/unit/test_cost.py @@ -1,4 +1,4 @@ -"""Cost-attribution tags (§3, BG §1.7, #196). +"""Cost-attribution tags (docs/verified-apis.md §3, BG §1.7, #196). ``CostTags`` is the pure, framework-free carrier for the FIXED four-dimension attribution set. These tests pin the two guarantees the acceptance criteria diff --git a/python/tests/unit/test_donkey_surface.py b/python/tests/unit/test_donkey_surface.py index 42e926a4..7f41262f 100644 --- a/python/tests/unit/test_donkey_surface.py +++ b/python/tests/unit/test_donkey_surface.py @@ -22,7 +22,7 @@ def test_uninstalled_adapter_raises_curated_import_error( # Force "framework not installed" regardless of what happens to be present in # the dev env, so the assertion is deterministic: access must raise # ImportError with the exact install command, never a bare - # ModuleNotFoundError (§3.2). + # ModuleNotFoundError (BG §1.8). monkeypatch.setattr("donkey_kit.donkey._framework_installed", lambda _probe: False) fab = Donkey(_cfg()) with pytest.raises(ImportError) as exc: @@ -125,7 +125,7 @@ def test_run_without_id_generates_a_run_of_one() -> None: assert current_correlation_id() is None -# --- cost-attribution tags on the public surface (§3, BG §1.7, #196) -------- +# --- cost-attribution tags on the public surface (docs/verified-apis.md §3, BG §1.7, #196) -------- def test_from_env_sets_config_level_cost_tags( @@ -162,7 +162,7 @@ def test_from_env_kwargs_merge_over_env_tags( assert fab.config.cost == CostTags(team="kwarg-team", env="prod") -# --- on_model_substitution override on the public surface (§3, #309) -------- +# --- on_model_substitution override on the public surface (BG §1.1, #309) -------- def test_from_env_sets_on_model_substitution( @@ -181,7 +181,7 @@ def test_from_env_sets_on_model_substitution( def test_model_substituted_is_a_public_export() -> None: # The typed error the "raise" mode surfaces must be importable from the - # top-level package so a caller can `except ModelSubstituted` (§3, #309). + # top-level package so a caller can `except ModelSubstituted` (BG §1.1, #309). from donkey_kit import ModelSubstituted from donkey_kit.core.errors import DonkeyError @@ -237,7 +237,7 @@ def test_client_returns_async_by_default_and_blocking_on_request() -> None: def test_both_clients_carry_the_same_governed_configuration() -> None: """The blocking client is a transport swap, not a different contract: same - base URL and same verified client_id/client_secret headers (§2/§3).""" + base URL and same verified client_id/client_secret headers (docs/verified-apis.md §2/§3).""" pytest.importorskip("openai") with Donkey(_cfg()) as fab: @@ -248,7 +248,7 @@ def test_both_clients_carry_the_same_governed_configuration() -> None: assert str(built.base_url) == "https://proxy" assert built.default_headers["client_id"] == "cid" assert built.default_headers["client_secret"] == "csecret" - assert built.max_retries == 0 # retries belong to the transport (§2.3) + assert built.max_retries == 0 # retries belong to the transport (BG §1.1) def test_blocking_transport_is_lazy_shared_and_closed_by_the_context_manager() -> None: diff --git a/python/tests/unit/test_errors.py b/python/tests/unit/test_errors.py index 2ae3615d..b27d48e8 100644 --- a/python/tests/unit/test_errors.py +++ b/python/tests/unit/test_errors.py @@ -1,4 +1,4 @@ -"""errors.classify behaviour (§2.4). Policy rejections must be terminal.""" +"""errors.classify behaviour (BG §1.2). Policy rejections must be terminal.""" from __future__ import annotations @@ -32,7 +32,7 @@ def test_401_is_auth_error() -> None: def test_generic_4xx_is_terminal_policy_violation() -> None: err = classify(_resp(400)) assert isinstance(err, PolicyViolation) - assert err.remediation # required, non-empty (§2.4) + assert err.remediation # required, non-empty (BG §1.2) def test_429_is_token_budget_with_retry_after() -> None: @@ -48,7 +48,7 @@ def test_injection_protection_header_is_prompt_injection_blocked() -> None: err = classify(_resp(400, {"x-injection-protection": "blocked"})) assert isinstance(err, PromptInjectionBlocked) assert err.policy == "prompt-injection-protection" - assert err.remediation # required, non-empty (§2.4) + assert err.remediation # required, non-empty (BG §1.2) def test_400_without_injection_header_is_not_prompt_injection() -> None: @@ -115,7 +115,7 @@ def test_content_safety_action_allow_is_not_content_safety_blocked() -> None: # carries no nested provider envelope must NOT be coerced into a subclass whose # contract we haven't verified. It is surfaced as a generic PolicyViolation that # names what was observed and says the shape is unconfirmed. A 403 is auth ONLY -# when it carries the verified www-authenticate challenge (docs §4). +# when it carries the verified www-authenticate challenge (docs/verified-apis.md §4). def test_unrecognised_403_falls_through_to_honest_policy_violation() -> None: @@ -149,7 +149,7 @@ def test_unrecognised_403_names_the_observed_policy_headers() -> None: def test_403_with_www_authenticate_is_still_auth_error() -> None: """Regression guard for the #184 split: a 403 carrying a ``www-authenticate`` - challenge is the verified client-id-enforcement shape (docs §4) and must + challenge is the verified client-id-enforcement shape (docs/verified-apis.md §4) and must remain an AuthError, so ``donkey doctor``'s credentials diagnosis (#202) stays intact.""" err = classify(_resp(403, {"www-authenticate": 'Bearer realm="anypoint"'})) @@ -169,11 +169,11 @@ def test_5xx_is_retryable_upstream() -> None: def test_policy_violation_is_not_a_retryable_type() -> None: # A PolicyViolation must never be an UpstreamModelError (which the transport - # would retry). Distinct branches of the taxonomy (§2.4). + # would retry). Distinct branches of the taxonomy (BG §1.2). assert not issubclass(PolicyViolation, UpstreamModelError) -# --- correlation/call id read-back (§2.3, #195) ----------------------------- +# --- correlation/call id read-back (BG §1.1, #195) ----------------------------- # classify() derives the run id and the per-call id from the response's own # request headers, so a caller bridging an openai error gets them for free. diff --git a/python/tests/unit/test_gateway.py b/python/tests/unit/test_gateway.py new file mode 100644 index 00000000..0aab8eba --- /dev/null +++ b/python/tests/unit/test_gateway.py @@ -0,0 +1,103 @@ +"""Base-only tests for the out-of-process ``gateway`` fixture's machinery (#278). + +These exercise the request spy and the missing-extra guard **without** booting a +real server, so they run under the base ``[dev]`` install (no ``[local]`` extra): + +- ``_record`` redacts the consumer-auth secret and any bearer, and keeps method, + path and every other header. +- ``_RecordingApp`` records one entry per HTTP request and forwards the ASGI + call untouched (lifespan/other scopes pass through, unrecorded). +- taking the fixture without the ``[local]`` extra raises an ``ImportError`` + naming the exact ``pip install`` — never a bare ``ModuleNotFoundError``. + +The end-to-end path (a real port, a stock client, ``set_scenarios``) lives in +``tests/conformance/test_gateway_fixture.py`` behind the ``local_gateway`` marker. +""" + +from __future__ import annotations + +import builtins +from typing import Any + +import pytest + +from donkey_kit.conformance.gateway import ( + RecordedRequest, + _record, + _RecordingApp, + start_gateway, +) + + +def _http_scope( + headers: list[tuple[bytes, bytes]], *, method: str = "POST", path: str = "/responses" +) -> dict[str, Any]: + return {"type": "http", "method": method, "path": path, "headers": headers} + + +def test_record_redacts_secret_and_bearer_keeps_the_rest() -> None: + req = _record( + _http_scope( + [ + (b"client_id", b"public-abc"), + (b"client_secret", b"super-secret-value"), + (b"authorization", b"Bearer tok"), + (b"content-type", b"application/json"), + ] + ) + ) + assert req.method == "POST" + assert req.path == "/responses" + # The public id and content-type survive verbatim; only the secret and bearer + # are redacted — so a captured request is safe to print in a failure message. + assert req.headers["client_id"] == "public-abc" + assert req.headers["content-type"] == "application/json" + assert req.headers["client_secret"] == "***" + assert req.headers["authorization"] == "***" + + +def test_record_redaction_is_case_insensitive() -> None: + # Both case variants of the real `client_secret` request header are redacted; + # the name is matched case-insensitively (ASGI lowercases names, but a proxy + # or client could send any case). + req = _record(_http_scope([(b"Client_Secret", b"nope"), (b"CLIENT_SECRET", b"nope")])) + assert set(req.headers.values()) == {"***"} + + +async def test_recording_app_records_http_and_forwards() -> None: + seen: list[str] = [] + + async def inner(scope: Any, receive: Any, send: Any) -> None: + seen.append(scope["type"]) + + async def receive() -> dict[str, Any]: # pragma: no cover - not driven here + return {} + + async def send(message: dict[str, Any]) -> None: # pragma: no cover + return None + + app = _RecordingApp(inner) + await app(_http_scope([(b"client_secret", b"s")], method="GET", path="/models"), receive, send) + await app({"type": "lifespan"}, receive, send) + + # Both scopes reach the inner app; only the HTTP one is recorded. + assert seen == ["http", "lifespan"] + assert len(app.requests) == 1 + (only,) = app.requests + assert isinstance(only, RecordedRequest) + assert only.method == "GET" + assert only.path == "/models" + assert only.headers["client_secret"] == "***" + + +def test_missing_local_extra_names_the_pip_install(monkeypatch: pytest.MonkeyPatch) -> None: + real_import = builtins.__import__ + + def _no_uvicorn(name: str, *args: Any, **kwargs: Any) -> Any: + if name == "uvicorn" or name.startswith("uvicorn."): + raise ImportError("No module named 'uvicorn'") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _no_uvicorn) + with pytest.raises(ImportError, match=r'pip install "donkey-kit\[local\]"'): + start_gateway() diff --git a/python/tests/unit/test_governed_state_shapes.py b/python/tests/unit/test_governed_state_shapes.py index 6a453f53..7b68cac6 100644 --- a/python/tests/unit/test_governed_state_shapes.py +++ b/python/tests/unit/test_governed_state_shapes.py @@ -6,7 +6,7 @@ logic faithfully represent the direct Anypoint control-plane contract — the join `ExchangeRegistry` will perform once its (still-blocked) live fetch path is wired (docs/verified-apis.md §12). The raw→domain mapping lives here in the test, -not in the SDK, keeping the §0.3 discipline: shapes are verified, the live fetch +not in the SDK, keeping the verification discipline: shapes are verified, the live fetch endpoint is not yet. """ @@ -74,7 +74,7 @@ def test_policies_drive_governed_verdict() -> None: ), ] # Use STRICT minus the ruleset-pass check (ruleset result-read shape is still - # UNVERIFIED, §6); everything else is real and should pass. + # UNVERIFIED, docs/verified-apis.md §6); everything else is real and should pass. criteria = GovernanceCriteria( require_governance_pass=False, required_policies=STRICT.required_policies, @@ -88,7 +88,8 @@ def test_policies_drive_governed_verdict() -> None: def test_missing_required_policy_fails_and_explains() -> None: # Same instance but demand a policy it does NOT have → not governed, with a - # retained reason (the "filtered vs broken credential" distinction, §6.1.2). + # retained reason (the "filtered vs broken credential" distinction, the governed-state check + # API). checks = [Check("required_policies", False, "rate-limiting not applied")] report = evaluate(checks, GovernanceCriteria(allow_unknown=False)) assert report.governed is False diff --git a/python/tests/unit/test_langgraph_correlation.py b/python/tests/unit/test_langgraph_correlation.py index 4af54a3e..fbf939f7 100644 --- a/python/tests/unit/test_langgraph_correlation.py +++ b/python/tests/unit/test_langgraph_correlation.py @@ -1,4 +1,4 @@ -"""LangGraph correlation-id propagation (#195, §2.3). +"""LangGraph correlation-id propagation (#195, BG §1.1). Proves the #195 acceptance criterion "propagation reaches every LangGraph node via contextvar (no threading through state)": a run id bound once with diff --git a/python/tests/unit/test_lastcall.py b/python/tests/unit/test_lastcall.py index 241b6314..b5086723 100644 --- a/python/tests/unit/test_lastcall.py +++ b/python/tests/unit/test_lastcall.py @@ -1,10 +1,10 @@ """LastCall — the gateway's own metadata about the most recent governed call -(§3, #362). Base-only surface (Surface 1): httpx + fixtures, no framework. +(BG §1.1, #362). Base-only surface (Surface 1): httpx + fixtures, no framework. Covers the issue's acceptance criteria: - * one named record, parsed from the LIVE-VERIFIED §3 headers (request_id, + * one named record, parsed from the LIVE-VERIFIED BG §1.1 headers (request_id, api_instance_id, environment_id) on a 200; - * absent/unparseable headers leave fields None, never raise (§0.3); + * absent/unparseable headers leave fields None, never raise (verification discipline); * populated IDENTICALLY by the async and blocking transports; * contextvar concurrency scope — a fan-out does not clobber (hazard #2); * three honest states OBSERVED / UNOBSERVED / UNAVAILABLE (hazard #3); @@ -41,7 +41,7 @@ from donkey_kit.core.transport import DonkeyAsyncClient, DonkeyClient from donkey_kit.simulator.fixtures import load, replay_headers -# The LIVE-VERIFIED §3 identity headers (responses.success.headers.txt, 2026-08-28). +# The LIVE-VERIFIED BG §1.1 identity headers (responses.success.headers.txt, 2026-08-28). _REQUEST_ID = "req_f85003861d5348c9a1d152c276082b07" _DECORATOR = "api-instance-21133858.3e6ce455-e3e8-4402-b830-9fcf07d9207b.svc" _API_INSTANCE_ID = "21133858" @@ -51,7 +51,7 @@ "x-request-id": _REQUEST_ID, "x-envoy-decorator-operation": _DECORATOR, } -# The LIVE-VERIFIED §3 routing headers (responses.success.headers.txt, #309). +# The LIVE-VERIFIED BG §1.1 routing headers (responses.success.headers.txt, #309). _ROUTING_HEADERS = { "x-llm-proxy-routing-type": "ModelBased", "x-llm-proxy-routing-fallback": "false", @@ -73,7 +73,7 @@ def _reset_last_call(): lastcall._last_call.reset(token) -# --- parsing the decorator-operation header (never raises, §0.3) ------------ +# --- parsing the decorator-operation header (never raises, verification discipline) ------------ @pytest.mark.parametrize( @@ -415,7 +415,8 @@ def test_parse_usage_absent_detail_fields_are_none_not_zero() -> None: [None, {}, "usage", 17, {"input_tokens": "x"}, {"input_tokens": True}], ) def test_parse_usage_non_dict_or_garbage_is_all_none(bad) -> None: - # Never raises, never a fabricated 0; a bool is not accepted as an int (§0.3). + # Never raises, never a fabricated 0; a bool is not accepted as an int (verification + # discipline). assert set(parse_usage(bad).values()) == {None} @@ -450,7 +451,8 @@ def test_usage_from_response_non_2xx_is_all_none() -> None: def test_usage_from_response_unread_or_empty_body_never_raises() -> None: - # An SSE / bodyless response's .json() raises; it is caught → all None (§0.3). + # An SSE / bodyless response's .json() raises; it is caught → all None (verification + # discipline). resp = httpx.Response(200, headers={"content-type": "text/event-stream"}) assert set(usage_from_response(resp).values()) == {None} @@ -559,12 +561,12 @@ def test_no_unverified_warning_on_the_read_path() -> None: LastCall.from_response(httpx.Response(200, headers=_IDENTITY_HEADERS)) observe_last_call(httpx.Response(200, headers=_IDENTITY_HEADERS)) _ = current_last_call() - # The usage path traces to the same verified §3 row — no warning either. + # The usage path traces to the same verified BG §1.1 row — no warning either. parse_usage(_RESPONSES_USAGE) usage_from_response(_json_response(200, _RESPONSES_USAGE)) -# --- gateway routing & fallback (§3, #309) ---------------------------------- +# --- gateway routing & fallback (BG §1.1, #309) ---------------------------------- @pytest.mark.parametrize( @@ -577,7 +579,7 @@ def test_no_unverified_warning_on_the_read_path() -> None: (" true ", True), # whitespace tolerated (None, None), # header absent — a non-proxy / simulated response ("", None), # empty is not a guess - ("maybe", None), # unrecognised is unknown, never a default False (§0.3) + ("maybe", None), # unrecognised is unknown, never a default False (verification discipline) ], ) def test_parse_fallback_shapes(raw, expected) -> None: @@ -619,7 +621,7 @@ def test_from_response_parses_all_five_routing_fields_from_the_live_fixture() -> def test_missing_routing_headers_report_none_and_do_not_crash() -> None: # AC2: a non-proxy or simulated response with none of the routing headers must - # leave every field None/None and never raise (§0.3). + # leave every field None/None and never raise (verification discipline). record = LastCall.from_response(httpx.Response(200)) assert record.served_provider is None assert record.served_model is None diff --git a/python/tests/unit/test_llm_client_openai3_injection.py b/python/tests/unit/test_llm_client_openai3_injection.py new file mode 100644 index 00000000..b0e21539 --- /dev/null +++ b/python/tests/unit/test_llm_client_openai3_injection.py @@ -0,0 +1,121 @@ +"""openai 3.x drives our httpx client end to end (#18, BG §1.8). + +The scare with `openai>=3.0` is that it retyped ``AsyncOpenAI(http_client=…)`` +to ``httpx2.AsyncClient`` — a class from a *separate* distribution +(``httpx2``), distinct from the ``httpx`` our ``DonkeyAsyncClient`` / +``DonkeyClient`` subclass. That is a **typecheck-only** mismatch: when an +``http_client`` is injected, openai builds and sends every request *through +that client*, so ``httpx2`` never touches our code path. These tests pin that +runtime truth so a future openai release that actually broke injection would +fail CI loudly (floors-never-ceilings, docs/verified-apis.md §8.1) — rather +than us discovering it in a customer sandbox. + +Guarded by ``importorskip("openai")`` at module top so the base-only CI job +(``.[dev]`` only, no ``[llm]``) skips it cleanly; it runs in the full/nightly +matrix where openai is installed and always the newest release. +""" + +from __future__ import annotations + +import pytest + +openai = pytest.importorskip("openai") + +import httpx # noqa: E402 + +from donkey_kit.core.config import DonkeyConfig # noqa: E402 +from donkey_kit.core.transport import DonkeyAsyncClient, DonkeyClient # noqa: E402 +from donkey_kit.llm.client import LLMClient # noqa: E402 + +# A base URL with NO `/v1` — the verified ingress shape (docs/verified-apis.md §2). +# The trailing slash lets the OpenAI SDK append `chat/completions` directly. +_BASE_URL = "https://gw.example.internal/openai-sdk/" + +# Explicit correlation/call-id header names so no UnverifiedValueWarning fires and +# the asserted keys are deterministic (the placeholder names warn once per key). +_CFG = DonkeyConfig( + llm_proxy_url=_BASE_URL, + llm_proxy_client_id="cid-123", + llm_proxy_client_secret="csecret-456", + correlation_header="x-correlation-id", + call_id_header="x-donkey-request-id", +) + +_CANNED_COMPLETION = { + "id": "chatcmpl-1", + "object": "chat.completion", + "created": 0, + "model": "gpt-4o", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "hi there"}, + "finish_reason": "stop", + } + ], + "usage": {"prompt_tokens": 5, "completion_tokens": 2, "total_tokens": 7}, +} + + +def _recording_handler(seen: dict[str, object]): + """A MockTransport handler that records the request our transport actually + sent, then returns a canned chat completion.""" + + def handler(request: httpx.Request) -> httpx.Response: + seen["request"] = request + return httpx.Response(200, json=_CANNED_COMPLETION) + + return handler + + +def _assert_governed(seen: dict[str, object]) -> None: + request = seen["request"] + # openai built and sent the request through OUR client — the object our + # transport saw is a plain httpx.Request, not something from httpx2. + assert isinstance(request, httpx.Request) + assert type(request).__module__.split(".")[0] == "httpx" + # The verified client_id/client_secret pair is the real auth (docs/verified-apis.md §2/§3). + assert request.headers["client_id"] == "cid-123" + assert request.headers["client_secret"] == "csecret-456" + # The OpenAI SDK still sets Authorization from its required api_key slot; that + # slot is the sentinel the proxy ignores, never a real credential. + assert request.headers["authorization"] == "Bearer client-id-enforced" + # Correlation + per-call ids stamped by the transport. + assert request.headers.get("x-correlation-id") + assert request.headers.get("x-donkey-request-id") + # Base URL used verbatim: no `/v1` injected at the ingress (docs/verified-apis.md §2). + url = str(request.url) + assert url == "https://gw.example.internal/openai-sdk/chat/completions" + assert "/v1/" not in url + + +async def test_async_openai_client_drives_our_httpx_client_end_to_end() -> None: + seen: dict[str, object] = {} + shared = DonkeyAsyncClient(_CFG, None, transport=httpx.MockTransport(_recording_handler(seen))) + async with shared: + client = LLMClient(_CFG, shared).client() # AsyncOpenAI + resp = await client.chat.completions.create( + model="gpt-4o", messages=[{"role": "user", "content": "hi"}] + ) + assert resp.choices[0].message.content == "hi there" + assert resp.usage.total_tokens == 7 + _assert_governed(seen) + + +def test_sync_openai_client_drives_our_httpx_client_end_to_end() -> None: + seen: dict[str, object] = {} + # The async client is required by LLMClient's constructor but unused on the + # sync path; the blocking transport is what we drive here. + async_shared = DonkeyAsyncClient( + _CFG, None, transport=httpx.MockTransport(lambda r: httpx.Response(200)) + ) + sync_client = DonkeyClient(_CFG, transport=httpx.MockTransport(_recording_handler(seen))) + with sync_client: + llm = LLMClient(_CFG, async_shared, sync_http_client=lambda: sync_client) + client = llm.client(sync=True) # OpenAI + resp = client.chat.completions.create( + model="gpt-4o", messages=[{"role": "user", "content": "hi"}] + ) + assert resp.choices[0].message.content == "hi there" + assert resp.usage.total_tokens == 7 + _assert_governed(seen) diff --git a/python/tests/unit/test_llm_proxy_contract.py b/python/tests/unit/test_llm_proxy_contract.py index 809a62b4..36f10005 100644 --- a/python/tests/unit/test_llm_proxy_contract.py +++ b/python/tests/unit/test_llm_proxy_contract.py @@ -4,7 +4,7 @@ and docs/verified-apis.md §2/§3/§4. Unlike the shape-only A2D fixtures, these are the direct Anypoint data-plane -contract, so they may drive real SDK behavior (the §8.2 fixture-derived error +contract, so they may drive real SDK behavior (the BG §1.5 fixture-derived error table). The raw→exception mapping under test lives in core/errors.classify(). """ @@ -38,8 +38,8 @@ def _load(name: str) -> object: def test_success_response_is_openai_passthrough_with_usage() -> None: - """§2: proxy returns the OpenAI Responses object verbatim, incl. token usage - (the basis for cost attribution) — no /v1 rewriting of the body.""" + """docs/verified-apis.md §2: proxy returns the OpenAI Responses object verbatim, incl. token + usage (the basis for cost attribution) — no /v1 rewriting of the body.""" body = _load("responses.success.body.json") assert isinstance(body, dict) assert body["object"] == "response" @@ -52,7 +52,7 @@ def test_success_response_is_openai_passthrough_with_usage() -> None: def test_success_headers_carry_gateway_governance_and_identity() -> None: - """§3: attribution/telemetry is on the response headers, not a bespoke + """docs/verified-apis.md §3: attribution/telemetry is on the response headers, not a bespoke request header the SDK must invent.""" h = _headers("responses.success.headers.txt") assert h["server"] == "Anypoint Flex Gateway" @@ -64,7 +64,7 @@ def test_success_headers_carry_gateway_governance_and_identity() -> None: def test_client_id_enforcement_rejection_classifies_as_auth() -> None: - """§4 family 1 — Anypoint policy rejection: flat string `error`, 401 + + """docs/verified-apis.md §4 family 1 — Anypoint policy rejection: flat string `error`, 401 + www-authenticate. classify() maps 401/403 → AuthError.""" h = _headers("reject.client-id-missing.headers.txt") body = _load("reject.client-id-missing.body.json") @@ -77,7 +77,7 @@ def test_client_id_enforcement_rejection_classifies_as_auth() -> None: def test_two_error_envelope_families_are_distinguishable() -> None: - """§4: the fixture-derived discriminator the §8.2 table needs. + """docs/verified-apis.md §4: the fixture-derived discriminator the BG §1.5 table needs. - Anypoint policy rejection → `error` is a STRING. - Upstream provider (OpenAI) passthrough → `error` is an OBJECT with @@ -93,8 +93,8 @@ def test_two_error_envelope_families_are_distinguishable() -> None: def test_upstream_400_classifies_as_upstream_request_error() -> None: - """§4 discriminator (fixture-driven, §8.2): the model-not-found 400 is an - upstream provider passthrough (nested `error` object), so classify() returns + """docs/verified-apis.md §4 discriminator (fixture-driven, BG §1.5): the model-not-found 400 is + an upstream provider passthrough (nested `error` object), so classify() returns UpstreamRequestError — NOT a gateway PolicyViolation — carrying the provider code/type for actionability.""" body = _load("reject.model-not-found.body.json") @@ -108,8 +108,8 @@ def test_upstream_400_classifies_as_upstream_request_error() -> None: def test_pii_detection_rejection_classifies_as_pii_not_auth() -> None: - """§4 live capture: the PII policy rejects with 403 + a NESTED error object - whose type is ``pii_detected`` and NO ``www-authenticate`` header. Despite + """docs/verified-apis.md §4 live capture: the PII policy rejects with 403 + a NESTED error + object whose type is ``pii_detected`` and NO ``www-authenticate`` header. Despite the 403, this is NOT an auth failure — classify() must return PIIDetected and surface the flagged entity types parsed from the message.""" h = _headers("reject.pii-detected.headers.txt") @@ -127,7 +127,7 @@ def test_pii_detection_rejection_classifies_as_pii_not_auth() -> None: def test_token_rate_limit_rejection_is_429_with_header_only_budget() -> None: - """§4 live capture: the token-rate-limit policy rejects with 429 and an + """docs/verified-apis.md §4 live capture: the token-rate-limit policy rejects with 429 and an EMPTY body; the reset window is header-only (x-token-reset, in ms) with NO standard retry-after. classify() → TokenBudgetExceeded with retry_after derived from x-token-reset.""" diff --git a/python/tests/unit/test_pure_logic.py b/python/tests/unit/test_pure_logic.py index b1e822d3..46f87986 100644 --- a/python/tests/unit/test_pure_logic.py +++ b/python/tests/unit/test_pure_logic.py @@ -69,5 +69,5 @@ def test_governance_unknown_check_respects_allow_unknown() -> None: lenient = evaluate(checks, GovernanceCriteria(allow_unknown=True)) assert strict.governed is False assert lenient.governed is True - # Reason is retained either way (§6.1.2). + # Reason is retained either way (the governed-state check API). assert any("ruleset" in r for r in strict.reasons_failed()) diff --git a/python/tests/unit/test_registry_shapes.py b/python/tests/unit/test_registry_shapes.py index bfecfb90..98ad5a3b 100644 --- a/python/tests/unit/test_registry_shapes.py +++ b/python/tests/unit/test_registry_shapes.py @@ -3,10 +3,10 @@ The fixtures under ``tests/fixtures/a2d/`` are faithful captures from the A2D platform's MCP tools (see that directory's README). These tests prove the SDK's ``McpServerHandle`` / ``ToolDescriptor`` / ``AssetRef`` types can represent -real-world shapes (§8.2) — WITHOUT wiring the (still-blocked) ``ExchangeRegistry`` +real-world shapes (BG §1.5) — WITHOUT wiring the (still-blocked) ``ExchangeRegistry`` to any live endpoint. The raw→domain mapping lives here in the test, not in the SDK, precisely because the direct Anypoint REST contract is still UNVERIFIED -(§0.3); only the shapes are confirmed. +(verification discipline); only the shapes are confirmed. """ from __future__ import annotations diff --git a/python/tests/unit/test_rejection_contract.py b/python/tests/unit/test_rejection_contract.py index 625d5ffc..2e26454c 100644 --- a/python/tests/unit/test_rejection_contract.py +++ b/python/tests/unit/test_rejection_contract.py @@ -6,7 +6,7 @@ moved); rows 3/4/6 (injection, content-moderation, upstream-5xx) and rows 7/8 (regex-prompt-guard, content-safety — #289) live in ``rejections/``. The #289 shapes are documented (pinned from the policy pages) but not yet live-captured; -their live capture is blocked on #253 (see ``rejections/README.md`` and docs §4). +their live capture is blocked on #253 (see ``rejections/README.md`` and docs/verified-apis.md §4). The discriminator is the error ``type`` + specific headers, NEVER the status code alone — which is exactly why rows 7/8 (both 403) must not be swallowed by the 401/403 → auth rule. diff --git a/python/tests/unit/test_simulator_app.py b/python/tests/unit/test_simulator_app.py index e191db8e..cb7d4ce6 100644 --- a/python/tests/unit/test_simulator_app.py +++ b/python/tests/unit/test_simulator_app.py @@ -113,8 +113,8 @@ async def test_stream_replays_the_sse_capture_with_event_stream_media_type() -> assert resp.status_code == 200 assert resp.headers["content-type"].startswith("text/event-stream") # The captured SSE sample is a single, truncated `response.created` event — - # replayed verbatim, NOT a complete stream ending in `data: [DONE]` (§0.3: the - # rest is not fabricated). Assert byte-identity, not consumability. + # replayed verbatim, NOT a complete stream ending in `data: [DONE]` (verification discipline: + # the rest is not fabricated). Assert byte-identity, not consumability. assert resp.content == fx.load("stream").body assert resp.headers[SIMULATOR_HEADER] == "true" # A streaming call is still a happy path: it carries the same prose budget diff --git a/python/tests/unit/test_simulator_base_only.py b/python/tests/unit/test_simulator_base_only.py index b969ea66..efdab496 100644 --- a/python/tests/unit/test_simulator_base_only.py +++ b/python/tests/unit/test_simulator_base_only.py @@ -1,4 +1,4 @@ -"""The base-only guarantee (§1.1, BG §1.4): importing the simulator package must +"""The base-only guarantee (the layered architecture, BG §1.4): importing the simulator package must NOT require the ``[local]`` extra. Unguarded on purpose — this runs in the base-only CI job under ``[dev]`` only, diff --git a/python/tests/unit/test_simulator_scenarios.py b/python/tests/unit/test_simulator_scenarios.py index 02e2ffee..7a371978 100644 --- a/python/tests/unit/test_simulator_scenarios.py +++ b/python/tests/unit/test_simulator_scenarios.py @@ -157,7 +157,7 @@ def test_budget_happy_path_header_reflects_counter() -> None: def test_budget_default_cost_is_fixture_usage() -> None: # The default per-call cost is the happy-path fixture's own total_tokens (68), - # not a fabricated number (§0.3). + # not a fabricated number (verification discipline). s = BudgetScenario(limit=1000, window_ms=60_000) s.on_call("hi") assert "932 tokens remaining of 1000 limit" in s.happy_path_headers()[RATELIMIT_HEADER] diff --git a/python/tests/unit/test_telemetry.py b/python/tests/unit/test_telemetry.py index ed736adb..a765be2a 100644 --- a/python/tests/unit/test_telemetry.py +++ b/python/tests/unit/test_telemetry.py @@ -81,7 +81,7 @@ def test_policy_decision_values_are_the_documented_literals() -> None: def test_routing_keys_are_the_pinned_and_stable_literal_strings() -> None: # The served model uses the pinned semconv key; the two routing facts the - # semconv has no key for live under the stable donkey.* namespace (§3, #309). + # semconv has no key for live under the stable donkey.* namespace (BG §1.1, #309). # gen_ai.response.model is pinned; the donkey.routing.* pair is public API. assert telemetry.GEN_AI_RESPONSE_MODEL == "gen_ai.response.model" assert telemetry.DONKEY_ROUTING_TYPE == "donkey.routing.type" @@ -154,7 +154,7 @@ def test_build_genai_attributes_keeps_zero_token_counts() -> None: def test_build_genai_attributes_emits_the_routing_facts() -> None: - # §3, #309: the served model, routing strategy and fallback flag land on the + # BG §1.1, #309: the served model, routing strategy and fallback flag land on the # span beside the request model. attrs = telemetry.build_genai_attributes( request_model="gpt-5.1", diff --git a/python/tests/unit/test_transport.py b/python/tests/unit/test_transport.py index 399e9822..4efbb8f4 100644 --- a/python/tests/unit/test_transport.py +++ b/python/tests/unit/test_transport.py @@ -1,5 +1,5 @@ """Transport: header injection, retry policy, no-retry-on-policy-rejection, -401 refresh (§2.3). Uses httpx MockTransport so no network is touched.""" +401 refresh (BG §1.1). Uses httpx MockTransport so no network is touched.""" from __future__ import annotations @@ -76,7 +76,8 @@ def handler(request: httpx.Request) -> httpx.Response: cfg = DonkeyConfig(application_name="hr-agent", business_group="finance") async with _client(handler, cfg) as client: - with pytest.warns(UnverifiedValueWarning): # placeholder header names warn (§0.3) + # placeholder header names warn (verification discipline) + with pytest.warns(UnverifiedValueWarning): with run_context("run-1"): await client.get("https://x/thing") @@ -87,8 +88,8 @@ def handler(request: httpx.Request) -> httpx.Response: def test_proxy_auth_headers_carry_verified_client_id_secret() -> None: - """§2/§3 (LIVE): the direct-proxy auth is a client_id/client_secret request- - header pair — verified names, no warning, no bearer.""" + """docs/verified-apis.md §2/§3 (LIVE): the direct-proxy auth is a client_id/client_secret + request-header pair — verified names, no warning, no bearer.""" cfg = DonkeyConfig( llm_proxy_url="https://proxy", llm_proxy_client_id="cid", @@ -144,7 +145,7 @@ def handler(request: httpx.Request) -> httpx.Response: async with _client(handler, DonkeyConfig(max_retries=3)) as client: resp = await client.get("https://x") assert resp.status_code == 400 - assert calls["n"] == 1 # terminal — NOT retried (§2.4) + assert calls["n"] == 1 # terminal — NOT retried (BG §1.2) async def test_401_triggers_single_token_refresh() -> None: @@ -157,7 +158,7 @@ def handler(request: httpx.Request) -> httpx.Response: async with _client(handler, DonkeyConfig(), StaticToken("t")) as client: resp = await client.get("https://x") assert resp.status_code == 200 - assert calls["n"] == 2 # refreshed once, retried once (§2.2) + assert calls["n"] == 2 # refreshed once, retried once (BG §1.1) # --- blocking twin (client(sync=True)) ------------------------------------ @@ -216,7 +217,7 @@ def handler(request: httpx.Request) -> httpx.Response: assert seen == ["run-7", "run-7"] -# --- the two ids: run correlation id vs per-call id (§2.3, #195) ------------ +# --- the two ids: run correlation id vs per-call id (BG §1.1, #195) ------------ # X-Correlation-Id groups a run (shared); X-Donkey-Request-Id pinpoints one # request within it (unique, but stable across that request's own retries). @@ -266,7 +267,7 @@ def handler(request: httpx.Request) -> httpx.Response: async def test_config_overrides_correlation_and_call_id_header_names() -> None: """A customer whose gateway reads different inbound names points the SDK at - them via config; the override is used verbatim (§0.3 escape hatch, #195).""" + them via config; the override is used verbatim (verification discipline escape hatch, #195).""" seen: dict[str, str] = {} def handler(request: httpx.Request) -> httpx.Response: @@ -338,7 +339,7 @@ def handler(request: httpx.Request) -> httpx.Response: async def test_classify_round_trip_emits_no_unverified_warning() -> None: - """#363 AC4 / §0.3: read-back must never emit an ``UnverifiedValueWarning`` — + """#363 AC4 / verification discipline: read-back must never emit an ``UnverifiedValueWarning`` — that warning belongs at injection time. The extensions fallback touches ``.placeholder``, never ``Unverified.get()``. Covers both the stamped path and a response with no stamp at all (a hand-built stock-client response).""" @@ -486,7 +487,7 @@ def handler(request: httpx.Request) -> httpx.Response: with _sync_client(handler, DonkeyConfig(max_retries=3)) as client: resp = client.get("https://x") assert resp.status_code == 400 - assert calls["n"] == 1 # terminal — NOT retried (§2.4) + assert calls["n"] == 1 # terminal — NOT retried (BG §1.2) def test_sync_401_is_terminal_because_there_is_no_token_to_refresh() -> None: @@ -635,7 +636,7 @@ def test_sync_swap_transport_takes_effect_on_the_next_request() -> None: assert client.get("https://x").status_code == 200 -# --- policy refusals are terminal: no retry (#183, §2.4) ------------------- +# --- policy refusals are terminal: no retry (#183, BG §1.2) ------------------- # classify() maps EVERY 429 to TokenBudgetExceeded (a PolicyViolation), so a 429 # is terminal like any other policy refusal — retrying it just burns the same # already-exhausted budget window (Scenario B: 50k records overnight). The @@ -648,12 +649,12 @@ async def test_does_not_retry_429_budget_refusal() -> None: def handler(request: httpx.Request) -> httpx.Response: calls["n"] += 1 - return httpx.Response(429) # empty body, no retry-after (docs §4) + return httpx.Response(429) # empty body, no retry-after (docs/verified-apis.md §4) async with _client(handler, DonkeyConfig(max_retries=3)) as client: resp = await client.get("https://x") assert resp.status_code == 429 - assert calls["n"] == 1 # terminal on the first hit — never retried (§2.4, #183) + assert calls["n"] == 1 # terminal on the first hit — never retried (BG §1.2, #183) async def test_does_not_retry_403_policy_rejection() -> None: @@ -876,7 +877,7 @@ async def test_llm_refusal_records_refuse_decision_and_policy_type(monkeypatch) exporter = _use_tracer(monkeypatch) def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(429) # empty body → TokenBudgetExceeded (docs §4) + return httpx.Response(429) # empty body → TokenBudgetExceeded (docs/verified-apis.md §4) client = DonkeyAsyncClient(_LLM_CFG, None, transport=httpx.MockTransport(handler)) async with client: @@ -1305,8 +1306,9 @@ def test_sync_streaming_span_closes_when_abandoned_mid_iteration(monkeypatch) -> assert dict(span.attributes)["gen_ai.request.model"] == "gpt-4o" -# --- cost-attribution tags on the wire + in spans (§3, BG §1.7, #196) -------- -# All header NAMES are UNVERIFIED placeholders (docs §3); these tests set the +# --- cost-attribution tags on the wire + in spans +# --- (docs/verified-apis.md §3, BG §1.7, #196) -------- +# All header NAMES are UNVERIFIED placeholders (docs/verified-apis.md §3); these tests set the # ``cost_*_header`` config overrides so the asserted header keys are deterministic # and no placeholder warning is triggered. The span-attribute side carries the # full value regardless of the header question (AC #4). @@ -1547,7 +1549,7 @@ def test_sync_send_forwards_capture_content_flag(monkeypatch, capture: bool) -> assert seen == [capture, capture] -# --- gateway routing & fallback: never double-retry, opt-in raise (§3, #309) -- +# --- gateway routing & fallback: never double-retry, opt-in raise (BG §1.1, #309) -- # The gateway's Enhanced Resilience routing is the FIRST recovery layer; an SDK # retry stacked on a response it already failed over just multiplies latency # against an outage the gateway is already handling. And a silent model @@ -1763,7 +1765,7 @@ def test_transport_error_raises_gateway_unavailable_sync(exc: httpx.TransportErr async def test_gateway_unavailable_carries_the_sent_ids() -> None: # The run correlation id and the per-call id the client stamped on the failed # request are carried, so an availability failure quotes the same ids a - # response error would — even with no response (§2.3, #195). + # response error would — even with no response (BG §1.1, #195). client = _client(_raiser(httpx.ConnectError("refused"))) with warnings.catch_warnings(): warnings.simplefilter("ignore", UnverifiedValueWarning) diff --git a/spec/archive/donkey-development-kit-build-plan-v1.md b/spec/archive/donkey-development-kit-build-plan-v1.md deleted file mode 100644 index 4f0640d8..00000000 --- a/spec/archive/donkey-development-kit-build-plan-v1.md +++ /dev/null @@ -1,2082 +0,0 @@ -# Agent Fabric SDK — Build Plan & Implementation Spec (v1, ARCHIVED) - -> ## ⚠️ ARCHIVED — not the authority for what to build -> -> This is the **v1** build plan, superseded by -> [`../agent-fabric-sdk-build-plan.md`](../agent-fabric-sdk-build-plan.md) and -> [`../agent-fabric-sdk-build-guide.md`](../agent-fabric-sdk-build-guide.md). -> Its three-pillar model, provisioning control plane, and eight-adapter -> conformance roster were all **cut**. Do not plan, scope, or review work -> against this document. -> -> **It is retained for one reason:** roughly 500 `§N.N` citations in -> `python/` source, tests, docstrings, and `docs/` point into this -> document's numbering. Until those are migrated, a bare `§N.N` citation -> anywhere in this repo resolves **here**, not into the new plan. See -> "Citation convention" in the new build plan. -> -> Sections that describe **still-live, CI-enforced invariants** — `§0.3` -> verification discipline, `§1.1` layered architecture and framework-free -> core, `§2.1` config resolution, `§8.1` conformance kit, `§8.4` extras are -> floors-never-ceilings — remain accurate and are carried forward by -> reference. Everything about the three pillars (`§3`, `§4`, `§5`), -> publication (`§7`), and the provisioning control plane is historical. - -**Audience:** an engineering agent (Claude Opus 4.8 / Sonnet 5) implementing the SDK, plus the human tech lead reviewing scope. -**Status:** ARCHIVED design spec — superseded, retained for citation resolution. -**Date of research:** August 2026. All third-party framework APIs cited here were verified around this date and **must be re-verified before coding** (see §0.3). - ---- - -## 0. Read this before writing any code - -### 0.1 What this SDK is - -A Python and TypeScript SDK that lets an agent developer, working in their own IDE and their own agent framework, consume three MuleSoft platform capabilities without adopting Mule: - -1. **Governed model access** — the Omni Gateway LLM Proxy as a drop-in model provider for eight agent frameworks. -2. **Governed tool access** — Anypoint Exchange as a live registry, resolving to ready-to-bind MCP toolsets. -3. **Provisioning as code** — declarative MCP Bridge instances and policy bindings, applied from CI with plan/diff/apply semantics. - -### 0.2 What this SDK is explicitly NOT - -Do not build these. Each was considered and rejected for a stated reason. - -| Not building | Why | -|---|---| -| A Python/TS runtime for Agent Broker orchestration | Agent networks compile to Mule apps on CloudHub 2.0; brokers are A2A servers. Reimplementing the guided-determinism graph engine is a competing product, not an SDK. | -| Authoring of gateway policy *logic* | Omni Gateway policies are Rust→WASM on Envoy via the PDK. Cannot be expressed in Python or TS. §6.9 specifies a **contract** with companion custom policies and the SDK's client half of it — the policy logic itself lives in a separate repo, and no `.rs` file belongs in this one. | -| An agent-network YAML/Agent Script generator | Schema is new and moving (Agent Network 2.0 / `.agent` files). The Anypoint CLI plugin and DX MCP Server already cover it. Wrap the CLI later if demanded. | -| A wrapper abstraction over the eight frameworks | Adapters return **native** framework objects. See §3.1. | -| Runtime policy application from application code | Inverts the platform-team ownership model. Provisioning is a CI-time concern. See §5.4. | -| A central scanner that publishes to Exchange directly | Repository scanning **is** in scope (§7.10), but the scanner opens PRs; publication happens in each repo's own CI. A central publisher holding write scopes makes its operator the org's publisher of record. See §7.10.3. | - -### 0.3 Mandatory verification step (do this first, before M0) - -Several APIs referenced below are recent or move fast. **Do not code against my descriptions. Verify each, record the verified signature in `docs/verified-apis.md` with a date and source URL, and only then implement.** - -Verify: - -- **Anypoint OAuth token endpoint** — exact path, region host variants (US/EU/Canada/Japan Hyperforce), connected-app scope names needed for Exchange read, API Manager write, and policy management. Some operations require an *admin* connected app with user context rather than pure client credentials. -- **LLM Proxy endpoint shape** — base URL format, whether `/v1` is included, auth header name (bearer vs. custom), whether an SLA/consumer credential pair or a single API key is used, streaming support, and which OpenAI request fields pass through vs. get stripped. -- **Token attribution headers** — the exact header names the gateway reads for business-group and client-application attribution. These are the single most important unknown; without them the SDK's core value proposition (cost attribution per agent) does not work. -- **Policy rejection response shapes** — status codes and bodies returned when token rate limiting, prompt-injection protection, content safety, or PII detection blocks a request. Capture real responses as fixtures (§8.2). -- **MCP Bridge provisioning API** — whether API Manager exposes a documented REST endpoint for creating MCP Bridge instances, or whether it is UI/wizard-only with an internal endpoint. **This determines whether §5 is viable at all.** If UI-only, fall back to wrapping the Terraform provider (§5.5). -- **Terraform provider coverage** — `mulesoft/anypoint` v1.x reportedly covers MCP servers and AI agent resources. Enumerate exactly what it already does before duplicating it. -- **Framework APIs** — every constructor in §3.3. `agent-framework` (Python) in particular changed its top-level class name recently; the August 2026 docs show `from agent_framework import Agent` with a `client=` kwarg, not `ChatAgent`. - -**Six further M0 items specific to governance and local mode are listed in §6.7, and six more for publication in §7.9. They gate §6 and §7 entirely — do all of them in the same pass.** - -If any verification fails, **stop and report** rather than inventing an endpoint. A fabricated endpoint that returns 404 in a customer's sandbox destroys trust in the whole package. - -### 0.4 Naming and legal - -`MuleSoft`, `Anypoint`, `Omni Gateway`, and `Agent Fabric` are Salesforce trademarks. **"Agent Fabric" is a specific MuleSoft product name, not a generic term**, so the project name "Agent Fabric SDK" reads as a first-party SDK for that product. That is fine — desirable, even — if this ships with MuleSoft's endorsement or as a MuleSoft-owned project. If it does not, the name will be read as an official-status claim, which is a real trademark exposure and will also confuse users about who supports it. - -Two workable paths: - -1. **Endorsed.** Confirm with MuleSoft (see the week-1 conversation in §10) and use the name as-is. -2. **Unaffiliated.** Keep the descriptive form in the docs — "an SDK for Agent Fabric" — but ship under a distinct, org-scoped project name so the package itself does not read as first-party. - -Working names in this document — `agent-fabric` (import `agent_fabric`) / `@yourorg/agent-fabric`, CLI `agent-fabric` — assume path 1. Under path 2, rename the distributions and keep the import name. Either way, put a support statement in the README stating exactly who maintains the project and what the support expectations are. - ---- - -## 1. Architecture - -### 1.1 Layer diagram - -``` -┌────────────────────────────────────────────────────────────────┐ -│ integrations/ (one optional extra per framework) │ -│ langgraph · adk · strands · agent_framework · openai · crewai │ -│ anthropic · llamaindex │ -│ — each returns NATIVE framework objects, never wrappers │ -└───────────────┬────────────────────────────────────────────────┘ - │ -┌───────────────▼───────────────┬────────────────┬───────────────┐ -│ llm/ │ registry/ │ tools/ │ -│ OpenAI-compatible client │ Exchange │ MCP session │ -│ factory + model catalog │ discovery, │ mgmt, tool │ -│ │ typed assets │ filtering │ -└───────────────┴───────────────┴────────────────┴───────────────┘ -┌────────────────────────────────────────────────────────────────┐ -│ core/ config · auth · transport · errors · telemetry · region │ -│ ZERO framework dependencies. httpx / fetch only. │ -└────────────────────────────────────────────────────────────────┘ -┌────────────────────────────────────────────────────────────────┐ -│ provisioning/ (separate entry point, CI-oriented) │ -│ declarative specs · plan/diff/apply · governance lint · CLI │ -└────────────────────────────────────────────────────────────────┘ -``` - -**Hard rule:** `core` has no dependency on any agent framework. `integrations/*` may depend on exactly one framework each. Nothing in `integrations/` may be imported by `core`, `llm`, `registry`, or `tools`. Enforce with an import-linter rule in CI. - -### 1.2 Repository layout - -Monorepo, two publishable trees. - -``` -agent-fabric-sdk/ -├── python/ -│ ├── pyproject.toml # hatchling; optional extras per framework -│ ├── src/agent_fabric/ -│ │ ├── __init__.py # Fabric, FabricConfig, exceptions -│ │ ├── core/ -│ │ │ ├── config.py # FabricConfig, env resolution, regions -│ │ │ ├── auth.py # AuthProvider protocol + AnypointConnectedApp -│ │ │ ├── transport.py # shared httpx.AsyncClient factory, hooks -│ │ │ ├── errors.py # exception taxonomy (§2.4) -│ │ │ ├── telemetry.py # OTel spans, correlation IDs -│ │ │ └── cache.py # TTL cache for tokens + registry lookups -│ │ ├── llm/ -│ │ │ ├── client.py # raw AsyncOpenAI-compatible client -│ │ │ └── catalog.py # ModelHandle, list/resolve models -│ │ ├── registry/ -│ │ │ ├── exchange.py # search, get asset, resolve -│ │ │ ├── governance.py # GovernanceCriteria, explain() -│ │ │ ├── publication.py # Publication, AssetType, descriptors -│ │ │ └── introspect.py # descriptor='auto' generation -│ │ │ └── models.py # AssetRef, McpServerHandle, AgentHandle -│ │ ├── policies/ # §6.10 — NOT core/; import-linter keeps it out -│ │ │ ├── base.py # PolicyPlugin protocol, PolicyRef, RequestContext -│ │ │ ├── registry.py # discovery, activation, ordering, collision checks -│ │ │ ├── attestation.py # §6.9.2 client half: canonical JSON, HMAC, codec -│ │ │ └── builtin/ # plugins for the four §6.9 companion policies -│ │ ├── tools/ -│ │ │ ├── session.py # MCP streamable-HTTP session mgmt -│ │ │ └── filter.py # allow/deny, tag + domain filtering -│ │ ├── integrations/ -│ │ │ ├── langgraph.py -│ │ │ ├── adk.py -│ │ │ ├── strands.py -│ │ │ ├── agent_framework.py -│ │ │ ├── openai_agents.py # fabric.openai — OpenAI Agents SDK -│ │ │ ├── anthropic.py -│ │ │ ├── crewai.py -│ │ │ └── llamaindex.py -│ │ └── provisioning/ -│ │ ├── spec.py # pydantic models for the YAML spec -│ │ ├── planner.py # desired vs. actual → Plan -│ │ ├── applier.py -│ │ ├── lint.py # governance ruleset preflight -│ │ ├── publish.py # Exchange publication, digest, --if-changed -│ │ ├── policy.py # `agent-fabric policy new|check` (§6.10.5) -│ │ └── cli.py # `agent-fabric` typer CLI -│ └── tests/ -│ ├── unit/ -│ ├── contract/ # respx fixtures from real responses -│ ├── conformance/ # SAME suite run against every adapter -│ └── integration/ # docker-compose local-mode gateway -├── typescript/ -│ ├── package.json # pnpm workspace root -│ └── packages/ -│ ├── core/ # @yourorg/agent-fabric-core -│ ├── langgraph/ -│ ├── adk/ -│ ├── llamaindex/ -│ ├── strands/ -│ └── cli/ -├── examples/ # one runnable example per framework -├── docs/ -│ ├── verified-apis.md # OUTPUT OF §0.3 — keep current -│ └── unsupported-boundary.md # §9.3 -└── .github/workflows/ -``` - -### 1.3 Language parity — be honest about this - -Of the eight targeted frameworks, TypeScript equivalents exist for only some: - -| Framework | Python | TypeScript | -|---|---|---| -| LangGraph | yes | yes (LangGraph.js) | -| Google ADK | yes | yes (`@google/adk`) | -| Strands | yes | yes (`@strands-agents/sdk`) | -| OpenAI Agents SDK | yes | yes (`@openai/agents`) | -| Anthropic SDK | yes | yes (`@anthropic-ai/sdk`) | -| LlamaIndex | yes | yes (LlamaIndex.TS) | -| CrewAI | yes | no | -| Microsoft Agent Framework | yes | no (.NET, Python, Go) | - -**Ship Python for all eight. Ship TypeScript for six** — LangGraph, ADK, Strands, OpenAI Agents SDK, Anthropic SDK, LlamaIndex — and add the Vercel AI SDK as one further TS-native target. Do not promise TS CrewAI or Microsoft Agent Framework in any README. - -### 1.4 Framework tiering — also be honest - -The roster deliberately drops AutoGen and Semantic Kernel: Microsoft's own documentation positions Agent Framework as the direct successor to both — built by the same teams, merging AutoGen's abstractions with Semantic Kernel's enterprise features — so carrying all three would mean shipping two sunset-path adapters. Microsoft Agent Framework covers that lineage; CrewAI, the OpenAI Agents SDK, and the Anthropic SDK take the freed capacity. - -Plan accordingly: - -- **Tier 1 (full support, conformance-gated, blocking CI):** LangGraph, Google ADK, Strands, Microsoft Agent Framework, OpenAI Agents SDK, Anthropic SDK, CrewAI. -- **Tier 2 (supported, conformance-gated, non-blocking CI):** LlamaIndex. -- **Tier 3:** none. (AutoGen and Semantic Kernel are out of scope — see the note above.) - -Two Tier 1 targets carry known constraints, tracked as documented conformance exemptions (§8.1) rather than as a lower tier: CrewAI reaches models through LiteLLM (like ADK), so per-run correlation degrades to per-client; and the Anthropic SDK depends on the proxy exposing an Anthropic-native Messages API route, which is an open M0 verification item (§0.3). - -This is a scope decision, not a technical one — raise it with the human lead before M1. - ---- - -## 2. `core` — the foundation - -### 2.1 Configuration - -```python -# src/agent_fabric/core/config.py -from dataclasses import dataclass -from typing import Literal - -Region = Literal["us", "eu", "ca", "jp"] - -@dataclass(frozen=True) -class FabricConfig: - # --- Anypoint control plane (registry + provisioning) --- - client_id: str | None = None # env: ANYPOINT_CLIENT_ID - client_secret: str | None = None # env: ANYPOINT_CLIENT_SECRET - org_id: str | None = None # env: ANYPOINT_ORG_ID - environment: str = "Sandbox" # env: ANYPOINT_ENV - region: Region = "us" # env: ANYPOINT_REGION - base_url: str | None = None # override; else derived from region - - # --- LLM proxy (data plane) --- - llm_proxy_url: str | None = None # env: AGENT_FABRIC_LLM_PROXY_URL - llm_proxy_key: str | None = None # env: AGENT_FABRIC_LLM_PROXY_KEY - - # --- Attribution (see §0.3 for real header names) --- - application_name: str | None = None # env: AGENT_FABRIC_APP_NAME - business_group: str | None = None # env: AGENT_FABRIC_BUSINESS_GROUP - - # --- Behaviour --- - timeout_s: float = 60.0 - max_retries: int = 3 - registry_cache_ttl_s: int = 300 - telemetry: bool = True - - @classmethod - def from_env(cls) -> "FabricConfig": ... - def validated(self) -> "FabricConfig": - """Raise ConfigError listing every missing field at once, not one at a time.""" -``` - -Resolution order: explicit kwarg → env var → `.agent-fabric.toml` in cwd or `$XDG_CONFIG_HOME` → default. Never read `.env` implicitly; document that the user calls `load_dotenv()` themselves. - -`validated()` must report **all** missing fields in one error. A config error that surfaces one missing variable per run is the most common source of first-five-minutes abandonment. - -### 2.2 Auth - -```python -class AuthProvider(Protocol): - async def token(self) -> str: ... - async def invalidate(self) -> None: ... - -class AnypointConnectedApp(AuthProvider): - """OAuth2 client_credentials against the Anypoint token endpoint. - - Caches the token in memory with a 60s safety margin before expiry. - On 401 from any downstream call, invalidate() then retry exactly once. - """ -``` - -Also implement `StaticToken` (for CI where a token is injected) and `ChainedAuth`. Keep the interface small so a customer can plug in their own vault. - -**Verify (§0.3):** the token endpoint path, and which operations require an admin connected app with user context. Document per-operation scope requirements in a table in `docs/`. - -The LLM proxy credential is **separate** from the control-plane credential and must not be conflated. A developer may legitimately have proxy access and no Exchange access. - -### 2.3 Transport — the single place headers get injected - -This is the most important piece of engineering in the SDK. Every framework has a different mechanism for setting request headers, and several have none. The solution is one shared HTTP client. - -```python -def build_http_client(cfg: FabricConfig, auth: AuthProvider | None) -> httpx.AsyncClient: - """Returns an httpx.AsyncClient with: - - request event hook injecting: - * correlation ID (uuid4 per logical agent run, propagated via contextvar) - * attribution headers (application, business group) [names: see §0.3] - * bearer token, refreshed lazily - - retry on 429/502/503/504 with exponential backoff + jitter, - honouring Retry-After when present - - NO retry on 4xx policy rejections (see §2.4) — those are terminal - - timeout from config - """ -``` - -Every framework adapter that accepts a custom HTTP client **must** be given this one. For frameworks that only accept a `default_headers` dict, pass a snapshot and accept that the correlation ID will be per-client rather than per-run — document the degradation explicitly per adapter in §3.3. - -Correlation ID lives in a `contextvar` so a single agent run's fan-out of model calls and tool calls share one trace ID end to end. Expose `with fabric.run_context(run_id=...)` for callers who want to supply their own. - -### 2.4 Error taxonomy - -Mapping gateway policy rejections to catchable, actionable exceptions is the SDK's clearest value over raw HTTP. Get the shape right even before the real status codes are verified. - -```python -class FabricError(Exception): - """Base. Carries correlation_id, request_id, raw response.""" - -class ConfigError(FabricError): ... -class AuthError(FabricError): ... # 401/403 on control plane - -class PolicyViolation(FabricError): - """Base for gateway-enforced refusals. NEVER retried.""" - policy: str # e.g. "token-rate-limit" - remediation: str # human-readable next step - -class TokenBudgetExceeded(PolicyViolation): ... # retry_after: float | None -class PromptInjectionBlocked(PolicyViolation): ... -class ContentSafetyBlocked(PolicyViolation): ... -class PIIDetected(PolicyViolation): - entities: list[str] # if the gateway reports them - -class UpstreamModelError(FabricError): ... # provider-side failure, retryable -class ToolInvocationError(FabricError): ... -class RegistryError(FabricError): ... -class ProvisioningError(FabricError): ... -``` - -Two design points that matter: - -1. **`PolicyViolation` must be distinguishable from a transient error at the framework boundary.** Most agent frameworks will retry or loop on an exception. A prompt-injection block that gets retried three times is three policy violations in the audit log and a confused developer. Every adapter must surface `PolicyViolation` in a way the host framework will not silently retry — for frameworks with middleware/hooks, install a hook that converts it to a terminal agent state. -2. **`remediation` is a required field.** "Token budget exceeded for business group `finance`; limit resets in 42m; request an increase in API Manager" is worth more than a stack trace. - -The concrete mapping from HTTP response → exception class lives in one function, `errors.classify(response)`, driven by a table populated from real captured fixtures (§8.2). Do not hand-write guesses into the table. - -`classify()` consults activated policy plugins (§6.10.3) before that table, so a custom policy can map its own rejection shape without an edit to `core/`. A plugin declines by returning `None`; it can never turn a rejection into a success (§6.10.4). With no plugins installed — the default, and a tested configuration — behaviour is exactly the fixture-driven table described above. - -### 2.5 Telemetry - -OpenTelemetry, optional dependency, off by default in libraries and on by default in the CLI. - -Spans: `fabric.llm.chat`, `fabric.registry.resolve`, `fabric.tool.call`, `fabric.provision.apply`. Attributes: model, proxy route, business group, application, token counts if the gateway returns them, policy decisions. - -The point is that a developer can correlate their local trace with what the platform team sees in Omni Gateway's observability view via the shared correlation ID. Say so in the docs — it is a headline feature. - ---- - -## 3. Pillar 1 — governed model access - -### 3.1 The one design rule - -**Adapters return native framework objects.** - -```python -model = fabric.langgraph.chat_model("gpt-4o") # -> a real ChatOpenAI instance -``` - -Not a `FabricChatModel` implementing `BaseChatModel`. Reasons: - -- Zero surface area to maintain when the framework adds features. -- Users keep every existing framework capability (structured output, streaming, caching, callbacks) with no gaps. -- Reviewers can read the returned object and know exactly what it does. -- If the SDK is later abandoned, users' code degrades to three lines of manual configuration, not a rewrite. - -The cost: no cross-framework portability of agent code. Accept it. Portability was never the ask. - -### 3.2 Public surface - -```python -from agent_fabric import Fabric - -fabric = Fabric.from_env() # or Fabric(config=FabricConfig(...)) - -# Raw, framework-free -client = fabric.llm.client() # AsyncOpenAI pointed at the proxy -models = await fabric.llm.list_models() # [ModelHandle(id, provider, ...)] - -# Per framework (each an optional extra) -fabric.langgraph.chat_model("gpt-4o", temperature=0.2) -fabric.adk.model("gpt-4o") -fabric.strands.model("gpt-4o") -fabric.agent_framework.chat_client("gpt-4o") -fabric.openai.model("gpt-4o") # OpenAI Agents SDK -fabric.anthropic.client() # AsyncAnthropic; model id is per-call -fabric.crewai.llm("gpt-4o") -fabric.llamaindex.llm("gpt-4o") -``` - -Accessing an adapter whose extra is not installed raises `ImportError` with the exact install command: -`pip install "agent-fabric[langgraph]"`. Implement via `__getattr__` on `Fabric` with a lazy import and a curated message. Do not let a bare `ModuleNotFoundError` escape. - -Every adapter accepts `**kwargs` forwarded verbatim to the underlying constructor, so users are never blocked by the SDK lagging a framework feature. - -### 3.3 Per-framework integration points - -Verified around August 2026. **Re-verify each before implementing (§0.3)** and record in `docs/verified-apis.md`. - -#### LangGraph / LangChain (Python + TS) - -```python -from langchain_openai import ChatOpenAI - -def chat_model(self, model: str, **kw) -> ChatOpenAI: - return ChatOpenAI( - model=model, - base_url=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - default_headers=self._attribution_headers(), - http_async_client=self._http_client(), # our client, our hooks - max_retries=0, # we retry in transport - **kw, - ) -``` - -Header injection: full (`default_headers` + custom http client). Best-case adapter. - -#### Google ADK (Python + TS) - -ADK is Gemini-first and reaches other providers through the `LiteLlm` wrapper (`google.adk.models.lite_llm.LiteLlm`), which takes LiteLLM-format model strings. - -```python -from google.adk.models.lite_llm import LiteLlm - -def model(self, model: str, **kw) -> LiteLlm: - return LiteLlm( - model=f"openai/{model}", # LiteLLM's OpenAI-compatible route - api_base=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - extra_headers=self._attribution_headers(), - **kw, - ) -``` - -Header injection: via LiteLLM's `extra_headers`. **Cannot inject our httpx client** — LiteLLM owns the transport. Consequence: retries and correlation-ID-per-run degrade. Document this. Consider a LiteLLM custom logger callback to recover trace correlation. - -ADK requires `litellm>=1.84`. Pin a floor, not a ceiling. - -#### CrewAI (Python; Tier 1) - -CrewAI reaches models through its own `crewai.LLM` class, which wraps LiteLLM — so, as with ADK, an OpenAI-compatible proxy is addressed with the `openai/` model prefix. - -```python -from crewai import LLM - -def llm(self, model: str, **kw) -> LLM: - return LLM( - model=f"openai/{model}", # LiteLLM's OpenAI-compatible route - base_url=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - extra_headers=self._attribution_headers(), - **kw, - ) -``` - -Header injection: via LiteLLM's `extra_headers`. **Cannot inject our httpx client** — LiteLLM owns the transport, exactly as with ADK. Consequence: retries and correlation-ID-per-run degrade to per-client. This is a documented, asserted conformance exemption (§8.1 `correlation_id_propagated`), shared with ADK. - -#### Microsoft Agent Framework (Python; .NET and Go out of scope for v1) - -Current Python surface is `from agent_framework import Agent` with `Agent(client=, name=..., instructions=...)`. The OpenAI-compatible chat client class name and its base-URL kwarg **must be verified** — this package is young and renamed classes recently. - -```python -def chat_client(self, model: str, **kw): - from agent_framework.openai import OpenAIChatClient # VERIFY name/path - return OpenAIChatClient( - model_id=model, # VERIFY kwarg - base_url=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - default_headers=self._attribution_headers(), - **kw, - ) -``` - -Agent Framework has first-class **middleware** for intercepting agent actions. Use it: ship `fabric.agent_framework.policy_middleware()` that catches `PolicyViolation` and terminates the run cleanly rather than letting the agent loop retry. This is the best policy-integration story of any of the eight — make it the flagship example. - -#### OpenAI Agents SDK (Python + TS; Tier 1) - -The OpenAI Agents SDK (pip `openai-agents`, import `agents`) models a provider as an `OpenAIChatCompletionsModel` wrapping an `AsyncOpenAI` client. Because we construct that client ourselves, header AND transport injection are both available. - -```python -from openai import AsyncOpenAI -from agents import OpenAIChatCompletionsModel - -def model(self, model: str, **kw) -> OpenAIChatCompletionsModel: - return OpenAIChatCompletionsModel( - model=model, - openai_client=AsyncOpenAI( - base_url=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - default_headers=self._attribution_headers(), - http_client=self._http_client(), - max_retries=0, # we retry in transport (§2.3) - ), - **kw, - ) -``` - -Header injection: full — the preferred pattern anywhere a framework accepts a pre-built OpenAI client. Pass the returned model into `agents.Agent(model=...)` and drive it with `Runner` per the SDK's own docs. - -#### Anthropic SDK (Python + TS; Tier 1) - -Returns a native `anthropic.AsyncAnthropic` client bound to the proxy. **Divergence, by design (§11.10 — the framework wins):** Anthropic's native surface is a *client*, and the model id is a per-call argument, so this adapter exposes `client()` rather than the `model(...)` factory the OpenAI-compatible adapters use. - -```python -from anthropic import AsyncAnthropic - -def client(self, **kw) -> AsyncAnthropic: - return AsyncAnthropic( - base_url=self._cfg.llm_proxy_url, # UNVERIFIED route — see note below - api_key=self._cfg.llm_proxy_key, - default_headers=self._attribution_headers(), - http_client=self._http_client(), - max_retries=0, - **kw, - ) -``` - -Header injection: full. **Open M0 dependency (§0.3):** the Omni Gateway LLM proxy is verified OpenAI-compatible; whether it also exposes an **Anthropic-native Messages API route** is unverified. If it does not, this adapter's requests will not reach a working upstream — so the SDK emits a one-time `UnverifiedValueWarning` on first use and keeps `base_url` overridable. Confirm the route in M0 before relying on this adapter. - -#### LlamaIndex (Python + TS) - -```python -from llama_index.llms.openai_like import OpenAILike - -def llm(self, model: str, **kw) -> OpenAILike: - return OpenAILike( - model=model, - api_base=self._cfg.llm_proxy_url, - api_key=self._cfg.llm_proxy_key, - is_chat_model=True, - is_function_calling_model=True, - default_headers=self._attribution_headers(), - **kw, - ) -``` - -**Gotcha:** `OpenAILike` defaults `is_chat_model=False`, which silently routes to the completions endpoint and fails against a chat-only proxy. Always set it. This is the single most common LlamaIndex-with-a-gateway bug. - -#### Strands Agents (Python + TS) - -```python -from strands.models.openai import OpenAIModel - -def model(self, model: str, **kw) -> OpenAIModel: - return OpenAIModel( - client_args={ - "base_url": self._cfg.llm_proxy_url, - "api_key": self._cfg.llm_proxy_key, - "default_headers": self._attribution_headers(), - "http_client": self._http_client(), - }, - model_id=model, - **kw, - ) -``` - -`client_args` is forwarded to the underlying OpenAI client, so header and transport injection are both available. Strands also has lifecycle **hooks** (`BeforeToolCallEvent` and friends) — use them for the same policy-termination pattern as Agent Framework. - -### 3.4 Model catalog - -`fabric.llm.list_models()` returns logical model names the proxy exposes, not raw provider names. Prefer the proxy's own `/models` endpoint if it has one; fall back to the registry. Cache with the configured TTL. - -`ModelHandle` should carry enough capability metadata to feed any framework that requires explicit capability flags and to let a developer branch on function-calling support. If the platform does not expose capability metadata, ship a small bundled JSON capability table keyed by well-known model IDs, clearly marked as a heuristic, and let users override it. - ---- - -## 4. Pillar 2 — governed tool access (the differentiating feature) - -This is the feature that makes someone install the package. Prioritise it accordingly. - -### 4.1 Target developer experience - -```python -tools = await fabric.tools.discover(domain="hr", tags=["approved"]) -agent = create_react_agent(fabric.langgraph.chat_model("gpt-4o"), tools.langgraph()) -``` - -Two lines from "our enterprise has a governed tool catalog" to "my LangGraph agent can use it." Everything in this section exists to make those two lines work. - -Discovery is also the **filter and search** entry point. The same call narrows the catalog by name/description (glob), governance, domain, tags, and asset type, so an agent binds only the tools it needs rather than the entire catalog: - -```python -# governed-only + name search: only governed tools whose name/description matches "*accounts*" -tools = await fabric.tools.discover(governed_only=True, search="*accounts*") - -# full filter surface -tools = await fabric.tools.discover( - search="*accounts*", # glob over asset name + description; None = no text filter - governed_only=True, # True = default criteria; or a GovernanceCriteria (§6.1) - domain="hr", - tags=["approved"], - asset_types=["mcp"], - environment="Production", - limit=50, -) -``` - -`fabric.tools.discover(...)` is the high-level facade over `ExchangeRegistry.search()` (§4.2). On the facade, `search` is the text/glob filter (maps to the registry's `query`) and `governed_only` is the governance predicate (maps to the registry's `governed`) — the facade uses the clearer names. **Verify (M0/M2):** whether Exchange supports a server-side wildcard/substring query, or the glob must be applied client-side after a coarse Exchange query — see §4.2. - -### 4.2 Registry discovery - -```python -class ExchangeRegistry: - async def search( - self, *, query: str | None = None, asset_types: list[AssetType] | None = None, - tags: list[str] | None = None, domain: str | None = None, - environment: str | None = None, limit: int = 50, - ) -> list[AssetRef]: ... - - async def resolve_mcp(self, ref: AssetRef | str) -> McpServerHandle: ... - async def resolve_agent(self, ref: AssetRef | str) -> AgentHandle: ... -``` - -`query` matches against asset **name and description** and must accept a glob (`*accounts*`, `get_*`). Apply it server-side if Exchange exposes a wildcard/substring search parameter; otherwise fetch the coarse candidate set (by `asset_types`/`tags`/`domain`) and filter the glob client-side. **Verify in M0** which path Exchange supports; either way the developer-facing `search=` behaviour is identical. - -`AssetRef` is `group_id/asset_id/version` plus name, type, tags, description, and the Exchange URL. Accept a shorthand string form (`"com.acme/vendor-shipment-mcp/1.0.0"`) everywhere a ref is taken. - -`McpServerHandle` carries the consumer endpoint URL, transport type, auth requirements, and the tool descriptors if the registry exposes them without a live connection. - -Caching: registry lookups are cached with `registry_cache_ttl_s`. Provide `fabric.registry.refresh()` and honour a `FABRIC_NO_CACHE=1` escape hatch. - -**Pinning matters.** Resolving `latest` at agent startup means a platform-team asset change silently alters agent behaviour in production. Default to requiring an explicit version; allow `version="latest"` only with a logged warning. Ship `fabric.tools.lock()` which writes a `fabric.lock` file of resolved versions and digests, and `discover(locked=True)` which refuses to resolve anything not in the lockfile. Teams shipping agents to production will need this and nobody else provides it. - -### 4.3 MCP session management - -MCP servers created by MCP Bridge are gateway endpoints speaking streamable HTTP, protected by gateway policies. The session layer must handle: - -- **Auth.** Client-credentials OAuth is the machine-to-machine case. Strands' `MCPClient` already builds streamable HTTP with a `client_credentials` grant internally; other frameworks need headers supplied. Provide `McpServerHandle.auth_headers()` returning a ready dict, refreshed on 401. -- **Connection lifecycle.** MCP clients are stateful and several frameworks connect lazily. Do not open connections in `discover()`. Return handles; connect on first tool use. -- **Multi-server aggregation.** `ToolSet` wraps N `McpServerHandle`s. Tool-name collisions across servers must be resolved by prefixing with the server's short name, and the mapping surfaced in `ToolSet.name_map` so a developer can debug why the model called `hr__get_employee`. -- **Filtering.** `ToolSet.filter(allow=[...], deny=[...], predicate=fn)`. Enterprise MCP servers can expose dozens of tools; handing 60 tool descriptors to a model degrades it and inflates token cost. Make filtering prominent in the docs, and log the descriptor token count at debug level. - -### 4.4 Per-framework tool binding - -| Framework | Binding | -|---|---| -| LangGraph | `langchain_mcp_adapters.client.MultiServerMCPClient({...}).get_tools()` — build the connection dict from handles, transport `"streamable_http"`, headers injected | -| Google ADK | `McpToolset(connection_params=StreamableHTTPConnectionParams(url=..., headers=...), tool_filter=[...])`; pass the toolset object straight into `LlmAgent(tools=[...])` | -| MS Agent Framework | its MCP client/tool class for streamable HTTP — **verify name**; docs reference hosted MCP tools and MCP clients for tool integration | -| OpenAI Agents SDK | `agents.mcp.MCPServerStreamableHttp(params={"url": ..., "headers": ...})`; pass into `agents.Agent(mcp_servers=[...])` — **verify class name** | -| Anthropic SDK | streamable-HTTP MCP servers supplied via the SDK's `mcp_servers` tool integration — **verify name** | -| CrewAI | `crewai_tools.MCPServerAdapter(server_params)` yielding native CrewAI tools — **verify class name** | -| LlamaIndex | `llama_index.tools.mcp.BasicMCPClient` + `McpToolSpec(...).to_tool_list_async()` | -| Strands | `MCPClient(lambda: streamablehttp_client(url, headers=...))`; implements `ToolProvider`, so it can be passed directly into `Agent(tools=[...])` with automatic lifecycle management | - -Each returns the framework's native tool type. `ToolSet` exposes one method per installed integration: - -```python -ts = await fabric.tools.discover(domain="hr") -ts.langgraph() # -> list[BaseTool] -ts.adk() # -> list[McpToolset] -ts.strands() # -> list[MCPClient] -ts.llamaindex() # -> list[FunctionTool] -# etc. -``` - -Note the shape difference: ADK and Strands want toolset/provider objects, LangGraph and LlamaIndex want flat tool lists. Do not force a uniform return type; match each framework's idiom and document the difference in the method docstring. - -### 4.5 Agent handles (A2A) - -Secondary but cheap. Agent Broker is an A2A server, and A2A-compliant agents in the registry can be consumed from Python. Ship `AgentHandle.as_tool()` which wraps a remote A2A agent as a callable tool in each framework, using the community `a2a-sdk` for protocol handling rather than implementing JSON-RPC yourself. - -This is what lets a Python agent delegate to an Agent Broker without the developer learning A2A. Keep it in scope for M2 but behind a `[a2a]` extra. - ---- - -## 5. Pillar 3 — provisioning as code - -**Gate:** do not start M3 until §0.3 has confirmed a usable MCP Bridge provisioning API. If it is UI-only, jump to §5.5. - -### 5.1 Spec format - -A single declarative YAML file, versioned in the user's repo, validated by pydantic models. `Governance.export()` (§6.3) emits fragments of exactly this format — the two features share one schema, and that is deliberate. Do not let them diverge. - -```yaml -apiVersion: fabric/v1 -kind: FabricSpec -metadata: - name: hr-agent-tools - environment: Sandbox - businessGroup: 5a1b... - -mcpBridges: - - name: hr-tools - gateway: managed-omni-eu-1 - apis: - - assetId: employee-api - version: 1.2.0 - upstream: https://internal.acme.com/employees - tools: - - name: get_employee - method: GET - resource: /employees/{id} - description: Fetch an employee record by ID. - inputSchema: auto # derive from the OAS/RAML spec - - name: search_employees - method: GET - resource: /employees - description: Search employees by department or title. - inputSchema: - type: object - properties: - department: {type: string} - required: [department] - policies: - - assetId: rate-limiting-sla-based - version: 1.2.0 - config: - rateLimits: - - maximumRequests: 100 - timePeriodInMilliseconds: 60000 -``` - -Design notes: - -- **`inputSchema: auto`** derives the schema from the API's published spec in Exchange. This is the highest-leverage feature in the whole provisioning module — it is what makes bulk conversion of 200 APIs tractable. Implement it as a local OAS/RAML → JSON Schema transform so it can run offline in CI and produce a reviewable diff. -- **DataWeave is a hard boundary.** MCP Bridge's HTTP mapping uses DataWeave expressions for non-trivial parameter extraction. The spec must accept a raw `httpMapping.dataweave` string passthrough field and make no attempt to generate or parse DataWeave. Document this clearly as a limitation. -- **No secrets in the spec.** Reference them: `upstreamAuth: ${secret:anypoint-secrets-manager/hr-api-key}`. Resolve at apply time. - -### 5.2 Plan / apply - -``` -$ agent-fabric plan -f fabric.yaml - + mcpBridge hr-tools (create) - ~ mcpBridge finance-tools (update: +1 tool, ~1 description) - + tool list_invoices - ~ tool get_invoice description changed - - policy rate-limiting-sla-based on hr-tools (remove) - -3 changes. Run `apply` to proceed. - -$ agent-fabric apply -f fabric.yaml --auto-approve -``` - -Requirements: - -- **Read-before-write, always.** Fetch current state, diff, render, then apply. Never blind-PUT. -- **Idempotent.** Re-running `apply` with no spec change produces zero API calls that mutate. -- **`--dry-run` and `--out plan.json`** for CI gating. The GitHub Action posts the plan as a PR comment; `apply` runs only on merge. -- **Partial-failure handling.** Apply resources in dependency order, stop on first failure, report exactly what was applied and what was not. Do not attempt automatic rollback — report state and let the operator re-plan. Silent partial rollback in a control plane is worse than a clear stop. -- **Drift detection.** `agent-fabric drift` compares live state against the spec and exits non-zero. Run it on a schedule; it catches console changes that bypass the pipeline. - -### 5.3 Governance lint - -Governance rulesets resolve locally with per-rule severities. Wire `agent-fabric lint` to validate API specs against project and centralized rulesets **before** anything is published, and fail the PR on `error` severity. - -This is small, cheap, uncontroversial, and the fastest way to get a platform team to say yes to the SDK. Build it in M1 even though the rest of provisioning is M3. - -### 5.4 Policy ownership — the political constraint - -Applying policies from application code inverts the ownership model platform teams bought Omni Gateway for. Design so that: - -- The **spec file lives in the repo** and is reviewable. -- **Apply runs in CI** under a connected app whose scopes the platform team controls. -- The SDK supports an **allow-list catalog**: `fabric.yaml` can reference only policy assets the platform team has approved, enumerated in a separate `policy-catalog.yaml` that the platform team owns in a different repo. - -Ship the allow-list mechanism in v1 even if nobody asks for it. Its absence is what gets the SDK banned in a security review. - -**A policy plugin (§6.10) is not an approval.** Installing and enabling a plugin teaches the SDK to speak a custom policy's wire contract — emit its headers, type its rejections, validate its config. It grants nothing. A policy absent from `policy-catalog.yaml` is still refused by `agent-fabric apply`, plugin or no plugin. Keep the two mechanisms in separate files with separate owners, and say so in the docs, because the first reviewer to mistake one for the other will conclude the allow-list is bypassable. - -### 5.5 Fallback if there is no provisioning API - -If §0.3 finds MCP Bridge is wizard-only: - -1. Cut §5 from v1 entirely. Do not reverse-engineer internal endpoints — they will break, and doing so in an enterprise product is a support liability. -2. Keep §5.3 (lint), which needs no provisioning API. -3. Emit **Terraform** from the same `fabric.yaml` spec (`agent-fabric generate --target terraform`), and let the official provider do the applying. The `inputSchema: auto` derivation is still the valuable part, and it survives this pivot intact. - -Option 3 is a genuinely good outcome. Do not treat it as a failure mode. - ---- - -## 6. Governance profiles and environment targeting - -Two requested capabilities live here: **governed-only discovery** (§6.1) and the **governance object** attached to agent and MCP-server construction (§6.2 onward). Both are worth building. The second needs one structural change and one corrected expectation before it is safe to implement — read §6.3 and §6.4 before writing code. - -### 6.1 Governed-only discovery - -#### 6.1.1 "Governed" is not a flag — it is a computed predicate - -Exchange is a catalog. Publication to Exchange says nothing about whether an asset is fronted by a gateway, has policies applied, or passes the org's rulesets. There is no single boolean to query. "Governed" must be computed by joining state across three systems, and the join is environment-scoped: an asset governed in Production may be ungoverned in Sandbox. - -```python -# src/agent_fabric/registry/governance.py -from dataclasses import dataclass, field - -@dataclass(frozen=True) -class GovernanceCriteria: - """Definition of 'governed'. Every field is a separate, independently - checkable condition. Defaults are deliberately moderate; orgs override.""" - - require_api_instance: bool = True # an API Manager instance exists in this env - require_deployed: bool = True # the instance is deployed to a gateway, not just configured - require_any_policy: bool = True # at least one policy is applied - required_policies: list[str] = field(default_factory=list) - # asset IDs that MUST be present, e.g. - # ["client-id-enforcement", "rate-limiting-sla-based"] - forbidden_policies: list[str] = field(default_factory=list) - require_governance_pass: bool = False # passes org governance rulesets with no `error` findings - require_gateways: list[str] = field(default_factory=list) - # only assets behind these named gateways - require_tags: list[str] = field(default_factory=list) - require_lifecycle: list[str] = field(default_factory=list) - # e.g. ["published", "approved"] if Exchange exposes lifecycle - allow_unknown: bool = False # if a check cannot be evaluated, does the asset pass? - -STRICT = GovernanceCriteria( - require_governance_pass=True, - required_policies=["client-id-enforcement"], - allow_unknown=False, -) -``` - -`allow_unknown` matters more than it looks. If the platform does not expose, say, ruleset results through an API, `require_governance_pass=True` with `allow_unknown=False` silently filters the entire catalog to zero results and the developer has no idea why. Every filtered-out asset must therefore carry a reason. - -#### 6.1.2 API surface - -```python -tools = await fabric.tools.discover(domain="hr", governed_only=True) # default criteria -tools = await fabric.tools.discover(domain="hr", governed_only=STRICT) # explicit criteria -tools = await fabric.tools.discover(search="*accounts*", governed_only=True) # + name/desc glob -assets = await fabric.registry.search(query="*accounts*", asset_types=["mcp"], governed=True) - -# Always available for debugging: -report = await fabric.registry.explain(ref, criteria=STRICT) -# GovernanceReport(governed=False, checks=[ -# Check("api_instance_exists", True, "instance 19283 in Sandbox"), -# Check("deployed", True, "gateway managed-omni-eu-1"), -# Check("required_policies", False, "missing: client-id-enforcement"), -# Check("governance_pass", None, "UNKNOWN: rulesets API returned 403"), -# ]) -``` - -`explain()` is not optional polish. Without it, `governed=True` returning an empty list is indistinguishable from a broken credential, and the SDK gets blamed. Make it a first-class documented method and reference it in the empty-result warning message. - -**Default:** `governed_only` on the facade (and `governed` on `registry.search()`) defaults to `None` (no filtering) in v1, with a startup log line stating that discovery is unfiltered. Both accept `True` (default `GovernanceCriteria`) or an explicit `GovernanceCriteria` such as `STRICT`. Flipping the default to `True` is a breaking change for a later major version — consider it, but do not surprise people in a minor release. - -#### 6.1.3 Implementation — avoid the N+1 - -Naive implementation queries API Manager once per Exchange asset. On a 500-asset catalog that is 500 sequential calls and a discovery step that takes minutes. - -Correct approach: - -1. One call to list **all** API Manager instances for `(org, environment)`. Build an in-memory index keyed by `(groupId, assetId, version)` — and also by `(groupId, assetId)` for version-agnostic matching, since instance version and asset version can diverge. -2. One call per instance-set to fetch applied policies, or a single bulk policies call if one exists. **Verify in M0** whether policies can be fetched in bulk; if not, fetch policies only for the candidate assets that survived steps 1 and the cheap Exchange-side filters. -3. Apply Exchange-side filters (tags, lifecycle, asset type) first, before any API Manager calls, to shrink the candidate set. -4. Cache the whole index under `registry_cache_ttl_s`, keyed by environment. - -Expose `fabric.registry.warm(environment=...)` so a long-running agent process can build the index at startup rather than on first discovery. - -**Verify in M0:** whether "deployed to a gateway" is readable per instance, and whether governance ruleset results are exposed through an API at all. If ruleset results are UI-only, ship `require_governance_pass` as permanently `UNKNOWN` and say so in the docstring rather than quietly dropping the field. - ---- - -### 6.2 The governance object — what it is - -```python -from agent_fabric import Governance, GatewayTarget, PolicyBinding - -gov = Governance( - name="hr-agent", - gateway=GatewayTarget.from_env(), # resolves local | sandbox | prod - policies=[ - PolicyBinding("rate-limiting-sla-based", "1.4.0", config={ - "rateLimits": [{"maximumRequests": 100, "timePeriodInMilliseconds": 60000}], - }), - PolicyBinding("openai-token-policy", "1.0.0", config={ - "maxTokensPerMinute": 50_000, - }), - PolicyBinding("prompt-injection-protection", "1.0.0"), - ], -) - -model = fabric.langgraph.chat_model("gpt-4o", governance=gov) -tools = await fabric.tools.discover(domain="hr", governed=True, governance=gov) -``` - -`GatewayTarget` is the environment-varying part: - -```python -@dataclass(frozen=True) -class GatewayTarget: - mode: Literal["local", "managed", "self-managed"] - base_url: str # http://localhost:8081 | https://.gw... | https:// - environment: str | None = None # None for local - gateway_name: str | None = None # control-plane gateway/registration name - connected: bool = True # local mode gateways are unconnected - - @classmethod - def from_env(cls) -> "GatewayTarget": - """FABRIC_TARGET=local|sandbox|production selects a profile from - .agent-fabric.toml. Local requires no Anypoint credentials beyond - whatever registration the local gateway itself needs (§6.4).""" -``` - -Profiles live in `.agent-fabric.toml`, committed to the repo: - -```toml -[targets.local] -mode = "local" -base_url = "http://localhost:8081" - -[targets.sandbox] -mode = "managed" -base_url = "https://hr-agent.sandbox.eu1.gw.mulesoft.com" -environment = "Sandbox" -gateway_name = "managed-omni-eu-1" - -[targets.production] -mode = "self-managed" -base_url = "https://gw.internal.acme.com" -environment = "Production" -gateway_name = "prod-k8s-omni" -``` - -`PolicyBinding` takes any policy, stock or custom. When an activated plugin (§6.10) matches the `assetId`, construction additionally runs that plugin's `validate_config`, so a bad config fails on the laptop rather than at `agent-fabric apply`. With no matching plugin the binding is still valid — the SDK does not require a plugin to declare a policy, only to speak its wire contract. - -This part of the request is sound and maps cleanly onto a pattern developers already know from Terraform workspaces and Kustomize overlays. Build it as specified. - ---- - -### 6.3 Structural change: one object, three verbs — not one deploy - -The request as stated is "set the governance object, then deploying it applies it locally / in sandbox / in prod." That is the right ergonomics and the wrong lifecycle, for the reason already established in §5.4: if application code applies policies to a shared gateway, the app team has taken the platform team's job, and the SDK gets rejected in security review. - -The fix keeps the ergonomics. **The same `Governance` object supports three verbs with three different trust levels:** - -| Verb | Target | Who runs it | What it does | -|---|---|---|---| -| `simulate()` | local | developer, laptop | Starts an ephemeral local Omni Gateway, applies the config, returns a live `base_url`. Nothing shared is touched. | -| `export()` | sandbox, prod | developer, laptop | Compiles the object into a `fabric.yaml` fragment (§5.1). Writes a file. Touches nothing. | -| `resolve()` | sandbox, prod | agent process, runtime | **Read-only.** Looks up the already-provisioned gateway route, verifies the expected policies are actually applied, returns the `base_url`. Raises `GovernanceDrift` if reality does not match the declaration. | - -There is deliberately **no `apply()` on the runtime object.** Applying to sandbox and production goes through `agent-fabric apply` in CI, against the reviewed spec, under platform-controlled credentials, filtered by the allow-list catalog (§5.4). The path is: - -``` -agent code declares Governance - → export() → fabric.yaml → PR review - → CI: agent-fabric plan → posted as PR comment - → merge: agent-fabric apply → gateway provisioned - → runtime: resolve() → verifies + returns URL -``` - -`resolve()` is the piece that makes this feel like the user's original request rather than a bureaucratic downgrade. The developer still writes one governance object; at runtime it *checks* rather than *mutates*, and a mismatch fails loudly at startup instead of silently running ungoverned. That verification is a genuinely better property than blind application, and it should be marketed as such. - -**Escape hatch:** `Governance.apply(target, i_am_the_platform_team=True)` may exist for platform teams automating their own gateways. Make the kwarg name that explicit, require the connected app to hold write scopes it will not have by default, and log every use at WARNING. - ---- - -### 6.4 Corrected expectation: local is not sandbox with a different URL - -The request states the difference between local and sandbox/prod is the URL. **It is not, and building on that assumption will produce a local dev loop that passes and a sandbox deploy that fails.** Three material differences: - -**1. Local Mode and Connected Mode are different feature sets.** Local Mode gateways are configured by declarative YAML files on disk and do not talk to the control plane. Anything control-plane-dependent is therefore unavailable locally. Verify the exact list in M0, but expect at minimum: - -| Capability | Local Mode | Connected Mode | -|---|---|---| -| Routing, CORS, headers, TLS | yes | yes | -| Basic rate limiting / spike control | yes | yes | -| Custom WASM (PDK) policies | yes | yes | -| SLA-based rate limiting, client-ID enforcement, contracts | **no** (needs API Manager client apps) | yes | -| **LLM Proxy** | **probably not** — verify | yes | -| **MCP Bridge** | **probably not** — delivered as an API Manager guided experience | yes | -| Control-plane analytics, token usage reporting | no | yes | - -If LLM Proxy and MCP Bridge are Connected-Mode-only, the local gateway **cannot** be a faithful stand-in for the two capabilities this SDK is mostly about. That is the single most important finding M0 must produce. Plan for it now: - -- **If confirmed unavailable locally:** `simulate()` still validates routing, auth, rate limiting, custom policies and request/response shape — real value for the dev loop — but LLM traffic is served by a **local mock proxy** the SDK ships (an OpenAI-compatible stub that replays fixtures and simulates policy rejections from §8.2). Label it unambiguously in logs: `LOCAL SIMULATION — LLM proxy is mocked, not a real gateway`. Never let a developer believe they tested the real thing. -- **If available locally:** great, use it, and drop the mock to a fallback. - -**A third option this section originally missed.** "Local Mode" and "runs on my laptop" are **orthogonal axes**, and conflating them is what produced the false dichotomy above. Omni Gateway is Flex Gateway (verified: API type `flexGateway`, v1.13.2), and a Flex Gateway can run **self-managed in Connected Mode anywhere a container runs** — including a developer's laptop and a CI runner — drawing its configuration from API Manager rather than from YAML on disk. In that topology the LLM Proxy policies, `client-id-enforcement`, SLA-based rate limiting and control-plane analytics are all **real**, because it is the same gateway binary and the same control plane as production. It costs an Anypoint org, a dev environment, and a per-run API instance. See §6.8 — it is the highest-fidelity local loop available, and it makes the "LLM Proxy is Connected-Mode-only" finding a cost question rather than a capability ceiling. - -**2. Policies are not portable across modes.** Classify every policy in a shipped table: - -```python -class PolicyPortability(Enum): - BOTH = "both" - CONNECTED_ONLY = "connected_only" - LOCAL_ONLY = "local_only" - UNKNOWN = "unknown" -``` - -`simulate()` must print, before it starts, exactly which declared policies are being skipped and why: - -``` -Starting local Omni Gateway (target: local) - applied cors 1.0.0 - applied rate-limiting 1.3.0 - applied acme-custom-redaction 0.2.0 (WASM) - SKIPPED rate-limiting-sla-based 1.4.0 connected-mode only (needs API Manager client apps) - SKIPPED prompt-injection-protection 1.0.0 connected-mode only - ! 2 of 5 policies are not exercised locally. Local pass does not imply sandbox pass. -``` - -A silent skip here is the worst possible failure mode for this feature. Make the warning loud, non-suppressible by default, and repeated in the summary at teardown. - -**3. Identity and secrets differ.** No connected app, no client credentials, no Secrets Manager locally. Ship a `LocalCredentialShim` that generates throwaway credentials and injects them into both the gateway config and the SDK client, so `${secret:...}` references in the spec resolve to dev values from a gitignored `.agent-fabric.local.toml`. - ---- - -### 6.5 `simulate()` — the local harness - -```python -async with gov.simulate() as env: # mode="local" by default - model = fabric.langgraph.chat_model("gpt-4o", gateway=env.gateway) - tools = await fabric.tools.discover(domain="hr", gateway=env.gateway) - agent = create_react_agent(model, tools.langgraph()) - result = await agent.ainvoke({"messages": [("user", "…")]}) -assert env.policy_events("rate-limiting").count == 1 -``` - -`simulate()` takes a `mode` selecting one of three fidelity rungs — `"mock"`, `"local"`, `"connected"` — described in §6.8. The rest of this section covers the `"local"` rung: a Local Mode gateway from declarative YAML. **`env.fidelity` must always be queryable**, so a test that requires a real LLM proxy can assert it got one rather than silently passing against a mock. - -Implementation: - -1. Render the `Governance` object to Local Mode declarative YAML into a temp dir. -2. `docker compose up` the Omni Gateway image plus, if needed, the mock LLM proxy and any upstream mocks. Wait for readiness on the health endpoint with a bounded timeout and a clear error if the image cannot be pulled. -3. Yield an `Environment` exposing `gateway` (a `GatewayTarget` pointing at the mapped localhost port), `logs()`, and `policy_events(policy_name)` parsed from gateway logs so tests can assert a policy actually fired. -4. Tear down, unless `FABRIC_KEEP_LOCAL=1`. - -Design constraints that matter: - -- **Port allocation must be dynamic**, not hardcoded 8081, so parallel test workers do not collide. -- **Prefer config hot-reload over container restart** between test cases. Container startup measured in seconds is fine for a dev loop and unacceptable in a test matrix. Make the fixture session-scoped by default with per-test config reload. -- **Do not run this in unit tests.** Gate behind a pytest marker (`@pytest.mark.local_gateway`) that is off by default. -- **Verify licensing in M0.** Omni Gateway local mode likely requires a registration artifact obtained from the control plane. If so, contributors and CI runners without an Anypoint org cannot run `simulate()` at all — which changes it from "the default dev loop" to "an opt-in dev loop for licensed users," and the mock proxy becomes the primary local experience. This is a gating question for the feature's value, not a detail. - -**Docker is a hard dependency for this feature only.** It must remain an optional extra (`[local]`) and the rest of the SDK must work without it. - ---- - -### 6.6 MCP servers written in Python or TypeScript - -Worth stating plainly, because the request implies it: if a developer writes an MCP server *in their agent framework* and attaches a governance object, the SDK can put a gateway in front of it locally, and can generate the gateway configuration for sandbox and production. **It cannot deploy the server itself to sandbox or production.** CloudHub 2.0 runs Mule applications; there is no MuleSoft-hosted runtime for a Python or TypeScript process. - -So the honest split: - -| | Local | Sandbox / Prod | -|---|---|---| -| The MCP server process | SDK runs it in docker-compose | **User's own platform** — K8s, Cloud Run, Lambda, ECS | -| The gateway in front of it | SDK spins up local Omni Gateway | SDK exports config; CI applies it | -| Upstream URL in the config | container DNS name | user-supplied, from their deployment | - -To make the gap less painful, `agent-fabric export --with-runtime` may emit a Dockerfile and a plain Kubernetes manifest as a starting point. Frame it as a convenience, never as a deployment platform. Do not build a deployment orchestrator — that is a different product, and §0.2 exists to stop this kind of drift. - ---- - -### 6.7 Verification checklist added to M0 - -Add to §0.3: - -- Can Local Mode run the LLM Proxy? Can it run MCP Bridge? **(gating for §6.4/§6.5)** -- Does Local Mode require a control-plane-issued registration or licence artifact? Can CI run it? -- Which policies are Connected-Mode-only? Produce the portability table in §6.4 from real data, not inference. -- Is "deployed to gateway" readable per API instance via a documented API? **(gating for `require_deployed`)** -- Are governance ruleset results exposed via API? **(gating for `require_governance_pass`)** -- Can applied policies be fetched in bulk for an environment, or only per instance? **(determines whether §6.1.3 is fast or slow)** -- Can a Flex Gateway register in **Connected Mode from a laptop or a CI runner**, and what does the registration artifact contain? **(gating for §6.8 — the highest-fidelity local loop)** -- Can an API instance be created and destroyed per developer or per CI run cheaply, and does it count against a licensed instance quota? **(determines whether §6.8 is per-developer or one shared dev gateway)** - ---- - -### 6.8 Local testing against a self-managed Connected Mode gateway - -§6.4 and §6.5 assume a local gateway means a **Local Mode** gateway. That assumption produced a genuine gap, because *how a gateway is configured* and *where it runs* are independent: - -| | Configured by local YAML (**Local Mode**) | Configured by the control plane (**Connected Mode**) | -|---|---|---| -| **Runs remotely** — private space, cloud, customer K8s | rare, but possible | `mode="managed"` / `mode="self-managed"` — what §6.2 already models | -| **Runs on a laptop or CI runner** | §6.5 `simulate()` | **this section — nothing covered it** | - -The bottom-right cell is the important one. Omni Gateway is Flex Gateway, and a Flex Gateway container registered to the control plane in Connected Mode behaves identically wherever it runs: it fetches its API instances and policies from API Manager. Run that container on a laptop against a dedicated dev environment and the LLM Proxy policies (`llm-proxy-core`, `model-based-routing`, `openai-transcoding-policy`), `client-id-enforcement`, SLA-based rate limiting and token-usage analytics are **all real**. - -**This changes the shape of the project's biggest governance risk.** §10 rates "Local Mode cannot run LLM Proxy or MCP Bridge" as Medium-high likelihood, High impact, with the outcome "`simulate()` becomes a mock, not a replica." A connected self-managed local gateway makes that a **cost** question rather than a **capability ceiling** — the capability exists locally; it just requires an org and a dev environment. - -#### 6.8.1 The three-rung fidelity ladder - -| Rung | Gateway | Config from | LLM Proxy | Connected-only policies | `resolve()` testable | Needs an org | Offline | -|---|---|---|---|---|---|---|---| -| `mock` | none — in-process stub | fixtures (§8.2) | mocked | none | no | no | yes | -| `local` | Flex Gateway container, Local Mode | declarative YAML | probably not — M0 | **no** | **no** — no control plane | no* | yes | -| `connected` | **Flex Gateway container, Connected Mode, self-managed** | **API Manager** | **real** | **real** | **yes** | yes + dev env | no | - -\* subject to the §6.7 licence-artifact question. - -```python -async with gov.simulate(mode="connected") as env: - assert env.fidelity == "connected" # never a silent downgrade - model = fabric.langgraph.chat_model("gpt-4o", gateway=env.gateway) - resp = await model.ainvoke("hello") - - # Only reachable on this rung: real policies, real drift detection, real usage. - await gov.resolve(fabric) # exercises GovernanceDrift - assert env.policy_events("llm-token-rate-limit").count == 0 - assert env.usage().total_tokens > 0 # real attribution data -``` - -**No new types are needed.** §6.2's `GatewayTarget` already expresses this: `mode="self-managed"`, `connected=True`, `base_url="http://localhost:8081"`. The `connected` flag was introduced to carry exactly this distinction. What is missing is the harness, the lifecycle management and the documentation — not the model. - -#### 6.8.2 Two capabilities this unlocks that no other rung can - -**`resolve()` becomes locally testable.** In Local Mode there is no control plane, so there are no API Manager instances, no applied-policy list, and `GovernanceDrift` detection (§6.3) cannot be exercised *at all* — not in a dev loop, not in CI, only against a shared sandbox. On the `connected` rung there are real instances with real applied policies, so `governance_resolve_drift` becomes a first-class local test. - -**Attribution headers become verifiable.** The business-group attribution header name is the **single highest-priority M0 unknown** (§0.3) and it gates every governed call across all nine adapters. Verifying it requires a real gateway that actually reads the header and real analytics that report the attributed usage. A connected local gateway provides both, on a laptop, against a dev environment — which is a far tighter loop than probing a shared sandbox. - -#### 6.8.3 Lifecycle is the hard part, not the container - -Starting the container is easy. Not leaving debris in the control plane is not. - -**Ephemeral API instances.** Each run needs an API instance in the dev environment. Fifty developers times several runs a day produces orphaned instances indefinitely. So: deterministic naming (`fabric-dev--`), create-if-absent rather than create-always, teardown on exit **and on exception**, and a **TTL reaper** (`agent-fabric dev reap`) for instances leaked by a hard kill. Report what was reaped; never silently delete something a developer is using. - -**A hard environment fence.** The harness creates and destroys control-plane objects, which is a mutation — legitimate here because it is a dev-loop concern rather than the runtime path (working instruction #11), but it must be fenced. Refuse to run against any environment not explicitly marked as a dev target in config, and refuse outright if the environment name resolves to production. A dev harness that can be pointed at production by a typo in an env var is a defect, not a feature. - -**Two supported topologies**, because the right one depends on instance quota: - -| | Per-developer gateway | One shared dev gateway | -|---|---|---| -| Control-plane objects | one gateway + instances per dev | one gateway, instances per dev | -| Isolation | full | policies collide; noisy neighbours | -| Registration | one artifact per dev | one shared artifact | -| Default when | quota is comfortable | quota is tight | - -Default to per-developer; support shared for orgs where a registered gateway is a licensed, counted resource. The §6.7 quota question decides which is the documented default. - -**Registration artifacts are credentials.** `flexctl registration create` produces a file containing secrets that let the holder register a gateway into the org. Treat it as such: gitignored, never logged, never baked into an image, generated per developer, and supplied to CI from secrets. `agent-fabric dev up` must refuse to proceed if it finds a registration artifact tracked by git. - -**Real traffic costs real tokens.** Unlike `local` and `mock`, this rung sends real requests through the real proxy to a real model. Apply a token-budget policy in the dev environment **by default** rather than documenting the risk, default the harness to conservative `max_tokens`, and print cumulative spend at teardown. A dev loop that quietly consumes a production LLM budget will be switched off after the first invoice. - -#### 6.8.4 Where each rung belongs - -| Context | Rung | Why | -|---|---|---| -| Unit tests, OSS contributors, no org | `mock` | offline, no credentials, fast | -| Routing, CORS, custom WASM policies, request shape | `local` | real gateway, no control-plane cost | -| Policy behaviour, `resolve()`, attribution, usage | `connected` | the only rung where these are real | -| Pre-merge CI gate | `connected` | highest fidelity before a shared sandbox | -| Release gating | live sandbox | see §8.3 | - -Keep `local` as the default for `simulate()`. `connected` is opt-in, because it requires credentials and spends money — but it should be the rung the docs recommend for anyone with an org, and the one CI uses on the main branch. - -#### 6.8.5 Added to the M0 verification checklist - -Both items are in §6.7. They decide whether §6.8 is per-developer or shared, and how CI obtains a registration. - -### 6.9 Companion custom policies — the SDK-coupled contract - -§0.2 excludes authoring gateway policy *logic* from this SDK, and that exclusion stands: Omni Gateway policies are Rust→WASM on Envoy via the PDK and cannot be expressed in Python or TypeScript. This section is not a reversal. It specifies the **contract** between the SDK and a small set of custom policies shipped from a **separate companion repository**, of which this SDK implements only the client half. - -The reason to specify it here rather than leave it to the policy repo is that the contract is worth more to the SDK than to the gateway. Every stock policy verified in §4 keys on a single HTTP request — a credential, a body, a header. But an agent run is not one request. It is one model call, then a tool call, then six more model calls, then a delegation to another agent. The gateway sees fifteen unrelated requests from one `client_id` and cannot tell a converged run from a runaway loop, or a declared tool from an injected one. - -There are exactly three facts the SDK knows that the gateway structurally cannot: - -1. **Run shape.** The correlation-ID contextvar in `core/transport.py` (§2.3) already fans one logical run across every model and tool call. Step index, recursion depth, and the parent run in an A2A delegation chain exist only in the client. -2. **The declared toolset.** `ToolSet.filter(allow=, deny=)` (§4.3) runs in the developer's process. It is advisory. A prompt-injected agent can call any tool its credential reaches, and the gateway will allow it. -3. **Provenance.** Framework and version, SDK version, resolved model, lockfile digest (§4.2). - -A custom policy that consumes any of those is, by construction, useless without the SDK — which is the property that makes this workstream worth funding. Nothing below can be reproduced with `curl`. - -#### 6.9.1 Ownership boundary — read this before writing any Rust - -| Artifact | Repo | Owner | -|---|---|---| -| Policy logic (Rust→WASM, PDK), policy Exchange assets | **companion repo**, e.g. `agent-fabric-policies` | platform / gateway team | -| Header emission, signing, error mapping, conformance scenarios | **this SDK** | SDK team | -| Applying a policy to an API instance | CI, from a reviewed spec (§5.4) | platform team | - -Three rules follow, and none of them is negotiable: - -- **No Rust in this repository.** If a PR to this repo adds a buildable Rust crate, it is in the wrong repo. The SDK's entire contribution is a signed header on the way out and a typed exception on the way back. The one permitted exception is the inert scaffolding templates behind `agent-fabric policy new` (§6.10.5), which are never compiled here. -- **The wire format is a published spec, not an implementation detail.** Version it in the header value itself (`x-agent-run: v1.…`) so the SDK and the policies release independently. Maintain a compatibility matrix in the companion repo; the SDK must tolerate a gateway running the previous major version. -- **§5.4 and working instruction #11 still apply.** These policies are applied by the platform team in CI. Nothing in the SDK's runtime path applies, mutates, or requests a policy. The SDK emits headers and reads decisions. - -**Everything here is UNVERIFIED.** Not one row below has been confirmed against a real gateway, and §6.9.6 lists the PDK capability questions that decide whether the design is buildable at all. Per working instruction #2, no SDK code path ships against this contract until its row in `docs/verified-apis.md` is filled in. - -#### 6.9.2 `agent-run-attestation` — the primitive; build this one first - -A request-side policy requiring a signed run-context header, without which the request is rejected (or, in `observe` mode, annotated and passed). - -``` -x-agent-run: v1.. - -claims = { - "run_id": "0f9c…", # the §2.3 correlation ID, verbatim - "parent_run_id": "a41e…", # A2A / broker delegation chain; null at the root - "step": 7, # 7th governed call in this run - "depth": 2, # 2 agents deep - "agent_id": "checkout-agent", - "business_group": "payments", - "framework": "langgraph@0.2.74", - "sdk_version": "0.1.0", - "toolset_digest": "sha256:…", # §6.9.5; null when no tools are bound - "iat": 1756800000, "exp": 1756800300, "nonce": "…" -} -``` - -**Sign with the `client_secret` the SDK already holds.** API Manager already knows that secret for the client application — `client-id-enforcement` 1.3.3 is verified live on the `openai-sdk` instance (§4) — so the policy can verify without any new key infrastructure, key rotation story, or JWKS endpoint. This is the single most important design decision in the section, and it is also the least certain: whether a PDK policy can read the contract secret for the authenticated client application is an open capability question (§6.9.6). If it cannot, fall back to a policy-configured shared signing key per business group, and accept the rotation burden. - -Configuration surface: - -```yaml -mode: observe | enforce # start at observe, always -maxClockSkewSeconds: 60 -maxTokenAgeSeconds: 300 -replayWindowSeconds: 300 # nonce cache; see the shared-state question in §6.9.6 -requireSignature: true # false = accept unsigned claims, for the migration window -emitTracingLabels: true # project run_id / agent_id / depth into tracing (§2.5) -``` - -What it buys, in order of value: - -- **It closes the §3 blocker.** "Business-group attribution header name" has been UNVERIFIED since M0 and is rated High impact in §10 because it kills cost attribution. This makes it moot. Rather than waiting for MuleSoft to expose a header the gateway reads, you define the header, you sign it, and the policy reads it. Attribution becomes a contract you control instead of a platform unknown you are blocked on. -- **It makes every other policy in this section possible.** §6.9.3, §6.9.4 and §6.9.5 are all consumers of these claims. Build them in that order or not at all. -- **It gives `mode: enforce` as a real capability.** An instance with this policy is SDK-only by construction, which is a legitimate thing for a platform team to want and impossible to achieve with the stock policy set. -- **It joins the client trace to the gateway trace.** §2.5 promises a developer can correlate a local OTel trace with what the platform team sees. Today that rests on `X-Correlation-Id` surviving; `emitTracingLabels` makes it structural. - -**Ship the free version first.** The stock `llm-token-rate-limit` policy takes a DataWeave `keySelector`, verified in §4 as `#[attributes.headers['client_id']]`. The moment the SDK emits a plain `x-agent-run-id` header, that becomes `#[attributes.headers['x-agent-run-id']]` and stock rate limiting is run-scoped with **zero custom code**. Do that in M1. The custom policy earns its keep only when you need signing, replay protection, and step/depth counting — do not let it block the 80% that is a one-line config change. - -SDK side. Note the split: run *state* is core, because the correlation contextvar already is; the attestation *codec* is a plugin (§6.10), because `core/` must not know this policy exists. - -- `policies/attestation.py` — canonical JSON serialisation (key order is part of the signature; get this wrong and every request fails verification), HMAC, and the header codec. Exposed as a `PolicyPlugin` whose `contribute_headers` emits `x-agent-run`. -- `core/telemetry.py` — `step` and `depth` counters on the run contextvar, incremented per governed call. -- `core/transport.py` — no policy-specific code at all. The header arrives through the plugin hook in `_apply_base_headers` (§6.10.3), which both `FabricAsyncClient` and `FabricClient` already call. -- `Fabric.run_context(run_id=…)` gains `parent_run_id=` so an A2A delegation continues the chain rather than starting a new one. - -The plain `x-agent-run-id` header from the paragraph above is the exception: it is core, unconditional, and needs no plugin, because it is just the existing correlation ID under a second name that a `keySelector` can read. - -#### 6.9.3 `agent-run-budget` — the runaway-loop killer - -`client_id`-keyed token limiting is the wrong granularity for agents. One ReAct loop that will not converge exhausts the budget for every other agent sharing that credential, and the failure lands on whichever agent asks next — not on the one at fault. Keying on `run_id` from §6.9.2 contains the blast radius to the run that caused it. - -```yaml -maxTokensPerRun: 50000 -maxSteps: 40 -maxDepth: 4 # delegation depth; the A2A cycle guard -runTtlSeconds: 900 -onExceeded: reject | truncate # truncate = allow the in-flight call, block the next -``` - -`maxDepth` deserves its own mention: §4.5 wraps a remote A2A agent as a callable tool, so agent-calls-agent is a supported pattern, and there is currently nothing anywhere in the stack that stops A delegating to B delegating back to A. Client-side depth counting is defeated by the first participant that does not use the SDK. Gateway-side is not. - -The rejection body is the other half of the value. Verified stock behaviour for a token-limit block is `429` with an **empty body**, no `retry-after`, and budget state smuggled into `x-token-reset` as milliseconds (§4). Against that, this policy returns: - -```json -{"error": {"type": "run_budget_exceeded", "policy": "agent-run-budget", - "run_id": "0f9c…", "consumed": 48213, "limit": 50000, - "step": 41, "limit_kind": "maxSteps", - "remediation": "Run exceeded 40 steps — likely a non-converging tool loop. Inspect the run trace before raising the limit."}} -``` - -`PolicyViolation.remediation` is a required field (§2.4) that the SDK currently has to synthesise, because the gateway says nothing. Here it comes from the policy, which is the only party that knows which limit tripped. - -**Two hard parts, stated up front.** Token counts live in the response `usage` block, and on a streaming response they arrive only in the terminal SSE event — so accumulation means inspecting a stream without buffering it. And a cross-request counter needs state shared across gateway workers; if the PDK offers no shared store, per-worker approximation is the fallback and the configured limit becomes a soft ceiling that must be documented as such. Both are §6.9.6 questions. Scope this policy only after they are answered. - -#### 6.9.4 `fabric-error-envelope` — normalise the rejection shapes - -`errors.classify()` is a heuristic table because the gateway speaks four dialects, and §4 says so explicitly: *neither the status code nor the shape of the `error` value is a sufficient discriminator*. A flat `{"error": "…"}` for auth, a nested provider object passed through verbatim, a nested `{"type": "pii_detected"}` for PII that is a `403` but not an auth failure, and an empty body for `429`. The current implementation disambiguates on `www-authenticate` presence and error `type` strings, which works, and which will break the first time a policy is upgraded. - -A response-side policy that rewrites **non-2xx responses only** into one versioned envelope collapses that table into a schema check: - -```json -{"error": {"envelope": "fabric/v1", "type": "pii_detected", "policy": "llm-pii-detection-policy@1.0.0", - "message": "…", "remediation": "…", "retry_after_s": 42, - "correlation_id": "0f9c…", "run_id": "0f9c…", - "details": {"entities": [{"pii_type": "Email", "start": 31, "end": 48}]}, - "doc_url": "https://…/errors/pii_detected"}} -``` - -**Make it content-negotiated.** Rewrite only when the request carried `x-fabric-accept-envelope: v1`. Any existing non-SDK consumer of that instance keeps byte-identical passthrough behaviour, so this can be applied to a live instance without a breaking-change review — and the good error experience is an SDK-exclusive property rather than a migration event for somebody else's integration. - -Three constraints on the implementation: - -- **Never touch a 2xx**, and never touch `text/event-stream`. Streaming is verified working (§2) and a response-rewriting policy is exactly the sort of thing that breaks it. -- **Preserve the original status code.** The envelope changes the body, not the semantics. -- **`details` is open.** The envelope's contract is the outer keys; per-policy payloads go in `details` so a new policy does not require a new envelope version. - -Payoff for the SDK: `classify()` becomes "parse the envelope if present, else fall back to today's heuristic table." The fallback stays forever — the SDK must work against an instance with no companion policies — but it stops being the primary path, and a new gateway policy stops being a fixture-capture exercise. - -#### 6.9.5 `toolset-contract` — the one with real differentiation - -This policy sits on the **MCP ingress** (`https://…/mcp//`, verified §6), not the LLM proxy, and it is the strongest of the four because it closes a security gap rather than a DX gap. - -Today, tool filtering is client-side and therefore advisory. §4.3 recommends filtering to keep descriptor counts down, which is a token-cost argument, not a security one — and the security consequence is not written down anywhere: a prompt-injected agent can invoke any tool its credential can reach, because the gateway has never been told which tools the agent declared. - -The policy takes `toolset_digest` from the signed claims (§6.9.2), resolves it against the manifest published to Exchange, and enforces two things: - -1. the invoked tool name is in the declared set, and -2. its arguments validate against the published `inputSchema`. - -The digest infrastructure already exists. §7.3 derives descriptors from code, §7.5 computes a content digest for `--if-changed`, and the conformance kit already asserts `descriptor_auto_stable` — two introspections of one server produce an identical digest. This policy is largely a consumer of work the plan has already committed to. - -It is unforgeable without the SDK because the SDK is what computes the digest at publication time and signs it at call time. That is the cleanest example in this section of the property being asked for. - -**Stretch, and the reason to call any of this a fabric:** have the LLM proxy return a short-lived, run-scoped capability token on a successful completion, and have the MCP ingress require it. Governed tools then become callable only from inside a governed model run — a tool call with a valid credential but no active run is rejected. No hand-rolled client can do this, because it requires carrying state from an LLM response header into an MCP session, which is precisely what `core/transport.py` and `McpServerHandle` already are. Treat it as a design spike, not a deliverable: it couples the two data planes, and if the coupling is wrong it fails closed on every tool call in production. - -#### 6.9.6 Cross-cutting rules - -**Policy ordering is part of the contract.** Attestation must run before anything reading run claims, and the error envelope must sit outermost so it catches rejections from every policy including attestation itself. Record the required order in the companion repo and assert it in `Governance.resolve()` (§6.3) — a correct set of policies in the wrong order is drift, and `GovernanceDrift` should say so. - -**Every policy ships `observe | enforce`, and defaults to `observe`.** Roll out in observe, read the analytics, then enforce. A policy that hard-requires an SDK header on day one breaks whatever is already calling that instance, and the rollback is a control-plane change under someone else's change window. - -**Fail closed, but only in the direction that is safe.** A missing or invalid attestation in `enforce` mode is a rejection. A missing *optional* claim is not — `toolset_digest` is null for an agent with no tools bound, and that must stay a normal request rather than a policy incident. - -**LiteLLM degrades this, as it degrades everything.** ADK and CrewAI cannot be handed the SDK's httpx client, so their correlation ID is per-client rather than per-run — already a documented, asserted conformance exemption (§8.1 `correlation_id_propagated`, §10). Under attestation, those two adapters still authenticate and still attribute per client application, but `step` and `depth` are not trustworthy, which means §6.9.3's `maxSteps` and `maxDepth` cannot be enforced for them. Decide explicitly whether that is an exemption or a hard blocker for enforcement on an instance those frameworks call. Do not discover it during a rollout. - -**This is the rare governance feature that is fully testable locally.** The §6.4 portability table says custom WASM (PDK) policies work in **both** Local and Connected Mode — unlike `client-id-enforcement`, SLA rate limiting, or the LLM proxy itself. So these four are exercisable on the `local` rung of `simulate()` (§6.5, §6.8): a real gateway, real policy enforcement, no control-plane cost, no shared sandbox, no org required. `attribution_headers_present` and `correlation_id_propagated` stop being assertions against a mock and become assertions against a gateway that actually rejects. That is a stronger dev loop than anything else in §6 offers, and it is an argument for building §6.9.2 earlier than its business value alone would justify. - -#### 6.9.7 Added to the M0 verification checklist - -None of this is buildable until the PDK's actual capabilities are known. Each row gates a specific design decision above; add them to `docs/verified-apis.md` and answer them before any Rust is written. - -| Question | Gates | If the answer is no | -|---|---|---| -| Can a PDK policy read the contract `client_secret` for the authenticated client application? | §6.9.2 signing key | Per-BG configured shared key; own the rotation story | -| Does the PDK expose state shared across gateway workers (counters, nonce cache)? | §6.9.2 replay window, §6.9.3 budgets | Per-worker approximation; document limits as soft ceilings | -| Can a policy inspect a streaming (SSE) response body without buffering it? | §6.9.3 token accumulation | Budget enforcement is non-streaming-only; say so loudly | -| Can a policy rewrite a response body conditionally on a request header? | §6.9.4 content negotiation | Envelope becomes instance-wide and therefore a breaking change | -| Can a policy make an outbound call (Exchange manifest lookup), and with what latency budget? | §6.9.5 digest resolution | Ship the manifest in policy config; accept staleness | -| Is policy execution order controllable and readable per API instance? | §6.9.6 ordering, drift detection | Order becomes convention, unassertable by `resolve()` | -| Do custom WASM policies genuinely run in Local Mode as §6.4 claims? | the entire local dev-loop argument | Falls back to the `connected` rung; needs an org | - -### 6.10 `PolicyPlugin` — the interface custom policies implement - -§6.9 specifies four policies. This section specifies the **interface** they implement, and the reason it exists is that the SDK must not know about those four. - -Custom policies already exist in the wild. §6.4's own `simulate()` output shows `acme-custom-redaction 0.2.0 (WASM)` sitting alongside the stock set — a customer policy this project will never see. If the SDK hard-codes §6.9's four into `transport.py` and `errors.py`, then every customer with their own PDK policy is locked out of header emission, typed rejections, drift verification, and local simulation, and the four become a permanent maintenance tax on `core/`. One extension point costs less than four special cases and serves an unbounded set. - -#### 6.10.1 The protocol - -A plugin teaches the SDK three things about one policy: what to send, how to read its rejection, and how to declare it. Every method is optional — a plugin that only classifies errors is a legitimate plugin. - -```python -# src/agent_fabric/policies/base.py — framework-free (§1.1) - -@dataclass(frozen=True) -class PolicyRef: - asset_id: str # Exchange assetId, e.g. "agent-run-attestation" - versions: str # range this plugin speaks, e.g. ">=1.0,<2.0" - group_id: str | None = None # None = the stock-policy org - -Surface = Literal["llm", "mcp", "a2a", "control_plane"] - -@dataclass(frozen=True) -class RequestContext: - cfg: FabricConfig - run: RunContext # run_id, parent_run_id, step, depth (§6.9.2) - surface: Surface - -class PolicyPlugin(Protocol): - ref: PolicyRef - surfaces: frozenset[Surface] # where this policy applies; nothing else calls it - portability: Literal["both", "connected_only"] # feeds the §6.4 table - - def contribute_headers(self, ctx: RequestContext) -> Mapping[str, str]: - """Request headers this policy needs. Called per request, on the hot path — - keep it pure, synchronous, and allocation-light. MUST NOT do I/O.""" - - def classify(self, response: httpx.Response) -> FabricError | None: - """This policy's rejection -> a typed exception, or None for 'not mine'. - None is the ONLY way to decline. A plugin cannot turn a rejection into a - success (§6.10.4).""" - - def validate_config(self, config: Mapping[str, object]) -> None: - """Raise ConfigError on an invalid PolicyBinding config, at construction - time rather than at apply time. Also drives `agent-fabric lint` (§5.3).""" - - def export_fragment(self, binding: PolicyBinding) -> Mapping[str, object]: - """The `policies[]` entry this binding compiles to in fabric.yaml - (§5.1), used by Governance.export() (§6.3).""" - - def simulate_fragment(self, binding: PolicyBinding) -> Mapping[str, object] | None: - """Declarative Local Mode config, or None meaning connected-only — which - is what puts the policy in simulate()'s non-suppressible SKIPPED report - (§6.4). Returning a fragment for a connected-only policy is a bug that - makes a developer trust a green local run.""" -``` - -`PolicyBinding` is unchanged from §6.2. What changes is that constructing one now looks for a plugin matching its `assetId` and calls `validate_config`, so a typo in a policy config fails on the developer's laptop instead of in the CI apply step. - -#### 6.10.2 Discovery is automatic; activation is not - -Plugins are found through the `agent_fabric.policies` entry-point group, so a companion package registers itself by being installed and this SDK never depends on it. - -**Discovery does not imply activation.** Auto-loading an installed package that can inject headers into every governed request and intercept error classification is a supply-chain hole, and this plan is already careful about exactly that class of risk in §7.10.3. So a discovered plugin is inert until it is named: - -```toml -# .agent-fabric.toml — committed, reviewable -[policies] -enabled = ["agent-run-attestation", "acme-pii-redaction"] -``` - -Also accept `Fabric(policies=[...])` for programmatic use and tests. Discovered-but-not-enabled plugins are logged once at INFO with the line needed to enable them — discoverable, not silent, not automatic. - -Three determinism rules, all enforced at `Fabric` construction and never at request time: - -- **Ordering is explicit.** Plugins run in the order listed in `enabled`. Never rely on entry-point iteration order. -- **Header collisions are fatal.** A plugin may not write a header another plugin or the core set (§2.3) already writes. Raise `ConfigError` naming both plugins at startup. Silent last-writer-wins on an attribution header is a bug nobody will find. -- **Version mismatch is fatal.** If a plugin's `ref.versions` does not cover the version `resolve()` reports as applied on the gateway, that is `GovernanceDrift` (§6.3), not a warning. A v1 client signing for a v2 policy fails every request; better to fail at startup with the reason. - -#### 6.10.3 Where the SDK calls it - -Five hooks, five existing call sites. No new machinery. - -| Hook | Call site | Notes | -|---|---|---| -| `contribute_headers` | `_apply_base_headers`, both transports (§2.3) | After core headers; collision-checked at startup, not here | -| `classify` | `errors.classify()` (§2.4) | Plugins first, in order; built-in table is the fallback | -| `validate_config` | `PolicyBinding.__init__`, `agent-fabric lint` (§5.3) | Fails on the laptop, not in CI | -| `export_fragment` | `Governance.export()` (§6.3) | Compiles to the §5.1 spec format | -| `simulate_fragment` | `Governance.simulate()` (§6.5) | `None` → the §6.4 SKIPPED report | - -`classify()` inverting to plugins-first is the one behavioural change to shipped code, and it is what §6.9.4 needs: the envelope parser is just a plugin that returns `None` when the response carries no envelope, leaving today's verified heuristic table (§4) untouched as the fallback. Both paths keep working, forever, on an instance with no companion policies at all. - -#### 6.10.4 A plugin must not be able to break a governed call - -This is the part to get right, because a governance SDK whose extension point can silently disable governance is worse than one with no extension point. - -- **A plugin cannot suppress a rejection.** `classify` returning `None` means "not mine" and falls through. There is no return value meaning "treat this 403 as success". If a plugin raises, log it and fall through to the built-in table — never swallow the response. -- **A plugin cannot make a call ungoverned.** `contribute_headers` adds; it cannot remove or overwrite core headers. The core set in §2.3 is written last and wins by construction. -- **No I/O on the request path.** `contribute_headers` is synchronous and pure. A plugin needing remote data fetches it at construction and caches it; the alternative is a policy plugin adding a network round-trip to every model call. -- **Failure is loud and early.** Every validation above happens at `Fabric` construction. Nothing about plugin resolution may fail for the first time on request number four hundred. -- **Zero plugins is a supported, tested configuration.** The base test job (working instruction #9) runs with none installed. - -**A plugin is not an approval.** §5.4's allow-list catalog is what says a policy may be applied to a shared gateway, and it is owned by the platform team in a different repo. Installing a plugin only teaches the SDK to *speak* to a policy; if the policy is not in the allow-list, `agent-fabric apply` still refuses. Keep those two mechanisms visibly separate — conflating them hands the app team the platform team's job, which is precisely the failure §5.4 exists to prevent. - -#### 6.10.5 `agent-fabric policy` — scaffolding, not authoring - -The interface above creates a synchronisation problem: a custom policy now has three artifacts that must agree — the PDK config schema on the gateway, the plugin's `validate_config`, and the `fabric.yaml` fragment. Hand-maintaining three copies of one schema guarantees drift. - -``` -$ agent-fabric policy new acme-pii-redaction --surface llm - created policies/acme-pii-redaction/policy.yaml # one declaration: config schema + metadata - created policies/acme-pii-redaction/plugin.py # PolicyPlugin stub, schema generated - created policies/acme-pii-redaction/test_plugin.py # the §8.1 plugin scenarios, parametrised - created policies/acme-pii-redaction/pdk/ # PDK project skeleton + generated JSON schema - next implement the Rust handler in pdk/src/lib.rs — the SDK does not generate policy logic (§0.2) -``` - -One declaration, three generated artifacts, and the last line of output states the boundary. **This does not author policy logic** — §0.2 stands, the Rust handler body is written by a human. What is generated is the schema in three representations and the client-side plumbing, which is exactly the mechanical part. - -The scaffold lands in the **user's** working directory, which is where the §6.9.1 "no Rust in this repository" rule and a generator that emits a `pdk/` skeleton stop contradicting each other: this repo ships the generator and its templates, never a compiled policy. A team may keep the generated `pdk/` beside their plugin or move it to a dedicated policy repo; `policy check` works either way, because it reads the deployed schema from the control plane rather than from the neighbouring directory. - -``` -$ agent-fabric policy check --target sandbox - ✓ acme-pii-redaction 0.2.0 plugin schema matches the deployed policy - ✗ agent-run-attestation 1.1.0 plugin declares maxTokenAgeSeconds; gateway does not -``` - -`check` is grounded in something already verified: `api-mgr:policy:describe --policyVersion -o json` returns the policy's `configuration[]` (§4), so the deployed schema is readable and diffable against the plugin's declared one. That turns "the policy was upgraded and nobody told the SDK team" from a production incident into a CI failure. Wire it into `agent-fabric lint` (§5.3) and run it on the schedule that already runs `agent-fabric drift` (§5.2). - -#### 6.10.6 Scope - -In: the protocol, discovery and activation, the five call sites, the safety rules, `policy new` / `policy check`. Ships with §6.9's four as the first four plugins — in `policies/`, **not** in `core/`, so the import-linter rule (§1.1) keeps them out of the framework-free layer and proves the extension point is real rather than a facade over four hard-coded cases. - -Out: policy logic (§0.2), a Python→WASM compiler (§0.2), and any runtime path that applies a policy (§5.4, working instruction #11). - ---- - -## 7. Publication — registering code-first assets into Exchange - -Symmetric with §6 by design: one declarative object, the same three-verb lifecycle, the same "declare in code, apply in CI, verify at runtime" discipline. The symmetry holds for two of the three verbs. **It breaks at runtime, and §7.4 explains why — read that before implementing.** - -Publication is lower risk than §5, because Exchange has a well-trodden publication mechanism (Exchange API, Anypoint CLI, and the Exchange Maven plugin) rather than a possibly-UI-only wizard. Confidence here is higher than anywhere else in this spec. - -### 7.1 Scope: code-first assets only - -Publication exists for assets that **originate in the developer's code**: - -- an MCP server written in Python or TypeScript, -- an agent written in one of the eight frameworks and exposed over A2A, -- an agent exposed as a tool without an A2A surface. - -It does **not** exist for assets the platform already owns. If an MCP server was created by MCP Bridge from an existing API (§5), that capability is already in Exchange. Publishing a second, code-derived descriptor for it creates two catalog entries for one capability, which is exactly the failure a registry is supposed to prevent. - -**Implement a collision check.** Before publishing, search Exchange for an existing asset with the same endpoint, name, or derived tool signature. On a probable match, refuse and print the existing asset's coordinates. Override with an explicit `--allow-duplicate` flag that logs at WARNING. - -### 7.2 The object - -```python -from agent_fabric import Publication, AssetType, Contact - -pub = Publication( - asset_type=AssetType.MCP_SERVER, # MCP_SERVER | A2A_AGENT | AGENT | API - group_id="${ANYPOINT_ORG_ID}", - asset_id="hr-tools-mcp", - version="1.3.0", # semver; see §7.5 on immutability - name="HR Tools", - description="Employee lookup and leave-balance tools for HR agents.", - - # Discovery metadata - tags=["hr", "internal", "agent-tool"], - categories={"Domain": "People", "Lifecycle": "Production"}, - contact=Contact(team="People Platform", email="people-plat@acme.com"), - - # Type-specific descriptor — exactly one, matching asset_type - descriptor="auto", # introspect the live server (§7.3) - - # Documentation pages, published alongside the asset - docs=[ - ("home", "docs/exchange/overview.md"), - ("getting-started", "docs/exchange/quickstart.md"), - ], - - # Where it actually lives — metadata only, see §7.6 - endpoint="https://hr-tools.internal.acme.com/mcp", - governance=gov, # optional link to the §6 object -) -``` - -`AssetType` drives which descriptor is required and how it is generated: - -| `asset_type` | Descriptor | Generated from | -|---|---|---| -| `MCP_SERVER` | MCP tool manifest — server info, tool names, descriptions, JSON Schema inputs | live `tools/list` against the running server | -| `A2A_AGENT` | A2A Agent Card | declared skills, endpoint, auth schemes, input/output modes | -| `AGENT` | agent descriptor (no A2A surface) | framework-specific introspection, best-effort | -| `API` | OpenAPI / AsyncAPI | user-supplied file; no generation | - -**M0 gate:** verify whether Exchange exposes first-class asset types for MCP servers and AI agents. Agent Fabric's Agent Registry is built on Exchange, so this is likely — but if it does not, publication degrades to a generic or custom asset type carrying tags, which weakens discoverability and **breaks `asset_types=["mcp"]` filtering in §6.1**. The two features share this dependency; verify once, record once. - -### 7.3 `descriptor="auto"` — deriving the spec from code - -Hand-maintained catalog descriptors go stale within one sprint. Generation is the entire value of this object, exactly as `inputSchema: auto` is the value of §5. - -#### 7.3.1 The key insight: do not write a type-hint-to-JSON-Schema converter - -Every one of the eight frameworks **already derives JSON Schema from function signatures, type hints, and docstrings**, because that is how tool calling works at all. `@tool` in LangChain and Strands, `FunctionTool` in ADK and LlamaIndex, `@function_tool` in the OpenAI Agents SDK, `@mcp.tool()` in the MCP Python SDK — each builds a schema and attaches it to a tool object. ADK's documentation is explicit that the docstring is what the model reads. - -The SDK's job is therefore **to ask the framework for the schema it already computed**, not to re-derive it. Re-implementing type-hint inference would produce a second, subtly different schema from the one the model actually sees, which is worse than useless — it means the catalog documents something other than the running behaviour. - -Concretely, per framework, read the already-populated fields: - -| Framework | Tool object | Name / description / schema | -|---|---|---| -| MCP Python SDK (FastMCP) | registered tool | `.name`, `.description`, `.inputSchema` | -| LangChain / LangGraph | `BaseTool` | `.name`, `.description`, `.args_schema` (pydantic → `.model_json_schema()`) | -| Strands | `@tool` function | tool spec including input schema | -| Google ADK | `FunctionTool` | declaration with parameters | -| LlamaIndex | `FunctionTool` | `.metadata.name`, `.description`, `.fn_schema` | -| OpenAI Agents SDK | `FunctionTool` | `.name`, `.description`, `.params_json_schema` | -| Anthropic SDK | tool param dict | `name`, `description`, `input_schema` | -| CrewAI | `BaseTool` | `.name`, `.description`, `.args_schema` (pydantic → `.model_json_schema()`) | -| MS Agent Framework | tool / `AIFunction` | declaration + JSON schema | - -**These are per-framework adapters and they will break on upstream releases.** Some of these attributes are semi-public. Put every one of them in the conformance kit and the nightly matrix (§8.4), same as the model and tool adapters. - -#### 7.3.2 Three derivation modes, ranked by fidelity - -```python -descriptor="auto" # object introspection — the default -descriptor="auto:live" # live protocol introspection — highest fidelity -descriptor="auto:static" # AST only — lowest fidelity, no code execution -descriptor="auto:check" # generate, diff against committed file, fail on mismatch -``` - -**1. `auto:live` — highest fidelity.** Start the server, perform the MCP initialize handshake, call `tools/list` (plus `resources/list` and `prompts/list`). This is exactly what a client sees, so it captures dynamically registered tools and any runtime filtering. Requires the server to actually run, which may need credentials and network. - -**2. `auto` (object introspection) — the default, and the answer to "build the spec from the code."** Import the user's module, locate the tool and agent objects, read their already-computed schemas. No server, no network, works in CI. Fidelity is near-identical to `auto:live` because it reads the same objects the server would expose. - -Caveat that must be documented prominently: **importing user code executes it.** Module-level side effects — database connections, API calls, config loading — will fire. Mitigate with an explicit entrypoint rather than package-wide scanning: - -```toml -[publication.entrypoints] -"hr-tools-mcp" = "acme.hr.server:mcp" # module:attribute -"hr-agent" = "acme.hr.agent:build_agent" # a zero-arg factory is also accepted -``` - -Document the contract in one line: **tool definitions must be import-safe.** Most already are. - -**3. `auto:static` — AST parsing, genuinely lower fidelity.** Parse decorators, signatures and docstrings without executing anything. Useful only where importing user code is unacceptable (untrusted CI, a security policy against executing PR code during analysis). - -Be honest about what it cannot see, because the gap is not marginal: - -- tools registered in a loop, or from a config file or database -- tools registered conditionally behind a feature flag -- tools attached dynamically at startup -- schemas built from imported or generated pydantic models -- any tool whose description is computed rather than literal - -A static descriptor that silently lists 4 of 11 tools is worse than no descriptor. So: `auto:static` must emit a **completeness warning** whenever it encounters dynamic registration patterns it cannot resolve, and must never be the default. - -**Cross-check mode.** `agent-fabric publish --cross-check` runs `auto` and `auto:live` and diffs them. Disagreement means either dynamic registration the object graph does not reflect, or a broken framework adapter. Worth running in CI for anything important. - -#### 7.3.3 The honest ceiling: shape is derivable, meaning is not - -Type hints give the **shape**. `department: str` becomes `{"type": "string"}` and says nothing about what a valid department is, which values exist, or when the tool should be used instead of a similar one. Only a human supplies that, and it is precisely the part that determines whether a model calls the tool correctly. - -So auto-derivation automates the mechanical 80% and cannot touch the 20% that matters most. Design accordingly: - -- **Fail publication on missing or tautological descriptions.** A description equal to the identifier, or a near-match after normalising underscores and case, is a failure, not a warning. `"search_employees"` as the description of `search_employees` makes a useless tool look documented. -- **Report description quality** in `preview()`: which tools have descriptions under N characters, which parameters have none, which enums are untyped `str`. Give the developer a checklist, not a verdict. -- Optionally offer `agent-fabric describe --suggest`, which drafts descriptions with an LLM through the governed proxy and writes them into the **source** as docstrings for human review. Drafts into source, never straight into the catalog. Keep this opt-in and clearly labelled. - -#### 7.3.4 A2A agent cards derive less well than MCP manifests - -Worth stating plainly, because the asymmetry is easy to miss. An **MCP tool manifest is close to 1:1 with functions**, so derivation works well. An **A2A skill is not a function** — it is a coarser capability with examples, input/output modes, and its own description, and one skill typically spans several functions. - -Auto-deriving skills from functions produces a card advertising forty micro-skills, which is a bad agent card. So: - -- Derive the **mechanical** card fields automatically: endpoint, transport, auth schemes, capability flags, input/output modes, version. -- Require **explicit skill declaration or grouping**. Provide a decorator (`@skill("leave-management", examples=[...])`) or a config mapping functions to skills, and fail publication if an `A2A_AGENT` has no declared skills rather than inventing them. - -Also note the card must be **served** at the agent's well-known path for A2A clients to find it. Ship `pub.agent_card_handler()` returning an ASGI/Express-compatible route and one example per web framework. The developer mounts it; do not auto-mount. - -#### 7.3.5 Committed vs generated - -Support both, and make the CI-friendly mode obvious. `descriptor="auto:check"` generates, compares against a committed descriptor file, and fails on mismatch. That is what teams will actually run in CI: the descriptor is reviewable in the PR diff, and drift between code and descriptor is caught before publication rather than by `verify()` after it. - -### 7.4 Runtime verb: `verify()`, not `publish()` - -The request asks for publication to work at runtime the way governance does. **The symmetry does not survive contact with how Exchange works, and implementing runtime publication would be actively harmful.** Four reasons, all concrete: - -1. **Exchange versions are immutable.** Publishing the same version twice fails. A process that publishes on startup must therefore bump versions, which means every pod restart mints a new catalog version. A ten-replica deployment races to mint ten. -2. **The catalog would reflect what happens to be running.** A registry's value is that it is a curated, reviewed statement of what exists. Runtime self-registration turns it into a log of process starts. This is the specific outcome enterprises adopt a registry to avoid. -3. **Privilege escalation.** Exchange publication needs write scopes. Granting them to a production agent's runtime credential means any compromise of that agent can rewrite the enterprise catalog. -4. **No review.** Publication is a governance act. It belongs behind a PR, like policy. - -So the three verbs are: - -| Verb | Runs where | What it does | -|---|---|---| -| `preview()` | developer laptop | Generates the descriptor and renders the Exchange entry as it would appear. Writes nothing. | -| `export()` | developer laptop | Compiles into the `fabric.yaml` spec (§5.1). CI publishes on merge via `agent-fabric apply`. | -| `verify()` | agent process, runtime | **Read-only.** Fetches the published descriptor from Exchange, introspects the live server, and compares. Raises `PublicationDrift` on mismatch. | - -`verify()` is the genuine mirror of `resolve()`, and it is more useful than runtime publishing would have been. It catches the single most common way an agent registry rots: **a team ships a new tool and the catalog still describes the old surface.** A `verify()` call at startup, or a scheduled `agent-fabric verify` in CI, turns catalog staleness from an invisible slow decay into a failing check with a diff: - -``` -PublicationDrift: com.acme/hr-tools-mcp/1.3.0 - live server exposes 7 tools, Exchange descriptor lists 5 - + get_leave_balance (live, not in Exchange) - + approve_leave (live, not in Exchange) - ~ search_employees input schema changed: +department (required) - Run `agent-fabric publish --bump minor` to update the catalog. -``` - -Default `verify()` to warn-and-continue, configurable to raise. A drifted catalog should not take down production traffic, but it must be loud. - -### 7.5 Versioning and idempotency - -The predictable failure mode is CI publishing a new version on every merge until the catalog holds four hundred patch versions of one asset. Prevent it structurally: - -- **Content digest.** Compute a stable hash over the canonical descriptor plus metadata. Store it as an Exchange field or tag on publish. -- **`--if-changed` is the CI default.** Compare the digest against the latest published version; if identical, skip and exit zero. -- **Version strategy is explicit config**, one of: `pinned` (from the object), `from-package` (read the Python/npm package version), or `semantic-auto` (patch for metadata-only changes, minor for added tools, **major for removed or narrowed tools** — a removed tool is a breaking change for every consuming agent). Default to `pinned`, since implicit version bumps in a shared catalog are the kind of magic that erodes trust. -- **Never delete or overwrite.** Deprecation is a metadata change on the existing version, not a delete. If Exchange supports lifecycle states, use them. - -### 7.6 Publication ≠ deployment ≠ governance - -Three independent facts that this object handles exactly one of: - -| Fact | Meaning | Owned by | -|---|---|---| -| Published in Exchange | discoverable | §7 | -| Deployed and reachable | someone can call it | the user's own platform (§6.6) | -| Fronted by a gateway with policies | governed | §6 | - -An asset published by §7 alone **will not pass `governed=True` filtering in §6.1**, and that is correct behaviour, not a bug. Publication plus Governance together produce a fully registered, governed asset; either alone is a half-state. - -Make this visible rather than leaving developers to discover it. `agent-fabric status` should render all three facts per asset: - -``` -com.acme/hr-tools-mcp/1.3.0 - published yes Exchange, 2026-08-14 - reachable yes https://hr-tools.internal.acme.com/mcp (200, 7 tools) - governed NO no API Manager instance in Production - -> `agent-fabric apply -f fabric.yaml --target production` -``` - -### 7.7 Catalog ownership - -The same political constraint as §5.4. In most orgs, publishing into the shared Exchange organisation requires approval, and an SDK that makes it a one-liner will be seen as a threat to the catalog rather than a contribution to it. - -Defaults that keep the SDK welcome: - -- Publish to the **developer's own business group** by default. Publishing to the root or a shared org requires explicit `group_id` config plus a `--target shared` flag. -- Support an **allow-list of publishable asset types and target groups**, owned by the platform team in a separate repo, mirroring the policy catalog in §5.4. -- CI-only publication under a connected app whose Exchange write scopes the platform team controls. The developer's own credential should not have them. - -### 7.8 Asset auto-detection and `agent-fabric init` - -Asset type detection is reliable, because the framework objects are unambiguous. Scan the entrypoint or project and classify: - -| Detected | Inferred type | -|---|---| -| A FastMCP / MCP server instance with registered tools | `MCP_SERVER` | -| An A2A server app, or an agent with declared skills and an A2A route | `A2A_AGENT` | -| A framework agent object (LangGraph compiled graph, ADK `LlmAgent`, Strands `Agent`, `agent_framework.Agent`, `agents.Agent` (OpenAI Agents SDK), CrewAI `Agent`/`Crew`, LlamaIndex `FunctionAgent`) with no A2A surface | `AGENT` | -| An OpenAPI / AsyncAPI file with no agent or server object | `API` | - -Two things this must get right: - -**One project can produce several assets.** A single repo commonly exposes an MCP server *and* an A2A agent, or several agents. Detection returns a **list** of candidate publications, not one. Model it that way from the start; retrofitting multi-asset support later is painful because the CLI, config, and digest layout all assume cardinality. - -**Never silently pick a type.** Exchange versions are immutable (§7.5), so an asset published under the wrong type is a catalog error you cannot cleanly delete. Detection proposes; the developer confirms once; the answer is written to config and never re-inferred. - -``` -$ agent-fabric init -Scanning acme/hr/ … - - Found 2 publishable assets: - - 1. MCP_SERVER acme.hr.server:mcp - 7 tools · all have descriptions · 2 params undocumented - 2. A2A_AGENT acme.hr.agent:build_agent - LangGraph agent with an A2A route - ! no skills declared — an agent card needs explicit skills (§7.3.4) - - Write these to .agent-fabric.toml? [Y/n] -``` - -`init` should also run the §7.3.3 quality report and the §7.1 collision check at this point, so the developer sees every problem before their first publish rather than one per attempt. - -Ambiguous or unrecognised projects exit with a clear message naming the entrypoint config to set manually. Detection failing is fine; detection guessing wrong is not. - -### 7.9 Added to the M0 verification checklist - -- Does Exchange have first-class asset types for MCP servers and AI agents, or must they be published as generic/custom types? **(gates §7.2 and, jointly, `asset_types` filtering in §6.1)** -- Which publication mechanism is supported for non-Mule assets: Exchange REST API, Anypoint CLI, or the Maven plugin? Pick one and note whether it requires a JVM at publish time — a Maven-only path makes the Python and TS CI story materially worse. -- Can arbitrary metadata or tags be attached for the content digest (§7.5)? -- Are asset lifecycle states (draft / published / deprecated) exposed via API? **(gates `require_lifecycle` in §6.1.1 and deprecation in §7.5)** -- Can documentation pages be published programmatically, and in what markup? -- Does Exchange accept an MCP tool manifest and an A2A agent card as native descriptor formats, or must they be attached as files? **(shapes §7.3 output)** -- Can an asset be published in a **draft / unlisted / staged** state, and can provenance be recorded on it? **(gates the `stage` tier in §7.10.2)** - -### 7.10 Repository and organisation-wide scanning - -Everything above §7.10 assumes one repo, one developer, one interactive command. That answers "help me publish my asset." It does not answer **"we have two hundred repos and an empty catalog,"** which is the harder and more common problem: bootstrapping is a one-time org-wide cost that no individual developer is motivated to pay, and a registry that stays empty for six months never recovers its credibility. - -So: scanning is in scope. Fully automatic publication of everything it finds is not, and the reasons are structural rather than cautious. - -#### 7.10.1 What is already derivable, restated precisely - -| Surface | Derivable without hand-authoring? | Where | -|---|---|---| -| MCP tool names, descriptions, input schemas | **Yes, fully.** Read the schema the framework already computed. | §7.3.1 | -| Asset type (`MCP_SERVER` / `A2A_AGENT` / `AGENT` / `API`) | **Yes.** Framework objects are unambiguous. | §7.8 | -| A2A card mechanical fields — endpoint, transport, auth schemes, I/O modes | **Yes.** | §7.3.4 | -| **A2A skills** | **No, and deliberately so.** Deriving one skill per function produces a card advertising forty micro-skills. | §7.3.4 | -| **Tool *meaning*** — which values are valid, when to call this tool over a similar one | **No.** Shape is derivable; meaning is not. | §7.3.3 | - -An MCP server can therefore be scanned to a complete, publishable descriptor with no human input. An A2A agent cannot, and a scanner must not pretend otherwise. That asymmetry drives §7.10.8. - -#### 7.10.2 The autonomy ladder — four tiers, default `propose` - -| Tier | What it writes | Who reviews | Default for | -|---|---|---|---| -| `report` | nothing, anywhere | nobody | always, on first run | -| `propose` | **a PR in each target repo** adding `.agent-fabric.toml`, a committed descriptor, and the publish workflow | that repo's own owners, via normal PR review | **the default** | -| `stage` | publishes into a separate **discovery** business group, tagged as scanner-derived, never the shared catalog | a catalog curator, out of band | opt-in per org | -| `auto` | publishes to the repo's own business group on merge to the default branch | nobody — CI only | opt-in **per repo** | - -One rule makes this safe, and it is not negotiable: **a tier above `propose` can only be enabled by a file committed in the target repo.** Never by a flag on the scanner. If a scanner operator can raise the tier centrally, then one person with a token becomes the org's publisher of record and §7.7 is void — which is the exact outcome that gets the SDK banned in a platform review. - -```toml -# .agent-fabric.toml, committed in the TARGET repo — the only place this can be raised -[scan] -tier = "auto" # report | propose | stage | auto -assets = ["hr-tools-mcp"] # explicit; `auto` never applies to newly detected assets -``` - -A newly detected asset always starts at `propose`, even in a repo set to `auto`. Otherwise adding a file to an `auto` repo silently publishes a new catalog entry, and Exchange versions are immutable (§7.5) — there is no clean undo. - -#### 7.10.3 Static-first, because scanning is executing - -§7.3.2's caveat — importing user code runs it — is per-repo advice at single-repo scale. At fleet scale it is a security boundary. A central scanner that imports code from two hundred unreviewed repos is arbitrary code execution across the organisation, in one process, with an Anypoint write credential in its environment. Nobody should accept that, and no amount of documentation makes it acceptable. - -Therefore: - -- **Fleet scanning defaults to `auto:static`** (AST, no execution) and cannot be switched to `auto` for repos it does not own. -- **Escalation to `auto` happens inside the target repo's own CI**, where the repo already trusts its own code, never in the central scanner. -- Consequently, fleet-scan descriptors carry the §7.3.2 completeness caveat and are **proposals, not publications**. - -Which yields the invariant that shapes the whole feature: - -> **The central scanner never publishes. It opens pull requests. Every publication happens from the target repo's CI, from a reviewed descriptor, under credentials the platform team controls.** - -This is not a compromise imposed on the feature — it is strictly better than central publication. The completeness warning lands in a PR where the owning team can see it and fix it by moving to `auto` in their own CI, rather than silently producing a catalog entry listing 4 of 11 tools. - -#### 7.10.4 The `propose` PR - -``` -$ agent-fabric scan fleet --config scan-fleet.yaml --tier propose - -Scanning 214 repositories (github.com/acme) … - 208 scanned · 6 skipped (no supported framework detected) - - Detected 47 candidate assets in 31 repositories: - 38 MCP_SERVER complete descriptor derived - 6 A2A_AGENT mechanical fields only — skills required (§7.3.4) - 3 AGENT no A2A surface - - Quality gate (§7.3.3) - 31 pass · 7 have missing or tautological descriptions · 9 have undocumented params - - Cross-repo dedupe (§7.10.6) - 2 clusters where >1 repo exposes the same capability — issues opened, no PRs - - Opening 29 PRs (2 clusters held back) … done - acme/hr-service#412 1 asset MCP_SERVER ready - acme/finance-tools#89 2 assets MCP_SERVER 1 quality failure - … -``` - -Each PR adds only reviewable files, and never a credential: - -``` -+ .agent-fabric.toml entrypoints + asset types, from detection -+ descriptors/hr-tools-mcp.json the derived descriptor, reviewable in the diff -+ .github/workflows/fabric-publish.yml auto:check on PR, publish --if-changed on merge -``` - -The PR body states what was derived, what could not be, and what the reviewer must supply — the seven-line version of §7.3.3's quality report. A PR that says "3 tools need descriptions, here is the file and the line" converts catalog bootstrapping into ordinary code review, which is the only process that scales to two hundred repos. - -#### 7.10.5 Fleet configuration and ownership - -```yaml -# scan-fleet.yaml — owned by the platform team -apiVersion: fabric/v1 -kind: ScanFleet -hosts: - - provider: github - org: acme - include_topics: ["agent", "mcp"] # opt-in by repo topic - exclude: ["acme/archived-*", "acme/*-sandbox"] - default_branch_only: true -ownership: - from: CODEOWNERS # -> Publication.contact - fallback_team: "Platform Engineering" -publication: - group_id: per_repo_business_group # never the shared catalog from a scan - tier: propose # ceiling; a repo may only lower it -quality: - block_proposal_below: warning # do not PR an asset that cannot pass review -``` - -`ownership.from: CODEOWNERS` is worth more than it looks. §7.1's `Contact` is the field most likely to be left blank, and an uncontactable catalog entry is nearly as bad as a missing one. CODEOWNERS is already maintained, already accurate, and already reviewed. - -#### 7.10.6 Cross-repo dedupe is where fleet scanning beats per-repo `init` - -§7.1's collision check compares one candidate against Exchange. A fleet scanner sees **all candidates simultaneously**, which catches the duplicate class that per-repo publishing structurally cannot: three teams that each wrapped the same upstream API. - -Group candidates by tool-signature digest and endpoint. For any cluster of more than one, **do not open PRs.** Open a single issue naming every repo in the cluster and asking which one owns the capability. A scanner that resolves this by picking the first repo alphabetically produces exactly the duplicate catalog §7.1 exists to prevent, only faster and at scale. - -#### 7.10.7 Ship the coverage report first - -The read-only, bidirectional inventory needs **no write scopes at all**: - -``` -$ agent-fabric scan report --config scan-fleet.yaml - -In code, not in the catalog 38 assets across 27 repos -In the catalog, no live code 6 assets -> deprecation candidates (§7.5) -In both, descriptor drifted 11 assets -> `publish --bump minor` (§7.4) -In both, current 52 assets - -Catalog coverage: 52 of 90 detected capabilities (58%) -``` - -This is the §5.3 pattern applied to publication: small, cheap, uncontroversial, requires no verification of a write path, and it is the fastest way to get a platform team to say yes to the rest. It is also the artifact that makes the case for funding the bootstrap at all, since "our catalog covers 58% of what we actually run" is a number an engineering director can act on. - -Build it before the PR machinery. - -#### 7.10.8 Guardrails that survive contact with two hundred repos - -- **The quality gate is not downgradable by the scanner.** An asset failing §7.3.3 is never staged or auto-published — at most proposed, with the failures in the PR body. Bulk scanning multiplies §10's "auto-generated descriptions make useless tools look documented" risk by the size of the fleet. -- **A2A agents are `propose`-only, permanently.** Skills cannot be derived (§7.3.4), so no tier auto-publishes an `A2A_AGENT`. The PR asks for `@skill(...)` declarations. -- **Every scanner-derived asset records its provenance** — scanner version, commit SHA, derivation mode, date — so a curator can tell a reviewed entry from a bootstrapped one, and can re-run the bootstrap later without re-litigating which is which. -- **Digest-gated re-scan.** Re-scanning uses §7.5's content digest, so an unchanged repo produces no PR, no issue and no write. Without this, a scheduled fleet scan is a PR spam generator and will be turned off within a week. -- **One PR and one issue per repo, updated in place.** Same discipline as the nightly matrix (§8.4). -- **Rate limits and concurrency are a design constraint, not an afterthought.** Two hundred shallow clones plus Exchange lookups will hit both the Git host's and Anypoint's limits. Bounded concurrency, sparse checkout, and a persistent cache keyed on commit SHA. - -#### 7.10.9 What scanning is not - -| Not building | Why | -|---|---| -| Zero-touch publication of everything found, with no review | Exchange versions are immutable (§7.5) and catalog ownership is a political constraint (§7.7). The closest offered is `auto`, per-repo opt-in, still quality-gated. | -| A central scanner holding Exchange write scopes | Would make the scanner operator the org's publisher of record. Publication happens in the target repo's CI (§7.10.3). | -| Importing untrusted repo code centrally | Arbitrary code execution across the org. Static-first, escalate only inside the owning repo's CI. | -| A secret scanner, SAST, or code-quality tool | Different products. The scanner detects publishable agent assets and nothing else. | -| Deployment or reachability discovery | The SDK does not deploy anything (§6.6). `reachable` stays a probe in `status`, not an inference. | - -#### 7.10.10 Added to the M0 verification checklist - -- Can an asset be published in a **draft / unlisted / staged** state? **(gates the `stage` tier — without it, `stage` collapses into `propose`)** -- Can **provenance metadata** be attached to a published asset? Shares the §7.9 metadata/tags verification. -- Is there an Exchange API to list **all** assets in an org efficiently enough for a bidirectional coverage report over a large catalog, or must it be paginated per business group? - ---- - -## 8. Testing - -### 8.1 The adapter conformance kit — the most important test asset - -One suite, defined once, executed identically against every framework adapter. Any new framework is "supported" only when it passes all of it. - -```python -# tests/conformance/suite.py -CONFORMANCE_SCENARIOS = [ - "simple_completion", # one turn, no tools, assert text returned - "streaming_completion", # tokens arrive incrementally - "single_tool_call", # model calls one MCP tool, result returned - "multi_tool_multi_server", # two servers, name collision resolved - "tool_filtering", # denied tool is absent from descriptors - "policy_violation_terminal", # gateway 4xx -> PolicyViolation, NOT retried - "attribution_headers_present", # assert headers on every outbound request - "correlation_id_propagated", # same ID on model call and tool call - "governed_filter_excludes", # ungoverned asset absent when governed=True - "governance_resolve_drift", # declared policy missing on gateway -> GovernanceDrift - "governance_target_switch", # same code, FABRIC_TARGET=local|sandbox, correct base_url - "descriptor_auto_stable", # two introspections of one server -> identical digest - "publication_verify_drift", # added tool on live server -> PublicationDrift - "publication_idempotent", # --if-changed with no change -> zero writes, exit 0 - "descriptor_matches_framework",# derived schema == the schema the model receives - "fidelity_never_downgrades", # simulate(mode="connected") fails rather than falling back - "connected_local_real_policy", # connected-mode local gateway enforces a connected-only policy - "dev_instance_cleaned_up", # ephemeral API instance destroyed on exit AND on exception - "scan_never_publishes", # fleet scan issues ZERO writes to Exchange - "scan_tier_needs_repo_optin", # tier > propose ignored unless committed in target repo - "scan_static_completeness", # loop-registered tools -> completeness warning in the PR body - "scan_fleet_dedupe", # same capability in 2 repos -> 1 issue, 0 PRs - "auto_vs_live_agree", # object and live introspection produce one digest - "dynamic_tools_detected", # loop-registered tools present in `auto`, warned in `auto:static` - "asset_type_detection", # fixture projects per framework classify correctly - "attestation_header_signed", # §6.9.2 header present, canonical, signature verifies - "attestation_step_increments", # step/depth advance across a run's model + tool calls - "attestation_chain_preserved", # A2A delegation continues parent_run_id, no new root - "error_envelope_preferred", # §6.9.4 envelope parsed when present, heuristic when not - "no_plugins_is_supported", # §6.10.4 zero plugins installed -> full base behaviour - "plugin_cannot_suppress", # a plugin returning None/raising still yields PolicyViolation - "plugin_header_collision", # two plugins claiming one header -> ConfigError at startup - "plugin_version_drift", # plugin range excludes the applied version -> GovernanceDrift -] -``` - -Parametrise over adapters with pytest. A framework that cannot satisfy `correlation_id_propagated` (likely ADK, via LiteLLM) records a documented, asserted exemption in a `KNOWN_LIMITATIONS` table rather than silently skipping. Publish that table in the README — it is credibility, not embarrassment. - -### 8.2 Contract fixtures - -Write a `scripts/probe.py` that runs against a real sandbox and records every response shape into `tests/contract/fixtures/`: token issuance, model list, a successful completion, a streaming completion, and one deliberately-triggered instance of each policy rejection. Redact secrets on write. - -Unit and contract tests replay these with `respx` (Python) / `msw` (TS). **The `errors.classify` table is generated from these fixtures**, not from guesses. Re-run the probe monthly; a fixture diff is an early warning that the platform changed. - -### 8.3 Integration - -Three tiers, matching the §6.8 fidelity ladder. The distinction that matters: LLM Proxy and MCP Bridge are **Connected Mode** features, which is not the same as "not testable locally" — a self-managed Connected Mode gateway runs on a laptop or a CI runner and has both for real (§6.8). - -| Tier | Marker | What it runs | Needs an org | -|---|---|---|---| -| `mock` | default, always on | in-process stub replaying §8.2 fixtures | no | -| `local` | `@pytest.mark.local_gateway` | `docker-compose`: Flex Gateway in Local Mode, declarative config, upstream mock | no* | -| `connected` | `@pytest.mark.connected_gateway` | Flex Gateway container in **Connected Mode**, registered to a dev environment | **yes** | -| live sandbox | `FABRIC_SANDBOX_TESTS=1` | the shared sandbox, for release gating | yes | - -\* subject to the §6.7 licence-artifact question. - -Coverage on the `local` tier is partial by design — connected-only policies are skipped and loudly reported (§6.4). The `connected` tier closes that gap and is the pre-merge gate on the main branch. Keep every org-requiring tier off by default so contributors without an Anypoint org can still run the full `mock` suite. - -### 8.4 Nightly framework matrix - -Eight frameworks, each releasing independently, several pre-1.0. Run the conformance suite nightly against the **latest** release of each framework in addition to the pinned version. Open an issue automatically on failure. Without this you find out a framework broke when a user reports it. - -Pin floors, not ceilings, in `pyproject.toml`. Never `<` pin a framework — it forces users into dependency hell. - ---- - -## 9. Delivery - -### 9.1 Milestones - -| Milestone | Scope | Est. (2 engineers) | -|---|---|---| -| **M0 — Verify** | §0.3 in full. `docs/verified-apis.md`. `scripts/probe.py` + captured fixtures. Go/no-go on §5. | 2 weeks | -| **M1 — Model access** | `core` complete. `llm` client + catalog. Adapters for Tier 1 (LangGraph, ADK, Strands, Agent Framework, OpenAI Agents SDK, Anthropic SDK, CrewAI). Conformance kit. `lint` command. Docs site skeleton. **First public release, 0.1.0.** | 4 weeks | -| **M2 — Tool access** | `registry` + `tools` + `ToolSet`. Bindings for all eight. Lockfile. A2A agent handles. LlamaIndex (Tier 2) adapter. Governed-only discovery, name/glob `search` + filters, and `explain()` (§6.1). **0.2.0.** | 6 weeks | -| **M2.5 — Governance object** | `Governance`, `GatewayTarget`, target profiles, `resolve()` with drift detection, policy portability table, `simulate()` across all three fidelity rungs incl. the self-managed Connected Mode local gateway and its dev-instance lifecycle (§6.2–§6.6, §6.8). **0.3.0.** | 5 weeks | -| **M2.6 — Policy interface** | `policies/`: the `PolicyPlugin` protocol, discovery + activation, the five call sites, plugin-first `classify()`, `agent-fabric policy new\|check` (§6.10). Then the client half of §6.9: canonical JSON + HMAC codec, header emission in both transports, step/depth counters, `parent_run_id` on `run_context()`, and the four §6.9 policies as the first four plugins. **0.3.2.** | 3 weeks | -| **M2.7 — Publication** | `Publication`, `preview()`, `verify()` with drift diff, digest + `--if-changed`, collision check, `agent-fabric status` (§7). **0.3.5.** | 3 weeks | -| **M2.8 — Derivation** | Per-framework descriptor adapters (§7.3.1), `auto` / `auto:live` / `auto:static` / `auto:check`, cross-check, quality report, asset detection + `agent-fabric init` (§7.8). **0.3.8.** | 4 weeks | -| **M2.9 — Fleet scanning** | `agent-fabric scan` (single repo, non-interactive), coverage report, fleet enumeration, autonomy tiers, `propose` PRs, cross-repo dedupe, scheduled re-scan (§7.10). **0.3.9.** | 3 weeks | -| **M3 — TypeScript** | TS core + LangGraph.js, ADK TS, Strands TS, OpenAI Agents (JS), Anthropic SDK (TS), LlamaIndex.TS, Vercel AI SDK. Shared conformance scenarios ported. **0.4.0.** | 4 weeks | -| **M4 — Provisioning** | Spec, plan/apply/drift, `inputSchema: auto`, policy allow-list, GitHub Action. Plus `Governance.export()` and `Publication.export()`. Or the Terraform-generation pivot. **0.5.0.** | 5 weeks | -| **M5 — Hardening** | Perf, telemetry polish, error-message pass, migration guide, examples for all eight, security review. **1.0.0.** | 3 weeks | - -`export()` — for both the governance and publication objects — lands with M4, since it emits the M4 spec format. Ship M2.5 and M2.7 with their local and runtime verbs only, and document `export()` as coming. - -**M2.6 splits into two halves with very different dependencies, and only one of them is fenced.** The `PolicyPlugin` interface (§6.10) depends on nothing outside this repo, is worth building for customers who already have their own PDK policies — §6.4's `acme-custom-redaction` is not hypothetical — and is the thing that keeps `core/` from accreting special cases. Build it. The four builtin plugins are the SDK's half of the §6.9 contract: they depend on the companion policy repo existing and on §6.9.7 being answered, so they are excluded from the total below. A deliverable that blocks on a different team's Rust has no business on this plan's critical path. - -What is fenced off from both is the one-line version: emit a plain `x-agent-run-id` header in M1 so a platform team can point the stock `llm-token-rate-limit` `keySelector` at it and get run-scoped budgets with no custom policy and no plugin at all (§6.9.2). Do that in M1 regardless of whether the rest of M2.6 is ever funded. - -Roughly ten to eleven months for two engineers, up from six. M2.9 depends entirely on M2.8 — a scanner is only as good as the derivation underneath it — but its coverage report (§7.10.7) is read-only and needs no write path, so that one piece can ship alongside M2.7 as a platform-team on-ramp. M2.8 is the single highest-leverage block in the plan after M1 — it is what makes the catalog self-maintaining instead of self-rotting — but it is also the block most exposed to upstream framework churn, so it needs the nightly matrix in place before it starts. Ship M1 publicly rather than waiting — the LLM proxy adapters are useful alone, and early feedback will reorder M2–M4. - -### 9.2 Definition of done, per adapter - -1. Passes the full conformance kit, or has an asserted documented exemption. -2. A runnable example in `examples//` that works against a sandbox with only env vars set. -3. A docs page with the manual equivalent — the three lines of code the adapter replaces. Users must be able to eject. -4. Listed in the nightly matrix. -5. Version floor declared, no ceiling. - -### 9.3 The unsupported boundary - -Maintain `docs/unsupported-boundary.md` listing every platform API the SDK calls, classified: - -- **Documented and public** — safe. -- **Documented but no SLA for third-party use** — will break, we'll fix. -- **Undocumented** — should be empty. If anything lands here, it needs a written justification and an owner. - -Link it from the README above the fold. Enterprise buyers will ask; having the answer pre-written converts a two-week procurement stall into a five-minute conversation. - ---- - -## 10. Risk register - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| MuleSoft ships a first-party Python/TS SDK | Medium-high | Existential | Talk to MuleSoft product in week 1 of M0. Ask for a written answer. Offer to be a design partner. | -| MCP Bridge has no provisioning API | Medium | M4 only | Pivot to Terraform generation (§5.5). M1–M3 unaffected. | -| Attribution header names not exposed | Low-medium | High — kills cost attribution | Surfaced in M0. If unavailable, escalate to MuleSoft; ship with correlation-ID-only telemetry and document the gap. **§6.9.2 makes this moot if built** — a signed, self-defined attestation header the companion policy reads replaces the platform header the SDK is waiting on. | -| Client-side tool filtering mistaken for tool authorization | **High** | **High — a prompt-injected agent calls any tool its credential reaches** | Document `ToolSet.filter` as advisory in §4.3, not a security control. Gateway-enforced fix is §6.9.5, and it needs the attestation primitive first. | -| Runaway agent loop exhausts a shared credential's token budget | High | Medium — the failure lands on an innocent agent | Run-scoped keying: `keySelector` on the SDK's run-id header (free, M1), then §6.9.3 for step/depth limits. | -| A2A delegation cycle (A→B→A) with no depth guard | Medium | Medium — unbounded spend, no client-side fix | Client-side depth counting is defeated by the first non-SDK participant. `maxDepth` in §6.9.3 is the only real guard. | -| Companion policy contract drifts from the SDK's emitted header | Medium | High — every governed call rejects at once | Versioned wire format (`v1.…`), compat matrix in the companion repo, SDK tolerates the previous major, `observe` mode default (§6.9.1, §6.9.6). | -| Custom policies do not actually run in Local Mode as §6.4 claims | Medium | Medium — kills the §6.9.6 local dev-loop argument | M0 gate (§6.9.7). Falls back to the `connected` rung, which needs an org. | -| **A third-party policy plugin weakens governance** — suppresses a rejection, overwrites an attribution header | Medium | **High — the extension point disables the product** | `classify` can only decline, never succeed; core headers written last and win; collisions fatal at startup; conformance `plugin_cannot_suppress` (§6.10.4). | -| Auto-loaded plugin becomes a supply-chain vector | Medium | High — installed code injecting headers into every governed call | Discovery is automatic, **activation is not**: inert until named in a committed `.agent-fabric.toml` (§6.10.2). | -| Policy plugin mistaken for policy approval | Medium | High — reviewer concludes the §5.4 allow-list is bypassable | Separate files, separate owners, stated in §5.4 and §6.10.4; `apply` refuses a non-allow-listed policy regardless of plugins. | -| Policy config schema drifts from the plugin after a gateway upgrade | High over time | Medium — every apply fails, cause non-obvious | `agent-fabric policy check` diffs the plugin schema against `api-mgr:policy:describe` output; wired into `lint` and the scheduled drift run (§6.10.5). | -| The plugin interface is a facade over four hard-coded policies | Medium | Medium — customers with their own PDK policies stay locked out | The four §6.9 policies ship **as plugins in `policies/`**, under the same import-linter rule as any third party (§6.10.6). | -| Framework churn breaks adapters | Certain | Medium | Nightly matrix (§8.4). Native-object design (§3.1) minimises blast radius. | -| Anthropic-native proxy route unavailable | Medium | Medium — Anthropic adapter cannot reach a working upstream | M0 gate (§3.3). One-time `UnverifiedValueWarning`; `base_url` overridable; the seven other adapters are unaffected. | -| CrewAI/ADK LiteLLM transport blocks per-run correlation | Certain | Low | Documented, asserted conformance exemption (§8.1); LiteLLM logger callback may recover it later. | -| Anypoint API changes break control-plane calls | Medium | Medium | Contract fixtures + monthly probe re-run (§8.2). | -| Scope creep into agent-network authoring | High | High | It is in §0.2. Point at it in every scope discussion. | -| **Local Mode cannot run LLM Proxy or MCP Bridge** | **Medium-high** | **Medium — was High before §6.8** | M0 gate (§6.7). Ship the mock proxy either way; label simulation loudly (§6.4). **A self-managed Connected Mode gateway running locally has both for real (§6.8)**, so this became a cost question rather than a capability ceiling. | -| Local Mode needs a control-plane licence artifact | Medium | Medium — blocks OSS contributors and CI | M0 gate. Fall back to mock-proxy-only local loop; `connected` rung for anyone with an org. | -| **Orphaned dev API instances accumulate in the control plane** | **High if unguarded** | **Medium — quota exhaustion and a mess nobody owns** | Deterministic naming, create-if-absent, teardown on exception, `agent-fabric dev reap` TTL reaper (§6.8.3). | -| Dev harness pointed at a production environment by a stray env var | Low-medium | **High** | Hard environment fence: refuse any environment not explicitly marked a dev target, and refuse outright on production (§6.8.3). | -| Gateway registration artifact leaked or committed | Medium | High — lets the holder register a gateway into the org | Treated as a credential: gitignored, never logged, per-developer; `dev up` refuses if git tracks it (§6.8.3). | -| Connected local dev loop silently spends LLM budget | Medium-high | Medium — feature switched off after the first invoice | Token-budget policy applied in the dev environment by default, conservative `max_tokens`, cumulative spend printed at teardown (§6.8.3). | -| A test expecting `connected` silently runs on `mock` | Medium | High — a false pass on the policy behaviour that matters most | `env.fidelity` queryable; `simulate(mode=...)` fails rather than downgrading; conformance `fidelity_never_downgrades` (§6.8.1). | -| Developers trust a green `simulate()` run | High | High — ungoverned code reaches prod | Non-suppressible skipped-policy warning (§6.4). `resolve()` drift check at startup as the real gate. | -| `governed=True` silently returns empty | High | Medium — SDK gets blamed | `explain()` (§6.1.2) plus a reason attached to every exclusion. | -| Governed-discovery join is slow on large catalogs | Medium | Medium | Bulk index + Exchange-side prefilter (§6.1.3). `warm()` at startup. | -| Governance object read as app-team policy authorship | Medium | High — security review rejection | Three-verb split, no runtime `apply()` (§6.3). | -| Exchange lacks first-class MCP/agent asset types | Medium | High — weakens both §7 and `governed` filtering in §6.1 | Single M0 gate serving both features (§7.9). Fall back to tagged custom types. | -| Catalog spam from per-merge publishing | High if unguarded | Medium — erodes registry trust | Content digest + `--if-changed` as CI default + `pinned` version strategy (§7.5). | -| Auto-generated descriptions make useless tools look documented | High | Medium | Fail publication on missing or tautological descriptions (§7.3.3). | -| Framework tool-object internals change, breaking descriptor derivation | High | Medium | Conformance scenario `descriptor_matches_framework` + nightly matrix (§8.4). | -| Importing user code for introspection triggers side effects | Medium | Medium | Explicit entrypoints, documented import-safety contract, `auto:static` fallback (§7.3.2). | -| `auto:static` silently under-reports tools | Medium | High — incomplete catalog looks complete | Completeness warning on unresolved dynamic registration; never the default (§7.3.2). | -| Auto-derived A2A cards advertise dozens of micro-skills | High if unguarded | Medium | Require explicit skill declaration; fail rather than invent (§7.3.4). | -| **Fleet scanner treated as a central publisher, holding org-wide Exchange write scopes** | **Medium** | **High — one operator becomes the org's publisher of record; §7.7 is void** | Scanner never publishes; it opens PRs. Tier above `propose` raised only by a file in the target repo (§7.10.2, §7.10.3). | -| Central scanner importing untrusted repo code | Medium | High — arbitrary code execution across the org with a write credential in scope | Fleet scanning is static-first and cannot be switched to import mode for repos it does not own (§7.10.3). | -| Bulk scan floods the catalog with low-quality entries | High if unguarded | High — erodes registry trust at fleet scale | Quality gate not downgradable by the scanner; A2A permanently `propose`-only; cross-repo dedupe holds back clusters (§7.10.6, §7.10.8). | -| Scheduled fleet re-scan becomes a PR spam generator | High if unguarded | Medium — feature gets switched off | Digest-gated re-scan; one PR and one issue per repo, updated in place (§7.10.8). | -| `auto:static` under-reporting is invisible at fleet scale | Medium | Medium — 200 partial descriptors look like 200 complete ones | Completeness warning surfaced in the proposal PR body, where the owning team sees it (§7.10.3). | -| Publication seen as a threat to catalog ownership | Medium | High | Own-business-group default, shared-target flag, platform-owned allow-list (§7.7). | -| Maven-only publication path for non-Mule assets | Medium | Medium — JVM in Python/TS CI | M0 gate (§7.9). Prefer Exchange REST API; document the JVM requirement if unavoidable. | -| Security review rejects policy-from-code | Medium | High | Allow-list catalog + CI-only apply, shipped in v1 (§5.4). | - ---- - -## 11. Working instructions for the implementing model - -1. **Verify before you build.** §0.3 is not optional and not parallelisable with implementation. If a doc page contradicts this spec, the doc page wins — update this spec and note the change. -2. **Never invent an endpoint, header name, or class name.** If you cannot verify it, write the code path with an explicit `NotImplementedError("blocked on verification: ")` and report it. A stub that raises is honest; a guess that 404s is not. -3. **Generate from OpenAPI where a spec exists.** Hand-written HTTP clients drift. If MuleSoft publishes an OpenAPI document for a surface you need, generate the client and commit the generator config. -4. **Write `scripts/probe.py` before `errors.py`.** The error taxonomy must be derived from observed responses. -5. **Build the conformance kit before the second adapter.** Building it after three adapters means retrofitting three adapters. -6. **Keep `core` framework-free.** Add the import-linter CI rule on day one, not at M5. -7. **Every error message names the next action.** No bare exception types reaching users. -8. **Type everything.** `mypy --strict` on Python, `strict: true` on TS, both blocking in CI. This SDK's differentiator over the existing MCP-tool-based workflow is that it is typed and testable. Untyped code forfeits the entire premise. -9. **Optional extras are genuinely optional.** CI must include a job that installs only the base package and runs the base test suite, to catch accidental top-level framework imports. -10. **When a framework's idiom conflicts with SDK consistency, the framework wins.** Users came from the framework, not from us. -11. **Nothing in the runtime path mutates shared state.** `resolve()` and `verify()` read. `simulate()` touches only ephemeral local resources. Every mutation of a gateway or the catalog happens in CI, from a reviewed spec, under platform-controlled credentials. If you find yourself writing a network call that creates or updates a control-plane or Exchange resource outside `provisioning/`, stop — it is in the wrong module. -12. **An extension point may add governance, never remove it.** The `PolicyPlugin` interface (§6.10) exists so `core/` does not accrete a special case per custom policy. It is not a hook for changing what the SDK enforces. A plugin can contribute a header, type a rejection, validate a config; it cannot overwrite a core header, cannot turn a rejection into a success, and cannot do I/O on the request path. If you find yourself adding a plugin return value that means "allow this," stop — you are building a governance bypass with a plugin API on the front of it. diff --git a/spec/donkey-development-kit-build-guide.md b/spec/donkey-development-kit-build-guide.md index 7de96f51..a176a37a 100644 --- a/spec/donkey-development-kit-build-guide.md +++ b/spec/donkey-development-kit-build-guide.md @@ -6,7 +6,7 @@ > **Note on API names.** Every Python signature below is a *proposal* to give the team something concrete to react to. Adjust freely. What must not change is the behaviour and the acceptance criteria. -> **Note on gateway contracts.** Header and body shapes quoted below (`x-token-*`, `pii_detected`, `x-injection-protection`) come from the public Omni Gateway policy docs (v1.11–v1.13). Re-verify against the current docs version before hard-coding anything — that is exactly the repo's existing §0.3 discipline, keep it. +> **Note on gateway contracts.** Header and body shapes quoted below (`x-token-*`, `pii_detected`, `x-injection-protection`) come from the public Omni Gateway policy docs (v1.11–v1.13). Re-verify against the current docs version before hard-coding anything — that is exactly the repo's existing verification discipline, keep it. --- diff --git a/spec/donkey-development-kit-build-plan.md b/spec/donkey-development-kit-build-plan.md index 64eb6e7a..2e87fdba 100644 --- a/spec/donkey-development-kit-build-plan.md +++ b/spec/donkey-development-kit-build-plan.md @@ -4,10 +4,9 @@ [`donkey-development-kit-build-guide.md`](donkey-development-kit-build-guide.md) are the spec for what gets built and in what order. -**Date:** September 2026. Supersedes -[`archive/donkey-development-kit-build-plan-v1.md`](archive/donkey-development-kit-build-plan-v1.md), -whose three-pillar model, provisioning control plane, and eight-adapter -conformance roster are cut. +**Date:** September 2026. Supersedes the retired v1 plan, whose three-pillar +model, provisioning control plane, and eight-adapter conformance roster are +cut. ## What this document is, and what the other two are @@ -17,30 +16,28 @@ Three documents, one job each. Read them in this order: |---|---|---| | **This plan** | Phases, milestones, label taxonomy, implementation order, standing invariants | `Phase N`, or the section name | | **[Build guide](donkey-development-kit-build-guide.md)** | Feature-by-feature scope: what each capability is, its scenario, its acceptance bar | `BG §1.1` … `BG §3.5` | -| **[Archived v1 plan](archive/donkey-development-kit-build-plan-v1.md)** | Nothing current. Retained so legacy citations resolve | bare `§N.N` | The per-issue detail is **not** in any of them. It is in the GitHub issues, because the issue is the plan — see the backlog index below. ## Citation convention -The v1 plan's numbering is load-bearing: roughly **500 `§N.N` citations** -across ~70 files in `python/` and `docs/` point into it. The build guide -independently numbers its own sections `1.1`–`3.5`. Those two schemes -collide — v1 `§1.1` is the layered-architecture rule, build guide `1.1` is -the LLM client — so they are kept textually distinct: +Two active forms, kept textually distinct because their numbers would +otherwise collide (the layered-architecture rule versus the build guide's +LLM-client section): - **`BG §1.1`** — a build guide section. Always prefixed `BG`. This is the - form to use in all new code, tests, issues, and commit messages that - reference feature scope. -- **bare `§1.1`** — a section of the **archived v1 plan**. Every existing - citation in the tree means this. Do not add new ones. + form to use in all code, tests, issues, and commit messages that reference + feature scope. - **`Phase N`** — a milestone in this plan. Named, never numbered with `§`. -Migrating the ~500 legacy citations is deliberately **out of scope** for the -document swap and still needs its own issue. Until that lands, a bare `§N.N` -is a valid pointer into the archive, not a dangling reference — which is the -whole reason the archive is retained rather than deleted. +The five **standing invariants** below keep a bare `§` label — `§0.3`, +`§1.1`, `§2.1`, `§8.1`, `§8.4` — as their stable historical name. Those five +headings are the **only** bare `§` anchors that remain in the tree; the +~500 legacy citations that used to point into the retired v1 plan have been +migrated to `BG §N.N`, a named invariant, or a `Phase N` (issue #266). Cite +an invariant by its `§` label or its section name; do not introduce new bare +`§` numbers anywhere else. ## The product thesis @@ -85,7 +82,7 @@ target; two are standing and unversioned. | `Phase 3 — Platform capabilities (0.3.0)` | 6 | Policy handshake, to-the-code push, structured output + eval hooks. Largely gated on the Upstream gaps milestone. Build guide 3.2-3.4. | | `Phase 4 — Enterprise readiness (0.4.0)` | 10 | Security review, performance budget, error-message pass, migration and deprecation policy, compliance evidence, log shipping, residency, workload identity, support model. | | `Phase 5 — Complete rollout (1.0.0)` | 6 | TypeScript port, remaining adapters by demand, go-to-market, 1.0 stability guarantees. Build guide 3.5, gated on Python PMF. | -| `Verification` | 6 | The §0.3 worklist: never invent an endpoint, header, class name or kwarg. Cross-phase, unversioned — each row blocks specific feature work. | +| `Verification` | 6 | The verification-discipline worklist: never invent an endpoint, header, class name or kwarg. Cross-phase, unversioned — each row blocks specific feature work. | | `Upstream gaps` | 6 | Product asks to the Omni Gateway team. Filed in Phase 1, landing whenever the gateway ships them. Build guide 3.1. | `Verification` and `Upstream gaps` do not complete and are not versioned. @@ -96,8 +93,9 @@ needs it starts will block that phase. Started early, it resolves in time. ## Standing invariants These survive the strategy change unaltered and several are enforced in -CI. They are stated here because the document that used to own them is now -archived; the `§` anchors resolve into that archive. +CI. Each keeps its bare `§` label as a stable historical name, and the +headings below are now their canonical home — the `§` anchors resolve here, +not into any archive (the v1 plan that used to own them is deleted, #266). ### Verification discipline (`§0.3`) — the most important rule @@ -648,7 +646,7 @@ which part of the guide is its spec. | Label | Meaning | |---|---| -| `blocked-on-verification` | Cannot be closed by writing code (§0.3) | +| `blocked-on-verification` | Cannot be closed by writing code (verification discipline) | | `upstream-gap` | A product ask to the Omni Gateway team, not SDK code | | `epic` | Tracking issue spanning several child issues | | `six-piece-minimum` | Part of the minimum set that justifies installing the SDK | diff --git a/website/.gitignore b/website/.gitignore index db2ba226..6b1f88bb 100644 --- a/website/.gitignore +++ b/website/.gitignore @@ -5,6 +5,12 @@ out/ next-env.d.ts .vercel/ +# Auto-generated by `next dev`/`next build` (Next 16 agent-rules feature; +# node_modules/next/dist/server/lib/generate-agent-files.js). Regenerated on +# every build, so they are artifacts, not sources — like next-env.d.ts above. +AGENTS.md +CLAUDE.md + # Logs npm-debug.log* yarn-debug.log* diff --git a/website/README.md b/website/README.md index d4f883be..b936e79e 100644 --- a/website/README.md +++ b/website/README.md @@ -1,8 +1,10 @@ # Donkey Development Kit — documentation site -A [Nextra](https://nextra.site) (Next.js + MDX) documentation site for the -Donkey Development Kit. Content lives in `pages/**/*.mdx`; navigation is -declared in the `_meta.js` files next to the pages. +A [Nextra 4](https://nextra.site) (Next.js App Router + MDX) documentation site +for the Donkey Development Kit. Content lives in `content/**/*.mdx`; navigation +is declared in the `_meta.js` files next to the pages. A single catch-all route, +`app/[[...mdxPath]]/page.tsx`, renders every page, and the theme is composed in +`app/layout.tsx` (Nextra 4 removed `theme.config.tsx`). ## Local development @@ -19,6 +21,11 @@ npm run build # `output: 'export'` → writes a static site to ./out npm run preview:pages ``` +Search is [Pagefind](https://pagefind.app) (Nextra 4 replaced the built-in +FlexSearch). It indexes the built HTML, so the `postbuild` hook runs +`pagefind --site out --output-subdir _pagefind` after every `next build` — which +means **search only works against the built export, not `npm run dev`.** + To preview exactly as GitHub Pages serves it — under the project sub-path: ```bash @@ -74,7 +81,7 @@ future custom domain serve at the root without the sub-path. ## Structure ``` -pages/ +content/ index.mdx Introduction — what the SDK is quickstart.mdx First governed request feature-overview.mdx The three pillars at a glance @@ -86,7 +93,7 @@ pages/ reference/ Configuration, unsupported boundary ``` -## Editing rules (inherited from the SDK — §0.3) +## Editing rules (inherited from the SDK — verification discipline) **Never document an endpoint, header, or class name that isn't verified.** Where a value is unconfirmed, say so on the page (see the "Verification policy" page). diff --git a/website/app/[[...mdxPath]]/page.tsx b/website/app/[[...mdxPath]]/page.tsx new file mode 100644 index 00000000..b7ee963d --- /dev/null +++ b/website/app/[[...mdxPath]]/page.tsx @@ -0,0 +1,35 @@ +import { generateStaticParamsFor, importPage } from 'nextra/pages' +import { useMDXComponents as getMDXComponents } from '../../mdx-components' + +// Catch-all route that renders every .mdx under content/. generateStaticParams +// enumerates all routes at build time, which is what makes `output: 'export'` +// (static GitHub Pages) work. +export const generateStaticParams = generateStaticParamsFor('mdxPath') + +type PageProps = { + params: Promise<{ mdxPath?: string[] }> +} + +export async function generateMetadata(props: PageProps) { + const params = await props.params + const { metadata } = await importPage(params.mdxPath) + return metadata +} + +const Wrapper = getMDXComponents().wrapper + +export default async function Page(props: PageProps) { + const params = await props.params + const result = await importPage(params.mdxPath) + const { default: MDXContent, toc, metadata, sourceCode } = result + return ( + + {/* Nextra 4 no longer wraps page markup in `.nextra-content`; this + `.af-prose` div is the stable hook styles/globals.css keys the prose + refinements off (see the typography section there). */} +
+ +
+ + ) +} diff --git a/website/app/layout.tsx b/website/app/layout.tsx new file mode 100644 index 00000000..8313efee --- /dev/null +++ b/website/app/layout.tsx @@ -0,0 +1,106 @@ +import type { Metadata } from 'next' +import { Inter } from 'next/font/google' +import { Footer, Layout, Navbar } from 'nextra-theme-docs' +import { Head } from 'nextra/components' +import { getPageMap } from 'nextra/page-map' +import 'nextra-theme-docs/style.css' +import '../styles/globals.css' + +const SITE_NAME = 'Donkey Development Kit' +const SITE_DESCRIPTION = + 'An SDK for Agent Fabric — governed model access from your own agent framework, with budget, refusals, telemetry and simulation on one skeleton.' + +// next/image's unoptimized loader does not prefix basePath for /public assets, +// so reference the served copy under website/public/img/ with the base path +// applied manually — same convention as components/index.tsx. +const BASE_PATH = process.env.NEXT_PUBLIC_BASE_PATH ?? '' + +const inter = Inter({ + subsets: ['latin'], + display: 'swap', + variable: '--font-inter', +}) + +// Replaces theme.config.tsx `head()`: the Metadata API builds the same +// `${title} — ${SITE_NAME}` titles (root page keeps the bare site name via +// `default`) and the shared description / OpenGraph tags. Per-page descriptions +// come from each page's frontmatter, resolved by generateMetadata in the +// [[...mdxPath]] route. +export const metadata: Metadata = { + title: { + default: SITE_NAME, + template: `%s — ${SITE_NAME}`, + }, + description: SITE_DESCRIPTION, + openGraph: { + title: SITE_NAME, + description: SITE_DESCRIPTION, + }, +} + +const navbar = ( + + + {SITE_NAME} + + } + projectLink="https://github.com/Donkey-Development-Kit/donkey-development-kit" + /> +) + +const footer = ( +
+ + Donkey Development Kit — an SDK for Agent Fabric. “Agent Fabric”, + “Anypoint”, and “Omni Gateway” are Salesforce trademarks; this project is + descriptive (the trademark/support boundary). + +
+) + +export default async function RootLayout({ + children, +}: { + children: React.ReactNode +}) { + const pageMap = await getPageMap() + return ( + + {/* Violet accent, close to the reference docs look (was theme.config `color`). */} + + + + {children} + + + + ) +} diff --git a/website/pages/_meta.js b/website/content/_meta.js similarity index 89% rename from website/pages/_meta.js rename to website/content/_meta.js index 0dfa0562..1615dbfa 100644 --- a/website/pages/_meta.js +++ b/website/content/_meta.js @@ -19,6 +19,11 @@ export default { testing: 'Testing & conformance', telemetry: 'Telemetry & cost', cli: 'CLI & decorators', + '-- scenarios': { + type: 'separator', + title: 'Scenarios — the pieces together', + }, + scenarios: 'Scenarios', '-- phase-2': { type: 'separator', title: 'Phase 2 — differentiate', diff --git a/website/pages/a2a.mdx b/website/content/a2a.mdx similarity index 100% rename from website/pages/a2a.mdx rename to website/content/a2a.mdx diff --git a/website/pages/budget.mdx b/website/content/budget.mdx similarity index 100% rename from website/pages/budget.mdx rename to website/content/budget.mdx diff --git a/website/content/cli.mdx b/website/content/cli.mdx new file mode 100644 index 00000000..975eca9f --- /dev/null +++ b/website/content/cli.mdx @@ -0,0 +1,141 @@ +--- +description: Decorators and a four-command CLI so the six pieces are reachable in one line. +--- + +import { Callout } from 'nextra/components' + +# CLI & decorators + + + **Phase 1 — the decorators and the four-command CLI are shipped.** Both + decorators below (`@donkey.governed` / `@donkey.tool`) have landed (#200), and + the CLI now exposes its four commands — `init`, `doctor`, `mock`, and `test` + (#201, #202) — with consistent global flags and exit codes. See + [Roadmap](/roadmap) and [Verification policy](/concepts/verification). + + +Two on-ramps. Everything on the [Roadmap](/roadmap)'s six-piece list attaches +to the same transport hooks, and these are how you reach all of it without +wiring each piece by hand. + +## Decorators + +`@donkey.governed` runs a function inside a [`donkey.run()`](/telemetry) scope: + +```python +@donkey.governed(team="support") +async def handle_ticket(ticket): ... +``` + +One decorator gives that function a run/correlation ID, cost tags, an OTel span, +and typed refusals — the four things you would otherwise set up per call site. +It wraps **both sync and async** callables, works bare (`@donkey.governed`) or +parametrised (`@donkey.governed(team=…, project=…, env=…, enduser_id=…)`), and +deliberately takes **no `id=`**: each call opens its own run (a "run of one"), +and a fixed id pinned across calls would collapse unrelated runs into one +correlation. When you need a specific id, use `donkey.run(id=…)` directly. + +```python +@donkey.tool +async def lookup_crm(customer_id: str) -> dict: + """Look up a customer record by id.""" + ... +``` + +`@donkey.tool` marks a function as a governed tool **without changing how it's +called** — it returns the same function with a `__donkey_tool__` marker and +records a `ToolSpec` (name, qualname, signature, docstring, `is_async`) in a +process-global registry you read with `registered_tools()`. A tool with **no +docstring is rejected at decoration time** (`ValueError`) — an undescribed tool +is useless to a model and to the registry. `ToolSpec` and `registered_tools` +are exported from `donkey_kit`. + +The registry exists in Phase 1; its **consumers** are Phase 2 — the +[scanner](/publishing) reads these markers to derive a manifest from your code, +and the [A2A](/a2a) agent-card generator reads the same markers to build a card. +Marking tools early costs nothing and saves the migration later. + +## The CLI + +```bash +donkey init # writes a commented .donkey-kit.toml, names every missing env var at once +donkey doctor # checks creds, reaches the gateway, reports budget state +donkey mock # the local simulator — --scenario scripts failures +donkey test # a thin front end to pytest --donkey-conformance +``` + +All four share three global flags, which precede the subcommand: + +```bash +donkey --config ./cfg.toml init # write/read a config file at a non-default path +donkey --env Production doctor # override the Anypoint environment +donkey --json init # machine-readable output where a command supports it +``` + +Every command exits non-zero on failure, so any of them drops into CI as a +preflight. A command that needs an optional extra (`donkey mock` → `[local]`, +`donkey test` → `[test]`, `donkey doctor` → `[llm]`) prints the exact +`pip install` line and exits `1` — never a stack trace. + +### `donkey init` bootstraps your config + +`donkey init` resolves your current configuration (kwargs → env vars → +`.donkey-kit.toml` → defaults) and writes a **commented** `.donkey-kit.toml` +with the values it found. Three properties matter: + +- **It names every missing required field at once** — control-plane *and* proxy + — reusing the same validation the SDK runs at a live call, so `init` and a + real request never disagree about what is required. +- **It never writes a secret.** `client_secret`, `llm_proxy_client_secret`, and + `llm_proxy_key` are emitted as commented `env` pointers, not values — a + committed config file is the wrong home for a credential. +- **It is idempotent.** An existing file is left untouched unless you pass + `--force`. + +### `donkey test` runs the conformance suite against your agent + +`donkey test` is a thin front end to `pytest --donkey-conformance` — it does +not re-implement the runner. Point it at your agent factory and pass any +trailing pytest arguments straight through; pytest's exit code becomes +`donkey test`'s own. + +```bash +donkey test --agent my.pkg:make_agent -k governance -x +``` + +### `donkey doctor` is the one that pays for itself + +A governed call can fail for several reasons that all look identical from the +outside. `doctor` exists to tell them apart: + +- **Wrong credentials** — the `client_id`/`client_secret` pair is rejected. +- **Wrong URL** — the credentials are fine but the base URL is not a proxy + instance. (The trailing `/v1` mistake lands here; the governed proxy has no + `/v1` segment.) +- **Credentials fine, model not in the allow-list** — nothing is misconfigured + at all; your platform team has not granted that model. + +Each answer comes with the same remediation string the matching typed +exception carries, so the fix is in the output rather than in a runbook. This +is the difference between an afternoon of guessing and thirty seconds. + +## Acceptance bar + +- `donkey --help` lists exactly the four commands: `init`, `doctor`, `mock`, + `test`. +- Global `--config` / `--env` / `--json` are consistent across the commands + that support them, and every command exits non-zero on failure. +- `donkey init` writes a commented `.donkey-kit.toml` with the resolved values, + lists every missing required field in one run, never writes a secret, and does + not clobber an existing file without `--force`. +- `donkey doctor` distinguishes those three failures from each other — not one + generic "could not connect." Each verdict prints the remediation string from + the corresponding typed refusal, so the CLI and the exception never disagree. +- `donkey test` forwards to `pytest --donkey-conformance` and propagates its + exit code. + +--- + +**Status: Phase 1 — the `@donkey.governed` / `@donkey.tool` decorators (#200) +and the four-command CLI (`init`, `doctor`, `mock`, `test`) are shipped (#201, +#202).** diff --git a/website/pages/concepts/_meta.js b/website/content/concepts/_meta.js similarity index 100% rename from website/pages/concepts/_meta.js rename to website/content/concepts/_meta.js diff --git a/website/pages/concepts/attribution.mdx b/website/content/concepts/attribution.mdx similarity index 100% rename from website/pages/concepts/attribution.mdx rename to website/content/concepts/attribution.mdx diff --git a/website/pages/concepts/environments.mdx b/website/content/concepts/environments.mdx similarity index 100% rename from website/pages/concepts/environments.mdx rename to website/content/concepts/environments.mdx diff --git a/website/pages/concepts/governance.mdx b/website/content/concepts/governance.mdx similarity index 100% rename from website/pages/concepts/governance.mdx rename to website/content/concepts/governance.mdx diff --git a/website/pages/concepts/observability.mdx b/website/content/concepts/observability.mdx similarity index 98% rename from website/pages/concepts/observability.mdx rename to website/content/concepts/observability.mdx index 3c3179b5..a24c3fd5 100644 --- a/website/pages/concepts/observability.mdx +++ b/website/content/concepts/observability.mdx @@ -80,8 +80,7 @@ The proxy routes to several providers (OpenAI, Azure OpenAI, Gemini, Bedrock, Anthropic), so `gen_ai.system` is taken **only** from the verified `x-llm-proxy-llm-provider` response header. If that header is absent on a given response, the attribute is omitted rather than defaulted — a wrong provider -label is worse than a missing one ([Verification policy](/concepts/verification), -§0.3). +label is worse than a missing one ([Verification policy](/concepts/verification)). ## Grouping a run: the run id on the span, the call id on the error @@ -120,7 +119,7 @@ present. the join key is not yet confirmed, so those request-header **names** are overridable placeholders — set `correlation_header` / `call_id_header` in config if your gateway expects different names - ([Verification policy](/concepts/verification), §0.3). + ([Verification policy](/concepts/verification)). ## Streaming, refusals, and exceptions @@ -149,7 +148,7 @@ takes, it produces **exactly one span**, and that span closes: event.** For OpenAI-style Chat Completions that means requesting it with `stream_options={"include_usage": true}`; without it the provider emits no usage event, so the counts are omitted rather than guessed - ([Verification policy](/concepts/verification), §0.3). + ([Verification policy](/concepts/verification)). ## What this piece does *not* do yet diff --git a/website/pages/concepts/verification.mdx b/website/content/concepts/verification.mdx similarity index 91% rename from website/pages/concepts/verification.mdx rename to website/content/concepts/verification.mdx index dbcb08bb..821d9b0a 100644 --- a/website/pages/concepts/verification.mdx +++ b/website/content/concepts/verification.mdx @@ -14,7 +14,7 @@ Anypoint sandbox / an installed framework**, or it is **not used** — the code path that would need it raises `NotImplementedError("blocked on verification: …")` or a clear `ConfigError` instead of guessing. How each value is confirmed — and how a blocked value flips to verified — is documented for contributors in -[ARCHITECTURE.md → Verification discipline](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/ARCHITECTURE.md#verification-discipline-03). +[ARCHITECTURE.md → Verification discipline](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/ARCHITECTURE.md#verification-discipline). ## What is verified today @@ -31,7 +31,7 @@ how a blocked value flips to verified — is documented for contributors in | Area | Status | What the SDK does about it | |---|---|---| -| Framework **constructor signatures** (§8) | Unverified per framework | Adapters target recorded class names; a mismatch surfaces via the verification harness / nightly matrix rather than silently misbehaving. Agent Framework's client raises `blocked on verification` today. | +| Framework **constructor signatures** (docs/verified-apis.md §8) | Unverified per framework | Adapters target recorded class names; a mismatch surfaces via the verification harness / nightly matrix rather than silently misbehaving. Agent Framework's client raises `blocked on verification` today. | | MCP tool discovery + binding ([Phase 2](/tool-access)) | Unverified | `donkey.tools.discover` and the binding classes raise `blocked on verification`. | | Exchange publish API ([Phase 2](/publishing)) | Unverified | `donkey publish` is blocked; whether Exchange exposes first-class MCP/agent asset types is unconfirmed. | | Token-exchange endpoint for [on-behalf-of](/identity) (Phase 2) | Unverified | The exact endpoint and header the gateway expects are not guessed. | @@ -49,4 +49,4 @@ hasn't confirmed raises `blocked on verification` rather than guessing, so a governed call either works against the verified contract or fails loudly — it never silently reaches an unconfirmed endpoint. The mechanism behind that and the full verified-versus-blocked ledger are documented for contributors in -[ARCHITECTURE.md → Verification discipline](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/ARCHITECTURE.md#verification-discipline-03). +[ARCHITECTURE.md → Verification discipline](https://github.com/Donkey-Development-Kit/donkey-development-kit/blob/main/ARCHITECTURE.md#verification-discipline). diff --git a/website/pages/errors.mdx b/website/content/errors.mdx similarity index 74% rename from website/pages/errors.mdx rename to website/content/errors.mdx index d5a10ec4..382a0add 100644 --- a/website/pages/errors.mdx +++ b/website/content/errors.mdx @@ -66,6 +66,37 @@ DonkeyError # base of the whole tree something the gateway told the SDK; `GatewayUnavailable` is the gateway not being there to tell it anything. +## Cookbook: every exception — discriminator, retryable, next step + +One row per exception you can catch, with the three facts you need to write a +handler: **what tells it apart** (the discriminator), **whether retrying it can +ever succeed**, and **the next step** its `.remediation` names. "Retryable" here +means *by you* — the transport already retries the only class that is safe to +(`UpstreamModelError`), and treats every governance refusal as terminal so it can +never burn an exhausted budget or replay a blocked prompt. + +| Exception | Discriminator | Retryable? | Next step (`.remediation`) | +|---|---|---|---| +| `AuthError` | `401`, or `403` + `www-authenticate` | **No** — terminal. The async client refreshes its token and retries **once** on a `401`, then surfaces it. | Check the consumer `client_id` / `client_secret` pair and that the consumer is authorized in API Manager. | +| `PIIDetected` | `403`, nested `type: "pii_detected"`, **no** `www-authenticate` | **No** — a `PolicyViolation`, never retried. | Remove or redact the flagged values (`.entities`), or relax the policy's entity list in API Manager. | +| `TokenBudgetExceeded` | `429`, empty body, `x-token-*` headers | **Not immediately** — never auto-retried; only worth retrying *after* the window resets. | Wait for `.retry_after` (seconds) / the reset, then retry — or request an increase in API Manager. | +| `PromptInjectionBlocked` | header `x-injection-protection: blocked`, **or** a top-level `matched_patterns` list (regex prompt guard) | **No** — a `PolicyViolation`, never retried. | Review and sanitise the untrusted input, or adjust the policy's sensitivity / deny-list in API Manager. | +| `ContentSafetyBlocked` | `403` + `x-llm-proxy--…-action: reject` (Azure Content Safety / Bedrock Guardrails) — *body shape pending [#253]* | **No** — a `PolicyViolation`, never retried. | Revise the flagged content (`.categories`), or adjust the policy's categories / severity thresholds in API Manager. | +| `PolicyViolation` (generic) | a `4xx` matching **no** documented rejection shape | **No** — terminal; the shape is simply not yet typed. | Inspect `.response`; file an issue with the status/headers/body so the shape can be typed ([#184]/[#253]). | +| `UpstreamRequestError` | non-`429` `4xx`, nested `error` with `code`/`type`/`param` | **No** — a client-side request mistake passed through the gateway, terminal. | Fix the flagged model or parameter (`.code` / `.param`); if `model_not_found`, request the model in API Manager. | +| `UpstreamModelError` | `5xx` | **Yes** — the transport already retries `502` / `503` / `504`; a persistent `5xx` is safe for you to retry too. | Transient provider failure — retry, then escalate if it persists. | +| `GatewayUnavailable` | transport failure — DNS, refused connection, TLS, timeout — with **no** HTTP response | **Not automatically** — terminal here (failover is separate); you may retry or fall back. | Check host reachability, `.base_url`, and network egress; run [`donkey doctor`](/cli). | + +Two more `DonkeyError`s are **client-side signals**, not gateway refusals, so +they sit outside the retry question: `BudgetReserveReached` is raised *before* a +call by [`donkey.budget.pace()`](/budget) and is meant to be recovered from +(`await donkey.budget.wait_for_reset()`, then continue), and `ModelSubstituted` +reports that a call *succeeded* against a different model than requested (opt-in +via `on_model_substitution="raise"`). `ConfigError` is raised locally, pre-flight, +and reports every missing field at once — fix the config and re-run. + +[#184]: https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/184 + ## When the gateway can't be reached at all Every rejection above describes something the gateway *told* the SDK. @@ -182,5 +213,5 @@ exact bodies are tracked in [#253]. `classify()` therefore types them from the header/field discriminator only, and no verification row flips to `verified` until a live round-trip confirms the shape. Any *other* content-moderation / federated-guardrail shape still falls through to a generic `PolicyViolation` -rather than an invented type — this is deliberate (§0.3): we don't invent a +rather than an invented type — this is deliberate (verification discipline): we don't invent a discriminator we haven't observed. See [Verification policy](/concepts/verification). diff --git a/website/pages/feature-overview.mdx b/website/content/feature-overview.mdx similarity index 80% rename from website/pages/feature-overview.mdx rename to website/content/feature-overview.mdx index 17c573a3..50c7d8e7 100644 --- a/website/pages/feature-overview.mdx +++ b/website/content/feature-overview.mdx @@ -96,6 +96,45 @@ That costs nothing here — the proxy authenticates on the header pair, not a fetched token — but it does mean the control-plane surfaces (`registry`, `tools`) remain async. +## One-line on-ramps: decorators + +Two decorators fold the run scope and the tool registry into a single line, so +the six-piece minimum is reachable without threading anything through your +framework. + +`@donkey.governed` runs a callable inside a [`donkey.run()`](/telemetry) scope — +a fresh run/correlation id per call (a "run of one"), the optional per-run cost +tags, the OTel span, and typed refusals — with nothing to wire up: + +```python +@donkey.governed(team="support") +async def handle_ticket(ticket): ... +``` + +It wraps **both sync and async** callables, works bare (`@donkey.governed`) or +parametrised (`@donkey.governed(team=…, project=…, env=…, enduser_id=…)`), and +deliberately takes **no `id=`**: each call opens its own run, and a fixed id +pinned across calls would collapse unrelated runs into one correlation. When you +need a specific id, use `donkey.run(id=…)` directly. + +`@donkey.tool` marks a callable as a governed tool **without changing how it's +called** — it returns the same function with a `__donkey_tool__` marker and +records a `ToolSpec` (name, qualname, signature, docstring, `is_async`) in a +process-global registry you read with `registered_tools()`: + +```python +@donkey.tool +async def lookup_crm(customer_id: str) -> dict: + """Look up a customer record by id.""" + ... +``` + +A tool with **no docstring is rejected at decoration time** (`ValueError`) — an +undescribed tool is useless to a model and to the registry. That one marker is +what the Phase 2 [scanner](/publishing) and the [A2A](/a2a) agent-card generator +will both read, so the single annotation pays off across surfaces. `ToolSpec` +and `registered_tools` are exported from `donkey_kit`. + ## A typed governed error taxonomy `classify()` maps the proxy's eight rejection shapes to typed exceptions — @@ -134,7 +173,7 @@ The governed proxy has **no** catalog endpoint (`GET /models` → `404`, verifie rather than fabricating a path. - **Honesty note (§0.3).** The proxy *contract* the adapters target is + **Honesty note (verification discipline).** The proxy *contract* the adapters target is live-verified, but the exact framework *constructor signatures* are still being confirmed against installed versions. Where a name can't be resolved, the adapter raises `blocked on verification` rather diff --git a/website/pages/frameworks/_meta.js b/website/content/frameworks/_meta.js similarity index 100% rename from website/pages/frameworks/_meta.js rename to website/content/frameworks/_meta.js diff --git a/website/pages/frameworks/adk.mdx b/website/content/frameworks/adk.mdx similarity index 96% rename from website/pages/frameworks/adk.mdx rename to website/content/frameworks/adk.mdx index aba77484..8b68d483 100644 --- a/website/pages/frameworks/adk.mdx +++ b/website/content/frameworks/adk.mdx @@ -7,7 +7,7 @@ Google's Agent Development Kit (ADK) is Gemini-first; it reaches the Agent Fabric LLM proxy through ADK's `LiteLlm` model wrapper, which speaks LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()`**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected > here (see below). diff --git a/website/pages/frameworks/agent-framework.mdx b/website/content/frameworks/agent-framework.mdx similarity index 96% rename from website/pages/frameworks/agent-framework.mdx rename to website/content/frameworks/agent-framework.mdx index 0b5c6b82..a9d065a1 100644 --- a/website/pages/frameworks/agent-framework.mdx +++ b/website/content/frameworks/agent-framework.mdx @@ -7,7 +7,7 @@ Microsoft Agent Framework gets a governed chat client at the Agent Fabric LLM proxy, plus first-class policy middleware for terminating a run cleanly on a governance rejection instead of letting the agent loop retry. -> **Supported at `connection_kwargs()`.** This adapter also ships the strongest +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** This adapter also ships the strongest > policy-integration story of the eight frameworks (`policy_middleware()`), > but its native client class is young and **UNVERIFIED** — read the callout > below before depending on it. diff --git a/website/pages/frameworks/anthropic.mdx b/website/content/frameworks/anthropic.mdx similarity index 97% rename from website/pages/frameworks/anthropic.mdx rename to website/content/frameworks/anthropic.mdx index a939f721..26d4ec88 100644 --- a/website/pages/frameworks/anthropic.mdx +++ b/website/content/frameworks/anthropic.mdx @@ -7,7 +7,7 @@ The Anthropic SDK gets a governed `AsyncAnthropic` client pointed at the Agent Fabric LLM proxy, with the SDK's shared transport and verified headers injected directly into the client constructor. -> **Supported at `connection_kwargs()`**, with one important divergence from +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one important divergence from > every other adapter in this roster: this factory returns a bare **client**, > not a model-bound object — see below before you reach for it. diff --git a/website/pages/frameworks/crewai.mdx b/website/content/frameworks/crewai.mdx similarity index 96% rename from website/pages/frameworks/crewai.mdx rename to website/content/frameworks/crewai.mdx index c3201b6c..d7f49db1 100644 --- a/website/pages/frameworks/crewai.mdx +++ b/website/content/frameworks/crewai.mdx @@ -7,7 +7,7 @@ CrewAI gets a governed `LLM`, backed by LiteLLM under the hood, pointed at the Agent Fabric LLM proxy through LiteLLM's own model-string and kwarg conventions rather than raw OpenAI ones. -> **Supported at `connection_kwargs()`**, with one documented exemption: LiteLLM +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`)**, with one documented exemption: LiteLLM > owns its own transport, so the SDK's shared httpx client is not injected > here (see below) — the same exemption as the [Google ADK](/frameworks/adk) > adapter. diff --git a/website/pages/frameworks/index.mdx b/website/content/frameworks/index.mdx similarity index 100% rename from website/pages/frameworks/index.mdx rename to website/content/frameworks/index.mdx diff --git a/website/pages/frameworks/langgraph.mdx b/website/content/frameworks/langgraph.mdx similarity index 100% rename from website/pages/frameworks/langgraph.mdx rename to website/content/frameworks/langgraph.mdx diff --git a/website/pages/frameworks/llamaindex.mdx b/website/content/frameworks/llamaindex.mdx similarity index 96% rename from website/pages/frameworks/llamaindex.mdx rename to website/content/frameworks/llamaindex.mdx index 8df639c3..5b9710b6 100644 --- a/website/pages/frameworks/llamaindex.mdx +++ b/website/content/frameworks/llamaindex.mdx @@ -8,7 +8,7 @@ LlamaIndex gets a governed `OpenAILike` LLM pointed at the Agent Fabric LLM prox with the one flag that a chat-only gateway absolutely requires forced on for you. -> **Supported at `connection_kwargs()`.** Full header injection; watch the +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header injection; watch the > `is_chat_model` gotcha below if you ever build the client by hand. ## Install diff --git a/website/pages/frameworks/openai.mdx b/website/content/frameworks/openai.mdx similarity index 96% rename from website/pages/frameworks/openai.mdx rename to website/content/frameworks/openai.mdx index 70541c49..8bc75732 100644 --- a/website/pages/frameworks/openai.mdx +++ b/website/content/frameworks/openai.mdx @@ -9,7 +9,7 @@ The OpenAI Agents SDK (pip package `openai-agents`) gets a governed SDK constructs the OpenAI client itself, wired with its shared http client and verified headers, and hands it to the Agents SDK ready-made. -> **Supported at `connection_kwargs()`.** Full header AND transport injection — +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** Full header AND transport injection — > the adapter builds the `AsyncOpenAI` client itself, so both travel together > as one object. diff --git a/website/pages/frameworks/strands.mdx b/website/content/frameworks/strands.mdx similarity index 96% rename from website/pages/frameworks/strands.mdx rename to website/content/frameworks/strands.mdx index 090972a1..6cbe9671 100644 --- a/website/pages/frameworks/strands.mdx +++ b/website/content/frameworks/strands.mdx @@ -8,7 +8,7 @@ Strands Agents gets a governed `OpenAIModel`, with proxy connection details forwarded through Strands' `client_args`, which Strands passes straight through to the underlying OpenAI client. -> **Supported at `connection_kwargs()`.** `client_args` gives full header AND +> **Supported at `connection_kwargs()` — not conformance-tested (`BG §1.8`).** `client_args` gives full header AND > transport injection, on par with the LangGraph adapter. ## Install diff --git a/website/pages/hitl.mdx b/website/content/hitl.mdx similarity index 100% rename from website/pages/hitl.mdx rename to website/content/hitl.mdx diff --git a/website/pages/identity.mdx b/website/content/identity.mdx similarity index 100% rename from website/pages/identity.mdx rename to website/content/identity.mdx diff --git a/website/pages/index.mdx b/website/content/index.mdx similarity index 98% rename from website/pages/index.mdx rename to website/content/index.mdx index 1c4144f4..9cbad254 100644 --- a/website/pages/index.mdx +++ b/website/content/index.mdx @@ -42,7 +42,7 @@ never a wrapper you have to code around. **"Agent Fabric" is a MuleSoft (Salesforce) product name**, not a generic term. This project is an SDK *for* that product; it is descriptive, not a - first-party release. See [§0.4 in the README](https://github.com/Donkey-Development-Kit/donkey-development-kit). + first-party release. See [the trademark/support boundary in the README](https://github.com/Donkey-Development-Kit/donkey-development-kit). ## The one design rule diff --git a/website/pages/policies.mdx b/website/content/policies.mdx similarity index 100% rename from website/pages/policies.mdx rename to website/content/policies.mdx diff --git a/website/pages/publishing.mdx b/website/content/publishing.mdx similarity index 100% rename from website/pages/publishing.mdx rename to website/content/publishing.mdx diff --git a/website/pages/quickstart.mdx b/website/content/quickstart.mdx similarity index 89% rename from website/pages/quickstart.mdx rename to website/content/quickstart.mdx index 4bf8e1e9..f136d413 100644 --- a/website/pages/quickstart.mdx +++ b/website/content/quickstart.mdx @@ -143,7 +143,22 @@ The *same code* runs against your Omni Gateway proxy — you only change the environment. Stop the simulator and set the three real governed-access values. The proxy authenticates on a `client_id` / `client_secret` **header pair** (consumer auth) — **not** a bearer token, and separate from any Anypoint -control-plane credential: +control-plane credential. + + + **Don't have these values yet?** They come from the platform, not the SDK — + create the proxy, then request access to mint the consumer credentials. The + official docs own each step (linking rather than restating so this page can't + drift from them): + + - **[Create the model proxy](https://docs.mulesoft.com/general/model-proxy-create-model-proxy)** + on the Omni Gateway — its consumer endpoint is your `DONKEY_LLM_PROXY_URL` + (with **no `/v1`**). + - **[Request access](https://docs.mulesoft.com/exchange/to-request-access)** in + Exchange — the proxy is protected by a client-ID enforcement policy, so + requesting access registers a client application and mints the pair that + becomes `DONKEY_LLM_PROXY_CLIENT_ID` / `DONKEY_LLM_PROXY_CLIENT_SECRET`. + ```bash export DONKEY_LLM_PROXY_URL="https:////" # note: no /v1 diff --git a/website/pages/reference/_meta.js b/website/content/reference/_meta.js similarity index 100% rename from website/pages/reference/_meta.js rename to website/content/reference/_meta.js diff --git a/website/pages/reference/configuration.mdx b/website/content/reference/configuration.mdx similarity index 100% rename from website/pages/reference/configuration.mdx rename to website/content/reference/configuration.mdx diff --git a/website/pages/reference/unsupported-boundary.mdx b/website/content/reference/unsupported-boundary.mdx similarity index 86% rename from website/pages/reference/unsupported-boundary.mdx rename to website/content/reference/unsupported-boundary.mdx index d539ecf6..31f90d2d 100644 --- a/website/pages/reference/unsupported-boundary.mdx +++ b/website/content/reference/unsupported-boundary.mdx @@ -16,17 +16,17 @@ Every platform API the SDK calls is classified: | **Documented, no SLA for third-party use** | May break; we'll fix. | | **Undocumented** | Should be empty. Anything here needs a written justification and an owner. | -## Support statement (§0.4) +## Support statement (the trademark/support boundary) **Before any public release**, the maintainer and support expectations must be - stated (README §0.4). If this project is **not** published with MuleSoft's + stated (README, the trademark/support boundary). If this project is **not** published with MuleSoft's endorsement, it must ship under a distinct, org-scoped distribution name so it doesn't read as a first-party, official-status SDK. "Agent Fabric", "Anypoint", and "Omni Gateway" are Salesforce trademarks. -## Why this matters (§0.3) +## Why this matters (verification discipline) The SDK's rule against inventing endpoints exists to keep this boundary honest: a call the SDK makes is either against a classified, known API or it doesn't happen diff --git a/website/pages/roadmap.mdx b/website/content/roadmap.mdx similarity index 100% rename from website/pages/roadmap.mdx rename to website/content/roadmap.mdx diff --git a/website/content/scenarios/_meta.js b/website/content/scenarios/_meta.js new file mode 100644 index 00000000..39d9f864 --- /dev/null +++ b/website/content/scenarios/_meta.js @@ -0,0 +1,6 @@ +export default { + index: 'Overview', + 'support-triage': 'Support triage', + 'nightly-batch': 'Nightly batch', + 'internal-copilot': 'Internal copilot', +} diff --git a/website/content/scenarios/index.mdx b/website/content/scenarios/index.mdx new file mode 100644 index 00000000..3ac148d6 --- /dev/null +++ b/website/content/scenarios/index.mdx @@ -0,0 +1,39 @@ +--- +description: Three end-to-end walkthroughs — support triage, an overnight batch job, and an internal copilot — each showing the governance layer a bare base_url cannot give you. +--- + +import { Callout } from 'nextra/components' + +# Scenarios + +The feature pages tell you what each piece *is*. These pages show three of them +working together on a real job, start to finish — the same three scenarios the +build guide uses to justify the skeleton (`BG §1.8`). Each one is runnable +today against the [local simulator](/simulator), with no Anypoint credentials +and no real gateway, so you can watch the governance branch execute in your own +terminal before it ever runs on a customer's data. + +| Scenario | The job | What it exercises | +| --- | --- | --- | +| [Support triage](/scenarios/support-triage) | Draft replies to a queue of support tickets; one carries PII | Typed refusals ([`PIIDetected`](/errors)), [correlation IDs](/telemetry), [OTel spans](/telemetry) — the shipped LangGraph demo, end to end | +| [Nightly batch](/scenarios/nightly-batch) | Enrich 50,000 records overnight against a windowed budget, unattended | [Budget pacing](/budget) (`pace()` / `wait_for_reset()`) and resume, driven by the [simulator's `budget` scenario](/simulator#scenario-scripting) | +| [Internal copilot](/scenarios/internal-copilot) | An internal assistant whose output must clear a content-safety guardrail | [`ContentSafetyBlocked`](/errors) via the `donkey-sim/content-safety` sentinel, plus per-run correlation | + + + **"Scenario" means two different things in these docs — don't conflate them.** + These pages are the three *product* scenarios (a job you'd actually run). The + [simulator's `--scenario` flag](/simulator#scenario-scripting) names three + *failure-injection rules* (`pii_block`, `budget`, `injection`). The pages + below use those rules as the engine, but the job is the story. + + +## What's runnable, and what's honest about being blocked + +Support triage runs **completely** today — it is the Phase-1 acceptance +artefact (`BG §1.8`), timed in CI so its first-run experience can't rot. The +nightly-batch and internal-copilot walkthroughs run their **governed-call and +refusal-handling** paths against the simulator now; where a step depends on a +surface that is still [blocked on verification](/concepts/verification) — MCP +tool access, agent identity, the kill switch — the page says so in the reader's +terms and shows the shape without inventing an endpoint (verification discipline). A "blocked" +note here means *known, deliberate, and tracked*, never *guessed*. diff --git a/website/content/scenarios/internal-copilot.mdx b/website/content/scenarios/internal-copilot.mdx new file mode 100644 index 00000000..206fbeee --- /dev/null +++ b/website/content/scenarios/internal-copilot.mdx @@ -0,0 +1,127 @@ +--- +description: An internal assistant whose output must clear a content-safety guardrail — the ContentSafetyBlocked branch driven by the simulator, with per-run correlation joining the refusal to your logs. +--- + +import { Callout, Steps } from 'nextra/components' + +# Internal copilot + +An internal copilot answers employee questions against company systems. Its +output has to clear a **content-safety guardrail** before it reaches a person, +and when the guardrail fires you need the refusal to (a) surface as something +you can branch on and (b) carry an id that joins the block back to the run in +your own logs. This page runs that content-safety branch against the [local +simulator](/simulator), and is honest about the parts of a full internal +copilot that are still [blocked on verification](/concepts/verification). + +## What it demonstrates + +- **The content-safety branch executes** as a typed + [`ContentSafetyBlocked`](/errors), driven by the `donkey-sim/content-safety` + sentinel — no need to craft a prompt that a real guardrail would reject. +- **Per-run correlation joins the refusal to your logs.** Inside a + [`donkey.run(id=...)`](/telemetry) block the bound id becomes the + `X-Correlation-Id` on every request *and* lands on + `ContentSafetyBlocked.correlation_id`, so a log line for the block joins to + the gateway's own record with no extra wiring. + +## Run it + + + +### Install the extras + +```bash +pip install "donkey-kit[llm,local]" +``` + +### Boot the simulator + +```bash +donkey mock --port 8080 +``` + +No `--scenario` needed: the [`donkey-sim/` sentinel](/simulator) lets +*you* pick which call fails by setting the request's `model`. The selectable +shapes include `content-safety`. + +### Force a content-safety block and catch it typed + +```python +import asyncio +import openai +from donkey_kit import Donkey, ContentSafetyBlocked +from donkey_kit.core.errors import classify + +async def ask(donkey, question, *, run_id): + client = donkey.llm.client() # the native AsyncOpenAI, governed transport + async with donkey.run(id=run_id): + try: + return await client.chat.completions.create( + # the sentinel forces the captured content-safety 403 from the simulator + model="donkey-sim/content-safety", + messages=[{"role": "user", "content": question}], + ) + except openai.APIStatusError as e: + governed = classify(e.response) # -> a DonkeyError subclass + if isinstance(governed, ContentSafetyBlocked): + print(f"[{run_id}] blocked by guardrail:", governed.remediation) + print(f"[{run_id}] correlation id:", governed.correlation_id) + raise governed from e + +asyncio.run(...) # DONKEY_LLM_PROXY_URL=http://localhost:8080, throwaway creds +``` + + + + + **The raw client raises `openai.APIStatusError`, not a `DonkeyError`.** + `donkey.llm.client()` is the real OpenAI SDK, so you bridge into the taxonomy + with `classify(e.response)` — see [Bridging from the raw + client](/errors#bridging-from-the-raw-client). An adapter that wraps calls in + `typed_refusals()` (as the [support-triage](/scenarios/support-triage) demo + does) surfaces the typed refusal directly instead. + + +## What a full internal copilot also needs — and what's blocked + +A production internal copilot wants more than a content-safety branch. Three of +those pieces are **not yet buildable** because they depend on surfaces still +blocked on verification — this page shows the *shape* without inventing +an endpoint: + + + **Governed access to internal tools** — reaching company systems through + governed MCP tools rather than ad-hoc HTTP — is a [Phase 2 surface](/roadmap). + Exchange→MCP tool discovery is still blocked on verification; the SDK raises + `NotImplementedError("blocked on verification: …")` at call time rather than + guessing an endpoint. See [Tool access](/tool-access). + + + + **Agent identity and a kill switch** — a verifiable identity for the copilot, + and the ability to disable it centrally — are platform capabilities the SDK's + job is to make *reachable and typed*, not to reimplement. Both are + [Phase 2](/roadmap) and gated on verifying the platform's own contract; see + [Identity](/identity). Until then the SDK does not fabricate a stand-in. + + +## Verification status + +The content-safety **discriminator** (the vendor `…-action: reject` header) is +typed by `donkey_kit.core.errors.classify()`, but its exact body is +**documented-but-not-live-captured** — pinned from the policy pages and pending +a live sandbox round-trip ([Verification policy](/concepts/verification)). The +simulator replays the captured fixture so the branch runs today; no verification +row flips to `verified` until a live capture confirms the shape. The correlation +mechanism (`donkey.run()` → `X-Correlation-Id` → `.correlation_id`) is shipped +and framework-agnostic. + +## Where to go next + +- [Typed refusals](/errors) — the `ContentSafetyBlocked` shape and the + documented-but-not-captured caveat, plus the retryable cookbook. +- [Telemetry & cost](/telemetry) — how the run id ties spans, logs, and the + gateway record together. +- [Support triage](/scenarios/support-triage) — the same governance surfacing a + refusal directly through a framework adapter. diff --git a/website/content/scenarios/nightly-batch.mdx b/website/content/scenarios/nightly-batch.mdx new file mode 100644 index 00000000..2ac0d698 --- /dev/null +++ b/website/content/scenarios/nightly-batch.mdx @@ -0,0 +1,128 @@ +--- +description: Enrich tens of thousands of records overnight against a windowed token budget, unattended — pacing before the wall and resuming after the window resets, tested end to end against the simulator. +--- + +import { Callout, Steps } from 'nextra/components' + +# Nightly batch + +50,000 product records, enriched overnight against a governed model, budget +window resetting every hour, no human awake. Without a budget object the script +runs flat out, takes a `429` partway through, crashes, and someone re-runs it +from record zero in the morning — spending the budget twice to do the same work. + +The fix is to make the remaining budget a [first-class object](/budget) and pace +against it: slow down *before* the wall, wait for the window to reset, and carry +on. This page runs that loop end to end against the [local +simulator](/simulator) in about ninety seconds, instead of "we'll find out +tonight." + +## What it demonstrates + +- **`pace()` raises before the request that would cross your reserve**, not + after a `429` comes back — the distinction that is the whole feature. +- **The window actually resets and the job resumes**, driven by the simulator's + `budget` [scenario](/simulator#scenario-scripting), which runs a *real* + wall-clock-windowed token counter and serves the captured `token-rate-limit` + **429** on exhaustion. +- **The batch finishes unattended** — nobody re-runs anything. + +## Run it + + + +### Install the extras + +```bash +pip install "donkey-kit[llm,local]" +``` + +### Boot the simulator with a one-minute budget window + +In one terminal, shrink the hour-long window to a minute so the whole +pace-exhaust-reset cycle plays out in seconds: + +```bash +donkey mock --port 8080 --scenario budget:limit=20000,window=60s +``` + +The happy-path `200` carries the live `x-llm-proxy-ratelimit` prose window +(decreasing as you spend); once the window's 20,000 tokens are gone, calls get +the `token-rate-limit` **429** with `x-token-remaining` / `x-token-reset` +recomputed from the real milliseconds left, until the window rolls over. + +### Point the SDK at it and run the batch + +In a second terminal: + +```bash +export DONKEY_LLM_PROXY_URL=http://localhost:8080 +export DONKEY_LLM_PROXY_CLIENT_ID=local # simulator ignores auth +export DONKEY_LLM_PROXY_CLIENT_SECRET=local +python enrich.py +``` + + + +## The batch loop + +The pacing and resume logic is a handful of lines. `pace()` guards each batch; +on `BudgetReserveReached` you wait for the window and continue from where you +left off: + +```python +import asyncio +from donkey_kit import Donkey, BudgetReserveReached + +async def enrich_all(records, enrich): + async with Donkey.from_env() as donkey: + i = 0 + while i < len(records): + batch = records[i : i + 200] + try: + async with donkey.budget.pace(reserve=0.05): + await enrich(donkey, batch) + except BudgetReserveReached: + # We're within 5% of the window's limit — don't take the 429. + await donkey.budget.wait_for_reset() # sleeps until reset_at + continue # retry the same batch + checkpoint(batch) # only advance on success + i += 200 +``` + + + `pace(reserve=0.05)` raises `BudgetReserveReached` **before** issuing the + request that would cross the last 5% of the window — so you never spend the + request that earns the `429`. `wait_for_reset()` sleeps until + `donkey.budget.reset_at`, computed from the gateway's `x-token-reset` header + (milliseconds, converted for you). See [Budget & pacing](/budget). + + +## The honest limitation + + + **Budget is only visible in-band.** The gateway reports it on response + headers; there is **no endpoint that answers "what is my remaining budget?"**. + So `donkey.budget.remaining` is only as fresh as your last call, and a + brand-new process knows nothing until its first request completes — which is + why `donkey.budget.observed_at` is part of the public surface. A budget-query + endpoint is filed as an [upstream gap](/roadmap) against the gateway. + + +## Verification status + +The budget object, `pace()`, and `wait_for_reset()` are **shipped** (Phase 1). +The windowed-counter behaviour you're pacing against here is the +[simulator's](/simulator) — a faithful replay of the observed live contract +(prose window on the `200`, numeric `x-token-*` trio on the `429`), never a +header shape the gateway does not emit ([Verification +policy](/concepts/verification)). The end-to-end assertion against the simulator +is exactly what this scenario runs. + +## Where to go next + +- [Budget & pacing](/budget) — the full `Budget` object and its two helpers. +- [Local simulator](/simulator) — the `budget` scenario and how the windowed + counter is computed. +- [Internal copilot](/scenarios/internal-copilot) — a content-safety guardrail + and per-run correlation for an internal assistant. diff --git a/website/content/scenarios/support-triage.mdx b/website/content/scenarios/support-triage.mdx new file mode 100644 index 00000000..b7e95e5d --- /dev/null +++ b/website/content/scenarios/support-triage.mdx @@ -0,0 +1,122 @@ +--- +description: Draft replies to a queue of support tickets, one carrying PII, with typed refusals, correlation IDs, and OTel spans — the shipped LangGraph demo, end to end, no gateway. +--- + +import { Callout, Steps } from 'nextra/components' + +# Support triage + +A support agent drafts a one-line reply to each ticket in a queue. Most are +routine; one contains a customer's SSN. On a bare `base_url` that PII-laden call +sails straight through to the model. Through a governed proxy it comes back as a +**typed refusal** — and the branch that masks and re-routes it is a branch you +can watch execute *before* it runs on real data. + +This is the shipped Phase-1 acceptance demo (`BG §1.8`): `pip install` to a +drafted reply against the [local simulator](/simulator) only — no Anypoint +credentials, no real gateway — exercising four governance pieces at once. + +## What it demonstrates + +- **Runs with no gateway.** The demo boots the [local simulator](/simulator) + in-process on an ephemeral port and points the SDK at it; the simulator + ignores auth, so the credentials are throwaway placeholders. +- **The PII-masking branch actually executes.** The simulator runs the + `pii_block:every=5` [scenario](/simulator#scenario-scripting) — every fifth + `POST /responses` is served the captured `pii-detected` **403**. The fifth + ticket (the one with the SSN) is blocked, and the refusal surfaces out of the + LangGraph run as a typed [`PIIDetected`](/errors), not a framework-wrapped + generic error. +- **Correlation reaches every node for free.** Each ticket runs inside a + [`donkey.run(id=...)`](/telemetry) block; the run id shows up in a graph + node's own logs without being threaded through graph state, because LangGraph + runs nodes on context-copying `asyncio` tasks. +- **Every governed call emits an OTel span** — refusals included — exported to a + bundled local OTLP collector with [zero config](/telemetry): setting the + standard `OTEL_EXPORTER_OTLP_ENDPOINT` is all it takes. + +## Run it + + + +### Install the extras + +```bash +pip install "donkey-kit[langgraph,local,otel]" +``` + +`langgraph` brings the framework and the adapter, `local` the simulator, `otel` +the OpenTelemetry export path. + +### Run the demo + +```bash +python -m examples.langgraph.main +``` + +No environment setup: the demo boots its own simulator, sets its own throwaway +credentials, and stands up its own local OTLP collector. You'll see one drafted +reply per routine ticket, a `BLOCKED by policy — PIIDetected` line for the +PII-laden one, the remaining budget, and the count of spans exported. + + + +## The governed core + +The whole demo is ordinary framework code; the SDK touches it in exactly two +places. First, the model is built off a shared `Donkey` so it rides that +instance's governed transport: + +```python +model = donkey.langgraph.chat_model("gpt-4o") +``` + +Second, the node that calls the model wraps the call in `typed_refusals()`, so a +proxy rejection comes back as a `DonkeyError` subclass instead of a +framework-wrapped generic error: + +```python +from donkey_kit.integrations.langgraph import typed_refusals + +async def _call_model(state): + with typed_refusals(): + reply = await model.ainvoke(state["messages"]) + return {"messages": [reply]} +``` + +The caller binds a run id per ticket and catches the typed refusal: + +```python +async with donkey.run(id=f"ticket-{i}"): + try: + result = await agent.run(f"Draft a one-line support reply to: {ticket}") + except PIIDetected as refusal: + print(f" ticket {i}: BLOCKED by policy — PIIDetected ({refusal.policy})") +``` + + + **Honest note on the block.** The simulator triggers on the request *count* + (`every=5`), not by scanning content — it replays a captured fixture, it is + not a PII detector. The real gateway does the detection; here the SSN in + ticket five just makes the blocked ticket read true. See + [It replays; it does not evaluate](/simulator#it-replays-it-does-not-evaluate). + + +## Verification status + +The proxy *contract* the demo depends on — the base URL shape (no `/v1`), the +`client_id`/`client_secret` header pair, the attribution headers, and the four +live-verified rejection shapes including PII — is **live-verified** +([Verification policy](/concepts/verification)). `ChatOpenAI` / `StateGraph` are +the frameworks' own classes and `.ainvoke` is their documented API: +construction via the SDK factory is the verified surface, and everything after +is the framework's own runtime. + +## Where to go next + +- [Nightly batch](/scenarios/nightly-batch) — the same governance, applied to an + unattended overnight job that paces itself against a budget window. +- [Typed refusals](/errors) — the full exception taxonomy the block above lands + in, with a per-exception "retryable?" cookbook. +- [Testing & conformance](/testing) — run this same agent factory through the + conformance suite (`pytest --donkey-conformance`). diff --git a/website/pages/simulator.mdx b/website/content/simulator.mdx similarity index 100% rename from website/pages/simulator.mdx rename to website/content/simulator.mdx diff --git a/website/pages/telemetry.mdx b/website/content/telemetry.mdx similarity index 93% rename from website/pages/telemetry.mdx rename to website/content/telemetry.mdx index 00a73722..3991ceb8 100644 --- a/website/pages/telemetry.mdx +++ b/website/content/telemetry.mdx @@ -223,12 +223,31 @@ function, and concurrent runs never leak into each other. It works with or without OpenTelemetry installed. `donkey.run(...)` is a **dual sync/async** context manager (plain `with` works too); nested blocks rebind then restore. +### The decorator form + +When a whole function should be one run, `@donkey.governed` is the decorator +equivalent of wrapping its body in `donkey.run()`: + +```python +@donkey.governed(team="support") +async def handle_ticket(ticket): + await triage_agent.run(ticket) +``` + +Each call opens its own run — a fresh run/correlation id (a "run of one") — and +binds the optional per-run cost tags, the OTel span, and typed refusals, exactly +the scope `donkey.run()` establishes. It wraps **both sync and async** callables +and is usable bare (`@donkey.governed`) or parametrised. There is deliberately +**no `id=`**: pinning one id across every call would collapse unrelated runs into +a single correlation, so when you need a specific id, reach for `donkey.run(id=…)` +directly. + Whether the gateway **reads** an inbound `X-Correlation-Id` / `X-Donkey-Request-Id` is unverified — `x-correlation-id` is confirmed only as a gateway **response** echo. So those request-header **names** are overridable placeholders (`correlation_header` / `call_id_header`), not guesses - ([Verification policy](/concepts/verification), §0.3). + ([Verification policy](/concepts/verification)). ## Cost-attribution tags diff --git a/website/pages/testing.mdx b/website/content/testing.mdx similarity index 54% rename from website/pages/testing.mdx rename to website/content/testing.mdx index 80d57234..a1709a12 100644 --- a/website/pages/testing.mdx +++ b/website/content/testing.mdx @@ -1,5 +1,5 @@ --- -description: In-process refusal injection plus a pytest plugin you run against your own agent. +description: In-process refusal injection, a pytest plugin you run against your own agent, and an out-of-process gateway fixture. --- import { Callout } from 'nextra/components' @@ -7,15 +7,18 @@ import { Callout } from 'nextra/components' # Testing & conformance - **Phase 1 — both ship.** In-process refusal injection (`donkey.simulate()`) - and the pytest conformance plugin - ([#191](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/191)) are - both shipped. The plugin runs your agent **in process** through its own - `Donkey` — there is no separate server mode. See [Roadmap](/roadmap) and + **Phase 1 — all three ship.** In-process refusal injection + (`donkey.simulate()`), the pytest conformance plugin + ([#191](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/191)), + and an out-of-process `gateway` fixture + ([#278](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/278)) + are all shipped. The conformance plugin runs your agent **in process** through + its own `Donkey`; the `gateway` fixture serves the simulator on a **real port** + for a subject that does not import the SDK. See [Roadmap](/roadmap) and [Verification policy](/concepts/verification). -Two test-time tools, and the second one is the more important of the pair. +Three test-time tools, and the second one is the most important of the set. ## `simulate()` — in-process, no server @@ -77,6 +80,45 @@ It asks questions you probably cannot currently answer about your own code: The output is a table of scenario → pass / fail / **exempt**. +## The `gateway` fixture — an out-of-process server tests assert against + +`simulate()` and the conformance plugin both run your subject **in process**. +Some subjects cannot be: a containerised agent, a Node service, an A2A client, a +manual `curl` — none of which import `donkey_kit`. Those need a real listener on a +real port, and they need the other half of the question `simulate()` cannot +answer: *what did the gateway actually receive?* "Did my agent stop after the +refusal, or retry four more times?" is answerable only from the gateway's side. + +The `gateway` fixture (shipped in `donkey-kit[test]`, beside `donkey`) boots the +[local simulator](/simulator) on an **ephemeral port**, hands your test its +`.url`, and records every request in a spy you can assert on: + +```python +async def test_agent_stops_after_a_refusal(gateway): + gateway.set_scenarios("pii_block:every=1") + app = deploy(env={"DONKEY_LLM_PROXY_URL": gateway.url}) + await app.run(ticket_with_card_number) + # A policy refusal is terminal — the agent must not retry it. + assert gateway.requests_received == 1 +``` + +- `gateway.url` is a real `http://127.0.0.1:` any process can point + `DONKEY_LLM_PROXY_URL` at — it does not import the SDK. +- The port is bound to `0`, so parallel `pytest -n` runs never collide. +- `gateway.requests_received` counts requests; `gateway.requests` exposes each + one's method, path and headers, with `client_secret` **redacted**. +- `gateway.set_scenarios(...)` arms the same + [scenario scripting](/simulator#scenario-scripting) (`pii_block`, `injection`, + `budget`) **per test**, not just per process. +- Every response carries `x-donkey-simulator: true`, so a fixture reply is never + mistaken for a real gateway. +- The server is torn down when the test exits, including on failure. + +It serves the **same captured rejection fixtures** as `simulate()` and the +conformance plugin, so the three cannot drift apart. Because it starts a real +server, it needs the `[local]` extra (`starlette` + `uvicorn`); taking the +fixture without it raises an `ImportError` naming the exact `pip install`. + ## Why this is the moment the SDK earns trust A team installs the SDK and on day one gets a failing test that says *"your @@ -97,14 +139,16 @@ them to take a claim on faith. ## How this differs from the simulator -Both exist, they share fixtures, and they are not the same tool: +They all exist, they share fixtures, and they are not the same tool: -| | [Local simulator](/simulator) | `simulate()` | -|---|---|---| -| Shape | A **server** on a port | **In-process** context manager | -| Use it for | Manual dev, demos, integration tests, any client | Fast unit tests | -| Needs a network | Yes (localhost) | No | -| Works with a stock OpenAI client | Yes | No — it hooks the `Donkey` transport | +| | [`donkey mock`](/simulator) | `gateway` fixture | `simulate()` | +|---|---|---|---| +| Shape | A **server** you run | A **server** a test starts | **In-process** context manager | +| Port | You pick it (`--port`) | Ephemeral (`0`), on `.url` | None | +| Use it for | Manual dev, demos, any client | Testing an out-of-process subject | Fast unit tests | +| Needs a network | Yes (localhost) | Yes (localhost) | No | +| Works with a stock OpenAI client | Yes | Yes | No — it hooks the `Donkey` transport | +| Asserts on what the gateway received | No | Yes (`requests_received`) | No | They read the **same rejection fixtures**, so they cannot drift apart. @@ -122,9 +166,14 @@ They read the **same rejection fixtures**, so they cannot drift apart. your factory (or `--donkey-known-limitations=module:NAME`), validated at collection time so a bad key or empty reason fails the run loudly; a silent skip is never possible. ✅ +- The `gateway` fixture binds an ephemeral port, exposes `.url` to a process that + does not import `donkey_kit`, records requests with `client_secret` redacted, + takes scenarios per test, and tears down on exit. ✅ --- -**Status: Phase 1 — `simulate()` and the conformance plugin -([#191](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/191)) are -both shipped.** +**Status: Phase 1 — `simulate()`, the conformance plugin +([#191](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/191)), +and the `gateway` fixture +([#278](https://github.com/Donkey-Development-Kit/donkey-development-kit/issues/278)) +are all shipped.** diff --git a/website/pages/tool-access/_meta.js b/website/content/tool-access/_meta.js similarity index 100% rename from website/pages/tool-access/_meta.js rename to website/content/tool-access/_meta.js diff --git a/website/pages/tool-access/a2a.mdx b/website/content/tool-access/a2a.mdx similarity index 100% rename from website/pages/tool-access/a2a.mdx rename to website/content/tool-access/a2a.mdx diff --git a/website/pages/tool-access/binding.mdx b/website/content/tool-access/binding.mdx similarity index 100% rename from website/pages/tool-access/binding.mdx rename to website/content/tool-access/binding.mdx diff --git a/website/pages/tool-access/discovery.mdx b/website/content/tool-access/discovery.mdx similarity index 100% rename from website/pages/tool-access/discovery.mdx rename to website/content/tool-access/discovery.mdx diff --git a/website/pages/tool-access/index.mdx b/website/content/tool-access/index.mdx similarity index 100% rename from website/pages/tool-access/index.mdx rename to website/content/tool-access/index.mdx diff --git a/website/pages/tool-access/lockfile.mdx b/website/content/tool-access/lockfile.mdx similarity index 100% rename from website/pages/tool-access/lockfile.mdx rename to website/content/tool-access/lockfile.mdx diff --git a/website/pages/use-with-your-agent.mdx b/website/content/use-with-your-agent.mdx similarity index 100% rename from website/pages/use-with-your-agent.mdx rename to website/content/use-with-your-agent.mdx diff --git a/website/mdx-components.tsx b/website/mdx-components.tsx new file mode 100644 index 00000000..5bd01020 --- /dev/null +++ b/website/mdx-components.tsx @@ -0,0 +1,16 @@ +import type { MDXComponents } from 'nextra/mdx-components' +import { useMDXComponents as getDocsMDXComponents } from 'nextra-theme-docs' + +// Nextra 4 requires this root file (Next.js's MDX convention picks it up). It +// merges the docs-theme MDX components with any per-page overrides. The site's +// own presentation components (Hero/Card/CardGrid/Figure/Badge) are still +// imported explicitly at the top of each .mdx from ../components, exactly as +// under Nextra 3, so they are not registered globally here. +const docsComponents = getDocsMDXComponents() + +export function useMDXComponents(components?: MDXComponents) { + return { + ...docsComponents, + ...components, + } +} diff --git a/website/next.config.mjs b/website/next.config.mjs index 68bdc592..f709fa5e 100644 --- a/website/next.config.mjs +++ b/website/next.config.mjs @@ -1,8 +1,9 @@ import nextra from 'nextra' +// Nextra 4 is App-Router only: `theme` / `themeConfig` are gone — the docs theme +// is composed from //