Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ node_modules/
/CLAUDE.md
/.claude/skills
/.claude/settings.json
/.claude/workflows
/.claude/commands

# Playwright MCP scratch output (local agent tooling)
.playwright-mcp/
Expand Down
60 changes: 39 additions & 21 deletions docs/verified-apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,14 @@
> (§2) and the token-rate-limit (`429`, empty body, header-only budget) + PII
> (`403`, `type:pii_detected`) rejection contracts (§4) are now also
> `VERIFIED (LIVE)` — both policies were applied to `openai-sdk` for capture and
> removed. Still `UNVERIFIED`: prompt-injection / regex-prompt-guard /
> content-safety rejection bodies (§4, typed from the policy pages, pending live
> capture #253) and the **framework constructor/binding names** (§§8–10). §11
> removed. The regex-prompt-guard (`403` +
> `matched_patterns`) and Azure content-safety (`403` + `…-action: reject`)
> rejection shapes were also `VERIFIED (LIVE)` on 2026-09-22 against the deployed
> `ddk-injection-guard` / `ddk-azure-content-safety` proxies (#253). Still
> `UNVERIFIED`: the Injection Protection body (`x-injection-protection: blocked`,
> a distinct policy — no proxy deployed) and any other content-moderation /
> Bedrock-guardrail shape (§4, pending live capture #253), and the **framework
> constructor/binding names** (§§8–10). §11
> records A2D shapes
> (`VERIFIED-SHAPE-ONLY`), used only to validate SDK value types; no blocked
> code path was wired to them.
Expand Down Expand Up @@ -208,9 +213,14 @@ point the SDK at it via the `cost_*_header` overrides.

## 4. Policy rejection response shapes (capture as fixtures, BG §1.5)

**Four rejection shapes** are LIVE-VERIFIED (2026-08-28) from the `openai-sdk`
proxy (client-id-enforcement, upstream passthrough, PII, token-rate-limit).
Fixtures in `tests/fixtures/anypoint/llm_proxy/reject.*`. The critical lesson:
**Six rejection shapes** are now LIVE-VERIFIED: four from the `openai-sdk` proxy
(client-id-enforcement, upstream passthrough, PII, token-rate-limit; 2026-08-28),
plus **Regex Prompt Guard** and **Azure Content Safety** — both `403` policy
blocks confirmed 2026-09-22 against the deployed provisioning proxies
`ddk-injection-guard` (instance 21179713) and `ddk-azure-content-safety`
(instance 21180957), rows added to the policy table below (#253).
Fixtures in `tests/fixtures/anypoint/llm_proxy/reject.*` and
`tests/fixtures/rejections/reject.*`. The critical lesson:
**neither the status code nor the mere shape of the `error` value is a
sufficient discriminator** — the same nested-object envelope is emitted by both
the upstream provider AND a gateway policy (PII), and a policy block can be a
Expand Down Expand Up @@ -253,26 +263,32 @@ whose message names the observed status and any `x-llm-proxy-*` policy headers a
states the **shape is unconfirmed** (#184), rather than being mis-typed. The full
eight-shape taxonomy is indexed in `tests/fixtures/rejections/README.md`.

The Regex Prompt Guard and content-safety/guardrail shapes below are
**DOCUMENTED, pending live capture (#253)** — typed from the policy pages, not
from a live sandbox round-trip, so **no `_verify.py` row flips to
`verified=True`** for them:
The Regex Prompt Guard and content-safety/guardrail shapes below are now
**VERIFIED (LIVE), 2026-09-22 (#253)** — confirmed against the deployed
provisioning proxies (`ddk-injection-guard`, `ddk-azure-content-safety`), not
only typed from the policy pages. They have **no `_verify.py` constant** —
`classify()` reads them straight from the response — so the verification record
is these rows, not an `Unverified(...)` flip:

| Shape | HTTP | Discriminator | Maps to | Source |
|---|---|---|---|---|
| Regex Prompt Guard | `403` | top-level `matched_patterns` list (flat-string `error`) | `PromptInjectionBlocked` (`policy="regex-prompt-guard"`) | [Regex Prompt Guard policy](https://docs.mulesoft.com/gateway/latest/policies-included-regex-prompt-guard); DOCUMENTED, pending live capture (#253) |
| Content safety / guardrails | `403` | `x-llm-proxy-azure-content-safety-action` / `x-llm-proxy-bedrock-guardrail-action` == `reject`; reasons in the sibling `…-reason` header | `ContentSafetyBlocked` (parses `categories`) | [Azure Content Safety policy](https://docs.mulesoft.com/gateway/latest/policies-included-azure-content-safety); [Amazon Bedrock Guardrails policy](https://docs.mulesoft.com/gateway/latest/policies-included-bedrock-guardrails); DOCUMENTED, pending live capture (#253) |
| Regex Prompt Guard | `403` | top-level `matched_patterns` list (flat-string `error`) | `PromptInjectionBlocked` (`policy="regex-prompt-guard"`) | [Regex Prompt Guard policy](https://docs.mulesoft.com/gateway/latest/policies-included-regex-prompt-guard) (v1.11.4); VERIFIED (LIVE) 2026-09-22 on `ddk-injection-guard` (instance 21179713) |
| Content safety / guardrails | `403` | `x-llm-proxy-azure-content-safety-action` / `x-llm-proxy-bedrock-guardrail-action` == `reject`; reasons in the sibling `…-reason` header | `ContentSafetyBlocked` (parses `categories`) | [Azure Content Safety policy](https://docs.mulesoft.com/gateway/latest/policies-included-azure-content-safety) (v1.13.0); [Amazon Bedrock Guardrails policy](https://docs.mulesoft.com/gateway/latest/policies-included-bedrock-guardrails) (v1.13.0); VERIFIED (LIVE) 2026-09-22 on `ddk-azure-content-safety` (instance 21180957, Azure verified; Bedrock still documented-only) |
| Unrecognised refusal (fall-through) | any non-429 `4xx` | matches **none** of the discriminators above; no nested `error` envelope; no `www-authenticate` (e.g. an unrecognised `403`) | generic `PolicyViolation` (`policy="unknown"`), message says **shape unconfirmed** and names the observed status + `x-llm-proxy-*` headers | UNVERIFIED — no known contract; capture + type via #184/#253 |

The **injection body** and any **other content-moderation / federated-guardrail**
shape remain uncaptured. Rather than guess a type for them, `classify()` surfaces
any such unrecognised refusal **honestly** — a `PolicyViolation` whose message
states the shape is unconfirmed and whose remediation asks the operator to file
the observed status/headers/body so the shape can be typed (#184). This is the
last row above; it is deliberately **not** an `AuthError`, even for a `403`, once
the verified `www-authenticate` auth shape is excluded. The official policy
pages above establish the documented contracts; direct sandbox capture remains
tracked in #253 and is still required before any verification status changes.
The Regex Prompt Guard (row 7) and Azure Content Safety (row 8) shapes are now
LIVE-captured (2026-09-22, #253). Still **uncaptured**: the **Injection
Protection** body (`x-injection-protection: blocked`, a policy *distinct* from
Regex Prompt Guard — no proxy running it is deployed) and any **other
content-moderation / federated-guardrail** shape (e.g. Bedrock Guardrails).
Rather than guess a type for them, `classify()` surfaces any such unrecognised
refusal **honestly** — a `PolicyViolation` whose message states the shape is
unconfirmed and whose remediation asks the operator to file the observed
status/headers/body so the shape can be typed (#184). This is the last row
above; it is deliberately **not** an `AuthError`, even for a `403`, once the
verified `www-authenticate` auth shape is excluded. Closing these last shapes
needs an Injection Protection (and a Bedrock Guardrails) proxy deployed to
capture against; that residual keeps #253 open.

**Budget window emission — two forms, keyed on outcome not status class
(LIVE-VERIFIED).** The gateway signals its token-rate-limit window in two
Expand Down Expand Up @@ -326,6 +342,8 @@ the prose parser from #352) now populates identically from a simulated or a live
| LLM proxy core | `llm-proxy-core` `1.0.5` | applied VERIFIED (LIVE) | on `openai-sdk`; rejection body not yet triggered | 2026-08-28 | `policy:list` |
| Token rate limiting | interface `llm-token-rate-limit` `1.0.2` (impl `-policy-flex` `1.0.4`) | VERIFIED (LIVE) | Two emission forms: `429` limit-exceeded → **empty body**, numeric trio `x-token-limit`/`x-token-remaining`/`x-token-reset`(ms), no `retry-after`; `200`/`403` under the same policy → the window as prose in a single `x-llm-proxy-ratelimit` header (ms reset). See "Budget window emission" above. | 2026-08-28 | applied + live probe |
| PII detection | interface `llm-pii-detection-policy` `1.0.0` (impl `-flex` `1.0.2`) | VERIFIED (LIVE) | `403`, nested `{"error":{message,type:"pii_detected"}}`, no `www-authenticate` | 2026-08-28 | applied + live probe |
| Regex Prompt Guard | `regex-prompt-guard-policy` `1.0.0` | VERIFIED (LIVE) | `403`, flat-string `error` + top-level `matched_patterns` list; body matched the committed fixture byte-for-byte | 2026-09-22 | live probe, `ddk-injection-guard` instance 21179713 |
| Azure Content Safety | `azure-content-safety-policy` `1.0.0` | VERIFIED (LIVE) | `403`, `x-llm-proxy-azure-content-safety-action: reject` + `-phase` + `-reason` headers; body `{"error":…,"categories":[…]}` (categories prompt-dependent, e.g. `severity_hate,severity_violence`, `prompt_shield`) | 2026-09-22 | live probe, `ddk-azure-content-safety` instance 21180957 (private-space gateway) |

**Apply note (verified):** these LLM policies apply against the schema-bearing
**interface** asset id/version (`llm-token-rate-limit` `1.0.2`,
Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "donkey-kit"
version = "0.1.0.dev5"
version = "0.1.0.dev6"
description = "An SDK for consuming Agent Fabric capabilities (governed model and tool access) from your own agent framework."
readme = "README.md"
requires-python = ">=3.10"
Expand Down
2 changes: 1 addition & 1 deletion python/src/donkey_kit/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
PublicationAssetType,
)

__version__ = "0.1.0.dev5"
__version__ = "0.1.0.dev6"

__all__ = [
"STRICT",
Expand Down
31 changes: 19 additions & 12 deletions python/src/donkey_kit/core/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -473,11 +473,16 @@ def classify(
(flagged reasons parsed from the sibling ``...-reason`` header). Also
checked before the auth rule.

These two shapes are **documented, not yet live-captured** (same posture as
#181's header-based injection typing; docs/verified-apis.md §4). Exact strings are pinned from
the policy pages; no ``_verify.py`` row flips to ``verified=True`` until a
live sandbox round-trip confirms them (#253). Any other content-moderation /
federated-guardrail shape still falls through to a generic
Both shapes are now **live-verified** (2026-09-22, #253): Regex Prompt Guard
against ``ddk-injection-guard`` and Azure Content Safety against
``ddk-azure-content-safety`` (docs/verified-apis.md §4). These shapes have no
``_verify.py`` constant — ``classify()`` reads them straight from the
response — so the record is the §4 rows, not a ``verified=True`` flip. Amazon
Bedrock Guardrails (same ``...-action: reject`` header family) and the
Injection Protection body (``x-injection-protection: blocked``, a distinct
policy) stay documented-only — no proxy running them is deployed to capture
(#253). Any other content-moderation / federated-guardrail shape still falls
through to a generic
:class:`PolicyViolation` whose message names what was observed and says the
shape is unconfirmed (#184). Because auth is the verified 401 / ``www-authenticate``
shape (docs/verified-apis.md §4), a **403 carrying no ``www-authenticate`` header** and matching
Expand Down Expand Up @@ -522,9 +527,10 @@ def classify(

# Content-safety / guardrails policy: 403 + a vendor `...-action: reject`
# header (Azure Content Safety / Amazon Bedrock Guardrails, docs/verified-apis.md §4).
# Documented, pending live capture (#253). Checked before the 401/403 → auth
# rule because a moderation block is not an auth failure. Keyed on the
# header, not the body, so the body-less Bedrock reject is caught too.
# Azure verified LIVE 2026-09-22 (#253); Bedrock still documented-only.
# Checked before the 401/403 → auth rule because a moderation block is not an
# auth failure. Keyed on the header, not the body, so the body-less Bedrock
# reject is caught too.
cs = _content_safety_reject(response)
if cs is not None:
vendor, categories = cs
Expand All @@ -542,7 +548,8 @@ def classify(

# Regex Prompt Guard policy: 403 + a top-level `matched_patterns` list
# (flat-string `error`, so NOT the nested upstream envelope; docs/verified-apis.md §4).
# Documented, pending live capture (#253). Checked before the 401/403 → auth
# Verified LIVE 2026-09-22 against ddk-injection-guard (#253); body matched
# the committed fixture byte-for-byte. Checked before the 401/403 → auth
# rule so a deny-list block is not mis-typed as an auth failure.
matched = body.get("matched_patterns") if body is not None else None
if isinstance(matched, list):
Expand Down Expand Up @@ -730,9 +737,9 @@ def _json_body(response: httpx.Response) -> dict[str, Any] | None:

# Content-safety / guardrails policies report their verdict in a pair of vendor
# headers — an ``...-action`` (``allow``|``reject``) and a comma-separated
# ``...-reason``. Both are ``x-llm-proxy-<vendor>-...`` (docs/verified-apis.md §4). Pinned from the
# Azure Content Safety and Amazon Bedrock Guardrails policy pages; documented,
# pending live capture (#253).
# ``...-reason``. Both are ``x-llm-proxy-<vendor>-...`` (docs/verified-apis.md §4).
# Azure Content Safety verified LIVE 2026-09-22 against ddk-azure-content-safety
# (#253); Amazon Bedrock Guardrails still documented-only (no proxy deployed).
_CONTENT_SAFETY_VENDORS: tuple[tuple[str, str, str], ...] = (
(
"x-llm-proxy-azure-content-safety-action",
Expand Down
4 changes: 2 additions & 2 deletions python/tests/fixtures/fixtures.lock
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
"anypoint/llm_proxy/responses.success.body.json": "sha256:c47727c6ffb0d2dec80ed37d182825d5707de043517a890999fefa4e407dc90e",
"anypoint/llm_proxy/responses.success.headers.txt": "sha256:263ce673cad339d8cc06e095924fb5ff1310f84d47f591f18eb1a8c90fb8027b",
"rejections/reject.content-moderation.headers.txt": "sha256:5df6a849d1ea245db54617e66f1d380566270f23102ca0cb8046a1c8b990cbda",
"rejections/reject.content-safety.body.json": "sha256:4c9c71529bd564d3c72e4e6dc383fa527088290c9016b3afb2f8706e5e32fbf1",
"rejections/reject.content-safety.headers.txt": "sha256:9bfd3b7d8e34b08a1df5771ce4e545dc000473ab9f240da623892e049bc587fa",
"rejections/reject.content-safety.body.json": "sha256:678a01904e36159a5089f4e0784fc3a61b9fcdeefda04e6a16c07b8e81fea267",
"rejections/reject.content-safety.headers.txt": "sha256:d991dbc094133c574968b5f735d000398fb556bbae5c65139fba11ce7e3bfdec",
"rejections/reject.injection-protection.headers.txt": "sha256:7964c425ed91c8498b6f25474d6cd91c0350ec3803496f75c40fb71004930ea2",
"rejections/reject.regex-prompt-guard.body.json": "sha256:9bf2ca369b9250edbac362f3d74904d76d8c6e338c1c9ef53fe33b9848db7f23",
"rejections/reject.regex-prompt-guard.headers.txt": "sha256:01c9848a2470e700602713dc0530656c4ff6be647b69441308b24e78ab3e11ae",
Expand Down
Loading
Loading