Repository navigation
Add a TRAPI resilience/red-team probe, test plan, and intake regression tests - #186
Merged
Merged
Conversation
Adds scripts/resilience_probe.py, a Test-only diagnostic that throws the kinds of malformed, abusive, resource-exhausting, and semantically explosive TRAPI requests a bad actor sends in the wild at Shepherd's own front door, so we can find and close gaps in Test before they reach Prod. Each probe declares the guard it expects the code to have, sends the request, and reports whether the guard held plus the out-of-band signal to watch (event-loop lag, pool exhaustion, response size, recovery). Categories: malformed/schema-abuse, oversized/structural, resource-exhaustion, overly-broad/high-cardinality, topological (n-hop/cycle/clique/star/ pathfinder), cache abuse, constraint stacking, hostile identifiers, enumeration/IDOR/forgery, SSRF via the async callback URL, and surface hygiene. Heavy categories that run a full pipeline are gated behind --enable-heavy and bounded by flags. Safety: refuses to run without an explicit acknowledgement flag, aborts on any Prod-looking host, and only ever points callback probes at a local sink or documentation-range IPs. docs/RESILIENCE_TESTING_PLAN.md is the companion plan: the query-type catalog, the per-case hypotheses, and the candidate findings to confirm against Test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
Adds two categories aimed at non-malicious-but-weird traffic, which is what the weekend incident most resembles: - agent: the "completely weird stuff" an LLM agent or buggy script emits -- natural-language bodies, markdown-fenced JSON, trailing prose, Python-repr / JSON5, extra envelopes, wrong nesting, garbage metadata fields, hallucinated curies/prefixes/predicates, status-poll storms, and orphaned async jobs with a callback host that never resolves. The bar is two-fold: never 500, and answer with a clear enough error that the agent self-corrects instead of retrying in a loop. - injection: prompt-injection-shaped text in query fields, to confirm it stays inert data (logged as a value, echoed escaped) since our logs and responses get fed to other LLMs downstream. Also adds per-batch latency min/med/max to the repeated-probe summary so the cache-abuse comparison and the poll-storm cost are visible. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
- encoding category: BOM-prefixed JSON, UTF-16 body (orjson is UTF-8 only), and double-encoded JSON (a JSON string containing the JSON). All must decode or reject cleanly, never crash the parser. - agent.mixed-trapi-version: a Frankenstein query mixing 2.0 constraints with retired 1.x qualifier_constraints, to confirm the retired-field check catches the 1.x half. - agent.curie-category-mismatch: schema-valid but contradictory (a CHEBI chemical typed as a Gene), to confirm a worker doesn't choke on the nonsense. - agent.abandon-and-poll: the full realistic agent failure -- POST asyncquery with a callback never served, then hot-poll the status endpoint for a job that will never be delivered. New runner flow drives the POST then the polls. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
Pins the deterministic, intake-only guards the resilience probe checks, so the behavior stays a tested contract rather than a one-time finding. Covers the cheap categories that resolve before any worker runs (no Redis/Postgres): - unparseable bodies (markdown-fenced JSON, trailing prose, Python-repr, trailing-comma JSON5, raw prose, UTF-16, UTF-8 BOM) -> 422 "not valid JSON" - double-encoded JSON (a JSON string containing the JSON) -> 422 non-object - schema-invalid bodies (natural-language question, missing message, extra envelope, empty query graph) -> 400 - a mixed 1.x/2.0 query -> 400 naming the retired field - schema-valid-but-weird bodies (hallucinated curies/prefixes/predicates, a curie/category mismatch) are accepted at intake, not crashed - an injection-shaped curie is passed through as inert data, unchanged - unknown agent-added fields currently pass through (pinned so a future decision to reject them is deliberate, not silent) Follows test_query_body_parsing's pattern: run_query stubbed, rejected bodies asserted to never register a query, accepted bodies checked via /asyncquery so the call returns at intake instead of polling Postgres. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
httpx/h11 validate Content-Length against the body on the client side and refuse to send a short body with a huge declared length, so the probe failed in the client and never reached the server (a false FAIL). Send it over a raw socket instead so it actually tests whether the server trusts the header and blocks waiting for bytes that never arrive (slowloris-style) versus bounding the read. A clean 4xx is the safe outcome; a socket timeout is surfaced as the finding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
compose.test.yml was folded into compose.yml (its resource limits are now the default), so the bring-up is a plain `docker compose up` rather than a two-file overlay. Update docs/RESILIENCE_TESTING_PLAN.md §1 and the observability note, and the one probe signal string that referenced the old overlay, to match -- including the new >=6 CPU requirement and the compose.override.yml escape hatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
surface.unknown-ara was posting an empty body to the real default ARA, so it
got a 422 (bad body) instead of testing ARA routing. Add an abs_path field on
Probe that bypasses the /{ara} prefix, and point this probe at
/__no_such_ara__/query with a valid body, so the result reflects a 404 for an
unmounted ARA rather than a body-parse error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
Codecov Report✅ All modified and coverable lines are covered by tests. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Reformat the probe and the intake regression tests to satisfy the Black check-format CI job (psf/black@stable, target py312). Formatting only; no behavior change — the 30 intake tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
maximusunc
pushed a commit
that referenced
this pull request
Oct 27, 2026
- The resilience probe, plan and intake tests (#186) keep trapi_1.6's port from #187, made from #186's final state and adapted to 1.6's intake. - #179 points the ARAX pathfinder's BiolinkHelper at the TRAPI 2.0 models' Biolink version (shepherd_utils.trapi, not on this branch); it keeps arax_biolink_version (v4.2.5) here, matching the in-process ARAX's 4.2.5. - The compose changes (ARS behind the `ars` profile, compose.test.yml's limits folded into compose.yml) come in as they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TtsBikU9y9AFQ5uu9tcioz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A Test-only diagnostic for Shepherd's TRAPI front door and the findings from running it. It throws the kinds of malformed, abusive, resource-exhausting, and non-conforming (LLM-agent-generated) requests we see in the wild at our own endpoints, so we can find and close gaps in Test before they reach Prod — prompted by the bad-actor traffic that hit the Translator services over the weekend.
Contents
scripts/resilience_probe.py— the probe. 14 categories / ~60 probes: malformed/schema-abuse, oversized/structural, resource-exhaustion, overly-broad, topological (n-hop/cycle/clique/star/pathfinder), cache, constraint-stacking, hostile identifiers, LLM-agent traffic, prompt-injection-shaped data, encoding games, enumeration/IDOR/forgery, SSRF, and surface hygiene. Each probe declares the guard it expects, sends the request, and reports whether the guard held plus the out-of-band signal to watch (event-loop lag, pool exhaustion, response size, recovery). Heavy categories that run a full pipeline are gated behind--enable-heavyand bounded by flags.--i-understand-this-is-test, aborts on any Prod-looking host, and only ever points callback probes at a local sink or documentation-range IPs.docs/RESILIENCE_TESTING_PLAN.md— the query-type catalog, per-case hypotheses, rules of engagement, and sequencing.tests/unit/test_resilience_intake.py— regression tests pinning the deterministic intake-level guards (clean 400/422 for unparseable / schema-invalid / mixed-version bodies; schema-valid-but-weird accepted as inert; injection-shaped curie passed through unchanged). 30 tests, no Redis/Postgres needed — they followtest_query_body_parsing's stub-run_querypattern.Findings (from running it against local Test)
Each has repro + root cause + suggested fix in its issue:
/asyncqueryaccepts arbitrarycallbackURLs (cloud-metadata IP, internal Redis,file://,gopher://) with no scheme/host validation;finish_querylater POSTs to them./query//asyncquery(10 MB accepted, parsed on the event loop); a lyingContent-Lengthhangs a worker (slowloris); 50k-ids query costs several seconds.workflowraises an unhandled exception (connection drop / 500) instead of a clean 400.GET /response/{id}returns 500 instead of 404 for an unknown id (no data leak).Triage first: #180 (SSRF).
What held up well
Every intake-validation case returned clean 400/422 or treated input as inert; enumeration of status/response ids and forged callbacks leak nothing; poll-storm and abandon-poll stay cheap; routing and methods are correct. Those guards are now locked in by the regression tests.
Notes
develaftercompose.test.ymlwas folded intocompose.yml; the plan and probe signals were updated to match (plaindocker compose up).Closes nothing automatically; see #180, #181, #182, #183, #184.
Generated by Claude Code