Skip to content

Port the TRAPI resilience probe, plan, and intake tests to trapi_1.6 - #187

Merged
maximusunc merged 1 commit into
trapi_1.6from
red-team-resilience-probe-trapi16
Oct 27, 2026
Merged

maximusunc merged 1 commit into
trapi_1.6from
red-team-resilience-probe-trapi16

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

What this is

Ports the resilience tooling (merged to devel in #186) onto trapi_1.6, adapted for 1.6's intake contract. Same Test-only probe, plan, and intake regression tests — tuned so the expectations and tests are correct for 1.6 rather than 2.0.

The key difference: 1.6 has no TOM-2.0 validate_query

On 1.6 the only intake rejection is at parse time (parse_query_body → 422 for non-JSON / non-object / wrong content-type). Bodies that 2.0 rejects with a 400 — retired 1.x spellings (which are simply valid on 1.6), a missing/empty query graph, type confusion, an extra envelope, a natural-language body — are accepted at intake here and left to the pipeline. Everything else is identical.

What changed vs the devel version

  • scripts/resilience_probe.py — adds a --trapi-version {2.0,1.6} flag (default 2.0). On --trapi-version 1.6 the ~7 validation-dependent probes (retired-1x, empty-graph, no-message, type-confusion-*, nl-question, extra-envelope, mixed-trapi-version) switch to a permissive "accepted at intake — must not 500 or hang" expectation, with a version note in their signal. All other categories (SSRF, enum, oversized/slowloris, topology, overbroad, cache, constraints, identifier, encoding, surface, workflow) are version-agnostic and unchanged. One probe is the same on both branches — the flag keeps them unifiable.
  • tests/unit/test_resilience_intake.py — rewritten to 1.6's contract: the parse-level 422 tests stay; the 2.0 "schema-invalid → 400" and "mixed-version rejects the retired field" tests become "schema-shaped body is accepted at intake." Verified against 1.6's intake under a TestClient (run_query stubbed); 27 pass.
  • docs/RESILIENCE_TESTING_PLAN.md — a TRAPI-version note up top (run with --trapi-version 1.6) and inline caveats on the validation-dependent items.

Findings

The findings themselves are version-agnostic and the probe shows them on 1.6 too (same callback SSRF, same unbounded request.body(), same workflow TypeError, same get_message KeyError). They're already tracked against devel in #180–#184; this PR is the tooling only, so those issues cover 1.6 as well. Run --trapi-version 1.6 against a 1.6 Test instance to confirm.

Notes

  • Tooling + tests only — no server behavior changes.
  • Usage: python scripts/resilience_probe.py --base-url <test> --i-understand-this-is-test --trapi-version 1.6 [...]

Generated by Claude Code

Brings the resilience tooling from devel onto trapi_1.6, adapted for 1.6's
intake contract. 1.6 has no TOM-2.0 validate_query layer, so the only intake
rejection is at parse time (non-JSON / non-object / wrong content-type -> 422);
bodies 2.0 rejects with a 400 (retired 1.x spellings, missing/empty query
graph, type confusion, extra envelope, natural-language body) are accepted at
intake on 1.6 and left to the pipeline.

- resilience_probe.py: add a --trapi-version {2.0,1.6} flag (default 2.0). On
  1.6 the ~7 validation-dependent probes switch to a permissive "accepted at
  intake -- must not 500 or hang" expectation, with a version note in their
  signal. All other categories (SSRF, enum, oversized/slowloris, topology,
  overbroad, cache, constraints, identifier, encoding, surface, workflow) are
  version-agnostic and unchanged.
- test_resilience_intake.py: rewritten to 1.6's contract -- parse-level 422s
  kept; the 2.0 "schema-invalid -> 400" and "mixed-version rejects retired
  field" tests become "schema-shaped body accepted at intake". Verified against
  1.6 intake under a TestClient (run_query stubbed); 27 pass.
- RESILIENCE_TESTING_PLAN.md: a TRAPI-version note up top and inline caveats on
  the validation-dependent items.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ne6qXVt5WT6yq9sTmyEnA
@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (trapi_1.6@d17055e). Learn more about missing BASE report.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d17055e...c824521. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit aae48ad into trapi_1.6 Oct 27, 2026
2 checks passed
@maximusunc
maximusunc deleted the red-team-resilience-probe-trapi16 branch October 27, 2026 13:34
maximusunc pushed a commit that referenced this pull request Oct 27, 2026
- The resilience probe, plan and intake tests (#186) keep trapi_1.6's port
  from #187, made from #186's final state and adapted to 1.6's intake.
- #179 points the ARAX pathfinder's BiolinkHelper at the TRAPI 2.0 models'
  Biolink version (shepherd_utils.trapi, not on this branch); it keeps
  arax_biolink_version (v4.2.5) here, matching the in-process ARAX's 4.2.5.
- The compose changes (ARS behind the `ars` profile, compose.test.yml's
  limits folded into compose.yml) come in as they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtsBikU9y9AFQ5uu9tcioz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants