[maximusunc]

Why

Every worker failure looked the same from outside. handle_task_failure always sent status: "ERROR" to finish_query, and /query answered that with a 500. So when BTE refused a Pathfinder query, callers saw a crash, even though BTE was declining a query it doesn't support. TRAPI's code for that is 501 Not Implemented.

Through the ARS it was worse: a failed ARA query's child was recorded as a clean D/200 with 0 results, so the failure didn't show up at all.

Found while load-testing the mixed ARA profile with HelmsDeep: BTE's Pathfinder refusals were counted as errors and failed every checkpoint.

What

Only the ARA's own worker knows whether it broke or just doesn't do this kind of query, so the worker now decides the status code. The server gets no ARA-specific logic.

• shepherd_utils/query_errors.py (new): QueryError(description, status_code) and UnsupportedQueryError (501). The code is stored in the existing status text as "ERROR 501", so there is no schema change. A bare "ERROR" still means 500, so queries that fail the old way read exactly as before. Codes outside 400–599 are ignored, so a worker can't make a failed query answer 2xx/3xx.
• Lifecycle: run_task_lifecycle passes the exception to handle_task_failure. For a QueryError, that sends finish_query the coded status plus its description, which is shown to callers. Any other exception still sends "ERROR", and its text isn't forwarded because it may not be safe to show callers. A QueryError is logged without a traceback, since it isn't a crash.
• finish_query: stores the description in shepherd_brain.description. That column already existed but nothing wrote to it. Both the status and the description are passed on to the ARS intake.
• /query and /arax/query: answer with the worker's code. The description becomes the response's description unless the ARA already set one.
• /asyncquery_status: shows the description (it already read that column).
• ars_premerge intake: a failed ARA query's child becomes E/<code> (or E/500 when no code was named), which is how upstream records an ARA answering non-200. TIMEOUT queries are unchanged and still deliver their partial results as D/200. I added this to docs/ARS_PARITY_REGISTER.md.
• workers/bte and workers/bte_lookup: raise UnsupportedQueryError("BTE does not support Pathfinder type queries."), with the same message as before.
• Version 2.2.0: bumped in pyproject.toml and shepherd_server/openapi-config.yaml, following the same two-file pattern as the 2.1.0 bump. devel (including Monitor: show image version per worker; exclude running tasks from backlog #199) is merged into this branch. Image tags and SHEPHERD_VERSION come from the release tag via release.yml, so nothing else needs bumping.

Behavior changes to review

• Plain ARA failures in the ARS are now E/500 instead of D/200. One side effect: if a workflow fails partway through, whatever results it gathered before the failure are no longer merged.
• A lost response now stores its "Response lost: …" description alongside its ERROR status. I updated test_finish_async_query_with_a_lost_response to match.

Releasing

After merging, publish a GitHub release tagged v2.2.0. release.yml builds and pushes the images from that.

Not in this PR

If a client gives /asyncquery its own callback URL, the body we POST back still doesn't say the query failed. That was already true for every failure, not just this one. Fixing it means changing how finish_query builds that body without decoding it, so it's left for a follow-up.

Testing

• New tests/unit/test_query_errors.py: status encoding, handle_task_failure with and without a QueryError, and run_task_lifecycle passing the exception through.
• New cases in test_query_status.py ("ERROR 501" → 501 with the worker's description on /query and /asyncquery_status; invalid codes fall back to 500; an ARA's own description wins), test_finish_query.py (status and description reach Postgres and the ARS intake task), and ars/test_ars_premerge.py (E/501, E/500, TIMEOUT still D/200).
• test_bte.py and test_bte_lookup.py assert UnsupportedQueryError with status_code == 501.
• Full unit suite locally, after merging devel and bumping the version: 2019 passed, 0 failed. 86 ARAX parity tests could not run because the reference data they need isn't available in this environment. They fail before reaching anything this PR touches.
• black --check passes on the whole tree.

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.13483% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.23%. Comparing base (d1c6221) to head (4e059dc).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines	Patch %	Lines
shepherd_server/base_routes.py	83.33%	1 Missing and 1 partial ⚠️
shepherd_utils/db.py	33.33%	1 Missing and 1 partial ⚠️
shepherd_utils/query_errors.py	94.28%	1 Missing and 1 partial ⚠️
workers/ars_premerge/worker.py	92.85%	0 Missing and 1 partial ⚠️

Files with missing lines	Coverage Δ
shepherd_server/aras/arax.py	76.56% <100.00%> (ø)
shepherd_utils/shared.py	81.85% <100.00%> (+0.34%)	⬆️
workers/bte/worker.py	73.17% <100.00%> (+0.67%)	⬆️
workers/bte_lookup/worker.py	87.01% <100.00%> (+0.05%)	⬆️
workers/finish_query/worker.py	91.86% <100.00%> (+0.26%)	⬆️
workers/ars_premerge/worker.py	76.03% <92.85%> (+0.92%)	⬆️
shepherd_server/base_routes.py	87.25% <83.33%> (-0.25%)	⬇️
shepherd_utils/db.py	72.92% <33.33%> (-0.22%)	⬇️
shepherd_utils/query_errors.py	94.28% <94.28%> (ø)

... and 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca0ed67...4e059dc. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.