Fail over ColPali query embeddings to the next endpoint

[Adityav369]

Problem

With multiple ColPali API endpoints configured (morphik_embedding_api_domain), the query paths — embed_for_query (text) and generate_embeddings (image queries):

• picked next(iter(self.healthy_endpoints)) — an arbitrary element of a set, not config order; and
• never marked an endpoint unhealthy on failure, and never tried another endpoint.

So if one embedding server became unreachable, every use_colpali=true query could fail with EmbeddingUnavailableError, even though a healthy endpoint was configured. Ingestion was unaffected because the distributed ingestion path already fails over.

Fix

• Text and image queries share one _embed_query_input path: try endpoints in config order (healthy first, unhealthy as a last resort), mark an endpoint unhealthy when it is unavailable, and fall through to the next. An endpoint that succeeds is marked healthy again. The existing 60s cooldown/re-probe logic is reused.
• Queries use a 5s connect timeout so an unreachable host is detected quickly instead of waiting out the 30s connect timeout (×2 attempts). Read/write/pool timeouts are unchanged from the client default: query embeddings can legitimately take over a minute to return under heavy ingestion load, so those must not be shortened.
• Non-retryable client errors (e.g. 401) still raise immediately without failover or health changes.
• _call_api_endpoint gains an optional timeout override; the ingestion path is unchanged.

Tests

core/tests/unit/test_week1_ingest_fixes.py (20 passed):

• updated test_query_path_retries_once_only: retry once per endpoint
• new: text query fails over on a dead endpoint + skips it during cooldown
• new: image query fails over on a dead endpoint
• new: config order is respected among healthy endpoints
• new: no failover on a 401
• new: query timeout only shortens connect

🤖 Generated with Claude Code