Skip to content

Add request timeout, error handling, and retry to KGClient - #148

Open
apdavison wants to merge 2 commits into
HumanBrainProject:masterfrom
apdavison:kg-client-robustness
Open

apdavison wants to merge 2 commits into
HumanBrainProject:masterfrom
apdavison:kg-client-robustness

Conversation

@apdavison

@apdavison apdavison commented Sep 23, 2026

Copy link
Copy Markdown
Member

The aim is to make the KGClient more robust, with better handling of network errors and dropped responses from the KG.

  • kg_core sets no timeout on its outbound HTTP requests and exposes no supported way to configure one, so a request that loses its connection mid-flight (e.g. switching wifi networks) hangs forever instead of failing. This patches kg_core to apply a default 300s timeout, configurable via a new request_timeout kwarg on KGClient.
  • Network failures (lost connection, no response within the timeout) now raise a clear KGConnectionError instead of a raw requests exception. Two independent layers produce it, so translation keeps working even if kg_core's private internals change and the deeper patch can no longer be applied.
  • Read-only and query-only KGClient methods (list, query, instance_from_full_uri, retrieve_query, user_info, spaces, space_info, is_released, token) now retry automatically with exponential backoff on KGConnectionError, via new max_retries/retry_backoff kwargs (defaults: 2 retries, 5s base). Methods that modify the KG are deliberately never auto-retried, since replaying a write risks duplicating or corrupting data if the original request actually succeeded but its response was lost.

The default timeout is long: 300 seconds. This is to ensure we wait long enough to get the KG response, but we don't wait forever. This is based on the following timings:

Method: raw HTTP GET requests to the KG core API's /v3-beta/instances endpoint for the File type (openMINDS type with very many instances), using requests directly with a large client-side timeout (well beyond any expected server response time).

Pre-production (core.kg-ppd.ebrains.eu):

  • size=1, no count: dropped at 50.09s and 50.12s (two separate trials). Connection aborted with no HTTP response ("RemoteDisconnected").
  • Same query, once "warm": succeeded consistently at ~11-12s (repeat identical queries appear to hit a query/result cache).
  • Fresh query, size=50: succeeded at 23.16s.
  • Fresh query, size=500: dropped again at 50.09s (same failure, independent of page size/shape).
  • Conclusion: ppd has a hard, consistent ~50-second cutoff (three drops landed within 30ms of each other) - almost certainly a gateway/reverse-proxy in front of the KG core API, not the KG's own query engine. Anything that doesn't get a response within ~50s is silently killed with no HTTP status at all.

Production (core.kg.ebrains.eu):

  • size=1, no count: 193.91s, succeeded.
  • Same query repeated: 32.92s, 27.95s, 23.57s, 24.43s - much faster once warm, but still slower than ppd's warm case.
  • size=500, fresh: 119.69s, succeeded.
  • size=1, returnTotalResults=true: 121.24s, succeeded (total count: 956,369 File instances).
  • size=1000, returnTotalResults=true, fresh: 78.62s, succeeded.
  • size=5000, returnTotalResults=true, fresh: 142.90s, succeeded.
  • Conclusion: no drops in 9 trials up to ~194s. Either there's no fixed gateway cutoff on production, or its threshold is well above what was tested.

Implications:

  • The ~50s ppd cutoff is environment-specific, not a general KG timeout to design around.
  • A client-side default timeout needs to tolerate production's observed range (up to ~194s) without spuriously killing a legitimate slow-but-successful query: hence the 300s default chosen for request_timeout, comfortably above the slowest observed case.
  • Because a query that eventually succeeds when "warm" (e.g. ~12-30s) can fail cold (timeout, or a dropped connection on ppd), retrying a failed read/query-only request is often enough to succeed quickly on the second attempt - the motivation for the retry behaviour added on top of the timeout/error-handling work.

…dling

kg_core sets no timeout on its outbound HTTP requests and exposes no supported way to configure one, so a request that loses its connection mid-flight (e.g. after the machine sleeps and resumes on a different network) hangs forever instead of failing. This patches kg_core's private _do_request to apply a 300s default timeout, and adds a request_timeout kwarg to KGClient for overriding it; because kg_core offers no per-client hook, this is necessarily a process-wide setting, documented as such.

Network failures now raise a new KGConnectionError instead of a raw requests exception. Two independent layers produce it: the _do_request patch gives specific messages for our own timeout firing versus the connection being dropped before a response arrived (e.g. "RemoteDisconnected", which we saw the KG's own gateway trigger around 50s on core.kg-ppd for slow queries such as against File); a separate translate_network_errors decorator, applied to every KGClient method that calls into kg_core, gives a plainer version of the same error and keeps working even if kg_core's internals change enough to break the _do_request patch.
KG queries can be slow (e.g. against File, which has hundreds of thousands of instances) and transient connection errors or timeouts are common but often succeed on a second attempt. This adds a retry_on_connection_error decorator, applied only to methods that are read-only or execute a query without persisting anything (list, query, instance_from_full_uri, retrieve_query, user_info, spaces, space_info, is_released, token), retrying with plain exponential backoff (no jitter needed given how long the backoff already is relative to reconnection time) on KGConnectionError. Methods that modify the KG keep only the existing error translation and are never auto-retried, since replaying a write risks duplicating or corrupting data if the original request actually succeeded but its response was lost.

max_retries and retry_backoff are new KGClient constructor kwargs, defaulting to 2 retries and a 5s base backoff. Unlike request_timeout, this is genuine per-instance state rather than a process-wide global, since retrying is ordinary fairgraph code and doesn't need to patch kg_core internals.

MockKGClient in test/utils.py gained matching _max_retries/_retry_backoff class attributes, since TestSpaceInfoOffline calls KGClient methods (now decorated) directly against a MockKGClient instance rather than a real KGClient.
@apdavison apdavison added the enhancement New feature or request label Sep 23, 2026
@apdavison apdavison moved this from Todo to In Progress in fairgraph development Sep 23, 2026
@apdavison apdavison added this to the 0.16 milestone Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant