Conversation
…dling kg_core sets no timeout on its outbound HTTP requests and exposes no supported way to configure one, so a request that loses its connection mid-flight (e.g. after the machine sleeps and resumes on a different network) hangs forever instead of failing. This patches kg_core's private _do_request to apply a 300s default timeout, and adds a request_timeout kwarg to KGClient for overriding it; because kg_core offers no per-client hook, this is necessarily a process-wide setting, documented as such. Network failures now raise a new KGConnectionError instead of a raw requests exception. Two independent layers produce it: the _do_request patch gives specific messages for our own timeout firing versus the connection being dropped before a response arrived (e.g. "RemoteDisconnected", which we saw the KG's own gateway trigger around 50s on core.kg-ppd for slow queries such as against File); a separate translate_network_errors decorator, applied to every KGClient method that calls into kg_core, gives a plainer version of the same error and keeps working even if kg_core's internals change enough to break the _do_request patch.
KG queries can be slow (e.g. against File, which has hundreds of thousands of instances) and transient connection errors or timeouts are common but often succeed on a second attempt. This adds a retry_on_connection_error decorator, applied only to methods that are read-only or execute a query without persisting anything (list, query, instance_from_full_uri, retrieve_query, user_info, spaces, space_info, is_released, token), retrying with plain exponential backoff (no jitter needed given how long the backoff already is relative to reconnection time) on KGConnectionError. Methods that modify the KG keep only the existing error translation and are never auto-retried, since replaying a write risks duplicating or corrupting data if the original request actually succeeded but its response was lost. max_retries and retry_backoff are new KGClient constructor kwargs, defaulting to 2 retries and a 5s base backoff. Unlike request_timeout, this is genuine per-instance state rather than a process-wide global, since retrying is ordinary fairgraph code and doesn't need to patch kg_core internals. MockKGClient in test/utils.py gained matching _max_retries/_retry_backoff class attributes, since TestSpaceInfoOffline calls KGClient methods (now decorated) directly against a MockKGClient instance rather than a real KGClient.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The aim is to make the KGClient more robust, with better handling of network errors and dropped responses from the KG.
request_timeoutkwarg onKGClient.KGConnectionErrorinstead of a rawrequestsexception. Two independent layers produce it, so translation keeps working even if kg_core's private internals change and the deeper patch can no longer be applied.KGClientmethods (list,query,instance_from_full_uri,retrieve_query,user_info,spaces,space_info,is_released,token) now retry automatically with exponential backoff onKGConnectionError, via newmax_retries/retry_backoffkwargs (defaults: 2 retries, 5s base). Methods that modify the KG are deliberately never auto-retried, since replaying a write risks duplicating or corrupting data if the original request actually succeeded but its response was lost.The default timeout is long: 300 seconds. This is to ensure we wait long enough to get the KG response, but we don't wait forever. This is based on the following timings:
Method: raw HTTP GET requests to the KG core API's
/v3-beta/instancesendpoint for theFiletype (openMINDS type with very many instances), usingrequestsdirectly with a large client-side timeout (well beyond any expected server response time).Pre-production (core.kg-ppd.ebrains.eu):
Production (core.kg.ebrains.eu):
Implications:
request_timeout, comfortably above the slowest observed case.