You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Intermittent ReadFile stalls after guest idle — not reproduced, bisector included #73
On a self-hosted single-node k3s deployment I've observed intermittent multi-second (at times tens of seconds) ReadFile latency on paths that had been reading normally moments earlier. Occurrences correlate loosely with guest inactivity. Working hypothesis class: a wake-up path — suspended guest or idle connection state — blocking on the read. I have not been able to reproduce it under controlled conditions, so I'm filing this for visibility with capture tooling attached, rather than a speculative fix.
Investigation to date
Probes driven by a purpose-built bisector against a real router + guest:
Has anyone observed multi-second ReadFile stalls correlated with guest idle or suspend/resume?
Is there existing latency instrumentation on the guest read path (metrics or trace spans) that could confirm or refute wake-up attribution? If yes, pointers appreciated. If not, that instrumentation may be worth adding independently — it would turn the next occurrence into evidence rather than anecdote.
If it recurs on our side, I'll attach receiver-side latency distributions captured with the tooling below.
Capture tooling
Available on fork branch barceloneta-k3s under .specs/tools/:
Thank you for the careful probe data. Answers to the three asks:
A wake can explain stalls of this size. A resume that restores a snapshot takes seconds, and a cold image pull takes longer. Until now, the first guest call to a suspended environment depended on the router's own parking budget. ate-env-api: wait for a suspended environment to wake instead of failing its first call #86, merged as 7be7449, makes ate-env-api wait for the wake with bounded budgets. On the first wait it logs a line such as waiting for environment ate-env/<id> to wake (state SUSPENDED, wake budget 2m0s). Please retest on main. A stall that comes with this log line is a wake.
env has no metrics or traces on the guest path today. Substrate's router records route latency with a resume attribute, and it logs ResumeActor result with the actor state for each routed request. These two sources can attribute a stall to a resume. Metrics in ate-env-api are a good follow-up; a PR is welcome.
Two connection notes. ate-env-api opens a new router connection for each guest call, thus a stale pooled connection on that hop cannot cause the stall. The client channel to ate-env-api is long-lived, and the Python client sets no gRPC keepalive. If a load balancer or NAT drops idle connections in your setup, the first call after idle can wait for TCP retransmission. Try channel keepalive options to rule this out.
Keep the issue open until a capture shows the cause.
Activity
tomergee commentedon Oct 16, 2026
Thank you for the careful probe data. Answers to the three asks:
waiting for environment ate-env/<id> to wake (state SUSPENDED, wake budget 2m0s). Please retest onmain. A stall that comes with this log line is a wake.ResumeActor resultwith the actor state for each routed request. These two sources can attribute a stall to a resume. Metrics in ate-env-api are a good follow-up; a PR is welcome.Keep the issue open until a capture shows the cause.