Skip to content

Intermittent ReadFile stalls after guest idle — not reproduced, bisector included #73

Open

Description

Summary

On a self-hosted single-node k3s deployment I've observed intermittent multi-second (at times tens of seconds) ReadFile latency on paths that had been reading normally moments earlier. Occurrences correlate loosely with guest inactivity. Working hypothesis class: a wake-up path — suspended guest or idle connection state — blocking on the read. I have not been able to reproduce it under controlled conditions, so I'm filing this for visibility with capture tooling attached, rather than a speculative fix.

Investigation to date

Probes driven by a purpose-built bisector against a real router + guest:

Probe Result
Missing-path reads, cold and warm NotFound in ~25 ms, no stalls
Sequential read load (150 ops) clean
Forced suspend → immediate warm read guest auto-resumed; read completed in 185 ms
Composite (follow-stream + poller + mid-run suspend) 60/60 reads clean

Asks

  1. Has anyone observed multi-second ReadFile stalls correlated with guest idle or suspend/resume?
  2. Is there existing latency instrumentation on the guest read path (metrics or trace spans) that could confirm or refute wake-up attribution? If yes, pointers appreciated. If not, that instrumentation may be worth adding independently — it would turn the next occurrence into evidence rather than anecdote.
  3. If it recurs on our side, I'll attach receiver-side latency distributions captured with the tooling below.

Capture tooling

Available on fork branch barceloneta-k3s under .specs/tools/:

  • wedge — targeted bisector: segment-read patterns (s1/s2/s3), plus getenv, suspend, warm-hang, load, soak modes
  • e03probe — transcript-egress latency probe: follow-stream vs polling vs segmented reads, plus microbenchmarks (shell echo round-trip, StartProcess → first chunk, ReadFile at 4 KB / 256 KB / 1 MB)

Both emit JSON lines on stdout, so runs are directly diffable.

Environment

  • Single-node k3s; guests exercised both run-to-completion and under suspend/resume
  • Standard client → api → router → guest gRPC path

Activity

  1. tomergee commented on Oct 16, 2026

    @tomergee
    Collaborator

    Thank you for the careful probe data. Answers to the three asks:

    1. A wake can explain stalls of this size. A resume that restores a snapshot takes seconds, and a cold image pull takes longer. Until now, the first guest call to a suspended environment depended on the router's own parking budget. ate-env-api: wait for a suspended environment to wake instead of failing its first call #86, merged as 7be7449, makes ate-env-api wait for the wake with bounded budgets. On the first wait it logs a line such as waiting for environment ate-env/<id> to wake (state SUSPENDED, wake budget 2m0s). Please retest on main. A stall that comes with this log line is a wake.
    2. env has no metrics or traces on the guest path today. Substrate's router records route latency with a resume attribute, and it logs ResumeActor result with the actor state for each routed request. These two sources can attribute a stall to a resume. Metrics in ate-env-api are a good follow-up; a PR is welcome.
    3. Two connection notes. ate-env-api opens a new router connection for each guest call, thus a stale pooled connection on that hop cannot cause the stall. The client channel to ate-env-api is long-lived, and the Python client sets no gRPC keepalive. If a load balancer or NAT drops idle connections in your setup, the first call after idle can wait for TCP retransmission. Try channel keepalive options to rule this out.

    Keep the issue open until a capture shows the cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions