You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(supervisor): shared supervisor uses about 4 threads and 7 gateway sockets per sandbox #5660
I ran a mini scale test of OpenShell on Kubernetes with a shared supervisor pool, at up to 450 sandboxes. The supervisor's thread, file-descriptor, and socket counts grew in a straight line with the number of sandboxes, and memory grew until the supervisor ran out of memory. I need one shared supervisor to host many sandboxes without its cost growing this fast.
Problem Statement
Each sandbox in a shared supervisor gets its own:
OS thread, plus a private Tokio runtime with 2 worker threads (INSTANCE_WORKER_THREADS = 2), with extra blocking threads added on demand
TLS connections to the gateway, about 7 sockets per sandbox
What we measured:
Sandboxes
Threads
File descriptors
Sockets
256 (idle)
~1,040
2,574
1,798
256 (under load)
~1,182
No OS limits were hit. Idle exec sessions are fine: 450 held sessions added no threads, so #5217 works.
Why This Is a Problem
Cost grows with every sandbox. Hosting more sandboxes on one supervisor is the point of the shared model, but each one adds about 4 threads and 7 sockets whether it's busy or idle.
Memory. The supervisor is a glibc build with the default allocator. glibc gives busy threads their own malloc arenas, up to 8 per CPU core. With about 1,000 threads, memory spreads across many arenas and is slow to return to the OS, which likely added to the memory growth behind the out-of-memory kill.
Gateway load. About 7 TLS connections per sandbox means a large shared supervisor holds thousands of connections to the gateway.
Impact / Why This Matters
The number of sandboxes one supervisor can host is limited by threads and memory, not by real work.
Operators have to give supervisors much more memory, or run more of them, than the workload needs.
Acceptance Criteria
Threads per idle sandbox in a shared supervisor drop well below 4. For example, sandboxes could share worker threads while keeping their own identity and logs.
Gateway connections per sandbox drop, for example by sharing a connection across sandboxes.
Supervisor memory at 256 and 450 sandboxes is measured before and after, and stays flat at a steady sandbox count.
Reproduction Steps
Deploy OpenShell on Kubernetes with a shared supervisor pool.
Create 256 sandboxes.
On the supervisor pod, count threads, file descriptors, and sockets: ls /proc/<pid>/task | wc -l, ls /proc/<pid>/fd | wc -l, and ls -l /proc/<pid>/fd | grep -c socket.
Repeat at different sandbox counts. All three grow roughly linearly.
Environment
OpenShell: mini scale test build; code paths confirmed on main at 624acee
Deployment: Kubernetes (Helm), shared supervisor pool, glibc supervisor image
Activity