Repository navigation
fix: exec the OCI runtime instead of spawning it - #188
Conversation
0483c24 to
1f3ca49
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Mounts without options can remain writable, and the new process test is not portable across all enabled Unix targets.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Replaces runtime spawning with exec, enforces read-only networking mounts, and prepares v0.3.0 documentation and metadata.
Changes:
- Executes the OCI runtime in place to prevent orphaned processes.
- Makes networking mounts read-only regardless of the deprecated flag.
- Adds regression tests and deployment guidance.
File summaries
| File | Description |
|---|---|
src/main.rs |
Implements runtime exec and unconditional mount rewriting. |
src/networking_mounts.rs |
Tracks and applies read-only mount changes. |
tests/runtime_invocation.rs |
Tests PID, signal, error, and exit propagation. |
tests/integration.rs |
Verifies default networking mounts are read-only. |
README.md |
Documents wrappers and changed mount behavior. |
CHANGELOG.md |
Adds v0.3.0 release notes. |
Cargo.toml |
Bumps the crate to v0.3.0. |
Cargo.lock |
Synchronizes the package version. |
Review details
- Files reviewed: 7/8 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
1f3ca49 to
49b8478
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The documentation/release notes contain a couple of now-inaccurate or confusing statements (wrapper example uses a deprecated no-op flag; changelog overstates “only written when changed”) that should be corrected before release.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/main.rs:160
spec_modifiedis set to true whenever--oi-env/--oi-env-forceflags are present, even thoughmodify_env_varscan be a no-op (e.g., variable already exists andforce=false). That meansconfig.jsonmay still be rewritten despite the PR goal of avoiding no-op saves; consider returning aboolfrommodify_env_vars(likemodify_networking_mounts) so the save is gated on an actual change.
spec_modified |= modify_networking_mounts(&mut spec);
if !env_var_overrides.is_empty() {
modify_env_vars(&mut spec, env_var_overrides);
spec_modified = true;
}
- Files reviewed: 7/8 changed files
- Comments generated: 2
- Review effort level: Lite
call_oci_runtime spawned the runtime as a child and waited on it, leaving the
interceptor in the process tree for the lifetime of every runtime call.
containerd's shim invokes the runtime through go-runc, which builds commands with
Go's exec.CommandContext, and a cancelled or timed-out call delivers SIGKILL to
the direct child only -- so the interceptor died and the real runtime underneath
it was orphaned while the shim believed the call had been cancelled. For a
`runc delete` that leaves the task undeleted and its containerd-shim-runc-v2
running indefinitely, holding roughly 5 MiB until the host reboots. A 2 GiB cork
worker was measured carrying 66 such shims against 5 running containers.
Exec'ing means the runtime inherits our pid and we leave the process tree
entirely. It also fixes a runtime killed by a signal being reported as exit code
-1, which process::exit truncates to 255, hiding signal death from callers that
distinguish the two.
This is necessary but not sufficient on its own, and the documentation now says
so. Docker generates its own wrapper script for any runtime whose runtimeArgs is
non-empty, and that generated wrapper does not exec:
#!/bin/sh
/usr/local/bin/oci-interceptor --flags $@
It re-inserts exactly the layer this change removes, so an operator following the
old README example got the fixed binary and still leaked shims. The only complete
daemon.json example no longer uses runtimeArgs, and the README documents the
wrapper-script alternative with "$@" quoted -- Docker's generated wrapper uses a
bare $@, which word-splits and glob-expands. It also stops presenting the CI
integration job as a reference deployment: that job uses runtimeArgs because the
per-runtime flag assertions need it, which means CI never exercises the shape a
real host should use.
Separately, /etc/hosts, /etc/hostname and /etc/resolv.conf are now made read-only
unconditionally rather than only under --oi-readonly-networking-mounts. Docker
bind-mounts these from outside the container's writable layer, so where XFS
project quotas restrict that layer they are an escape hatch for filling the host
volume, which on a shared host affects every other container. Note this is not a
barrier between containers -- each gets its own copy, so tampering only affects
the container doing it, and a process with code execution bypasses these files
anyway via HOSTALIASES, RES_OPTIONS, LD_PRELOAD or by querying a resolver
directly. There is no use case here for writable networking mounts, so there is
no opt-out; --oi-readonly-networking-mounts is retained as an accepted no-op so
a daemon.json still passing it keeps working, which keeps the rollout order
independent. The mounts themselves are left in place: Docker points a container
on a user-defined network at its embedded DNS resolver by writing
`nameserver 127.0.0.11` into the bind-mounted /etc/resolv.conf, so removing that
mount would break sibling-container name resolution (see #71).
Mount options are normalised rather than appended to: any existing ro or rw is
dropped and ro re-appended last, so a mount that arrives with no options at all,
or with a rw following a ro, still ends up read-only. The kernel honours the last
of a conflicting pair, so appending without normalising would have left both
cases writable.
modify_networking_mounts now reports whether it changed anything, so the spec is
only rewritten to disk when a mount was actually modified.
tests/runtime_invocation.rs drives a stub runtime and covers the process-tree
behaviour and the paths around it: that the runtime inherits the interceptor's
pid, that signalling the interceptor does not orphan it, that a runtime which
cannot be exec'd is reported as such, that exit codes survive the exec, and that
a signal-killed runtime is not reported as exit 255. All but the exit-code test
fail against the previous implementation, which reports unix_wait_status(65280)
for a SIGTERM'd runtime.
49b8478 to
37b85f3
Compare
Problem
A cork worker was found running 71
containerd-shim-runc-v2processes against 5 containers — 66 orphans holding roughly 310 MiB, 16% of RAM on a 2 GiB host, accumulating until reboot and driving host-wide OOM kills.call_oci_runtimespawned the runtime as a child and waited on it, leaving the interceptor in the process tree for the lifetime of every runtime call. containerd's shim invokes the runtime through go-runc, which builds commands with Go'sexec.CommandContext; a cancelled or timed-out call delivers SIGKILL to the direct child only. So the interceptor died and the real runtime was orphaned beneath it, while the shim believed the call had been cancelled. For arunc deletethat leaves the task undeleted and its shim running indefinitely.Changes
The runtime is now
exec'd, so it inherits our pid and the interceptor leaves the process tree entirely. This also fixes a runtime killed by a signal being reported as exit code-1, whichprocess::exittruncates to 255.Networking mounts are now always read-only.
/etc/hosts,/etc/hostnameand/etc/resolv.confare bind-mounted from outside the container's writable layer, so where XFS project quotas restrict that layer they are an escape hatch for filling the host volume — which on a shared host affects every other container. There is no use case here for writable networking mounts and therefore no opt-out;--oi-readonly-networking-mountsis retained as an accepted no-op so adaemon.jsonstill passing it keeps working, which keeps the rollout order-independent. Mount options are normalised rather than appended to: any existingroorrwis dropped androre-appended last, so a mount arriving with no options at all, or with arwfollowing aro, still ends up read-only — the kernel honours the last of a conflicting pair.Worth being precise about what this is not: it is not a barrier between containers. Docker gives each container its own copy of these files, so tampering only affects the container doing it, and a process with code execution bypasses them anyway via
HOSTALIASES,RES_OPTIONS,LD_PRELOAD, or by querying a resolver directly. Treat it as quota enforcement, with in-container tamper resistance as a side effect.The mounts themselves are left in place: Docker writes
nameserver 127.0.0.11into the bind-mounted/etc/resolv.conffor containers on user-defined networks, so removing that mount would break sibling name resolution (see #71).Documentation. The only complete
daemon.jsonexample usedruntimeArgs, and Docker generates its own non-exec/bin/shwrapper for exactly that condition — re-inserting the layer this change removes, so an operator following the README got the fixed binary and still leaked shims. The example is now flagless, and there is a "Passing flags" section giving the wrapper-script alternative with"$@"quoted (Docker's generated wrapper uses a bare$@, which word-splits). The README also no longer presents the CIintegrationjob as a reference deployment — it usesruntimeArgsbecause the per-runtime flag assertions need it, which means CI never exercises the shape a real host should use.Verification
Reproduced in a real containerd process tree by hanging the runtime and SIGKILLing the shim's direct child, exactly as
exec.CommandContextdoes:daemon.jsonruntimeArgspathOnly the bottom-right cell is clean, with the shim's direct child being the runtime. This fix is necessary but not sufficient: deployments must also move flags out of
runtimeArgs. The corresponding ansible change does this.tests/runtime_invocation.rsdrives a stub runtime and covers five properties: the runtime inherits the interceptor's pid; signalling the interceptor does not orphan it; a runtime that cannot beexec'd is reported as such; exit codes survive theexec; and a signal-killed runtime is not reported as exit 255. All but the exit-code test fail against the previous implementation, which reportsunix_wait_status(65280)— 255 — for a SIGTERM'd runtime.The 11 Docker-backed integration tests were run against a real daemon with all six named runtimes, including a new
networking_mounts_readonly_without_any_flag. A sensitivity control — swapping in a no-op interceptor — fails exactly the 7 spec-rewriting tests, confirming the suite would catch lost interception.36 tests pass.
cargo fmt --checkandcargo clippy --all-targetswith-D warningsare clean.Release
Cut as v0.3.0:
CHANGELOG.mdhas a v0.3.0 section andCargo.tomlis bumped, perRELEASING.md. The changelog calls out the deployment change explicitly — upgrading the binary alone does not stop the leak.