Skip to content

fix: exec the OCI runtime instead of spawning it - #188

Merged
hi-liang merged 2 commits into
mainfrom
fix/exec-oci-runtime
Sep 6, 2026
Merged

hi-liang merged 2 commits into
mainfrom
fix/exec-oci-runtime

Conversation

@hi-liang

@hi-liang hi-liang commented Sep 6, 2026 •

Copy link
Copy Markdown
Member

Problem

A cork worker was found running 71 containerd-shim-runc-v2 processes against 5 containers — 66 orphans holding roughly 310 MiB, 16% of RAM on a 2 GiB host, accumulating until reboot and driving host-wide OOM kills.

call_oci_runtime spawned the runtime as a child and waited on it, leaving the interceptor in the process tree for the lifetime of every runtime call. containerd's shim invokes the runtime through go-runc, which builds commands with Go's exec.CommandContext; a cancelled or timed-out call delivers SIGKILL to the direct child only. So the interceptor died and the real runtime was orphaned beneath it, while the shim believed the call had been cancelled. For a runc delete that leaves the task undeleted and its shim running indefinitely.

Changes

The runtime is now exec'd, so it inherits our pid and the interceptor leaves the process tree entirely. This also fixes a runtime killed by a signal being reported as exit code -1, which process::exit truncates to 255.

Networking mounts are now always read-only. /etc/hosts, /etc/hostname and /etc/resolv.conf are bind-mounted from outside the container's writable layer, so where XFS project quotas restrict that layer they are an escape hatch for filling the host volume — which on a shared host affects every other container. There is no use case here for writable networking mounts and therefore no opt-out; --oi-readonly-networking-mounts is retained as an accepted no-op so a daemon.json still passing it keeps working, which keeps the rollout order-independent. Mount options are normalised rather than appended to: any existing ro or rw is dropped and ro re-appended last, so a mount arriving with no options at all, or with a rw following a ro, still ends up read-only — the kernel honours the last of a conflicting pair.

Worth being precise about what this is not: it is not a barrier between containers. Docker gives each container its own copy of these files, so tampering only affects the container doing it, and a process with code execution bypasses them anyway via HOSTALIASES, RES_OPTIONS, LD_PRELOAD, or by querying a resolver directly. Treat it as quota enforcement, with in-container tamper resistance as a side effect.

The mounts themselves are left in place: Docker writes nameserver 127.0.0.11 into the bind-mounted /etc/resolv.conf for containers on user-defined networks, so removing that mount would break sibling name resolution (see #71).

Documentation. The only complete daemon.json example used runtimeArgs, and Docker generates its own non-exec /bin/sh wrapper for exactly that condition — re-inserting the layer this change removes, so an operator following the README got the fixed binary and still leaked shims. The example is now flagless, and there is a "Passing flags" section giving the wrapper-script alternative with "$@" quoted (Docker's generated wrapper uses a bare $@, which word-splits). The README also no longer presents the CI integration job as a reference deployment — it uses runtimeArgs because the per-runtime flag assertions need it, which means CI never exercises the shape a real host should use.

Verification

Reproduced in a real containerd process tree by hanging the runtime and SIGKILLing the shim's direct child, exactly as exec.CommandContext does:

daemon.json runtimeArgs exec'ing wrapper at path
spawn + wait (before) orphaned orphaned
exec (after) orphaned clean

Only the bottom-right cell is clean, with the shim's direct child being the runtime. This fix is necessary but not sufficient: deployments must also move flags out of runtimeArgs. The corresponding ansible change does this.

tests/runtime_invocation.rs drives a stub runtime and covers five properties: the runtime inherits the interceptor's pid; signalling the interceptor does not orphan it; a runtime that cannot be exec'd is reported as such; exit codes survive the exec; and a signal-killed runtime is not reported as exit 255. All but the exit-code test fail against the previous implementation, which reports unix_wait_status(65280) — 255 — for a SIGTERM'd runtime.

The 11 Docker-backed integration tests were run against a real daemon with all six named runtimes, including a new networking_mounts_readonly_without_any_flag. A sensitivity control — swapping in a no-op interceptor — fails exactly the 7 spec-rewriting tests, confirming the suite would catch lost interception.

36 tests pass. cargo fmt --check and cargo clippy --all-targets with -D warnings are clean.

Release

Cut as v0.3.0: CHANGELOG.md has a v0.3.0 section and Cargo.toml is bumped, per RELEASING.md. The changelog calls out the deployment change explicitly — upgrading the binary alone does not stop the leak.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Mounts without options can remain writable, and the new process test is not portable across all enabled Unix targets.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Replaces runtime spawning with exec, enforces read-only networking mounts, and prepares v0.3.0 documentation and metadata.

Changes:

  • Executes the OCI runtime in place to prevent orphaned processes.
  • Makes networking mounts read-only regardless of the deprecated flag.
  • Adds regression tests and deployment guidance.
File summaries
File Description
src/main.rs Implements runtime exec and unconditional mount rewriting.
src/networking_mounts.rs Tracks and applies read-only mount changes.
tests/runtime_invocation.rs Tests PID, signal, error, and exit propagation.
tests/integration.rs Verifies default networking mounts are read-only.
README.md Documents wrappers and changed mount behavior.
CHANGELOG.md Adds v0.3.0 release notes.
Cargo.toml Bumps the crate to v0.3.0.
Cargo.lock Synchronizes the package version.
Review details
  • Files reviewed: 7/8 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/networking_mounts.rs Resolved
Comment thread tests/runtime_invocation.rs Resolved
Comment thread Cargo.toml Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation/release notes contain a couple of now-inaccurate or confusing statements (wrapper example uses a deprecated no-op flag; changelog overstates “only written when changed”) that should be corrected before release.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/main.rs:160

  • spec_modified is set to true whenever --oi-env / --oi-env-force flags are present, even though modify_env_vars can be a no-op (e.g., variable already exists and force=false). That means config.json may still be rewritten despite the PR goal of avoiding no-op saves; consider returning a bool from modify_env_vars (like modify_networking_mounts) so the save is gated on an actual change.
        spec_modified |= modify_networking_mounts(&mut spec);
        if !env_var_overrides.is_empty() {
            modify_env_vars(&mut spec, env_var_overrides);
            spec_modified = true;
        }
  • Files reviewed: 7/8 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread CHANGELOG.md Resolved
Comment thread README.md Resolved
call_oci_runtime spawned the runtime as a child and waited on it, leaving the
interceptor in the process tree for the lifetime of every runtime call.
containerd's shim invokes the runtime through go-runc, which builds commands with
Go's exec.CommandContext, and a cancelled or timed-out call delivers SIGKILL to
the direct child only -- so the interceptor died and the real runtime underneath
it was orphaned while the shim believed the call had been cancelled. For a
`runc delete` that leaves the task undeleted and its containerd-shim-runc-v2
running indefinitely, holding roughly 5 MiB until the host reboots. A 2 GiB cork
worker was measured carrying 66 such shims against 5 running containers.

Exec'ing means the runtime inherits our pid and we leave the process tree
entirely. It also fixes a runtime killed by a signal being reported as exit code
-1, which process::exit truncates to 255, hiding signal death from callers that
distinguish the two.

This is necessary but not sufficient on its own, and the documentation now says
so. Docker generates its own wrapper script for any runtime whose runtimeArgs is
non-empty, and that generated wrapper does not exec:

    #!/bin/sh
    /usr/local/bin/oci-interceptor --flags $@

It re-inserts exactly the layer this change removes, so an operator following the
old README example got the fixed binary and still leaked shims. The only complete
daemon.json example no longer uses runtimeArgs, and the README documents the
wrapper-script alternative with "$@" quoted -- Docker's generated wrapper uses a
bare $@, which word-splits and glob-expands. It also stops presenting the CI
integration job as a reference deployment: that job uses runtimeArgs because the
per-runtime flag assertions need it, which means CI never exercises the shape a
real host should use.

Separately, /etc/hosts, /etc/hostname and /etc/resolv.conf are now made read-only
unconditionally rather than only under --oi-readonly-networking-mounts. Docker
bind-mounts these from outside the container's writable layer, so where XFS
project quotas restrict that layer they are an escape hatch for filling the host
volume, which on a shared host affects every other container. Note this is not a
barrier between containers -- each gets its own copy, so tampering only affects
the container doing it, and a process with code execution bypasses these files
anyway via HOSTALIASES, RES_OPTIONS, LD_PRELOAD or by querying a resolver
directly. There is no use case here for writable networking mounts, so there is
no opt-out; --oi-readonly-networking-mounts is retained as an accepted no-op so
a daemon.json still passing it keeps working, which keeps the rollout order
independent. The mounts themselves are left in place: Docker points a container
on a user-defined network at its embedded DNS resolver by writing
`nameserver 127.0.0.11` into the bind-mounted /etc/resolv.conf, so removing that
mount would break sibling-container name resolution (see #71).

Mount options are normalised rather than appended to: any existing ro or rw is
dropped and ro re-appended last, so a mount that arrives with no options at all,
or with a rw following a ro, still ends up read-only. The kernel honours the last
of a conflicting pair, so appending without normalising would have left both
cases writable.

modify_networking_mounts now reports whether it changed anything, so the spec is
only rewritten to disk when a mount was actually modified.

tests/runtime_invocation.rs drives a stub runtime and covers the process-tree
behaviour and the paths around it: that the runtime inherits the interceptor's
pid, that signalling the interceptor does not orphan it, that a runtime which
cannot be exec'd is reported as such, that exit codes survive the exec, and that
a signal-killed runtime is not reported as exit 255. All but the exit-code test
fail against the previous implementation, which reports unix_wait_status(65280)
for a SIGTERM'd runtime.
@hi-liang
hi-liang force-pushed the fix/exec-oci-runtime branch from 49b8478 to 37b85f3 Compare September 6, 2026 18:36
@hi-liang
hi-liang merged commit 65be4fc into main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants