diff --git a/README.md b/README.md index fd135c39..e2ee294d 100644 --- a/README.md +++ b/README.md @@ -347,6 +347,40 @@ same facts as one object with the fields `url`, `state`, `draft`, `head`, as in `quest wait`. Any PR state exits 0, invalid arguments exit 2, and other failures exit 1. +## Gate expensive commands + +`quest gate [args...]` holds an expensive command until the host has +room for it, across all of a user's repositories and worktrees: + +```sh +quest gate bun run test +quest gate cargo build --release +``` + +Admission is sized from the host, with no configuration. Every platform waits +while the host-wide 1-minute load average exceeds the host's CPU count, or while +available memory is below one command's budget: 4 GiB, or a quarter of RAM on +smaller hosts, where RAM is capped by any container memory limit. On Linux, +available memory includes reclaimable host and cgroup file cache and is bounded +by the remaining budget of each enclosing memory cgroup. Load and memory count +work from every session and user. The load average lags new work, so the gate +admits at most one waiting command every 5 seconds per host rather than +releasing every waiter at once. Windows reports no load +average and macOS counts reclaimable cache as used memory, so those signals are +skipped there. + +Admission is the gate's only job: it holds no slot or lock while the command +runs and does not track the command's descendants. A nested invocation runs +immediately inside the admitted command. Internal compiler or test worker +parallelism remains the command's responsibility. + +This repository's `just check`, `just test`, `just worker-check`, and +`just worker-test` run through the gate. + +Arguments, working directory, environment and terminal streams pass through to +the executable, and waiting happens before it starts. There is no fairness +guarantee or timeout: a command waits as long as the host stays busy. + ## Prunable local checkouts The experimental `quest worktree` commands share one disk budget across this diff --git a/assets/skills/complete.md b/assets/skills/complete.md index caa06c20..67a8a2b7 100644 --- a/assets/skills/complete.md +++ b/assets/skills/complete.md @@ -27,7 +27,6 @@ Each sub-agent blocks on its own waits and reports back only when done or blocke For GitHub review or merge activity, it uses `quest wait` and resumes with the returned cursor as described by `quest skill merge`, rather than polling GitHub. Limit the number of active agents to the CPU core count. -Other sessions share this machine: hold new agents while the load average exceeds the core count. Tell each merge sub-agent that you watch its merge queue, so it reports once auto-merge is enabled. Wait on each such PR with `quest wait`, as `quest skill merge` diff --git a/assets/skills/spawn.md b/assets/skills/spawn.md index 42ce8836..e67b5380 100644 --- a/assets/skills/spawn.md +++ b/assets/skills/spawn.md @@ -29,7 +29,6 @@ Start each quest in a background sub-agent with its own isolated worktree, since Each agent reports back only when done or blocked. Limit the number of active agents to the physical CPU core count. -Other sessions share this machine: hold new agents while the load average exceeds the core count. When done, each agent's claim PR holds its work, still a draft if decisions or unfinished work remain. Its report lists every open decision and suggested follow-up with a recommendation. diff --git a/justfile b/justfile index dfb91c6d..dadcba80 100644 --- a/justfile +++ b/justfile @@ -8,8 +8,14 @@ default: install: bun install --frozen-lockfile +# Expensive recipes wait for host room through `quest gate`; nested calls run at once. +gate := "bun run --silent quest gate just" + # Check code, repository tooling, and the quest tree. check: + {{gate}} _check + +_check: bun run check bun run --silent quest check bun run --silent quest --root examples/export check @@ -21,6 +27,9 @@ check: # Run tests, including the shell hooks and the lifecycle of both builds. test: + {{gate}} _test + +_test: bun run test bash .claude/hooks/direnv.test.sh bash scripts/install-check.test.sh @@ -52,10 +61,16 @@ run *args: # Check generated binding types, TypeScript, formatting, and deployment bundle. worker-check: + {{gate}} _worker-check + +_worker-check: bun run --cwd cloudflare check # Run Worker integration tests in the Cloudflare runtime. worker-test: + {{gate}} _worker-test + +_worker-test: bun run --cwd cloudflare test # Serve the Worker on localhost with seeded data and a fake GitHub, for checking UI. diff --git a/quest/m0/README.md b/quest/m0/README.md index 3fa45d15..589735b7 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -35,7 +35,7 @@ ownership-manifest design: ## Required -- [Expensive command gate](/quest/m0/local-gate.md) - shared CPU admission across worktrees and repositories +- [Gate contention measurements](/quest/m0/local-gate.md) - evidence that gated checks stay responsive under shared load - [Worktree process ownership](/quest/m0/worktree.md) - supervise checkout commands and release checkouts automatically - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - reclaim completed and suspended checkouts through their harness - [Agent workflow helpers](/quest/m0/helpers/README.md) - CLI commands for CI waits, PR status, claims, quest moves, and serialized merges diff --git a/quest/m0/local-gate.md b/quest/m0/local-gate.md index 1977ccd7..998c5fd1 100644 --- a/quest/m0/local-gate.md +++ b/quest/m0/local-gate.md @@ -1,33 +1,59 @@ -# [M] Bound expensive commands with quest gate +# [S] Measure quest gate under representative contention ## Goal -`quest gate ` admits a bounded number of expensive commands for one user -on one host, shared across repositories and worktrees. Concurrent agent sessions -remain useful without multiplying full-machine builds and test stacks. +`quest gate ` holds expensive commands for one user on one host until the +host has CPU and memory room, shared across repositories and worktrees. +Concurrent agent sessions remain useful without multiplying full-machine builds +and test stacks. + +Load and memory admission and its automatic use by `just check` / `just test` +have landed. What remains is measuring representative build/test contention +with and without the gate, for example how many concurrent Worker suites hit +their 5-second timeouts, before claiming a responsiveness improvement. ## Plan -- Decided 2026-10-15: gate expensive command phases, not agent lifetimes. Use a - simple configurable concurrency limit, with a CPU-count default. Repository - entrypoints invoke the gate automatically; ordinary reasoning and I/O do not - occupy a CPU slot. No adaptive scheduler or new daemon is needed for this scope. -- Wait before launching the command, its stack, or its test timeout clocks. - Preserve arguments, terminal I/O, exit status, and cancellation. Nested gated - calls must not deadlock or silently multiply the admitted concurrency. -- A slot remains owned while expensive descendants run. Cover interruption, - crashes, concurrent acquisition, and recovery without relying on stale PID - guesses. Unsupported ownership semantics fail explicitly. -- Reuse existing compiler scheduling such as mbx; this is admission for whole - expensive phases, not a replacement compiler scheduler. Internal worker - concurrency remains a separate consumer responsibility. -- Exercise contention across processes and repositories, cancellation, nesting, - and exact exit propagation with small fixtures. Measure admission overhead - and representative contention before claiming a responsiveness improvement. -- Update CLI help and existing usage docs inline. Keep the shared primitive - outside repository-specific policy and independent of the hosted runner. +- Decided 2026-10-15: gate expensive command phases, not agent lifetimes. + Repository entrypoints invoke the gate automatically; ordinary reasoning and + I/O are not gated. No adaptive scheduler or new daemon is needed for this scope. +- Decided 2026-10-15 (user): gate checks and CI-like suites on host load, not + agents, because load from every session and user is what starves test timers. + The agent-level load-average rule was removed from the spawn and complete skills. +- Decided 2026-10-21 (user): admission is load and memory only, on every + platform. This supersedes the 2026-10-15 decisions to add Linux systemd scope + slots on top: every review round on the scope handshake, `cgroup.kill` inode + pinning, and manager probes found another race, for little admission benefit. + Exact descendant ownership and orphan cleanup move to the + [worktree process-ownership quest](/quest/m0/worktree.md). The removed + backend, its probes, and its opt-in systemd suite are recoverable from commit + 5684307 in [kixelated/quest#128](https://github.com/kixelated/quest/pull/128). +- Decided 2026-10-15 (user): Bun's `--no-orphans` stays a research option. Its + best-effort registration and bounded PID-tree walks do not establish exact + ownership by themselves. +- Decided 2026-10-15: at most one admission per 5-second window across all + callers, claimed by exclusively creating a per-window file. A claim is + refused while the previous window's claim exists, so admissions are always a + full window apart (revised 2026-10-21 after a Codex finding). The load average + lags new work and is sampled every 5 seconds, so waiters that all see it fall + must not start together. Nothing is held across the command, so a crash + leaves no stale lock. Claims live in `$XDG_RUNTIME_DIR`, or a per-hostname + cache directory, because a home directory may be shared between hosts. +- Decided 2026-10-15 (user): limits come from host specs, with no configuration + file. The load limit is the host's CPU count, because the load average counts + every CPU, including ones outside this process's affinity (revised + 2026-10-21 from the usable core count). Admission waits while available + memory is below `min(4 GiB, RAM / 4)`: 4 GiB is a typical build or test + stack, and a small host still runs one command. RAM is capped by a cgroup + memory limit, and cgroup headroom subtracts inactive file cache from usage, + since it is reclaimable. macOS counts reclaimable cache as used memory, so + only its load is gated; Windows reports no load average, so only its memory is. +- Nested gated calls run immediately, identified by a marker the gate sets in + the admitted command's environment. Fairness is not promised. +- Measure admission overhead alongside contention; the earlier 37 ms figure + included the removed systemd probe. ## Related - [Worktree process ownership](/quest/m0/worktree.md) - owns checkout command descendants, with its own process-ownership backend -- [Own-PC runner](/quest/m0/kix/runner.md) - reuse local command supervision +- [Own-PC runner](/quest/m0/kix/runner.md) - can gate expensive job phases on host room diff --git a/quest/m0/worktree.md b/quest/m0/worktree.md index 554ca376..dd0c07a2 100644 --- a/quest/m0/worktree.md +++ b/quest/m0/worktree.md @@ -48,5 +48,5 @@ work, stealing a live lock, or deleting scratch a running process still uses. ## Related -- [Expensive command gate](/quest/m0/local-gate.md) - independent CPU and memory admission +- [Gate contention measurements](/quest/m0/local-gate.md) - independent CPU and memory admission - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - native agents and workflow adoption diff --git a/src/cli/gate.ts b/src/cli/gate.ts new file mode 100644 index 00000000..f4d17a97 --- /dev/null +++ b/src/cli/gate.ts @@ -0,0 +1,266 @@ +// Host-wide command admission, sized from the host itself: every platform waits +// for CPU load and free memory before starting the command. The gate owns no +// descendants and holds nothing while the command runs. +import { spawn } from "node:child_process"; +import { closeSync, existsSync, mkdirSync, openSync, readdirSync, readFileSync, rmSync } from "node:fs"; +import { availableParallelism, constants, cpus, homedir, hostname, loadavg, totalmem } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { setTimeout as sleep } from "node:timers/promises"; + +/** The machine-wide inputs to load admission, replaceable in tests. */ +export interface Host { + /** The 1-minute load average. */ + load(): number; + /** Bytes available to a new command without swapping. */ + memory(): number; + now(): number; + /** Atomically claim an admission window; false when another caller took it. */ + claim(window: number): boolean; + sleep(ms: number, signal: AbortSignal): Promise; +} + +export interface Limits { + /** The highest 1-minute load average that still admits a command. */ + load: number; + /** The least available memory, in bytes, that still admits a command. */ + memory: number; +} + +export type Waiting = "load" | "memory"; + +/** + * Admissions across all callers are at least this far apart. The load average + * lags new work, so waiters that all see it fall must not start together. + */ +export const SETTLE_MS = 5000; + +/** The memory one gated build or test stack is assumed to need. */ +export const COMMAND_MEMORY = 4 * 2 ** 30; + +/** Usable memory: physical RAM, or a smaller cgroup/container limit. */ +function usableMemory(): number { + if (process.platform === "linux") return linuxMemory().total; + const constrained = process.constrainedMemory?.() ?? 0; + return constrained > 0 ? Math.min(totalmem(), constrained) : totalmem(); +} + +/** + * Derive the limits from the host: no configuration. The load average counts + * runnable threads on every CPU, including ones outside this process's + * affinity, so its limit is one per host CPU. Each admission reserves one + * command's memory, though never more than a quarter of a small host's RAM. + */ +export function hostLimits( + processors = Math.max(availableParallelism(), cpus().length), + memory = usableMemory(), +): Limits { + return { + load: processors, + memory: Math.min(COMMAND_MEMORY, memory / 4), + }; +} + +/** A memory.stat counter, or 0 where the file or key is absent. */ +function cgroupStat(group: string, key: string): number { + try { + const stat = readFileSync(join(group, "memory.stat"), "utf8"); + return Number(stat.match(new RegExp(`^${key} (\\d+)$`, "m"))?.[1] ?? 0); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return 0; + throw error; + } +} + +/** Count reclaimable host cache and the headroom of every enclosing memory cgroup. */ +export function linuxMemory(proc = "/proc", cgroups = "/sys/fs/cgroup"): { total: number; available: number } { + const meminfo = readFileSync(join(proc, "meminfo"), "utf8"); + const bytes = (key: string) => { + const value = meminfo.match(new RegExp(`^${key}:\\s+(\\d+) kB$`, "m"))?.[1]; + if (value === undefined) throw new Error(`gate cannot read Linux ${key}`); + return Number(value) * 1024; + }; + let total = bytes("MemTotal"); + let available = bytes("MemAvailable"); + for (const line of readFileSync(join(proc, "self/cgroup"), "utf8").trim().split("\n")) { + const [, controllers, path] = line.match(/^\d+:([^:]*):(.*)$/) ?? []; + if (controllers === undefined || path === undefined) throw new Error("gate cannot read Linux memory cgroups"); + const unified = controllers === ""; + if (!unified && !controllers?.split(",").includes("memory")) continue; + const root = resolve(cgroups, unified ? "" : "memory"); + const usage = unified ? "memory.current" : "memory.usage_in_bytes"; + // Usage includes inactive file cache, which the kernel reclaims without + // swapping; subtract it, as container runtimes do for their working set. + const cache = unified ? "inactive_file" : "total_inactive_file"; + const limits = unified + ? ["memory.max", "memory.high"] + : ["memory.limit_in_bytes", "memory.soft_limit_in_bytes"]; + // Child cgroups inherit their parents' limits and shared memory usage. The + // walk ends at the mount point, which a container without a cgroup + // namespace bind-mounts from its own cgroup: its membership path may not + // exist below the mount, but the container's limit still applies there. + const start = resolve(root, `.${path}`); + // A cgroup namespace shows a cgroup outside it as `/..`, which is not mounted. + const inside = relative(root, start); + if (inside === ".." || inside.startsWith(`..${sep}`)) continue; + for (let group = start; ; group = dirname(group)) { + for (const file of limits) { + try { + const limit = Number(readFileSync(join(group, file), "utf8").trim()); + if (!Number.isFinite(limit) || limit <= 0) continue; // "max", or an unset v1 soft limit. + const current = Number(readFileSync(join(group, usage), "utf8").trim()); + if (!Number.isFinite(current) || current < 0) + throw new Error("gate cannot read Linux cgroup memory usage"); + const used = Math.max(0, current - cgroupStat(group, cache)); + total = Math.min(total, limit); + available = Math.min(available, Math.max(0, limit - used)); + } catch (error) { + // The memory controller may not be enabled in every ancestor. + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + } + if (group === root || group === dirname(group)) break; + } + } + return { total, available }; +} + +/** + * Claims belong to this host, since a home directory may be shared between + * machines: the user's runtime directory where one exists, otherwise a + * per-host cache directory. + */ +export function claimDirectory(env = process.env, home = homedir(), host = hostname()): string { + return env.XDG_RUNTIME_DIR + ? join(env.XDG_RUNTIME_DIR, "quest", "gate") + : join(home, ".cache", "quest", "gate", host); +} + +/** Claims are empty files, named by window, in one per-user, per-host directory. */ +export function fileHost(directory: string, now = Date.now): Host { + return { + load: () => loadavg()[0], + // macOS counts reclaimable cache as used, so only its load is gated. + memory: () => + process.platform === "darwin" + ? Infinity + : process.platform === "linux" + ? linuxMemory().available + : process.availableMemory(), + now, + claim(window) { + if (window !== Math.floor(now() / SETTLE_MS)) return false; + mkdirSync(directory, { recursive: true }); + const file = join(directory, String(window)); + try { + closeSync(openSync(file, "wx")); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "EEXIST") return false; + throw error; + } + // The previous window's claim may have admitted a command moments ago, + // so admissions stay a full window apart. Only an admitted claim removes + // older ones, and none can be admitted while this one exists, so that + // claim is still here. A descheduled caller may also have created an old + // claim after cleanup; it must not admit work from that expired sample. + // A refused claim is removed, so it never holds back the next window. + if (existsSync(join(directory, String(window - 1))) || window !== Math.floor(now() / SETTLE_MS)) { + rmSync(file, { force: true }); + return false; + } + for (const name of readdirSync(directory)) { + if (Number(name) < window) rmSync(join(directory, name), { force: true }); + } + return true; + }, + sleep: (ms, signal) => sleep(ms, undefined, { signal }), + }; +} + +/** + * Wait until the host has room for a command and this caller wins the current + * window. No fairness promise, adaptive scheduler, or admission timeout. + */ +export async function admit(limits: Limits, host: Host, signal: AbortSignal, waiting?: (reason: Waiting) => void) { + for (;;) { + signal.throwIfAborted(); + const now = host.now(); + const window = Math.floor(now / SETTLE_MS); + if (host.load() > limits.load) waiting?.("load"); + else if (host.memory() < limits.memory) waiting?.("memory"); + else if (host.claim(window)) return; + // Waiters wake together at the boundary, and exactly one claims the window. + await host.sleep((window + 1) * SETTLE_MS - now, signal); + } +} + +/** Marks an admitted command, so nested gated calls inside it run at once. */ +const ADMITTED = "QUEST_GATE_ADMITTED"; + +/** + * Run a command in place, with inherited streams and its exact exit status. + * Aborting with a signal name forwards that signal, so the command's own + * handling of it runs; any other reason sends SIGTERM. + */ +export function run(command: string[], signal: AbortSignal): Promise { + return new Promise((resolve, reject) => { + const child = spawn(command[0], command.slice(1), { + env: { ...process.env, [ADMITTED]: "1" }, + stdio: "inherit", + }); + const cancel = () => + child.kill( + typeof signal.reason === "string" && signal.reason in constants.signals + ? (signal.reason as NodeJS.Signals) + : "SIGTERM", + ); + signal.addEventListener("abort", cancel, { once: true }); + if (signal.aborted) cancel(); + child.once("error", (error) => { + signal.removeEventListener("abort", cancel); + reject(error); + }); + child.once("close", (code, killed) => { + signal.removeEventListener("abort", cancel); + resolve(code ?? (killed ? 128 + constants.signals[killed] : 1)); + }); + }); +} + +const notices: Record = { + load: "quest gate: waiting for the load average to fall", + memory: "quest gate: waiting for free memory", +}; + +/** Run from the CLI. */ +export async function gate(command: string[]): Promise { + const controller = new AbortController(); + let interrupted = 0; + const interrupt = (name: "SIGINT" | "SIGTERM" | "SIGHUP") => { + interrupted = 128 + constants.signals[name]; + controller.abort(name); + }; + const handlers = { + SIGINT: () => interrupt("SIGINT"), + SIGTERM: () => interrupt("SIGTERM"), + SIGHUP: () => interrupt("SIGHUP"), + }; + for (const [name, handler] of Object.entries(handlers)) process.on(name, handler); + try { + // A nested invocation executes inside the already admitted command. + if (process.env[ADMITTED] !== "1") { + const shown = new Set(); + await admit(hostLimits(), fileHost(claimDirectory()), controller.signal, (reason) => { + if (shown.has(reason)) return; + shown.add(reason); + process.stderr.write(`${notices[reason]}\n`); + }); + } + const result = await run(command, controller.signal); + return interrupted || result; + } catch (error) { + if (interrupted) return interrupted; + throw error; + } finally { + for (const [name, handler] of Object.entries(handlers)) process.off(name, handler); + } +} diff --git a/src/cli/main.ts b/src/cli/main.ts index 19e29605..17da850d 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -18,6 +18,7 @@ import { unblocked, } from "../core"; import { claims, questBranch } from "./claims"; +import { gate } from "./gate"; import { gitHubDependencies, GitHubInputError, pullRequest } from "./github"; import { landed } from "./landed"; import { type Report, moveQuest, removeQuest } from "./move"; @@ -69,6 +70,23 @@ export function main(argv: string[], out: Output): number | Promise { .exitOverride(); const root = () => program.opts<{ root: string }>().root; + program + .command("gate") + .summary("Run an expensive command once the host has room for it") + .description( + "Wait while the 1-minute load average exceeds the host's CPU count or available memory is below " + + "min(4 GiB, RAM/4), admitting one waiting command every 5 seconds. Nested calls run immediately. " + + "Limits come from the host; there is no configuration.", + ) + .argument("", "Executable to run") + .argument("[args...]", "Arguments forwarded verbatim") + .action((command: string, args: string[]) => { + pending = gate([command, ...args]).catch((error: unknown) => { + warn(error instanceof Error ? error.message : String(error)); + return 1; + }); + }); + program .command("check") .summary("Report every structural mistake in the tree; exit non-zero if any") @@ -458,7 +476,7 @@ export function main(argv: string[], out: Output): number | Promise { }); try { - program.parse(argv, { from: "user" }); + program.parse(gateBoundary(argv), { from: "user" }); return pending ?? code; } catch (error) { if (error instanceof CommanderError) { @@ -503,6 +521,24 @@ function positiveSeconds(value: string): number { return seconds; } +/** + * Stop option parsing before a gated executable, keeping its own flags intact. + * Quest's global options still apply on either side of `gate`. + */ +export function gateBoundary(argv: string[]): string[] { + const skipRoot = (index: number) => { + while (argv[index] === "--root" || argv[index]?.startsWith("--root=")) { + index += argv[index] === "--root" ? 2 : 1; + } + return index; + }; + const index = skipRoot(0); + if (argv[index] !== "gate") return argv; + const command = skipRoot(index + 1); + if (!argv[command] || ["--", "-h", "--help", "-V", "--version"].includes(argv[command])) return argv; + return [...argv.slice(0, command), "--", ...argv.slice(command)]; +} + /** Parses a positive, safe integer option named `name`. */ function positiveInteger(name: string): (value: string) => number { return (value) => { diff --git a/test/gate.test.ts b/test/gate.test.ts new file mode 100644 index 00000000..bc70af9a --- /dev/null +++ b/test/gate.test.ts @@ -0,0 +1,243 @@ +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { constants } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test, vi } from "vitest"; +import { + admit, + COMMAND_MEMORY, + claimDirectory, + fileHost, + gate, + type Host, + hostLimits, + linuxMemory, + run as runCommand, + SETTLE_MS, +} from "../src/cli/gate"; +import { gateBoundary, main } from "../src/cli/main"; +import { run, tempDir } from "./fixture"; + +const GiB = 2 ** 30; + +afterEach(() => vi.unstubAllEnvs()); + +test("limits derive from the host's CPUs and memory", () => { + // The host-wide load average is compared with every host CPU, not the + // process's affinity subset, so an idle assigned CPU is not held back. + expect(hostLimits(24, 64 * GiB)).toEqual({ load: 24, memory: COMMAND_MEMORY }); + // A small host still runs one command, and reserves a quarter of its RAM. + expect(hostLimits(4, 2 * GiB)).toEqual({ load: 4, memory: GiB / 2 }); + expect(hostLimits().load).toBeGreaterThanOrEqual(1); +}); + +test("load and memory pressure wait, then admit one caller per settle window", async () => { + const events: string[] = []; + const taken = new Set([4]); // Another waiter won the first window once the host had room. + let now = 2 * SETTLE_MS + 500; + let load = 9; + let memory = 0; + const host: Host = { + load: () => load, + memory: () => memory, + now: () => now, + claim: (window) => { + events.push(`claim ${window}`); + if (taken.has(window)) return false; + taken.add(window); + return true; + }, + sleep: async (ms) => { + events.push(`sleep ${ms}`); + now += ms; + if (load > 4) load = 3; + else memory = GiB; + }, + }; + const signal = new AbortController().signal; + await admit({ load: 4, memory: GiB }, host, signal, (reason) => events.push(reason)); + expect(events).toEqual([ + "load", + `sleep ${SETTLE_MS - 500}`, + "memory", + `sleep ${SETTLE_MS}`, + "claim 4", + `sleep ${SETTLE_MS}`, + "claim 5", + ]); +}); + +test("a window admits one claim and cleanup never admits an expired sample again", () => { + const directory = join(tempDir(), "claims"); + let now = 5 * SETTLE_MS; + const host = fileHost(directory, () => now); + expect(host.claim(5)).toBe(true); + expect(host.claim(5)).toBe(false); + now = 7 * SETTLE_MS; + expect(host.claim(7)).toBe(true); + expect(existsSync(join(directory, "5"))).toBe(false); + // A waiter sampled 5 before descheduling, then resumes after cleanup. + expect(host.claim(5)).toBe(false); + expect(existsSync(join(directory, "5"))).toBe(false); + expect(host.load()).toBeGreaterThanOrEqual(0); + expect(host.memory()).toBeGreaterThan(0); +}); + +test("a claim that crosses rollover during filesystem work does not admit", () => { + const directory = join(tempDir(), "claims"); + const successor = fileHost(directory, () => 6 * SETTLE_MS); + let calls = 0; + const delayed = fileHost(directory, () => { + if (++calls === 1) return 5 * SETTLE_MS; + // The pending claim may still admit, so the next window must wait for it. + expect(successor.claim(6)).toBe(false); + return 6 * SETTLE_MS; + }); + expect(delayed.claim(5)).toBe(false); + // The expired claim admitted nothing and is gone, so the next window is free. + expect(existsSync(join(directory, "5"))).toBe(false); + expect(successor.claim(6)).toBe(true); +}); + +test("admissions stay a full window apart across a window boundary", () => { + const directory = join(tempDir(), "claims"); + let now = 5 * SETTLE_MS - 1; + const host = fileHost(directory, () => now); + expect(host.claim(4)).toBe(true); + // A waiter at the next boundary, 1 ms later, must not start alongside it. + now = 5 * SETTLE_MS; + expect(host.claim(5)).toBe(false); + // The refused claim is removed: it admitted nothing, so it holds back no one. + expect(existsSync(join(directory, "5"))).toBe(false); + expect(host.claim(5)).toBe(false); + now = 6 * SETTLE_MS; + expect(host.claim(6)).toBe(true); + expect(existsSync(join(directory, "4"))).toBe(false); +}); + +test.each(["v1", "v2"])("Linux memory counts reclaimable cache and bounds %s container headroom", (version) => { + const proc = tempDir(); + const cgroups = tempDir(); + mkdirSync(join(proc, "self")); + writeFileSync( + join(proc, "meminfo"), + `MemTotal: ${(64 * GiB) / 1024} kB\nMemFree: 1024 kB\nMemAvailable: ${(24 * GiB) / 1024} kB\n`, + ); + const unified = version === "v2"; + writeFileSync(join(proc, "self/cgroup"), unified ? "0::/container/child\n" : "3:memory:/container/child\n"); + const root = unified ? cgroups : join(cgroups, "memory"); + const container = join(root, "container"); + const child = join(container, "child"); + mkdirSync(child, { recursive: true }); + const limit = unified ? "memory.max" : "memory.limit_in_bytes"; + const usage = unified ? "memory.current" : "memory.usage_in_bytes"; + const cache = unified ? "inactive_file" : "total_inactive_file"; + writeFileSync(join(child, limit), unified ? "max" : String(128 * GiB)); + writeFileSync(join(child, usage), String(GiB)); + expect(linuxMemory(proc, cgroups)).toEqual({ total: 64 * GiB, available: 24 * GiB }); + // The child is unlimited, but shares an almost-full 8 GiB parent budget. + writeFileSync(join(container, limit), String(8 * GiB)); + writeFileSync(join(container, usage), String(8 * GiB - GiB / 4)); + expect(linuxMemory(proc, cgroups)).toEqual({ total: 8 * GiB, available: GiB / 4 }); + // Inactive file cache inside the budget is reclaimable, so it counts as available. + writeFileSync(join(container, "memory.stat"), `active_file 1\n${cache} ${6 * GiB}\nanon 2\n`); + expect(linuxMemory(proc, cgroups)).toEqual({ total: 8 * GiB, available: 6 * GiB + GiB / 4 }); + writeFileSync(join(container, "memory.stat"), `${cache} 0\n`); + writeFileSync(join(container, usage), String(9 * GiB)); + expect(linuxMemory(proc, cgroups).available).toBe(0); +}); + +test("Linux memory reads the container cgroup mounted as the root", () => { + const proc = tempDir(); + const cgroups = tempDir(); + mkdirSync(join(proc, "self")); + writeFileSync(join(proc, "meminfo"), `MemTotal: ${(64 * GiB) / 1024} kB\nMemAvailable: ${(24 * GiB) / 1024} kB\n`); + writeFileSync(join(cgroups, "memory.max"), String(8 * GiB)); + writeFileSync(join(cgroups, "memory.current"), String(2 * GiB)); + const limited = { total: 8 * GiB, available: 6 * GiB }; + // With a cgroup namespace, the container's own cgroup is `/`. + writeFileSync(join(proc, "self/cgroup"), "0::/\n"); + expect(linuxMemory(proc, cgroups)).toEqual(limited); + // Without one, the host path is absent below the bind-mounted container cgroup. + writeFileSync(join(proc, "self/cgroup"), "0::/system.slice/docker-abc.scope\n"); + expect(linuxMemory(proc, cgroups)).toEqual(limited); + // A cgroup outside the namespace is not mounted, so nothing visible bounds it. + writeFileSync(join(proc, "self/cgroup"), "0::/../sibling\n"); + expect(linuxMemory(proc, cgroups)).toEqual({ total: 64 * GiB, available: 24 * GiB }); +}); + +test("admission claims stay on this host", () => { + expect(claimDirectory({ XDG_RUNTIME_DIR: "/run/user/1000" }, "/home/u", "a")).toBe( + join("/run/user/1000", "quest", "gate"), + ); + // A home directory shared between machines keeps each host's claims apart. + expect(claimDirectory({}, "/home/u", "a")).toBe(join("/home/u", ".cache", "quest", "gate", "a")); + expect(claimDirectory({}, "/home/u", "b")).not.toBe(claimDirectory({}, "/home/u", "a")); +}); + +test("cancelling a waiting caller rejects admission", async () => { + const controller = new AbortController(); + const busy: Host = { + load: () => 9, + memory: () => Infinity, + now: () => 0, + claim: () => true, + sleep: async () => controller.abort(new Error("cancelled")), + }; + await expect(admit({ load: 1, memory: 1 }, busy, controller.signal)).rejects.toThrow("cancelled"); +}); + +test("an admitted command runs in place and is marked nested", async () => { + const script = "process.exit(process.env.QUEST_GATE_ADMITTED === '1' ? 7 : 1)"; + expect(await runCommand([process.execPath, "-e", script], new AbortController().signal)).toBe(7); +}); + +test("a nested gate runs at once and keeps the exact exit status", async () => { + // A gated recipe that calls another one must not wait for admission twice. + vi.stubEnv("QUEST_GATE_ADMITTED", "1"); + expect(await gate([process.execPath, "-e", "process.exit(23)"])).toBe(23); +}); + +test("a command cancelled before listener registration still stops", async () => { + const controller = new AbortController(); + controller.abort(); + const result = await runCommand([process.execPath, "-e", "setInterval(() => {}, 1000)"], controller.signal); + expect(result).not.toBe(0); +}); + +// Windows has no POSIX signals to forward; kill() always terminates there. +test.skipIf(process.platform === "win32")("cancellation forwards the received signal to the command", async () => { + const controller = new AbortController(); + controller.abort("SIGINT"); + const result = await runCommand([process.execPath, "-e", "setInterval(() => {}, 1000)"], controller.signal); + expect(result).toBe(128 + constants.signals.SIGINT); +}); + +test("the gated command keeps its own flags, even ones Quest also parses", () => { + expect(gateBoundary(["gate", "bun", "--help", "--root", "x"])).toEqual([ + "gate", + "--", + "bun", + "--help", + "--root", + "x", + ]); + expect(gateBoundary(["--root", "r", "gate", "ls"])).toEqual(["--root", "r", "gate", "--", "ls"]); + expect(gateBoundary(["gate", "--help"])).toEqual(["gate", "--help"]); + expect(gateBoundary(["check"])).toEqual(["check"]); + // Quest's own global options may also follow the subcommand. + expect(gateBoundary(["gate", "--root", "r", "ls"])).toEqual(["gate", "--root", "r", "--", "ls"]); + expect(gateBoundary(["gate", "--root=r", "ls"])).toEqual(["gate", "--root=r", "--", "ls"]); + expect(gateBoundary(["gate", "--version"])).toEqual(["gate", "--version"]); +}); + +test("global options after gate keep working", async () => { + expect(run("gate", "--version").stdout).toMatch(/^quest /); + vi.stubEnv("QUEST_GATE_ADMITTED", "1"); + const out = { stdout: () => {}, stderr: () => {} }; + expect(await main(["gate", "--root", ".", process.execPath, "-e", "process.exit(5)"], out)).toBe(5); +}); + +test("gate has command help and requires an executable", () => { + expect(run("gate", "--help").stdout).toContain("Arguments forwarded verbatim"); + expect(run("gate").code).toBe(2); +});