Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,9 @@ Non-severe review findings deferred on purpose (see the review-round rule: only
## Vault hardening (M1-7)
- **Electron targets:** an app attached over CDP (F4) runs in its own persistent profile: no ephemeral context there. The app's own storage keeps whatever the scene typed; document it for Electron projects and prefer a throwaway app profile (`--user-data-dir` per run).
- **Terminals in a desktop app's demo (decided 2026-10-11):** an app its project lets open terminals (`terminals: true`, chosen when it's added, warned, approved with it) gets new ptys; the user's terminals open at launch are never reached (read, written, chmod, revoked). Stated gaps: a terminal the user opens while a launch lives (any app's) can be read (keystrokes, passwords), written and revoked by it, since Seatbelt can't tell its ptys from the user's: a pty broker (Kiframe opens them, the app gets descriptors) would close it. Closing a terminal that was open at launch can make the app's next pty fail (its number is reused, still denied). Helpers some apps run through a pty without terminals (askpass, some language servers) fail. An app the project already names keeps its project's setting: remove and add it again to change it. Its checks (adding it, a quiet trial of an update) run without terminals: an app that quits without a pty at start can't be added yet.
- **Socket paths in a desktop app's sandbox:** an app's socket in its data folder (VS Code's instance socket) must fit macOS's 104 characters; in `~/.kiframe` the path is 88 + the user's name long (VS Code 1.140). A long user name quits such an app at once ("quit at once": FAILURE-CATALOGUE #25). Shorter sandbox names (and the sweep reading both forms) would add room; a work area set by the user (KIFRAME_WORK_DIR) must stay short.
- **A desktop window smaller than its viewport: its take refused (design first, PR 5 2026-10-11):** VS Code's window (1200×800) is smaller than its viewport (1440×900): some screencast frames are of the window as it is (2700×1800), the take takes the first frame's size, TakeMeta refuses it ("frameSize must equal viewport × deviceScaleFactor"); going full screen worked around it. A frame gate (keep only the viewport at the scale asked) went through a redesign and six review rounds without settling: a larger window of the same shape is scaled into the asked size and passes; a take's size vs its first window; a popup's unread size; a display change mid-take. Next: a reviewed design, maybe at the root (the window sized to the viewport, which moves the user's window: decided against 2026-10-08, to weigh again). Its work: scratchpad patch `vscode-fixes-full.patch` (not in the repo).
- **The guard's way back from a site (design first, PR 5 2026-10-11):** a main window that went to a site goes back once; on CI the back failed ("Page.getNavigationHistory: Not attached to an active page", the page moving process) and the window was blanked (the embed test's flake). A retry loop driven by the window's address went through the same review rounds without settling: it stops on Chromium's error page or an about:blank entry (both "allowed"), a same-URL pushState reads as no history, a cut-off back may land late. Next: a reviewed design with the app's own check (`allows`) as its goal, never `frameAllowed`.
- **Copy and paste in a desktop app's demo:** the confinement denies the pasteboard (the user's clipboard may hold a password), so an app's own copy and paste does nothing in a run. A demo that needs it: a run-only pasteboard (a private named pasteboard can't be forced on an app) or typing the pasted text.
- **What a confined app opens through LaunchServices:** an app can't start without `launchservicesd`, and through it it can open a file or a URL that then runs unconfined (a script it wrote, the user's browser). Stated in §0.9 (accident, not a hostile app); an Endpoint Security watcher or a per-run user account would close it.
- **Desktop-app sweeps read processes more than once:** the startup sweep runs one `ps` per leftover sandbox (one `ps -axo pid=,lstart=` table would answer every owner), and a run's close reads the process table twice (`orphanedGroup` and `sweepProcesses`). Startup and close time only (#58 review).
Expand Down
4 changes: 4 additions & 0 deletions docs/FAILURE-CATALOGUE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,7 @@
| 22 | Content past the snapshot's cut | Wikipedia (desktop app, 2026-10-05) | "Search Messi, then go to his WC26 campaign": the article's snapshot is 1.14 M characters, the agent reads the first 20 k. It never saw the 2026 part: ~35 snapshots (25 failed `within` guesses: Wikipedia's sections have no names), the search page, the API, the printable page, until the 12 min ran out ($0.11). The user saw it as "scroll not working". With `find` (rerun, traced) it scrolled to the 2026 part in 3 min, then took a snapshot "to check where the page is now": a snapshot starts at the page's top whatever the scroll, and it thought 283 s about why nothing moved. Model time 704 s of 730 (tools 26 s) | — | Done: `snapshot` takes `find` (the elements that mention a text, with their ancestors and refs, searched in the scrubbed snapshot); the cut says to use it; every snapshot says where the view is (how far down, the section) and that it covers the whole page. The thinking itself: capped next (its own PR) |
| 23 | The recording fails where the replay passed | Wikipedia (desktop app, 2026-10-05/06) | "Search Messi": `save_scene`'s replay passed, `record_scene` failed with "locator.press: Timeout 15000ms exceeded" on the search step, every run (the agent worked around it, minutes each time). Typed at a person's pace, Wikipedia swaps its search box for a combobox mid-typing: the text still goes to the focused new field, but `submit` pressed Enter on the target's locator (a "Search Wikipedia" searchbox, gone). The replay typed at once, before the swap | — | Done: when the focused text field holds the typed text and the target is gone (or doesn't hold it: a modal took the keys), Enter goes where the text is; otherwise the target, as before. `save_scene` replays with the recording's typing and settling (pointer at once) and pixel ratio, so what a person's typing changes fails before the recording |
| 24 | A terminal app's shell refused by the confinement | minmux (desktop app, confined, 2026-10-11) | Every terminal pane said "posix_spawnp failed": Seatbelt denied /dev/ptmx, so no pty could be made; Kif spent the run on diagnostics. Measuring it found the old rule let any confined app print into every terminal the user had open (input injection refused by macOS) | Per app, opt-in (user decision): `terminals: true` allows new ptys and denies the user's terminals open at launch (every way: read, write, chmod, ioctl, revoke); every other app gets no terminal at all | Done (stated gap: terminals opened while it runs, BACKLOG) |
| 25 | An app quits at once, silently, confined | VS Code (desktop app, confined, 2026-10-11) | "Code quit at once": exit 1, nothing said (its `--verbose`: `Lifecycle#kill()` right after start). Its instance socket lives in its data folder, inside the sandbox; under the driver's work folder that path was ~170 characters, past macOS's 104 for a socket. Found with the launcher's debug output (KIFRAME_ELECTRON_DEBUG: the app's own output and exit code) and VS Code's `--verbose` | The driver uses a short work folder (`/tmp/kf-…`); in `~/.kiframe` the path is 88 + the user's name long (95 here) | Stated limit (BACKLOG): a user name over ~16 characters, or an app with a longer socket name, quits the same way |
| 26 | A desktop app's take refused: its frames of two shapes | VS Code (desktop app, confined, 2026-10-11) | Every `record_scene`: "frameSize must equal viewport × deviceScaleFactor". Its window (1200×800) is smaller than the viewport it's shown at (1440×900): some screencast frames are of the window as it is (2700×1800), and the take took the first one's size. Kif got a take only by going full screen (frames still changed mid-take) | Going full screen (Kif found it) | Open: design first (BACKLOG) |
| 27 | A shell's temp files refused | VS Code's terminal (confined, 2026-10-11) | Its zsh integration said `.zshrc:158: can't create temp file` on camera (zsh writes here-docs under /tmp/zsh: denied), in a path naming the user's login | Every desktop app's `TMPPREFIX` in its sandbox temp (VS Code reads its shell at start, terminals or not) | Done |
| 28 | Grounding slow in an editor | VS Code (desktop app, 2026-10-11) | 15.5 min and $0.18 for one scene: Ctrl+` didn't open the terminal, the command palette's input timed out on focus (in the replay too), a first-run dialog and the folder's trust dialog each took steps; the scene opens the terminal from the panel toggle | — | Open: prompt hints for editors (dismiss first-run dialogs in setup; prefer visible toggles to shortcuts) |
21 changes: 17 additions & 4 deletions packages/runtime/src/electron.ts
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,8 @@ function spawnOptions(sandbox: { root: string; env: NodeJS.ProcessEnv }): SpawnO
return {
detached: true,
env: sandbox.env,
stdio: ["ignore", "ignore", "pipe"],
// Its standard output read only for Kiframe's own debugging.
stdio: ["ignore", process.env.KIFRAME_ELECTRON_DEBUG === undefined ? "ignore" : "pipe", "pipe"],
cwd: sandbox.root,
}
}
Expand Down Expand Up @@ -356,6 +357,9 @@ async function makeSandbox(work: string): Promise<{
// Chromium on macOS: its temp folder (the single-instance lock's socket) in the sandbox too,
// never the user's /var/folders (denied by the confinement: such an app would quit).
MAC_CHROMIUM_TMPDIR: dirs.tmp,
// zsh's temp files (here-docs: VS Code's shell, its environment read at start) in its own
// temp, never /tmp (denied).
TMPPREFIX: join(dirs.tmp, "zsh"),
})
return {
root,
Expand Down Expand Up @@ -495,6 +499,14 @@ export async function launchElectronWith(
// Its spawn failing is said by the launch (never an unhandled 'error' event).
let spawnError: Error | undefined
child.on("error", (error) => (spawnError = error))
// What it says and how it ends: Kiframe's own debugging only (its output may name paths).
if (process.env.KIFRAME_ELECTRON_DEBUG !== undefined) {
child.stderr?.on("data", (chunk: Buffer) => process.stderr.write(`[app] ${chunk.toString()}`))
child.stdout?.on("data", (chunk: Buffer) =>
process.stderr.write(`[app out] ${chunk.toString()}`),
)
child.on("exit", (code, signalName) => debug("exit", `${code ?? signalName}`))
}
let browser: Browser | undefined
let ended: Promise<{ unread: boolean }> | undefined
// Once: the app's group is signalled only while it's known alive (its leader running, or its
Expand Down Expand Up @@ -720,21 +732,22 @@ function debuggingEndpoint(
const finish = (error?: Error, endpoint?: string) => {
clearTimeout(timer)
clearInterval(poll)
child.stderr?.removeAllListeners("data")
child.stderr?.removeListener("data", onData)
child.removeListener("exit", exited)
signal?.removeEventListener("abort", aborted)
if (error !== undefined) reject(error)
else resolve(endpoint as string)
}
child.stderr?.on("data", (chunk: Buffer) => {
const onData = (chunk: Buffer) => {
output = (output + chunk.toString("utf8")).slice(-8192)
// A whole line only (a chunk may end mid-address), on this machine only.
const found =
/DevTools listening on (ws:\/\/127\.0\.0\.1:\d+\/devtools\/browser\/[0-9a-f-]+)\r?\n/.exec(
output,
)?.[1]
if (found !== undefined) finish(undefined, found)
})
}
child.stderr?.on("data", onData)
// The profile's own file, inside the sandbox (never one elsewhere: an app that ignores
// --user-data-dir would write it where Kiframe never reads), read whole.
const poll = setInterval(() => {
Expand Down
2 changes: 2 additions & 0 deletions packages/runtime/test/electron.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1289,6 +1289,8 @@ steps:
expect(seen["pty-shell"], `terminals: ${terminals}`).toMatch(
terminals ? /^OK true/ : /^DENIED/,
)
// A shell's temp files in its own temp (every app's: VS Code reads its shell at start).
expect(seen["zsh-temp"], `terminals: ${terminals}`).toBe("OK here-ok")
// Its shells' prompts: the folder only, never the user's login or Mac's name.
if (terminals) {
expect(seen["zsh-prompt"]).toMatch(/^OK \S+ %$/)
Expand Down
2 changes: 2 additions & 0 deletions packages/runtime/test/fixtures/electron-app/main.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,8 @@ if (process.argv.includes("terminals")) {
.trim()
tryIt("zsh-prompt", () => shell("/bin/zsh", 'print -rP -- "$PS1"'))
tryIt("bash-prompt", () => shell("/bin/bash", 'echo "$PS1"'))
// A here-string (a temp file of zsh's: what VS Code's shell integration writes).
tryIt("zsh-temp", () => shell("/bin/zsh", "cat <<< here-ok"))
const data = app.getPath("userData")
fs.writeFileSync(join(data, "terminals.json"), JSON.stringify(out))
const asked = join(data, "later.txt")
Expand Down
14 changes: 10 additions & 4 deletions scripts/real-apps/drive.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import {
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs"
import { tmpdir } from "node:os"
Expand Down Expand Up @@ -282,7 +283,8 @@ async function addDesktop(spec: string, dir: string): Promise<void> {
await adding.getByRole("button", { name: "Check" }).click()
const status = adding.getByRole("status")
await page.waitForFunction(
(el) => /runs confined|couldn|can't|isn't|refused|failed/i.test(el?.textContent ?? ""),
(el) =>
/runs confined|couldn|can.t|isn.t|refused|failed|quit|the site/i.test(el?.textContent ?? ""),
await status.elementHandle(),
{ timeout: 120_000 },
)
Expand Down Expand Up @@ -341,9 +343,13 @@ const env = Object.fromEntries(
env.KIFRAME_TEST_KEYCHAIN = "memory"
// The agent's every event, timed, next to the report (what it called, with what, how long).
env.KIFRAME_TRACE = join(out, "trace.jsonl")
// A desktop app's sandboxes in the run's own work area (never the user's ~/.kiframe); its launch's
// timings and failures' details on main's stderr, kept next to the report.
env.KIFRAME_WORK_DIR = join(out, "work")
// A short path (never the user's ~/.kiframe): an app's socket in its sandbox must fit macOS's 104
// characters (VS Code's quit at once under the run's own folder).
const work = mkdtempSync("/tmp/kf-")
env.KIFRAME_WORK_DIR = work
process.on("exit", () => rmSync(work, { recursive: true, force: true }))
// A desktop app's launch timings and failures' details (its own output too) on main's stderr
// (main.log, next to the report).
env.KIFRAME_ELECTRON_TIMING = "1"
env.KIFRAME_ELECTRON_DEBUG = "1"
const profile = mkdtempSync(join(tmpdir(), "kiframe-real-"))
Expand Down
Loading