diff --git a/CONTEXT.md b/CONTEXT.md index 55200fbf..4b162204 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -31,7 +31,7 @@ The message thirdshift starts or resumes an agent session with. It carries the * _Avoid_: instructions, system prompt **Harness**: -The headless agent CLI that runs every agent session of a **Command**, named as its CLI is: Claude Code (`claude`), Codex (`codex`), Antigravity CLI (`agy`), Grok Build (`grok`), Muse Code (`muse`) or OpenCode (`opencode`). One per Command, its child Runs and every **Resume** and **Repair** included. Chosen by the command, else the **User config**, else Claude Code. +The headless agent CLI that runs every agent session of a **Command**, named as its CLI is: Claude Code (`claude`), Codex (`codex`), Antigravity CLI (`agy`), Grok Build (`grok`), Muse Code (`muse`) or OpenCode (`opencode`). One per Command, its child Runs and every **Resume** and **Repair** included. Chosen by the command, else, for a **Security run**, the User config's `security.harness`, else the **User config**'s default Harness, else Claude Code. _Avoid_: agent, provider, backend **Model**: @@ -69,7 +69,7 @@ One invocation of the factory on a single issue that is not a **Spec**, from lau A factory-owned checkout, on an **Issue branch** for a **Run** or detached for an **Architecture review**. Acquisition accounts for partial local effects: failed-acquisition recovery rechecks clean-only, instance-specific authority before and after every destructive attempt, including retries, advances only on verified transitions, and retains and names remaining work or uncertain partial effects. Failed-acquisition retention survives later Architecture review attempts. Successful acquisition carries checkout and registration identity through its lifetime; ordinary synchronization, observations and Failed run preservation share acquired-instance checks before and after every Git subprocess attempt, including retries, acting only on that acquired checkout and selected Issue branch and refusing changed or uncertain ownership. Failed run preservation retains work when ownership has changed or is uncertain and finishes after Command interruption without clearing it; ordinary synchronization remains interruptible. Confirmed Self-merge branch deletion uses the captured Launch repository independently of checkout validity. Cleanup checks stage-specific disposal authority before and after every destructive Git attempt, including retries: acquired identity and the captured cleanup head for checkout removal, removed-path absence and an unused expected head for local branch removal, then ref absence and the exact original local subsection for branch configuration removal. Only successful, verified transitions advance disposal; partial failures retain and name every remaining or uncertain resource. Live cleanup finishes after Command interruption without clearing it; stale disposal stays interruptible. Stale Architecture review scratch is disposable only with evidence of successful detached acquisition and unchanged ownership; unmarked or uncertain checkouts are retained. **Claim**: -The mark that the factory has taken an issue: thirdshift labels the issue `in-progress`, in place of `ready-for-agent` if it has that, when a **Run** or a **Spec run** starts on it, whether started on its **Issue URL** or by an **Architect run**, a **Pickup run** or a **Security run**. A **Ticket**'s Run in a Spec run and a **Base fix** make none. Ending a Claim finishes even after Command interruption. A Claim is released, `ready-for-agent` put back, only when the Run ends with nothing on `origin` to take over: no **Issue branch** and no pull request. Otherwise it stays while the issue is open, whether its pull request is ready for review or the Run failed, until the **Day shift** relabels it, and thirdshift takes the label off once the issue is closed. +The mark that the factory has taken an issue: thirdshift labels the issue `in-progress`, in place of `ready-for-agent` if it has that, when a **Run** or a **Spec run** starts on it, whether started on its **Issue URL** or by an **Architect run**, a **Pickup run** or a **Security run**. A **Ticket**'s Run in a Spec run and a **Base fix** make none. Ending a Claim finishes even after Command interruption. A failed Security fix keeps its Claim even with nothing on `origin`, so it stays with the Day shift. For other Runs, a Claim is released, `ready-for-agent` put back, only when the Run ends with nothing on `origin` to take over: no **Issue branch** and no pull request. Otherwise it stays while the issue is open, whether its pull request is ready for review or the Run failed, until the **Day shift** relabels it, and thirdshift takes the label off once the issue is closed. _Avoid_: lock (a lock is per machine and ends with its process), assignment **Spec run**: @@ -109,7 +109,7 @@ A **Pickup run**'s removal of `in-progress` from every closed issue in the repos _Avoid_: cleanup, garbage collection **Ready issue**: -An open issue a **Pickup run** may take: labelled `ready-for-agent`, with no label that makes an **Unready Ticket** and no **Claim**, not a sub-issue, not a **Base fix**'s issue, with no open blocker, never started (no **Issue branch** and no pull request), not a **Spec** whose **Tickets** are all closed, and left untouched long enough that whoever is shaping it has finished. On a Spec, the label says its Tickets are published. A `ready-for-agent` Ticket inside a Spec is not one: it is reached through its Spec, when the Spec is itself a Ready issue. +An open issue a **Pickup run** may take: labelled `ready-for-agent`, with no label that makes an **Unready Ticket** and no **Claim**, not a sub-issue, not a **Base fix**'s or a **Security run**'s fix issue, with no open blocker, never started (no **Issue branch** and no pull request), not a **Spec** whose **Tickets** are all closed, and left untouched long enough that whoever is shaping it has finished. On a Spec, the label says its Tickets are published. A `ready-for-agent` Ticket inside a Spec is not one: it is reached through its Spec, when the Spec is itself a Ready issue. _Avoid_: queued issue, backlog item **Architecture review**: @@ -156,7 +156,7 @@ A **Run** asked to end with its pull request merged rather than left for review, _Avoid_: auto-merge (GitHub's own feature, which thirdshift does not use) **Run notification**: -A message thirdshift sends when a **Run**, a **Spec run**, an **Architect run** or a **Security run** ends, whatever its outcome (ready, merged, failed or interrupted; for an Architect run that dispatched nothing, plan published, idea filed, idea already filed or review failed), to the address given with the email flag or the default in the **User config**. Each sends one only when asked to, by the flag or by the User config; a Spec run's notification lists each **Ticket**'s outcome, and a Ticket's **Run** never sends one of its own. After an **Inherited failure** it carries, beside the cause, what the Run says on stderr: where the checks fail on the **Base branch**, and the offer of a **Base fix** if nobody decided against one. An Architect run's notification tells how its **Architecture review** ended, naming the plan or idea issue, and how the Spec run or Run it dispatched ended, which sends none of its own; a skipped one sends none. A **Pickup run** that took a **Ready issue** sends the one the Spec run or Run it dispatched would have sent, which sends none of its own; a skipped one sends none. A **Security run**'s notification lists each **Security finding**'s severity, title and private link, never its write-up, since the message passes through a third party; a skipped one sends none. Failing to send one never changes the outcome. +A message thirdshift sends when a **Run**, a **Spec run**, an **Architect run** or a **Security run** ends, whatever its outcome (ready, merged, failed or interrupted; for an Architect run that dispatched nothing, plan published, idea filed, idea already filed or review failed), to the address given with the email flag or the default in the **User config**. Each sends one only when asked to, by the flag or by the User config; a Spec run's notification lists each **Ticket**'s outcome, and a Ticket's **Run** never sends one of its own. After an **Inherited failure** it carries, beside the cause, what the Run says on stderr: where the checks fail on the **Base branch**, and the offer of a **Base fix** if nobody decided against one. An Architect run's notification tells how its **Architecture review** ended, naming the plan or idea issue, and how the Spec run or Run it dispatched ended, which sends none of its own; a skipped one sends none. A **Pickup run** that took a **Ready issue** sends the one the Spec run or Run it dispatched would have sent, which sends none of its own; a skipped one sends none. A **Security run**'s notification lists each **Security finding**'s severity, title and private link, never its write-up, since the message passes through a third party. When a Run leaves a reproduced finding unfixed and nobody decided against fixing, it offers both the command and the User config setting that allow fixing; a skipped one sends none. Failing to send one never changes the outcome. _Avoid_: completion email, alert **User config**: diff --git a/README.md b/README.md index dbb00155..815ffa72 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ The **Claim** is the mark that the factory has taken an issue, so the issue list How the Run or the Spec run ends decides what becomes of its Claim: -- **Released**, when it fails with nothing on `origin` to take over: no Issue branch for the issue there and no pull request from one, open, merged or closed, or for a Spec run, no Spec branch and no Spec PR. That is a session that fails before it changes anything, as on a usage limit or an expired login, a Ctrl-C before any work, or a worktree that can't be created, on an issue that was never started before. A retry that fails that way on an issue an earlier Run left an Issue branch or a pull request for keeps its Claim. The issue's labels go back as they were before the Claim: `in-progress` comes off, and `ready-for-agent` goes back if the issue had it, and only then. So an outage costs a retry and doesn't strand the issue. stderr says `releasing the Claim on #: labelling it ready-for-agent, in place of in-progress`, or `releasing the Claim on #: removing in-progress`. A Failed run whose push failed is released too, since nothing of it reached `origin`: its work is only in the worktree it kept, which stops the next Run on that machine at the pre-flight checks until you push or delete the local Issue branch. +- **Released**, except for a failed Security fix, when it fails with nothing on `origin` to take over: no Issue branch for the issue there and no pull request from one, open, merged or closed, or for a Spec run, no Spec branch and no Spec PR. That is a session that fails before it changes anything, as on a usage limit or an expired login, a Ctrl-C before any work, or a worktree that can't be created, on an issue that was never started before. A retry that fails that way on an issue an earlier Run left an Issue branch or a pull request for keeps its Claim. The issue's labels go back as they were before the Claim: `in-progress` comes off, and `ready-for-agent` goes back if the issue had it, and only then. So an outage costs a retry and doesn't strand the issue. stderr says `releasing the Claim on #: labelling it ready-for-agent, in place of in-progress`, or `releasing the Claim on #: removing in-progress`. A Failed run whose push failed is released too, since nothing of it reached `origin`: its work is only in the worktree it kept, which stops the next Run on that machine at the pre-flight checks until you push or delete the local Issue branch. - **Kept**, on any other ending that leaves the issue open: a pull request ready for review, or a [Failed run](#failed-runs) that pushed its Issue branch or left a pull request. The issue stays `in-progress` until you relabel it, so a failure waits for you. - **Removed**, in a Merge run, once the Self-merge has left the issue closed, whether thirdshift closed it or GitHub did: `in-progress` comes off, after `removing in-progress from issue #` on stderr. @@ -149,7 +149,7 @@ thirdshift --email you@example.com https://github.com/acme/widgets/issues/7 It sends one [Run notification](#run-notifications), whatever the outcome. To have every Run on a machine send one, set `email.always` in the [User config](#user-config); `no-email` (or `--no-email`) then skips it for one Run. - **stdout** carries only the pull request's URL: on success, and on a Failed run that leaves an open pull request, a draft or, after a policy refusal, one ready for review. The exit code tells the two apart, so script it as `url=$(thirdshift "$issue") && echo "ready: $url"`. -- **stderr** carries everything else: errors, cleanup problems, and progress lines while sessions run. Each line starts with the local time it was printed, as in `thirdshift: 12:14:49 pushing issue-7`, so a quiet terminal shows how long the Run has been on its last step. The first line of a Run, a Spec run, an [Architect run](#architect-runs) or a [Pickup run](#pickup-runs) also names the date and the UTC offset, as in `thirdshift: 12:00:01 starting on https://github.com/acme/widgets/issues/7, 2026-10-03 -0400`, so a log that collects many commands, such as a crontab line's, reads as a dated list. Everything on stderr and stdout is kept in the command's [Command log](#logs) too. A successful Run's last line names the pull request too: `PR is ready for review`, or `PR is merged` after a Merge run, followed only by a `warning:` line if a [Run notification](#run-notifications) can't be sent. +- **stderr** carries everything else: errors, cleanup problems, and progress lines while sessions run. Each line starts with the local time it was printed, as in `thirdshift: 12:14:49 pushing issue-7`, so a quiet terminal shows how long the Run has been on its last step. The first line of a Run, a Spec run or a Pass also names the date and the UTC offset, as in `thirdshift: 12:00:01 starting on https://github.com/acme/widgets/issues/7, 2026-10-03 -0400`, so a log that collects many commands, such as a crontab line's, reads as a dated list. Everything on stderr and stdout is kept in the command's [Command log](#logs) too. A successful Run's last line names the pull request too: `PR is ready for review`, or `PR is merged` after a Merge run, followed only by a `warning:` line if a [Run notification](#run-notifications) can't be sent. - **Exit code** `0` means the Run ended with a pull request the factory stands behind, merged in a Merge run. Once the Self-merge has merged, the Run succeeds even if deleting the Issue branch on `origin`, closing the issue or removing `in-progress` from it then fails: the merge can't be undone, so each failed step is a `warning:` line on stderr naming the command to run by hand, and the Run still exits `0` with the URL on stdout. Ctrl-C likewise: before the merge it makes a Failed run, after it thirdshift finishes these steps and exits as merged. `2` means the command is one thirdshift can't use: the Issue URL is missing or isn't a GitHub Issue URL, there is an argument other than the URL and the Run flags, a Run flag is repeated or contradicts another, or `parallel` isn't followed by a whole number from 1 up; the error and the help text go to stderr, before any work. A [User config](#user-config) thirdshift can't use exits `1`, also before any work. Any other failure exits `1`. The other commands: @@ -194,7 +194,7 @@ from = "thirdshift@your-verified-domain.com" # the sender; onboarding@resend.de dir = "~/elsewhere/logs" # the root of the logs, instead of ~/.thirdshift/logs [activity] -quiet_skips = true # a skipped Architect run or Pickup run prints nothing +quiet_skips = true # a skipped Pass prints nothing [spec] parallel = 2 # how many Tickets a Spec run runs at once, instead of 3 @@ -203,6 +203,10 @@ parallel = 2 # how many Tickets a Spec run runs at once, instead of 3 limit = 5 # how many open issues labelled in-progress stop a Pickup run taking another, instead of 3 wait_minutes = 60 # minutes since the latest shaping event before an issue is Ready, instead of 30; 0 disables waiting +[security] +review = true # review each Run's change for Security findings +harness = "claude" # the Harness for Security runs; blank or missing follows harness.default + [harness] default = "claude" # the Harness every session runs on; claude unless set @@ -240,7 +244,7 @@ from = "onboarding@resend.dev" # the sender; default onboarding@resend.dev, whi dir = "~/.thirdshift/logs" # the root of the logs, each repository's in //; default ~/.thirdshift/logs [activity] -quiet_skips = false # a skipped Architect run or Pickup run prints nothing, leaving only its Activity log line; default false +quiet_skips = false # a skipped Pass prints nothing, leaving only its Activity log line; default false [spec] parallel = 3 # how many Tickets a Spec run runs at once; default 3 @@ -249,6 +253,11 @@ parallel = 3 # how many Tickets a Spec run runs at once; default 3 limit = 3 # how many open issues labelled in-progress stop a Pickup run taking another; default 3 wait_minutes = 30 # minutes since the latest shaping event before an issue is Ready; 0 disables waiting; default 30 +[security] +fix = false # Security runs may fix reproduced findings; default false +review = false # Runs review their change for Security findings; default false +harness = "" # the Harness a Security run uses unless its command names one; default blank, for harness.default or Claude Code + [harness] default = "claude" # the Harness every Run's sessions run on, claude, codex, agy, grok, muse or opencode; default claude @@ -283,10 +292,12 @@ Every key holds its real value, so a Run reading it does exactly what it does wi 2. Every Run a Merge run? (`merge.always`) 3. With that on, every Run may start a Base fix when the Base branch's CI is red? (`base.fix`). With it off, this is not asked, and `base.fix` is written at its default, `false`. 4. Every Run first fast-forwards your checkout of the Base branch? (`launch.pull`) -5. Run notifications? (`email.always`). If yes, the address (`email.to`), asked again until it has an `@`, then the sender (`email.from`). -6. With notifications on, the Resend API key, with input hidden. With none saved it asks `Resend API key (input hidden, Enter to skip):`; with one in the [Credentials](#email), `Resend API key (input hidden, Enter keeps the saved one):`, and a new one replaces it. Surrounding spaces are trimmed, and anything that doesn't start with `re_` is asked again. A key you give is saved in the Credentials, `~/.thirdshift/credentials.toml`, created with mode 600 (and `~/.thirdshift` with it) or edited in place, keeping its comments and anything else in it and changing only `resend.key`; `setup` then prints `wrote the Credentials `. The key is never printed, nor written to the User config. Skipping writes no Credentials and says how to add a key later: rerun `thirdshift setup`, or set `RESEND_API_KEY`. With `RESEND_API_KEY` set and not empty, which wins over the Credentials, nothing is asked, and it says the key comes from `RESEND_API_KEY`. With a key found or given, it offers to send a test email (default No), as `thirdshift email-test` does, once the files are written. +5. Security runs may fix reproduced findings? (`security.fix`), off by default. +6. Runs review their change for Security findings? (`security.review`), off by default. +7. Run notifications? (`email.always`). If yes, the address (`email.to`), asked again until it has an `@`, then the sender (`email.from`). +8. With notifications on, the Resend API key, with input hidden. With none saved it asks `Resend API key (input hidden, Enter to skip):`; with one in the [Credentials](#email), `Resend API key (input hidden, Enter keeps the saved one):`, and a new one replaces it. Surrounding spaces are trimmed, and anything that doesn't start with `re_` is asked again. A key you give is saved in the Credentials, `~/.thirdshift/credentials.toml`, created with mode 600 (and `~/.thirdshift` with it) or edited in place, keeping its comments and anything else in it and changing only `resend.key`; `setup` then prints `wrote the Credentials `. The key is never printed, nor written to the User config. Skipping writes no Credentials and says how to add a key later: rerun `thirdshift setup`, or set `RESEND_API_KEY`. With `RESEND_API_KEY` set and not empty, which wins over the Credentials, nothing is asked, and it says the key comes from `RESEND_API_KEY`. With a key found or given, it offers to send a test email (default No), as `thirdshift email-test` does, once the files are written. -Pressing Enter takes the default shown, which is the file's current value, or else the setting's default, and for the address the suggested email above. `logs.dir`, `activity.quiet_skips`, `spec.parallel` and `pickup.limit` are not asked about. The answers are written like everything else below: in place, keeping your comments. Ctrl-C, SIGTERM or SIGHUP during the questions or a Model/Effort check exits `1` with `interrupted` and writes nothing: neither the User config nor the Credentials. Ordinary and hidden questions stop without another line of input, and hidden input restores the saved terminal settings, including echo. With notifications off, nothing about a key is asked, and saved Credentials stay as they were, so `--email` on a single Run still works. Credentials a Run would refuse (see [Email](#email)) are refused before any question, exit `1`, and not touched. With no terminal, as from cron or `thirdshift setup . Set your defaults now? [Y/n]`. Yes (or Enter) asks the questions above, writes the file, and the Run carries on using your answers; a flag in the command, such as `--no-merge`, `--email` or `--no-email`, still wins over them. No writes every setting at its default, as `setup` with no terminal does, asks nothing about a key, writes no Credentials, says that `thirdshift setup` changes it, and the Run carries on; later Runs find the file and don't offer again. If the file can't be written, stderr gets a `warning:` line and the Run carries on with the defaults. Ctrl-C, SIGTERM or SIGHUP during the offer, the questions or a Model/Effort check exits `1` with `interrupted`, writes nothing and ends the command before any work, with no Run notification. A command thirdshift can't parse exits `2` before any offer. A Run with no terminal, as from cron, CI, `nohup` or an agent's shell, offers nothing, writes nothing, and runs on the defaults, so a later Run from a terminal still gets the offer. @@ -334,7 +345,7 @@ Grok reads both `AGENTS.md` and `CLAUDE.md` when both exist, along with `AGENT.m Interrupting a Command stops every session's process tree on every Harness, including commands that started their own process group or session. The same owner supervises all Model/Effort checks, from Command startup and terminal Setup: interruption stops the check and its owned commands, even after the CLI exits while its captured output pipes are still open. A cancelled Command check ends before any Claim, worktree or Command log. thirdshift snapshots the descendants with `ps` on Linux and macOS, sends each process group the Harness's stop signals with a 10-second grace period each, then takes another snapshot before SIGKILL to catch commands started in the meantime. If the CLI exits on an earlier signal, thirdshift kills the remaining commands immediately without another grace period. -Before any work, where an Origin match failure would stop it, and so before the Claim, the worktree and any Command log, a command checks what it chose: the Harness's CLI must be on `PATH`, else it fails naming the CLI and the setting that chose it; and on Claude a named Model gets a minimal test call, `claude -p --model ` with the Effort, if any, which must succeed, else the command fails with what Claude said. No test call is made when no Model is named. On Codex a named Model and Effort are checked against `codex debug models`, which costs no tokens: each matches regardless of case, a Model by its slug or its display name, and becomes the name Codex takes, so `model GPT-6.1-Sol effort Max` reaches Codex as `gpt-6.1-sol` and `max`. An unknown Model, or an Effort that Model doesn't support (with no Model, one no Model supports), fails naming the valid choices. On Grok Build, `grok models` lists the available Models without a model call and refreshes `models_cache.json` under `GROK_HOME` or `~/.grok`; thirdshift reads that cache for each Model's supported Efforts. Both are matched regardless of case and passed as Grok names them. With no Model named, a named Effort is checked against Grok's current default Model. Unknown Models and unsupported Efforts fail with the valid choices, and an unreadable catalog fails before work. The check has the same null stdin and fixed environment as sessions. What the command records is what the check settled on. A Pickup run or an Architect run makes these checks only once it has decided not to skip, and a Spec run makes them once for all its Tickets. +Before any work, where an Origin match failure would stop it, and so before the Claim, the worktree and any Command log, a command checks what it chose: the Harness's CLI must be on `PATH`, else it fails naming the CLI and the setting that chose it; and on Claude a named Model gets a minimal test call, `claude -p --model ` with the Effort, if any, which must succeed, else the command fails with what Claude said. No test call is made when no Model is named. On Codex a named Model and Effort are checked against `codex debug models`, which costs no tokens: each matches regardless of case, a Model by its slug or its display name, and becomes the name Codex takes, so `model GPT-6.1-Sol effort Max` reaches Codex as `gpt-6.1-sol` and `max`. An unknown Model, or an Effort that Model doesn't support (with no Model, one no Model supports), fails naming the valid choices. On Grok Build, `grok models` lists the available Models without a model call and refreshes `models_cache.json` under `GROK_HOME` or `~/.grok`; thirdshift reads that cache for each Model's supported Efforts. Both are matched regardless of case and passed as Grok names them. With no Model named, a named Effort is checked against Grok's current default Model. Unknown Models and unsupported Efforts fail with the valid choices, and an unreadable catalog fails before work. The check has the same null stdin and fixed environment as sessions. What the command records is what the check settled on. A Pass makes these checks only once it has decided not to skip, and a Spec run makes them once for all its Tickets. With `harness muse`, every session and Resume runs `muse exec --json --yolo`, with stdin null and `MUSE_NO_AUTO_UPDATE=1`. `--yolo` disables approvals and the sandbox and trusts the worktree ([ADR-0013](docs/adr/0013-every-harness-runs-unattended-and-fully-trusted.md)). Muse reads the project's `AGENTS.md`, falling back to `CLAUDE.md`, and also reads `~/.claude/CLAUDE.md`. The Factory skills are linked into `.agents/skills/` and kept out of git. The prompt's first line tells the model to load its `thirdshift-` with its skill tool. If the stream shows no `read_skill` load of that skill, thirdshift writes a warning to the progress lines and Command log; it neither fails nor retries the session. @@ -383,7 +394,7 @@ no Resend API key. Either: (a crontab line, CI secret, or a shell profile the Run's shell reads) ``` -The Credentials are read only when `RESEND_API_KEY` is unset or empty, and only by `email-test`, `setup`, and a Run, an Architect run or a Pickup run that asks for a notification. A missing file just means no key from it. One that isn't valid TOML, or holds anything but a quoted `resend.key`, such as a typo like `kye`, stops the command with exit `1`, naming the file and the offending key, and nothing is sent. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer. +The Credentials are read only when `RESEND_API_KEY` is unset or empty, and only by `email-test`, `setup`, and a Run or a Pass that asks for a notification. A missing file just means no key from it. One that isn't valid TOML, or holds anything but a quoted `resend.key`, such as a typo like `kye`, stops the command with exit `1`, naming the file and the offending key, and nothing is sent. One that others can read is still used, with a `warning:` line on stderr saying to `chmod 600` it. Nothing is sent to check the key itself. When Resend accepts the email, it prints `accepted by Resend; check your inbox` and exits `0`; that is all it can verify, so check that the email arrives. When Resend refuses it, for example for a bad key or a sender it won't send from, it prints Resend's error text word for word, with where the key came from, and exits `1`. It gives up after 30 seconds without an answer. #### Run notifications @@ -426,11 +437,11 @@ The folder already names the repository, so no file name repeats it. A **Session A Resume is logged as its session's kind plus `-resume`, e.g. `implement-resume.jsonl`. -A **Command log** is everything one command printed, stderr and stdout in the order printed, while the terminal still shows all of it. Its folder is the command typed: `thirdshift `, a Run or a Spec run, in `issue/`, `thirdshift pickup` in `pickup/`, named for the issue it took, and `thirdshift architect` in `architect/`. A Spec run's covers its Tickets' Runs, a Run's covers its [Base fix](#base-fix), and a Pickup run's or an Architect run's covers the Spec run or Run it dispatched: none of those keeps one of its own. So `ls -t ~/.thirdshift/logs/acme/widgets/commands/pickup | head` lists the recent passes on acme/widgets that took an issue. A command keeps its Command log once it starts work: a Pickup run or an Architect run skipped before doing any work keeps none, nor does a Run that fails before it starts work, as on the [Origin match](#what-a-run-does), and `setup`, `email-test`, `update`, `version` and `help` never keep one. Lines printed before the Command log's name is known, such as a Pickup run's lines on the issues it [passed over](#why-an-issue-was-passed-over) before it took one, are written first, and once it is created a progress line says `logging this command to `. A Command log that can't be written, as when its folder can't be created or the disk is full, is one `warning:` line on stderr, and the command carries on with the same outcome, stdout and exit code. +A **Command log** is everything one command printed, stderr and stdout in the order printed, while the terminal still shows all of it. Its folder is the command typed: `thirdshift `, a Run or a Spec run, in `issue/`, `thirdshift pickup` in `pickup/`, named for the issue it took, and `thirdshift architect` in `architect/`. A Spec run's covers its Tickets' Runs, a Run's covers its [Base fix](#base-fix), and a Pass's covers the Spec run or Run it dispatched: none of those keeps one of its own. So `ls -t ~/.thirdshift/logs/acme/widgets/commands/pickup | head` lists the recent passes on acme/widgets that took an issue. A command keeps its Command log once it starts work: a Pass skipped before doing any work keeps none, nor does a Run that fails before it starts work, as on the [Origin match](#what-a-run-does), and `setup`, `email-test`, `update`, `version` and `help` never keep one. Lines printed before the Command log's name is known, such as a Pickup run's lines on the issues it [passed over](#why-an-issue-was-passed-over) before it took one, are written first, and once it is created a progress line says `logging this command to `. A Command log that can't be written, as when its folder can't be created or the disk is full, is one `warning:` line on stderr, and the command carries on with the same outcome, stdout and exit code. The stamp is the local time the command started, with its UTC offset, as in `20261003T120000-0400`, in the machine's time zone, or `TZ`'s if set, so it agrees with `date` and `ls -l`. A command's Command log and all of its Session logs, its Tickets' Runs' and its Base fix's included, share that one stamp, so they sort together and each can be found from the other. When a Run fails, stderr ends with the path of its most recent Session log, the place to start looking, then that of its Command log. -The **Activity log**, `activity.log`, is a short running record of what the factory did on the repository ([ADR-0011](docs/adr/0011-thirdshift-keeps-a-per-repository-activity-log.md)). A Run, a Spec run, an Architect run or a Pickup run writes a line when it starts work, naming its Command log, and one when it ends, with its outcome; a Run you type by hand writes them too, and the run an Architect run or a Pickup run dispatched, a Spec run's Tickets and a Base fix write none of their own. A skipped Architect run or Pickup run writes a line only when its reason differs from the last line of its own kind, so a repository that sits idle shows one line, not one per pass, and the first skip after a pass that did work always shows: that is when the repository went idle. Every line starts with the local date and time: +The **Activity log**, `activity.log`, is a short running record of what the factory did on the repository ([ADR-0011](docs/adr/0011-thirdshift-keeps-a-per-repository-activity-log.md)). A Run, a Spec run or a Pass writes a line when it starts work, naming its Command log, and one when it ends, with its outcome; a Run you type by hand writes them too, and the run a Pass dispatched, a Spec run's Tickets and a Base fix write none of their own. A skipped Pass writes a line only when its reason differs from the last line of its own kind, so a repository that sits idle shows one line, not one per pass, and the first skip after a pass that did work always shows: that is when the repository went idle. Every line starts with the local date and time: ``` 2026-10-03 02:00:01 Pickup run skipped: no Ready issue on acme/widgets @@ -441,7 +452,7 @@ The **Activity log**, `activity.log`, is a short running record of what the fact Each line is appended whole, so passes that run at once on one repository never garble it. One that can't be written is one `warning:` line on stderr, and the command carries on as for a Command log. thirdshift never rotates it: collapsed skips keep it small. Logs written before this layout, under `sessions/` and `commands/` at the root of the logs, are not moved. -With `quiet_skips = true` in the `[activity]` section of the [User config](#user-config), a skipped Architect run or Pickup run prints nothing on stdout or stderr, its dated first line included, and leaves only its Activity log line. A pass that does work, or fails, prints as ever, so a scheduler's log file catches only what went wrong. Without it, a skipped pass prints as it always has, for a pass you type by hand. +With `quiet_skips = true` in the `[activity]` section of the [User config](#user-config), a skipped Pass prints nothing on stdout or stderr, its dated first line included, and leaves only its Activity log line. A pass that does work, or fails, prints as ever, so a scheduler's log file catches only what went wrong. Without it, a skipped pass prints as it always has, for a pass you type by hand. ## Foreign commits in a Merge run @@ -512,7 +523,7 @@ A Failed run: 1. Commits any uncommitted work as `thirdshift: failed run ()`, with a timestamp and the hostname, and pushes the Issue branch, so nothing is lost. If the branch has no changes against the Base branch, nothing is pushed. 2. Converts its open pull request, if any, back to a draft, so a pull request only claims to be ready when the factory stands behind it. The next successful Continuation marks it ready again. The exception is a Merge run's policy refusal: the pull request is ready, mergeable and green and only the Self-merge could not happen, so it stays ready for review, and no failure commit is pushed onto the head whose CI was watched. 3. Cleans up as usual, prints the reason to stderr and exits non-zero. If the push failed, the worktree and local Issue branch are kept instead, and stderr names the branch, its head commit and the worktree path, so you can recover the work or push it by hand. -4. Releases its [Claim](#when-the-claim-ends) if it left nothing on `origin`: no Issue branch and no pull request. Otherwise the issue stays `in-progress`. +4. Releases its [Claim](#when-the-claim-ends), unless it is a failed Security fix, if it left nothing on `origin`: no Issue branch and no pull request. Otherwise the issue stays `in-progress`. Merges, never rebases or force-pushes: a branch worked on from several servers never loses history. @@ -537,7 +548,7 @@ In a [Spec run](#spec-runs), a Ticket's Run says this on its own stderr, relayed A **Spec** is an issue with sub-issues, its **Tickets**. `thirdshift ` on a Spec is a **Spec run**: it works through the Tickets in the order their GitHub "blocked by" links allow, each Ticket's **Run** a **Merge run** into the **Spec branch**, then leaves one **Spec PR** from the Spec branch into the Base branch ready for review ([ADR-0006](docs/adr/0006-spec-runs-merge-tickets-into-a-spec-branch.md)). An issue with no sub-issues is an ordinary Run. -A Spec run makes the [Claim](#the-claim) on the Spec, as a Run does on its issue: the Spec is labelled `in-progress`, in place of `ready-for-agent`. Its Tickets' Runs change no label on their Tickets. The Claim [ends](#when-the-claim-ends) as a Run's does: it is released if the Spec run fails with no Spec branch on `origin` and no Spec PR, removed once a Self-merge of the Spec PR has left the Spec closed, and kept otherwise. The Spec branch is pushed before the first Ticket starts, so a Spec run that got as far as its Tickets keeps its Claim. +A Spec run makes the [Claim](#the-claim) on the Spec, as a Run does on its issue: the Spec is labelled `in-progress`, in place of `ready-for-agent`. Its Tickets' Runs change no label on their Tickets. The Claim [ends](#when-the-claim-ends) as a Run's does: it is released if the Spec run fails with no Spec branch on `origin` and no Spec PR, except for a failed Security fix, removed once a Self-merge of the Spec PR has left the Spec closed, and kept otherwise. The Spec branch is pushed before the first Ticket starts, so a Spec run that got as far as its Tickets keeps its Claim. The Spec PR opens as a draft as soon as the first Ticket lands, titled from the Spec, with `Closes #` and a Tickets checklist: one line per Ticket, ticked once it is done, with its pull request, or saying it is running, failed, blocked or unready. thirdshift rewrites the checklist between its `` markers as each Ticket starts and ends, leaving the rest of the body as it is. Once every Ticket is done, the **Spec review** rewrites the body, and thirdshift puts the checklist back, appending it if the markers are gone, before marking the Spec PR ready. @@ -592,13 +603,13 @@ The focus is optional free text, given as one argument, anywhere among the flags `merge`, `no-merge`, `base-fix`, `no-base-fix`, `parallel `, `harness `, `model ` and `effort ` (or the same with dashes) are for the run the plan is dispatched as, and mean what they do for `thirdshift `, so none of them is ever read as the focus. The [Harness, Model and Effort](#harness-model-and-effort) are the Architecture review's too. `--plan-only` stops the Architect run once the plan is marked ready, and dispatches nothing. `email`, optionally followed by an address, and `no-email` (or `--email` and `--no-email`) are for the Architect run's own [Run notification](#one-run-notification), with or without `--plan-only`. The word after `email` is the address only if it contains `@`, so a focus without one is never taken for it. -`base ` (or `--base `) names the Architect run's **Base branch**, so the branch your clone has checked out stops mattering: start it from a clone on another branch, on a detached HEAD, or with uncommitted changes. The Architecture review's worktree starts at the head of `` on `origin`, and the run the plan is dispatched as takes `` as its Base branch too: a Run's **Issue branch**, or a Spec run's **Spec branch**, is branched off it, and the pull request or **Spec PR** targets it. It goes before or after the focus and the other flags, `--plan-only` included, and the word after it is always the branch, never the focus. Without `base`, the Base branch is the branch checked out. `base` is a flag of `architect` and [`pickup`](#pickup-runs) only: `thirdshift ` doesn't take it. It is what lets an Architect run be started [on a schedule](#on-a-schedule). +`base ` (or `--base `) names the Architect run's **Base branch**, so the branch your clone has checked out stops mattering: start it from a clone on another branch, on a detached HEAD, or with uncommitted changes. The Architecture review's worktree starts at the head of `` on `origin`, and the run the plan is dispatched as takes `` as its Base branch too: a Run's **Issue branch**, or a Spec run's **Spec branch**, is branched off it, and the pull request or **Spec PR** targets it. It goes before or after the focus and the other flags, `--plan-only` included, and the word after it is always the branch, never the focus. Without `base`, the Base branch is the branch checked out. `base` is a flag of a Pass only: `thirdshift ` doesn't take it. It is what lets an Architect run be started [on a schedule](#on-a-schedule). A second focus, a repeated flag, `merge` with `no-merge`, `base-fix` with `no-base-fix`, `email` with `no-email`, `parallel` without a whole number from 1 up, `base` with no branch after it, any other argument starting with a dash, or an empty focus is an argument error (exit `2`). So is `merge`, `no-merge`, `base-fix`, `no-base-fix` or `parallel` with `--plan-only`, since nothing is dispatched for them to apply to. An Architect run: -1. Makes the checks a Run makes that don't need an issue, before creating anything: `origin` is a GitHub repository, git has a `user.name` and `user.email`, HEAD is not detached, and the Base branch exists on `origin` with your local copy not ahead of it. With `base `, a detached HEAD is fine, since the checkout no longer picks the Base branch, and the other two checks are made on ``: `base branch does not exist on origin; push it first`, or `local is commit(s) ahead of origin/; push them first`. There is no **Origin match**, since there is no Issue URL: the repository is the one `origin` names. If another Architect run on the repository, or a [Pickup run](#pickup-runs), is still running, it is [skipped](#one-at-a-time) here, and if not, it is skipped when an earlier [Architect plan is still open](#one-architect-plan-at-a-time), and if not, when an [Architect idea waits for triage](#an-architect-idea-waits-for-triage), and if not, when the repository has a [Ready issue](#a-ready-issue-goes-first). With `launch.pull = true` in the [User config](#user-config), it then fast-forwards your checkout of the Base branch, as a Run does, and so only when the Base branch is the branch checked out: with `base ` from a clone on another branch, your checkout is left alone. +1. Makes the checks a Run makes that don't need an issue, before creating anything: `origin` is a GitHub repository, git has a `user.name` and `user.email`, HEAD is not detached, and the Base branch exists on `origin` with your local copy not ahead of it. With `base `, a detached HEAD is fine, since the checkout no longer picks the Base branch, and the other two checks are made on ``: `base branch does not exist on origin; push it first`, or `local is commit(s) ahead of origin/; push them first`. There is no **Origin match**, since there is no Issue URL: the repository is the one `origin` names. If another Pass on the repository is still running, it is [skipped](#one-at-a-time) here, and if not, it is skipped when an earlier [Architect plan is still open](#one-architect-plan-at-a-time), and if not, when an [Architect idea waits for triage](#an-architect-idea-waits-for-triage), and if not, when the repository has a [Ready issue](#a-ready-issue-goes-first). With `launch.pull = true` in the [User config](#user-config), it then fast-forwards your checkout of the Base branch, as a Run does, and so only when the Base branch is the branch checked out: with `base ` from a clone on another branch, your checkout is left alone. 2. Creates a git worktree next to your clone, named `-architect`, detached at the head of the Base branch on `origin`, with no **Issue branch**. Your checkout, its uncommitted changes and its untracked files are never touched or scanned. 3. Runs the **Architecture review** there: a headless Claude Code session with the **Factory skills** loaded, started with the [Architecture review prompt](prompts/architecture-review.md). It looks for deepening opportunities, skips any an open issue already covers, and takes the top recommendation. If it is Strong, the review publishes it as the plan, labelled `needs-triage`: a **Spec** with **Tickets**, or a single Ticket when one session is enough. It may edit files in the worktree to check an idea, but commits and pushes nothing. It ends its final message with one line naming the plan: `Architecture review plan: `. thirdshift reads only that line. With [no Strong candidate](#no-strong-candidate) the line names another issue, which thirdshift labels an **Architect idea**, and steps 5 to 7 are skipped. 4. Removes the worktree once the session ends, whatever the outcome. The temporary directory the Factory skills were written out to goes as the Architect run ends. @@ -654,11 +665,11 @@ The rule doesn't ask whether anything will take the Ready issue. A `ready-for-ag ### One at a time -Only one Architect run or [Pickup run](#pickup-runs) per repository runs at a time on a machine. A `thirdshift architect` started while another on the same repository, or a `thirdshift pickup`, is still running, the Spec run or Run it dispatched included, is **skipped**: it prints `an Architect run or a Pickup run is already running on /` as a progress line on stderr, prints nothing on stdout, and exits `0`. Skipped is not a failure. A skipped run does nothing else: no launch pull, no worktree, no agent session, and no issue filed or labelled. This holds for every Architect run, however it was started, so two can never plan the same refactor. A Pickup run is skipped the same way while an Architect run or another Pickup run is still running, so the two never build one repository at once. +A **Pass** is a command started with no Issue URL that decides before any work whether it is skipped. Only one Pass per repository runs at a time on a machine. A Pass started while another on the same repository is still running, the Spec run or Run it dispatched included, is **skipped**: it prints `another Pass is already running on /` as a progress line on stderr, prints nothing on stdout, and exits `0`. Skipped is not a failure. A skipped Pass does nothing else: no launch pull, no worktree, no agent session, and no issue filed or labelled. This holds for every Pass, however it was started, so two never build one repository at once. -The repository is the one `origin` names, so two clones of one repository count as the same, and Architect runs on different repositories run at the same time. A Run or a Spec run started on an **Issue URL** is never skipped this way, and never makes an Architect run or a Pickup run skip. +The repository is the one `origin` names, so two clones of one repository count as the same, and Passes on different repositories run at the same time. A Run or a Spec run started on an **Issue URL** is never skipped this way, and never makes a Pass skip. -thirdshift holds the rule with an operating-system lock on a file under `~/.thirdshift/architect-locks/`, one for both kinds of run, which the run's process takes without waiting, after its checks in step 1, and holds until it exits. The operating system releases the lock when that process ends for any reason, a crash, a kill or a reboot included, so there is never a stale lock to clear: the next Architect run or Pickup run on the repository runs normally. The file itself stays, and means nothing on its own. A review worktree that a killed Architect run left behind is removed by the next one. +thirdshift holds the rule with an operating-system lock on a file under `~/.thirdshift/architect-locks/`, shared by every kind of Pass, which the Pass's process takes without waiting, after its checks in step 1, and holds until it exits. The directory keeps its original name so older thirdshift versions share the lock too. The operating system releases the lock when that process ends for any reason, a crash, a kill or a reboot included, so there is never a stale lock to clear: the next Pass on the repository runs normally. The file itself stays, and means nothing on its own. A review worktree that a killed Architect run left behind is removed by the next one. A skipped Architect run sends no [Run notification](#one-run-notification), even when asked, and neither does a skipped Pickup run ([sends none](#one-run-notification-for-the-issue-taken)). Both make the notification's checks before the lock is tried all the same, so a broken setup fails the pass with exit `1` rather than being skipped. @@ -703,7 +714,7 @@ The command's flags and the User config decide what a scheduled Architect run do The first leaves the pull request for you to review, and the second stops at the Architect plan, for you to read and run with `thirdshift `. -A scheduled Architect run is skipped when another on the repository, or a Pickup run, is [still running](#one-at-a-time), the Spec run or Run it dispatched included, when an [Architect plan is still open](#one-architect-plan-at-a-time), when an [Architect idea waits for triage](#an-architect-idea-waits-for-triage), or when the repository has a [Ready issue](#a-ready-issue-goes-first), so a `ready-for-agent` issue with no Pickup line to take it holds off every night until you run it: no new refactor is planned until last night's Architect plan is closed or loses its `architect-plan` label. Merging its pull request closes it; if its run failed, pick it up with `thirdshift ` first. Closing the pull request alone leaves the Architect plan open. A skip exits `0`, so the scheduler sees no failure, and the repository's [Activity log](#logs) has the line that says why, as does the log file unless `activity.quiet_skips` is set. With Run notifications on, by `email.always = true` in the User config or `email` on the line, an Architect run that did work sends one email, and a skipped one sends none, so a crontab line can fire every few minutes, as for [Weeding](#weeding), without flooding your inbox. A night without an email means it was skipped, or failed before the notification's checks passed: look in the log file. +A scheduled Architect run is skipped when another Pass on the repository is [still running](#one-at-a-time), the Spec run or Run it dispatched included, when an [Architect plan is still open](#one-architect-plan-at-a-time), when an [Architect idea waits for triage](#an-architect-idea-waits-for-triage), or when the repository has a [Ready issue](#a-ready-issue-goes-first), so a `ready-for-agent` issue with no Pickup line to take it holds off every night until you run it: no new refactor is planned until last night's Architect plan is closed or loses its `architect-plan` label. Merging its pull request closes it; if its run failed, pick it up with `thirdshift ` first. Closing the pull request alone leaves the Architect plan open. A skip exits `0`, so the scheduler sees no failure, and the repository's [Activity log](#logs) has the line that says why, as does the log file unless `activity.quiet_skips` is set. With Run notifications on, by `email.always = true` in the User config or `email` on the line, an Architect run that did work sends one email, and a skipped one sends none, so a crontab line can fire every few minutes, as for [Weeding](#weeding), without flooding your inbox. A night without an email means it was skipped, or failed before the notification's checks passed: look in the log file. Any scheduler that runs the command works; a systemd user timer needs lingering on (`loginctl enable-linger`) to fire while you are logged out. @@ -720,7 +731,7 @@ The `PATH` line, the `cd` and `base main` and the logins are as in [On a schedul Most passes are skipped, and that is what paces Weeding. A pass is skipped while: -- **a run is still going**: an Architect run or a Pickup run on the repository, the Spec run or Run it dispatched included ([One at a time](#one-at-a-time)); +- **a run is still going**: a Pass on the repository, the Spec run or Run it dispatched included ([One at a time](#one-at-a-time)); - **an Architect plan is open**: one whose pull request waits for review, one whose run failed, or one published with `--plan-only` and not yet built ([One Architect plan at a time](#one-architect-plan-at-a-time)); - **an Architect idea waits for triage**, since the factory has run out of Strong ideas ([An Architect idea waits for triage](#an-architect-idea-waits-for-triage)); - **the repository has a Ready issue**, so work a human shaped goes first ([A Ready issue goes first](#a-ready-issue-goes-first)). @@ -729,6 +740,59 @@ A skip exits `0` and sends no email, even with Run notifications on, so the sche On a repository that also has a [Pickup line](#a-pickup-run-on-a-schedule), put `--plan-only` on the Weeding line instead: the Architect run publishes the Architect plan and marks it ready, and a Pickup run builds it as an ordinary Ready issue, under the [Claim limit](#the-claim-limit), and merges it with `merge.always`, as in the [whole crontab](#a-pickup-run-on-a-schedule) below. +## Security runs + +`thirdshift secure` starts a **Security run**, report-only by default: it audits the **Base branch** at its head on `origin` and records each **Security finding** privately as a draft repository security advisory. Start it from a clone on the Base branch, or name that branch from any checkout: + +```sh +thirdshift secure +thirdshift secure base main harness claude +thirdshift secure base main harness codex model gpt-6.1-sol effort high +``` + +The command takes the Harness, Model and Effort words and the shared Pass words (`merge`, `no-merge`, `base-fix`, `no-base-fix`, `parallel`, `email`, `no-email`, `security-fix` and `no-security-fix`, with or without dashes). After recording findings, it tries to reproduce each untriaged one in a fresh throwaway worktree at the recorded audited commit, and keeps the outcome, severity when reproduced, notes and test in the private record. It preserves findings already triaged by the Day shift. It never publishes or closes an advisory. + +Fixing is off by default. `security-fix` on the command or `fix = true` under `[security]` allows one fix; `no-security-fix` overrides the setting. Both words are accepted on every command that starts Runs and passed to those Runs. With fixing allowed, a Security run takes the most severe reproduced finding still awaiting a fix before auditing, with ties in private-record order. After an audit reproduces findings, it goes on to the most severe one. + +A publishing session reads the private record and follows the reproduction's fix-size decision: a single **Ticket**, or a **Spec** with **Tickets** through `thirdshift-to-spec` and `thirdshift-to-tickets`. Every new issue is terse, saying only what the fix changes and linking the private record. thirdshift checks all the issues, swaps the top issue's `needs-triage` for `ready-for-agent`, adds `security-fix` to every fix issue (creating the label when missing), records the fix link privately and dispatches its **Run** or **Spec run**. The Security run ends as that run ends, including a **Merge run** when asked by the command or the User config. + +On a private repository, the finding's own issue is the fix's Ticket or Spec, keeping its write-up and proof-of-concept. A one-session fix needs no publishing session or second issue: thirdshift marks the finding's issue ready and dispatches it. For a bigger fix, the publishing session adds terse Tickets as that issue's native sub-issues, with their blocking links. + +A failed fix keeps its **Claim**, even if it pushed nothing, and stays open for the **Day shift**. No later Security run or Pickup run retries it. Security runs pause while that fix's issue is open; closing the issue lets them go on. + +`email`, optionally followed by an address, or `email.always` in the User config asks for one **Run notification** when the Security run ends, whether the audit succeeded, failed or was interrupted. `no-email` overrides the default. The notification says how the audit ended and lists each finding it recorded or matched to an existing private record: its severity when known, title and private link. It includes no finding write-up, trace or evidence, since it passes through Resend. Detailed failure causes stay in the local logs; the notification gives the audit's status and log paths. A recording failure still lists the records reached before it failed. When a run leaves a reproduced finding unfixed and neither the command nor the User config decided against fixing, its notification and stderr offer both `thirdshift secure security-fix` and `fix = true` under `[security]`. With `no-security-fix`, or the setting turned off, it offers nothing. A skipped Security run sends none. The usual address and Resend API key checks run before the skip checks or any work; a failed send is a warning and never changes the run's outcome. + +A Security run chooses its Harness from the command's `harness` word, then `[security] harness` in the User config, then `harness.default`, then Claude Code. A blank or missing Security setting keeps that default. Model and Effort come from the chosen Harness's own `[harness.]` section, with command words taking precedence. For example, `harness claude` overrides `[security] harness = "codex"` and uses `[harness.claude]`. + +Codex security sessions and their Resumes set `agents.max_concurrent_threads_per_session=8` and ask for fresh sub-agents with `fork_turns: "none"`, so the skill's verifiers stay independent. + +A Security run checks its gates in order: another **Pass** on the repository running on the machine, a **Ready issue**, a failed Security fix whose issue is still open, a **Security finding** waiting for the **Day shift**, then an unchanged **Base branch** since the last completed Security audit. An unreproduced draft advisory with no severity waits; closing it, publishing it or assigning severity ends that wait. A reproduced finding waits while fixing is disabled until its record is closed or published, or its fix Ticket is closed. On a private repository, an open finding issue with `needs-triage` waits until that label is removed or the issue is closed. The audited commit comes from the skill's own `run-metadata.json` under the audit root, with no separate state file. A skip exits `0`, starts no session, keeps no Command log and sends no Run notification; its reason goes into the repository's **Activity log** only when it changes, with the usual `activity.quiet_skips` behavior. Before starting work, thirdshift checks the chosen Harness and that **Node.js** is on `PATH`, since the embedded skill's report validators require it. + +The Security audit runs in a throwaway worktree detached at origin's Base branch head, leaving the Launch directory and local work untouched, including when `launch.pull` is set. Its Session prompt runs `thirdshift-security-audit` in full audit mode with the `quick` profile, auditing the whole repository except vendored and third-party code. It names a `SECURITY.md` or conventional threat-model document when present, reads compatible earlier runs and ends `incomplete` instead of asking for input. + +Each audit keeps a new output directory under `///audits/`, outside the worktree. The audit artifacts and private advisory descriptions contain evidence; keep the logs directory private. thirdshift runs both embedded validators before recording anything. A missing final line, an incomplete session or an invalid report fails the run and names its Session log. Command logs are in `commands/secure/`; Session logs are named `secure--security-audit.jsonl`; the Activity log records the start and ending as for other Passes. + +A finding's title becomes the advisory summary. Its description preserves its write-up, trace, evidence and validation plan, with the fingerprint and audited commit. thirdshift claims no severity, CWE or affected version range. It uses the package in the repository's manifest, or the `other` ecosystem when none is identified. Matching fingerprints in any advisory state are kept rather than recorded again. Rejected candidates stay in the local report and produce no advisory. Draft advisories require GitHub repository security manager or administrator access. On a private repository whose advisory endpoint is unavailable, the private record is an issue labelled `security-finding` and `needs-triage`. It is never replaced by a public finding issue. + +### Security review + +`security-review`, or `review = true` under `[security]`, enables one guidance review after a Run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. A Spec run reviews the whole Spec branch against its Base branch once; its Tickets' Runs get none. `no-security-review` overrides the setting. Both words, with or without dashes, are accepted on commands that start Runs and follow dispatched Runs and Base fixes. Setup asks about the setting; it is off by default. + +The session reads the relevant attack-class guidance and supporting code without delegated auditors or the full audit workflow. It fixes only findings reproduced by a failing proof-of-concept test, keeps that test and commits the fix. Unaddressed introduced findings appear under Security in the PR's Unaddressed findings. A Merge run, including a Spec run asked to merge its Spec PR, leaves the PR ready for review and fails with the findings named if any remain, or if the review was refused or incomplete. A Run or Spec run not asked to merge records the outcome and continues. An old finding, whose proof-of-concept also fails at the pinned merge base, is reported in a private file under the repository's logs (`security-reviews/`). thirdshift records it as a draft advisory, or a `security-finding`/`needs-triage` issue on a private repository, reusing existing fingerprints and preserving the Day shift's grades. New records include scored reproduction and fix-size metadata so a later Security run can take the fix when allowed. Its details stay out of the PR and it does not hold Self-merge; fixing belongs to a separate Security run. The separate Security audit retains its full audit workflow. + +### Fencing + +**Fencing** is the factory finding and fixing vulnerabilities with nobody starting it: Security runs started on a schedule with fixing allowed. thirdshift never schedules itself ([ADR-0009](docs/adr/0009-the-operating-system-schedules-thirdshift.md)). This Linux crontab entry starts one every five minutes; use your own paths, with `claude` and `gh` already logged in and Git able to push: + +```cron +PATH=/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin +*/5 * * * * cd /home/you/repos/widgets && thirdshift secure base main harness claude security-fix >> /home/you/thirdshift-secure.log 2>&1 +``` + +Each pass yields to a **Ready issue**, so work the **Day shift** shaped goes first. It is skipped while another Pass on the repository is running, while a fix has failed and its issue stays open, while a finding waits for triage and no reproduced finding can be fixed, or while the Base branch is unchanged and no fix is waiting. A failed fix pauses Fencing for the Day shift until its issue is closed; it is never retried by a later Security run or Pickup run. Triaging a finding lets audits go on; fixing allowed can still take a reproduced finding ahead of a different finding waiting for triage. + +A skip exits `0` and sends no Run notification. Its reason appears in the **Activity log** once per change of reason. Set `activity.quiet_skips = true` in the [User config](#user-config) to keep those skips out of the scheduler's log. `merge` on the command, or `merge.always = true` in the User config, lets each fix's Run merge its pull request; otherwise it stays ready for review. + ## Pickup runs `thirdshift pickup` starts a **Pickup run**: one pass, with no **Issue URL**, that takes the lowest-numbered **Ready issue** in the repository and runs it, so that an issue you have already marked ready needs no command typed for it. Start it from a clone of the repository, on the **Base branch**, or name the Base branch with `base `: @@ -746,7 +810,7 @@ A Ready issue is an open issue that: - has none of the labels that make an **Unready Ticket**, `ready-for-human`, `needs-info`, `wontfix` and `needs-triage`, so a contradictory label errs on the side of not running; - carries no [Claim](#the-claim): it is not labelled `in-progress`; - is not a sub-issue. A sub-issue is a **Ticket** of a **Spec**, and is never run on its own, whatever the Spec's labels: it is reached through its Spec, when the Spec is itself a Ready issue, so its work always goes through the **Spec branch**. Labelling one Ticket never promotes its Spec, either; -- is not labelled `base-fix`: the Run that opened a [Base fix](#base-fix)'s issue owns it; +- is not labelled `base-fix` or `security-fix`: the Run that opened a [Base fix](#base-fix)'s issue, or the Security run that dispatched a fix, owns it. A failed Security fix is still passed over if GitHub prevented its Claim from being made; - has no open blocker, by GitHub's "blocked by" links, never the text of its body. Once every blocker is closed, it can be taken; - was never started: no **Issue branch** for it is on `origin`, and no pull request from one exists, open, merged or closed. So a Pickup run never does a [Continuation](#continuation), and an issue whose Run failed waits for you; - is not a **Spec** whose **Tickets** are all closed. With nothing started on it, such a Spec was done some other way, and the [Spec run](#spec-runs) it would be dispatched as stops with nothing to do, so a Pickup run passes it over rather than take it on every pass. Waiting never makes it a Ready issue, so this comes before the next condition, however lately the Spec was labelled: close it, or take its `ready-for-agent` off. A Spec with at least one open Ticket is taken; @@ -755,7 +819,7 @@ A Ready issue is an open issue that: A Pickup run: 1. Makes the checks an [Architect run](#architect-runs) makes, before anything else but the [Run notification](#one-run-notification-for-the-issue-taken)'s own: `origin` is a GitHub repository, git has a `user.name` and `user.email`, HEAD is not detached unless `base ` names the Base branch, and the Base branch exists on `origin` with your local copy not ahead of it. A check that fails stops the pass with exit `1`, before any label is read or changed. -2. Tries the lock an Architect run takes, and is [skipped](#one-at-a-time) if an Architect run or another Pickup run on the repository is still running on the machine. +2. Tries the Pass lock, and is [skipped](#one-at-a-time) if another Pass on the repository is still running on the machine. 3. Takes `in-progress` off the repository's closed issues: the [Sweep](#the-sweep). 4. Counts the repository's open issues labelled `in-progress`, and is skipped if the repository is at its [Claim limit](#the-claim-limit). 5. Lists the repository's open issues labelled `ready-for-agent`, lowest number first, and takes the first that is a Ready issue, saying so on stderr: `taking Ready issue # "", as thirdshift <Issue URL> would`. Each one it passes over on the way gets [a line saying why](#why-an-issue-was-passed-over). One issue a pass: the rest wait for the next. @@ -767,7 +831,7 @@ A Pickup run: The dispatched run's ending is the Pickup run's: its exit code, its pull request's URL alone on stdout, and its last line on stderr, `PR <url> is ready for review` or `PR <url> is merged`. If it fails, the Pickup run fails as that [Failed run](#failed-runs) or Failed spec run does. After an [Inherited failure](#an-inherited-failure-links-the-base-branchs-checks), the command it offers is `thirdshift <Issue URL>` with the dispatched run's flags and `base-fix`, since another Pickup run would not take a started issue again. -A pass is **skipped** when the lock is held, when the repository is at its Claim limit, or when the repository has no Ready issue: it exits `0` with stdout empty and one line of reason on stderr, `an Architect run or a Pickup run is already running on <owner>/<repo>`, `at the Claim limit on <owner>/<repo>: <count> open issue(s) labelled in-progress, pickup.limit is <limit>` or `no Ready issue on <owner>/<repo>`, the last after the lines on the issues it passed over. Skipped is not a failure. A skipped pass starts no agent session, makes no Claim, and sends no Run notification. One skipped for the lock changes no label at all; the others have made the Sweep, whose lines come before the reason. +A pass is **skipped** when the lock is held, when the repository is at its Claim limit, or when the repository has no Ready issue: it exits `0` with stdout empty and one line of reason on stderr, `another Pass is already running on <owner>/<repo>`, `at the Claim limit on <owner>/<repo>: <count> open issue(s) labelled in-progress, pickup.limit is <limit>` or `no Ready issue on <owner>/<repo>`, the last after the lines on the issues it passed over. Skipped is not a failure. A skipped pass starts no agent session, makes no Claim, and sends no Run notification. One skipped for the lock changes no label at all; the others have made the Sweep, whose lines come before the reason. ### The Claim limit @@ -842,7 +906,7 @@ PATH=/home/you/.local/bin:/home/you/.cargo/bin:/usr/local/bin:/usr/bin:/bin The `PATH` line, the `cd` and `base main`, the log file's directory, the logins `claude` and `gh` need, and schedulers other than cron are as for an Architect run: see its [On a schedule](#on-a-schedule). What differs for a Pickup run: - **The interval** sets how soon a Ready issue is taken and how much work is started: a pass takes one issue, so `*/30` starts at most one every half hour. A pass lasts as long as the run it dispatched, and the passes that fire meanwhile are skipped, so the schedule builds one issue of a repository at a time on a machine, and the first pass after it ends takes the next, unless the repository is by then at its [Claim limit](#the-claim-limit): each pull request left for review, and each failure left for you, holds a Claim, and at 3 of them, or the `pickup.limit` you set, the passes take nothing until you have dealt with one. An issue you have just labelled also waits thirty minutes by default, or the `pickup.wait_minutes` you set, until it is settled. A Run you start by hand on an Issue URL is outside all this: it neither waits for a pass nor makes one skip. -- **One line per repository.** thirdshift keeps no list of repositories. A Pickup run and an Architect run on one repository each skip while the other is still running, the Spec run or Run it dispatched included, so both lines can go in one crontab and the two never build that repository at once. Passes on different repositories do run at the same time: give their lines different minutes, such as `15,45`, if they would compete for the machine. +- **One line per repository.** thirdshift keeps no list of repositories. Each Pass on one repository skips while another is still running, the Spec run or Run it dispatched included, so both lines can go in one crontab and two Passes never build that repository at once. Passes on different repositories do run at the same time: give their lines different minutes, such as `15,45`, if they would compete for the machine. - **A pass is skipped** when the [lock is held](#one-at-a-time), when the repository has no Ready issue, and when it is at its Claim limit. A skip exits `0`, so the scheduler sees no failure, and sends no email, even with Run notifications on. The repository's [Activity log](#logs) records a skip when its reason changes. Unless `activity.quiet_skips` is set, the log file holds each skip's line too, and before the line of a pass that found no Ready issue, as before the line of one that took an issue, [why each issue was passed over](#why-an-issue-was-passed-over). A pass skipped for the lock or the Claim limit looks at no issue, so it has no such lines. - **A Run notification**, by `email.always = true` in the [User config](#user-config) or `email` on the line, is sent [only for an issue taken](#one-run-notification-for-the-issue-taken). So a day without email doesn't say the passes are running: a pass that fails before it takes an issue, on a broken User config, a missing Resend key or a failed check, shows up only in the log file. @@ -867,7 +931,7 @@ The command's flags and the User config decide what a pass does with the issue i */30 * * * * cd ~/repos/widgets && thirdshift pickup base main no-merge >> ~/.thirdshift/logs/cron.log 2>&1 ``` -An issue whose run failed [keeps its Claim](#when-the-claim-ends) and waits for the **Day shift**: it stays `in-progress`, with the Issue branch and any draft pull request the run left, no later pass takes it, and it counts towards the Claim limit until it is closed or you take the label off. To send it round again by hand, run `thirdshift <Issue URL>` from the clone, with the Base branch checked out, since that command takes no `base <branch>`: it picks up where the failed run stopped, as a [Continuation](#continuation) does. Labelling it `ready-for-agent` again doesn't do it: a Pickup run never takes an issue that was started. The one failure a later pass does retry is one that left nothing on `origin`, such as a usage limit or an expired login: its Claim is released, so the issue is a Ready issue again once it has settled. +An issue whose run failed [keeps its Claim](#when-the-claim-ends) and waits for the **Day shift**: it stays `in-progress`, with the Issue branch and any draft pull request the run left, no later pass takes it, and it counts towards the Claim limit until it is closed or you take the label off. To send it round again by hand, run `thirdshift <Issue URL>` from the clone, with the Base branch checked out, since that command takes no `base <branch>`: it picks up where the failed run stopped, as a [Continuation](#continuation) does. Labelling it `ready-for-agent` again doesn't do it: a Pickup run never takes an issue that was started. For an ordinary Run, the one failure a later pass does retry is one that left nothing on `origin`, such as a usage limit or an expired login: its Claim is released, so the issue is a Ready issue again once it has settled. A failed Security fix is always left for the Day shift, even with nothing on `origin`. If you shape your issues with the upstream `to-spec` and `to-tickets`, from [mattpocock/skills](https://github.com/mattpocock/skills), unchanged: `ready-for-agent` on a **Spec** must mean its **Tickets** are published, every Ticket attached as a sub-issue and every "blocked by" link in place. Upstream, `to-spec` labels the Spec `ready-for-agent` when it publishes it, minutes or days before `to-tickets` attaches the Tickets, and a Spec with no Tickets yet looks exactly like a standalone Ticket, so a pass would start a plain Run on it. Label such a Spec `needs-triage` until its Tickets are attached, then swap that for `ready-for-agent`, as the copies of the two this repository's own Day shift uses, under `.agents/skills/`, do ([`docs/agents/triage-labels.md`](docs/agents/triage-labels.md)). The settling window (thirty minutes by default) is only a second line of defence: it covers Tickets attached within minutes of the label, not a Spec left labelled and without Tickets for longer. @@ -938,4 +1002,6 @@ The tag starts the release workflow, generated by [`dist`](https://github.com/ax thirdshift is released under the [MIT License](LICENSE). -The Factory skills in `skills/` are adapted from Matt Pocock's [mattpocock/skills](https://github.com/mattpocock/skills), changed to run headless with no human in the loop. They are used under the MIT License; its copyright notice is kept in [`skills/LICENSE`](skills/LICENSE) and is embedded in the binary and written out with the skills on every Run. The `thirdshift-pr` skill also reproduces part of Dex Horthy's `show-me` skill; see [`skills/thirdshift-pr/CREDITS.md`](skills/thirdshift-pr/CREDITS.md). +Most Factory skills in `skills/` are adapted from Matt Pocock's [mattpocock/skills](https://github.com/mattpocock/skills), changed to run headless with no human in the loop. They are used under the MIT License; its copyright notice is kept in [`skills/LICENSE`](skills/LICENSE) and is embedded in the binary and written out with the skills on every Run. The `thirdshift-pr` skill also reproduces part of Dex Horthy's `show-me` skill; see [`skills/thirdshift-pr/CREDITS.md`](skills/thirdshift-pr/CREDITS.md). + +The `thirdshift-security-audit` Factory skill is copied from Cloudflare's [security-audit-skill](https://github.com/cloudflare/security-audit-skill) at commit `c1c8a8c`, with only its frontmatter name changed. It is used under the MIT License; Cloudflare's copyright notice and licence are kept in [`skills/thirdshift-security-audit/LICENSE`](skills/thirdshift-security-audit/LICENSE), with the upstream repository and full commit in [`CREDITS.md`](skills/thirdshift-security-audit/CREDITS.md). Both are embedded and written out with the skill. diff --git a/docs/research/security-review-mode-trial.md b/docs/research/security-review-mode-trial.md new file mode 100644 index 00000000..6a178f97 --- /dev/null +++ b/docs/research/security-review-mode-trial.md @@ -0,0 +1,113 @@ +# Security review mode trial + +Research date: 2026-10-08. Implements [#549](https://github.com/JacobStephens2/thirdshift/issues/549), the report-only trial in [#427](https://github.com/JacobStephens2/thirdshift/issues/427). At Jacob's request, Codex graded the private results on 2026-10-09. Jacob adopted guidance for the optional Security review on 2026-10-09. This note records the measurements, grading summary, and mode decision. + +## Method + +The sample is the three most recent merged **code** pull requests on thirdshift's `main` at the start of the trial: [#529](https://github.com/JacobStephens2/thirdshift/pull/529), [#527](https://github.com/JacobStephens2/thirdshift/pull/527), and [#525](https://github.com/JacobStephens2/thirdshift/pull/525), all merged on 2026-10-07. The newer #553 imported the skill being evaluated into `issue-427`; it was excluded from the code sample. This is one factory repository and one language, Rust, with small diffs. It is not a cross-repository benchmark. + +| PR | Merge base | Reviewed head | Changed files | +|---|---|---|---:| +| #529 | `54c31ee80fe3fcc7af454b007f18b47d6e6976ad` | `b5c9980a4d23fd72ccc1796807cc617022e9c2cf` | 6 | +| #527 | `5ec943dcb0c4dfe26aa9f5d98916aa947f48ca42` | `ac60d24e79226ef4f1c7b54ebb8442d1826e8077` | 2 | +| #525 | `535c3070073e38a70f9b40f9056c3794a1a4b60c` | `33609e6c20326bb535e084382d85f38a2aa236b5` | 4 | + +GitHub's pull-request API supplied each original head and base SHA. `git merge-base <base> <head>` supplied the comparison's start; `git diff <merge-base> <head>` supplied its scope. Each of the twelve sessions got its own fresh local clone, made with `git clone --no-hardlinks`, with `origin` removed before checkout and the session's start. Each target was detached at its PR head. The controller checked remotes and tracked/untracked target changes before and after each session. No prior trial or audit was provided as context. + +Both modes used the [embedded Factory skill](../../skills/thirdshift-security-audit/SKILL.md), imported in #553, from Cloudflare's upstream commit [`c1c8a8c1471069fb0e188eeaff69b8e8db6564a8`](https://github.com/cloudflare/security-audit-skill/tree/c1c8a8c1471069fb0e188eeaff69b8e8db6564a8). Its body, companions, schema and validators are unchanged upstream copies; the name is `thirdshift-security-audit` ([credits](../../skills/thirdshift-security-audit/CREDITS.md)). + +- **Full audit:** the complete six-phase workflow, `quick` profile, scoped to the merge-base/head diff, with native artifacts outside the target. No strict agent-invocation budget was imposed. It kept the single hunter wave, final coverage critic and independent verifiers required by `quick`. +- **Guidance:** one session read the relevant attack classes and reviewed the same diff. Delegation and the full workflow were explicitly disabled. Findings were returned in the session's final JSON response, with the skill's verdict vocabulary, rather than native audit artifacts. + +Both prompts named the target, head, merge base and diff scope; excluded vendored and third-party code; requested applicable threat-model and domain context; and allowed supporting source reads only to resolve in-scope behavior. They forbade target edits, commits, pushes and any public record. They instructed the session to end `incomplete` instead of asking a question, gave it no prior runs, and required it to join or stop its own tasks before ending. The guidance response contract was `{status, asked_question, findings, limitations}`; the full audit response contract was `{status, asked_question, findings_file, limitations}`. A finding's title and detail were retained only privately. + +No OS-enforced sandbox for target code was provisioned. The prompts explicitly forbade target-controlled builds, tests, processes and fixtures, and permitted trusted source-reading tools and the skill's Node validators. Where execution was decisive, the skill had to retain a `needs_validation` blocker and safe validation plan. A complete source-only review is not a confirmed vulnerability or a complete security assessment. + +The Harnesses were Claude Code **2.1.292**, `claude-opus-5-5` at **medium**, and Codex **0.161.0**, `gpt-6.1-sol` at **xhigh**. Claude used the factory's `auto` permission mode and a private, session-local plugin containing the pinned skill. Codex used the factory's unsandboxed invocation ([Claude adapter](../../src/harness/claude.rs), [Codex adapter](../../src/harness/codex.rs)). The agent CLI being unsandboxed did not authorize target-code execution. Codex's concurrent sub-agent cap was raised to eight, and the full audit prompt required fresh sub-agents with `fork_turns="none"`. + +The launch arguments, with path placeholders, were: + +```sh +claude -p --permission-mode auto --model claude-opus-5-5 --effort medium \ + --output-format stream-json --verbose --plugin-dir <private-plugin> \ + --add-dir <private-trial-directory> +codex exec --json --dangerously-bypass-approvals-and-sandbox \ + -m gpt-6.1-sol -c 'model_reasoning_effort="xhigh"' \ + -c agents.max_concurrent_threads_per_session=8 \ + -c 'project_doc_fallback_filenames=["CLAUDE.md"]' \ + -o <private-final-response> - +``` + +Prompts were supplied on stdin. The controller ran at most **two top-level trial sessions** at once, each under a **2,700-second (45-minute) timeout**. Delegated audit agents belonged to those sessions. For each PR in order #529, #527, #525, it queued guidance on Claude then Codex, followed by full audit on Claude then Codex. A freed slot started the next queued trial. Wall time covers the CLI's start through its exit, including initialization, permission review and sub-agent waits, but excludes clone preparation. A timeout terminates only that trial's captured process group; it is not a clean result. No safety refusal was retried on another Model. + +## Measurements + +Each cell is one independent session; there were twelve CLI sessions, and no cell was rerun. **C / N / R** means `confirmed` / `needs_validation` / `rejected` final records. **Refusal / Model change / question** records visible events for that particular session. All twelve exited 0 before their timeout, reported `complete`, left their target unchanged and kept it without remotes. All six full audits passed both native validators. These completion statements describe the scoped source-only workflow, not exhaustive coverage. + +### Claude Code + +Cost is the CLI’s reported `total_cost_usd`, rounded to four decimal places; it is a usage estimate, not an invoice. The reported total includes delegated agents; Anthropic documents that `usage` excludes them while `total_cost_usd` and `modelUsage` include them ([cost accounting](https://code.claude.com/docs/en/agent-sdk/cost-tracking#get-the-total-cost-of-a-query)). Sub-agents are distinct `local_agent` task starts in the private stream, cross-checked against delegation calls; resumed calls and background shell tasks are not counted as additional agents. + +| PR | Mode | Wall seconds | Cost USD | Sub-agents | C / N / R | Refusal / Model change / question | +|---|---|---:|---:|---:|---|---| +| #529 | Guidance | 64.181 | $0.4574 | 0 | 0 / 0 / 2 | 0 / 0 / no | +| #529 | Full audit, quick | 365.615 | $5.0737 | 8 | 0 / 0 / 0 | 0 / 0 / no | +| #527 | Guidance | 35.393 | $0.3481 | 0 | 0 / 0 / 0 | 0 / 0 / no | +| #527 | Full audit, quick | 360.046 | $4.8014 | 7 | 0 / 0 / 0 | 0 / 0 / no | +| #525 | Guidance | 46.072 | $0.3877 | 0 | 0 / 0 / 0 | 0 / 0 / no | +| #525 | Full audit, quick | 320.071 | $4.5467 | 8 | 0 / 0 / 0 | 0 / 0 / no | + +### Codex + +Tokens are the sum of each parent and descendant thread’s final `total_token_usage`, matched by exact thread ID and the recorded `parent_thread_id` lineage. Total = input + output; cached input is already part of input, and reasoning output is already part of output. Neither subset is added again. The separate cached column makes repeated cached context visible. Codex reports no dollar cost. + +The `exec --json` stream in this installed CLI omitted the newer delegation calls, so its apparent zero-agent count was discarded. The private rollouts establish the actual descendant count, requested Model and Effort, `fork_turns="none"` on every audit spawn, question-tool calls, and terminal completion of every thread. This avoids counting only the parent’s tokens. Refusals were checked in provider failure events; Model changes were checked in Claude’s emitted Model fields, Codex’s requested Models and explicit reroute notices. Codex does not expose a separate server-returned Model here, so zero means no visible switch, not an assertion about provider internals. Source-reading output was excluded from these event checks. Questions combine actual question-tool calls with the final response’s `asked_question` value. + +| PR | Mode | Wall seconds | Total tokens | Cached input tokens | Sub-agents | C / N / R | Refusal / Model change / question | +|---|---|---:|---:|---:|---:|---|---| +| #529 | Guidance | 179.166 | 948,329 | 835,968 | 0 | 0 / 0 / 0 | 0 / 0 / no | +| #529 | Full audit, quick | 949.918 | 15,871,642 | 14,784,512 | 9 | 0 / 0 / 0 | 0 / 0 / no | +| #527 | Guidance | 185.451 | 1,069,677 | 946,432 | 0 | 0 / 0 / 0 | 0 / 0 / no | +| #527 | Full audit, quick | 1108.118 | 11,152,997 | 10,264,832 | 8 | 0 / 0 / 0 | 0 / 0 / no | +| #525 | Guidance | 182.523 | 686,666 | 577,408 | 0 | 0 / 0 / 0 | 0 / 0 / no | +| #525 | Full audit, quick | 1088.746 | 13,115,984 | 12,153,984 | 8 | 0 / 0 / 0 | 0 / 0 / no | + +### What this sample establishes + +| Harness | Mode | Median wall seconds | Median cost or total tokens | +|---|---|---:|---:| +| Claude Code | Guidance | 46.072 | $0.3877 | +| Claude Code | Full audit, quick | 360.046 | $4.8014 | +| Codex | Guidance | 182.523 | 948,329 tokens | +| Codex | Full audit, quick | 1088.746 | 13,115,984 tokens | + +Guidance was faster and used less reported cost or fewer tokens on each of these three PRs. Full audit added reconnaissance, a coverage ledger, independent agents and validated native artifacts. Each Harness made its own coverage plan, so the same `quick` profile did not impose identical ledger granularity. + +There are no independently confirmed vulnerabilities or seeded ground-truth defects in this source-only sample. Counts alone therefore establish neither recall nor precision, and an empty final array is not evidence that the code is secure. The private results have now been graded by AI source review at the operator’s request; the grading summary below does not establish runtime detection accuracy. One run per cell, one Rust repository, small related diffs and fixed execution order limit generalization. Shared provider caches and other machine activity were not controlled. Dollar estimates and raw tokens are different measures and are not a cross-Harness price comparison. The trial supports a discussion of overhead and unattended operation; it does not choose the Security review’s mode. + +## Private grading record + +Every reported finding, including rejected candidates, is grouped by PR, mode and Harness in: + +`/home/jacob/.local/state/thirdshift/security-review-trial-549-7451fsff/findings.md` + +The file has mode **600**, inside a directory with mode **700**. Grades now accompany each finding and each empty-result cell, with the original finding JSON preserved. Raw session transcripts and native audit artifacts remain private on this machine. Finding titles, details, fingerprints, source traces and validation plans are absent from this note and from the pull request. No advisory, issue or label was created for the trial, and nothing was pushed to another repository. + +## Grading summary (2026-10-09) + +Codex compared the two submitted candidates against the original merge base and reviewed head, read the relevant ownership contracts and ADR, and checked all twelve private analysis records and the six full-audit validator results. This was source review; no target code was built or executed, and it was not a new exhaustive audit of the three PRs. Candidate details and grading rationales remain in the private record. + +| Dimension | Assessment | +|---|---| +| Operational feasibility | Pass within the sample: all twelve sessions completed, and all six full-audit artifact validations passed. | +| Submitted candidates | Both were correctly rejected as introduced vulnerabilities under the trial's documented scope, with a rationale caveat retained privately. Neither is a detected vulnerability or an actionable false positive. | +| Empty-result cells | No actionable candidate submitted; no false-negative rate or security-assurance score can be assigned. | +| Detection efficacy | Inconclusive: there were no seeded defects or independently established positive cases. Precision and recall are unmeasured. | +| Coverage process | Full audit provides a coverage ledger and independent review artifacts; guidance does not provide that artifact contract. Passing validators does not prove adequate coverage. | +| Efficiency | Guidance was faster and used less reported cost or aggregate tokens on all three PRs. | + +**Recommendation:** use guidance provisionally for the opt-in Security review of a change, based on its lower overhead and no demonstrated additional actionable yield from full audit in this sample. This is an engineering recommendation, not evidence of equivalent coverage or detection ability. The separate whole-repository Security audit retains its full-audit workflow. + +The existing Spec evidence requirements still apply: fixes require a failing proof-of-concept test, unaddressed introduced findings hold Self-merge, and pre-existing vulnerabilities stay in private records. A future effectiveness comparison should include known positive and negative cases with safe independent validation. + +**Mode decision (2026-10-09):** Jacob adopted guidance for the opt-in Security review in a Run or a Spec run. The review uses one session to apply the relevant attack-class guidance to the change and supporting code, without the full six-phase audit workflow or delegated auditors. This choice unblocks Tickets #550–#552 once recorded in their specifications. It does not change the separate whole-repository Security audit's full-audit workflow or the evidence and merge requirements above. diff --git a/prompts/README.md b/prompts/README.md index 6b0f3837..1df3b83c 100644 --- a/prompts/README.md +++ b/prompts/README.md @@ -25,7 +25,11 @@ With `harness codex`, each session runs `codex exec --json --dangerously-bypass- | Continuation, with an open pull request | Starts the implement session when the Run is a Continuation of an Issue branch whose pull request is open. | [continuation-pr.md](continuation-pr.md) | | Spec review | In a Spec run, starts the Spec review once every Ticket has landed on the Spec branch, before the Spec PR, a draft until then, is marked ready. | [spec-review.md](spec-review.md) | | Architecture review | Starts the Architecture review, the session an Architect run opens with, in a worktree at the head of the Base branch. The line naming the focus is left out when the command gives none. | [architecture-review.md](architecture-review.md) | +| Security review | When enabled, runs one guidance review after a Run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Ticket Runs get none; Base fixes get one too. Introduced findings, refusals or incomplete reviews hold Self-merge. | [security-review.md](security-review.md) | +| Security audit | Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. | [security-audit.md](security-audit.md) | +| Security reproduction | After a Security audit, tries to reproduce one recorded finding in a fresh throwaway worktree at its audited commit. The test is copied into the private record only after a complete outcome. | [security-reproduction.md](security-reproduction.md) | | Conflict Repair | Starts a Repair session when merging the Base branch into the Issue branch leaves conflicts. | [conflict-repair.md](conflict-repair.md) | +| Security fix publishing | With fixing allowed, publishes a terse Ticket or Spec with Tickets for the most severe reproduced finding, reusing a private finding's issue. thirdshift checks and marks it ready before dispatching its Run or Spec run. | [security-fix.md](security-fix.md) | | Conflict Repair, on Foreign commits | In a Merge run, starts a Repair session when merging Foreign commits from the Issue branch on origin into the local one leaves conflicts. | [foreign-conflict-repair.md](foreign-conflict-repair.md) | | Review Repair | In a Merge run, starts a Repair session once Foreign commits are merged into the Issue branch, to review them from the head the Run last knew as its own before they can be merged. | [review-repair.md](review-repair.md) | | CI-fix Repair | Starts a Repair session when CI fails on the pull request's head commit, listing each failed check. | [ci-fix-repair.md](ci-fix-repair.md) | diff --git a/prompts/security-audit.md b/prompts/security-audit.md new file mode 100644 index 00000000..7f66f074 --- /dev/null +++ b/prompts/security-audit.md @@ -0,0 +1,21 @@ +<!-- Generated by src/prompts_page.rs from the prompts in src/prompt.rs and their titles and when sentences in src/prompts_page.rs; don't edit. Regenerate with UPDATE_PROMPTS=1 cargo test prompts_page --> + +# Security audit + +Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. + +``` +/thirdshift-security-audit +Use the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile. +Audit the whole repository at commit `<audited commit>`, the head of Base branch `<base>` on origin. All vendored and third-party code are out of scope. +Read the repository's threat-model document `SECURITY.md`. +Output directory: `<output directory>`. +Audit root: `<audit root>`. +Read compatible earlier runs under the audit root before planning this audit. +This session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing. +If a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking. +Write the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing. +End your final message with exactly `Security audit: complete` after writing valid artifacts, or `Security audit: incomplete` when incomplete. + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +``` diff --git a/prompts/security-fix.md b/prompts/security-fix.md new file mode 100644 index 00000000..f148ae5a --- /dev/null +++ b/prompts/security-fix.md @@ -0,0 +1,21 @@ +<!-- Generated by src/prompts_page.rs from the prompts in src/prompt.rs and their titles and when sentences in src/prompts_page.rs; don't edit. Regenerate with UPDATE_PROMPTS=1 cargo test prompts_page --> + +# Security fix publishing + +With fixing allowed, publishes a terse Ticket or Spec with Tickets for the most severe reproduced finding, reusing a private finding's issue. thirdshift checks and marks it ready before dispatching its Run or Spec run. + +``` +Publish the fix for this reproduced Security finding on Base branch `<base>`. +Read the private record at <private record URL> and the record below. +Follow the completed reproduction's fix size: single publishes one Ticket; spec publishes a Spec with Tickets using `thirdshift-to-spec` and `thirdshift-to-tickets`. +On a public repository, publish one new top issue labelled `needs-triage`. Create that label if missing. On a private repository, reuse the finding's issue as the top issue and preserve its body and evidence; add the bigger fix's Tickets as its native sub-issues. +Every new issue is terse: it says only what the fix changes and links the private record. It carries none of the write-up, trace, evidence, reproduction notes, proof-of-concept test or exploit details, even paraphrased. Keep those in the private record. This overrides the skills' templates. +A Spec's Tickets must be new, open, labelled `ready-for-agent`, linked as native sub-issues with their native blocking links, and have no sub-issues of their own. Read the links back before finishing. +This session changes no repository source, commits and pushes nothing, opens no pull request, and does not implement the fix. thirdshift checks every issue, marks the top issue ready, labels all fix issues security-fix and dispatches its Run or Spec run. +End your final message with exactly `Security fix Ticket: <Issue URL>` for single or `Security fix Spec: <Issue URL>` for spec, naming the top issue. + +Private record: +<recorded Security finding> + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +``` diff --git a/prompts/security-reproduction.md b/prompts/security-reproduction.md new file mode 100644 index 00000000..b5be193e --- /dev/null +++ b/prompts/security-reproduction.md @@ -0,0 +1,21 @@ +<!-- Generated by src/prompts_page.rs from the prompts in src/prompt.rs and their titles and when sentences in src/prompts_page.rs; don't edit. Regenerate with UPDATE_PROMPTS=1 cargo test prompts_page --> + +# Security reproduction + +After a Security audit, tries to reproduce one recorded finding in a fresh throwaway worktree at its audited commit. The test is copied into the private record only after a complete outcome. + +``` +Reproduce this recorded Security finding at audited commit `<audited commit>`. +Read the `thirdshift-security-audit` skill's likelihood-and-impact severity rubric. +Write a proof-of-concept test from the finding's validation plan, then run it against the untouched code in this throwaway worktree. Use harmless payloads only, never a deployed site or a real third-party service. Do not fix or change repository source. +If the test reproduces the finding, score its severity with the skill's rubric and judge whether one session can hold the fix (single) or it needs a Spec (spec). +Test file: `<test file>`. +Write the test's full text to this file, even when not reproduced. In your final message, give reproduction notes: the exact command, its result, and, when reproduced, likelihood, impact and the fix-size reasoning. +End your final message with exactly `Security reproduction: reproduced <severity> <size>`, where severity is critical, high, medium, low or informational and size is single or spec, or `Security reproduction: not reproduced`. +This session commits, pushes, opens and publishes nothing. + +Recorded finding: +<recorded Security finding> + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +``` diff --git a/prompts/security-review.md b/prompts/security-review.md new file mode 100644 index 00000000..5d91e0f2 --- /dev/null +++ b/prompts/security-review.md @@ -0,0 +1,24 @@ +<!-- Generated by src/prompts_page.rs from the prompts in src/prompt.rs and their titles and when sentences in src/prompts_page.rs; don't edit. Regenerate with UPDATE_PROMPTS=1 cargo test prompts_page --> + +# Security review + +When enabled, runs one guidance review after a Run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Ticket Runs get none; Base fixes get one too. Introduced findings, refusals or incomplete reviews hold Self-merge. + +``` +Use the `thirdshift-security-audit` skill in guidance mode. +Review the change for <Issue URL> on branch <branch> against Base branch <base> with `git diff <merge base commit>...HEAD`, including supporting code. +Read the relevant security attack-class guidance and the repository's SECURITY.md or threat model when present. +Use one session. Do not delegate auditors or run the full six-phase audit, validators, coverage ledger or audit artifacts. +Fix only a Security finding you can show with a failing proof-of-concept test; run it before fixing, keep it as a regression test, and rerun it afterward. Use harmless local payloads; touch no deployed site or real third-party service. Commit each fix. +Merge base commit: `<merge base commit>`. +Classify introduced findings against this pinned merge base: run each proof-of-concept on HEAD and the untouched merge base; a test that also fails there is pre-existing. Never include pre-existing vulnerability details, titles, fingerprints, tests or private-record links in the pull request or final message, and do not fix those here. +Private report file: `<private report file>`. +Write a JSON array to that file, even when empty ([]). Each entry is an old finding with exactly these fields: fingerprint (stable across reviews and audits, no newlines or backticks), title (one line), description (the private write-up and evidence), proof_of_concept (test: full test text, command: exact command run at both commits, head_exit_code: positive failing exit code, merge_base_exit_code: positive failing exit code, notes: reproduction evidence and scoring rationale, severity: critical/high/medium/low/informational scored with the skill's likelihood-and-impact rubric, fix_size: single/spec judged from whether one session can hold the fix). All text fields are nonempty. Do not commit or copy this report into the worktree. thirdshift records it privately with the Security run's fingerprint matching and reproduced-outcome format so a later Security run can fix it when allowed; existing records and the Day shift's grades are preserved. Fixing old findings belongs to a separate Security run. +Update this branch's pull request: list each unaddressed introduced finding under Security in its Unaddressed findings, with evidence and why it is unaddressed. Preserve Standards and Spec entries and the rest of the body. Do not merge. Delivery pushes any new commits and marks the PR ready after this session. +If refused, incomplete, or missing a prerequisite, say so and do not claim completion. +After a complete review, end your final message with exactly one line of the form: +Security review: {"unaddressed_count":0,"findings":[],"pre_existing_count":0} +unaddressed_count counts only unaddressed findings introduced by this change; findings is an array of their short titles, with one title per finding. pre_existing_count is the number of entries in the private report. No old-finding details belong in this line. + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +``` diff --git a/prompts/spec-review.md b/prompts/spec-review.md index baa0f94a..70df5f9f 100644 --- a/prompts/spec-review.md +++ b/prompts/spec-review.md @@ -14,7 +14,7 @@ Review with the `thirdshift-code-review` skill using <base> as the fixed point a For each Standards or Spec finding that says behaviour is wrong, run its test or command as written before deciding the finding. If it fails as the finding says, fix the code and keep the test. If it passes, you may decline the finding, citing the run. A passing run counts only when it exercises that finding; a green suite does not. Address the Standards and Spec findings you agree with, using the `thirdshift-tdd` skill where it fits, and commit. Write the reviewers' reports, with each axis's final files-read list, to `<review reports directory>/standards.md` (Standards) and `<review reports directory>/spec.md` (Spec). -Push branch <Spec branch>. Do not rebase or force-push. +Do not push: thirdshift pushes branch <Spec branch> after the optional Security review. Do not rebase or force-push. Update PR <pull request URL> using the `thirdshift-pr` skill, rewriting its body to cover the whole Spec. Leave out its Tickets checklist, or keep it between its markers as it is: thirdshift puts it back. Leave the PR a draft: thirdshift marks it ready once you are done. diff --git a/site/prompts/index.html b/site/prompts/index.html index 70070308..1244d2a8 100644 --- a/site/prompts/index.html +++ b/site/prompts/index.html @@ -121,7 +121,7 @@ <h3 id="prompt-spec-review-title">Spec review</h3> For each Standards or Spec finding that says behaviour is wrong, run its test or command as written before deciding the finding. If it fails as the finding says, fix the code and keep the test. If it passes, you may decline the finding, citing the run. A passing run counts only when it exercises that finding; a green suite does not. Address the Standards and Spec findings you agree with, using the `thirdshift-tdd` skill where it fits, and commit. Write the reviewers' reports, with each axis's final files-read list, to `<mark class="ph"><review reports directory></mark>/standards.md` (Standards) and `<mark class="ph"><review reports directory></mark>/spec.md` (Spec). -Push branch <mark class="ph"><Spec branch></mark>. Do not rebase or force-push. +Do not push: thirdshift pushes branch <mark class="ph"><Spec branch></mark> after the optional Security review. Do not rebase or force-push. Update PR <mark class="ph"><pull request URL></mark> using the `thirdshift-pr` skill, rewriting its body to cover the whole Spec. Leave out its Tickets checklist, or keep it between its markers as it is: thirdshift puts it back. Leave the PR a draft: thirdshift marks it ready once you are done. @@ -155,6 +155,66 @@ <h3 id="prompt-architecture-review-title">Architecture review</h3> Architecture review idea: <URL of the issue you filed> Architecture review already filed: <URL of the open issue that already covers it> +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +</code></pre> + </article> + <article class="job-sheet" id="prompt-security-review" aria-labelledby="prompt-security-review-title"> + <h3 id="prompt-security-review-title">Security review</h3> + <p>When enabled, runs one guidance review after a Run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Ticket Runs get none; Base fixes get one too. Introduced findings, refusals or incomplete reviews hold Self-merge.</p> + <p class="sent-by">Sent by <a href="/#unit-3">3 · Y · Review</a></p> + <pre class="job-text"><code>Use the `thirdshift-security-audit` skill in guidance mode. +Review the change for <mark class="ph"><Issue URL></mark> on branch <mark class="ph"><branch></mark> against Base branch <mark class="ph"><base></mark> with `git diff <mark class="ph"><merge base commit></mark>...HEAD`, including supporting code. +Read the relevant security attack-class guidance and the repository's SECURITY.md or threat model when present. +Use one session. Do not delegate auditors or run the full six-phase audit, validators, coverage ledger or audit artifacts. +Fix only a Security finding you can show with a failing proof-of-concept test; run it before fixing, keep it as a regression test, and rerun it afterward. Use harmless local payloads; touch no deployed site or real third-party service. Commit each fix. +Merge base commit: `<mark class="ph"><merge base commit></mark>`. +Classify introduced findings against this pinned merge base: run each proof-of-concept on HEAD and the untouched merge base; a test that also fails there is pre-existing. Never include pre-existing vulnerability details, titles, fingerprints, tests or private-record links in the pull request or final message, and do not fix those here. +Private report file: `<mark class="ph"><private report file></mark>`. +Write a JSON array to that file, even when empty ([]). Each entry is an old finding with exactly these fields: fingerprint (stable across reviews and audits, no newlines or backticks), title (one line), description (the private write-up and evidence), proof_of_concept (test: full test text, command: exact command run at both commits, head_exit_code: positive failing exit code, merge_base_exit_code: positive failing exit code, notes: reproduction evidence and scoring rationale, severity: critical/high/medium/low/informational scored with the skill's likelihood-and-impact rubric, fix_size: single/spec judged from whether one session can hold the fix). All text fields are nonempty. Do not commit or copy this report into the worktree. thirdshift records it privately with the Security run's fingerprint matching and reproduced-outcome format so a later Security run can fix it when allowed; existing records and the Day shift's grades are preserved. Fixing old findings belongs to a separate Security run. +Update this branch's pull request: list each unaddressed introduced finding under Security in its Unaddressed findings, with evidence and why it is unaddressed. Preserve Standards and Spec entries and the rest of the body. Do not merge. Delivery pushes any new commits and marks the PR ready after this session. +If refused, incomplete, or missing a prerequisite, say so and do not claim completion. +After a complete review, end your final message with exactly one line of the form: +Security review: {"unaddressed_count":0,"findings":[],"pre_existing_count":0} +unaddressed_count counts only unaddressed findings introduced by this change; findings is an array of their short titles, with one title per finding. pre_existing_count is the number of entries in the private report. No old-finding details belong in this line. + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +</code></pre> + </article> + <article class="job-sheet" id="prompt-security-audit" aria-labelledby="prompt-security-audit-title"> + <h3 id="prompt-security-audit-title">Security audit</h3> + <p>Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root.</p> + <p class="sent-by">Sent by a Security run, before any unit</p> + <pre class="job-text"><code>/thirdshift-security-audit +Use the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile. +Audit the whole repository at commit `<audited commit>`, the head of Base branch `<mark class="ph"><base></mark>` on origin. All vendored and third-party code are out of scope. +Read the repository's threat-model document `SECURITY.md`. +Output directory: `<output directory>`. +Audit root: `<audit root>`. +Read compatible earlier runs under the audit root before planning this audit. +This session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing. +If a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking. +Write the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing. +End your final message with exactly `Security audit: complete` after writing valid artifacts, or `Security audit: incomplete` when incomplete. + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +</code></pre> + </article> + <article class="job-sheet" id="prompt-security-reproduction" aria-labelledby="prompt-security-reproduction-title"> + <h3 id="prompt-security-reproduction-title">Security reproduction</h3> + <p>After a Security audit, tries to reproduce one recorded finding in a fresh throwaway worktree at its audited commit. The test is copied into the private record only after a complete outcome.</p> + <p class="sent-by">Sent by a Security run, before any unit</p> + <pre class="job-text"><code>Reproduce this recorded Security finding at audited commit `<audited commit>`. +Read the `thirdshift-security-audit` skill's likelihood-and-impact severity rubric. +Write a proof-of-concept test from the finding's validation plan, then run it against the untouched code in this throwaway worktree. Use harmless payloads only, never a deployed site or a real third-party service. Do not fix or change repository source. +If the test reproduces the finding, score its severity with the skill's rubric and judge whether one session can hold the fix (single) or it needs a Spec (spec). +Test file: `<test file>`. +Write the test's full text to this file, even when not reproduced. In your final message, give reproduction notes: the exact command, its result, and, when reproduced, likelihood, impact and the fix-size reasoning. +End your final message with exactly `Security reproduction: reproduced <severity> <size>`, where severity is critical, high, medium, low or informational and size is single or spec, or `Security reproduction: not reproduced`. +This session commits, pushes, opens and publishes nothing. + +Recorded finding: +<recorded Security finding> + You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. </code></pre> </article> @@ -169,6 +229,25 @@ <h3 id="prompt-conflict-repair-title">Conflict Repair</h3> Resolve the conflicts, finish the merge, and push <mark class="ph"><branch></mark>. Do not rebase or force-push. +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +</code></pre> + </article> + <article class="job-sheet" id="prompt-security-fix" aria-labelledby="prompt-security-fix-title"> + <h3 id="prompt-security-fix-title">Security fix publishing</h3> + <p>With fixing allowed, publishes a terse Ticket or Spec with Tickets for the most severe reproduced finding, reusing a private finding's issue. thirdshift checks and marks it ready before dispatching its Run or Spec run.</p> + <p class="sent-by">Sent by a Security run, before any unit</p> + <pre class="job-text"><code>Publish the fix for this reproduced Security finding on Base branch `<mark class="ph"><base></mark>`. +Read the private record at <private record URL> and the record below. +Follow the completed reproduction's fix size: single publishes one Ticket; spec publishes a Spec with Tickets using `thirdshift-to-spec` and `thirdshift-to-tickets`. +On a public repository, publish one new top issue labelled `needs-triage`. Create that label if missing. On a private repository, reuse the finding's issue as the top issue and preserve its body and evidence; add the bigger fix's Tickets as its native sub-issues. +Every new issue is terse: it says only what the fix changes and links the private record. It carries none of the write-up, trace, evidence, reproduction notes, proof-of-concept test or exploit details, even paraphrased. Keep those in the private record. This overrides the skills' templates. +A Spec's Tickets must be new, open, labelled `ready-for-agent`, linked as native sub-issues with their native blocking links, and have no sub-issues of their own. Read the links back before finishing. +This session changes no repository source, commits and pushes nothing, opens no pull request, and does not implement the fix. thirdshift checks every issue, marks the top issue ready, labels all fix issues security-fix and dispatches its Run or Spec run. +End your final message with exactly `Security fix Ticket: <mark class="ph"><Issue URL></mark>` for single or `Security fix Spec: <mark class="ph"><Issue URL></mark>` for spec, naming the top issue. + +Private record: +<recorded Security finding> + You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. </code></pre> </article> @@ -1079,302 +1158,5658 @@ <h4 id="file-thirdshift-resolving-merge-conflicts-agents-openai-yaml"><code>thir short_description: "Resolve merge and rebase conflicts" </code></pre> </article> - <article class="job-sheet" id="skill-thirdshift-tdd" aria-labelledby="skill-thirdshift-tdd-title"> - <h3 id="skill-thirdshift-tdd-title">thirdshift-tdd</h3> - <p class="sent-by">Used by <a href="/#unit-2">2 · M · Implement</a></p> - <h4 id="file-thirdshift-tdd-skill-md"><code>thirdshift-tdd/SKILL.md</code></h4> + <article class="job-sheet" id="skill-thirdshift-security-audit" aria-labelledby="skill-thirdshift-security-audit-title"> + <h3 id="skill-thirdshift-security-audit-title">thirdshift-security-audit</h3> + <p class="sent-by">Used by the Security audit, in a Security run, or a Security review</p> + <h4 id="file-thirdshift-security-audit-skill-md"><code>thirdshift-security-audit/SKILL.md</code></h4> <pre class="job-text"><code>--- -name: thirdshift-tdd -description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests. +name: thirdshift-security-audit +description: Security guidance and vulnerability review for codebases, APIs, services, CLI tools, libraries, and daemons. Use for security questions, focused reviews, vulnerability research, security audits, or pen tests. Run the complete workflow only for explicit codebase audit or pen-test requests, full/comprehensive/end-to-end reviews, or requested report artifacts. --- -# Test-Driven Development +# Security Audit -TDD is the red → green loop. This skill is the reference that makes that loop produce tests worth keeping: what a good test is, where tests go, the anti-patterns, and the rules of the loop. Every section applies on every cycle: consult them before and during the loop, not after. +Find vulnerabilities that violate a real trust boundary, then give owners the source evidence, safe reproduction, priority, and smallest effective fix. This is a defensive, source-first workflow. A candidate without a concrete affected principal, resource, or security outcome is not a confirmed finding. -When exploring the codebase, read `CONTEXT.md` (if it exists) so test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching. +## Operating modes -## What a good test is +This skill is guidance by default. Loading it does not authorize the complete audit workflow or file creation. -Tests verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't. A good test reads like a specification: "user can checkout with valid cart" tells you exactly what capability exists, and it survives refactors because it doesn't care about internal structure. +- **Guidance mode**: For security questions, focused reviews, methodology, triage, or investigation of specific findings, use only the relevant parts of this skill. Do not automatically run all six phases, create an output directory, or write audit artifacts. You may launch focused agents when useful; they return results to the current task. +- **Full audit mode**: Use the complete workflow when the user explicitly asks to audit or pen-test a codebase, asks for a full, comprehensive, or end-to-end security review, or requests report artifacts. Run all six phases and write the files defined below. -See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines. +If the request could mean either mode, ask one focused question before creating files or starting the complete workflow. -## Seams: where tests go +## Platform terminology -A **seam** is the public boundary you test at: the interface where you observe behavior without reaching inside. Tests live at seams, never against internals. +This skill is agent-neutral: -**Test only at pre-agreed seams.** Before writing any test, write down the seams under test. No test is written at a seam you haven't written down. Have the seams listed in the PR body only. You can't test everything, so agreeing the seams up front is how testing effort lands on the critical paths and complex logic instead of every edge case. +- **Parent** is the agent that coordinates the run and owns shared state. +- **Task tool** is the platform's delegation or sub-agent mechanism. +- **`research` agent** is a delegated agent for focused source exploration and factual verification. +- **`general` agent** is a delegated agent for broad investigation and bounded local execution. +- **`subagent_type:`** in a heading names which of these two delegated agent roles runs that work. -Decide: "What's the public interface, and which seams should we test?" +Use equivalent platform capabilities while preserving role, write-isolation, prompt, and independence boundaries. -When the shape of that interface is itself in question (how deep the module is, where the seam belongs, what the interface should expose), use the `thirdshift-codebase-design` skill for the vocabulary. It is the shared source of the module, interface, depth, seam, adapter, leverage and locality terms, and it is a reference to consult, not a session to run. +## Universal execution safety -## Anti-patterns +These rules apply in both operating modes. Source inspection is read-only. Run target-controlled builds, tests, processes, browsers, emulators, fuzzers, and fixture processing only inside an OS-enforced sandbox that provides all of these controls: -- **Implementation-coupled**: mocks internal collaborators, tests private methods, or verifies through a side channel (querying the database instead of using the interface). The tell: the test breaks when you refactor but behavior hasn't changed. -- **Tautological**: the assertion recomputes the expected value the way the code does (`expect(add(a, b)).toBe(a + b)`, a snapshot derived by hand the same way, a constant asserted equal to itself), so it passes by construction and can never disagree with the code. Expected values must come from an independent source of truth: a known-good literal, a worked example, the spec. -- **Horizontal slicing**: writing all tests first, then all implementation. Bulk tests verify _imagined_ behavior: you test the _shape_ of things rather than user-facing behavior, the tests go insensitive to real changes, and you commit to test structure before understanding the implementation. Work in **vertical slices** instead: one test → one implementation → repeat, each test a **tracer bullet** that responds to what the last cycle taught you. +- no external network; use only an isolated loopback namespace when the check needs local client/server traffic; +- an empty environment populated from an explicit allowlist with safe values, with scratch-local `HOME`, temporary directories, and caches; +- a read-only target and toolchain, with the target-controlled process able to write only inside its assigned `scratch/` directory; and +- explicit low CPU, memory, process, file-size, disk, and wall-clock limits. -## Rules of the loop +The agent, outside the target-controlled process, may make a disposable source copy in an assigned `scratch/` directory when a build must write beside source. In guidance mode, do not retain target-controlled files. In full audit mode, only trusted parent-side code may promote the minimum non-secret result to retained `artifacts/` using the procedure under Write isolation. Never expose a retained output directory (other than the agent's own assigned `scratch/`), another agent's directory, the host home directory, credentials, sockets, or shared services to target code. Do not install dependencies or let builds fetch them. Use only tools and dependencies already available locally. If every control cannot be enforced, do not execute target code: report the missing sandbox capability as a needs-validation blocker and give a safe validation plan. -- **Red before green.** Write the failing test first, then only enough code to pass it. Don't anticipate future tests or add speculative features. -- **One slice at a time.** One seam, one test, one minimal implementation per cycle. -- **Refactoring is not part of the loop.** It belongs to the review stage (see the `thirdshift-code-review` skill), not the red → green implementation cycle. -</code></pre> - <h4 id="file-thirdshift-tdd-agents-openai-yaml"><code>thirdshift-tdd/agents/openai.yaml</code></h4> - <pre class="job-text"><code>interface: - display_name: "TDD" - short_description: "Test-driven red-green-refactor" -</code></pre> - <h4 id="file-thirdshift-tdd-mocking-md"><code>thirdshift-tdd/mocking.md</code></h4> - <pre class="job-text"><code># When to Mock +Use dummy principals, fixtures, and secrets. Do not probe deployed endpoints, external services, shared infrastructure, production identities, other users' data, or live control planes. Do not test availability against a live or shared process, publish artifacts, alter releases, spend paid API quota, or continue beyond the minimum local effect needed to establish a defect. If the decisive fact is outside source or the sandboxed fixture, report it as needing validation. -Mock at **system boundaries** only: +## Full audit setup -- External APIs (payment, email, etc.) -- Databases (sometimes - prefer test DB) -- Time/randomness -- File system (sometimes) +In full audit mode, resolve these values before reconnaissance: -Don't mock: +- **Skill directory**: the absolute directory containing this `SKILL.md`. +- **Target**: the absolute repository root under review. +- **Repo name**: a stable repository identifier from the directory or local Git remote. +- **Output directory**: a new writable directory outside the target, defaulting to `~/security-audit-skill/<repo-name>/run-<N>`, where `<N>` is the next unused integer. Use a directory inside the target only when the user explicitly selects it and the parent verifies that version control ignores the whole directory. Otherwise stop and request an external path. +- **Source ref**: the reviewed commit and whether the worktree is dirty. Do not treat unreviewed generated or modified files as another revision. -- Your own classes/modules -- Internal collaborators -- Anything you control +### Write isolation -## Designing for Mockability +The parent creates and is the only writer of shared run files: -At system boundaries, design interfaces that are easy to mock: +- `run-metadata.json` +- `architecture.md` +- `coverage-ledger.json` +- `findings.json` +- `REPORT.md` +- `FINDINGS-DETAIL.md` +- `NEEDS-VALIDATION.md` -**1. Use dependency injection** +Each hunter or verifier receives a unique root under `<output-dir>/agents/<agent-id>/`, with separate `scratch/` and `artifacts/` directories. Canonical agent IDs match `^[a-z0-9][a-z0-9_-]{0,63}$` and must not equal a Windows device name such as `con`, `prn`, `aux`, `nul`, `com1` through `com9`, or `lpt1` through `lpt9`. Lowercase IDs prevent case-fold collisions. The agent and every target-controlled process may write only to `scratch/`; retained `artifacts/` is parent-owned, is never exposed to the sandbox, and is writable only by trusted parent-side promotion code. Agents may not change shared files, target source, retained artifacts, or another agent's directory. Do not use `/tmp` or the host home directory as a writable fallback. -Pass external dependencies in rather than creating them internally: +Before execution, the parent opens and retains trusted, non-inheritable directory descriptors for the agent's `scratch/` and `artifacts/` roots, and records an allowlist of expected scratch-relative artifact files plus explicit per-file and cumulative byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox and all its processes terminate, trusted parent-side code promotes each allowlisted file separately: -```typescript -// Easy to mock -function processPayment(order, paymentClient) { - return paymentClient.charge(order.total); -} +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, type, link count, or size. +7. For the destination, walk every parent component from the retained artifacts-root descriptor with no-follow directory-relative operations; require each existing component to be a real directory, and create any missing directory exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened destination is a regular file with link count exactly one, and copy from the verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or open or promote a symlink, FIFO, socket, device, directory, hard-linked file, changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch entry; if it is decisive evidence, retain `needs_validation` with the exact promotion blocker. -// Hard to mock -function processPayment(order) { - const client = new StripeClient(process.env.STRIPE_KEY); - return client.charge(order.total); -} -``` +[HUNTING.md](HUNTING.md) and [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) carry this procedure as one identical fenced block for hunter and verifier prompts; it states the same rules in the same order as this list. -**2. Prefer SDK-style interfaces over generic fetchers** +For a reproduced check, record the command, exact test input, sandbox limits, and only the allowlisted environment variable names plus safe non-secret values needed to reproduce it. Never capture or copy the ambient environment, inherited variables, credential values, authentication state, or unrelated host paths. Launch from an empty environment rather than trying to redact one after execution. -Create specific functions for each external operation instead of one generic function with conditional logic: +Before delegation, the parent writes `run-metadata.json` with at least `run_id`, `repo`, `target`, `source_ref`, `profile`, `scope_paths`, `budget` (null if unset), `execution_policy: "sandboxed-source-and-local-only"`, selected companion files, prior-run paths, shared-file owners, and `run_status: "in_progress"`. Update metadata only when those facts change; candidate state belongs in the coverage ledger and `findings.json`. -```typescript -// GOOD: Each function is independently mockable -const api = { - getUser: (id) => fetch(`/users/${id}`), - getOrders: (userId) => fetch(`/users/${userId}/orders`), - createOrder: (data) => fetch('/orders', { method: 'POST', body: data }), -}; +## Full audit planning -// BAD: Mocking requires conditional logic inside the mock -const api = { - fetch: (endpoint, options) => fetch(endpoint, options), -}; -``` +The coverage, prior-run, profile, and budget requirements in this section apply only in full audit mode. -The SDK approach means: -- Each mock returns one specific shape -- No conditional logic in test setup -- Easier to see which endpoints a test exercises -- Type safety per endpoint -</code></pre> - <h4 id="file-thirdshift-tdd-tests-md"><code>thirdshift-tdd/tests.md</code></h4> - <pre class="job-text"><code># Good and Bad Tests +### Coverage and prior runs -## Good Tests +No one pass is complete. Build a deterministic coverage plan before hunting and update it after every agent result. [RECONNAISSANCE.md](RECONNAISSANCE.md) defines the stable coverage units and [HUNTING.md](HUNTING.md) defines coverage-critic waves. The parent alone updates the ledger. -**Integration-style**: Test through real interfaces, not mocks of internal parts. +If prior runs exist, read every compatible `coverage-ledger.json` and `findings.json` before planning the current run: -```typescript -// GOOD: Tests observable behavior -test("user can checkout with valid cart", async () => { - const cart = createCart(); - cart.add(product); - const result = await checkout(cart, paymentMethod); - expect(result.status).toBe("confirmed"); -}); -``` +1. Compare the relevant current source with each prior record and unit. A prior source ref alone is not evidence that a path is unchanged. +2. Carry a prior `confirmed` record into the current candidate set only when its relevant source and conditions are unchanged and its evidence still meets the current contract. Link it to a current ledger unit seeded `planned`, preserve its fingerprint, exclude only that carried root cause from hunters, and send the carried record through the current final verification path; the Phase 3 verifier that re-checks it becomes that unit's assignment owner and moves it to `candidate`. +3. When relevant source for a prior `confirmed` record changed, create a current planned revalidation unit. Do not put that record on the hunter exclusion list. It remains confirmed only if current independent validation establishes the current path and result. +4. Make prior `needs_validation`, `deferred`, `blocked`, `out_of_scope`, and any changed-source unit current work. A still-external `needs_validation` record may be carried only after the current source trace is checked and linked by fingerprint to a current `planned` unit whose verifier re-check supplies its owner and evidence; the record keeps the unresolved blocker. These prior states never suppress a current unit. +5. A prior same-source covered unit may inform priority, but it remains visible in the current ledger. A prior `rejected` record suppresses only the unchanged failed claim, not coverage of its unit; changed evidence creates current work. +6. Read the prior profile and scope. A prior `quick` or scoped ledger contributes only its recorded evidence and gaps, never an implied "rest is fine." -Characteristics: +If no prior ledger exists, say so in the final coverage statement. Never imply that one run exhausts the target. -- Tests behavior users/callers care about -- Uses public API only -- Survives internal refactors -- Describes WHAT, not HOW -- One logical assertion per test +### Run profiles and scope -## Bad Tests +During full audit setup, pick a profile from the user's request or propose one from the target's size and stakes. Record it in `run-metadata.json` (`profile`, `scope_paths`) and state it in the report. The default is `standard`. -**Implementation-detail tests**: Coupled to internal structure. +- **`quick`** — a bounded pass for small targets, re-runs, or a fast first look. Coarsen ledger units to surface × boundary × attack class (subsystem uses the fixed canonical `profile/quick/all-in-scope-subsystems` identifier), run exactly one hunter wave followed by exactly one final coverage-critic pass, and use one fresh verifier per candidate for both candidate validation and final record verification. Do not launch a follow-up hunter wave: record the critic's accepted discoveries and reassignments as `deferred`. +- **`standard`** — the workflow as written. +- **`deep`** — for high-stakes or large targets. Split ledger units per subsystem and lifecycle mode, run critic waves to a clean pass, keep candidate validation and final record verification as separate fresh agents, and give `prior_covered_same_source` units an independent second pass. -```typescript -// BAD: Tests implementation details -test("checkout calls paymentService.process", async () => { - const mockPayment = jest.mock(paymentService); - await checkout(cart, payment); - expect(mockPayment.process).toHaveBeenCalledWith(cart.total); -}); -``` +A **scoped run** audits a subset: named paths, one subsystem, one companion domain, or the diff between two source refs. Seed ledger units only for in-scope surfaces and record everything else as `out_of_scope` — never as `covered`. A scoped or `quick` run must present itself as partial coverage. -Red flags: +Profiles change breadth and redundancy, never the evidence bar. Do not scale away the candidate gate, the source/local execution boundary, `needs_validation` discipline, schema validation, or independent verification of `confirmed` records. -- Mocking internal collaborators -- Testing private methods -- Asserting on call counts/order -- Test breaks when refactoring without behavior change -- Test name describes HOW not WHAT -- Verifying through external means instead of interface +#### Cost budget -```typescript -// BAD: Bypasses interface to verify -test("createUser saves to database", async () => { - await createUser({ name: "Alice" }); - const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]); - expect(row).toBeDefined(); -}); +The ledger makes spend countable: one unit is roughly one hunter assignment, and one surviving candidate is one or two verifier assignments depending on profile. When the user sets a budget — or the parent proposes one for a large target — record `budget` in `run-metadata.json` as a maximum number of agent invocations across all phases. -// GOOD: Verifies through interface -test("createUser makes user retrievable", async () => { - const user = await createUser({ name: "Alice" }); - const retrieved = await getUser(user.id); - expect(retrieved.name).toBe("Alice"); -}); -``` +Apply the strict budget gate before launching any reconnaissance agent. Reserve the four baseline reconnaissance calls, one final post-wave critic for `quick` or one post-wave plus one distinct final-clean critic for `standard`/`deep`, and at least one verifier call. Add focused reconnaissance only after repeating this gate for each extra call. If the requested budget cannot fund that minimum, launch no agent: ask for a larger budget, narrower scope, or different profile. If the request remains unchanged, set `run_status: "incomplete"` with `incomplete_reason: "budget_cannot_fund_reconnaissance_and_reserves"` and report that no audit pass ran. -**Tautological tests**: Expected value restates the implementation, so the test passes by construction. +Spend it in this order: -```typescript -// BAD: Expected value is recomputed the way the code computes it -test("calculateTotal sums line items", () => { - const items = [{ price: 10 }, { price: 5 }]; - const expected = items.reduce((sum, i) => sum + i.price, 0); - expect(calculateTotal(items)).toBe(expected); -}); +1. Count reconnaissance, every post-wave critic, and the separate final-clean critic as agent invocations. +2. **Reserve critics and validation before hunting.** For `quick`, reserve its one post-wave final critic. Before every `standard` or `deep` hunter wave, reserve one immediate post-wave critic plus one distinct final-clean critic. Also reserve verifier cost from the profile (about 1 or 2 agents per expected candidate; when in doubt reserve 30% of the balance after critic reservation). Never assign hunters into either reserve. +3. Assign hunters to units in priority order until the hunting allowance is spent. Spend the reserved post-wave critic immediately after that wave; keep the final-clean and validation reserves intact. +4. Before a later wave, reserve its new post-wave critic again. If the remaining budget cannot cover the required critic calls and validation reserve, launch no hunters from that wave, mark its planned units `deferred` with reason `budget_cannot_reserve_critics_and_validation`, and use the retained final-clean critic to record the resulting gap. -// GOOD: Expected value is an independent, known literal -test("calculateTotal sums line items", () => { - expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15); -}); -``` -</code></pre> - </article> - <article class="job-sheet" id="skill-thirdshift-to-spec" aria-labelledby="skill-thirdshift-to-spec-title"> - <h3 id="skill-thirdshift-to-spec-title">thirdshift-to-spec</h3> - <p class="sent-by">Used by the Architecture review, in an Architect run</p> - <h4 id="file-thirdshift-to-spec-skill-md"><code>thirdshift-to-spec/SKILL.md</code></h4> - <pre class="job-text"><code>--- -name: thirdshift-to-spec -description: "Turn the current session into a spec and publish it to the project issue tracker: no interview, just synthesis of what you've already settled. Only for an Architecture review." -disable-model-invocation: false ---- +Before wave 1, update the pre-recon estimate with seeded units, implied hunter count, mandatory critic calls, validation reserve, and whether the remaining budget covers the plan. If it clearly cannot, say so and propose either a tighter scope or a coarser profile instead of silently thinning evidence. If later facts consume the required final-critic reserve, launch no hunters, mark all planned work deferred, set the run incomplete with reason `critic_budget_exhausted`, and make no complete-coverage claim. -This skill takes the current session context and codebase understanding and produces a spec. Nobody is there to interview or to ask; just synthesize what you already know. +A strict total-agent budget can still be exceeded by an unexpectedly large candidate set or by a material Phase 5 replacement that needs another independent verifier. If the remaining budget cannot validate every candidate, stop hunting, validate candidates in fingerprint order while the budget permits, and set `run_status: "incomplete"` plus `incomplete_reason: "validation_budget_exhausted"`. Keep each unvalidated fingerprint linked to a `candidate` ledger unit with that unresolved reason. Do not put an unvalidated candidate in `findings.json`, relabel it `needs_validation`, or report the run as complete. Phase 6 may produce a partial report only if its first section states that candidate validation is incomplete and lists the affected fingerprints and units. Never exceed a user-set strict budget silently. -The issue tracker and triage label vocabulary should have been provided to you. If `docs/agents/issue-tracker.md` is missing, fall back to `gh issue create`. If `docs/agents/triage-labels.md` is missing, use the label names as written here, creating a label the tracker lacks (`gh label create`). +## Core principles -The spec is all this skill writes. Write nothing to the repository: no spec file, commits or branches. A `CONTEXT.md` or ADR change the spec needs is work for one of its tickets. +### Require a boundary and result -## Process +For every candidate, name the lower-trust principal, accepted input or action, intended control, crossed boundary, affected principal or resource, and concrete observed or owner-observable result. Do not elevate a missing best practice, guessed deployment behavior, generic parser crash, or self-impact into a security finding. -1. Explore the repo to understand the current state of the codebase, if you haven't already. Use the project's domain glossary vocabulary throughout the spec, and respect any ADRs in the area you're touching. +### Use bounded local evidence -2. Sketch out the seams at which the feature will be tested. Existing seams should be preferred to new ones. Use the highest seam possible. If new seams are needed, propose them at the highest point you can. The fewer seams across the codebase, the better - the ideal number is one. +Static analysis establishes the source path. Sandboxed local tests resolve behavior when all execution controls are available: a minimal function harness, existing unit test, small parser fixture, dummy-tenant integration test, locally rendered configuration, or bounded isolated-loopback client. Stop at a wrong return value, unauthorized dummy record, sanitizer finding, policy difference, or other minimum effect. Do not extend the local check beyond the minimum boundary result or produce persistence, post-fault, or concealment material. -Decide the seams yourself and record them, with the reason for each, under Testing Decisions. +### Respect source visibility -3. Write the spec using the template below, then publish it to the project issue tracker. Apply the `needs-triage` triage label and no other: thirdshift marks the spec ready once the session has ended and the plan passes its checks. +Deployment controls, proxy behavior, provider settings, browser headers, identity policy, broker ACLs, packaging, and topology are real controls. If they are required and absent from the repository, do not assume either presence or absence. Use `needs_validation` with the exact missing fact and a safe owner-observed or local plan. -<spec-template> +### Separate priority from certainty -## Problem Statement +Only `confirmed` records receive severity. Likelihood and impact must reflect the demonstrated conditions and result; overall severity cannot exceed demonstrated impact. `needs_validation` means a specific source-grounded boundary hypothesis is blocked, not a low-confidence confirmed vulnerability, and it has no severity. -The problem that the user is facing, from the user's perspective. +Calibrate overall severity with these anchors: -## Solution +- **critical** — an unauthenticated actor gains code execution, full data-store access, or takeover of arbitrary accounts. +- **high** — an actor fully defeats an explicit security control with real consequences: authentication bypass, cross-tenant read or write, stored script execution affecting other users, authenticated code execution, or an unauthenticated remote stop of a shared service. +- **medium** — a real boundary violation with limited blast radius, uncommon preconditions, or consequences confined to a narrow resource set. +- **low** — disclosure of non-secret internals, or an effect requiring sustained effort for minimal gain. +- **informational** — a confirmed but minimal-impact observation, useful mainly as a prerequisite inside a larger finding. -The solution to the problem, from the user's perspective. +The high/medium discriminator: does the demonstrated result fully defeat an explicit control for an action with real consequences, or only weaken it? If you cannot state the concrete damage, the severity is lower than it feels. -## User Stories +### Recommend the smallest effective source fix -A LONG, numbered list of user stories. Each user story should be in the format of: +For each confirmed finding, identify the invariant the code must enforce and the narrowest source change that enforces it at the last trusted decision point. Prefer specific repository-relative changes and regression tests over generic hardening advice. The audit describes fixes; it does not modify target source. -1. As an <actor>, I want a <feature>, so that <benefit> +## Full audit workflow -<user-story-example> -1. As a mobile bank customer, I want to see balance on my accounts, so that I can make better informed decisions about my spending -</user-story-example> +In full audit mode, follow all six phases in order: -This list of user stories should be extremely extensive and cover all aspects of the feature. +1. **Reconnaissance** — map the source, trust boundaries, local build paths, companion selections, prior evidence, and initial deterministic coverage ledger with [RECONNAISSANCE.md](RECONNAISSANCE.md). +2. **Coverage-led hunting waves** — assign isolated hunters from the ledger and collect structured candidate results with [HUNTING.md](HUNTING.md), [ATTACK-CLASSES.md](ATTACK-CLASSES.md), and the selected domain companions. +3. **Candidate validation** — consolidate fingerprints and give every candidate to a fresh source verifier as defined in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md). +4. **Structured output** — write all final `confirmed`, `needs_validation`, and `rejected` records to `findings.json`; validate it with `report-schema.json` and `validate-findings.cjs`, and validate the coverage claim with `validate-coverage-ledger.cjs`. +5. **Independent record verification** — use fresh agents to verify final source claims and reconcile corrections or state changes. +6. **Target-neutral report** — derive `REPORT.md`, `FINDINGS-DETAIL.md`, and `NEEDS-VALIDATION.md` from the final records, with no live-probe instructions. -## Implementation Decisions +Do not end the run before one of exactly two terminal states: (a) all Phase 6 artifacts are written and both validators pass, or (b) `run_status: "incomplete"` is recorded with its exact reason and the gap is disclosed in the report. Never stop mid-phase. -A list of implementation decisions that were made, each with its reason. This can include: +## Anti-patterns -- The modules that will be built/modified -- The interfaces of those modules that will be modified -- Architectural decisions -- Schema changes -- API contracts -- Specific interactions +1. Checklist deviations presented as vulnerabilities. +2. Defense-in-depth advice with no reachable boundary violation. +3. Live or shared-environment testing where bounded local evidence is insufficient. +4. Guessing provider, proxy, browser, identity, or deployment behavior not present in source. +5. Treating intended same-principal authority or self-impact as a cross-boundary result. +6. Reporting a parser or runtime effect stronger than the observed effect. +7. Emitting prose-only hunter results that cannot be deduplicated or verified. +8. Re-reporting carried same-source prior confirmed records or using them as exemplars that anchor the hunt. +9. Assigning severity to `needs_validation` records. +10. Writing the report before independent verification or letting prose and JSON disagree. +</code></pre> + <h4 id="file-thirdshift-security-audit-ai-and-llm-md"><code>thirdshift-security-audit/AI-AND-LLM.md</code></h4> + <pre class="job-text"><code># AI, LLM, and Agent Hunting -Do NOT include specific file paths or code snippets. They may end up being outdated very quickly. +#### When to use this file -Exception: if a prototype produced a snippet that encodes a decision more precisely than prose can (state machine, reducer, schema, type shape), inline it within the relevant decision and note briefly that it came from a prototype. Trim to the decision-rich parts, not a working demo, just the important bits. +Reach for this file when a language model participates in a trust-sensitive decision: chatbots and assistants, RAG pipelines, persistent agent memory, agent/tool-calling loops, MCP servers and clients, code that builds prompts from untrusted input, or code that consumes model output and acts on it. The important data flow is *untrusted content → model or memory → capability, authority, or sink*. -## Testing Decisions +Use this alongside `ATTACK-CLASSES.md`, not instead of it. Transport, access control, query construction, filesystem use, and output rendering remain ordinary trust boundaries. This file covers the model-specific delegation layer. Split large targets by retrieval, memory, tool dispatch, MCP, and output handling. -A list of testing decisions that were made, each with its reason. Include: +## Core discipline (include in every agent prompt for this domain) -- A description of what makes a good test (only test external behavior, not implementation details) -- The seams the feature will be tested at -- Which modules will be tested -- Prior art for the tests (i.e. similar types of tests in the codebase) +``` +- Prompt injection alone is not a finding. Require a code-level boundary failure: content reaches another principal's context, invokes authority the requester lacks, discloses data they cannot read, or drives a sink they cannot reach directly. +- Model output, memory, tool descriptions, and MCP responses are untrusted inputs. Point to the code that grants authority, trusts output, writes durable state, or feeds a sink. +- A guardrail prompt is not a security boundary. Count only deterministic checks, resource-scoped authorization, isolation, binding, and constrained credentials. +- State the attacker, affected principal, effective execution identity, resource, exact action, authority used, and observable impact. An intentional direct request to use the requester's existing authority is not a delegation defect merely because a model executes it. +- Authorization and action binding are separate controls. Attacker-controlled content that causes an action under an affected principal's valid authority is an action-binding failure when that principal did not intentionally request or approve the exact action. +- Classify every candidate as `confirmed` only after source evidence and bounded local validation establish the boundary and result. Use `needs_validation` when a required provider, deployment, model, renderer, or identity behavior is not observable locally. +``` -## Out of Scope +## Context, retrieval, and memory attack classes (subagent_type: `general`) -A description of the things that are out of scope for this spec. +**Indirect injection through retrieved or ingested content** +An attacker can write a RAG document, indexed page, file, email, issue body, tool response, or metadata that enters a different principal's model context. Trace who can write each source, how retrieval scopes it, whose session consumes it, and what capability is enabled there. Check isolation, resource authorization, and binding to the consuming principal's intent separately. The defect is a missing deterministic control, not persuasive text by itself. -## Further Notes +**Cross-session or cross-tenant context bleed** +Conversation history, embeddings, retrieval results, or prompt caches are keyed too broadly. Verify tenant and ACL filters in the query itself and every cache key. A tenant field stored on an object is not enforcement if an alternate query, shared cache, or batch path omits it. -Any further notes about the feature. +**Persistent memory poisoning** +Attacker-controlled content or model summaries are written into memory that later shapes another task, user, or privileged session. Review who may create, update, merge, and delete memory; its provenance and tenant scope; whether low-trust observations become durable instructions or facts; and whether retrieval distinguishes user preferences from tool policy. Memory intentionally saved by a user and used only for that user's intentional, allowed requests is not a cross-boundary finding. -</spec-template> +**Prompt role and provenance confusion** +Prompt assembly lets untrusted text impersonate a system message, prior turn, tool result, policy, or memory record. Look for string concatenation, untyped history, caller-controlled role fields, and serialization round trips that lose source labels. Confirm that the forged provenance changes a deterministic trust decision or reaches a meaningful capability. + +## Tool and action attack classes (subagent_type: `general`) + +**Tool-argument injection into a downstream sink** +Model-produced arguments reach SQL, shell, file, URL-fetch, or privileged APIs without handler-side validation. Treat the tool schema as input parsing, then follow each field from decoded call to sink. Structured output narrows shape; it does not establish authorization, safe paths, safe URLs, or query semantics. + +**Excessive agency and confused-deputy authority** +The agent uses a service identity or broad credential, while the tool handler does not re-check the requesting principal's permission on the named resource. Verify both the effective identity and whether the caller could perform that exact operation through the normal product interface. A shared credential with enforced per-user query scope is not a defect. + +**Action-confirmation and approval binding** +A user approves one described action but execution can use changed arguments, a different resource, a different principal, or a later model turn. An action-binding defect also exists when attacker-controlled content causes a side effect under a victim's valid authority without the victim's intentional request or approval, even if generic authorization permits the victim to perform it. Review whether intent or confirmation binds the normalized tool name, complete argument object, requester, target, amount, expiry, and batch membership. Check retries and resumed sessions: an approval must not authorize a mutated or duplicate side effect. + +**Tool-schema and dispatcher disagreement** +The schema accepts aliases, extra fields, duplicate keys, coercions, nested free-form objects, or out-of-range values that the dispatcher or handler interprets differently. Compare schema validation, canonicalization, generated bindings, and handler defaults. Validate again where values become resource selectors or security-relevant options. + +**Unbounded delegated action loops** +A bounded request can enqueue repeated spend, send, mutation, or external API work without a per-request budget, per-action authorization, cancellation, or idempotency control. Confirm impact on shared cost, quotas, other users, or durable state. Do not test by exhausting a service; use code-level accounting and a locally bounded loop. + +## MCP and sub-agent trust classes (subagent_type: `general`) + +**Sub-agent and MCP trust inheritance** +A delegated task receives the full session, credentials, memory, or capabilities rather than the least authority required. Check the principal and tenant carried into each call, capability narrowing, credential audience, and whether delegated results are treated as untrusted on return. + +**MCP server and tool identity confusion** +Calls or results are routed by attacker-influenceable server names, tool names, request IDs, resource URIs, or model-selected aliases rather than the authenticated connection and outstanding request. Check whether two servers can claim the same tool or resource identity, whether reconnect changes the binding, and whether a response from one server can satisfy another server's pending call. + +**MCP metadata and schema as policy** +Tool descriptions, resource metadata, prompts, completion hints, or schemas supplied by an MCP peer are trusted as policy or authorization. These fields can guide the model but cannot grant capability. Find the deterministic allowlist, server identity check, and handler authorization that remain authoritative when metadata conflicts. + +## Output and disclosure attack classes (subagent_type: `general`) + +**Insecure output rendering** +Model output reaches an executing HTML, Markdown, template, URL, or command sink without the sink's required encoding and policy. For browser rendering, verify auto-loaded resources and CSP or sanitization in `CLIENT-SIDE.md`; renderer behavior outside the repository makes the candidate `needs_validation`. + +**Sensitive context extraction** +The assembled context contains credentials, another user's data, private source, or policy values that themselves grant access, and user-influenced output exposes them. Read prompt assembly and data-fetch code. Disclosure of generic instructions or behavior that does not cross a data boundary is not a finding. + +## Universal moves (apply across the above) + +- Draw four maps first: each execution identity, each capability, every writable context or memory source, and each output destination. Then connect the principal at the start to the authority at the end. +- Start at side-effecting tools and work backward through dispatcher, schema, confirmation, model context, retrieval, and ingestion. Start at durable memory reads and trace every writer. +- Compare direct, queued, retry, resume, batch, and delegated paths for the same action. The strongest gate must apply after arguments are final and before every side effect. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the crossed boundary and observable result: attacker, affected principal or shared resource, execution identity, target, and unauthorized or unrequested action or disclosure. +2. For confused-deputy authority claims, prove the tool lacks requester-and-resource authorization and that the attacker cannot perform the same action normally. For action-binding claims, instead prove attacker-controlled content caused an action under the affected principal's authority that the principal did not intentionally request or approve. Valid generic authorization does not establish that intent. +3. For memory or retrieval claims, cite both the attacker-controlled write and the later cross-principal read or privileged decision. A shared record without a reachable consumer is not enough. +4. For action binding, establish the intentional request or normalized approved object, if any, and compare it with the object the handler uses. Confirm a locally observable unrequested action, mutation, duplicate, or authority change without extending the test into harmful execution. For schema disagreement, compare the normalized validated object with the handler's object. +5. For MCP identity claims, verify the authenticated connection, request correlation, tool namespace, and effective credential. Mark `needs_validation` if external server identity or deployment routing is required. +6. Return `confirmed` findings only with a complete source trace and meaningful result. Return `needs_validation` for a specific unresolved boundary fact and state the bounded local or owner-observed check needed to resolve it. </code></pre> - <h4 id="file-thirdshift-to-spec-agents-openai-yaml"><code>thirdshift-to-spec/agents/openai.yaml</code></h4> - <pre class="job-text"><code>interface: - display_name: "To Spec" - short_description: "Turn a session into a spec" -policy: - allow_implicit_invocation: true + <h4 id="file-thirdshift-security-audit-attack-classes-md"><code>thirdshift-security-audit/ATTACK-CLASSES.md</code></h4> + <pre class="job-text"><code># Attack Classes + +#### Attack classes — choose and split based on Phase 1 + +Select attack classes relevant to the application type. Not every class applies to every codebase. The list below is a starting point; add application-specific classes from Phase 1 and split large codebases per subsystem. Frame work as finding, validating, fixing, and prioritizing vulnerabilities. Keep validation to source review and bounded local fixtures; do not develop payload chains, test availability on live services, or take action in shared environments. + +Use `confirmed` only when source evidence and bounded validation establish the full boundary and meaningful result. Use `needs_validation` when a specific deployment, provider, platform, identity, or runtime fact is unavailable; state the missing fact and the safe owner-observed or local check that resolves it. + +> **Native / binary / kernel targets** (C/C++/Rust-unsafe, kernel modules, parsers and decoders, FFI, concurrent runtimes, binary loaders, JITs, firmware): use the memory-safety, integer/ABI, concurrency, binary-loader, and privileged-interface classes in [MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md). +> +> **AI / LLM / agent targets** (chatbots, RAG, persistent memory, tool-calling agents, MCP servers/clients, prompt assembly, or model-controlled actions): use the context, memory-poisoning, action-binding, tool-schema, MCP-identity, and output classes in [AI-AND-LLM.md](AI-AND-LLM.md). +> +> **HTTP, web, and identity targets** (ordinary web apps, APIs, reverse proxies, CDNs, gateways, custom HTTP parsers, sessions, CSRF, JWT, OAuth/OIDC, SAML, MFA, passkeys, account recovery/linking, API keys, or mTLS): use [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md). +> +> **Client-side and browser targets** (SPAs, browser extensions, embedded webviews, service workers, browser storage, cross-window messaging, CORS, WebSockets, or DOM rendering): use [CLIENT-SIDE.md](CLIENT-SIDE.md). +> +> **Supply-chain and release targets** (dependency resolution, generated inputs, CI, release/signing/promotion, updates, plugins, or extensions): use [SUPPLY-CHAIN-AND-RELEASE.md](SUPPLY-CHAIN-AND-RELEASE.md). +> +> **Cloud and deployment targets** (IAM, infrastructure as code, containers/Kubernetes, service mesh, serverless/edge, ingress, provider events, or runtime configuration): use [CLOUD-AND-DEPLOYMENT.md](CLOUD-AND-DEPLOYMENT.md). +> +> **Protocol, RPC, and messaging targets** (gRPC, GraphQL transports, Protobuf/Cap'n Proto/Thrift, custom protocols, queues, brokers, pub/sub, webhooks, or streaming RPC): use [PROTOCOLS-RPC-AND-MESSAGING.md](PROTOCOLS-RPC-AND-MESSAGING.md). +> +> **Resource-exhaustion and availability targets** (untrusted work can consume shared CPU, memory, disk, connections, workers, queues, quotas, or operator-owned spend): use [RESOURCE-EXHAUSTION-AND-AVAILABILITY.md](RESOURCE-EXHAUSTION-AND-AVAILABILITY.md). +> +> **Data-isolation and lifecycle targets** (multi-tenant stores, caches/search, object links, analytics, export/backup, migration, deletion, retention, or restore): use [DATA-ISOLATION-AND-LIFECYCLE.md](DATA-ISOLATION-AND-LIFECYCLE.md). +> +> **Desktop, mobile, and local-IPC targets** (native apps, deep links, webview bridges, exported components, privileged helpers, local daemons, Unix sockets/XPC/Binder/D-Bus): use [DESKTOP-MOBILE-AND-LOCAL-IPC.md](DESKTOP-MOBILE-AND-LOCAL-IPC.md). + +**Injection** (subagent_type: `general`) +Trace untrusted input from entry point to dangerous sink. What counts as a "dangerous sink" depends on the application: +- Web apps: SQL queries, HTML output, shell commands, template engines, file paths, HTTP redirects, deserialization +- Libraries: any function that processes caller-supplied data without validation — buffer operations, parsers, format strings +- CLI tools: shell command construction, file path handling, environment variable interpolation +- Services: query construction, message serialization, log injection, LDAP/XPATH queries +- Client-side (browser/JS): DOM XSS, prototype pollution, `postMessage`/origin trust, and other browser-side classes — covered by the [CLIENT-SIDE.md](CLIENT-SIDE.md) companion blocks when selected + +Do not stop at the obvious direct paths. Look for indirect injection: data stored safely, then retrieved and used in a dangerous context by different code. Look for injection through field names, keys, headers, and metadata — not just values. Look for injection into secondary systems (logs, caches, search indexes, analytics). + +**Access control** (subagent_type: `general`) +Verify that a caller cannot do something outside its authority. Go beyond checking whether permission checks exist — verify they check the *right* permission for the *right* resource via the *right* mechanism: +- Is there a path to the same state change that checks a different (weaker) permission? +- Can a field in the request body override what the permission system intended to restrict? +- Are there endpoints that gate on authentication but forget authorization? +- Does the same resource have multiple access paths with inconsistent checks? +- What about bulk/batch/export/import operations — do they enforce per-item permissions? + +For complex access models, split into separate agents for auth bypass vs authorization logic. + +**Resource and file handling** (subagent_type: `general`) +- Path traversal (reading/writing outside intended directories) — including through symlinks, encoded sequences, and null bytes +- SSRF (making the application fetch attacker-controlled URLs) — including through redirects, DNS rebinding, and URL parser differentials +- Unsafe deserialization, archive extraction (zip slip), temp file handling +- Memory safety (if applicable): buffer overflows, use-after-free, integer overflow +- Race conditions on file operations (TOCTOU between check and use) + +**Cryptography and secrets** (subagent_type: `general`) +- Weak randomness for security-critical values (tokens, keys, nonces) +- Hardcoded secrets, secrets in logs, error messages, URLs, or client-visible responses +- Broken key derivation, missing HMAC verification, nonce reuse +- Timing side-channels on secret comparison +- Misuse of crypto primitives (ECB mode, unauthenticated encryption, static IVs, etc.) +- What happens when crypto operations fail? Does the error path fall back to no-crypto? + +**Business logic** (subagent_type: `general`) +Hunt logic errors by hand: standard scanners cannot find them, and they yield high-impact findings. For each major workflow: +- **State machine violations**: Can you skip steps? Go backwards? Reach an invalid state? What happens if you replay a completed flow? What about partial failure — if step 2 of 3 fails, is step 1 rolled back? +- **Race conditions with business impact**: Concurrent operations that produce invalid states (double-spend, double-approve, lost updates). Focus on operations that check-then-act non-atomically. +- **Numeric/quantity manipulation**: Negative values, zero values, overflow, precision loss, type coercion between string and number. +- **Access boundary violations**: Not "does the permission check exist" but "is it the right check for the business rule?" Can input to one operation bypass a restriction enforced on a different operation for the same effect? +- **Implicit trust assumptions**: Data from storage, config, other components, or plugins assumed safe because "we validated it on the way in." What if a different code path wrote it? +- **Time-based logic**: Expiry checks, scheduling, rate windows, clock skew. What happens at exact boundary moments? What about timezone differences between components? +- **Default and fallback behavior**: What is the security posture when config is missing? When a feature flag is off? When a dependency is unavailable? When the system is mid-migration? + +**Feature abuse and data leakage** (subagent_type: `general`) +Legitimate features used for unintended purposes. Look for bugs in the design, not only in the code: +- **Export/backup as exfiltration**: Can a low-privilege user trigger an export, snapshot, or backup that includes data above their access level? Can they export other users' data? Does the export include deleted/draft/private content? Revision history that was supposed to be pruned? +- **Import/restore as injection**: Can import overwrite existing data? Can it create records that bypass normal validation? Can it inject content into collections the user has no write access to? Does it respect the same permission model as the UI? +- **Search/filter/sort as oracle**: Can search queries reveal whether content exists that the user cannot directly access? Do filter parameters let users probe statuses, roles, or fields they should not know about? Does sorting by a hidden field reveal its values through result ordering? +- **Enumeration through side effects**: Do error messages differ between "does not exist" and "no access"? Do response times differ? Response sizes? HTTP status codes? Can you enumerate users through password reset, invite, or registration flows? +- **Preview/draft/staging leakage**: Are preview tokens scoped to one item or do they unlock broader access? Can draft content be discovered through search, RSS feeds, sitemaps, or API listing endpoints? Can cache headers cause a CDN to serve private content publicly? +- **Notification/webhook as SSRF**: Can a user set a notification URL, webhook URL, or callback URL that the server fetches? Is it validated against internal networks? What about after a redirect? + +**Chained vulnerabilities and trust boundaries** (subagent_type: `general`) +Individually allowed or contained behavior can become a vulnerability when another component or lifecycle step relies on a stronger guarantee: +- **Multi-step boundary failures**: Map what a low-privilege principal may read, write, invoke, and retain, then connect only concrete outputs to later trust decisions. Confirm each prerequisite and do not assume a downstream effect. +- **Cross-component trust gaps**: Component A validates input and passes it to component B. Compare the exact guarantee A produces with what B assumes, including truncation, type coercion, normalization, tenant scope, and plugin/extension access. +- **Second-order use**: Data safe when stored may become dangerous in a later context. A field name becomes a JSON path, a slug becomes a file path, escaped text enters raw rendering, or a stored string becomes a URL, regex, template, or policy expression. +- **Scope and capability growth**: Token, API-key, plugin, OAuth, MCP, or AI capabilities become broader after delegation, refresh, caching, role change, or composition. Name the concrete operation the resulting principal should not have. +- **Timing and ordering**: Review setup, migration, soft-delete, revoke/cache expiry, check/use, and validate/consume windows. Confirm stale state is accepted before reporting. +- **Rollback and recovery**: Undelete, restore, revision rollback, and cancellation must apply current ownership, validation, and authorization. Confirm which invalid state is restored. + +**Wildcard** (subagent_type: `general`) +You are not given a category. Find vulnerabilities outside the standard classes already assigned. + +Read code that looks boring or disconnected from security. Follow incomplete, experimental, compatibility, and fallback features, but retain the same concrete boundary and validation requirements as every other class. + +Use these starting points, but do not limit yourself to them: +- What is the strangest code in the codebase? Why does it exist? What happens if it is abused? +- Are there any features that feel half-finished, experimental, or bolted on? Those have the weakest security because they got the least review. +- What happens if you use the API in a way the frontend never would? The UI constrains users, but the API does not. What API calls are possible but never made by the client? +- Are there any hidden or undocumented endpoints, parameters, headers, or features? Look at route registrations, middleware, and config for things that are not in the docs. +- What happens when you mix features that were not designed to work together? Localization + preview + caching. Import + plugins + webhooks. OAuth + impersonation + API keys. +- Is there anything interesting in the git history? Reverted security fixes, commented-out auth checks, secrets that were committed then removed (still in history). +- Which valid-account actions affect other users, shared integrity, availability, or operator-owned cost? Verify containment, quotas, authorization, and recovery around those actions. +- Which operations are irreversible or require elevated confirmation? Bind authorization and approval to the final principal, action, and resource. +- What assumptions does the code make about the environment? That the database is local, that the clock is accurate, that DNS is trustworthy, that the filesystem is case-sensitive? +- Look at the test files — what are they **not** testing? Compare the edge cases the developer thought about (tests exist) with the ones they did not (no tests). + +Pursue anomalies inside your assigned scope until the invariant is settled. If something looks strange, read it until you can state whether it is safe. If a function has a comment explaining why it is safe, verify the explanation. If a variable is named `temp` or `hack` or `legacy`, read it closely. + +**Obvious things** (subagent_type: `general`) +Other agents hunt subtle bugs. This agent checks the basic exposures that are easy to overlook because everyone assumes someone else already checked them: +- Are there any hardcoded passwords, API keys, tokens, or secrets in the source? (grep for `password`, `secret`, `apikey`, `token`, `Bearer`, `-----BEGIN`, common default passwords) +- Are there any TODO/FIXME/HACK/XXX comments that reference security? (`TODO: add auth`, `FIXME: validate input`, `HACK: skip permission check`) +- Is debug mode / dev mode properly gated? Can it be enabled in production via environment variable, query parameter, or header? +- Are there test/example/seed credentials that work in production? +- Is there a `/debug`, `/admin`, `/test`, `/status`, `/health`, `/metrics`, `/env`, `/.env`, `/config` endpoint that is unprotected? +- Are there any `.env`, `.env.local`, `credentials.json`, `*.pem`, `*.key` files checked into the repo? +- Does the `.gitignore` actually cover secrets, uploads, and local config? +- Are dependencies pinned? Are there known CVEs in the dependency tree? (check lockfiles) +- Are there any `eval()`, `exec()`, `child_process`, `Function()`, `vm.runInContext`, `import()` with dynamic input? +- Are CORS headers set to `*` or overly permissive? Is `Access-Control-Allow-Credentials` combined with a wildcard origin? +- Are cookies missing `HttpOnly`, `Secure`, or `SameSite` attributes? +- Are there any open redirects? (parameters named `redirect`, `return`, `next`, `url`, `goto`, `continue` that feed into redirects without validation) +- Is TLS enforced? Are there any HTTP-only endpoints? +- Are error responses in production returning stack traces, internal paths, or SQL errors? + +This agent does not need to be creative. It needs to be thorough and literal. Check every item. Report each result. + +**Important**: For any finding this agent reports, it must verify the full code path, not just surface appearance. If a cookie is missing `HttpOnly`, check whether the cookie contains security-sensitive data and whether JS needs to read it by design. If an error message contains a field name, check whether the field is ever actually populated with sensitive data. A flag is not a finding — trace the impact before reporting. </code></pre> - </article> - <article class="job-sheet" id="skill-thirdshift-to-tickets" aria-labelledby="skill-thirdshift-to-tickets-title"> - <h3 id="skill-thirdshift-to-tickets-title">thirdshift-to-tickets</h3> - <p class="sent-by">Used by the Architecture review, in an Architect run</p> - <h4 id="file-thirdshift-to-tickets-skill-md"><code>thirdshift-to-tickets/SKILL.md</code></h4> - <pre class="job-text"><code>--- -name: thirdshift-to-tickets -description: Break a plan, spec, or the current session into a set of tracer-bullet tickets, each declaring its blocking edges, published to the issue tracker with native sub-issue and blocking links. Only for an Architecture review. -disable-model-invocation: false ---- + <h4 id="file-thirdshift-security-audit-client-side-md"><code>thirdshift-security-audit/CLIENT-SIDE.md</code></h4> + <pre class="job-text"><code># Client-Side and Browser Hunting -# To Tickets +#### When to use this file -Break a plan, spec, or session into a set of **tickets**: tracer-bullet vertical slices, each declaring the tickets that **block** it. +Reach for this file when meaningful trust decisions or untrusted rendering happen in a browser: single-page apps, browser extensions, embedded webviews, service workers, offline applications, and code that renders attacker-influenceable content into the DOM, receives cross-window messages, or uses browser storage. These paths include sources the server never sees, such as URL fragments, `window.name`, `postMessage`, and previously cached content. -The issue tracker and triage label vocabulary should have been provided to you. If `docs/agents/issue-tracker.md` is missing, fall back to the `gh` CLI. If `docs/agents/triage-labels.md` is missing, use the label names as written here, creating a label the tracker lacks (`gh label create`). +Use alongside `ATTACK-CLASSES.md`. This file covers browser sources and sinks, origin boundaries, browser persistence, and cross-site state oracles. Use `DESKTOP-MOBILE-AND-LOCAL-IPC.md` for the native side of a webview bridge, and `WEB-PROTOCOL-AND-AUTH.md` for server-side CSRF, sessions, and auth callbacks. -The tickets are all this skill writes. Write nothing to the repository: no ticket files, commits or branches. A `CONTEXT.md` or ADR change the work needs is part of a ticket's work, named in its acceptance criteria. +## Core discipline (include in every agent prompt for this domain) + +``` +- A client-side candidate needs a controllable source and an executing or disclosing sink. Name both and show attacker-influenced data reaching the sink. +- The impact must reach a victim's session, another origin, or shared persistence. Self-injection and disclosure of the attacker's own data are not findings. +- Framework escaping, browser same-origin policy, CSP, COOP/CORP, service-worker scope, and modern noopener defaults are real controls. Verify them before assigning impact. +- Browser storage and caches are shared by origin and may outlive login state. Identify who writes, who reads, and which account, tenant, or worker lifecycle clears each record. +- Use `confirmed` only for complete source evidence plus bounded local browser tests. Use `needs_validation` when renderer, extension permission, deployed header, or browser-policy behavior is required but unavailable. +``` + +## DOM and object-state attack classes (subagent_type: `general`) + +**DOM-based XSS** +Trace `location` fields, `document.referrer`, `window.name`, message data, storage, and browser-controlled document state into `innerHTML`, `outerHTML`, `document.write`, string-evaluating APIs, executable URLs, jQuery HTML APIs, or framework escape hatches. Interpolation escaped by the framework is not a finding. + +**DOM clobbering** +Attacker-injected `id` or `name` attributes shadow a global, form property, configuration object, or initialization flag later trusted by code. Require both a markup path that preserves the attribute and a security-relevant use of the clobbered value. + +**Prototype pollution and gadget chain** +An attacker-controlled key reaches a recursive write such as deep merge or path assignment and modifies prototype state. Then a reachable gadget consumes the polluted property to change authorization, execution, navigation, or rendering. `JSON.parse`, a shallow copy, or pollution without a gadget is not enough. + +## Cross-origin messaging and network attack classes (subagent_type: `general`) + +**`postMessage` origin and source trust** +A handler performs a sensitive action with `event.data` without an exact origin allowlist and, where multiple frames share an origin, the expected `event.source`. On the send side, sensitive data sent to `*` reaches an unintended embedder. Weak substring, prefix, suffix, or unanchored-regex origin matching is not an origin check. + +**Cross-site WebSocket request use** +A WebSocket upgrade accepts ambient cookies from an untrusted origin without an `Origin` check or channel-specific token, allowing the victim's session to read or mutate data. Confirm both the upgrade behavior and a security-relevant message handler. + +**Credentialed CORS trust** +The server reflects or weakly matches `Origin` while allowing credentials and returns sensitive responses. A bare wildcard with credentials is rejected by browsers; report only the actual reflected/allowed origin path and cross-origin data or mutation. + +## Service-worker and browser-storage attack classes (subagent_type: `general`) + +**Service-worker registration and scope takeover** +Attacker-influenceable content can become the registered worker script, control a path that receives an over-broad `Service-Worker-Allowed` scope, or alter update imports without integrity control. Verify the final script URL, response MIME type, origin, scope, and who controls every imported script. A normal same-origin worker with intended scope is not a defect. + +**Service-worker cache and identity confusion** +The worker caches personalized responses without including account, tenant, authorization state, or request mode in its policy, then serves them after account switch or logout. Review fetch-event routing, cache names and keys, navigation fallbacks, cache cleanup, and whether error/offline paths return another user's prior response. + +**Browser-storage disclosure and stale authorization** +Tokens, private responses, draft data, or authorization decisions remain in `localStorage`, `sessionStorage`, IndexedDB, Cache Storage, extension storage, or client state and become readable by another account or less-trusted same-origin component. Storage of a token alone is not a finding; require a realistic reader with less authority, or continued use after revocation/logout. + +**Cross-context storage and broadcast confusion** +`storage` events, `BroadcastChannel`, shared workers, or origin-wide caches carry identity or commands between tabs without binding them to the current session. Check account switching, private/public windows, tenant changes, and stale tabs that can overwrite newer auth state. + +## Cross-site information leak classes (subagent_type: `general`) + +**XS-Leaks and cross-origin state oracles** +An attacker page can distinguish protected cross-origin state through resource load/error events, frame or window state, redirect behavior, timing, cache state, or response size while the browser attaches victim credentials. Require one concrete secret-bearing predicate such as whether a private object, role, or account exists. Generic timing variance or public-resource availability is not a finding. + +**Window and opener state disclosure** +A cross-origin window's permitted metadata or navigation result reveals protected state, or a retained opener/named-window relationship lets an attacker-controlled page influence a privileged navigation. Check COOP, frame protections, `noopener`, exact origin, and whether the observable state is confidential. + +## UI-redress and navigation attack classes (subagent_type: `general`) + +**Clickjacking** +A framed, state-changing action lacks effective `frame-ancestors`, `X-Frame-Options`, or equivalent UI isolation. Require the sensitive action and confirm it can complete in the framed state; missing headers on read-only content are hardening notes. + +**Client-side navigation confusion** +A client source controls redirect or navigation without scheme and destination policy, including executable `javascript:` or `data:` destinations. Reverse tabnabbing applies only where code explicitly keeps `window.opener`, uses `window.open` without isolation, or supports a browser without implicit `noopener`. + +## Universal moves (apply across the above) + +- Start from DOM, navigation, worker, message, and storage sinks, then trace backward to browser-only and server-controlled sources. Record the browser policy that should stop the path. +- Test account switch, logout, worker update, offline fallback, and stale-tab state with a local test origin and dummy accounts. Do not use production users, origins, or shared services. +- For XS-Leaks, list only predicates proved by source and local browser behavior. Then identify the response headers or rendering choice that would remove the oracle. + +## Validation rules (apply before reporting ANY finding here) + +1. Cite the source, sink, browser policy, affected origin/session, and observable mutation or disclosure. +2. For prototype pollution, prove the recursive write and a security-relevant gadget. For DOM clobbering, prove the markup survives and the shadowed value is used. +3. For service workers and storage, prove lifecycle reachability: an attacker-controlled write or cache entry must reach a different account, tenant, or later authorization state. +4. For messaging, CORS, WebSocket, and XS-Leaks, show exact origin/source validation and the protected state or action exposed. Confirm that CSP, COOP/CORP, cookies, and SameSite policy do not already block it. +5. Return `confirmed` findings only with a complete client path and bounded local evidence. Return `needs_validation` with the precise deployed header, extension permission, browser version, or renderer behavior an owner must verify. +</code></pre> + <h4 id="file-thirdshift-security-audit-cloud-and-deployment-md"><code>thirdshift-security-audit/CLOUD-AND-DEPLOYMENT.md</code></h4> + <pre class="job-text"><code># Cloud and Deployment Hunting + +#### When to use this file + +Reach for this file when the repository defines cloud identity, infrastructure, containers, Kubernetes, service mesh, serverless functions, edge workers, ingress, object storage, managed services, or environment-specific configuration. This domain asks whether deployed components receive the intended identity, isolation, network reachability, secrets, and policy. Source often expresses intent rather than live fact, so separate source-confirmed defects from deployment validation needs. + +Use `SUPPLY-CHAIN-AND-RELEASE.md` for build and promotion trust, `WEB-PROTOCOL-AND-AUTH.md` for HTTP proxy semantics, and `DATA-ISOLATION-AND-LIFECYCLE.md` for data-store tenant scope. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Do not infer a live exposure from a manifest alone. Establish which environment consumes it, what defaults or overlays modify it, and whether the source path is active. +- Map each workload's identity to specific operations and resources. Broad policy is a finding only when lower-trust input can reach an unauthorized action. +- Ingress, proxies, service mesh, metadata services, and admission policy are real boundaries, but only count a control when its configuration and attachment are visible. +- Secret references are not secret disclosure. Require a lower-trust reader, output, artifact, log path, or unsafe fallback. +- Use `confirmed` for active in-repo configurations and local rendering/policy validation. Use `needs_validation` for account policy, network attachment, runtime admission, hosted metadata, or drift that needs owner observation. +``` + +## Workload identity and IAM attack classes (subagent_type: `general`) + +**Workload identity overreach** +A workload, pod, function, edge worker, or node identity can act on tenants, accounts, resources, or APIs beyond its role, and untrusted request or job input selects that target. Review cloud policy conditions, resource patterns, service-account attachment, namespace mapping, and fallback credentials. + +**Cross-account or cross-tenant role confusion** +Role assumption, external IDs, token exchange, workload federation, or resource policies accept identity claims not bound to the intended source account, audience, repository, namespace, or workload. Establish both trust policy and caller-controlled claim. + +**Application authorization delegated to cloud metadata** +An app trusts caller-supplied identity headers, tags, labels, account IDs, or resource metadata without verifying they came from the cloud control plane or a trusted proxy. Cloud IAM and application authorization are separate checks. + +## Ingress, network, and control-plane attack classes (subagent_type: `general`) + +**Unexpected service or management-plane reachability** +An ingress, service, listener, security group, load-balancer annotation, port mapping, or server bind exposes an admin, debug, metrics, node, control-plane, or internal API to a lower-trust network. Missing network controls alone are `needs_validation`; a repository-controlled public route to a sensitive handler can be `confirmed`. + +**Trusted-proxy and mesh identity bypass** +A backend accepts forwarded identity, mTLS subject, or authorization metadata from peers outside the intended ingress/sidecar, or an alternate port and health/legacy path bypasses the mesh. Verify header stripping, peer reachability, and fail-open behavior when the proxy is absent. + +**Metadata and internal-service reachability** +An untrusted URL, destination, or protocol selection reaches instance/container metadata, control-plane sockets, or internal APIs with workload credentials. Trace URL parsing and redirect handling under `ATTACK-CLASSES.md`; here establish deployed network, metadata-version, and identity boundaries. + +## Container and orchestration attack classes (subagent_type: `general`) + +**Host or control-plane capability exposure** +A lower-trust workload can select privileged mode, capabilities, host namespaces, host paths, device mounts, container runtime sockets, or service-account tokens that cross into node/control-plane authority. Bare absence of seccomp or read-only filesystem is hardening unless a reachable operation crosses that boundary. + +**Admission and policy path inconsistency** +One deployment route enforces image identity, namespace, resource, secret, or privilege policy while another controller, job, upgrade, restore, or compatibility path does not. Confirm the alternate route and resulting deployed object. + +**Namespace and label trust confusion** +Network, admission, secret, or workload-identity policy relies on labels, annotations, names, or namespaces that a less-trusted principal can set. Compare who controls selectors with what authority matching grants. + +## Configuration and secret lifecycle attack classes (subagent_type: `general`) + +**Security-control precedence drift** +Development values, chart defaults, environment variables, command-line flags, feature gates, sidecar injection, or per-region overlays disable authentication, transport security, tenant scoping, or audit policy in a deployed environment. Render the final configuration for each maintained deployment, not just the base file. + +**Secret exposure across workload boundaries** +Secrets enter logs, crash reports, process arguments, shared environment, broad volumes, build outputs, service discovery, or read APIs accessible to another workload or tenant. Check secret type and authority; a public endpoint or key ID is not a credential. + +**Credential renewal and outage fallback** +Failure to mount, refresh, rotate, or revoke a workload credential causes stale credentials to remain active or an app to accept a less trusted identity mode. Review startup, readiness, reconnect, and cached-client behavior. + +## Managed storage, events, and edge attack classes (subagent_type: `general`) + +**Object and signed-URL policy confusion** +Bucket/container policy, object keys, CDN origins, or signed URLs fail to bind principal, operation, object namespace, audience, or expiry. Review list/version operations and write paths as well as reads. + +**Event-source identity confusion** +A function or worker trusts event body fields as source identity without validating provider-signed envelope, subscription/topic, account, region, and replay state. Compare push, pull, retry, and dead-letter paths. + +**Edge/runtime boundary mismatch** +An edge or serverless runtime assumes a secret, API, filesystem, isolation, or tenant policy that differs from the origin runtime, and fallback to origin changes authority or cache behavior. Confirm which configuration selects each path. + +## Universal moves (apply across the above) + +- Render every maintained environment and make a matrix of external port, workload identity, network peers, mounted secrets, and cloud resources. Differences require an owner or policy explanation. +- Follow a lower-trust request, object, label, or event into cloud policy. Show which workload credential performs the final operation and what condition should scope it. +- Diff normal deploy, migration, restore, node maintenance, failover, and local/emulator paths. Review behavior when mesh, admission, identity, secret, or policy service is unavailable. + +## Validation rules (apply before reporting ANY finding here) + +1. Establish the active source path and effective deployment object; otherwise use `needs_validation` and state which rendered manifest or owner-observed attachment is missing. +2. Name the lower-trust caller/workload, cloud or application identity, controllable selector, affected resource, and unauthorized operation or disclosure. +3. Verify provider and orchestrator defaults at the pinned version. Do not assume a public IP, reachable metadata service, permissive firewall, or absent admission attachment. +4. Local validation may render templates, evaluate policy, inspect container/user namespaces in an isolated fixture, or run an emulator with dummy identities. Do not probe live endpoints or alter shared cloud resources. +5. Return `confirmed` only with a complete active source trace and concrete boundary result. Return `needs_validation` with the exact deployed policy, identity attachment, overlay, network, or drift observation needed. +</code></pre> + <h4 id="file-thirdshift-security-audit-credits-md"><code>thirdshift-security-audit/CREDITS.md</code></h4> + <pre class="job-text"><code># Upstream and licence + +Copied from Cloudflare's [security-audit-skill](https://github.com/cloudflare/security-audit-skill), at commit [`c1c8a8c1471069fb0e188eeaff69b8e8db6564a8`](https://github.com/cloudflare/security-audit-skill/tree/c1c8a8c1471069fb0e188eeaff69b8e8db6564a8). + +The skill and its supporting files are unchanged except for the `SKILL.md` frontmatter name, changed to `thirdshift-security-audit` (ADR 0012). + +Used under the MIT License. Cloudflare's copyright notice and licence are kept in [LICENSE](LICENSE) in this directory. +</code></pre> + <h4 id="file-thirdshift-security-audit-data-isolation-and-lifecycle-md"><code>thirdshift-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md</code></h4> + <pre class="job-text"><code># Data Isolation and Lifecycle Hunting + +#### When to use this file + +Reach for this file when the target stores multi-tenant or access-controlled data, derives search/index/cache/analytics copies, issues object links, exports or restores records, migrates schemas, or promises deletion, revocation, and retention behavior. This domain follows one data item through every copy and state transition. Use `ATTACK-CLASSES.md` for endpoint-level access control and `CLOUD-AND-DEPLOYMENT.md` for provider-level storage policy. + +Split large targets by primary storage, cache/search, object/blob storage, analytics/logging, export/backup, deletion/revocation, and migration. + +## Core discipline (include in every agent prompt for this domain) + +``` +- A tenant or owner field on a record is not isolation. Find the query, key, path, policy, or row-level control that enforces it for each read and write path. +- Trace derived copies. Sanitized primary data can become unsafe in search, cache, analytics, export, previews, logs, replicas, and backups with different ACL and retention rules. +- Deletion and revocation are lifecycle contracts. Check current, historical, cached, indexed, exported, restored, and queued copies within the product's stated boundary. +- Privacy or retention preference is not automatically a security vulnerability. Require an explicit data-access boundary or deletion/revocation guarantee and an unauthorized reader or later operation. +- Use `confirmed` for complete source-visible lineage and bounded dummy-tenant tests. Use `needs_validation` when external storage policy, retention, CDN behavior, replica lag, or backup access is unavailable. +``` + +## Tenant and object-isolation attack classes (subagent_type: `general`) + +**Missing tenant or owner enforcement** +A read, update, delete, list, count, or bulk query identifies an object without binding it to the authenticated tenant/owner, or trusts body fields to supply that identity. Compare direct lookup, nested relationship, background, admin, import, and legacy paths. + +**Composite-key and namespace collision** +Cache keys, object paths, database uniqueness, search document IDs, temporary files, or deduplication keys omit tenant or environment. Two principals can overwrite or retrieve the same logical key even though application records carry separate owners. + +**Policy and query disagreement** +Row-level policy, ORM default scopes, authorization filters, and raw/bypass clients apply different predicates. Check joins, aggregates, aliases, views, transactions, `unscoped` or service clients, and error paths where context is missing. + +**Blob and signed-reference overreach** +Object keys, attachment IDs, version IDs, shared links, or signed URLs permit operations or namespaces beyond the issuing principal's access, or remain valid after the underlying ACL changes. Bind operation, exact object/version, audience, expiry, and tenant. + +## Derived-data and disclosure attack classes (subagent_type: `general`) + +**Search, cache, and index ACL drift** +A primary record's ACL or lifecycle changes without invalidating a searchable, cached, embedded, thumbnail, RSS, preview, or index copy. Validate filtering at retrieval time as well as document ingestion and invalidation. + +**Analytics, logs, traces, and diagnostics as alternate readers** +Private content or credentials are emitted into systems with broader access, longer retention, or tenant mixing. Confirm the data class and realistic reader; field names, public identifiers, and operator-only content under intended policy are not enough. + +**Enumeration and aggregate oracles** +Counts, filters, ordering, errors, unique constraints, timings, notification behavior, or existence checks disclose protected object or account state. Require a concrete confidential predicate and observable distinction, not general response variance. + +## Export, backup, restore, and migration attack classes (subagent_type: `general`) + +**Export and backup scope expansion** +An export, snapshot, portability package, report, or backup includes other tenants, inaccessible object fields, soft-deleted data, secret values, or history above the requester's access. Check per-item authorization after selection and authorization to download the final artifact. + +**Import and restore authority expansion** +Restore/import bypasses owner, schema, ACL, uniqueness, or validation rules, overwrites existing resources, or recreates records in a tenant the requester cannot write. Validate archive contents as untrusted and authorize the resulting operation rather than trusting prior provenance. + +**Migration default and ownership confusion** +Old records lack tenant/ACL/lifecycle fields, incompatible IDs collide, or partial rollout makes new and old readers apply different defaults. Review backfill, dual-read/write, compatibility, rollback, and resumed-migration paths. + +**Backup and replication boundary drift** +Encryption keys, storage accounts, cross-region replicas, restoration environments, or support snapshots have broader identity or tenant scope than primary data. Source can confirm only in-repo policy; hosted access and retention require `needs_validation`. + +## Deletion, revocation, and lifecycle attack classes (subagent_type: `general`) + +**Soft-delete and tombstone bypass** +Direct lookup, search, relation traversal, object link, background processor, or restore ignores the lifecycle predicate and returns or acts on a deleted/revoked record. Check whether soft-deleted identifiers can be re-registered before all references are gone. + +**Stale authorization and derived copy use** +Membership removal, ACL update, consent withdrawal, secret revocation, or role downgrade does not invalidate sessions, caches, subscriptions, jobs, or materialized data that continue to authorize future operations. + +**Retention and queued-work overrun** +Deletion completes in primary storage while queued processors, retries, exports, analytics, or generated artifacts recreate or retain the data beyond the promised boundary. Find idempotent deletion and tombstone propagation. + +**Restore reintroduces invalid state** +Backup, undo, undelete, or replica recovery restores data, credentials, memberships, or permissions that current policy no longer allows. Re-authorize restored state and reapply lifecycle changes made after the snapshot. + +## Universal moves (apply across the above) + +- Pick one protected record and draw primary write, query, cache, index, event, export, backup, deletion, and restore paths. Mark principal and tenant at every edge. +- Compare two dummy tenants through the same local service methods, then repeat after ACL change, deletion, account switch, and restore. Do not use real user data. +- Start at bypass clients, background jobs, migrations, global uniqueness, and cache keys. These paths commonly omit request-scoped identity that interactive endpoints carry. + +## Validation rules (apply before reporting ANY finding here) + +1. Name attacker or lower-trust principal, protected data/state, affected owner/tenant, alternate copy or operation, and unauthorized disclosure or mutation. +2. Cite both intended source-of-truth policy and the path that omits or disagrees with it. Confirm another layer does not enforce the same tenant/lifecycle condition. +3. Use local dummy tenants and non-sensitive fixtures to prove cross-scope access or stale lifecycle behavior. Stop at the minimum observable record or operation. +4. If external cache, object storage, replicas, analytics, backup, or retention policy is required, classify `needs_validation` and state the owner-observed check. +5. Return `confirmed` only with complete lineage and concrete boundary impact. Return `needs_validation` with the exact unresolved storage, ACL, invalidation, retention, or restore fact. +</code></pre> + <h4 id="file-thirdshift-security-audit-desktop-mobile-and-local-ipc-md"><code>thirdshift-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md</code></h4> + <pre class="job-text"><code># Desktop, Mobile, and Local IPC Hunting + +#### When to use this file + +Reach for this file when the target is a desktop or mobile app, privileged helper, updater, local daemon, webview host, deep-link handler, browser native-messaging host, or local IPC client/server. Relevant untrusted actors may be a downloaded document, remote web content, another local app, another OS user, a sandboxed process, or a lower-privilege account. State that starting capability instead of treating all local users as equivalent. + +Use `CLIENT-SIDE.md` for browser-side webview behavior, `MEMORY-SAFETY-AND-BINARY.md` for native memory and loader safety, and `SUPPLY-CHAIN-AND-RELEASE.md` for update authenticity. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Establish the realistic local or remote-content attacker: another app, another OS user, a sandboxed child, an untrusted document, or a remote origin. Self-harm within the same account and authority is not a boundary violation. +- Paths, process names, bundle/package IDs, and claimed sender fields are not peer authentication. Use OS peer credentials, code identity, capability handles, or protected channel state. +- The native bridge or helper must authorize each operation and final resource after parsing. A trusted UI or broker does not make attacker-influenceable arguments trusted. +- OS sandbox, signing, entitlements, permissions, keychain ACLs, exported-component policy, and prompt behavior are real controls when pinned and visible. +- Use `confirmed` for source evidence plus bounded local/emulator tests. Use `needs_validation` when signing, manifest merge, OS version, device policy, installer ACL, or packaging is required but not observable. +``` + +## Deep-link, callback, and navigation attack classes (subagent_type: `general`) + +**Custom-scheme and deep-link ambiguity** +Another app or page can invoke a route that mutates state, imports data, completes authentication, or selects an account without a current-session and one-time callback binding. Review URI normalization, duplicate query fields, scheme/host/path matching, exported activity/handler policy, and stale/replayed links. + +**App and account handoff confusion** +OAuth, SSO, magic-link, invite, device pairing, passwordless, or payment callbacks return to the wrong installed app, profile, tenant, or pending transaction. Bind state to the initiating app identity, current session, account, provider, operation, and expiry. + +**File-open and intent authority confusion** +An associated file, share intent, drag/drop item, pasteboard/clipboard record, notification action, or open-file event triggers a privileged operation without confirming content type, sender trust where applicable, current user intent, and final target. + +## Webview and native-bridge attack classes (subagent_type: `general`) + +**Navigation-origin to bridge confusion** +Remote or attacker-controlled frames can reach a JavaScript/native bridge intended only for packaged content. Validate origin at call time and after every navigation, redirect, subframe creation, popup, and error/fallback page. URL-prefix checks and initial-load checks are insufficient. + +**Over-broad native bridge capabilities** +Web content can select arbitrary files, commands, IPC methods, credentials, or system actions through a generic bridge. Check method allowlists, normalized arguments, user/tenant authority, gesture/confirmation requirements, and return-value disclosure. + +**Webview file and universal access** +Remote content can read app-local files, privileged custom schemes, or internal origins because file access, universal access, mixed content, debug interfaces, or custom protocol handlers join origins unexpectedly. Missing a restrictive setting without reachable protected content is hardening. + +## Local IPC and exported-component attack classes (subagent_type: `general`) + +**IPC peer-authentication gaps** +Unix sockets, named pipes, XPC, Binder, D-Bus, native messaging, RPC, shared memory, or loopback listeners accept a lower-trust peer without checking OS credentials, code identity, sandbox token, or channel ownership. Require a meaningful method or disclosure behind the channel. + +**Claimed principal versus channel identity** +The authenticated process/channel belongs to one app or user, but request fields select another user, tenant, profile, or capability. Bind each method and resource to the peer credential rather than a caller-declared identifier. + +**Exported service, activity, receiver, or provider overreach** +A mobile component or local automation endpoint is externally invokable and performs an operation intended for the app itself. Review final merged manifests, intent filters, permission/signature level, path grants, and alternate aliases. Manifest status unknown after packaging requires `needs_validation`. + +**IPC lifecycle and correlation confusion** +Predictable request IDs, reused handles, stale channels, inherited descriptors, world-writable socket paths, or restart behavior lets one peer answer, cancel, or reuse another peer's operation. Review creation permissions and cleanup of socket files, locks, ports, and shared mappings. + +## Privileged-helper and local-file attack classes (subagent_type: `general`) + +**Privileged helper as confused deputy** +A low-privilege caller can select a privileged command, file, service, user, or system setting without per-operation authorization. Review sudo/polkit/UAC/XPC helper rules and ensure the helper independently validates normalized arguments. + +**Install, update, and repair path trust** +A privileged installer/helper reads manifests, scripts, packages, symlinks, working directories, or repair state writable by a lower-trust actor after authorization. Bind authorization to immutable content and safe destination paths. + +**Local file ownership and TOCTOU** +The app checks a file/path then follows replacement, symlink, mount, or case/normalization changes during a privileged read/write. Use descriptor-relative operations and verify final ownership. Focus `MEMORY-SAFETY-AND-BINARY.md` on parsing after the file is opened. + +**Credential-store and local-secret boundary mismatch** +A keychain/keystore item, token file, backup, log, clipboard, notification preview, or local config is readable by another app/profile/user with less authority. Plaintext readable only by the same intended OS account is not automatically a vulnerability; state the lower-trust reader and credential power. + +## Application-state and device-lifecycle attack classes (subagent_type: `general`) + +**Account switch, logout, and device restore leakage** +Cached data, background tasks, widgets, notifications, local databases, webview storage, or biometric approvals survive logout/account change and appear under a later account. Review backup/restore and multi-profile behavior. + +**Pending-action and user-presence confusion** +Notification, widget, shortcut, share sheet, biometric prompt, or deferred operation authorizes a different action than displayed, executes after expiry, or uses another profile's pending state. Bind confirmation to normalized action, resource, account, and current foreground state. + +## Universal moves (apply across the above) + +- Enumerate every process, app component, local endpoint, URI scheme, file association, webview origin, and helper. Record OS identity, runtime privilege, caller, and callable operation. +- Read final packaging inputs: merged manifest, entitlements, installer rules, native-messaging registration, protocol handlers, and ACL creation. Source declarations can be overwritten downstream. +- Validate with dummy profiles and non-sensitive local fixtures on an isolated machine/emulator. Do not interact with other users' apps, credentials, or production services. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the attacker starting capability, OS/app principal crossed, entry channel, accepted argument or state, and unauthorized operation or disclosure. +2. Confirm OS sandbox, peer credential, signing, entitlement, permission, user-consent, and installer controls that apply. Unknown packaging/runtime facts require `needs_validation`. +3. For webview bridges, cite both navigation/origin control and privileged native sink. For IPC, cite peer authentication and per-resource authorization. For helpers, verify final normalized destination. +4. Keep local tests bounded and use dummy content/accounts. Stop after proving the boundary result; do not extend proof into persistence or broader system modification. +5. Return `confirmed` only with a complete source and local evidence chain. Return `needs_validation` with the exact OS, manifest, signing, ACL, or device-lifecycle fact required. +</code></pre> + <h4 id="file-thirdshift-security-audit-hunting-md"><code>thirdshift-security-audit/HUNTING.md</code></h4> + <pre class="job-text"><code># Vulnerability Hunting + +### Phase 2: Run coverage-led hunting waves + +The parent assigns `planned` ledger units to `general` agents. Use enough focused hunters to cover the units without combining unrelated boundaries. One hunter may own closely related units in one subsystem; no unit may be silently unassigned because of an agent-count limit — a unit the budget cannot reach is explicitly `deferred` with reason `budget_cannot_reserve_critics_and_validation`. + +When a budget or profile caps hunter count, assign units in priority order and record the ordering rationale in the ledger. Rank by: (1) unauthenticated or lowest-trust entry surfaces before authenticated ones; (2) boundaries protecting the most valuable resources (credentials, cross-tenant data, code execution, release authority); (3) prior-run gaps, revalidation targets, and changed source before same-source re-passes; (4) units whose class historically yields confirmed findings for this target type over speculative ones. Ties break lexicographically by `coverage_id` so runs stay deterministic. + +Before launch, the parent changes assigned units to `in_progress`, sets a canonical lowercase `agent_id`, and creates that agent's `scratch/` and parent-owned `artifacts/`. Hunters read source and parent-provided context, write only to their unique `scratch/`, and return one structured result through the Task tool. They never write retained artifacts or edit target source, `architecture.md`, `coverage-ledger.json`, `findings.json`, or another agent's files. + +## Required hunter prompt + +Every hunter prompt contains these parts in this order: + +1. A two-sentence role preamble: the hunter's goal is to find source-grounded security invariant failures in its assigned units, and it must return exactly one JSON object matching the structured-result contract at the end of this prompt. +2. `architecture.md` verbatim. +3. Assigned coverage IDs, subsystem, boundary, repository-relative starting paths, and each unit's assignment block map from `coverage-ledger.json`. +4. The exact selected blocks, copied verbatim: each selected ordinary attack-class block from `ATTACK-CLASSES.md`, and from each selected companion its `Core discipline`, each chosen attack-class subsection, `Universal moves`, and `Validation rules`. Ordinary blocks are self-contained and carry no companion-style `Core discipline`, `Universal moves`, or `Validation rules` sections. Do not send block or companion names alone. +5. Explicit excluded ordinary and companion blocks with a reason for each exclusion. +6. The core hunting method below, followed by the promotion procedure block. +7. The core validation rules below. +8. Carried same-source prior confirmed exclusions, each limited to fingerprint, title, and root cause, plus peer-owned current coverage IDs that this hunter must not duplicate. +9. The unique scratch/artifact paths, safe agent ID, predeclared promotion allowlist and byte limits, and the structured-result contract, including the Structured hunter result block below and the `confirmed` and `needs_validation` branches of `report-schema.json` copied verbatim. + +A prompt may select several companion blocks when the same path crosses several domains. Keep their constraints together. Scope is the hunter's coverage obligation, not permission to duplicate excluded work. If an unexpected different boundary appears, return it under `uncovered` so the parent creates a stable ledger unit and assigns it in the next wave. + +#### Core hunting method — include in every hunter prompt + +```text +## Defensive vulnerability-finding method + +Your goal is to find source-grounded security invariant failures and the smallest fix, +not to expand harm beyond the boundary result. Stay within source review and bounded local execution. +Do not contact deployed endpoints, provider APIs, registries, identity systems, +message brokers, shared services, or other users. Use local dummy data only. + +READ THE CODE AT DEPTH. Follow each assigned input through parsing, identity, +authorization, normalization, state, derived copies, and the final sink. Read sibling, +legacy, batch, retry, cancellation, migration, and error paths that produce the same +effect. Compare sibling controls for equivalence, not only presence, and compare what +one component guarantees with what the next component assumes. + +WORK FROM A CONCRETE INVARIANT: +1. Name the lower-trust principal and starting capability. +2. Name the accepted value, action, state transition, or resource selector. +3. Locate the control that should reject, bind, isolate, limit, or revoke it. +4. Trace the exact source path after that decision. +5. Stop at the smallest affected dummy record, wrong return value, process-integrity + effect, or locally observable shared-resource effect. +6. State a source-level change and regression case that enforce the invariant. + +DEPTH BOUND: trace only paths that can reach your assigned boundary or whose +guarantees that boundary relies on. Stop a line of investigation as soon as the +invariant is settled either way, and record the result in your structured output — +a covered, candidate, or blocked disposition, or an `uncovered` entry — instead of +continuing to search. + +TEST SAD PATHS AND DISAGREEMENTS. Check absent, empty, zero, negative, maximum, +over-limit, duplicate, mixed encoding, stale, revoked, reordered, concurrent, +partially migrated, failed dependency, and rollback state only where the interface +accepts them. Compare canonicalization and units at every parser or policy handoff. +For multi-step issues, treat each output as a prerequisite and do not assume a later +boundary. If any prerequisite is not established, record a blocker. + +When a proposed high or critical candidate reveals a reusable root cause, search paths +owned by the assigned coverage IDs for lexical, structural, and logical variants. +Consolidate the same root cause, but establish each variant's conditions and impact +independently. Do not investigate peer-owned units. Return a variant with no current +coverage unit as `uncovered`. + +USE THE NARROWEST LOCAL CHECK THAT SETTLES THE CLAIM. Target-controlled builds, +tests, processes, browsers, emulators, fuzzers, and fixture processing may run only +inside the parent-approved OS-enforced sandbox. It must disable external networking, +start from an empty allowlisted environment, expose target and tools read-only, permit +writes only to your scratch directory, and apply low CPU, memory, process, file-size, +disk, and wall-clock limits. Isolated loopback is allowed only for a local fixture. +If any control is unavailable, do not execute: return needs_validation with that exact +blocker. Prefer an existing unit test, minimal function harness, dummy-tenant service +call, small malformed fixture, deterministic race schedule, or locally rendered policy. +Do not install or fetch tools. + +Record the exact input, command, limits, and minimum result. For the environment, +record only allowlisted variable names and safe non-secret values needed to reproduce +the check. Never capture the ambient environment, inherited variables, credentials, +authentication state, or unrelated host paths. The target-controlled process writes +only in scratch. After the sandbox and all its processes terminate, only trusted +parent-side code may promote predeclared scratch-relative files, following the +promotion procedure block included verbatim in this prompt. You and target code never +write retained artifacts. If promotion is unavailable or fails for decisive evidence, +return needs_validation with the exact promotion blocker. +Never stress availability, invoke a live target, use a real credential, publish an +artifact, or continue past the minimum observed effect. + +A deployment, browser, provider, broker, OS, proxy, package, secret, or identity fact +outside source is not proof either way. If one such fact is decisive, return a +needs_validation record with the exact missing observation and safe owner-observed check. +``` + +#### Promotion procedure — copy this promotion procedure verbatim into every hunter prompt + +```text +Artifact promotion procedure (trusted parent-side code only): +Reference only for you: the parent performs these steps; you never perform them. + +Before execution, the parent opens and retains trusted, non-inheritable directory +descriptors for the agent's scratch/ and artifacts/ roots, and records an allowlist +of expected scratch-relative artifact files plus explicit per-file and cumulative +byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox +and all its processes terminate, trusted parent-side code promotes each allowlisted +file separately: + +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or + symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with + no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and + within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, + type, link count, or size. +7. For the destination, walk every parent component from the retained + artifacts-root descriptor with no-follow directory-relative operations; require + each existing component to be a real directory, and create any missing directory + exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened + destination is a regular file with link count exactly one, and copy from the + verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or + open or promote a symlink, FIFO, socket, device, directory, hard-linked file, + changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch + entry; if it is decisive evidence, retain `needs_validation` with the exact + promotion blocker. +``` + +#### Core validation rules — include in every hunter prompt + +```text +## Candidate gate + +1. A candidate needs a complete repository-relative source trace and evidence for the + claimed root cause, including the strongest source-visible control. +2. A proposed confirmed record needs a bounded local observed result, meaningful impact + across a stated boundary, complete conditions, and no visible preventing layer. +3. Do not strengthen a crash into code execution, ordinary work into shared availability, + or a same-principal action into privilege gain. +4. If a required fact is not source-visible or locally observable, use + needs_validation. Name exact blockers; do not give it severity or speculative completion. +5. A missing best practice with no affected principal/resource is excluded or hardening, + not a finding. A candidate disproved by source is not needs_validation. +6. Use the same source-derived fingerprint for the same root cause in every state. + It must match `^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$` and must not include a line, + wave, agent, severity, or verdict. +7. Return an empty candidate array when nothing survives these gates. +``` + +## Local validation boundaries + +Local execution is for confirmation, not impact expansion: + +- **Allowed only in the required OS sandbox:** offline builds with present dependencies; isolated-loopback processes using dummy state; unit and integration tests; small fixture processing; sanitizers; bounded fuzz/regression tests; deterministic concurrency checks; local browser/emulator tests with dummy accounts; rendered manifests and policy evaluation with dummy identities; mocked external or paid calls. +- **Disallowed:** live or deployed traffic; requests to services not started for this isolated check; network dependency installation; real accounts or credentials; production data; shared queues, cloud resources, runners, registries, signing or release services; publishing; stress, saturation, or cost generation; any work after the minimum dummy-data boundary result. + +The sandbox starts with an empty environment, gives target code no external network or host writable path, and enforces explicit low resource and time limits for every check, not only checks expected to be expensive. Scratch output remains target-controlled after exit. Promote it only with the no-follow, path-confined, regular-file, bounded-size host procedure in `SKILL.md`. Missing any sandbox or promotion capability does not erase a source-grounded candidate; represent the exact blocker in `needs_validation`. + +## Structured hunter result + +Return exactly one JSON object, with no surrounding prose: + +```json +{ + "units": [ + { + "coverage_id": "one assigned ID", + "disposition": "covered|candidate|blocked", + "reviewed_paths": ["repo/relative/path"], + "checks": [ + { + "agent_id": "canonical owner of this check", + "reviewed_paths": ["repo/relative/path owned by this check"], + "invariant": "specific control checked for this unit", + "method": "source|local", + "result": "what source or the bounded check established", + "artifact": "agents/<agent-id>/artifacts/file for local, null for source" + } + ], + "candidate_fingerprints": [], + "unresolved": [] + } + ], + "candidates": [], + "hardening": ["concrete non-finding note"], + "uncovered": [ + { + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "reason": "why this needs its own deterministic coverage unit" + } + ] +} +``` + +Each `candidates` entry is schema-shaped except that it uses `proposed_verdict` in place of `verdict`: + +- `proposed_verdict: "confirmed"`: include every field required by the `confirmed` branch of `report-schema.json` other than `verdict`: `fingerprint`, title, description, `root_cause`, `intended_behavior`, ordered `trace`, `evidence`, `conditions`, target-neutral `execution`, `remediation`, `severity`, and `confidence`. The execution instructions describe only the bounded local check already performed. `payloads` holds the minimum test input, fixture, or native invocation. `observed_result` records actual local output. Overall severity must not exceed observed impact. +- `proposed_verdict: "needs_validation"`: include every field required by that schema branch other than `verdict`: `fingerprint`, title, description, `claimed_root_cause`, ordered `trace`, `evidence`, nonempty `blockers`, and `validation_plan` with at least one applicable `local` or `deployment` step. Do not invent an inapplicable context. Do not include severity, execution, remediation, reason, or a confirmed `root_cause`. `deployment` is an owner-observed check, not a request to probe a live target. + +Every assigned coverage ID appears exactly once in `units`. A `covered` unit needs an owner, nonempty `reviewed_paths` and `checks`, no unresolved fact, and no candidate. A `candidate` unit has the same owned evidence and is the only state that carries linked fingerprints. A `blocked` unit is an owned partial review with nonempty paths, checks, and unresolved facts but no fingerprint. All source paths are repository-relative, never absolute or traversal paths. A trace with several entries begins at `entrypoint`, ends at `sink`, and labels intermediate steps `propagation`. Every check has its own canonical lowercase `agent_id` and nonempty `reviewed_paths`; the unit-level list is exactly the union of those owned paths. A `source` check uses `artifact: null`. A `local` check uses one successfully parent-promoted regular file beneath `agents/<check.agent_id>/artifacts/`; this permits a verifier to add independently owned evidence without taking ownership from the hunter. Never link scratch, an output-root file, or another check owner's artifact. + +## Parent consolidation and ledger update + +The parent validates each unit result, maps it to exactly one assigned `coverage_id`, and updates only that ledger unit. Reject duplicate or absent IDs, unsafe unit or check agent IDs, source checks with artifacts, and local artifacts that trusted parent-side code did not promote into the check owner's artifacts subtree. Copy the unit's `reviewed_paths`, its `checks` into the unit's `local_checks`, linked artifact paths, candidate fingerprints, and unresolved facts into the ledger. Retain each hunter's `hardening` list in a parent bookkeeping field on the relevant units (outside the semantic fields) so Phase 6 can report it. A failed, malformed, or unsupported conclusion leaves that unit `planned` for reassignment. Untouched budget/profile units become unassigned `deferred` units with empty evidence and a reason; do not hide partial evidence in `deferred`. Run `validate-coverage-ledger.cjs` after the update; an invalid ledger cannot drive another assignment. This per-unit contract allows one hunter to close one unit while returning a candidate or blocker for another. + +Consolidate candidate entries by fingerprint and then by root cause. One root cause that exposes several entry paths or effects is one candidate with the strongest complete trace. Related but independent missing controls use separate fingerprints. Record duplicate fingerprints in the relevant ledger unit and do not send duplicate candidates to validation. + +## Coverage-critic waves + +Immediately after each hunter wave, spend the reserved invocation on one fresh `research` post-wave coverage critic. It receives `architecture.md`, the full coverage ledger including each assignment block map, current candidate fingerprints and states, and the prior-ledger gap summary. It reads source but does not write or run targets. Require exactly this JSON: + +```json +{ + "missing_units": [ + { + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "selected_companion_blocks": ["FILE.md#section"], + "excluded_blocks": [{"block": "FILE.md#section", "reason": "..."}], + "reason": "source-backed coverage gap" + } + ], + "reassign_ids": ["existing-id-that-did-not-close"], + "resolved_prior_leads": ["fingerprint"], + "stop": false +} +``` + +The critic checks for unmapped entry points, unchecked parallel paths, missing lifecycle modes, selected companion classes without a unit, unjustified exclusions, units closed without paths/checks, and prior `needs_validation` or changed-source gaps that no unit addresses. It proposes coverage, not findings. `stop` is the critic's own assessment: `true` only when it accepts no `missing_units` and no `reassign_ids`; the parent's loop condition below, not `stop` alone, decides whether another wave runs. For each fingerprint in `resolved_prior_leads`, the parent marks the linked unit or prior-lead entry resolved and records the critic's source-backed reason. + +The parent rejects proposed units outside the review scope or source/local boundary, derives canonical IDs for accepted units, and deduplicates them against current units. A prior same-source completed unit may supply evidence; prior `deferred`, `blocked`, `out_of_scope`, or changed-source units become current work and never suppress an accepted unit. Fail rather than merge a canonical ID collision. For each legitimate `reassign_id` with live `blocked`, `covered`, or `candidate` evidence, append that exact terminal record to the unit's `attempts` with the critic's source-backed `reassignment_reason`. Preserve its owner, checks, artifacts, fingerprints, and unresolved facts in that archive. Increment the live `wave`; the next hunter must be a fresh owner and receives an `in_progress` unit with empty live evidence. The hunter's terminal result writes only its new evidence into the live fields. Never copy an archived owner's checks or artifacts into the new live attempt. Sort IDs and validate the ledger before another assignment. In `standard` and `deep`, when the post-wave critic reports no accepted `missing_units` or legitimate `reassign_ids` and no `planned` units remain, spend the separately reserved invocation on a distinct final-clean critic. Complete coverage only when that critic also returns no accepted work. If it finds work, queue it and repeat the wave, post-wave critic, and final-clean process. If time or resources force an early stop, mark every untouched unit `deferred`, preserve the critic's reason, and disclose the gap in the report. Never use a silent wave or agent cap as evidence of complete coverage. + +The run profile bounds this loop. A `quick` run has exactly one hunter wave followed by exactly one final critic pass. Add each accepted `missing_unit` to the current ledger and mark it `deferred` with reason `quick_profile_final_critic`. For each legitimate evidence-bearing `reassign_id`, archive the live terminal state in `attempts`, increment `wave`, and set the live state to unassigned `deferred` with empty evidence and reason `quick_profile_final_critic`. Do not launch a second hunter wave or another critic. In a scoped run, the critic still reports out-of-scope gaps it notices, but the parent records them as `out_of_scope` with the critic's reason instead of assigning them. The early-stop rule above is the same mechanism: `quick` is a pre-declared early stop, not evidence of complete coverage. + +A budget bounds it the same way. Before assigning each wave, compare remaining budget against its hunter count, the validation reserve, the immediate post-wave critic, and the retained final-clean critic (`quick` reserves only its single final post-wave critic). Shrink the hunter wave to fit, taking units in priority order. If those mandatory reserves do not fit, launch no hunter from that wave and mark its planned units `deferred` with reason `budget_cannot_reserve_critics_and_validation`. Critic-proposed units enter the same ranked queue rather than extending the budget. If surviving candidates exceed the validation reserve, follow the incomplete-run rule in `SKILL.md`: stop hunting, validate in fingerprint order, retain unvalidated units as unresolved candidates, and never present them as findings. +</code></pre> + <h4 id="file-thirdshift-security-audit-license"><code>thirdshift-security-audit/LICENSE</code></h4> + <pre class="job-text"><code>MIT License + +Copyright (c) 2025-2026 Cloudflare, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +</code></pre> + <h4 id="file-thirdshift-security-audit-memory-safety-and-binary-md"><code>thirdshift-security-audit/MEMORY-SAFETY-AND-BINARY.md</code></h4> + <pre class="job-text"><code># Memory Safety, Binary, and Kernel Hunting + +#### When to use this file + +Reach for this file when the target processes untrusted bytes in a memory-unsafe or privileged context: C/C++/Objective-C, Rust `unsafe`, FFI, kernel modules and drivers, parsers and decoders, network daemons, firmware, binary loaders, language runtimes, and JITs. Use `PROTOCOLS-RPC-AND-MESSAGING.md` for protocol authorization and state-machine logic, and this file for process integrity, memory safety, ABI boundaries, and loader behavior. + +Pick relevant classes from Phase 1 and split large targets by parser, allocator/lifetime, FFI, concurrency, loader, runtime, or privileged interface. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Re-derive every bound and lifetime from attacker-controlled inputs and all callers. Validate against the worst accepted case, not a typical test vector. +- A panic, sanitizer finding, or crash proves a defect only when a realistic untrusted input reaches it. Do not infer memory corruption, code execution, or shared availability impact from a label alone. +- Validate in a local harness with sanitizers, deterministic concurrency tests, existing fuzz targets, and debugger-assisted fault classification. Stop after proving the violated invariant and observable impact; do not develop post-corruption techniques. +- Assembly, JIT code, custom allocators, intra-object accesses, and foreign libraries can escape sanitizer coverage. Identify which relevant instructions are instrumented. +- Classify as `confirmed` only after source evidence and bounded local validation establish the defect and effect. Use `needs_validation` when ABI, allocator, architecture, feature, deployment, or reachability facts remain unknown. +``` + +## Bounds, integer, and representation attack classes (subagent_type: `general`) + +**Out-of-bounds read or write** +A length, offset, index, or terminator reaches a fixed or allocated buffer without a correct bound. Recalculate available headroom after prefixes, alignment, padding, and terminators. Check both source and destination capacity, and whether a short input is read before its declared length is trusted. + +**Integer overflow, underflow, truncation, and signedness** +Review attacker-controlled arithmetic before allocation, copy, loop, indexing, and pointer operations. High-hit patterns include `a - b` with `b > a`, `count * element_size`, additions near the type maximum, negative values converted to unsigned, 64-bit lengths narrowed to 32-bit fields, and sentinel values such as `-1` becoming a large size. Confirm which checked representation is later used. + +**Unit and pointer-depth confusion** +Code mixes bytes, elements, code units, pages, words, wire units, or nested pointer element sizes. Compare the unit at parse, validation, allocation, API boundary, and copy. A bounds check using the same wrong unit as the allocation is still wrong. + +**Uninitialized or partially initialized data** +A buffer, padding, struct field, or vector capacity is returned, compared, hashed, serialized, or passed across a trust boundary before initialization. Require an observable consumer and realistic output length; stack allocation by itself is not disclosure. + +## Lifetime, type, and concurrency attack classes (subagent_type: `general`) + +**Use-after-free, stale view, and double free** +Owners are released while callbacks, wait queues, timers, iterators, borrowed slices, or cached raw pointers can still use them. Review every error, cancellation, close, and realloc path. For embedded notification anchors, each free path must drain or detach all observers. + +**Type confusion and invalid downcast** +A tag, vtable, union discriminator, object kind, or foreign handle is checked differently from the representation later read. Look for unchecked dynamic casts, stale tags after reuse, and serialized types whose validated element differs from the element consumed. Confirm a wrong-type read or write locally without extending the test beyond the violated invariant. + +**Reference-count and ownership races** +Non-atomic retain/release, a check followed by an unlocked use, or inconsistent ownership across threads can free or mutate an object during access. Compare fast, error, shutdown, and compatibility paths for the same lock and ownership rules. + +**Shared-state races and TOCTOU** +Concurrent parser streams, global caches, lazy initialization, signal handlers, and resource teardown can invalidate bounds, policy, or pointers established earlier. Verify the race with a repeatable local schedule, barrier, or thread sanitizer; a hypothetical interleaving without a security-relevant state transition remains `needs_validation`. + +**Lock-order, deadlock, and starvation** +Externally reachable operations acquire locks in inconsistent order or hold them across callbacks and blocking I/O. Report under availability only when bounded input can stop shared progress; otherwise record it for fixing as a concurrency defect. + +## FFI and ABI attack classes (subagent_type: `general`) + +**Pointer-length and ownership contract mismatch** +Caller and callee disagree on who allocates, frees, pins, or mutates a buffer, how long a pointer remains valid, or whether a length is bytes or elements. Trace both sides of every `extern`, CGo/JNI/Python/native binding, and generated wrapper. Check null, zero length, aliasing, and callback retention. + +**Layout, alignment, and enum disagreement** +Foreign code receives a struct, bitfield, packed record, callback signature, integer width, enum, or calling convention that differs by architecture or build flag. Verify `repr`, packing, alignment, endianness, and ABI-specific types. An in-repo declaration mismatch can be confirmed locally; an opaque foreign implementation requires `needs_validation`. + +**Unwind, exception, and thread-affinity violations** +Exceptions or panics cross an ABI that forbids unwinding, callbacks run after teardown, or APIs requiring one runtime thread are invoked elsewhere. Review error conversion and cancellation. Confirm whether the process aborts or state is corrupted before assigning impact. + +## Binary loading and runtime attack classes (subagent_type: `general`) + +**Library, plugin, and executable search-order trust** +A privileged process loads a library, plugin, runtime image, or helper from a path writable by a less-trusted principal, or resolves a bare name through an attacker-influenceable working directory or environment. Compare intended installation ownership with each fallback and compatibility search path. A user loading their own plugin into their own process is not a boundary violation. + +**Missing artifact identity or signature binding** +A loader verifies one file or metadata record but maps a different image because path resolution, file replacement, architecture slices, or embedded resources are not bound to the check. Supply-channel authenticity belongs in `SUPPLY-CHAIN-AND-RELEASE.md`; this class covers the local verification-to-map gap. + +**Malformed binary metadata and relocation handling** +Offsets, counts, sections, relocations, symbols, bytecode, or debug metadata are trusted before range, overlap, and representation checks. Test parsers with bounded local fixtures and sanitizers. Separate memory corruption from a safely rejected malformed file. + +**JIT and generated-code consistency** +Validator, interpreter, optimizer, and generated code disagree about types, bounds, side effects, or lifetime. Diff optimized and unoptimized paths using the same local input. Confirm a process-integrity effect; output variance that stays within language semantics is not a finding. + +**Unload, reload, and teardown safety** +Live function pointers, callbacks, worker threads, or data views survive module unload or runtime reset. Review shutdown and failed-load cleanup as closely as startup. + +## Kernel and privileged-interface attack classes (subagent_type: `general`) + +**User-copy bounds and repeated reads** +A syscall, ioctl, driver, or kernel parser derives a trusted fact from user memory then reads the same mutable address again. Copy the full request once or revalidate the later copy. Also audit size, direction, and access checks at each user-copy primitive. + +**Privileged object lifecycle and dispatch consistency** +Externally reachable objects have unbalanced retain/release, teardown without observer drain, unchecked selector/table indices, or duplicated compatibility paths that omit a guard. Diff each dispatch and free path side by side. + +**Under-authorized powerful interfaces** +A device node, admin socket, helper, or management API validates shape but not the caller's authority over the resource. Establish actual interface ownership and reachability; permissions or sandbox policy outside the repository make this `needs_validation`. + +## Universal moves (apply across the above) + +- Audit fixes and duplicated paths for the same source-to-sink shape. A check in one caller, architecture, protocol role, feature flag, or compatibility path does not protect its siblings. +- Build a table for every parser or FFI boundary: accepted length/type, checked representation, allocation owner, consumer, thread, and teardown. Most native findings are one disagreement in that table. +- Use existing corpora and small locally generated boundary fixtures. Save exact sanitizer/runtime output and the input property that triggers it; avoid large resource consumption and any live target. + +## Validation rules (apply before reporting ANY finding here) + +1. Establish a realistic untrusted entry and exact operation that violates a bounds, type, lifetime, ABI, concurrency, loader, or authority invariant. +2. Classify the observable effect: invalid read, invalid write, stale alias, wrong object, uninitialized output, unauthorized image load, deadlock, or safe process termination. Do not claim a stronger effect than observed. +3. Run the narrowest local harness, existing test, sanitizer, or fuzzer needed to reproduce the effect. Verify sanitizer coverage of the faulting operation and record architecture/build conditions. +4. For concurrency, use a deterministic schedule or sanitizer trace. For binary loading, prove the checked identity differs from the mapped identity and name the lower-trust writer. +5. Return `confirmed` findings only with exact input, source trace, and observed result. Return `needs_validation` for a specific unresolved reachability, ABI, build, deployment, or runtime fact and state the bounded check needed. +</code></pre> + <h4 id="file-thirdshift-security-audit-protocols-rpc-and-messaging-md"><code>thirdshift-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md</code></h4> + <pre class="job-text"><code># Protocols, RPC, and Messaging Hunting + +#### When to use this file + +Reach for this file when the target uses gRPC, GraphQL transports, Cap'n Proto, Thrift, Protobuf, custom binary protocols, streaming RPC, webhooks, brokers, queues, pub/sub, or event buses. It covers peer identity, logical message interpretation, routing, replay, ordering, and delivery semantics. Use `MEMORY-SAFETY-AND-BINARY.md` for parser memory safety, `WEB-PROTOCOL-AND-AUTH.md` for HTTP framing, and `RESOURCE-EXHAUSTION-AND-AVAILABILITY.md` for availability impact. + +Split large systems by producer/consumer pair, external/internal peer role, synchronous RPC, streaming, and asynchronous message path. + +## Core discipline (include in every agent prompt for this domain) + +``` +- "Internal" is not authentication. Name the peer identity at every hop and show how it becomes the application principal used for authorization. +- Schema validation proves message shape, not provenance, resource authority, ordering, or safe values. Follow decoded fields to policy and side effects. +- Broker guarantees and application guarantees differ. Write down retry, ordering, acknowledgement, deduplication, and transaction behavior before evaluating state changes. +- Parser disagreement requires two concrete consumers, schema versions, or wire representations and one security-relevant divergent value. +- Use `confirmed` for source-complete paths plus bounded local producer/consumer tests. Use `needs_validation` for broker ACL, service-mesh identity, topic attachment, or compatibility behavior outside the repository. +``` + +## Framing, schema, and interpretation attack classes (subagent_type: `general`) + +**Message boundary and canonicalization disagreement** +Components disagree on length, compression, duplicate fields, unknown fields, encoding, numeric width, normalization, or envelope/body precedence. Compare generated and custom parsers, gateways, language bindings, and version converters. Confirm which principal, resource, or operation differs after decoding. + +**Union, enum, and default confusion** +Unknown variants, missing discriminators, zero values, default privileges, or compatibility mappings reach code that assumes a validated case. Review exhaustive dispatch, default branches, and how old consumers interpret newly added fields. + +**Envelope and payload identity mismatch** +Authorization uses trusted-looking routing or envelope metadata while the handler acts on a conflicting tenant, account, subject, object, or sender in the body. Identify which source is authoritative and ensure clients cannot override it. + +## RPC identity and authorization attack classes (subagent_type: `general`) + +**Interceptor and method-path inconsistency** +An authn/authz interceptor applies to unary methods but not streams, reflection, health, gateway-transcoded paths, compatibility services, or individual stream messages. Compare every registration and route to the same operation. + +**Peer identity to application-principal confusion** +mTLS, workload identity, bearer metadata, forwarded identity, or broker credentials authenticate a channel, but a caller-controlled field selects the user or tenant. The channel identity and claimed principal must be bound by deterministic policy. + +**Per-item and streaming authorization gaps** +A stream, subscription, batch, or bulk message is authorized once, then later items name different resources or continue after role, membership, or token revocation. Re-check where scope can change and bind subscriptions to their original principal. + +**Callback and reply-correlation confusion** +Predictable, reused, or cross-tenant correlation IDs let a response, webhook, cancellation, or acknowledgment satisfy another caller's pending operation. Bind each outstanding request to authenticated peer, tenant, operation, and lifecycle. + +## Broker and queue isolation attack classes (subagent_type: `general`) + +**Topic, routing-key, and subscription scope gaps** +A publisher or subscriber can select another tenant's topic, wildcard, consumer group, partition, reply queue, or dead-letter route. Check broker-enforced ACLs where visible and application-side namespace construction. Tenant text inside a payload is not isolation. + +**Dead-letter, retry, and diagnostic disclosure** +Messages routed to dead-letter queues, error topics, tracing, or operator views contain secrets or cross-tenant payloads accessible to a lower-trust consumer. Review policy and redaction at the failure path, not just normal delivery. + +**Untrusted producer treated as control plane** +A message body can declare itself an admin event, provider callback, replication record, or migration instruction without an independently authenticated producer and event type. Verify signatures and source/account/audience binding before privileged handling. + +## Replay, ordering, and transaction attack classes (subagent_type: `general`) + +**Duplicate delivery and idempotency gaps** +Retries or redelivery repeat a side effect because deduplication is absent, occurs after mutation, or uses a key that collides across tenants or operations. Confirm the broker's delivery model and the side effect that is not naturally idempotent. + +**Out-of-order and stale message acceptance** +Older state, revoked membership, canceled work, or pre-step-up authorization arrives after newer state and overwrites it. Review sequence/version checks, tombstones, partition changes, and restore/replay workflows. + +**Acknowledgment/commit ordering defects** +Acknowledgment occurs before durable commit and loses security-relevant work, or commit happens before an unreliable acknowledgment and duplicates a mutation. Evaluate transactional outbox/inbox behavior and failure recovery. + +**Partial multi-consumer transitions** +Several consumers jointly implement one authorization or business transition, but retries and partial failure leave only a subset committed. Identify invariants that must become durable atomically or compensate with current authorization. + +## Universal moves (apply across the above) + +- Draw producer → broker/transport → gateway → consumer → storage for each message family. At each hop record authenticated peer, authoritative tenant/resource fields, validation, and side effect. +- Feed the same small fixture to every in-repo schema version or language binding. Test duplicate, missing, unknown, boundary, replayed, and reordered messages without producing load. +- Compare normal, retry, dead-letter, replay, migration, reflection, stream, and gateway-transcoded routes. Security policy must survive transport changes. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the realistic producer or peer, accepted message, authenticated channel identity, affected principal/resource, and unauthorized mutation or disclosure. +2. For disagreement claims, cite both parsers/consumers and the divergent decoded value. Safe rejection by either side prevents confirmation. +3. For replay/order claims, establish actual delivery guarantees and reproduce the invariant failure with a bounded local/in-memory transport. +4. For authorization and isolation, verify all interceptor, broker ACL, gateway, and consumer layers visible in source. External attachments make the candidate `needs_validation`. +5. Return `confirmed` only with the complete message lifecycle and observed meaningful result. Return `needs_validation` with the exact broker, service identity, route, or delivery fact required. +</code></pre> + <h4 id="file-thirdshift-security-audit-reconnaissance-md"><code>thirdshift-security-audit/RECONNAISSANCE.md</code></h4> + <pre class="job-text"><code># Reconnaissance + +### Phase 1: Map the source and plan coverage + +The parent initializes `run-metadata.json`, applies the strict pre-reconnaissance budget gate in `SKILL.md`, then creates agent scratch roots and the shared ledger before hunting. If the gate fails, record the incomplete status in metadata and launch no reconnaissance agent. Reconnaissance reads the target and locally available build/configuration state only. It does not contact deployed endpoints, external identity providers, registries, brokers, cloud APIs, or other shared services. + +Launch several `research` agents in parallel. They return structured facts to the parent and do not write files. + +**Agent 1a: Product, stack, and local operation** + +```text +Read the target at <target>. Do not use network access. Return: +1. Product type, users, operators, and ordinary trust-sensitive actions. +2. Languages, frameworks, build system, runtimes, and locally visible deployment models. +3. Repository-relative entry points and subsystem boundaries. +4. Exact build and test commands that could run offline with local dependencies, their expected write locations, and the target-controlled inputs they process. Do not run them during reconnaissance. +5. Comparable software or protocol visible from local documentation and dependencies. If no useful comparison is source-grounded, say so. +6. Missing local toolchains or runtime facts that limit bounded execution. +Return only source facts with repository-relative file:line references. +``` + +**Agent 1b: Principals, authority, and controls** + +```text +Read all source that establishes identity, authorization, isolation, and privilege. Map: +1. Each lower-trust principal and the actions it has by design. +2. Authentication or peer identity at each entry surface. +3. Per-resource authorization and tenant/owner scope. +4. Process, browser, workload, CI, plugin, model/tool, device, or local-IPC authority. +5. Privilege changes, confirmation, revocation, recovery, and fallback paths. +6. Which controls are source-visible and which depend on an unobserved deployment fact. +Return trust boundaries and control locations with repository-relative file:line references. Do not infer live reachability. +``` + +**Agent 1c: Entry surfaces, copies, and sinks** + +```text +Inventory every source-visible place external or lower-trust input enters: +- HTTP/browser, RPC/message/protocol, files/archive/document, CLI/env/config, plugins/dependencies/CI, cloud events/IAM selectors, model context/tool arguments, mobile/deep-link/webview, and local IPC. +For each surface, follow major transformations, stored or derived copies, and security-relevant sinks. Record source-visible limits and parallel paths to the same effect. +Return repository-relative paths and line numbers. Be complete, but do not execute or send inputs. +``` + +**Agent 1d: Local execution and deployment visibility** + +```text +Read tests, build definitions, manifests, packaging, and maintained environment overlays. Return: +1. Small offline tests or existing fixtures that could validate trust boundaries with dummy data inside the required OS-enforced sandbox. +2. Processes that could use an isolated loopback network namespace without external or shared dependencies. +3. Commands that would fetch dependencies, publish artifacts, contact paid/provider APIs, or affect shared state; mark them prohibited for this run. +4. Deployed controls and attachments that source cannot establish and therefore require needs_validation if decisive. +5. The final active source path for each deployment mode only where the repository selects it deterministically. +6. Whether the local platform can enforce an empty allowlisted environment, no external network, read-only target/toolchain mounts, scratch-only writes, and explicit CPU, memory, process, file-size, disk, and wall-clock limits. Missing controls block target-controlled execution. +7. Whether trusted parent-side code can promote predeclared scratch files with path-confined no-follow descriptor traversal, nonblocking regular-file checks, no-follow traversal of every destination parent, exclusive regular-file destination creation, and explicit per-file and cumulative size bounds. Missing promotion controls block use of scratch files as evidence. +``` + +Add focused reconnaissance agents for materially distinct deployment modes or subsystems that these four do not map. Do not silently omit them: if the budget gate in `SKILL.md` blocks a focused agent, launch nothing for it, seed the unmapped area as a `deferred` ledger unit with reason `budget_cannot_reserve_critics_and_validation`, and disclose the gap in the report. + +## Prior-run input + +Before selecting work, the parent reads every available prior `coverage-ledger.json` and `findings.json` for the same repo: + +- Compare the source locations, controls, conditions, and source-derived identity for every prior record and unit against the current source. +- Carry an unchanged prior `confirmed` record into the current candidate set, with the same fingerprint, only when its relevant source, conditions, and qualifying evidence still apply. Link it to a current `planned` unit with `prior_status: "prior_confirmed_same_source"` and put only that root cause on the hunter exclusion list. The Phase 3 verifier that re-verifies the carried record becomes that unit's assignment owner; its source re-check is the unit's first check and moves the unit to `candidate` with the carried fingerprint. +- Build a current planned `prior_confirmed_changed_source` revalidation unit when any relevant source or condition changed. Do not exclude that root cause from hunting or assume the prior verdict still applies. +- Build current work units for every prior `needs_validation`, `deferred`, `blocked`, `out_of_scope`, and changed-source unit. These states are priority input, never deduplication or suppression keys. +- Carry a still-blocked prior `needs_validation` record with the same fingerprint only after current source supports its trace. Link it to a current `planned` unit with `prior_status: "prior_needs_validation"`; the record keeps the unresolved blocker. The Phase 3 verifier that re-checks the carried record becomes that unit's assignment owner; its re-check is the unit's first check and moves the unit to `candidate` with the carried fingerprint. Include the record in final verification. +- Treat prior rejected records as stale claims unless current evidence changes the failed trace or missing condition. An unchanged rejection suppresses only that exact claim, not review of the coverage unit. +- Record missing or incompatible ledgers instead of treating them as empty coverage. + +State paths and source refs used in `run-metadata.json`. Summarize only the coverage consequences in `architecture.md`. + +## Architecture summary and companion selection + +The parent synthesizes `<output-dir>/architecture.md`, with a hard cap of about 1,000 words. Include: + +1. Product, principals, normal authority, and protected resources. +2. The comparable-software baseline from Agent 1a, when one is source-grounded: what security trade-offs the comparable accepts. Use it to calibrate effort and severity, never to dismiss a demonstrated finding; if the comparable shares a defect pattern that has mattered in practice, that strengthens the finding. Omit this line when no meaningful comparable exists. +3. Tech stack, source-visible deployment paths, and offline build/test limits. +4. Entry surfaces and the important source-to-sink or lifecycle paths. +5. Trust boundaries and the strongest source-visible control on each. +6. Repository-relative starting paths. +7. Prior coverage gaps, changed-source and blocked revalidation targets, and same-source confirmed exclusions. +8. A short companion-selection summary derived from [ATTACK-CLASSES.md](ATTACK-CLASSES.md): selected files and the source-visible boundaries that require them. + +Keep the assignment-level ordinary block, selected companion blocks, and excluded blocks with reasons in each ledger unit, not in `architecture.md`. This keeps the architecture cap valid for large runs and makes the exact hunter prompt map machine-checkable. + +Do not select a companion file merely because the language or dependency name appears. Select it because reconnaissance found the trust-sensitive boundary described by its `When to use this file` section. Do not exclude a visible boundary just because another agent will review a related class. + +## Deterministic coverage ledger + +The parent writes `<output-dir>/coverage-ledger.json` as a top-level JSON array. Derive one unit for every material combination of entry surface, trust boundary, subsystem, and applicable ordinary or companion attack class at the granularity the run profile sets (`quick` uses one all-in-scope subsystem identity; `deep` adds lifecycle modes). For a scoped run, seed in-scope surfaces for assignment and retain discovered excluded surfaces as `out_of_scope` units so later full runs can turn them into current work. + +Each dimension has a human label and a stable source-derived value in `canonical_refs`. Use the same canonical reference for the same source object across runs even if its display label changes. Suitable references include a repository-relative entry path plus exported scope, a route or message identity defined in source, the source control that defines a boundary, a repository package path, and the exact attack-class block reference. A block reference is `FILE.md#` plus the exact class name as written in bold or as a heading in that file — a stable identifier matched against the file text, not a rendered HTML anchor. For companion section blocks, use the heading text before any parenthetical qualifier (for example `Core discipline`). Do not derive references by lowercasing or slugging display labels. + +Derive `coverage_id` without lossy slugs: + +1. Require every reference to be Unicode NFC with valid scalar values, visible content, no control, format, line/paragraph separator, or default-ignorable code point, and no surrounding whitespace. +2. Encode its UTF-8 bytes with RFC 3986 percent encoding: leave only `A-Z a-z 0-9 - . _ ~` unescaped and use uppercase `%HH` for every other byte. +3. Join encoded `surface`, `boundary`, `subsystem`, and `attack_class` references with `::`; append encoded `lifecycle` when present. + +Use the fixed canonical value `profile/quick/all-in-scope-subsystems` for the quick profile's coarsened subsystem dimension. Do not include wave number, agent, verdict, severity, or line number in a reference or ID. Sort units lexicographically by `coverage_id` before each assignment. Fail on every duplicate ID. If duplicate IDs have different semantic fields, treat that as a canonical identity collision; never merge or silently overwrite them. The validator also rejects one semantic tuple represented by different canonical references. + +Each unit records: + +```json +{ + "coverage_id": "...", + "canonical_refs": { + "surface": "src/router.ts#POST /users/:id", + "boundary": "src/authz.ts#requireOwner", + "subsystem": "packages/api", + "attack_class": "ATTACK-CLASSES.md#Access control" + }, + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "ordinary_attack_class_block": "ATTACK-CLASSES.md#Access control", + "selected_companion_blocks": ["FILE.md#section"], + "excluded_blocks": [{"block": "FILE.md#section", "reason": "..."}], + "prior_status": "new|prior_confirmed_same_source|prior_confirmed_changed_source|prior_needs_validation|prior_deferred|prior_blocked|prior_out_of_scope|prior_covered_same_source|prior_covered_changed_source|prior_rejected_claim_changed|none", + "attempts": [], + "wave": 1, + "status": "planned", + "agent_id": null, + "reviewed_paths": [], + "local_checks": [], + "result_fingerprints": [], + "unresolved": [] +} +``` + +When `lifecycle` is material, add both `canonical_refs.lifecycle` and a human `lifecycle` field. `ordinary_attack_class_block` is null only when no ordinary block applies. The selected companion list includes each applicable class plus its companion `Core discipline`, `Universal moves`, and `Validation rules`; `excluded_blocks` records every considered but unselected block and the source fact that excludes it. + +The parent may add bookkeeping fields but keeps the semantic fields above stable. In `prior_status`, `new` marks a surface first seen in this run when compatible prior ledgers exist; `none` marks a unit seeded when no compatible prior ledger is available. Prior `deferred`, `blocked`, `out_of_scope`, and changed-source units initialize as current `planned` work when now in scope. A prior same-source covered unit remains visible in the current ledger; assign changed source, important lifecycle paths, and exact conflicts first, then use the coverage critic to decide whether it needs another pass. + +`attempts` is an append-only archive for evidence-bearing assignments that a coverage critic reopens. Before reassignment, append the prior unit's exact `wave`, `status`, `agent_id`, `reviewed_paths`, `local_checks`, `result_fingerprints`, and `unresolved`, plus the critic's source-backed `reassignment_reason`. Only `blocked`, `covered`, and `candidate` states can be archived. Archived attempts retain the same state and evidence invariants as live units, use strictly increasing waves below the current wave, and retain their producing owners and artifacts. The next assignment increments `wave`, uses a fresh owner, and starts with empty live evidence. If the profile or budget prevents another assignment, increment `wave` and use live `deferred` state with null owner, empty evidence, and the stop reason. Never copy an archived owner's checks or artifacts into the live state. A later live terminal state contains only the new attempt's evidence; the archive remains unchanged. + +Enforce this state table exactly: + +| Status | Unit `agent_id` | `reviewed_paths` / `local_checks` | `result_fingerprints` | `unresolved` | +|---|---|---|---|---| +| `planned` | null | empty | empty | empty | +| `not_applicable`, `out_of_scope`, `deferred` | null | empty | empty | nonempty reason | +| `in_progress` | canonical owner | empty | empty | empty | +| `blocked` | canonical owner | both nonempty owned partial evidence | empty | nonempty blocker | +| `covered` | canonical owner | both nonempty | empty | empty | +| `candidate` | canonical owner | both nonempty | nonempty | optional | + +Canonical agent IDs match `^[a-z0-9][a-z0-9_-]{0,63}$` and are not Windows device names. Lowercase is mandatory, so one ledger cannot contain case-fold aliases. The unit `agent_id` records the assignment owner. Every check records its own `agent_id` and nonempty `reviewed_paths`; the unit-level `reviewed_paths` is exactly their union. A source-only check uses `artifact: null`. A local check requires a regular file promoted only by trusted parent-side code under exactly `agents/<check.agent_id>/artifacts/`. This lets hunter and verifier checks coexist in one unit. Scratch paths, output-root files, symlinks, special files, and another check owner's artifacts are not evidence. + +The ledger is the coverage claim. An architecture summary, agent count, or generic "auth reviewed" sentence is not coverage evidence. Phase 2 closes units only from the paths and checks in a hunter's structured result. + +Run `node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json` after seeding, after every parent update, and before Phase 6. The validator rejects input beyond 5 MiB, 64 nesting levels, 10,000 units, 1,000 entries in a nested collection, or 500,000 traversed values, and caps reported validation errors at 100. In practice the 5 MiB byte limit holds roughly 2,000-5,000 realistic units, so it binds before the 10,000-unit cap. Fix every error before assigning work or making a coverage claim. +</code></pre> + <h4 id="file-thirdshift-security-audit-resource-exhaustion-and-availability-md"><code>thirdshift-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md</code></h4> + <pre class="job-text"><code># Resource Exhaustion and Availability Hunting + +#### When to use this file + +Reach for this file when untrusted requests, messages, files, tenant state, or agent work can consume CPU, memory, disk, connections, worker slots, paid APIs, or queue capacity, or can deadlock/crash a shared service. This domain distinguishes a source-reviewable availability vulnerability from a general performance issue. Never validate by stressing a shared or live service. + +Use `MEMORY-SAFETY-AND-BINARY.md` for memory-integrity defects and `PROTOCOLS-RPC-AND-MESSAGING.md` for broker delivery logic. A reachable fatal error belongs here for shared impact even when the underlying parser is covered elsewhere. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Require an input-to-cost path, a missing effective bound, and impact on another user, shared service, safety function, or operator-owned spend. Self-limiting work in the requester's own process is not a service vulnerability. +- A missing rate limit is not enough. Check body/message/file caps, concurrency, queues, deadlines, database constraints, upstream gateways, and per-tenant quotas before calling a path unbounded. +- Do not run stress, saturation, or production tests. Use asymptotic analysis, small boundary fixtures, mocked paid calls, strict local resource limits, and deterministic cancellation tests. +- State attacker cost, service work, persistence, scope, and recovery. One bounded input with superlinear or persistent shared effect is materially different from sustained volume. +- Use `confirmed` for source-visible bounds failures demonstrated safely. Use `needs_validation` when upstream caps, deployed topology, autoscaling, paid quota, or recovery behavior is outside the repository. +``` + +## Computational amplification attack classes (subagent_type: `general`) + +**Superlinear parsing, matching, or evaluation** +Small accepted input drives catastrophic regex backtracking, nested parsing, recursive validation, symbolic evaluation, graph traversal, template expansion, or adversarial sort/hash behavior. Derive accepted depth/cardinality and complexity, then demonstrate a bounded growth curve locally. + +**Decompression and representation amplification** +Compressed, sparse, nested, aliased, or encoded input expands far beyond the checked transfer or file size. Verify limits after every expansion and across parser stages, including archives, images, fonts, structured documents, and protocol compression tables. + +**Database and downstream query amplification** +A small request creates broad scans, pathological joins, fan-out, unbounded sort/aggregation, or many downstream calls because query depth, filter cardinality, pagination, or expansion fields are not bounded. Confirm authorization does not intentionally permit the same resource scope. + +## Resource accumulation attack classes (subagent_type: `general`) + +**Unbounded buffering and cardinality** +Bodies, out-of-order streams, uploads, sessions, unique cache keys, metrics labels, log fields, subscriptions, or pending jobs accumulate without per-item and aggregate limits. Find cleanup and expiration on disconnect, timeout, cancellation, and partial parse. + +**File descriptor, handle, and temporary-resource leaks** +Malformed or canceled work misses cleanup and retains sockets, files, database cursors, timers, subprocesses, temporary files, or object references. Confirm the leak repeats through bounded local iterations and affects a shared pool. + +**Detached work after cancellation** +Client timeout, disconnect, canceled job, or failed authorization returns control but leaves database, model, network, or worker work running. Trace cancellation and deadline propagation through every layer. + +## Quota and scheduling attack classes (subagent_type: `general`) + +**Pre-authentication work imbalance** +Expensive parsing, key lookup, cryptography, decompression, or external requests happen before authentication and the earliest size/rate gate. Compare minimal requester effort to shared service cost and check upstream limits. + +**Quota-accounting scope and reset gaps** +Accounting uses attacker-influenceable IP, route, tenant, key prefix, task ID, or other dimension, allowing one principal's work to escape its intended budget or consume another principal's allocation. Review integer overflow, distributed races, retries, reconnects, and account switching. + +**Worker, pool, and priority starvation** +Low-priority or attacker-controlled jobs hold shared locks, workers, database pools, event-loop turns, or scheduler priority needed by unrelated users. Require a path that bypasses queue/concurrency fairness or retains a slot beyond its deadline. + +## Failure and recovery attack classes (subagent_type: `general`) + +**Reachable fatal error or deadlock** +An untrusted input reaches `panic`, abort, fatal assertion, unhandled exception, process exit, lock cycle, or infinite loop in a shared process. Confirm supervisor scope and whether one worker or the whole service becomes unavailable. A restarted isolated worker may reduce impact but does not erase the defect. + +**Retry storm and fail-open amplification** +Timeouts, dependency errors, partially processed messages, or health-check failures trigger synchronized or unbounded retries without jitter, ceilings, circuit breaking, or deduplication. Verify one bounded failure source can create persistent aggregate work. + +**Poison-record and head-of-line blocking** +One malformed record or message repeatedly fails at the front of a shared queue, partition, startup scan, migration, or recovery loop. Review skip/quarantine policy, offsets, and whether other tenants share the blocked unit. + +**Unsafe recovery and capacity rollback** +A restart, restore, fallback, or cleanup path rebuilds unbounded state, ignores current quotas, or restores the input that immediately repeats failure. Recovery correctness is part of availability. + +## Universal moves (apply across the above) + +- Build an input-to-resource table: earliest accepted size/cardinality, work before auth, downstream fan-out, persistence, shared pool, limit and cleanup owner, recovery. +- Compare aggregate limits with per-object limits. Ten thousand valid one-byte items may evade a per-message cap while exhausting tenant-wide or process-wide state. +- Validate only in an isolated fixture with strict CPU/memory/time limits and small growth points. Mock external and paid calls and stop once the missing bound or cancellation is observable. + +## Validation rules (apply before reporting ANY finding here) + +1. Name untrusted input, requester work, service amplification or retained resource, shared blast radius, and recovery. Missing limits without concrete shared impact are hardening. +2. Confirm no source-visible upstream, parser, queue, tenant, or framework bound prevents the path. Unknown deployed controls require `needs_validation`. +3. For superlinear behavior, establish the accepted complexity and bounded local growth. For leaks, show repeatable retention after cleanup should occur. For fatal paths, identify process/supervisor isolation. +4. Prioritize by low requester work, unauthenticated reachability, cross-tenant scope, persistence, and poor recovery; do not validate with availability impact. +5. Return `confirmed` only with safe local proof and meaningful shared effect. Return `needs_validation` with the exact upstream limit, topology, quota, or recovery observation an owner must check. +</code></pre> + <h4 id="file-thirdshift-security-audit-supply-chain-and-release-md"><code>thirdshift-security-audit/SUPPLY-CHAIN-AND-RELEASE.md</code></h4> + <pre class="job-text"><code># Supply Chain and Release Hunting + +#### When to use this file + +Reach for this file when the target resolves dependencies, builds from untrusted contributions, runs CI, creates release artifacts, signs or promotes builds, loads plugins, or updates deployed software. This domain covers trust handoffs from source and dependency to the artifact a user runs. Use `MEMORY-SAFETY-AND-BINARY.md` for flaws inside a local binary loader and `CLOUD-AND-DEPLOYMENT.md` for runtime workload authority. + +Split large targets into dependency resolution, CI isolation, artifact provenance, release authorization, and updater/plugin trust. + +## Core discipline (include in every agent prompt for this domain) + +``` +- A mutable or known-vulnerable dependency is not a finding by itself. Show who can influence resolution, which build consumes it, and what execution or release boundary follows. +- Follow integrity across every handoff: source identity, resolved inputs, build worker, artifact identity, test result, signature/attestation, promotion, and update consumer. +- CI configuration is authorization code. Establish which event triggered a workflow, whose code runs, which secrets and tokens exist, and what it may publish or mutate. +- A checksum fetched from the same untrusted location as the artifact does not establish independent integrity. Identify the trusted root and failure behavior. +- Use `confirmed` for in-repo control-flow failures with bounded local validation. Use `needs_validation` for branch protection, hosted-runner, registry, signing-service, or production promotion facts that are not observable. +``` + +## Dependency and build-input attack classes (subagent_type: `general`) + +**Dependency source and namespace confusion** +Resolver configuration can select an unintended public/private namespace, fallback registry, mirror, repository, or source URL. Review package names, source priority, lockfile and checksum use, alternate build files, platform-specific resolution, and first-install versus update behavior. + +**Mutable and unbound build inputs** +Builds consume branches, tags, unverified submodules, downloaded tools, generated assets, remote includes, floating CI actions, or container tags whose content can change without source review. Require a lower-trust writer and a path into trusted build output; reproducibility by itself does not prove authenticity. + +**Generated-source and codegen provenance gaps** +Schemas, vendored archives, generated clients, localization, documentation examples, or binary blobs produce executable or shipped content without the same review and integrity gate as source. Compare local regeneration with committed output and verify who controls input and generator. + +**Build-context inclusion** +Secrets, local configuration, repository metadata, test fixtures, or developer artifacts enter a package or image because the build context and ignore rules exceed intended release inputs. Confirm that the resulting artifact exposes a real credential, private data, or privileged configuration. + +## CI and automation attack classes (subagent_type: `general`) + +**Untrusted code in a privileged workflow** +A pull request, issue comment, fork, dependency update, or external event runs contributor-controlled code with protected secrets, write tokens, deployment authority, or a trusted runner. Compare trigger type, checkout ref, approval gate, environment protection, and permission narrowing. Do not assume repository-host defaults that are not in source. + +**Workflow command and expression confusion** +Attacker-controlled branch names, commit messages, issue fields, artifact names, matrix values, or generated output enter shell commands, template expressions, paths, or privileged workflow inputs without canonical validation. + +**Cache, artifact, and workspace trust mixing** +A lower-trust job can populate a cache, artifact, shared workspace, or output that a higher-trust job later restores and executes or releases. Review cache keys and namespaces, artifact producer identity, digest binding, retention, and whether promotion re-resolves by mutable name. + +**Automation identity overreach** +CI jobs receive permissions beyond the operation, repository, environment, or duration needed, and untrusted job inputs can select the affected resource. Missing least privilege alone is hardening; require a reachable privileged action. + +## Release and update attack classes (subagent_type: `general`) + +**Build-to-promotion substitution** +Tests, review, signature, and publication refer to mutable tags, filenames, channels, or artifact IDs rather than the same immutable digest. Check every copy, repack, architecture merge, and provenance step between build and release. + +**Release authorization and signing-policy gaps** +A release or signature is accepted from the wrong workflow, repository, branch, environment, key role, or threshold. Review identity claims inside attestations and verify the consumer validates them, not just a valid signature. Rotation, expiry, and revocation must fail closed where policy requires. + +**Update metadata and rollback confusion** +An updater authenticates payload bytes but not version, product, platform, channel, target path, expiry, or rollback state, or it accepts metadata and payload from different authorized transactions. Verify atomic installation and recovery behavior. A signature API call without policy binding is incomplete. + +**Plugin and extension trust expansion** +An extension package gains host authority beyond its declared scope, a lower-trust publisher can replace another publisher's identity, or install/update hooks run before authenticity and capability checks. Intended installation of arbitrary same-user plugins is not a privilege boundary. + +## Universal moves (apply across the above) + +- Walk backward from a released digest or installed update to every source, generated input, credential, worker, cache, test result, and authorization decision. +- Compare untrusted and protected workflow events side by side. Mark each persisted channel crossing between them and require an immutable identity plus producer trust. +- Review revoked key, failed download, missing attestation, partial platform release, rollback, and registry outage paths. The failure policy is part of release integrity. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the lower-trust actor, controllable source/cache/artifact/metadata, consuming trusted job or updater, and resulting unauthorized publication, code inclusion, secret disclosure, or privileged execution. +2. Prove artifact identity across the broken handoff. A different mutable name or unbound digest must reach a real consumer. +3. Verify built-in package-manager, repository-host, registry, and signing defaults for the pinned version. Unknown hosted controls require `needs_validation`. +4. Keep local validation bounded: use a harmless fixture repository, dummy credential marker, local registry/config, and non-production artifact namespace. Do not publish or alter a real release. +5. Return `confirmed` only with a complete source-visible handoff and meaningful result. Return `needs_validation` with the precise branch, runner, registry, signing, or deployment fact an owner must observe. +</code></pre> + <h4 id="file-thirdshift-security-audit-validation-and-reporting-md"><code>thirdshift-security-audit/VALIDATION-AND-REPORTING.md</code></h4> + <pre class="job-text"><code># Validation, Structured Output, Verification, and Reporting + +### Phase 3: Independently validate every candidate + +After the clean coverage-critic pass or an explicitly recorded early stop, consolidate Phase 2 candidates and carried same-source prior confirmations by stable fingerprint and root cause. Give every unique proposed `confirmed` and `needs_validation` candidate to a fresh `general` verifier that did not hunt it. A carried prior confirmation follows the same current verification path even though hunters exclude that unchanged root cause. A verifier may read hunter or prior artifacts but must re-read every cited current source location and independently run any decisive check it can reproduce safely. + +Assign each verifier a canonical lowercase unique ID and `<output-dir>/agents/<verifier-id>/scratch/` plus parent-owned `artifacts/`. The verifier writes only to `scratch/` and never writes retained artifacts. It receives only the candidate, its linked coverage-unit checks and artifact paths, architecture facts needed to interpret the path, exact relevant companion validation blocks, the promotion procedure block below, the source/local execution boundary, the `confirmed`, `needs_validation`, and `rejected` branches of `report-schema.json` copied verbatim, and prior records with the same fingerprint. It must not receive another verifier's conclusion. + +#### Candidate-verifier prompt + +```text +You did not write this candidate. Try to refute it from repository source and bounded +local evidence. Do not contact deployed endpoints or external/shared services. Run +target-controlled code only inside the approved OS-enforced sandbox: no external +network, empty allowlisted environment, read-only target and tools, scratch-only +writes, and explicit low resource and wall-clock limits. If any control is unavailable, +do not execute; retain the exact missing capability as a needs_validation blocker. +Treat every scratch entry as target-controlled after execution. After the sandbox and +all its processes terminate, only trusted parent-side code may promote a predeclared +scratch-relative file, following the promotion procedure block included verbatim in +this prompt. You and target code never write retained artifacts. If promotion is +unavailable or fails, do not use that file as evidence. + +1. Verify every trace and evidence file, positive line number, scope, and description. + Confirm the first entry is a real lower-trust entrypoint and the last is the + claimed sink or boundary effect. +2. Reconstruct the strongest source-visible validation, identity, authorization, + normalization, lifecycle, framework, and containment controls on the path. + Where the architecture summary names a comparable baseline, note whether it + shares the pattern — as calibration, never as grounds to dismiss. +3. For a proposed confirmed candidate, independently reproduce the minimum observed + result when possible. Verify inputs, interface shape, conditions, and affected + dummy principal/resource. Do not infer a stronger result or continue after it. +4. Verify that likelihood, impact, confidence, and the proposed source fix match only + what the evidence establishes. +5. For a proposed needs_validation candidate, decide whether the blocker is genuinely + outside source/local observation. If source refutes the trace, reject it. If the + missing fact remains decisive, keep needs_validation and make the local and + owner-observed plans exact and non-destructive. +6. Preserve the fingerprint for the same source-derived root cause across every state. + +Return exactly one JSON object and no surrounding prose: +{"decision": "confirmed|needs_validation|rejected", "record": { ... }} +where record exactly matches the decision's verdict branch of the schema included +in this prompt. A corrected record replaces the hunter's wording. +``` + +Copy this promotion procedure verbatim into every candidate-verifier prompt: + +```text +Artifact promotion procedure (trusted parent-side code only): +Reference only for you: the parent performs these steps; you never perform them. + +Before execution, the parent opens and retains trusted, non-inheritable directory +descriptors for the agent's scratch/ and artifacts/ roots, and records an allowlist +of expected scratch-relative artifact files plus explicit per-file and cumulative +byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox +and all its processes terminate, trusted parent-side code promotes each allowlisted +file separately: + +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or + symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with + no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and + within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, + type, link count, or size. +7. For the destination, walk every parent component from the retained + artifacts-root descriptor with no-follow directory-relative operations; require + each existing component to be a real directory, and create any missing directory + exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened + destination is a regular file with link count exactly one, and copy from the + verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or + open or promote a symlink, FIFO, socket, device, directory, hard-linked file, + changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch + entry; if it is decisive evidence, retain `needs_validation` with the exact + promotion blocker. +``` + +A verifier can promote `needs_validation` to `confirmed` only after independently establishing the complete path and bounded observed result. Demote proposed confirmation to `needs_validation` when a specific deployment or runtime fact remains unknown. Use `rejected` when source, local behavior, a visible control, missing meaningful impact, or an impossible prerequisite refutes the claim. `needs_validation` is never a parking place for a speculative idea. + +The parent checks that each verifier returned the same fingerprint unless it identified a genuinely different root cause. Merge corrections, record the decision in every linked coverage unit, and ensure there is one final record per fingerprint. Discard a malformed or prose-wrapped verifier result without repairing it; re-run that candidate with a fresh verifier when the budget permits, otherwise it remains an unvalidated ledger candidate under the incomplete-run rule. + +When verifier evidence updates a ledger check, set that check's `agent_id` to the verifier's canonical ID and list its nonempty repository-relative `reviewed_paths`. Keep the unit-level `reviewed_paths` equal to the union across checks. Use `method: "source"` with `artifact: null` for source-only review. Use `method: "local"` only with a file successfully promoted by trusted parent-side code below `agents/<check.agent_id>/artifacts/`. The unit retains its original assignment owner, so independently owned hunter and verifier checks can coexist. For a carried prior record's seeded `planned` unit there is no prior owner: the verifier that re-checks it becomes the unit's assignment owner, and its re-check is the unit's first check, moving the unit to `candidate` with the carried fingerprint. + +If a strict total-agent budget cannot cover every candidate, set the run status to incomplete and follow the deterministic budget rule in `SKILL.md`. An unvalidated candidate remains only in the ledger. It does not enter `findings.json` under any verdict. + +### Phase 4: Write and validate `findings.json` + +The parent writes all independently decided records to `<output-dir>/findings.json`, sorted by fingerprint. Include: + +- `confirmed`: source-grounded vulnerabilities with complete local execution evidence, conditions, specific remediation, likelihood/impact/overall severity, and confidence. +- `needs_validation`: source-grounded candidates with an exact unresolved blocker and at least one applicable local or owner-observed deployment plan. +- `rejected`: source-grounded candidates disproved during validation, retained so future runs do not repeat the unsupported claim without changed evidence. + +Read `report-schema.json` immediately before writing. It uses `additionalProperties: false`; do not carry hunter wrapper fields into a record. Keep these verdict contracts distinct: + +- A `confirmed` record uses `root_cause`, `intended_behavior`, `conditions`, `execution`, `remediation`, `severity`, and `confidence`. It must not use `claimed_root_cause`, `blockers`, `validation_plan`, or `reason`. `execution` is target-neutral and uses the target's native interface: API/HTTP input, CLI call, library call, message, file fixture, browser action, rendered policy, or local harness as applicable. `observed_result` is nonempty and factual. +- A `needs_validation` record uses `claimed_root_cause`, `trace`, `evidence`, `blockers`, and at least one nonempty `validation_plan.local` or `validation_plan.deployment` field. Include both only when both contexts can resolve distinct facts. It must not use severity, execution, remediation, reason, or confirmed root cause. +- A `rejected` record uses `claimed_root_cause`, `trace`, `evidence`, and `reason`. It must not use severity, execution, remediation, blockers, validation plan, or confirmed root cause. + +Every record has a stable fingerprint, title, description, and repository-relative source paths. A multi-step trace begins with `entrypoint`, ends with `sink`, and uses `propagation` only between them. One-entry traces use `entrypoint` or `sink`. Overall severity cannot exceed demonstrated impact. + +Run: + +```sh +node <skill-dir>/validate-findings.cjs <output-dir>/findings.json +node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json +``` + +Fix every structural and semantic error before continuing. The findings validator rejects input beyond 5 MiB, 1,000 top-level findings, or 64 nesting levels, and caps reported error output at 100 messages. Validator success proves format and ledger consistency only. + +### Phase 5: Verify the final records with fresh eyes + +Launch one fresh `research` verifier per final `confirmed` and `needs_validation` record, in parallel. This verifier checks the structured record, not the hunter write-up, and remains inside source/local boundaries. + +In a `quick` run, Phase 3 and Phase 5 merge: the Phase 3 verifier also performs these record checks and returns the final schema-shaped record, so each candidate gets one fresh independent reviewer instead of two. Every other profile keeps the two passes separate. Never skip independent review of a `confirmed` record in any profile. + +For `confirmed`, require it to check: + +1. Every repository-relative trace/evidence path, line, scope, and described operation. +2. Real entry interface and exact local input shape. +3. Every condition, parser/policy step, source-visible preventing layer, and observed local result. +4. Affected principal/resource and demonstrated impact. +5. Severity separation: realistic likelihood, demonstrated impact, overall no greater than impact. +6. Remediation strategy and any `code_changes`, including whether the fix enforces the invariant without merely moving trust. + +For `needs_validation`, require it to check: + +1. The source path is real and supports only the `claimed_root_cause` stated. +2. Every listed blocker is decisive and not already answerable locally. +3. The candidate names a boundary and a possible concrete result rather than a generic concern. +4. At least one validation-plan field is present and exact. `local` uses a bounded fixture; `deployment` asks an owner to observe a configuration, identity, route, policy, or runtime fact. Do not invent a plan for an inapplicable context, and never send audit traffic to a deployment. +5. The fingerprint matches prior/current records for the same root cause. + +Each verifier returns exactly one JSON object: `{"decision":"verified","fingerprint":"..."}` or `{"decision":"replace","reason":"...","record":{...}}`, with no surrounding prose. A replacement record must match its `confirmed`, `needs_validation`, or `rejected` schema branch. Treat a malformed or prose-wrapped Phase 5 result the same way as in Phase 3: discard it without repairing it and re-run with a fresh verifier when the budget permits. + +Do not apply a Phase 5 replacement as final when it promotes a record to a stronger verdict, including any promotion to `confirmed`, or materially changes the root cause, trace, execution input or observed result, demonstrated impact, or severity. Give that complete replacement to a new independent verifier that did not hunt, perform Phase 3 validation, or propose the Phase 5 replacement. The new verifier rechecks the current source and independently reproduces any decisive local result under the execution boundary, then returns `verified` or another replacement. Apply a material replacement only after this fresh verification. If another material replacement results, repeat with a fresh verifier. If budget or independence is unavailable, remove the disputed record from `findings.json`, keep its ledger unit as an unresolved candidate, and set `run_status: "incomplete"` with an exact `incomplete_reason`. Non-material wording or repository-line corrections may be applied directly when they do not change meaning or evidence. + +After every applied replacement, rerun both validators and update linked ledger decisions. If a final verifier identifies a separate root cause, assign a new fingerprint and send it through independent candidate validation before inclusion. Set `run_status: "complete"` only when every ledger candidate has an independent final disposition and every retained record passes Phase 5. + +Do not verify only `confirmed` records. A misleading `needs_validation` handoff wastes owner time and can preserve a false premise. + +### Phase 6: Produce target-neutral reports from final records + +Only after Phase 5 passes for every record retained in `findings.json`, derive prose from the final records, the ledger, and the hunter `hardening` notes retained in ledger bookkeeping. An incomplete run may report independently verified records, but it must identify each unresolved ledger candidate and must not present it as a finding. The prose files never change a verdict, severity, blocker, or demonstrated impact. + +#### `REPORT.md` + +Write: + +1. Run profile, scope, budget (if set) with agents spent versus planned, source ref, sandboxed source-and-local-only execution statement, prior-run use, and explicit deferred and out-of-scope coverage. Name carried same-source confirmations and changed-source revalidations. A `quick`, scoped, budget-limited, or incomplete run states plainly that it is a partial pass. If candidate validation exhausted a strict budget, state that the run is incomplete and list every unvalidated fingerprint and linked unit; do not describe those candidates as findings. If the budget prevented a mandatory critic, state which critic did not run and make no clean-coverage claim. +2. One short security posture summary. +3. A confirmed-findings table: severity, title, affected boundary, and one-line observed result. +4. Each confirmed finding: repository source location, lower-trust principal, target-native bounded reproduction, conditions, actual result, impact, priority rationale, and smallest source fix. +5. A separate `NEEDS VALIDATION` table. Give each lead's title, repository trace, exact blocker, bounded local next step, and safe owner-observed deployment check. Do not assign severity or call it a confirmed vulnerability. +6. Separate hardening notes and positive source patterns. +7. Coverage summary from the ledger: covered, candidate, blocked, and deferred counts, plus important exclusions and the final critic result. + +Do not describe rejected records as findings. Mention their fingerprints only when they explain a prior disagreement or coverage decision. + +#### `FINDINGS-DETAIL.md` + +For each confirmed `medium`, `high`, or `critical` record, copy the complete source path and target-neutral local reproduction: + +- ordered repository-relative trace and evidence; +- dummy attacker/principal and affected dummy resource; +- native input, invocation, or fixture and exact bounded instructions; +- observed output and the security invariant it proves; +- conditions and containment; +- source-level remediation and regression case. + +#### `NEEDS-VALIDATION.md` + +For every unresolved record, copy the source trace, verified evidence, exact blocker, affected boundary, and each applicable bounded local or owner-observed resolution plan. Keep these as prioritized leads without severity. Do not turn them into live test guidance or assume the missing deployment fact. + +HTTP is one possible native interface, not the default. A library finding may use a function call, a parser a fixture, a CLI a command, a desktop app an IPC or file action, and infrastructure a locally rendered policy. Do not require an endpoint, external account, or live environment that the target does not have. + +Keep the report proportional to the evidence. A clean run may have zero confirmed records. State that result and the remaining coverage/validation limits without inventing LOW findings. +</code></pre> + <h4 id="file-thirdshift-security-audit-web-protocol-and-auth-md"><code>thirdshift-security-audit/WEB-PROTOCOL-AND-AUTH.md</code></h4> + <pre class="job-text"><code># HTTP-Protocol and Authentication Hunting + +#### When to use this file + +Reach for this file when the target speaks HTTP at a parsing, caching, browser-authentication, or identity boundary: web applications, APIs, reverse proxies, CDNs, gateways, custom HTTP servers, and services implementing sessions, JWT, OAuth/OIDC, SAML, password recovery, MFA, passkeys, API keys, or mTLS. Use this with `ATTACK-CLASSES.md`: access-control review asks whether a principal may perform an operation; this file asks whether the HTTP or identity layer can confuse which principal, request, assurance level, or token the operation belongs to. + +Pick classes from Phase 1. Split a large target into request framing and cache policy, browser authentication, federated identity, strong authentication and recovery, service credentials, and session lifecycle. A single server behind an unobserved managed proxy has little source-confirmable smuggling surface; a proxy or custom parser has much more. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Framing and cache findings require two interpretations of the same request, response, or key. Name both components and the exact normalized value on each side. +- For every credential, find the signature or secret verification and every binding required for its role: issuer, audience, origin, RP, client, session, principal, resource, assurance, expiry, and one-time state. +- Host, Forwarded, X-Forwarded-*, Origin, Referer, redirect targets, callback state, and request-derived URLs are trust decisions. Trace each to the affected identity or response. +- A missing header, cookie attribute, MFA prompt, or rate limit is not a finding alone. Require an accepted invalid request, cross-principal impact, assurance downgrade, or credential disclosure. +- Classify `confirmed` only from complete source evidence and bounded local request/token tests. Use `needs_validation` when proxy, IdP, browser, certificate, secret, or deployed configuration is required but not visible. +``` + +## HTTP framing and cache attack classes (subagent_type: `general`) + +**Request framing and desynchronization** +Front end and back end disagree on request length or header normalization. Review multiple `Content-Length` values, `Transfer-Encoding`, HTTP/2 or HTTP/3 downgrade, header-name normalization, forbidden connection headers, and CR/LF conversion. Confirm which bytes one component assigns to a request and which bytes its peer assigns to the next request. + +**Web cache poisoning through unkeyed input** +A request value changes cached content or security-relevant headers but is absent from the cache key. Compare cache key construction with every response variant, including forwarded host/scheme, selected cookies, query normalization, language/device headers, and authorization state. + +**Cache deception and private-response caching** +Cache routing treats a private dynamic path as a public static asset, or caches a response whose identity and authorization inputs are missing from policy. Compare edge cacheability with application route parsing, suffix/path-parameter normalization, and response cache directives. + +**Host and forwarded-header trust** +Untrusted host/proxy metadata determines absolute URLs, tenant routing, callbacks, reset links, cache keys, or the client address used by authorization. Confirm who can supply the header and whether trusted ingress removes client-provided copies. + +**Response-header injection** +Untrusted data reaches `Location`, `Set-Cookie`, CSP, or another response header with unsafe control characters or normalization. Verify framework rejection before reporting and require a security-relevant response change. + +## Browser-session attack classes (subagent_type: `general`) + +**Ordinary CSRF** +A browser sends ambient credentials to a state-changing endpoint that accepts a cross-site request without an effective anti-CSRF token, same-site request binding, or strict Origin/Referer validation. Inventory every cookie-authenticated mutation, including form, JSON-like, multipart, method-override, and legacy routes. SameSite is effective only for the cookie and browser contexts actually used; login CSRF and cross-site subresource requests can have different requirements. + +**Session fixation and invalidation** +Session identifiers are not rotated on login, account switch, MFA completion, impersonation, or other privilege changes, or remain valid after logout, password change, revocation, and account disable. Check server sessions, refresh tokens, signed cookies, websocket state, cache copies, and fallback endpoints. + +**Cookie scope and transport** +A sensitive cookie has an over-broad `Domain` or `Path`, can cross an insecure transport, or conflicts with a sibling cookie that another component selects differently. Bare missing flags remain hardening notes unless a realistic less-trusted origin, network position, or browser path can gain or replace the credential. + +## Federated-identity attack classes (subagent_type: `general`) + +First establish role. Authorization-server controls such as redirect allowlisting and code issuance do not belong to a relying-party client. Verification and binding defects belong to the component consuming the artifact. + +**JWT verification and claim binding** +Check signature verification, server-pinned algorithm and key source, then `exp`, `nbf`, `aud`, and `iss`. Review `kid`, `jku`, and `x5u` as untrusted key selectors, duplicate/header normalization, and decode-without-verify paths. A valid token for another service is invalid here even when signed by a trusted issuer. + +**OAuth/OIDC request and callback binding** +Validate exact `redirect_uri` ownership where the target is the authorization server; session-bound `state`; PKCE and authorization-code binding where applicable; ID-token issuer/audience/signature/nonce; and selected-IdP binding in multi-provider flows. Compare initial callback, retry, mobile/deep-link, and account-link routes. + +**SAML signed-object and assertion binding** +Ensure the element whose signature is validated is the element used as identity. Review unsigned/fallback paths, safe XML parser configuration, canonicalization differences, and freshness/binding fields such as validity windows, audience/recipient, request correlation, and replay state. + +## MFA, passkey, and account-transition attack classes (subagent_type: `general`) + +**MFA enrollment and assurance downgrade** +Enrollment, replacement, disablement, recovery-code generation, trusted-device creation, and fallback login require the intended prior assurance. Check that a valid first factor cannot enroll or replace the second factor without policy-required fresh authentication, and that disabled or stale factors stop authorizing sessions. + +**Step-up binding and bypass** +A successful challenge upgrades the wrong session, account, tenant, action, or API request, or an alternate route omits the assurance check. Bind the challenge to principal, current session, assurance target, operation or resource when required, expiry, and one-time completion. Compare UI, API, batch, recovery, and resumed-flow paths. + +**WebAuthn and passkey verification** +At registration, bind challenge, RP ID, expected origin, credential, user/userHandle, algorithm, and policy-required user verification to the initiating session. At authentication, verify challenge, RP/origin, credential membership, signature, and intended user presence/verification. Check account-discovery and linking flows for userHandle or credential-to-account confusion. Signature-counter handling is meaningful only when the product treats regressions as a clone signal. + +**Account linking and identity collision** +Adding an IdP, passkey, email, phone, device, or external account to an existing account must require a current authenticated session, verified ownership of the new identity, policy-required step-up, and callback state bound to the account that initiated linking. Review unlink/relink and invite-acceptance paths for verified-identifier or tenant collisions. + +**Password reset and broader recovery** +Recovery tokens, support/admin recovery, backup codes, device migration, and email or phone change often become the weakest authentication path. Verify token randomness, user/action binding, expiry, one-time state, rate/accounting controls, delivery URL trust, and invalidation of prior tokens and sessions. Different responses that only reveal public account existence are not automatically security findings. + +## API-key and mTLS attack classes (subagent_type: `general`) + +**API-key scope and resource binding** +A key authenticates to broader tenants, resources, actions, or environments than its server-side record grants, or request parameters override those bindings. Review key lookup, prefix/full-secret verification, type confusion between publishable and secret keys, scope checks, rotation, revocation caches, and bulk endpoints. + +**API-key exposure and unsafe transport** +Keys appear in client bundles, URLs, redirects, logs, error paths, build artifacts, or responses accessible to a lower-trust principal. A public identifier called a key is not a secret. Confirm key type and the authority gained by disclosure. + +**mTLS peer and application-identity confusion** +A process trusts client-certificate identity headers from any network peer, verifies a chain but maps attacker-influenceable subject text to an account incorrectly, or accepts a certificate for the wrong trust domain, extended usage, audience, or validity policy. Where a trusted proxy terminates mTLS, verify only that proxy can connect, it removes incoming identity headers, and the backend binds the sanitized identity to the request. + +**Certificate lifecycle fallback** +Expired, revoked, missing, or renewal-failed certificates cause silent fallback to bearer-only or anonymous operation, or long-lived pooled connections retain authorization after revocation. Missing deployment revocation data makes the result `needs_validation`; an in-repo fail-open branch is source-confirmable. + +## Universal moves (apply across the above) + +- Walk issue → store → transmit → consume → refresh → revoke for every credential and challenge. Compare normal, error, retry, migration, legacy, and account-switch paths. +- Enumerate every door to the same identity and every route to the same sensitive operation. The effective policy is the weakest parallel path, not the most polished UI. +- Diff parser, proxy, router, cache, and application normalization side by side. For local validation, feed identical bounded request fixtures into each component rather than sending traffic to a live deployment. +- For recovery and linking, draw the account before/after graph. Each edge must name the current principal, proof of the new identity, required assurance, callback/session binding, and revocation effect. + +## Validation rules (apply before reporting ANY finding here) + +1. Apply a source-visibility gate. Proxy chains, edge cache keys, IdP policy, certificate trust, browser cookie behavior, secrets, and deployed auth modes may be outside the repository. Record a precise `needs_validation` candidate instead of asserting missing infrastructure behavior. +2. For framing and cache findings, name both components and the divergent parse/key. Confirm cross-request, cross-user, or private-response impact with bounded local fixtures. +3. For token, MFA, passkey, account-link, recovery, API-key, and mTLS findings, cite the verification line and missing principal/session/resource/origin/audience/action/assurance binding. Prove the server accepts the invalid transition or credential. +4. For CSRF, name the ambient credential, state-changing route, accepted cross-site request shape, browser cookie policy, and missing effective check. Read-only actions and routes requiring a non-ambient bearer token do not qualify. +5. Verify framework and library defaults. If version or configuration is unknown, use `needs_validation`; do not turn an unverified critical claim into a lower-severity confirmed finding. +6. Return `confirmed` only with a complete source trace and observable unauthorized identity, state, or disclosure. For `needs_validation`, name the missing fact and safe local or owner-observed check that resolves it. +</code></pre> + <h4 id="file-thirdshift-security-audit-report-schema-json"><code>thirdshift-security-audit/report-schema.json</code></h4> + <pre class="job-text"><code>{ + "$comment": "Top-level contract for findings.json. validate-findings.cjs interprets and checks this schema directly.", + "type": "array", + "items": { + "oneOf": [ + { + "type": "object", + "description": "A source-grounded vulnerability that was independently demonstrated.", + "properties": { + "verdict": { + "type": "string", + "const": "confirmed" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$", + "description": "A stable source-derived identifier that does not change between validation states." + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "intended_behavior": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "conditions": { + "type": "array", + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["authentication_level", "authorization_role", "user_interaction", "system_configuration", "network_routing", "environmental_dependency", "data_state", "timing_dependency", "third_party_dependency"] + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "description"], + "additionalProperties": false + } + }, + "execution": { + "type": "object", + "description": "Target-neutral reproduction in the target's native interface.", + "properties": { + "attacker_perspective": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "payloads": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string" + } + }, + "instructions": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "observed_result": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["attacker_perspective", "payloads", "instructions", "observed_result"], + "additionalProperties": false + }, + "remediation": { + "type": "object", + "properties": { + "strategy": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "code_changes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "file_name": { + "type": "string", + "minLength": 1 + }, + "fixed_code": { + "type": "string" + } + }, + "required": ["file_name", "fixed_code"], + "additionalProperties": false + } + } + }, + "required": ["strategy"], + "additionalProperties": false + }, + "severity": { + "type": "object", + "properties": { + "likelihood": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + }, + "impact": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + }, + "overall_severity": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + } + }, + "required": ["likelihood", "impact", "overall_severity"], + "additionalProperties": false + }, + "confidence": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["low", "medium", "high"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + } + }, + "required": ["verdict", "fingerprint", "title", "description", "root_cause", "intended_behavior", "trace", "evidence", "conditions", "execution", "remediation", "severity", "confidence"], + "additionalProperties": false + }, + { + "type": "object", + "description": "A source-grounded candidate whose decisive validation is blocked.", + "properties": { + "verdict": { + "type": "string", + "const": "needs_validation" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$" + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "claimed_root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "blockers": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "validation_plan": { + "type": "object", + "properties": { + "local": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "deployment": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "additionalProperties": false + } + }, + "required": ["verdict", "fingerprint", "title", "description", "claimed_root_cause", "trace", "evidence", "blockers", "validation_plan"], + "additionalProperties": false + }, + { + "type": "object", + "description": "A source-grounded candidate refuted during validation.", + "properties": { + "verdict": { + "type": "string", + "const": "rejected" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$" + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "claimed_root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["verdict", "fingerprint", "title", "description", "claimed_root_cause", "trace", "evidence", "reason"], + "additionalProperties": false + } + ] + } +} +</code></pre> + <h4 id="file-thirdshift-security-audit-validate-coverage-ledger-cjs"><code>thirdshift-security-audit/validate-coverage-ledger.cjs</code></h4> + <pre class="job-text"><code>#!/usr/bin/env node + +/** + * Validates coverage-ledger.json and its canonical coverage IDs. + * Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json> + */ + +const fs = require("node:fs"); +const path = require("node:path"); +const { TextDecoder } = require("node:util"); + +// Conservative bounds apply before JSON.parse and again to the parsed document. +const LIMITS = Object.freeze({ + inputBytes: 5 * 1024 * 1024, + units: 10000, + collectionItems: 1000, + objectFields: 1000, + nestingDepth: 64, + preflightValues: 500000, + validationErrors: 100, +}); +const MAX_INPUT_BYTES = LIMITS.inputBytes; +const MAX_UNITS = LIMITS.units; +const MAX_LIST_ITEMS = LIMITS.collectionItems; +const MAX_TEXT_LENGTH = 4096; +const REQUIRED_FIELDS = [ + "coverage_id", + "canonical_refs", + "surface", + "boundary", + "subsystem", + "attack_class", + "starting_paths", + "ordinary_attack_class_block", + "selected_companion_blocks", + "excluded_blocks", + "prior_status", + "attempts", + "wave", + "status", + "agent_id", + "reviewed_paths", + "local_checks", + "result_fingerprints", + "unresolved", +]; +const REF_FIELDS = ["surface", "boundary", "subsystem", "attack_class"]; +const STATUSES = new Set([ + "planned", + "not_applicable", + "out_of_scope", + "in_progress", + "covered", + "candidate", + "blocked", + "deferred", +]); +const ATTEMPT_STATUSES = new Set(["covered", "candidate", "blocked"]); +const ATTEMPT_FIELDS = [ + "wave", + "status", + "agent_id", + "reviewed_paths", + "local_checks", + "result_fingerprints", + "unresolved", + "reassignment_reason", +]; +const PRIOR_STATUSES = new Set([ + "new", + "prior_confirmed_same_source", + "prior_confirmed_changed_source", + "prior_needs_validation", + "prior_deferred", + "prior_blocked", + "prior_out_of_scope", + "prior_covered_same_source", + "prior_covered_changed_source", + "prior_rejected_claim_changed", + "none", +]); +const FINGERPRINT_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$/; +const AGENT_ID_PATTERN = /^[a-z0-9][a-z0-9_-]{0,63}$/; +const WINDOWS_RESERVED_AGENT_ID = /^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])$/; +const VISIBLE_CONTENT = /[^\p{White_Space}\p{Cc}\p{Cf}\p{Default_Ignorable_Code_Point}]/u; +const PATH_FORBIDDEN_CHARACTER = /[\p{Cc}\p{Cf}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/u; +const WINDOWS_RESERVED_COMPONENT = /^(?:con|prn|aux|nul|clock\$|conin\$|conout\$|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\.|$)/iu; +const UTF8_DECODER = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); +const UNSAFE_DIAGNOSTIC_CHARACTER = /[\p{Cc}\p{Cf}\p{Cs}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/gu; +const MAX_DIAGNOSTIC_STRING_LENGTH = 256; + +const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value, key); + +class JsonStructureError extends Error {} +class SafeInputError extends Error {} + +function escapeUnsafeDiagnosticCharacters(value) { + return String(value).replace(UNSAFE_DIAGNOSTIC_CHARACTER, (character) => { + const codePoint = character.codePointAt(0); + return codePoint <= 0xffff + ? `\\u${codePoint.toString(16).padStart(4, "0")}` + : `\\u{${codePoint.toString(16)}}`; + }); +} + +function safeQuote(value) { + let serialized; + if (typeof value === "string") { + const clipped = value.length > MAX_DIAGNOSTIC_STRING_LENGTH + ? `${value.slice(0, MAX_DIAGNOSTIC_STRING_LENGTH)}...` + : value; + serialized = JSON.stringify(clipped); + } else if (value === null || typeof value === "boolean") { + serialized = String(value); + } else if (typeof value === "number" && Number.isFinite(value)) { + serialized = String(value); + } else { + serialized = `"<${Array.isArray(value) ? "array" : typeof value}>"`; + } + return escapeUnsafeDiagnosticCharacters(serialized); +} + +function createErrorList() { + const errors = []; + Object.defineProperty(errors, "push", { + value(...messages) { + const remaining = LIMITS.validationErrors - this.length; + if (remaining > 0) { + Array.prototype.push.apply(this, messages.slice(0, remaining).map(escapeUnsafeDiagnosticCharacters)); + } + return this.length; + }, + }); + return errors; +} + +function isObject(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function preflightJsonText(contents) { + const containers = []; + let rootState = "value"; + let inString = false; + let escaped = false; + let totalValues = 0; + + function fail(message) { + throw new JsonStructureError(`input ${message}`); + } + + function currentContainer() { + return containers[containers.length - 1]; + } + + function countValue() { + totalValues++; + if (totalValues > LIMITS.preflightValues) { + fail(`exceeds ${LIMITS.preflightValues} total value limit`); + } + } + + function beginValue() { + const container = currentContainer(); + if (!container) { + if (rootState !== "value") fail("has malformed JSON structure"); + rootState = "end"; + } else if (container.type === "array") { + if (container.state !== "firstValueOrEnd" && container.state !== "value") { + fail("has malformed JSON array structure"); + } + container.items++; + const limit = container.topLevel ? MAX_UNITS : LIMITS.collectionItems; + if (container.items > limit) { + fail(container.topLevel + ? `exceeds ${limit} top-level unit limit` + : `exceeds ${limit} item array limit`); + } + container.state = "commaOrEnd"; + } else { + if (container.state !== "value") fail("has malformed JSON object structure"); + container.state = "commaOrEnd"; + } + countValue(); + } + + function beginString() { + const container = currentContainer(); + if (container && container.type === "object" && + (container.state === "firstKeyOrEnd" || container.state === "key")) { + container.fields++; + if (container.fields > LIMITS.objectFields) { + fail(`exceeds ${LIMITS.objectFields} field object limit`); + } + container.state = "colon"; + } else { + beginValue(); + } + inString = true; + } + + function beginContainer(type) { + beginValue(); + if (containers.length >= LIMITS.nestingDepth) { + fail(`exceeds nesting depth limit ${LIMITS.nestingDepth}`); + } + containers.push(type === "array" + ? { type, state: "firstValueOrEnd", items: 0, topLevel: containers.length === 0 } + : { type, state: "firstKeyOrEnd", fields: 0 }); + } + + function closeContainer(type) { + const container = currentContainer(); + if (!container || container.type !== type) fail("has mismatched JSON containers"); + const canClose = type === "array" + ? container.state === "firstValueOrEnd" || container.state === "commaOrEnd" + : container.state === "firstKeyOrEnd" || container.state === "commaOrEnd"; + if (!canClose) fail(`has malformed JSON ${type} structure`); + containers.pop(); + } + + function isWhitespace(character) { + return character === " " || character === "\t" || character === "\r" || character === "\n"; + } + + function isTokenDelimiter(character) { + return isWhitespace(character) || character === "," || character === ":" || + character === "[" || character === "]" || character === "{" || + character === "}" || character === "\""; + } + + for (let index = 0; index < contents.length; index++) { + const character = contents[index]; + if (inString) { + if (escaped) { + escaped = false; + } else if (character === "\\") { + escaped = true; + } else if (character === "\"") { + inString = false; + } + continue; + } + if (isWhitespace(character)) continue; + + if (character === "\"") { + beginString(); + } else if (character === "[") { + beginContainer("array"); + } else if (character === "{") { + beginContainer("object"); + } else if (character === "]") { + closeContainer("array"); + } else if (character === "}") { + closeContainer("object"); + } else if (character === ":") { + const container = currentContainer(); + if (!container || container.type !== "object" || container.state !== "colon") { + fail("has malformed JSON object structure"); + } + container.state = "value"; + } else if (character === ",") { + const container = currentContainer(); + if (!container || container.state !== "commaOrEnd") fail("has malformed JSON collection structure"); + container.state = container.type === "array" ? "value" : "key"; + } else { + beginValue(); + while (index + 1 < contents.length && !isTokenDelimiter(contents[index + 1])) index++; + } + } + + if (inString) fail("has an unterminated JSON string"); + if (containers.length > 0) fail("has truncated JSON structure"); + if (rootState !== "end") fail("has no JSON value"); +} + +function preflightDocument(root) { + const errors = createErrorList(); + const stack = [{ value: root, depth: 0, location: "$" }]; + let visited = 0; + + while (stack.length > 0) { + const { value, depth, location } = stack.pop(); + visited++; + if (visited > LIMITS.preflightValues) { + errors.push(`$: exceeds ${LIMITS.preflightValues} total values`); + return errors; + } + if (value === null || typeof value !== "object") continue; + if (depth >= LIMITS.nestingDepth) { + errors.push(`${location}: exceeds nesting depth limit ${LIMITS.nestingDepth}`); + return errors; + } + + if (Array.isArray(value)) { + const limit = location === "$" ? MAX_UNITS : MAX_LIST_ITEMS; + if (value.length > limit) { + errors.push(`${location}: exceeds ${limit} entries`); + return errors; + } + for (let index = value.length - 1; index >= 0; index--) { + stack.push({ value: value[index], depth: depth + 1, location: `${location}[${index}]` }); + } + continue; + } + + const keys = Object.keys(value); + if (keys.length > LIMITS.objectFields) { + errors.push(`${location}: exceeds ${LIMITS.objectFields} object fields`); + return errors; + } + for (let index = keys.length - 1; index >= 0; index--) { + const key = keys[index]; + stack.push({ value: value[key], depth: depth + 1, location: `${location}{${index}}` }); + } + } + + return errors; +} + +function hasValidUnicodeScalarValues(value) { + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff) { + if (index >= value.length) return false; + const second = value.charCodeAt(index++); + if (second < 0xdc00 || second > 0xdfff) return false; + } else if (first >= 0xdc00 && first <= 0xdfff) { + return false; + } + } + return true; +} + +function hasVisibleProse(value) { + return hasValidUnicodeScalarValues(value) && VISIBLE_CONTENT.test(value); +} + +function isVisibleText(value, maxLength = MAX_TEXT_LENGTH) { + return typeof value === "string" && + value.length > 0 && + value.length <= maxLength && + value.trim() === value && + hasVisibleProse(value); +} + +function isCanonicalRef(value) { + return isVisibleText(value, 1024) && + !PATH_FORBIDDEN_CHARACTER.test(value) && + value.normalize("NFC") === value; +} + +function encodeCanonicalRef(value) { + if (!isCanonicalRef(value)) throw new TypeError("invalid canonical reference"); + let encoded = ""; + for (const byte of Buffer.from(value, "utf8")) { + const unreserved = + (byte >= 0x41 && byte <= 0x5a) || + (byte >= 0x61 && byte <= 0x7a) || + (byte >= 0x30 && byte <= 0x39) || + byte === 0x2d || byte === 0x2e || byte === 0x5f || byte === 0x7e; + encoded += unreserved ? String.fromCharCode(byte) : `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; + } + return encoded; +} + +function canonicalCoverageId(refs) { + if (!isObject(refs)) throw new TypeError("canonical_refs must be an object"); + const fields = hasOwn(refs, "lifecycle") ? [...REF_FIELDS, "lifecycle"] : REF_FIELDS; + if (Object.keys(refs).length !== fields.length || fields.some((field) => !hasOwn(refs, field))) { + throw new TypeError("canonical_refs has missing or unexpected fields"); + } + return fields.map((field) => encodeCanonicalRef(refs[field])).join("::"); +} + +function isSafeRelativePath(value) { + if (typeof value !== "string" || value.length === 0 || !hasValidUnicodeScalarValues(value) || value.trim() !== value || PATH_FORBIDDEN_CHARACTER.test(value) || value.includes("\\") || value.includes(":")) return false; + if (path.posix.isAbsolute(value) || path.win32.isAbsolute(value) || /^[A-Za-z]:/.test(value) || value.startsWith("~")) return false; + return value.split("/").every((segment) => + segment !== "" && + segment !== "." && + segment !== ".." && + !/[ .]$/u.test(segment) && + !WINDOWS_RESERVED_COMPONENT.test(segment)); +} + +function isSafeAgentId(value) { + return typeof value === "string" && + AGENT_ID_PATTERN.test(value) && + !WINDOWS_RESERVED_AGENT_ID.test(value); +} + +function isOwnedArtifactPath(value, agentId) { + if (!isSafeAgentId(agentId) || !isSafeRelativePath(value)) return false; + const prefix = `agents/${agentId}/artifacts/`; + return value.startsWith(prefix) && value.length > prefix.length; +} + +function validateStringArray(value, location, errors, options = {}) { + const { allowEmpty = true, fingerprint = false, pathValue = false } = options; + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (!allowEmpty && value.length === 0) errors.push(`${location}: must not be empty`); + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + const seen = new Set(); + value.slice(0, MAX_LIST_ITEMS).forEach((entry, index) => { + const entryLocation = `${location}[${index}]`; + const valid = pathValue ? isSafeRelativePath(entry) : isVisibleText(entry); + if (!valid) errors.push(`${entryLocation}: invalid ${pathValue ? "repository-relative path" : "text"}`); + if (fingerprint && typeof entry === "string" && !FINGERPRINT_PATTERN.test(entry)) { + errors.push(`${entryLocation}: invalid fingerprint`); + } + if (typeof entry === "string" && seen.has(entry)) errors.push(`${entryLocation}: duplicate entry`); + if (typeof entry === "string") seen.add(entry); + }); +} + +function validateChecks(value, location, errors) { + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + value.slice(0, MAX_LIST_ITEMS).forEach((check, index) => { + const base = `${location}[${index}]`; + if (!isObject(check)) { + errors.push(`${base}: expected object`); + return; + } + for (const field of ["agent_id", "reviewed_paths", "invariant", "method", "result", "artifact"]) { + if (!hasOwn(check, field)) errors.push(`${base}: missing required field ${safeQuote(field)}`); + } + if (!isSafeAgentId(check.agent_id)) errors.push(`${base}.agent_id: expected a canonical lowercase agent ID`); + validateStringArray(check.reviewed_paths, `${base}.reviewed_paths`, errors, { allowEmpty: false, pathValue: true }); + if (!isVisibleText(check.invariant)) errors.push(`${base}.invariant: invalid text`); + if (check.method !== "source" && check.method !== "local") errors.push(`${base}.method: expected "source" or "local"`); + if (!isVisibleText(check.result)) errors.push(`${base}.result: invalid text`); + if (check.method === "source" && check.artifact !== null) { + errors.push(`${base}.artifact: source-only check must use null`); + } else if (check.method === "local") { + if (!isOwnedArtifactPath(check.artifact, check.agent_id)) { + errors.push(`${base}.artifact: local check requires an artifact owned by agent ${safeQuote(isSafeAgentId(check.agent_id) ? check.agent_id : "<agent-id>")}`); + } + } else if (check.method !== "source" && check.method !== "local" && check.artifact !== null && !isSafeRelativePath(check.artifact)) { + errors.push(`${base}.artifact: expected null or a safe output-relative path`); + } + }); +} + +function validateReviewedPathOwnership(unit, base, errors) { + if (!Array.isArray(unit.reviewed_paths) || !Array.isArray(unit.local_checks)) return; + const aggregatePaths = new Set(unit.reviewed_paths.filter((value) => typeof value === "string")); + const ownedPaths = new Set(); + for (const check of unit.local_checks) { + if (!isObject(check) || !Array.isArray(check.reviewed_paths)) continue; + for (const reviewedPath of check.reviewed_paths) { + if (typeof reviewedPath === "string") ownedPaths.add(reviewedPath); + } + } + for (const reviewedPath of aggregatePaths) { + if (!ownedPaths.has(reviewedPath)) errors.push(`${base}.reviewed_paths: ${safeQuote(reviewedPath)} has no check owner`); + } + for (const reviewedPath of ownedPaths) { + if (!aggregatePaths.has(reviewedPath)) errors.push(`${base}.local_checks: owned path ${safeQuote(reviewedPath)} is absent from aggregate reviewed_paths`); + } +} + +function validateExcludedBlocks(value, location, errors) { + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + const seen = new Set(); + value.slice(0, MAX_LIST_ITEMS).forEach((entry, index) => { + const base = `${location}[${index}]`; + if (!isObject(entry)) { + errors.push(`${base}: expected object`); + return; + } + if (!isVisibleText(entry.block)) errors.push(`${base}.block: invalid text`); + if (!isVisibleText(entry.reason)) errors.push(`${base}.reason: invalid text`); + if (typeof entry.block === "string" && seen.has(entry.block)) errors.push(`${base}.block: duplicate entry`); + if (typeof entry.block === "string") seen.add(entry.block); + }); +} + +function semanticKey(unit) { + return JSON.stringify([ + unit.surface, + unit.boundary, + unit.subsystem, + unit.attack_class, + hasOwn(unit, "lifecycle") ? unit.lifecycle : null, + ]); +} + +function hasValidSemanticFields(unit) { + return ["surface", "boundary", "subsystem", "attack_class"].every((field) => isVisibleText(unit[field])) && + (!hasOwn(unit, "lifecycle") || isVisibleText(unit.lifecycle)); +} + +function requireEmptyArray(unit, field, base, errors) { + if (Array.isArray(unit[field]) && unit[field].length > 0) { + errors.push(`${base}.${field}: unit with status ${safeQuote(unit.status)} must keep this array empty`); + } +} + +function requireNonemptyArray(unit, field, base, errors) { + if (!Array.isArray(unit[field]) || unit[field].length === 0) { + errors.push(`${base}.${field}: unit with status ${safeQuote(unit.status)} requires entries`); + } +} + +function validateStateInvariants(unit, base, errors) { + const emptyEvidence = () => { + requireEmptyArray(unit, "reviewed_paths", base, errors); + requireEmptyArray(unit, "local_checks", base, errors); + }; + const requireOwner = () => { + if (!isSafeAgentId(unit.agent_id)) errors.push(`${base}.agent_id: unit with status ${safeQuote(unit.status)} requires a canonical lowercase agent ID`); + }; + + if (unit.status !== "candidate") requireEmptyArray(unit, "result_fingerprints", base, errors); + + switch (unit.status) { + case "planned": + if (unit.agent_id !== null) errors.push(`${base}.agent_id: planned unit must be unassigned`); + emptyEvidence(); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "not_applicable": + case "out_of_scope": + case "deferred": + if (unit.agent_id !== null) errors.push(`${base}.agent_id: unit with status ${safeQuote(unit.status)} must be unassigned`); + emptyEvidence(); + requireNonemptyArray(unit, "unresolved", base, errors); + break; + case "in_progress": + requireOwner(); + emptyEvidence(); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "blocked": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireNonemptyArray(unit, "unresolved", base, errors); + break; + case "covered": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "candidate": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireNonemptyArray(unit, "result_fingerprints", base, errors); + break; + } +} + +function validateAttempts(value, unit, base, errors) { + if (!Array.isArray(value)) { + errors.push(`${base}.attempts: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${base}.attempts: exceeds ${MAX_LIST_ITEMS} entries`); + + const priorOwners = new Set(); + const priorArtifacts = new Set(); + let previousWave = 0; + value.slice(0, MAX_LIST_ITEMS).forEach((attempt, index) => { + const attemptBase = `${base}.attempts[${index}]`; + if (!isObject(attempt)) { + errors.push(`${attemptBase}: expected object`); + return; + } + for (const field of ATTEMPT_FIELDS) { + if (!hasOwn(attempt, field)) errors.push(`${attemptBase}: missing required field ${safeQuote(field)}`); + } + if (!Number.isInteger(attempt.wave) || attempt.wave < 1) { + errors.push(`${attemptBase}.wave: expected a positive integer`); + } else { + if (attempt.wave <= previousWave) errors.push(`${attemptBase}.wave: archived attempt waves must be strictly increasing`); + if (Number.isInteger(unit.wave) && attempt.wave >= unit.wave) { + errors.push(`${attemptBase}.wave: archived attempt wave must precede current wave ${safeQuote(unit.wave)}`); + } + previousWave = attempt.wave; + } + if (!ATTEMPT_STATUSES.has(attempt.status)) { + errors.push(`${attemptBase}.status: expected "covered", "candidate", or "blocked"`); + } + let hasFreshOwner = false; + if (!isSafeAgentId(attempt.agent_id)) { + errors.push(`${attemptBase}.agent_id: archived attempt requires a canonical lowercase agent ID`); + } else if (priorOwners.has(attempt.agent_id)) { + errors.push(`${attemptBase}.agent_id: assignment owner must be fresh for each attempt`); + } else { + hasFreshOwner = true; + } + validateStringArray(attempt.reviewed_paths, `${attemptBase}.reviewed_paths`, errors, { pathValue: true }); + validateChecks(attempt.local_checks, `${attemptBase}.local_checks`, errors); + validateReviewedPathOwnership(attempt, attemptBase, errors); + validateStringArray(attempt.result_fingerprints, `${attemptBase}.result_fingerprints`, errors, { fingerprint: true }); + validateStringArray(attempt.unresolved, `${attemptBase}.unresolved`, errors); + if (!isVisibleText(attempt.reassignment_reason)) errors.push(`${attemptBase}.reassignment_reason: invalid text`); + validateStateInvariants(attempt, attemptBase, errors); + + if (Array.isArray(attempt.local_checks)) { + attempt.local_checks.forEach((check, checkIndex) => { + if (!isObject(check)) return; + if (priorOwners.has(check.agent_id)) { + errors.push(`${attemptBase}.local_checks[${checkIndex}].agent_id: prior assignment owner evidence must remain in its earlier attempt`); + } + if (typeof check.artifact === "string" && priorArtifacts.has(check.artifact)) { + errors.push(`${attemptBase}.local_checks[${checkIndex}].artifact: artifact from an earlier attempt cannot be reused`); + } + if (check.method === "local" && typeof check.artifact === "string") priorArtifacts.add(check.artifact); + }); + } + if (hasFreshOwner) priorOwners.add(attempt.agent_id); + }); + + if (isSafeAgentId(unit.agent_id) && priorOwners.has(unit.agent_id)) { + errors.push(`${base}.agent_id: current assignment owner must be fresh after reassignment`); + } + if (Array.isArray(unit.local_checks)) { + unit.local_checks.forEach((check, index) => { + if (!isObject(check)) return; + if (priorOwners.has(check.agent_id)) { + errors.push(`${base}.local_checks[${index}].agent_id: prior assignment owner evidence must remain in its archived attempt`); + } + if (typeof check.artifact === "string" && priorArtifacts.has(check.artifact)) { + errors.push(`${base}.local_checks[${index}].artifact: artifact from an archived attempt cannot be reused`); + } + }); + } +} + +function collectUnitErrors(unit, index) { + const errors = createErrorList(); + const base = `$[${index}]`; + if (!isObject(unit)) return [`${base}: expected object`]; + + for (const field of REQUIRED_FIELDS) { + if (!hasOwn(unit, field)) errors.push(`${base}: missing required field ${safeQuote(field)}`); + } + for (const field of ["surface", "boundary", "subsystem", "attack_class"]) { + if (!isVisibleText(unit[field])) errors.push(`${base}.${field}: invalid text`); + } + if (hasOwn(unit, "lifecycle") && !isVisibleText(unit.lifecycle)) errors.push(`${base}.lifecycle: invalid text`); + + let expectedId = null; + if (!isObject(unit.canonical_refs)) { + errors.push(`${base}.canonical_refs: expected object`); + } else { + const expectedFields = hasOwn(unit.canonical_refs, "lifecycle") ? [...REF_FIELDS, "lifecycle"] : REF_FIELDS; + for (const field of expectedFields) { + if (!hasOwn(unit.canonical_refs, field)) { + errors.push(`${base}.canonical_refs: missing required field ${safeQuote(field)}`); + } else if (!isCanonicalRef(unit.canonical_refs[field])) { + errors.push(`${base}.canonical_refs.${field}: invalid canonical reference`); + } + } + if (Object.keys(unit.canonical_refs).some((field) => !expectedFields.includes(field))) { + errors.push(`${base}.canonical_refs: contains unexpected fields`); + } + if (hasOwn(unit, "lifecycle") !== hasOwn(unit.canonical_refs, "lifecycle")) { + errors.push(`${base}: lifecycle and canonical_refs.lifecycle must appear together`); + } + try { + expectedId = canonicalCoverageId(unit.canonical_refs); + } catch { + // The specific reference errors above are more useful. + } + } + if (!isVisibleText(unit.coverage_id, 65536)) { + errors.push(`${base}.coverage_id: invalid text`); + } else if (expectedId !== null && unit.coverage_id !== expectedId) { + errors.push(`${base}.coverage_id: expected canonical ID ${safeQuote(expectedId)}`); + } + + validateStringArray(unit.starting_paths, `${base}.starting_paths`, errors, { allowEmpty: false, pathValue: true }); + if (unit.ordinary_attack_class_block !== null && !isVisibleText(unit.ordinary_attack_class_block)) { + errors.push(`${base}.ordinary_attack_class_block: expected null or non-empty text`); + } + validateStringArray(unit.selected_companion_blocks, `${base}.selected_companion_blocks`, errors); + validateExcludedBlocks(unit.excluded_blocks, `${base}.excluded_blocks`, errors); + if (Array.isArray(unit.selected_companion_blocks) && Array.isArray(unit.excluded_blocks)) { + const selected = new Set(unit.selected_companion_blocks); + unit.excluded_blocks.forEach((entry, blockIndex) => { + if (isObject(entry) && selected.has(entry.block)) { + errors.push(`${base}.excluded_blocks[${blockIndex}].block: block is also selected`); + } + }); + } + + if (!PRIOR_STATUSES.has(unit.prior_status)) errors.push(`${base}.prior_status: invalid value ${safeQuote(unit.prior_status)}`); + if (!STATUSES.has(unit.status)) errors.push(`${base}.status: invalid value ${safeQuote(unit.status)}`); + if (!Number.isInteger(unit.wave) || unit.wave < 1) errors.push(`${base}.wave: expected a positive integer`); + if (unit.agent_id !== null && !isSafeAgentId(unit.agent_id)) errors.push(`${base}.agent_id: expected null or a safe agent ID`); + + validateAttempts(unit.attempts, unit, base, errors); + validateStringArray(unit.reviewed_paths, `${base}.reviewed_paths`, errors, { pathValue: true }); + validateChecks(unit.local_checks, `${base}.local_checks`, errors); + validateReviewedPathOwnership(unit, base, errors); + validateStringArray(unit.result_fingerprints, `${base}.result_fingerprints`, errors, { fingerprint: true }); + validateStringArray(unit.unresolved, `${base}.unresolved`, errors); + + validateStateInvariants(unit, base, errors); + + return errors; +} + +function readFileWithinLimit(file) { + const noFollow = fs.constants.O_NOFOLLOW; + const nonBlock = fs.constants.O_NONBLOCK; + if (!Number.isInteger(noFollow) || noFollow === 0 || !Number.isInteger(nonBlock) || nonBlock === 0) { + // Node exposes no race-safe fallback on these platforms, so reject all inputs. + throw new SafeInputError("OS no-follow and nonblocking input protection is unavailable"); + } + + let descriptor; + try { + descriptor = fs.openSync(file, fs.constants.O_RDONLY | noFollow | nonBlock); + } catch (error) { + if (error && (error.code === "ELOOP" || error.code === "EMLINK")) throw new SafeInputError("input must not be a symlink"); + throw error; + } + try { + const stat = fs.fstatSync(descriptor); + if (!stat.isFile()) throw new SafeInputError("input must be a regular file"); + if (stat.size > MAX_INPUT_BYTES) throw new SafeInputError(`input exceeds ${MAX_INPUT_BYTES} byte limit`); + + const chunks = []; + const buffer = Buffer.allocUnsafe(64 * 1024); + let bytesRead = 0; + while (true) { + const count = fs.readSync(descriptor, buffer, 0, buffer.length, null); + if (count === 0) break; + bytesRead += count; + if (bytesRead > MAX_INPUT_BYTES) throw new SafeInputError(`input exceeds ${MAX_INPUT_BYTES} byte limit`); + chunks.push(Buffer.from(buffer.subarray(0, count))); + } + try { + return UTF8_DECODER.decode(Buffer.concat(chunks, bytesRead)); + } catch { + throw new SafeInputError("input is not valid UTF-8"); + } + } finally { + fs.closeSync(descriptor); + } +} + +function validateDocument(ledger) { + const errors = createErrorList(); + if (!Array.isArray(ledger)) { + errors.push("$: expected a top-level array"); + return errors; + } + if (ledger.length > MAX_UNITS) { + errors.push(`$: exceeds ${MAX_UNITS} coverage units`); + return errors; + } + + errors.push(...preflightDocument(ledger)); + if (errors.length > 0) return errors; + + const ids = new Map(); + const semantics = new Map(); + let previousId = null; + for (let index = 0; index < ledger.length && errors.length < LIMITS.validationErrors; index++) { + const unit = ledger[index]; + errors.push(...collectUnitErrors(unit, index)); + if (errors.length >= LIMITS.validationErrors) break; + if (!isObject(unit) || typeof unit.coverage_id !== "string") continue; + + const key = hasValidSemanticFields(unit) ? semanticKey(unit) : null; + if (ids.has(unit.coverage_id)) { + const previous = ids.get(unit.coverage_id); + const qualifier = key !== null && previous.key !== null && previous.key !== key + ? "canonical identity collision with different semantic fields" + : "duplicate coverage ID"; + errors.push(`$[${index}].coverage_id: ${qualifier} at $[${previous.index}]`); + } else { + ids.set(unit.coverage_id, { index, key }); + } + if (key !== null && semantics.has(key) && semantics.get(key).id !== unit.coverage_id) { + const previous = semantics.get(key); + errors.push(`$[${index}].canonical_refs: semantic tuple already uses coverage ID ${safeQuote(previous.id)} at $[${previous.index}]`); + } else if (key !== null) { + semantics.set(key, { id: unit.coverage_id, index }); + } + if (previousId !== null && previousId > unit.coverage_id) { + errors.push(`$[${index}].coverage_id: units must be sorted lexicographically`); + } + previousId = unit.coverage_id; + } + return errors; +} + +function run(file) { + if (!file) { + console.error("Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json>"); + return 1; + } + + let contents; + try { + contents = readFileWithinLimit(file); + } catch (error) { + const reason = error instanceof SafeInputError ? error.message : "input could not be opened or read safely"; + console.error(`Failed to read coverage ledger: ${reason}`); + return 1; + } + + let ledger; + try { + preflightJsonText(contents); + } catch (error) { + const reason = error instanceof JsonStructureError ? error.message : "invalid JSON structure"; + console.error(`Failed to parse coverage ledger: ${reason}`); + return 1; + } + try { + ledger = JSON.parse(contents); + } catch { + console.error("Failed to parse coverage ledger: invalid JSON syntax"); + return 1; + } + + let errors; + try { + errors = validateDocument(ledger); + } catch { + console.error("Failed to validate coverage ledger: unexpected validation error"); + return 1; + } + for (const message of errors) console.error("ERROR:", message); + if (errors.length > 0) { + const cap = errors.length === LIMITS.validationErrors ? `; output capped at ${LIMITS.validationErrors}` : ""; + console.error(`FAIL: ${errors.length} validation error(s)${cap}`); + return 1; + } + console.log(`PASS: ${ledger.length} coverage units valid`); + return 0; +} + +module.exports = { + LIMITS, + PATH_FORBIDDEN_CHARACTER, + UNSAFE_DIAGNOSTIC_CHARACTER, + VISIBLE_CONTENT, + WINDOWS_RESERVED_COMPONENT, + canonicalCoverageId, + encodeCanonicalRef, + hasVisibleProse, + isSafeAgentId, + isSafeRelativePath, + preflightJsonText, + readFileWithinLimit, + safeQuote, + validateDocument, +}; + +if (require.main === module) process.exit(run(process.argv[2])); +</code></pre> + <h4 id="file-thirdshift-security-audit-validate-coverage-ledger-test-cjs"><code>thirdshift-security-audit/validate-coverage-ledger.test.cjs</code></h4> + <pre class="job-text"><code>const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const { spawnSync } = require("node:child_process"); +const test = require("node:test"); +const { + LIMITS, + canonicalCoverageId, + encodeCanonicalRef, + isSafeAgentId, + isSafeRelativePath, + preflightJsonText, + validateDocument, +} = require("./validate-coverage-ledger.cjs"); + +const validatorPath = path.join(__dirname, "validate-coverage-ledger.cjs"); +const CLI_TIMEOUT_MS = 5000; +const HOSTILE_CLI_TIMEOUT_MS = 15000; +const HAS_SAFE_INPUT_OPEN = Number.isInteger(fs.constants.O_NOFOLLOW) && + fs.constants.O_NOFOLLOW !== 0 && + Number.isInteger(fs.constants.O_NONBLOCK) && + fs.constants.O_NONBLOCK !== 0; + +function unit(overrides = {}) { + const canonicalRefs = overrides.canonical_refs || { + surface: "src/router.ts#POST /users/:id", + boundary: "src/authz.ts#requireOwner", + subsystem: "packages/api", + attack_class: "ATTACK-CLASSES.md#Access control", + }; + const value = { + coverage_id: canonicalCoverageId(canonicalRefs), + canonical_refs: canonicalRefs, + surface: "Update-user route", + boundary: "Object ownership", + subsystem: "API", + attack_class: "Access control", + starting_paths: ["src/router.ts", "src/authz.ts"], + ordinary_attack_class_block: "ATTACK-CLASSES.md#Access control", + selected_companion_blocks: [], + excluded_blocks: [{ block: "WEB-PROTOCOL-AND-AUTH.md#Cache behavior", reason: "The route is not cached." }], + prior_status: "new", + attempts: [], + wave: 1, + status: "planned", + agent_id: null, + reviewed_paths: [], + local_checks: [], + result_fingerprints: [], + unresolved: [], + }; + return Object.assign(value, overrides, { canonical_refs: canonicalRefs }); +} + +function errorsFor(value) { + return validateDocument(value); +} + +function runCli(contents, options = {}) { + const { nodeArgs = [], timeout = CLI_TIMEOUT_MS } = options; + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-")); + const ledgerPath = path.join(directory, "coverage-ledger.json"); + try { + fs.writeFileSync(ledgerPath, contents); + return spawnSync(process.execPath, [...nodeArgs, validatorPath, ledgerPath], { + encoding: "utf8", + timeout, + }); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +function cliOutput(result) { + return `${result.stdout}${result.stderr}`; +} + +const TERMINAL_CONTROL_PAYLOAD = "\u001b\u0007\u0085\u202e"; +const TERMINAL_CONTROL_BYTES = [ + Buffer.from([0x1b]), + Buffer.from([0x07]), + Buffer.from("\u0085"), + Buffer.from("\u202e"), +]; + +function assertNoInjectedControlBytes(output) { + const bytes = Buffer.isBuffer(output) ? output : Buffer.from(output, "utf8"); + for (const marker of TERMINAL_CONTROL_BYTES) { + assert.equal(bytes.indexOf(marker), -1, `found raw control bytes ${marker.toString("hex")}`); + } +} + +function sourceCheck(agentId = "hunter-1", overrides = {}) { + return { + agent_id: agentId, + reviewed_paths: ["src/router.ts"], + invariant: "The route checks object ownership.", + method: "source", + result: "The owner check applies before the update.", + artifact: null, + ...overrides, + }; +} + +function localCheck(agentId = "hunter-1", overrides = {}) { + return sourceCheck(agentId, { + method: "local", + result: "The bounded fixture accepted the other owner's object.", + artifact: `agents/${agentId}/artifacts/result.txt`, + ...overrides, + }); +} + +function archivedAttempt(overrides = {}) { + const value = { + wave: 1, + status: "blocked", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: [], + unresolved: ["The deployed policy is unavailable."], + reassignment_reason: "The critic found an unchecked parallel path.", + }; + return Object.assign(value, overrides); +} + +test("accepts an empty ledger and complete units", () => { + assert.deepEqual(errorsFor([]), []); + assert.deepEqual(errorsFor([unit()]), []); + + const missingAttempts = unit(); + delete missingAttempts.attempts; + assert(errorsFor([missingAttempts]).some((error) => error.includes('missing required field "attempts"'))); + + const covered = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert.deepEqual(errorsFor([covered]), []); +}); + +test("accepts a complete ledger through the CLI", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const result = runCli(JSON.stringify([unit()])); + assert.equal(result.status, 0, cliOutput(result)); + assert.match(result.stdout, /PASS: 1 coverage units valid/); +}); + +test("text preflight ignores structural characters and escapes inside strings", () => { + const value = unit({ + surface: "Route \\ slash [list] {object}, colon: quoted \"value\"", + excluded_blocks: [{ + block: "COMPANION.md#Literal [brackets] {braces}", + reason: "The text contains a backslash \\ before an escaped \"quote\".", + }], + }); + const contents = JSON.stringify([value]); + assert.doesNotThrow(() => preflightJsonText(contents)); + assert.deepEqual(JSON.parse(contents), [value]); + if (HAS_SAFE_INPUT_OPEN) { + const result = runCli(contents); + assert.equal(result.status, 0, cliOutput(result)); + } +}); + +test("text preflight enforces structural cardinality limits", () => { + const depthLimit = LIMITS.nestingDepth; + assert.doesNotThrow(() => preflightJsonText(`${"[".repeat(depthLimit)}0${"]".repeat(depthLimit)}`)); + assert.throws( + () => preflightJsonText(`${"[".repeat(depthLimit + 1)}0${"]".repeat(depthLimit + 1)}`), + /exceeds nesting depth limit 64/, + ); + + const tooManyUnits = `[${"null,".repeat(LIMITS.units)}null]`; + assert.throws(() => preflightJsonText(tooManyUnits), /exceeds 10000 top-level unit limit/); + + const tooManyItems = `[[${"null,".repeat(LIMITS.collectionItems)}null]]`; + assert.throws(() => preflightJsonText(tooManyItems), /exceeds 1000 item array limit/); + + const objectFields = Array.from( + { length: LIMITS.objectFields + 1 }, + (_, index) => `"field${index}":null`, + ).join(","); + assert.throws(() => preflightJsonText(`[{${objectFields}}]`), /exceeds 1000 field object limit/); + + const fullArray = `[${"null,".repeat(LIMITS.collectionItems - 1)}null]`; + const arraysNeeded = Math.floor(LIMITS.preflightValues / (LIMITS.collectionItems + 1)) + 1; + const tooManyValues = `[${Array.from({ length: arraysNeeded }, () => fullArray).join(",")}]`; + assert.throws(() => preflightJsonText(tooManyValues), /exceeds 500000 total value limit/); +}); + +test("text preflight rejects malformed structural truncation cleanly", () => { + assert.throws(() => preflightJsonText("["), /truncated JSON structure/); + assert.throws(() => preflightJsonText("[\"unterminated"), /unterminated JSON string/); + assert.throws(() => preflightJsonText("[{\"field\":1]"), /mismatched JSON containers/); +}); + +test("derives collision-free canonical IDs from exact UTF-8 references", () => { + assert.equal(encodeCanonicalRef("route:POST /users"), "route%3APOST%20%2Fusers"); + assert.notEqual(encodeCanonicalRef("route name"), encodeCanonicalRef("route-name")); + assert.equal( + canonicalCoverageId({ surface: "a", boundary: "b", subsystem: "c", attack_class: "d", lifecycle: "retry" }), + "a::b::c::d::retry", + ); + assert.throws(() => encodeCanonicalRef("e\u0301"), /invalid canonical reference/); + assert.throws(() => encodeCanonicalRef("bad\u0000ref"), /invalid canonical reference/); + assert.throws(() => encodeCanonicalRef("hidden\u200bref"), /invalid canonical reference/); +}); + +test("rejects noncanonical, duplicate, and colliding IDs", () => { + const wrong = unit({ coverage_id: "display-label-slug" }); + assert(errorsFor([wrong]).some((error) => error.includes("expected canonical ID"))); + + const duplicate = unit(); + assert(errorsFor([duplicate, unit()]).some((error) => error.includes("duplicate coverage ID"))); + + const collision = unit(); + const differentMeaning = unit({ surface: "Delete-user route" }); + assert(errorsFor([collision, differentMeaning]).some((error) => error.includes("canonical identity collision"))); +}); + +test("requires canonical references to be own properties", () => { + const inherited = Object.create(unit().canonical_refs); + const value = unit(); + value.canonical_refs = inherited; + assert(errorsFor([value]).some((error) => error.includes("missing required field"))); +}); + +test("rejects aliases for one semantic tuple", () => { + const first = unit(); + const refs = { ...first.canonical_refs, surface: "src/alias.ts#updateUser" }; + const alias = unit({ canonical_refs: refs }); + const ledger = [first, alias].sort((left, right) => left.coverage_id.localeCompare(right.coverage_id)); + assert(errorsFor(ledger).some((error) => error.includes("semantic tuple already uses coverage ID"))); +}); + +test("requires lexicographic order", () => { + const secondRefs = { + surface: "zzz", + boundary: "src/authz.ts#requireOwner", + subsystem: "packages/api", + attack_class: "ATTACK-CLASSES.md#Access control", + }; + assert(errorsFor([unit({ canonical_refs: secondRefs }), unit()]) + .some((error) => error.includes("sorted lexicographically"))); +}); + +test("validates assignment block maps", () => { + const overlap = unit({ + selected_companion_blocks: ["AI-AND-LLM.md#Tool calls"], + excluded_blocks: [{ block: "AI-AND-LLM.md#Tool calls", reason: "Claimed irrelevant." }], + }); + assert(errorsFor([overlap]).some((error) => error.includes("also selected"))); + + const noReason = unit({ excluded_blocks: [{ block: "AI-AND-LLM.md#Tool calls", reason: "" }] }); + assert(errorsFor([noReason]).some((error) => error.includes("reason"))); +}); + +test("requires owned artifacts for local checks and null artifacts for source checks", () => { + const local = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck()], + }); + assert.deepEqual(errorsFor([local]), []); + + const independentlyVerified = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts", "src/authz.ts"], + local_checks: [sourceCheck(), localCheck("verifier-1", { reviewed_paths: ["src/authz.ts"] })], + }); + assert.deepEqual(errorsFor([independentlyVerified]), []); + + const unownedPath = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts", "src/authz.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([unownedPath]).some((error) => error.includes("has no check owner"))); + + for (const [checkAgentId, artifact] of [ + [null, "agents/hunter-1/artifacts/result.txt"], + ["hunter-1", null], + ["hunter-1", "result.txt"], + ["hunter-1", "agents/hunter-2/artifacts/result.txt"], + ["../hunter", "agents/../hunter/artifacts/result.txt"], + ]) { + const value = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck(checkAgentId, { artifact })], + }); + assert.notEqual(errorsFor([value]).length, 0, `${checkAgentId}: ${artifact}`); + } + + const unownedSource = unit({ + status: "blocked", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["The boundary behavior is not source-visible."], + }); + assert(errorsFor([unownedSource]).some((error) => error.includes("unit with status \"blocked\" requires a canonical lowercase agent ID"))); + + const sourceWithArtifact = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { artifact: "agents/hunter-1/artifacts/source.txt" })], + }); + assert(errorsFor([sourceWithArtifact]).some((error) => error.includes("source-only check must use null"))); +}); + +test("requires canonical lowercase filesystem-safe agent IDs", () => { + for (const value of ["hunter-1", "verifier_2", "a0"]) assert.equal(isSafeAgentId(value), true, value); + for (const value of ["Hunter-1", "hunter.1", "hunter-1.", "hunter ", "con", "prn", "aux", "nul", "com1", "lpt9", "../hunter"]) { + assert.equal(isSafeAgentId(value), false, value); + } + + const caseAlias = unit({ status: "in_progress", agent_id: "Hunter-1" }); + assert(errorsFor([caseAlias]).some((error) => error.includes("canonical lowercase agent ID"))); +}); + +test("enforces state evidence", () => { + assert(errorsFor([unit({ status: "in_progress" })]).some((error) => error.includes("unit with status \"in_progress\" requires"))); + assert(errorsFor([unit({ status: "blocked" })]).some((error) => error.includes("unresolved"))); + assert(errorsFor([unit({ status: "candidate" })]).some((error) => error.includes("reviewed_paths"))); + + assert.deepEqual(errorsFor([unit({ status: "in_progress", agent_id: "hunter-1" })]), []); + const inProgressEvidence = unit({ + status: "in_progress", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([inProgressEvidence]).some((error) => error.includes("must keep this array empty"))); + + const assignedPlanned = unit({ + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([assignedPlanned]).some((error) => error.includes("planned unit must be unassigned"))); + + const candidate = unit({ + status: "candidate", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { invariant: "Ownership is required.", result: "No check exists." })], + result_fingerprints: ["src-router-missing-owner-check"], + unresolved: ["validation_budget_exhausted"], + }); + assert.deepEqual(errorsFor([candidate]), []); + + const blocked = unit({ + status: "blocked", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["The deployed policy is unavailable."], + }); + assert.deepEqual(errorsFor([blocked]), []); + const blockedFingerprint = { ...blocked, result_fingerprints: ["forbidden-fingerprint"] }; + assert(errorsFor([blockedFingerprint]).some((error) => error.includes("result_fingerprints"))); + + for (const status of ["not_applicable", "out_of_scope", "deferred"]) { + assert.deepEqual(errorsFor([unit({ status, unresolved: ["Reason recorded."] })]), []); + const invalid = unit({ + status, + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: ["forbidden-fingerprint"], + unresolved: ["Reason recorded."], + }); + const errors = errorsFor([invalid]); + assert(errors.some((error) => error.includes("must be unassigned")), status); + assert(errors.some((error) => error.includes("reviewed_paths")), status); + assert(errors.some((error) => error.includes("result_fingerprints")), status); + } + + const coveredFingerprint = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: ["forbidden-fingerprint"], + }); + assert(errorsFor([coveredFingerprint]).some((error) => error.includes("result_fingerprints"))); + + const coveredUnresolved = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["Unexpected unresolved claim."], + }); + assert(errorsFor([coveredUnresolved]).some((error) => error.includes("unresolved"))); +}); + +test("archives prior evidence when a critic assigns a fresh owner", () => { + const reassigned = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "in_progress", + agent_id: "hunter-2", + }); + assert.deepEqual(errorsFor([reassigned]), []); + + const finalClosure = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "covered", + agent_id: "hunter-2", + reviewed_paths: ["src/authz.ts"], + local_checks: [sourceCheck("hunter-2", { reviewed_paths: ["src/authz.ts"] })], + }); + assert.deepEqual(errorsFor([finalClosure]), []); +}); + +test("preserves candidate provenance when reassignment must be deferred", () => { + const candidateAttempt = archivedAttempt({ + status: "candidate", + result_fingerprints: ["src-router-missing-owner-check"], + unresolved: ["validation_budget_exhausted"], + }); + const deferred = unit({ + attempts: [candidateAttempt], + wave: 2, + status: "deferred", + unresolved: ["quick_profile_final_critic"], + }); + assert.deepEqual(errorsFor([deferred]), []); +}); + +test("rejects reassignment owner reuse and evidence mixing", () => { + const reusedOwner = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "in_progress", + agent_id: "hunter-1", + }); + assert(errorsFor([reusedOwner]).some((error) => error.includes("current assignment owner must be fresh"))); + + const mixedEvidence = unit({ + attempts: [archivedAttempt({ local_checks: [localCheck()] })], + wave: 2, + status: "covered", + agent_id: "hunter-2", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck()], + }); + const errors = errorsFor([mixedEvidence]); + assert(errors.some((error) => error.includes("prior assignment owner evidence must remain"))); + assert(errors.some((error) => error.includes("artifact from an archived attempt cannot be reused"))); + + const mixedHistory = unit({ + attempts: [ + archivedAttempt({ local_checks: [localCheck()] }), + archivedAttempt({ + wave: 2, + agent_id: "hunter-2", + local_checks: [localCheck()], + }), + ], + wave: 3, + status: "in_progress", + agent_id: "hunter-3", + }); + const historyErrors = errorsFor([mixedHistory]); + assert(historyErrors.some((error) => error.includes("prior assignment owner evidence must remain in its earlier attempt"))); + assert(historyErrors.some((error) => error.includes("artifact from an earlier attempt cannot be reused"))); + + const unordered = unit({ + attempts: [archivedAttempt(), archivedAttempt({ + wave: 1, + agent_id: "hunter-2", + local_checks: [sourceCheck("hunter-2")], + })], + wave: 3, + status: "in_progress", + agent_id: "hunter-3", + }); + assert(errorsFor([unordered]).some((error) => error.includes("strictly increasing"))); +}); + +test("rejects unsafe paths and malformed fingerprints", () => { + for (const value of [ + "/etc/passwd", + "../src/file.js", + "src/../file.js", + "src/con.txt", + "src/PRN", + "src/AUX.c", + "src/NUL", + "src/CLOCK$.txt", + "src/conin$.txt", + "src/conout$", + "src/COM1.log", + "src/lpt9", + "src/COM\u00b9.log", + "src/COM\u00b2.log", + "src/COM\u00b3.log", + "src/lpt\u00b9", + "src/lpt\u00b2", + "src/lpt\u00b3", + "src/file.js.", + "C:/src/file.js", + "src/file\n.js", + "src/file\u0085.js", + "src/file\u2028.js", + "src/file\u200b.js", + "src/file\u034f.js", + "src/file\ufe0f.js", + ]) { + assert.equal(isSafeRelativePath(value), false, value); + } + assert.equal(isSafeRelativePath("src/handler.js"), true); + assert.equal(isSafeRelativePath("src/caf\u00e9/handler.js"), true); + assert(errorsFor([unit({ starting_paths: ["../src/router.ts"] })]).some((error) => error.includes("repository-relative path"))); + assert(errorsFor([unit({ + status: "candidate", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { invariant: "Ownership is required.", result: "No check exists." })], + result_fingerprints: ["not stable"], + })]).some((error) => error.includes("invalid fingerprint"))); +}); + +test("rejects format, default-ignorable, and invalid-scalar prose", () => { + for (const invisible of ["\u200b", "\u034f", "\ufe0f", "\ud800"]) { + assert(errorsFor([unit({ surface: invisible })]).some((error) => error.includes("surface")), JSON.stringify(invisible)); + assert(errorsFor([unit({ + excluded_blocks: [{ block: "ATTACK-CLASSES.md#Access control", reason: invisible }], + })]).some((error) => error.includes("reason")), JSON.stringify(invisible)); + } +}); + +test("quotes input-derived controls in direct validation errors", () => { + const invalidStatus = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + const invalidPath = `src/${TERMINAL_CONTROL_PAYLOAD}.js`; + const value = unit({ + status: invalidStatus, + agent_id: "hunter-1", + reviewed_paths: [invalidPath], + local_checks: [sourceCheck()], + result_fingerprints: ["force-state-error"], + }); + const output = errorsFor([value]).join("\n"); + + assert.match(output, /\$\[0\]\.status/); + assert.match(output, /\$\[0\]\.reviewed_paths/); + assert.match(output, /\\u001b/); + assert.match(output, /\\u0007/); + assert.match(output, /\\u0085/); + assert.match(output, /\\u202e/); + assertNoInjectedControlBytes(output); +}); + +test("quotes input-derived controls in CLI validation errors", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const value = unit({ + status: `invalid-${TERMINAL_CONTROL_PAYLOAD}`, + result_fingerprints: ["force-state-error"], + }); + const result = runCli(JSON.stringify([value])); + + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /\$\[0\]\.status/); + assert.match(result.stderr, /\\u001b/); + assertNoInjectedControlBytes(result.stderr); +}); + +test("returns a generic syntax error without parser-supplied controls", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const malformed = Buffer.concat([ + Buffer.from("["), + Buffer.from(TERMINAL_CONTROL_PAYLOAD), + Buffer.from("]"), + ]); + const result = runCli(malformed); + + assert.equal(result.status, 1, cliOutput(result)); + assert.equal(result.stderr, "Failed to parse coverage ledger: invalid JSON syntax\n"); + assertNoInjectedControlBytes(result.stderr); +}); + +test("does not reflect controls from a failed CLI input path", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-path-")); + const missingPath = path.join(directory, `missing-${TERMINAL_CONTROL_PAYLOAD}.json`); + try { + const result = spawnSync(process.execPath, [validatorPath, missingPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /Failed to read coverage ledger:/); + assertNoInjectedControlBytes(result.stderr); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects invalid UTF-8 through the CLI", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const encoded = Buffer.from(JSON.stringify([unit()])); + const marker = Buffer.from("Update-user route"); + const markerOffset = encoded.indexOf(marker); + assert.notEqual(markerOffset, -1); + const malformed = Buffer.concat([ + encoded.subarray(0, markerOffset), + Buffer.from([0x80]), + encoded.subarray(markerOffset + marker.length), + ]); + + const result = runCli(malformed); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /input is not valid UTF-8/); + assert.doesNotMatch(output, /TypeError|stack|at validate-coverage-ledger/i); +}); + +test("rejects a FIFO through the CLI without blocking", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-fifo-")); + const fifoPath = path.join(directory, "coverage-ledger.json"); + try { + const created = spawnSync("mkfifo", [fifoPath], { encoding: "utf8", timeout: CLI_TIMEOUT_MS }); + assert.equal(created.status, 0, cliOutput(created)); + + const result = spawnSync(process.execPath, [validatorPath, fifoPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must be a regular file/); + assert.doesNotMatch(output, /stack|at validate-coverage-ledger/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects a symlink through the CLI", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-symlink-")); + const targetPath = path.join(directory, "target.json"); + const symlinkPath = path.join(directory, "coverage-ledger.json"); + try { + fs.writeFileSync(targetPath, JSON.stringify([unit()])); + fs.symlinkSync(targetPath, symlinkPath); + const result = spawnSync(process.execPath, [validatorPath, symlinkPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must not be a symlink/); + assert.doesNotMatch(output, /stack|at validate-coverage-ledger/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects deeply nested input without recursion failure", () => { + let nested = 0; + for (let depth = 0; depth < 20000; depth++) nested = [nested]; + assert(errorsFor(nested).some((error) => error.includes("exceeds nesting depth limit 64"))); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(`${"[".repeat(20000)}0${"]".repeat(20000)}`); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /exceeds nesting depth limit 64/); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); +}); + +test("rejects multi-megabyte nesting under a constrained Node heap", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const openContainers = "[".repeat(2000000); + const cases = [ + openContainers, + `${openContainers}0${"]".repeat(2000000)}`, + ]; + for (const contents of cases) { + const result = runCli(contents, { + nodeArgs: ["--max-old-space-size=64"], + timeout: HOSTILE_CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /exceeds nesting depth limit 64/); + assert.doesNotMatch(output, /heap out of memory|allocation failed|RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); + } +}); + +test("caps malformed 10000-unit validation output", () => { + assert.equal(errorsFor(Array.from({ length: LIMITS.units }, () => null)).length, LIMITS.validationErrors); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(JSON.stringify(Array.from({ length: LIMITS.units }, () => null))); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); +}); + +test("rejects malformed top-level data and excessive unit counts", () => { + assert.deepEqual(errorsFor({ units: [] }), ["$: expected a top-level array"]); + const tooMany = Array.from({ length: 10001 }, () => null); + const errors = errorsFor(tooMany); + assert.deepEqual(errors, ["$: exceeds 10000 coverage units"]); + + const oversizedCollection = unit({ extra: Array.from({ length: LIMITS.collectionItems + 1 }, () => null) }); + assert(errorsFor([oversizedCollection]).some((error) => error.includes("exceeds 1000 entries"))); +}); + +test("accepts a canonical ID derived from near-maximum multibyte references", () => { + const canonicalRefs = { + surface: "\u6f22".repeat(1024), + boundary: "\u00e9".repeat(1024), + subsystem: "packages/api", + attack_class: "\u6f22".repeat(1023) + "\u00e9", + }; + const value = unit({ canonical_refs: canonicalRefs }); + assert(value.coverage_id.length > 16384, `coverage_id length ${value.coverage_id.length}`); + assert(value.coverage_id.length <= 65536, `coverage_id length ${value.coverage_id.length}`); + assert.deepEqual(errorsFor([value]), []); + + if (HAS_SAFE_INPUT_OPEN) { + const result = runCli(JSON.stringify([value])); + assert.equal(result.status, 0, cliOutput(result)); + } +}); +</code></pre> + <h4 id="file-thirdshift-security-audit-validate-findings-cjs"><code>thirdshift-security-audit/validate-findings.cjs</code></h4> + <pre class="job-text"><code>#!/usr/bin/env node + +/** + * Validates findings.json against report-schema.json. + * Usage: node validate-findings.cjs <path-to-findings.json> + * + * This is a dependency-free interpreter for the JSON Schema keywords used by + * report-schema.json, plus finding-specific checks that are clearer in code. + */ + +const fs = require("fs"); +const path = require("path"); +const { TextDecoder } = require("util"); + +const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value, key); +const SUPPORTED_TYPES = new Set(["object", "array", "string", "integer", "number", "boolean", "null"]); +const SUPPORTED_KEYWORDS = new Set([ + "$comment", + "additionalProperties", + "const", + "description", + "enum", + "items", + "minimum", + "minItems", + "minLength", + "oneOf", + "pattern", + "properties", + "required", + "type", + "uniqueItems", + "visibleContent", +]); +const SEVERITY_RANK = new Map([ + ["informational", 0], + ["low", 1], + ["medium", 2], + ["high", 3], + ["critical", 4], +]); +const LIMITS = Object.freeze({ + inputBytes: 5 * 1024 * 1024, + nestingDepth: 64, + arrayItems: 1000, + canonicalKeyBytes: 1024 * 1024, + uniqueSetBytes: 5 * 1024 * 1024, + validationErrors: 100, +}); +const VISIBLE_CONTENT = /[^\p{White_Space}\p{Cc}\p{Cf}\p{Default_Ignorable_Code_Point}]/u; +const PATH_FORBIDDEN_CHARACTER = /[\p{Cc}\p{Cf}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/u; +const WINDOWS_RESERVED_COMPONENT = /^(?:con|prn|aux|nul|clock\$|conin\$|conout\$|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\.|$)/iu; +const UTF8_DECODER = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); +const UNSAFE_DIAGNOSTIC_CHARACTER = /[\p{Cc}\p{Cf}\p{Cs}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/gu; +const MAX_DIAGNOSTIC_STRING_LENGTH = 256; + +class JsonStructureError extends Error {} +class SafeInputError extends Error {} + +function escapeUnsafeDiagnosticCharacters(value) { + return String(value).replace(UNSAFE_DIAGNOSTIC_CHARACTER, (character) => { + const codePoint = character.codePointAt(0); + return codePoint <= 0xffff + ? `\\u${codePoint.toString(16).padStart(4, "0")}` + : `\\u{${codePoint.toString(16)}}`; + }); +} + +function safeQuote(value) { + let serialized; + if (typeof value === "string") { + const clipped = value.length > MAX_DIAGNOSTIC_STRING_LENGTH + ? `${value.slice(0, MAX_DIAGNOSTIC_STRING_LENGTH)}...` + : value; + serialized = JSON.stringify(clipped); + } else if (value === null || typeof value === "boolean") { + serialized = String(value); + } else if (typeof value === "number" && Number.isFinite(value)) { + serialized = String(value); + } else { + serialized = `"<${Array.isArray(value) ? "array" : typeof value}>"`; + } + return escapeUnsafeDiagnosticCharacters(serialized); +} + +function propertyPath(base, key) { + return /^[A-Za-z_][A-Za-z0-9_]*$/.test(key) + ? `${base}.${key}` + : `${base}[${safeQuote(key)}]`; +} + +function createErrorList() { + const errors = []; + Object.defineProperty(errors, "push", { + value(...messages) { + const remaining = LIMITS.validationErrors - this.length; + if (remaining > 0) { + Array.prototype.push.apply(this, messages.slice(0, remaining).map(escapeUnsafeDiagnosticCharacters)); + } + return this.length; + }, + }); + return errors; +} + +function typeOf(value) { + if (Array.isArray(value)) return "array"; + if (value === null) return "null"; + return typeof value; +} + +function deepEqual(left, right) { + if (left === right) return true; + if (typeOf(left) !== typeOf(right)) return false; + if (Array.isArray(left)) { + return left.length === right.length && left.every((value, index) => deepEqual(value, right[index])); + } + if (left !== null && typeof left === "object") { + const leftKeys = Object.keys(left); + const rightKeys = Object.keys(right); + return leftKeys.length === rightKeys.length && + leftKeys.every((key) => hasOwn(right, key) && deepEqual(left[key], right[key])); + } + return false; +} + +function codePointLength(value) { + let length = 0; + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff && index < value.length) { + const second = value.charCodeAt(index); + if (second >= 0xdc00 && second <= 0xdfff) index++; + } + length++; + } + return length; +} + +function hasValidUnicodeScalarValues(value) { + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff) { + if (index >= value.length) return false; + const second = value.charCodeAt(index++); + if (second < 0xdc00 || second > 0xdfff) return false; + } else if (first >= 0xdc00 && first <= 0xdfff) { + return false; + } + } + return true; +} + +function hasVisibleProse(value) { + return hasValidUnicodeScalarValues(value) && VISIBLE_CONTENT.test(value); +} + +function canonicalKey(value) { + const chunks = []; + let bytes = 0; + + function append(chunk) { + bytes += Buffer.byteLength(chunk); + if (bytes > LIMITS.canonicalKeyBytes) { + throw new Error(`canonical key exceeds ${LIMITS.canonicalKeyBytes} byte limit`); + } + chunks.push(chunk); + } + + function encode(item) { + const type = typeOf(item); + if (type === "null") { + append("null"); + } else if (type === "string") { + append(`string:${JSON.stringify(item)}`); + } else if (type === "number") { + append(`number:${Object.is(item, -0) ? "0" : String(item)}`); + } else if (type === "boolean") { + append(`boolean:${item ? "true" : "false"}`); + } else if (type === "array") { + append("array:["); + item.forEach((entry, index) => { + if (index > 0) append(","); + encode(entry); + }); + append("]"); + } else if (type === "object") { + append("object:{"); + Object.keys(item).sort().forEach((key, index) => { + if (index > 0) append(","); + append(JSON.stringify(key)); + append(":"); + encode(item[key]); + }); + append("}"); + } else { + append(`${type}:${String(item)}`); + } + } + + encode(value); + return { key: chunks.join(""), bytes }; +} + +function collectDataLimitErrors(value, location = "$data") { + const stack = [{ value, location, depth: value !== null && typeof value === "object" ? 1 : 0 }]; + const seen = new WeakSet(); + + while (stack.length > 0) { + const current = stack.pop(); + if (current.value === null || typeof current.value !== "object") continue; + if (current.depth > LIMITS.nestingDepth) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: exceeds ${LIMITS.nestingDepth} level nesting depth limit`)]; + } + if (seen.has(current.value)) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: input must not contain repeated or cyclic object references`)]; + } + seen.add(current.value); + + if (Array.isArray(current.value)) { + if (current.value.length > LIMITS.arrayItems) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: exceeds ${LIMITS.arrayItems} item array limit`)]; + } + for (let index = current.value.length - 1; index >= 0; index--) { + const child = current.value[index]; + if (child !== null && typeof child === "object") { + stack.push({ value: child, location: `${current.location}[${index}]`, depth: current.depth + 1 }); + } + } + } else { + const keys = Object.keys(current.value); + for (let index = keys.length - 1; index >= 0; index--) { + const key = keys[index]; + const child = current.value[key]; + if (child !== null && typeof child === "object") { + stack.push({ value: child, location: propertyPath(current.location, key), depth: current.depth + 1 }); + } + } + } + } + + return []; +} + +function collectSchemaErrors(schema, location = "schema") { + const errors = createErrorList(); + + function check(node, p) { + if (node === null || typeof node !== "object" || Array.isArray(node)) { + errors.push(`${p}: schema must be an object`); + return; + } + + for (const key of Object.keys(node)) { + if (!SUPPORTED_KEYWORDS.has(key)) errors.push(`${p}: unsupported schema keyword ${safeQuote(key)}`); + } + + if (hasOwn(node, "$comment") && typeof node.$comment !== "string") { + errors.push(`${p}.$comment: expected string`); + } + if (hasOwn(node, "description") && typeof node.description !== "string") { + errors.push(`${p}.description: expected string`); + } + if (hasOwn(node, "type") && (!SUPPORTED_TYPES.has(node.type))) { + errors.push(`${p}.type: unsupported type ${safeQuote(node.type)}`); + } + if (hasOwn(node, "properties")) { + if (node.properties === null || typeof node.properties !== "object" || Array.isArray(node.properties)) { + errors.push(`${p}.properties: expected object`); + } else { + for (const key of Object.keys(node.properties)) check(node.properties[key], propertyPath(`${p}.properties`, key)); + } + } + if (hasOwn(node, "required")) { + if (!Array.isArray(node.required) || node.required.some((key) => typeof key !== "string")) { + errors.push(`${p}.required: expected an array of strings`); + } else if (new Set(node.required).size !== node.required.length) { + errors.push(`${p}.required: entries must be unique`); + } + } + if (hasOwn(node, "additionalProperties") && typeof node.additionalProperties !== "boolean") { + errors.push(`${p}.additionalProperties: only boolean values are supported`); + } + if (hasOwn(node, "enum")) { + if (!Array.isArray(node.enum) || node.enum.length === 0) { + errors.push(`${p}.enum: expected a non-empty array`); + } else { + const seen = new Set(); + for (const value of node.enum) { + let key; + try { + key = canonicalKey(value).key; + } catch (error) { + errors.push(`${p}.enum: ${error.message}`); + break; + } + if (seen.has(key)) { + errors.push(`${p}.enum: entries must be unique`); + break; + } + seen.add(key); + } + } + } + if (hasOwn(node, "items")) check(node.items, `${p}.items`); + for (const keyword of ["minItems", "minLength"]) { + if (hasOwn(node, keyword) && (!Number.isInteger(node[keyword]) || node[keyword] < 0)) { + errors.push(`${p}.${keyword}: expected a non-negative integer`); + } + } + if (hasOwn(node, "minimum") && (typeof node.minimum !== "number" || !Number.isFinite(node.minimum))) { + errors.push(`${p}.minimum: expected a finite number`); + } + if (hasOwn(node, "pattern")) { + if (typeof node.pattern !== "string") { + errors.push(`${p}.pattern: expected string`); + } else { + try { + new RegExp(node.pattern); + } catch (error) { + errors.push(`${p}.pattern: invalid regular expression`); + } + } + } + if (hasOwn(node, "uniqueItems") && typeof node.uniqueItems !== "boolean") { + errors.push(`${p}.uniqueItems: expected boolean`); + } + if (hasOwn(node, "visibleContent")) { + if (typeof node.visibleContent !== "boolean") { + errors.push(`${p}.visibleContent: expected boolean`); + } else if (node.visibleContent === true && node.type !== "string") { + errors.push(`${p}.visibleContent: requires type "string"`); + } + } + if (hasOwn(node, "oneOf")) { + if (!Array.isArray(node.oneOf) || node.oneOf.length === 0) { + errors.push(`${p}.oneOf: expected a non-empty array`); + } else { + node.oneOf.forEach((branch, index) => check(branch, `${p}.oneOf[${index}]`)); + } + } + } + + check(schema, location); + return errors; +} + +function findDiscriminator(schema) { + if (!hasOwn(schema, "properties") || typeof schema.properties !== "object") return null; + for (const key of Object.keys(schema.properties)) { + const subSchema = schema.properties[key]; + if (subSchema && typeof subSchema === "object" && hasOwn(subSchema, "const")) { + return { key, value: subSchema.const }; + } + } + return null; +} + +function validate(value, schema, p, errors) { + if (errors.length >= LIMITS.validationErrors) return; + if (hasOwn(schema, "oneOf")) { + const results = schema.oneOf.map((branch) => collectUnchecked(value, branch, p)); + const passingIndexes = results + .map((branchErrors, index) => branchErrors.length === 0 ? index : -1) + .filter((index) => index !== -1); + + if (passingIndexes.length !== 1) { + errors.push(`${p}: must match exactly one schema in oneOf; matched ${passingIndexes.length}`); + if (passingIndexes.length === 0 && value !== null && typeof value === "object" && !Array.isArray(value)) { + const matchingDiscriminators = schema.oneOf + .map((branch, index) => ({ discriminator: findDiscriminator(branch), index })) + .filter(({ discriminator }) => discriminator && hasOwn(value, discriminator.key) && deepEqual(value[discriminator.key], discriminator.value)); + if (matchingDiscriminators.length === 1) { + errors.push(...results[matchingDiscriminators[0].index]); + } + } + } + } + + if (hasOwn(schema, "const") && !deepEqual(value, schema.const)) { + errors.push(`${p}: must equal ${safeQuote(schema.const)}, got ${safeQuote(value)}`); + } + if (hasOwn(schema, "enum") && !schema.enum.some((allowed) => deepEqual(value, allowed))) { + const allowed = schema.enum.map(safeQuote).join(", "); + errors.push(`${p}: invalid value ${safeQuote(value)} (expected one of ${allowed})`); + } + + if (hasOwn(schema, "type") && typeOf(value) !== schema.type && !(schema.type === "integer" && typeOf(value) === "number" && Number.isInteger(value))) { + errors.push(`${p}: expected ${schema.type}, got ${typeOf(value)}`); + return; + } + + if (typeOf(value) === "object") { + for (const req of hasOwn(schema, "required") ? schema.required : []) { + if (!hasOwn(value, req)) errors.push(`${p}: missing required field ${safeQuote(req)}`); + } + for (const key of Object.keys(value)) { + if (hasOwn(schema, "properties") && hasOwn(schema.properties, key)) { + validate(value[key], schema.properties[key], propertyPath(p, key), errors); + } else if (hasOwn(schema, "additionalProperties") && schema.additionalProperties === false) { + errors.push(`${p}: unexpected field ${safeQuote(key)}`); + } + } + } + + if (Array.isArray(value)) { + if (hasOwn(schema, "minItems") && value.length < schema.minItems) { + errors.push(`${p}: must have at least ${schema.minItems} item(s), got ${value.length}`); + } + if (hasOwn(schema, "uniqueItems") && schema.uniqueItems === true) { + const seen = new Set(); + let setBytes = 0; + for (let i = 0; i < value.length; i++) { + let canonical; + try { + canonical = canonicalKey(value[i]); + } catch (error) { + errors.push(`${p}[${i}]: ${error.message}`); + break; + } + if (seen.has(canonical.key)) { + errors.push(`${p}: items must be unique; duplicate at index ${i}`); + continue; + } + setBytes += canonical.bytes; + if (setBytes > LIMITS.uniqueSetBytes) { + errors.push(`${p}: canonical uniqueness set exceeds ${LIMITS.uniqueSetBytes} byte limit`); + break; + } + seen.add(canonical.key); + } + } + if (hasOwn(schema, "items")) { + value.forEach((item, index) => validate(item, schema.items, `${p}[${index}]`, errors)); + } + } + + if (typeof value === "string") { + if (hasOwn(schema, "minLength") && codePointLength(value) < schema.minLength) { + errors.push(`${p}: must have at least ${schema.minLength} character(s)`); + } + if (schema.visibleContent === true) { + if (!hasValidUnicodeScalarValues(value)) { + errors.push(`${p}: must contain only valid Unicode scalar values`); + } else if (!VISIBLE_CONTENT.test(value)) { + errors.push(`${p}: must contain a visible character`); + } + } + if (hasOwn(schema, "pattern") && !(new RegExp(schema.pattern).test(value))) { + errors.push(`${p}: must match pattern ${JSON.stringify(schema.pattern)}`); + } + } + + if (typeof value === "number" && hasOwn(schema, "minimum") && value < schema.minimum) { + errors.push(`${p}: must be at least ${schema.minimum}, got ${value}`); + } +} + +function collectUnchecked(value, schema, p) { + const errors = createErrorList(); + validate(value, schema, p, errors); + return errors; +} + +function collect(value, schema, p = "$data") { + const limitErrors = collectDataLimitErrors(value, p); + if (limitErrors.length > 0) return limitErrors; + return collectUnchecked(value, schema, p); +} + +function isSafeRelativeSourcePath(value) { + if (typeof value !== "string" || value.length === 0 || !hasValidUnicodeScalarValues(value) || value.trim() !== value || PATH_FORBIDDEN_CHARACTER.test(value) || value.includes("\\") || value.includes(":")) return false; + if (path.posix.isAbsolute(value) || path.win32.isAbsolute(value) || /^[A-Za-z]:/.test(value) || value.startsWith("~")) return false; + const segments = value.split("/"); + return segments.every((segment) => + segment !== "" && + segment !== "." && + segment !== ".." && + !/[ .]$/u.test(segment) && + !WINDOWS_RESERVED_COMPONENT.test(segment)); +} + +function collectFindingSemanticErrors(findings) { + const errors = createErrorList(); + if (!Array.isArray(findings)) return errors; + + const fingerprints = new Map(); + let previousFingerprint = null; + findings.forEach((finding, index) => { + if (errors.length >= LIMITS.validationErrors) return; + if (!finding || typeof finding !== "object" || Array.isArray(finding)) return; + const base = `$[${index}]`; + + if (hasOwn(finding, "fingerprint") && typeof finding.fingerprint === "string") { + if (fingerprints.has(finding.fingerprint)) { + errors.push(`${base}.fingerprint: duplicate of $[${fingerprints.get(finding.fingerprint)}].fingerprint`); + } else { + fingerprints.set(finding.fingerprint, index); + } + if (previousFingerprint !== null && previousFingerprint > finding.fingerprint) { + errors.push(`${base}.fingerprint: findings must be sorted lexicographically`); + } + previousFingerprint = finding.fingerprint; + } + + for (const field of ["trace", "evidence"]) { + if (!hasOwn(finding, field) || !Array.isArray(finding[field])) continue; + finding[field].forEach((entry, entryIndex) => { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) return; + if (hasOwn(entry, "line") && (!Number.isInteger(entry.line) || entry.line < 1)) { + errors.push(`${base}.${field}[${entryIndex}].line: must be a positive integer`); + } + if (hasOwn(entry, "file") && !isSafeRelativeSourcePath(entry.file)) { + errors.push(`${base}.${field}[${entryIndex}].file: must be a safe repository-relative source path`); + } + }); + } + if (finding.remediation && Array.isArray(finding.remediation.code_changes)) { + finding.remediation.code_changes.forEach((change, changeIndex) => { + if (change && hasOwn(change, "file_name") && !isSafeRelativeSourcePath(change.file_name)) { + errors.push(`${base}.remediation.code_changes[${changeIndex}].file_name: must be a safe repository-relative source path`); + } + }); + } + + if (Array.isArray(finding.trace) && finding.trace.length === 1) { + const kind = finding.trace[0] && finding.trace[0].kind; + if (kind !== "entrypoint" && kind !== "sink") { + errors.push(`${base}.trace[0].kind: a one-line trace must be "entrypoint" or "sink"`); + } + } else if (Array.isArray(finding.trace) && finding.trace.length > 1) { + const last = finding.trace.length - 1; + if (finding.trace[0] && finding.trace[0].kind !== "entrypoint") { + errors.push(`${base}.trace[0].kind: must be "entrypoint", got ${safeQuote(finding.trace[0].kind)}`); + } + if (finding.trace[last] && finding.trace[last].kind !== "sink") { + errors.push(`${base}.trace[${last}].kind: must be "sink", got ${safeQuote(finding.trace[last].kind)}`); + } + for (let traceIndex = 1; traceIndex < last; traceIndex++) { + if (finding.trace[traceIndex] && finding.trace[traceIndex].kind !== "propagation") { + errors.push(`${base}.trace[${traceIndex}].kind: must be "propagation", got ${safeQuote(finding.trace[traceIndex].kind)}`); + } + } + } + + const verdict = finding.verdict; + if (verdict === "confirmed") { + for (const forbidden of ["claimed_root_cause", "blockers", "validation_plan", "reason"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: confirmed finding must not contain ${safeQuote(forbidden)}`); + } + if (!finding.execution || typeof finding.execution !== "object" || typeof finding.execution.observed_result !== "string" || !hasVisibleProse(finding.execution.observed_result)) { + errors.push(`${base}: confirmed finding requires a visible execution observed_result`); + } + if (!finding.remediation || typeof finding.remediation !== "object" || typeof finding.remediation.strategy !== "string" || !hasVisibleProse(finding.remediation.strategy)) { + errors.push(`${base}: confirmed finding requires visible remediation`); + } + const overall = finding.severity && finding.severity.overall_severity; + const impact = finding.severity && finding.severity.impact && finding.severity.impact.score; + if (SEVERITY_RANK.has(overall) && SEVERITY_RANK.has(impact) && SEVERITY_RANK.get(overall) > SEVERITY_RANK.get(impact)) { + errors.push(`${base}.severity.overall_severity: cannot exceed demonstrated impact ${safeQuote(impact)}`); + } + } else if (verdict === "needs_validation") { + if (hasOwn(finding, "severity")) errors.push(`${base}: needs_validation finding must not contain "severity"`); + for (const forbidden of ["execution", "remediation", "reason", "root_cause"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: needs_validation finding must not contain ${safeQuote(forbidden)}`); + } + const plan = finding.validation_plan; + const hasLocalPlan = plan && typeof plan.local === "string" && hasVisibleProse(plan.local); + const hasDeploymentPlan = plan && typeof plan.deployment === "string" && hasVisibleProse(plan.deployment); + if (!hasLocalPlan && !hasDeploymentPlan) { + errors.push(`${base}.validation_plan: requires at least one visible local or deployment plan`); + } + } else if (verdict === "rejected") { + for (const forbidden of ["severity", "execution", "remediation", "blockers", "validation_plan", "root_cause"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: rejected finding must not contain ${safeQuote(forbidden)}`); + } + } + }); + + return errors; +} + +function validateDocument(findings, schema) { + const schemaErrors = collectSchemaErrors(schema); + if (schemaErrors.length > 0) return schemaErrors; + const limitErrors = collectDataLimitErrors(findings, "$"); + if (limitErrors.length > 0) return limitErrors; + const errors = collectUnchecked(findings, schema, "$"); + if (errors.length < LIMITS.validationErrors) { + errors.push(...collectFindingSemanticErrors(findings)); + } + return errors; +} + +function loadSchema(schemaPath) { + const schema = JSON.parse(fs.readFileSync(schemaPath, "utf8")); + const errors = collectSchemaErrors(schema); + if (errors.length > 0) throw new Error(`unsupported or invalid report schema:\n${errors.join("\n")}`); + return schema; +} + +function readFileWithinLimit(file) { + const noFollow = fs.constants.O_NOFOLLOW; + const nonBlock = fs.constants.O_NONBLOCK; + if (!Number.isInteger(noFollow) || noFollow === 0 || !Number.isInteger(nonBlock) || nonBlock === 0) { + throw new SafeInputError("OS no-follow and nonblocking input protection is unavailable"); + } + + let descriptor; + try { + descriptor = fs.openSync(file, fs.constants.O_RDONLY | noFollow | nonBlock); + } catch (error) { + if (error && (error.code === "ELOOP" || error.code === "EMLINK")) { + throw new SafeInputError("input must not be a symlink"); + } + throw error; + } + try { + const stat = fs.fstatSync(descriptor); + if (!stat.isFile()) { + throw new SafeInputError("input must be a regular file"); + } + if (stat.size > LIMITS.inputBytes) { + throw new SafeInputError(`input exceeds ${LIMITS.inputBytes} byte limit`); + } + + const chunks = []; + const buffer = Buffer.allocUnsafe(64 * 1024); + let bytesRead = 0; + while (true) { + const count = fs.readSync(descriptor, buffer, 0, buffer.length, null); + if (count === 0) break; + bytesRead += count; + if (bytesRead > LIMITS.inputBytes) { + throw new SafeInputError(`input exceeds ${LIMITS.inputBytes} byte limit`); + } + chunks.push(Buffer.from(buffer.subarray(0, count))); + } + try { + return UTF8_DECODER.decode(Buffer.concat(chunks, bytesRead)); + } catch { + throw new SafeInputError("input is not valid UTF-8"); + } + } finally { + fs.closeSync(descriptor); + } +} + +function enforceJsonTextLimits(contents) { + const containers = []; + let inString = false; + let escaped = false; + + function markArrayItem() { + const container = containers[containers.length - 1]; + if (!container || container.type !== "array" || !container.expectsItem) return; + container.expectsItem = false; + container.items++; + if (container.items > LIMITS.arrayItems) { + throw new JsonStructureError(`input exceeds ${LIMITS.arrayItems} item array limit`); + } + } + + for (let index = 0; index < contents.length; index++) { + const character = contents[index]; + if (inString) { + if (escaped) { + escaped = false; + } else if (character === "\\") { + escaped = true; + } else if (character === "\"") { + inString = false; + } + continue; + } + + if (character === "\"") { + markArrayItem(); + inString = true; + } else if (character === "[" || character === "{") { + markArrayItem(); + if (containers.length >= LIMITS.nestingDepth) { + throw new JsonStructureError(`input exceeds ${LIMITS.nestingDepth} level nesting depth limit`); + } + containers.push({ + type: character === "[" ? "array" : "object", + expectsItem: character === "[", + items: 0, + }); + } else if (character === "]" || character === "}") { + containers.pop(); + } else if (character === ",") { + const container = containers[containers.length - 1]; + if (container && container.type === "array") container.expectsItem = true; + } else if (!/\s/.test(character)) { + markArrayItem(); + } + } +} + +function run(file) { + if (!file) { + console.error("Usage: node validate-findings.cjs <path-to-findings.json>"); + return 1; + } + + let schema; + try { + schema = loadSchema(path.join(__dirname, "report-schema.json")); + } catch (error) { + console.error("Failed to load report-schema.json:", error.message); + return 1; + } + + let contents; + try { + contents = readFileWithinLimit(file); + } catch (error) { + const reason = error instanceof SafeInputError ? error.message : "input could not be opened or read safely"; + console.error(`Failed to read findings JSON: ${reason}`); + return 1; + } + + try { + enforceJsonTextLimits(contents); + } catch (error) { + const reason = error instanceof JsonStructureError ? error.message : "invalid JSON structure"; + console.error(`Failed to parse findings JSON: ${reason}`); + return 1; + } + + let findings; + try { + findings = JSON.parse(contents); + } catch { + console.error("Failed to parse findings JSON: invalid JSON syntax"); + return 1; + } + + let errors; + try { + errors = validateDocument(findings, schema); + } catch { + console.error("Failed to validate findings JSON: unexpected validation error"); + return 1; + } + for (const message of errors) console.error("ERROR:", escapeUnsafeDiagnosticCharacters(message)); + if (errors.length > 0) { + const cap = errors.length === LIMITS.validationErrors ? `; output capped at ${LIMITS.validationErrors}` : ""; + console.error(`FAIL: ${errors.length} validation error(s)${cap}`); + return 1; + } + console.log(`PASS: ${findings.length} findings valid`); + return 0; +} + +module.exports = { + LIMITS, + PATH_FORBIDDEN_CHARACTER, + UNSAFE_DIAGNOSTIC_CHARACTER, + VISIBLE_CONTENT, + WINDOWS_RESERVED_COMPONENT, + collect, + collectFindingSemanticErrors, + collectSchemaErrors, + hasVisibleProse, + isSafeRelativeSourcePath, + validateDocument, +}; + +if (require.main === module) process.exit(run(process.argv[2])); +</code></pre> + <h4 id="file-thirdshift-security-audit-validate-findings-test-cjs"><code>thirdshift-security-audit/validate-findings.test.cjs</code></h4> + <pre class="job-text"><code>const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const { spawnSync } = require("node:child_process"); +const test = require("node:test"); +const schema = require("./report-schema.json"); +const { + LIMITS, + collect, + collectSchemaErrors, + validateDocument, +} = require("./validate-findings.cjs"); + +const validatorPath = path.join(__dirname, "validate-findings.cjs"); +const CLI_TIMEOUT_MS = 5000; +const HOSTILE_CLI_TIMEOUT_MS = 15000; +const HAS_SAFE_INPUT_OPEN = Number.isInteger(fs.constants.O_NOFOLLOW) && + fs.constants.O_NOFOLLOW !== 0 && + Number.isInteger(fs.constants.O_NONBLOCK) && + fs.constants.O_NONBLOCK !== 0; +const TERMINAL_CONTROL_PAYLOAD = "\u001b\u0007\u0085\u202e\u034f\ufe0f"; +const TERMINAL_CONTROL_BYTES = [ + Buffer.from([0x1b]), + Buffer.from([0x07]), + Buffer.from("\u0085"), + Buffer.from("\u202e"), + Buffer.from("\u034f"), + Buffer.from("\ufe0f"), +]; + +function source(kind = "entrypoint", file = "src/handler.c", line = 10) { + return { kind, file, line, scope: "handle", description: "Attacker data reaches the operation." }; +} + +function evidence(file = "src/handler.c", line = 10) { + return { file, line, description: "The source performs the operation without the required check." }; +} + +function confirmed() { + return { + verdict: "confirmed", + fingerprint: "src-handler-missing-check", + title: "Missing ownership check", + description: "An attacker can reach an operation without the intended ownership check.", + root_cause: "handle omits the ownership check before changing the object.", + intended_behavior: "Only the object's owner can change it.", + trace: [source("entrypoint"), source("propagation", "src/model.c", 20), source("sink", "src/store.c", 30)], + evidence: [evidence()], + conditions: [], + execution: { + attacker_perspective: "An unprivileged remote user with their own account.", + payloads: ["An object identifier owned by another user."], + instructions: ["Submit the identifier through the public operation."], + observed_result: "The other user's object changes.", + }, + remediation: { strategy: "Check ownership before the state change." }, + severity: { + likelihood: { score: "medium", reason: "The operation is directly reachable." }, + impact: { score: "medium", reason: "The attacker changes one protected object." }, + overall_severity: "medium", + }, + confidence: { score: "high", reason: "The source path and result were reproduced." }, + }; +} + +function needsValidation() { + return { + verdict: "needs_validation", + fingerprint: "src-parser-size-hypothesis", + title: "Unchecked parsed size", + description: "A parsed size may reach an allocation without a limit.", + claimed_root_cause: "parse_size may pass an unbounded value to allocate.", + trace: [source()], + evidence: [evidence()], + blockers: ["The generated parser source is absent from this checkout."], + validation_plan: { + local: "Generate the parser and submit the smallest input that exceeds the documented limit.", + deployment: "In an approved test deployment, confirm the request reaches the generated parser and record the bounded observable result.", + }, + }; +} + +function rejected() { + return { + verdict: "rejected", + fingerprint: "src-router-auth-bypass", + title: "Authorization bypass in router", + description: "The candidate claimed a route bypassed authorization.", + claimed_root_cause: "dispatch was claimed to skip the authorization wrapper.", + trace: [source("sink")], + evidence: [evidence()], + reason: "All routes pass through the authorization wrapper before dispatch.", + }; +} + +function errorsFor(value) { + return validateDocument(value, schema); +} + +function runCli(contents, options = {}) { + const { nodeArgs = [], timeout = CLI_TIMEOUT_MS } = options; + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-")); + const findingsPath = path.join(directory, "findings.json"); + try { + fs.writeFileSync(findingsPath, contents); + return spawnSync(process.execPath, [...nodeArgs, validatorPath, findingsPath], { + encoding: "utf8", + timeout, + }); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +function cliOutput(result) { + return `${result.stdout}${result.stderr}`; +} + +function assertNoInjectedControlBytes(output) { + const bytes = Buffer.isBuffer(output) ? output : Buffer.from(output, "utf8"); + for (const marker of TERMINAL_CONTROL_BYTES) { + assert.equal(bytes.indexOf(marker), -1, `found raw control bytes ${marker.toString("hex")}`); + } +} + +function producerShapedFindings() { + const demonstrated = confirmed(); + demonstrated.conditions = [{ + kind: "authentication_level", + description: "The attacker needs a normal account.", + }]; + demonstrated.execution.payloads = ["", " \t\r\n", "\u0000\u001f\u007f", "\u034f", "\ufe0f", "\ud800", "\udc00", "[{,}]\\\""]; + demonstrated.remediation.code_changes = [{ + file_name: "src/handler.c", + fixed_code: "", + }]; + + const blocked = needsValidation(); + delete blocked.validation_plan.deployment; + + return [demonstrated, blocked, rejected()]; +} + +function rejectMutation(factory, mutate) { + const value = factory(); + mutate(value); + assert.notEqual(errorsFor([value]).length, 0); +} + +test("schema is an actual top-level array with exactly three branches", () => { + assert.equal(schema.type, "array"); + assert.equal(schema.items.oneOf.length, 3); + assert.deepEqual(schema.items.oneOf.map((branch) => branch.properties.verdict.const), [ + "confirmed", "needs_validation", "rejected", + ]); + const confirmedSchema = schema.items.oneOf[0].properties; + assert.equal(confirmedSchema.title.visibleContent, true); + assert.equal(confirmedSchema.execution.properties.payloads.items.minLength, undefined); + assert.equal(confirmedSchema.execution.properties.payloads.items.visibleContent, undefined); + assert.equal(confirmedSchema.remediation.properties.code_changes.items.properties.fixed_code.minLength, undefined); +}); + +test("accepts a producer-shaped findings document through the CLI", () => { + const result = runCli(JSON.stringify(producerShapedFindings())); + assert.equal(result.status, 0, cliOutput(result)); + assert.match(result.stdout, /PASS: 3 findings valid/); +}); + +test("accepts empty output and each complete branch", () => { + assert.deepEqual(errorsFor([]), []); + assert.deepEqual(errorsFor([confirmed(), needsValidation(), rejected()]), []); + const localOnly = needsValidation(); + delete localOnly.validation_plan.deployment; + assert.deepEqual(errorsFor([localOnly]), []); + const deploymentOnly = needsValidation(); + delete deploymentOnly.validation_plan.local; + assert.deepEqual(errorsFor([deploymentOnly]), []); +}); + +test("allows a one-line finding trace", () => { + const finding = confirmed(); + finding.trace = [source("entrypoint")]; + assert.deepEqual(errorsFor([finding]), []); +}); + +test("rejects empty required content", () => { + const cases = [ + [confirmed, (finding) => { finding.title = ""; }], + [confirmed, (finding) => { finding.title = " "; }], + [confirmed, (finding) => { finding.evidence = []; }], + [confirmed, (finding) => { finding.execution.payloads = []; }], + [confirmed, (finding) => { finding.execution.instructions = []; }], + [confirmed, (finding) => { finding.execution.observed_result = ""; }], + [confirmed, (finding) => { finding.remediation.strategy = ""; }], + [needsValidation, (finding) => { finding.blockers = []; }], + [needsValidation, (finding) => { finding.validation_plan = {}; }], + [needsValidation, (finding) => { finding.validation_plan = { local: " " }; }], + [rejected, (finding) => { finding.claimed_root_cause = ""; }], + ]; + for (const [factory, mutate] of cases) rejectMutation(factory, mutate); +}); + +test("preserves exact payload and replacement-code strings", () => { + const finding = confirmed(); + const payloads = ["", " \t\r\n", "\u0000\u001f\u007f", "\u034f", "\ufe0f", "\ud800", "\udc00"]; + const fixedCode = "\u0000 \t\r\n\u001f\u007f\u034f\ufe0f\ud800x\udc00"; + finding.execution.payloads = payloads.slice(); + finding.remediation.code_changes = [{ file_name: "src/handler.c", fixed_code: fixedCode }]; + + assert.deepEqual(errorsFor([finding]), []); + assert.deepEqual(finding.execution.payloads, payloads); + assert.equal(finding.remediation.code_changes[0].fixed_code, fixedCode); +}); + +test("rejects invalid scalars and whitespace, control, format, or default-ignorable prose", () => { + for (const invisible of ["\u0000\t\r\n\u001f\u007f\u200b", "\u034f", "\ufe0f", "\ud800", "\udc00", "visible\ud800"]) { + const cases = [ + [confirmed, (finding) => { finding.title = invisible; }], + [confirmed, (finding) => { finding.trace[0].scope = invisible; }], + [confirmed, (finding) => { finding.evidence[0].description = invisible; }], + [confirmed, (finding) => { finding.execution.instructions = [invisible]; }], + [confirmed, (finding) => { finding.remediation.strategy = invisible; }], + [confirmed, (finding) => { finding.severity.impact.reason = invisible; }], + [confirmed, (finding) => { finding.confidence.reason = invisible; }], + [needsValidation, (finding) => { finding.blockers = [invisible]; }], + [needsValidation, (finding) => { finding.validation_plan = { local: invisible }; }], + [rejected, (finding) => { finding.reason = invisible; }], + ]; + for (const [factory, mutate] of cases) rejectMutation(factory, mutate); + } +}); + +test("quotes input-derived controls in direct validation values and paths", () => { + const finding = confirmed(); + finding.trace[0].kind = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + finding.execution[`extra-${TERMINAL_CONTROL_PAYLOAD}`] = "value"; + + const cyclic = {}; + cyclic[`path-${TERMINAL_CONTROL_PAYLOAD}`] = cyclic; + const output = [ + ...errorsFor([finding]), + ...collect(cyclic, { type: "object" }, "$input"), + ].join("\n"); + + for (const escaped of ["\\u001b", "\\u0007", "\\u0085", "\\u202e", "\\u034f", "\\ufe0f"]) { + assert(output.includes(escaped), `missing escaped diagnostic ${escaped}`); + } + assertNoInjectedControlBytes(output); +}); + +test("rejects line zero", () => { + rejectMutation(confirmed, (finding) => { finding.trace[0].line = 0; }); + rejectMutation(rejected, (finding) => { finding.evidence[0].line = 0; }); +}); + +test("does not treat inherited or Object-prototype properties as schema properties", () => { + rejectMutation(confirmed, (finding) => { finding.constructor = "not allowed"; }); + const inherited = Object.create({ verdict: "confirmed" }); + assert(errorsFor([inherited]).some((error) => error.includes("exactly one"))); + assert(collect(Object.create({ constructor: "inherited" }), { + type: "object", + properties: { constructor: { type: "string" } }, + required: ["constructor"], + additionalProperties: false, + }).some((error) => error.includes("missing required"))); +}); + +test("oneOf requires exactly one passing branch", () => { + assert(collect("value", { oneOf: [{ type: "string" }, { minLength: 1 }] }, "$test") + .some((error) => error.includes("matched 2"))); + assert(collect(7, { oneOf: [{ type: "string" }, { minimum: 10 }] }, "$test") + .some((error) => error.includes("matched 0"))); +}); + +test("rejects duplicate fingerprints and unique array entries", () => { + const first = confirmed(); + const second = rejected(); + second.fingerprint = first.fingerprint; + assert(errorsFor([first, second]).some((error) => error.includes("duplicate of"))); + rejectMutation(needsValidation, (finding) => { finding.blockers = [finding.blockers[0], finding.blockers[0]]; }); +}); + +test("uses canonical Set uniqueness for structured entries at the array limit", () => { + const entries = Array.from({ length: LIMITS.arrayItems }, (_, id) => ({ id, label: String(id) })); + assert.deepEqual(collect(entries, { type: "array", uniqueItems: true }), []); + + const duplicate = entries.slice(0, -1); + duplicate.push({ label: "0", id: 0 }); + assert(collect(duplicate, { type: "array", uniqueItems: true }) + .some((error) => error.includes(`duplicate at index ${LIMITS.arrayItems - 1}`))); +}); + +test("bounds canonical uniqueness keys and Set storage", () => { + const oversizedKey = "x".repeat(LIMITS.canonicalKeyBytes + 1); + assert(collect([oversizedKey], { type: "array", uniqueItems: true }) + .some((error) => error.includes("canonical key exceeds"))); + + const itemLength = Math.floor(LIMITS.uniqueSetBytes / 6); + const largeUniqueItems = Array.from({ length: 6 }, (_, index) => `${index}${"x".repeat(itemLength)}`); + assert(collect(largeUniqueItems, { type: "array", uniqueItems: true }) + .some((error) => error.includes("canonical uniqueness set exceeds"))); +}); + +test("requires findings to be sorted by fingerprint", () => { + const first = confirmed(); + const second = rejected(); + assert(errorsFor([second, first]).some((error) => error.includes("sorted lexicographically"))); +}); + +test("rejects severity above demonstrated impact", () => { + rejectMutation(confirmed, (finding) => { + finding.severity.overall_severity = "high"; + finding.severity.impact.score = "medium"; + }); +}); + +test("rejects unsafe source paths", () => { + const badPaths = [ + "/etc/passwd", + "../src/file.c", + "src/../file.c", + "src//file.c", + "C:\\src\\file.c", + "src/file:name.c", + "src/file\nname.c", + "src/file\u0001name.c", + "src/file\u0085name.c", + "src/file\u2028name.c", + "src/file\u202ename.c", + "src/file\u2066name.c", + "src/file\u200dname.c", + "src/file\u034fname.c", + "src/file\ufe0fname.c", + "src/file\ud800name.c", + "src/file\udc00name.c", + "CON", + "src/con.txt", + "src/PRN", + "src/AUX.c", + "src/NUL", + "src/COM1.log", + "src/lpt9", + "src/CONIN$", + "src/CONOUT$.txt", + "src/CLOCK$.txt", + "src/COM\u00b9.log", + "src/LPT\u00b2.log", + "src /file.c", + "src./file.c", + "src/file.c ", + "src/file.c.", + ]; + for (const badPath of badPaths) { + rejectMutation(confirmed, (finding) => { finding.trace[0].file = badPath; }); + } + rejectMutation(rejected, (finding) => { finding.evidence[0].file = "NUL.txt"; }); + rejectMutation(confirmed, (finding) => { + finding.remediation.code_changes = [{ file_name: "src/file:name.c", fixed_code: "replacement" }]; + }); +}); + +test("accepts legitimate Unicode source paths and prose", () => { + const finding = confirmed(); + finding.title = "Finding \ud83d\ude00 cafe\u0301"; + finding.trace[0].file = "src/日本語/cafe\u0301-\ud83d\ude00.ts"; + finding.evidence[0].file = "src/mañana/файл.ts"; + finding.remediation.code_changes = [{ + file_name: "src/修正/éxito.ts", + fixed_code: "replacement", + }]; + assert.deepEqual(errorsFor([finding]), []); +}); + +test("CLI rejects input above the byte limit without an exception trace", () => { + const result = runCli(Buffer.alloc(LIMITS.inputBytes + 1, 0x20)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`input exceeds ${LIMITS.inputBytes} byte limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("CLI rejects invalid UTF-8 without replacement or an exception trace", () => { + const findings = producerShapedFindings(); + findings[0].execution.payloads = ["INVALID_UTF8"]; + const encoded = Buffer.from(JSON.stringify(findings)); + const marker = Buffer.from("INVALID_UTF8"); + const markerOffset = encoded.indexOf(marker); + assert.notEqual(markerOffset, -1); + const malformed = Buffer.concat([ + encoded.subarray(0, markerOffset), + Buffer.from([0x80]), + encoded.subarray(markerOffset + marker.length), + ]); + + const result = runCli(malformed); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /input is not valid UTF-8/); + assert.doesNotMatch(output, /TypeError|stack|at validate-findings/i); +}); + +test("quotes input-derived controls in CLI validation errors", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const finding = confirmed(); + finding.trace[0].kind = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + finding.execution[`extra-${TERMINAL_CONTROL_PAYLOAD}`] = "value"; + const result = runCli(JSON.stringify([finding])); + + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /\$\[0\]\.trace\[0\]\.kind/); + assert.match(result.stderr, /\\u001b/); + assert.match(result.stderr, /\\u202e/); + assertNoInjectedControlBytes(result.stderr); +}); + +test("returns a generic syntax error without parser-supplied controls", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const malformed = Buffer.concat([ + Buffer.from("["), + Buffer.from(TERMINAL_CONTROL_PAYLOAD), + Buffer.from("]"), + ]); + const result = runCli(malformed); + + assert.equal(result.status, 1, cliOutput(result)); + assert.equal(result.stderr, "Failed to parse findings JSON: invalid JSON syntax\n"); + assertNoInjectedControlBytes(result.stderr); +}); + +test("does not reflect controls from a failed CLI input path", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-path-")); + const missingPath = path.join(directory, `missing-${TERMINAL_CONTROL_PAYLOAD}.json`); + try { + const result = spawnSync(process.execPath, [validatorPath, missingPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /Failed to read findings JSON:/); + assertNoInjectedControlBytes(result.stderr); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects lone-surrogate prose without changing payload semantics", () => { + const findings = producerShapedFindings(); + findings[0].title = "\ud800"; + const result = runCli(JSON.stringify(findings)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /must contain only valid Unicode scalar values/); + assert.doesNotMatch(output, /stack|at validate-findings/i); +}); + +test("CLI rejects Unicode format controls in source paths", () => { + const findings = producerShapedFindings(); + findings[0].trace[0].file = "src/file\u202ename.c"; + const result = runCli(JSON.stringify(findings)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /must be a safe repository-relative source path/); + assert.doesNotMatch(output, /stack|at validate-findings/i); +}); + +test("CLI rejects a FIFO without blocking", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-fifo-")); + const fifoPath = path.join(directory, "findings.json"); + try { + const created = spawnSync("mkfifo", [fifoPath], { encoding: "utf8", timeout: CLI_TIMEOUT_MS }); + assert.equal(created.status, 0, cliOutput(created)); + + const result = spawnSync(process.execPath, [validatorPath, fifoPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must be a regular file/); + assert.doesNotMatch(output, /stack|at validate-findings/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects a symlink without following it", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-symlink-")); + const targetPath = path.join(directory, "target.json"); + const symlinkPath = path.join(directory, "findings.json"); + try { + fs.writeFileSync(targetPath, JSON.stringify(producerShapedFindings())); + fs.symlinkSync(targetPath, symlinkPath); + const result = spawnSync(process.execPath, [validatorPath, symlinkPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must not be a symlink/); + assert.doesNotMatch(output, /stack|at validate-findings/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects input above the nesting-depth limit without an exception trace", () => { + const levels = LIMITS.nestingDepth + 1; + const result = runCli(`${"[".repeat(levels)}0${"]".repeat(levels)}`); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`${LIMITS.nestingDepth} level nesting depth limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("CLI rejects an oversized array without an exception trace", () => { + const result = runCli(JSON.stringify(Array(LIMITS.arrayItems + 1).fill(null))); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`${LIMITS.arrayItems} item array limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("checks pattern and branch invariants", () => { + rejectMutation(rejected, (finding) => { finding.fingerprint = "not stable"; }); + rejectMutation(needsValidation, (finding) => { + finding.severity = { impact: { score: "low" }, overall_severity: "low" }; + }); +}); + +test("rejects unsupported and malformed schema keywords", () => { + assert(collectSchemaErrors({ type: "string", format: "uuid" }).some((error) => error.includes("format"))); + assert(collectSchemaErrors({ type: "string", pattern: "[" }).some((error) => error.includes("regular expression"))); + assert(collectSchemaErrors({ type: "string", visibleContent: "yes" }).some((error) => error.includes("expected boolean"))); + assert(collectSchemaErrors({ type: "array", visibleContent: true }).some((error) => error.includes("requires type"))); + assert.notEqual(validateDocument([], { type: "array", maxItems: 1 }).length, 0); +}); + +test("caps malformed 1000-finding validation output", () => { + assert.equal(errorsFor(Array.from({ length: LIMITS.arrayItems }, () => null)).length, LIMITS.validationErrors); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(JSON.stringify(Array.from({ length: LIMITS.arrayItems }, () => null))); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-findings/i); +}); + +test("caps amplified in-limit findings output under a constrained Node heap", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const findings = Array.from({ length: 750 }, () => ({ + verdict: "confirmed", + trace: Array.from({ length: LIMITS.arrayItems }, () => 0), + evidence: Array.from({ length: LIMITS.arrayItems }, () => 0), + })); + const contents = JSON.stringify(findings); + assert(contents.length > 3 * 1000 * 1000, `hostile input too small: ${contents.length}`); + assert(contents.length <= LIMITS.inputBytes, `hostile input over limit: ${contents.length}`); + + const result = runCli(contents, { + nodeArgs: ["--max-old-space-size=64"], + timeout: HOSTILE_CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /heap out of memory|allocation failed|RangeError|Maximum call stack/i); +}); + +test("keeps shared helpers aligned with the coverage-ledger validator", () => { + const findingsModule = require("./validate-findings.cjs"); + const ledgerModule = require("./validate-coverage-ledger.cjs"); + + for (const name of [ + "VISIBLE_CONTENT", + "PATH_FORBIDDEN_CHARACTER", + "WINDOWS_RESERVED_COMPONENT", + "UNSAFE_DIAGNOSTIC_CHARACTER", + ]) { + assert.equal(findingsModule[name].source, ledgerModule[name].source, `${name} source`); + assert.equal(findingsModule[name].flags, ledgerModule[name].flags, `${name} flags`); + } + + const sharedLimitKeys = Object.keys(findingsModule.LIMITS) + .filter((key) => Object.prototype.hasOwnProperty.call(ledgerModule.LIMITS, key)) + .sort(); + assert.deepEqual(sharedLimitKeys, ["inputBytes", "nestingDepth", "validationErrors"]); + for (const key of sharedLimitKeys) { + assert.equal(findingsModule.LIMITS[key], ledgerModule.LIMITS[key], `LIMITS.${key}`); + } + + const pathCorpus = [ + "src/handler.js", + "src/caf\u00e9/handler.js", + "src/\u65e5\u672c\u8a9e/\u0444\u0430\u0439\u043b.ts", + "src/cloc\u212a$.txt", + "src/CLOCK$.txt", + "src/con.txt", + "CON", + "src/COM\u00b9.log", + "src/lpt\u00b3", + "/etc/passwd", + "../src/file.c", + "src/../file.c", + "src//file.c", + "src\\file.c", + "src/file:name.c", + "~home/file.c", + "C:/file.c", + "src/file.c ", + "src/file.c.", + "src/file\u202ename.c", + "src/file\u200b.js", + "src/file\u034f.js", + "src/file\ufe0f.js", + "src/file\ud800name.c", + "src/file\udc00name.c", + ]; + for (const value of pathCorpus) { + assert.equal( + findingsModule.isSafeRelativeSourcePath(value), + ledgerModule.isSafeRelativePath(value), + `path verdict diverges for ${JSON.stringify(value)}`, + ); + } + assert.equal(findingsModule.isSafeRelativeSourcePath("src/cloc\u212a$.txt"), false); + assert.equal(ledgerModule.isSafeRelativePath("src/cloc\u212a$.txt"), false); + + const proseCorpus = [ + "Valid prose.", + "caf\u00e9", + "", + " \t\r\n", + "\u200b", + "\u034f", + "\ufe0f", + "\ud800", + "\udc00", + "visible\ud800", + ]; + for (const value of proseCorpus) { + assert.equal( + findingsModule.hasVisibleProse(value), + ledgerModule.hasVisibleProse(value), + `prose verdict diverges for ${JSON.stringify(value)}`, + ); + } +}); +</code></pre> + </article> + <article class="job-sheet" id="skill-thirdshift-tdd" aria-labelledby="skill-thirdshift-tdd-title"> + <h3 id="skill-thirdshift-tdd-title">thirdshift-tdd</h3> + <p class="sent-by">Used by <a href="/#unit-2">2 · M · Implement</a></p> + <h4 id="file-thirdshift-tdd-skill-md"><code>thirdshift-tdd/SKILL.md</code></h4> + <pre class="job-text"><code>--- +name: thirdshift-tdd +description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests. +--- + +# Test-Driven Development + +TDD is the red → green loop. This skill is the reference that makes that loop produce tests worth keeping: what a good test is, where tests go, the anti-patterns, and the rules of the loop. Every section applies on every cycle: consult them before and during the loop, not after. + +When exploring the codebase, read `CONTEXT.md` (if it exists) so test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching. + +## What a good test is + +Tests verify behavior through public interfaces, not implementation details. Code can change entirely; tests shouldn't. A good test reads like a specification: "user can checkout with valid cart" tells you exactly what capability exists, and it survives refactors because it doesn't care about internal structure. + +See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines. + +## Seams: where tests go + +A **seam** is the public boundary you test at: the interface where you observe behavior without reaching inside. Tests live at seams, never against internals. + +**Test only at pre-agreed seams.** Before writing any test, write down the seams under test. No test is written at a seam you haven't written down. Have the seams listed in the PR body only. You can't test everything, so agreeing the seams up front is how testing effort lands on the critical paths and complex logic instead of every edge case. + +Decide: "What's the public interface, and which seams should we test?" + +When the shape of that interface is itself in question (how deep the module is, where the seam belongs, what the interface should expose), use the `thirdshift-codebase-design` skill for the vocabulary. It is the shared source of the module, interface, depth, seam, adapter, leverage and locality terms, and it is a reference to consult, not a session to run. + +## Anti-patterns + +- **Implementation-coupled**: mocks internal collaborators, tests private methods, or verifies through a side channel (querying the database instead of using the interface). The tell: the test breaks when you refactor but behavior hasn't changed. +- **Tautological**: the assertion recomputes the expected value the way the code does (`expect(add(a, b)).toBe(a + b)`, a snapshot derived by hand the same way, a constant asserted equal to itself), so it passes by construction and can never disagree with the code. Expected values must come from an independent source of truth: a known-good literal, a worked example, the spec. +- **Horizontal slicing**: writing all tests first, then all implementation. Bulk tests verify _imagined_ behavior: you test the _shape_ of things rather than user-facing behavior, the tests go insensitive to real changes, and you commit to test structure before understanding the implementation. Work in **vertical slices** instead: one test → one implementation → repeat, each test a **tracer bullet** that responds to what the last cycle taught you. + +## Rules of the loop + +- **Red before green.** Write the failing test first, then only enough code to pass it. Don't anticipate future tests or add speculative features. +- **One slice at a time.** One seam, one test, one minimal implementation per cycle. +- **Refactoring is not part of the loop.** It belongs to the review stage (see the `thirdshift-code-review` skill), not the red → green implementation cycle. +</code></pre> + <h4 id="file-thirdshift-tdd-agents-openai-yaml"><code>thirdshift-tdd/agents/openai.yaml</code></h4> + <pre class="job-text"><code>interface: + display_name: "TDD" + short_description: "Test-driven red-green-refactor" +</code></pre> + <h4 id="file-thirdshift-tdd-mocking-md"><code>thirdshift-tdd/mocking.md</code></h4> + <pre class="job-text"><code># When to Mock + +Mock at **system boundaries** only: + +- External APIs (payment, email, etc.) +- Databases (sometimes - prefer test DB) +- Time/randomness +- File system (sometimes) + +Don't mock: + +- Your own classes/modules +- Internal collaborators +- Anything you control + +## Designing for Mockability + +At system boundaries, design interfaces that are easy to mock: + +**1. Use dependency injection** + +Pass external dependencies in rather than creating them internally: + +```typescript +// Easy to mock +function processPayment(order, paymentClient) { + return paymentClient.charge(order.total); +} + +// Hard to mock +function processPayment(order) { + const client = new StripeClient(process.env.STRIPE_KEY); + return client.charge(order.total); +} +``` + +**2. Prefer SDK-style interfaces over generic fetchers** + +Create specific functions for each external operation instead of one generic function with conditional logic: + +```typescript +// GOOD: Each function is independently mockable +const api = { + getUser: (id) => fetch(`/users/${id}`), + getOrders: (userId) => fetch(`/users/${userId}/orders`), + createOrder: (data) => fetch('/orders', { method: 'POST', body: data }), +}; + +// BAD: Mocking requires conditional logic inside the mock +const api = { + fetch: (endpoint, options) => fetch(endpoint, options), +}; +``` + +The SDK approach means: +- Each mock returns one specific shape +- No conditional logic in test setup +- Easier to see which endpoints a test exercises +- Type safety per endpoint +</code></pre> + <h4 id="file-thirdshift-tdd-tests-md"><code>thirdshift-tdd/tests.md</code></h4> + <pre class="job-text"><code># Good and Bad Tests + +## Good Tests + +**Integration-style**: Test through real interfaces, not mocks of internal parts. + +```typescript +// GOOD: Tests observable behavior +test("user can checkout with valid cart", async () => { + const cart = createCart(); + cart.add(product); + const result = await checkout(cart, paymentMethod); + expect(result.status).toBe("confirmed"); +}); +``` + +Characteristics: + +- Tests behavior users/callers care about +- Uses public API only +- Survives internal refactors +- Describes WHAT, not HOW +- One logical assertion per test + +## Bad Tests + +**Implementation-detail tests**: Coupled to internal structure. + +```typescript +// BAD: Tests implementation details +test("checkout calls paymentService.process", async () => { + const mockPayment = jest.mock(paymentService); + await checkout(cart, payment); + expect(mockPayment.process).toHaveBeenCalledWith(cart.total); +}); +``` + +Red flags: + +- Mocking internal collaborators +- Testing private methods +- Asserting on call counts/order +- Test breaks when refactoring without behavior change +- Test name describes HOW not WHAT +- Verifying through external means instead of interface + +```typescript +// BAD: Bypasses interface to verify +test("createUser saves to database", async () => { + await createUser({ name: "Alice" }); + const row = await db.query("SELECT * FROM users WHERE name = ?", ["Alice"]); + expect(row).toBeDefined(); +}); + +// GOOD: Verifies through interface +test("createUser makes user retrievable", async () => { + const user = await createUser({ name: "Alice" }); + const retrieved = await getUser(user.id); + expect(retrieved.name).toBe("Alice"); +}); +``` + +**Tautological tests**: Expected value restates the implementation, so the test passes by construction. + +```typescript +// BAD: Expected value is recomputed the way the code computes it +test("calculateTotal sums line items", () => { + const items = [{ price: 10 }, { price: 5 }]; + const expected = items.reduce((sum, i) => sum + i.price, 0); + expect(calculateTotal(items)).toBe(expected); +}); + +// GOOD: Expected value is an independent, known literal +test("calculateTotal sums line items", () => { + expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15); +}); +``` +</code></pre> + </article> + <article class="job-sheet" id="skill-thirdshift-to-spec" aria-labelledby="skill-thirdshift-to-spec-title"> + <h3 id="skill-thirdshift-to-spec-title">thirdshift-to-spec</h3> + <p class="sent-by">Used by the Architecture review, in an Architect run, or Security fix publishing, in a Security run</p> + <h4 id="file-thirdshift-to-spec-skill-md"><code>thirdshift-to-spec/SKILL.md</code></h4> + <pre class="job-text"><code>--- +name: thirdshift-to-spec +description: "Turn the current session into a spec and publish it to the project issue tracker: no interview, just synthesis of what you've already settled. For an Architecture review or Security fix publishing." +disable-model-invocation: false +--- + +This skill takes the current session context and codebase understanding and produces a spec. Nobody is there to interview or to ask; just synthesize what you already know. + +The issue tracker and triage label vocabulary should have been provided to you. If `docs/agents/issue-tracker.md` is missing, fall back to `gh issue create`. If `docs/agents/triage-labels.md` is missing, use the label names as written here, creating a label the tracker lacks (`gh label create`). + +The spec is all this skill writes. Write nothing to the repository: no spec file, commits or branches. A `CONTEXT.md` or ADR change the spec needs is work for one of its tickets. + +For **Security fix publishing**, the Session prompt overrides the template: keep the Spec terse, saying only what the fix changes and linking the private record. Keep all finding evidence in that record. On a private repository, use the finding's existing issue as the Spec and preserve its body and evidence; publish its Tickets with `thirdshift-to-tickets`. + +## Process + +1. Explore the repo to understand the current state of the codebase, if you haven't already. Use the project's domain glossary vocabulary throughout the spec, and respect any ADRs in the area you're touching. + +2. Sketch out the seams at which the feature will be tested. Existing seams should be preferred to new ones. Use the highest seam possible. If new seams are needed, propose them at the highest point you can. The fewer seams across the codebase, the better - the ideal number is one. + +Decide the seams yourself and record them, with the reason for each, under Testing Decisions. + +3. Write the spec using the template below, then publish it to the project issue tracker. Apply the `needs-triage` triage label and no other: thirdshift marks the spec ready once the session has ended and the plan passes its checks. + +<spec-template> + +## Problem Statement + +The problem that the user is facing, from the user's perspective. + +## Solution + +The solution to the problem, from the user's perspective. + +## User Stories + +A LONG, numbered list of user stories. Each user story should be in the format of: + +1. As an <actor>, I want a <feature>, so that <benefit> + +<user-story-example> +1. As a mobile bank customer, I want to see balance on my accounts, so that I can make better informed decisions about my spending +</user-story-example> + +This list of user stories should be extremely extensive and cover all aspects of the feature. + +## Implementation Decisions + +A list of implementation decisions that were made, each with its reason. This can include: + +- The modules that will be built/modified +- The interfaces of those modules that will be modified +- Architectural decisions +- Schema changes +- API contracts +- Specific interactions + +Do NOT include specific file paths or code snippets. They may end up being outdated very quickly. + +Exception: if a prototype produced a snippet that encodes a decision more precisely than prose can (state machine, reducer, schema, type shape), inline it within the relevant decision and note briefly that it came from a prototype. Trim to the decision-rich parts, not a working demo, just the important bits. + +## Testing Decisions + +A list of testing decisions that were made, each with its reason. Include: + +- A description of what makes a good test (only test external behavior, not implementation details) +- The seams the feature will be tested at +- Which modules will be tested +- Prior art for the tests (i.e. similar types of tests in the codebase) + +## Out of Scope + +A description of the things that are out of scope for this spec. + +## Further Notes + +Any further notes about the feature. + +</spec-template> +</code></pre> + <h4 id="file-thirdshift-to-spec-agents-openai-yaml"><code>thirdshift-to-spec/agents/openai.yaml</code></h4> + <pre class="job-text"><code>interface: + display_name: "To Spec" + short_description: "Turn a session into a spec" +policy: + allow_implicit_invocation: true +</code></pre> + </article> + <article class="job-sheet" id="skill-thirdshift-to-tickets" aria-labelledby="skill-thirdshift-to-tickets-title"> + <h3 id="skill-thirdshift-to-tickets-title">thirdshift-to-tickets</h3> + <p class="sent-by">Used by the Architecture review, in an Architect run, or Security fix publishing, in a Security run</p> + <h4 id="file-thirdshift-to-tickets-skill-md"><code>thirdshift-to-tickets/SKILL.md</code></h4> + <pre class="job-text"><code>--- +name: thirdshift-to-tickets +description: Break a plan, spec, or the current session into a set of tracer-bullet tickets, each declaring its blocking edges, published to the issue tracker with native sub-issue and blocking links. For an Architecture review or Security fix publishing. +disable-model-invocation: false +--- + +# To Tickets + +Break a plan, spec, or session into a set of **tickets**: tracer-bullet vertical slices, each declaring the tickets that **block** it. + +The issue tracker and triage label vocabulary should have been provided to you. If `docs/agents/issue-tracker.md` is missing, fall back to the `gh` CLI. If `docs/agents/triage-labels.md` is missing, use the label names as written here, creating a label the tracker lacks (`gh label create`). + +The tickets are all this skill writes. Write nothing to the repository: no ticket files, commits or branches. A `CONTEXT.md` or ADR change the work needs is part of a ticket's work, named in its acceptance criteria. + +For **Security fix publishing**, keep every Ticket terse: say only what the fix changes and link the private record. Keep all finding evidence in that record. A private finding's issue is the parent Spec, and its existing body and evidence stay unchanged. ## Process @@ -1477,7 +6912,7 @@ <h4 id="file-thirdshift-to-tickets-agents-openai-yaml"><code>thirdshift-to-ticke <section class="sec" aria-labelledby="licence-title"> <div class="wrap"> <div class="sec-head"><p class="eyebrow">Licence and credits</p><h2 id="licence-title">Whose skills these are</h2></div> - <p class="lede">The Factory skills are adapted from Matt Pocock's skills, under this licence. The <a href="#skill-thirdshift-pr">thirdshift-pr</a> skill credits Dex Horthy's <code>show-me</code> skill in its <a href="#file-thirdshift-pr-credits-md">CREDITS.md</a>.</p> + <p class="lede">Most Factory skills are adapted from Matt Pocock's skills, under the licence below. The <a href="#skill-thirdshift-pr">thirdshift-pr</a> skill credits Dex Horthy's <code>show-me</code> skill in its <a href="#file-thirdshift-pr-credits-md">CREDITS.md</a>. The <a href="#skill-thirdshift-security-audit">thirdshift-security-audit</a> skill is copied from <a href="https://github.com/cloudflare/security-audit-skill">Cloudflare's security-audit-skill</a>, under its own <a href="#file-thirdshift-security-audit-license">MIT licence</a>, with its upstream repository and commit recorded in <a href="#file-thirdshift-security-audit-credits-md">CREDITS.md</a>.</p> <h4 id="file-license"><code>LICENSE</code></h4> <pre class="job-text"><code>MIT License diff --git a/skills/thirdshift-security-audit/AI-AND-LLM.md b/skills/thirdshift-security-audit/AI-AND-LLM.md new file mode 100644 index 00000000..02ba9039 --- /dev/null +++ b/skills/thirdshift-security-audit/AI-AND-LLM.md @@ -0,0 +1,83 @@ +# AI, LLM, and Agent Hunting + +#### When to use this file + +Reach for this file when a language model participates in a trust-sensitive decision: chatbots and assistants, RAG pipelines, persistent agent memory, agent/tool-calling loops, MCP servers and clients, code that builds prompts from untrusted input, or code that consumes model output and acts on it. The important data flow is *untrusted content → model or memory → capability, authority, or sink*. + +Use this alongside `ATTACK-CLASSES.md`, not instead of it. Transport, access control, query construction, filesystem use, and output rendering remain ordinary trust boundaries. This file covers the model-specific delegation layer. Split large targets by retrieval, memory, tool dispatch, MCP, and output handling. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Prompt injection alone is not a finding. Require a code-level boundary failure: content reaches another principal's context, invokes authority the requester lacks, discloses data they cannot read, or drives a sink they cannot reach directly. +- Model output, memory, tool descriptions, and MCP responses are untrusted inputs. Point to the code that grants authority, trusts output, writes durable state, or feeds a sink. +- A guardrail prompt is not a security boundary. Count only deterministic checks, resource-scoped authorization, isolation, binding, and constrained credentials. +- State the attacker, affected principal, effective execution identity, resource, exact action, authority used, and observable impact. An intentional direct request to use the requester's existing authority is not a delegation defect merely because a model executes it. +- Authorization and action binding are separate controls. Attacker-controlled content that causes an action under an affected principal's valid authority is an action-binding failure when that principal did not intentionally request or approve the exact action. +- Classify every candidate as `confirmed` only after source evidence and bounded local validation establish the boundary and result. Use `needs_validation` when a required provider, deployment, model, renderer, or identity behavior is not observable locally. +``` + +## Context, retrieval, and memory attack classes (subagent_type: `general`) + +**Indirect injection through retrieved or ingested content** +An attacker can write a RAG document, indexed page, file, email, issue body, tool response, or metadata that enters a different principal's model context. Trace who can write each source, how retrieval scopes it, whose session consumes it, and what capability is enabled there. Check isolation, resource authorization, and binding to the consuming principal's intent separately. The defect is a missing deterministic control, not persuasive text by itself. + +**Cross-session or cross-tenant context bleed** +Conversation history, embeddings, retrieval results, or prompt caches are keyed too broadly. Verify tenant and ACL filters in the query itself and every cache key. A tenant field stored on an object is not enforcement if an alternate query, shared cache, or batch path omits it. + +**Persistent memory poisoning** +Attacker-controlled content or model summaries are written into memory that later shapes another task, user, or privileged session. Review who may create, update, merge, and delete memory; its provenance and tenant scope; whether low-trust observations become durable instructions or facts; and whether retrieval distinguishes user preferences from tool policy. Memory intentionally saved by a user and used only for that user's intentional, allowed requests is not a cross-boundary finding. + +**Prompt role and provenance confusion** +Prompt assembly lets untrusted text impersonate a system message, prior turn, tool result, policy, or memory record. Look for string concatenation, untyped history, caller-controlled role fields, and serialization round trips that lose source labels. Confirm that the forged provenance changes a deterministic trust decision or reaches a meaningful capability. + +## Tool and action attack classes (subagent_type: `general`) + +**Tool-argument injection into a downstream sink** +Model-produced arguments reach SQL, shell, file, URL-fetch, or privileged APIs without handler-side validation. Treat the tool schema as input parsing, then follow each field from decoded call to sink. Structured output narrows shape; it does not establish authorization, safe paths, safe URLs, or query semantics. + +**Excessive agency and confused-deputy authority** +The agent uses a service identity or broad credential, while the tool handler does not re-check the requesting principal's permission on the named resource. Verify both the effective identity and whether the caller could perform that exact operation through the normal product interface. A shared credential with enforced per-user query scope is not a defect. + +**Action-confirmation and approval binding** +A user approves one described action but execution can use changed arguments, a different resource, a different principal, or a later model turn. An action-binding defect also exists when attacker-controlled content causes a side effect under a victim's valid authority without the victim's intentional request or approval, even if generic authorization permits the victim to perform it. Review whether intent or confirmation binds the normalized tool name, complete argument object, requester, target, amount, expiry, and batch membership. Check retries and resumed sessions: an approval must not authorize a mutated or duplicate side effect. + +**Tool-schema and dispatcher disagreement** +The schema accepts aliases, extra fields, duplicate keys, coercions, nested free-form objects, or out-of-range values that the dispatcher or handler interprets differently. Compare schema validation, canonicalization, generated bindings, and handler defaults. Validate again where values become resource selectors or security-relevant options. + +**Unbounded delegated action loops** +A bounded request can enqueue repeated spend, send, mutation, or external API work without a per-request budget, per-action authorization, cancellation, or idempotency control. Confirm impact on shared cost, quotas, other users, or durable state. Do not test by exhausting a service; use code-level accounting and a locally bounded loop. + +## MCP and sub-agent trust classes (subagent_type: `general`) + +**Sub-agent and MCP trust inheritance** +A delegated task receives the full session, credentials, memory, or capabilities rather than the least authority required. Check the principal and tenant carried into each call, capability narrowing, credential audience, and whether delegated results are treated as untrusted on return. + +**MCP server and tool identity confusion** +Calls or results are routed by attacker-influenceable server names, tool names, request IDs, resource URIs, or model-selected aliases rather than the authenticated connection and outstanding request. Check whether two servers can claim the same tool or resource identity, whether reconnect changes the binding, and whether a response from one server can satisfy another server's pending call. + +**MCP metadata and schema as policy** +Tool descriptions, resource metadata, prompts, completion hints, or schemas supplied by an MCP peer are trusted as policy or authorization. These fields can guide the model but cannot grant capability. Find the deterministic allowlist, server identity check, and handler authorization that remain authoritative when metadata conflicts. + +## Output and disclosure attack classes (subagent_type: `general`) + +**Insecure output rendering** +Model output reaches an executing HTML, Markdown, template, URL, or command sink without the sink's required encoding and policy. For browser rendering, verify auto-loaded resources and CSP or sanitization in `CLIENT-SIDE.md`; renderer behavior outside the repository makes the candidate `needs_validation`. + +**Sensitive context extraction** +The assembled context contains credentials, another user's data, private source, or policy values that themselves grant access, and user-influenced output exposes them. Read prompt assembly and data-fetch code. Disclosure of generic instructions or behavior that does not cross a data boundary is not a finding. + +## Universal moves (apply across the above) + +- Draw four maps first: each execution identity, each capability, every writable context or memory source, and each output destination. Then connect the principal at the start to the authority at the end. +- Start at side-effecting tools and work backward through dispatcher, schema, confirmation, model context, retrieval, and ingestion. Start at durable memory reads and trace every writer. +- Compare direct, queued, retry, resume, batch, and delegated paths for the same action. The strongest gate must apply after arguments are final and before every side effect. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the crossed boundary and observable result: attacker, affected principal or shared resource, execution identity, target, and unauthorized or unrequested action or disclosure. +2. For confused-deputy authority claims, prove the tool lacks requester-and-resource authorization and that the attacker cannot perform the same action normally. For action-binding claims, instead prove attacker-controlled content caused an action under the affected principal's authority that the principal did not intentionally request or approve. Valid generic authorization does not establish that intent. +3. For memory or retrieval claims, cite both the attacker-controlled write and the later cross-principal read or privileged decision. A shared record without a reachable consumer is not enough. +4. For action binding, establish the intentional request or normalized approved object, if any, and compare it with the object the handler uses. Confirm a locally observable unrequested action, mutation, duplicate, or authority change without extending the test into harmful execution. For schema disagreement, compare the normalized validated object with the handler's object. +5. For MCP identity claims, verify the authenticated connection, request correlation, tool namespace, and effective credential. Mark `needs_validation` if external server identity or deployment routing is required. +6. Return `confirmed` findings only with a complete source trace and meaningful result. Return `needs_validation` for a specific unresolved boundary fact and state the bounded local or owner-observed check needed to resolve it. diff --git a/skills/thirdshift-security-audit/ATTACK-CLASSES.md b/skills/thirdshift-security-audit/ATTACK-CLASSES.md new file mode 100644 index 00000000..18a2beb4 --- /dev/null +++ b/skills/thirdshift-security-audit/ATTACK-CLASSES.md @@ -0,0 +1,130 @@ +# Attack Classes + +#### Attack classes — choose and split based on Phase 1 + +Select attack classes relevant to the application type. Not every class applies to every codebase. The list below is a starting point; add application-specific classes from Phase 1 and split large codebases per subsystem. Frame work as finding, validating, fixing, and prioritizing vulnerabilities. Keep validation to source review and bounded local fixtures; do not develop payload chains, test availability on live services, or take action in shared environments. + +Use `confirmed` only when source evidence and bounded validation establish the full boundary and meaningful result. Use `needs_validation` when a specific deployment, provider, platform, identity, or runtime fact is unavailable; state the missing fact and the safe owner-observed or local check that resolves it. + +> **Native / binary / kernel targets** (C/C++/Rust-unsafe, kernel modules, parsers and decoders, FFI, concurrent runtimes, binary loaders, JITs, firmware): use the memory-safety, integer/ABI, concurrency, binary-loader, and privileged-interface classes in [MEMORY-SAFETY-AND-BINARY.md](MEMORY-SAFETY-AND-BINARY.md). +> +> **AI / LLM / agent targets** (chatbots, RAG, persistent memory, tool-calling agents, MCP servers/clients, prompt assembly, or model-controlled actions): use the context, memory-poisoning, action-binding, tool-schema, MCP-identity, and output classes in [AI-AND-LLM.md](AI-AND-LLM.md). +> +> **HTTP, web, and identity targets** (ordinary web apps, APIs, reverse proxies, CDNs, gateways, custom HTTP parsers, sessions, CSRF, JWT, OAuth/OIDC, SAML, MFA, passkeys, account recovery/linking, API keys, or mTLS): use [WEB-PROTOCOL-AND-AUTH.md](WEB-PROTOCOL-AND-AUTH.md). +> +> **Client-side and browser targets** (SPAs, browser extensions, embedded webviews, service workers, browser storage, cross-window messaging, CORS, WebSockets, or DOM rendering): use [CLIENT-SIDE.md](CLIENT-SIDE.md). +> +> **Supply-chain and release targets** (dependency resolution, generated inputs, CI, release/signing/promotion, updates, plugins, or extensions): use [SUPPLY-CHAIN-AND-RELEASE.md](SUPPLY-CHAIN-AND-RELEASE.md). +> +> **Cloud and deployment targets** (IAM, infrastructure as code, containers/Kubernetes, service mesh, serverless/edge, ingress, provider events, or runtime configuration): use [CLOUD-AND-DEPLOYMENT.md](CLOUD-AND-DEPLOYMENT.md). +> +> **Protocol, RPC, and messaging targets** (gRPC, GraphQL transports, Protobuf/Cap'n Proto/Thrift, custom protocols, queues, brokers, pub/sub, webhooks, or streaming RPC): use [PROTOCOLS-RPC-AND-MESSAGING.md](PROTOCOLS-RPC-AND-MESSAGING.md). +> +> **Resource-exhaustion and availability targets** (untrusted work can consume shared CPU, memory, disk, connections, workers, queues, quotas, or operator-owned spend): use [RESOURCE-EXHAUSTION-AND-AVAILABILITY.md](RESOURCE-EXHAUSTION-AND-AVAILABILITY.md). +> +> **Data-isolation and lifecycle targets** (multi-tenant stores, caches/search, object links, analytics, export/backup, migration, deletion, retention, or restore): use [DATA-ISOLATION-AND-LIFECYCLE.md](DATA-ISOLATION-AND-LIFECYCLE.md). +> +> **Desktop, mobile, and local-IPC targets** (native apps, deep links, webview bridges, exported components, privileged helpers, local daemons, Unix sockets/XPC/Binder/D-Bus): use [DESKTOP-MOBILE-AND-LOCAL-IPC.md](DESKTOP-MOBILE-AND-LOCAL-IPC.md). + +**Injection** (subagent_type: `general`) +Trace untrusted input from entry point to dangerous sink. What counts as a "dangerous sink" depends on the application: +- Web apps: SQL queries, HTML output, shell commands, template engines, file paths, HTTP redirects, deserialization +- Libraries: any function that processes caller-supplied data without validation — buffer operations, parsers, format strings +- CLI tools: shell command construction, file path handling, environment variable interpolation +- Services: query construction, message serialization, log injection, LDAP/XPATH queries +- Client-side (browser/JS): DOM XSS, prototype pollution, `postMessage`/origin trust, and other browser-side classes — covered by the [CLIENT-SIDE.md](CLIENT-SIDE.md) companion blocks when selected + +Do not stop at the obvious direct paths. Look for indirect injection: data stored safely, then retrieved and used in a dangerous context by different code. Look for injection through field names, keys, headers, and metadata — not just values. Look for injection into secondary systems (logs, caches, search indexes, analytics). + +**Access control** (subagent_type: `general`) +Verify that a caller cannot do something outside its authority. Go beyond checking whether permission checks exist — verify they check the *right* permission for the *right* resource via the *right* mechanism: +- Is there a path to the same state change that checks a different (weaker) permission? +- Can a field in the request body override what the permission system intended to restrict? +- Are there endpoints that gate on authentication but forget authorization? +- Does the same resource have multiple access paths with inconsistent checks? +- What about bulk/batch/export/import operations — do they enforce per-item permissions? + +For complex access models, split into separate agents for auth bypass vs authorization logic. + +**Resource and file handling** (subagent_type: `general`) +- Path traversal (reading/writing outside intended directories) — including through symlinks, encoded sequences, and null bytes +- SSRF (making the application fetch attacker-controlled URLs) — including through redirects, DNS rebinding, and URL parser differentials +- Unsafe deserialization, archive extraction (zip slip), temp file handling +- Memory safety (if applicable): buffer overflows, use-after-free, integer overflow +- Race conditions on file operations (TOCTOU between check and use) + +**Cryptography and secrets** (subagent_type: `general`) +- Weak randomness for security-critical values (tokens, keys, nonces) +- Hardcoded secrets, secrets in logs, error messages, URLs, or client-visible responses +- Broken key derivation, missing HMAC verification, nonce reuse +- Timing side-channels on secret comparison +- Misuse of crypto primitives (ECB mode, unauthenticated encryption, static IVs, etc.) +- What happens when crypto operations fail? Does the error path fall back to no-crypto? + +**Business logic** (subagent_type: `general`) +Hunt logic errors by hand: standard scanners cannot find them, and they yield high-impact findings. For each major workflow: +- **State machine violations**: Can you skip steps? Go backwards? Reach an invalid state? What happens if you replay a completed flow? What about partial failure — if step 2 of 3 fails, is step 1 rolled back? +- **Race conditions with business impact**: Concurrent operations that produce invalid states (double-spend, double-approve, lost updates). Focus on operations that check-then-act non-atomically. +- **Numeric/quantity manipulation**: Negative values, zero values, overflow, precision loss, type coercion between string and number. +- **Access boundary violations**: Not "does the permission check exist" but "is it the right check for the business rule?" Can input to one operation bypass a restriction enforced on a different operation for the same effect? +- **Implicit trust assumptions**: Data from storage, config, other components, or plugins assumed safe because "we validated it on the way in." What if a different code path wrote it? +- **Time-based logic**: Expiry checks, scheduling, rate windows, clock skew. What happens at exact boundary moments? What about timezone differences between components? +- **Default and fallback behavior**: What is the security posture when config is missing? When a feature flag is off? When a dependency is unavailable? When the system is mid-migration? + +**Feature abuse and data leakage** (subagent_type: `general`) +Legitimate features used for unintended purposes. Look for bugs in the design, not only in the code: +- **Export/backup as exfiltration**: Can a low-privilege user trigger an export, snapshot, or backup that includes data above their access level? Can they export other users' data? Does the export include deleted/draft/private content? Revision history that was supposed to be pruned? +- **Import/restore as injection**: Can import overwrite existing data? Can it create records that bypass normal validation? Can it inject content into collections the user has no write access to? Does it respect the same permission model as the UI? +- **Search/filter/sort as oracle**: Can search queries reveal whether content exists that the user cannot directly access? Do filter parameters let users probe statuses, roles, or fields they should not know about? Does sorting by a hidden field reveal its values through result ordering? +- **Enumeration through side effects**: Do error messages differ between "does not exist" and "no access"? Do response times differ? Response sizes? HTTP status codes? Can you enumerate users through password reset, invite, or registration flows? +- **Preview/draft/staging leakage**: Are preview tokens scoped to one item or do they unlock broader access? Can draft content be discovered through search, RSS feeds, sitemaps, or API listing endpoints? Can cache headers cause a CDN to serve private content publicly? +- **Notification/webhook as SSRF**: Can a user set a notification URL, webhook URL, or callback URL that the server fetches? Is it validated against internal networks? What about after a redirect? + +**Chained vulnerabilities and trust boundaries** (subagent_type: `general`) +Individually allowed or contained behavior can become a vulnerability when another component or lifecycle step relies on a stronger guarantee: +- **Multi-step boundary failures**: Map what a low-privilege principal may read, write, invoke, and retain, then connect only concrete outputs to later trust decisions. Confirm each prerequisite and do not assume a downstream effect. +- **Cross-component trust gaps**: Component A validates input and passes it to component B. Compare the exact guarantee A produces with what B assumes, including truncation, type coercion, normalization, tenant scope, and plugin/extension access. +- **Second-order use**: Data safe when stored may become dangerous in a later context. A field name becomes a JSON path, a slug becomes a file path, escaped text enters raw rendering, or a stored string becomes a URL, regex, template, or policy expression. +- **Scope and capability growth**: Token, API-key, plugin, OAuth, MCP, or AI capabilities become broader after delegation, refresh, caching, role change, or composition. Name the concrete operation the resulting principal should not have. +- **Timing and ordering**: Review setup, migration, soft-delete, revoke/cache expiry, check/use, and validate/consume windows. Confirm stale state is accepted before reporting. +- **Rollback and recovery**: Undelete, restore, revision rollback, and cancellation must apply current ownership, validation, and authorization. Confirm which invalid state is restored. + +**Wildcard** (subagent_type: `general`) +You are not given a category. Find vulnerabilities outside the standard classes already assigned. + +Read code that looks boring or disconnected from security. Follow incomplete, experimental, compatibility, and fallback features, but retain the same concrete boundary and validation requirements as every other class. + +Use these starting points, but do not limit yourself to them: +- What is the strangest code in the codebase? Why does it exist? What happens if it is abused? +- Are there any features that feel half-finished, experimental, or bolted on? Those have the weakest security because they got the least review. +- What happens if you use the API in a way the frontend never would? The UI constrains users, but the API does not. What API calls are possible but never made by the client? +- Are there any hidden or undocumented endpoints, parameters, headers, or features? Look at route registrations, middleware, and config for things that are not in the docs. +- What happens when you mix features that were not designed to work together? Localization + preview + caching. Import + plugins + webhooks. OAuth + impersonation + API keys. +- Is there anything interesting in the git history? Reverted security fixes, commented-out auth checks, secrets that were committed then removed (still in history). +- Which valid-account actions affect other users, shared integrity, availability, or operator-owned cost? Verify containment, quotas, authorization, and recovery around those actions. +- Which operations are irreversible or require elevated confirmation? Bind authorization and approval to the final principal, action, and resource. +- What assumptions does the code make about the environment? That the database is local, that the clock is accurate, that DNS is trustworthy, that the filesystem is case-sensitive? +- Look at the test files — what are they **not** testing? Compare the edge cases the developer thought about (tests exist) with the ones they did not (no tests). + +Pursue anomalies inside your assigned scope until the invariant is settled. If something looks strange, read it until you can state whether it is safe. If a function has a comment explaining why it is safe, verify the explanation. If a variable is named `temp` or `hack` or `legacy`, read it closely. + +**Obvious things** (subagent_type: `general`) +Other agents hunt subtle bugs. This agent checks the basic exposures that are easy to overlook because everyone assumes someone else already checked them: +- Are there any hardcoded passwords, API keys, tokens, or secrets in the source? (grep for `password`, `secret`, `apikey`, `token`, `Bearer`, `-----BEGIN`, common default passwords) +- Are there any TODO/FIXME/HACK/XXX comments that reference security? (`TODO: add auth`, `FIXME: validate input`, `HACK: skip permission check`) +- Is debug mode / dev mode properly gated? Can it be enabled in production via environment variable, query parameter, or header? +- Are there test/example/seed credentials that work in production? +- Is there a `/debug`, `/admin`, `/test`, `/status`, `/health`, `/metrics`, `/env`, `/.env`, `/config` endpoint that is unprotected? +- Are there any `.env`, `.env.local`, `credentials.json`, `*.pem`, `*.key` files checked into the repo? +- Does the `.gitignore` actually cover secrets, uploads, and local config? +- Are dependencies pinned? Are there known CVEs in the dependency tree? (check lockfiles) +- Are there any `eval()`, `exec()`, `child_process`, `Function()`, `vm.runInContext`, `import()` with dynamic input? +- Are CORS headers set to `*` or overly permissive? Is `Access-Control-Allow-Credentials` combined with a wildcard origin? +- Are cookies missing `HttpOnly`, `Secure`, or `SameSite` attributes? +- Are there any open redirects? (parameters named `redirect`, `return`, `next`, `url`, `goto`, `continue` that feed into redirects without validation) +- Is TLS enforced? Are there any HTTP-only endpoints? +- Are error responses in production returning stack traces, internal paths, or SQL errors? + +This agent does not need to be creative. It needs to be thorough and literal. Check every item. Report each result. + +**Important**: For any finding this agent reports, it must verify the full code path, not just surface appearance. If a cookie is missing `HttpOnly`, check whether the cookie contains security-sensitive data and whether JS needs to read it by design. If an error message contains a field name, check whether the field is ever actually populated with sensitive data. A flag is not a finding — trace the impact before reporting. diff --git a/skills/thirdshift-security-audit/CLIENT-SIDE.md b/skills/thirdshift-security-audit/CLIENT-SIDE.md new file mode 100644 index 00000000..8ba673e9 --- /dev/null +++ b/skills/thirdshift-security-audit/CLIENT-SIDE.md @@ -0,0 +1,83 @@ +# Client-Side and Browser Hunting + +#### When to use this file + +Reach for this file when meaningful trust decisions or untrusted rendering happen in a browser: single-page apps, browser extensions, embedded webviews, service workers, offline applications, and code that renders attacker-influenceable content into the DOM, receives cross-window messages, or uses browser storage. These paths include sources the server never sees, such as URL fragments, `window.name`, `postMessage`, and previously cached content. + +Use alongside `ATTACK-CLASSES.md`. This file covers browser sources and sinks, origin boundaries, browser persistence, and cross-site state oracles. Use `DESKTOP-MOBILE-AND-LOCAL-IPC.md` for the native side of a webview bridge, and `WEB-PROTOCOL-AND-AUTH.md` for server-side CSRF, sessions, and auth callbacks. + +## Core discipline (include in every agent prompt for this domain) + +``` +- A client-side candidate needs a controllable source and an executing or disclosing sink. Name both and show attacker-influenced data reaching the sink. +- The impact must reach a victim's session, another origin, or shared persistence. Self-injection and disclosure of the attacker's own data are not findings. +- Framework escaping, browser same-origin policy, CSP, COOP/CORP, service-worker scope, and modern noopener defaults are real controls. Verify them before assigning impact. +- Browser storage and caches are shared by origin and may outlive login state. Identify who writes, who reads, and which account, tenant, or worker lifecycle clears each record. +- Use `confirmed` only for complete source evidence plus bounded local browser tests. Use `needs_validation` when renderer, extension permission, deployed header, or browser-policy behavior is required but unavailable. +``` + +## DOM and object-state attack classes (subagent_type: `general`) + +**DOM-based XSS** +Trace `location` fields, `document.referrer`, `window.name`, message data, storage, and browser-controlled document state into `innerHTML`, `outerHTML`, `document.write`, string-evaluating APIs, executable URLs, jQuery HTML APIs, or framework escape hatches. Interpolation escaped by the framework is not a finding. + +**DOM clobbering** +Attacker-injected `id` or `name` attributes shadow a global, form property, configuration object, or initialization flag later trusted by code. Require both a markup path that preserves the attribute and a security-relevant use of the clobbered value. + +**Prototype pollution and gadget chain** +An attacker-controlled key reaches a recursive write such as deep merge or path assignment and modifies prototype state. Then a reachable gadget consumes the polluted property to change authorization, execution, navigation, or rendering. `JSON.parse`, a shallow copy, or pollution without a gadget is not enough. + +## Cross-origin messaging and network attack classes (subagent_type: `general`) + +**`postMessage` origin and source trust** +A handler performs a sensitive action with `event.data` without an exact origin allowlist and, where multiple frames share an origin, the expected `event.source`. On the send side, sensitive data sent to `*` reaches an unintended embedder. Weak substring, prefix, suffix, or unanchored-regex origin matching is not an origin check. + +**Cross-site WebSocket request use** +A WebSocket upgrade accepts ambient cookies from an untrusted origin without an `Origin` check or channel-specific token, allowing the victim's session to read or mutate data. Confirm both the upgrade behavior and a security-relevant message handler. + +**Credentialed CORS trust** +The server reflects or weakly matches `Origin` while allowing credentials and returns sensitive responses. A bare wildcard with credentials is rejected by browsers; report only the actual reflected/allowed origin path and cross-origin data or mutation. + +## Service-worker and browser-storage attack classes (subagent_type: `general`) + +**Service-worker registration and scope takeover** +Attacker-influenceable content can become the registered worker script, control a path that receives an over-broad `Service-Worker-Allowed` scope, or alter update imports without integrity control. Verify the final script URL, response MIME type, origin, scope, and who controls every imported script. A normal same-origin worker with intended scope is not a defect. + +**Service-worker cache and identity confusion** +The worker caches personalized responses without including account, tenant, authorization state, or request mode in its policy, then serves them after account switch or logout. Review fetch-event routing, cache names and keys, navigation fallbacks, cache cleanup, and whether error/offline paths return another user's prior response. + +**Browser-storage disclosure and stale authorization** +Tokens, private responses, draft data, or authorization decisions remain in `localStorage`, `sessionStorage`, IndexedDB, Cache Storage, extension storage, or client state and become readable by another account or less-trusted same-origin component. Storage of a token alone is not a finding; require a realistic reader with less authority, or continued use after revocation/logout. + +**Cross-context storage and broadcast confusion** +`storage` events, `BroadcastChannel`, shared workers, or origin-wide caches carry identity or commands between tabs without binding them to the current session. Check account switching, private/public windows, tenant changes, and stale tabs that can overwrite newer auth state. + +## Cross-site information leak classes (subagent_type: `general`) + +**XS-Leaks and cross-origin state oracles** +An attacker page can distinguish protected cross-origin state through resource load/error events, frame or window state, redirect behavior, timing, cache state, or response size while the browser attaches victim credentials. Require one concrete secret-bearing predicate such as whether a private object, role, or account exists. Generic timing variance or public-resource availability is not a finding. + +**Window and opener state disclosure** +A cross-origin window's permitted metadata or navigation result reveals protected state, or a retained opener/named-window relationship lets an attacker-controlled page influence a privileged navigation. Check COOP, frame protections, `noopener`, exact origin, and whether the observable state is confidential. + +## UI-redress and navigation attack classes (subagent_type: `general`) + +**Clickjacking** +A framed, state-changing action lacks effective `frame-ancestors`, `X-Frame-Options`, or equivalent UI isolation. Require the sensitive action and confirm it can complete in the framed state; missing headers on read-only content are hardening notes. + +**Client-side navigation confusion** +A client source controls redirect or navigation without scheme and destination policy, including executable `javascript:` or `data:` destinations. Reverse tabnabbing applies only where code explicitly keeps `window.opener`, uses `window.open` without isolation, or supports a browser without implicit `noopener`. + +## Universal moves (apply across the above) + +- Start from DOM, navigation, worker, message, and storage sinks, then trace backward to browser-only and server-controlled sources. Record the browser policy that should stop the path. +- Test account switch, logout, worker update, offline fallback, and stale-tab state with a local test origin and dummy accounts. Do not use production users, origins, or shared services. +- For XS-Leaks, list only predicates proved by source and local browser behavior. Then identify the response headers or rendering choice that would remove the oracle. + +## Validation rules (apply before reporting ANY finding here) + +1. Cite the source, sink, browser policy, affected origin/session, and observable mutation or disclosure. +2. For prototype pollution, prove the recursive write and a security-relevant gadget. For DOM clobbering, prove the markup survives and the shadowed value is used. +3. For service workers and storage, prove lifecycle reachability: an attacker-controlled write or cache entry must reach a different account, tenant, or later authorization state. +4. For messaging, CORS, WebSocket, and XS-Leaks, show exact origin/source validation and the protected state or action exposed. Confirm that CSP, COOP/CORP, cookies, and SameSite policy do not already block it. +5. Return `confirmed` findings only with a complete client path and bounded local evidence. Return `needs_validation` with the precise deployed header, extension permission, browser version, or renderer behavior an owner must verify. diff --git a/skills/thirdshift-security-audit/CLOUD-AND-DEPLOYMENT.md b/skills/thirdshift-security-audit/CLOUD-AND-DEPLOYMENT.md new file mode 100644 index 00000000..5569f76c --- /dev/null +++ b/skills/thirdshift-security-audit/CLOUD-AND-DEPLOYMENT.md @@ -0,0 +1,86 @@ +# Cloud and Deployment Hunting + +#### When to use this file + +Reach for this file when the repository defines cloud identity, infrastructure, containers, Kubernetes, service mesh, serverless functions, edge workers, ingress, object storage, managed services, or environment-specific configuration. This domain asks whether deployed components receive the intended identity, isolation, network reachability, secrets, and policy. Source often expresses intent rather than live fact, so separate source-confirmed defects from deployment validation needs. + +Use `SUPPLY-CHAIN-AND-RELEASE.md` for build and promotion trust, `WEB-PROTOCOL-AND-AUTH.md` for HTTP proxy semantics, and `DATA-ISOLATION-AND-LIFECYCLE.md` for data-store tenant scope. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Do not infer a live exposure from a manifest alone. Establish which environment consumes it, what defaults or overlays modify it, and whether the source path is active. +- Map each workload's identity to specific operations and resources. Broad policy is a finding only when lower-trust input can reach an unauthorized action. +- Ingress, proxies, service mesh, metadata services, and admission policy are real boundaries, but only count a control when its configuration and attachment are visible. +- Secret references are not secret disclosure. Require a lower-trust reader, output, artifact, log path, or unsafe fallback. +- Use `confirmed` for active in-repo configurations and local rendering/policy validation. Use `needs_validation` for account policy, network attachment, runtime admission, hosted metadata, or drift that needs owner observation. +``` + +## Workload identity and IAM attack classes (subagent_type: `general`) + +**Workload identity overreach** +A workload, pod, function, edge worker, or node identity can act on tenants, accounts, resources, or APIs beyond its role, and untrusted request or job input selects that target. Review cloud policy conditions, resource patterns, service-account attachment, namespace mapping, and fallback credentials. + +**Cross-account or cross-tenant role confusion** +Role assumption, external IDs, token exchange, workload federation, or resource policies accept identity claims not bound to the intended source account, audience, repository, namespace, or workload. Establish both trust policy and caller-controlled claim. + +**Application authorization delegated to cloud metadata** +An app trusts caller-supplied identity headers, tags, labels, account IDs, or resource metadata without verifying they came from the cloud control plane or a trusted proxy. Cloud IAM and application authorization are separate checks. + +## Ingress, network, and control-plane attack classes (subagent_type: `general`) + +**Unexpected service or management-plane reachability** +An ingress, service, listener, security group, load-balancer annotation, port mapping, or server bind exposes an admin, debug, metrics, node, control-plane, or internal API to a lower-trust network. Missing network controls alone are `needs_validation`; a repository-controlled public route to a sensitive handler can be `confirmed`. + +**Trusted-proxy and mesh identity bypass** +A backend accepts forwarded identity, mTLS subject, or authorization metadata from peers outside the intended ingress/sidecar, or an alternate port and health/legacy path bypasses the mesh. Verify header stripping, peer reachability, and fail-open behavior when the proxy is absent. + +**Metadata and internal-service reachability** +An untrusted URL, destination, or protocol selection reaches instance/container metadata, control-plane sockets, or internal APIs with workload credentials. Trace URL parsing and redirect handling under `ATTACK-CLASSES.md`; here establish deployed network, metadata-version, and identity boundaries. + +## Container and orchestration attack classes (subagent_type: `general`) + +**Host or control-plane capability exposure** +A lower-trust workload can select privileged mode, capabilities, host namespaces, host paths, device mounts, container runtime sockets, or service-account tokens that cross into node/control-plane authority. Bare absence of seccomp or read-only filesystem is hardening unless a reachable operation crosses that boundary. + +**Admission and policy path inconsistency** +One deployment route enforces image identity, namespace, resource, secret, or privilege policy while another controller, job, upgrade, restore, or compatibility path does not. Confirm the alternate route and resulting deployed object. + +**Namespace and label trust confusion** +Network, admission, secret, or workload-identity policy relies on labels, annotations, names, or namespaces that a less-trusted principal can set. Compare who controls selectors with what authority matching grants. + +## Configuration and secret lifecycle attack classes (subagent_type: `general`) + +**Security-control precedence drift** +Development values, chart defaults, environment variables, command-line flags, feature gates, sidecar injection, or per-region overlays disable authentication, transport security, tenant scoping, or audit policy in a deployed environment. Render the final configuration for each maintained deployment, not just the base file. + +**Secret exposure across workload boundaries** +Secrets enter logs, crash reports, process arguments, shared environment, broad volumes, build outputs, service discovery, or read APIs accessible to another workload or tenant. Check secret type and authority; a public endpoint or key ID is not a credential. + +**Credential renewal and outage fallback** +Failure to mount, refresh, rotate, or revoke a workload credential causes stale credentials to remain active or an app to accept a less trusted identity mode. Review startup, readiness, reconnect, and cached-client behavior. + +## Managed storage, events, and edge attack classes (subagent_type: `general`) + +**Object and signed-URL policy confusion** +Bucket/container policy, object keys, CDN origins, or signed URLs fail to bind principal, operation, object namespace, audience, or expiry. Review list/version operations and write paths as well as reads. + +**Event-source identity confusion** +A function or worker trusts event body fields as source identity without validating provider-signed envelope, subscription/topic, account, region, and replay state. Compare push, pull, retry, and dead-letter paths. + +**Edge/runtime boundary mismatch** +An edge or serverless runtime assumes a secret, API, filesystem, isolation, or tenant policy that differs from the origin runtime, and fallback to origin changes authority or cache behavior. Confirm which configuration selects each path. + +## Universal moves (apply across the above) + +- Render every maintained environment and make a matrix of external port, workload identity, network peers, mounted secrets, and cloud resources. Differences require an owner or policy explanation. +- Follow a lower-trust request, object, label, or event into cloud policy. Show which workload credential performs the final operation and what condition should scope it. +- Diff normal deploy, migration, restore, node maintenance, failover, and local/emulator paths. Review behavior when mesh, admission, identity, secret, or policy service is unavailable. + +## Validation rules (apply before reporting ANY finding here) + +1. Establish the active source path and effective deployment object; otherwise use `needs_validation` and state which rendered manifest or owner-observed attachment is missing. +2. Name the lower-trust caller/workload, cloud or application identity, controllable selector, affected resource, and unauthorized operation or disclosure. +3. Verify provider and orchestrator defaults at the pinned version. Do not assume a public IP, reachable metadata service, permissive firewall, or absent admission attachment. +4. Local validation may render templates, evaluate policy, inspect container/user namespaces in an isolated fixture, or run an emulator with dummy identities. Do not probe live endpoints or alter shared cloud resources. +5. Return `confirmed` only with a complete active source trace and concrete boundary result. Return `needs_validation` with the exact deployed policy, identity attachment, overlay, network, or drift observation needed. diff --git a/skills/thirdshift-security-audit/CREDITS.md b/skills/thirdshift-security-audit/CREDITS.md new file mode 100644 index 00000000..f21b4652 --- /dev/null +++ b/skills/thirdshift-security-audit/CREDITS.md @@ -0,0 +1,7 @@ +# Upstream and licence + +Copied from Cloudflare's [security-audit-skill](https://github.com/cloudflare/security-audit-skill), at commit [`c1c8a8c1471069fb0e188eeaff69b8e8db6564a8`](https://github.com/cloudflare/security-audit-skill/tree/c1c8a8c1471069fb0e188eeaff69b8e8db6564a8). + +The skill and its supporting files are unchanged except for the `SKILL.md` frontmatter name, changed to `thirdshift-security-audit` (ADR 0012). + +Used under the MIT License. Cloudflare's copyright notice and licence are kept in [LICENSE](LICENSE) in this directory. diff --git a/skills/thirdshift-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md b/skills/thirdshift-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md new file mode 100644 index 00000000..0cba1fa4 --- /dev/null +++ b/skills/thirdshift-security-audit/DATA-ISOLATION-AND-LIFECYCLE.md @@ -0,0 +1,84 @@ +# Data Isolation and Lifecycle Hunting + +#### When to use this file + +Reach for this file when the target stores multi-tenant or access-controlled data, derives search/index/cache/analytics copies, issues object links, exports or restores records, migrates schemas, or promises deletion, revocation, and retention behavior. This domain follows one data item through every copy and state transition. Use `ATTACK-CLASSES.md` for endpoint-level access control and `CLOUD-AND-DEPLOYMENT.md` for provider-level storage policy. + +Split large targets by primary storage, cache/search, object/blob storage, analytics/logging, export/backup, deletion/revocation, and migration. + +## Core discipline (include in every agent prompt for this domain) + +``` +- A tenant or owner field on a record is not isolation. Find the query, key, path, policy, or row-level control that enforces it for each read and write path. +- Trace derived copies. Sanitized primary data can become unsafe in search, cache, analytics, export, previews, logs, replicas, and backups with different ACL and retention rules. +- Deletion and revocation are lifecycle contracts. Check current, historical, cached, indexed, exported, restored, and queued copies within the product's stated boundary. +- Privacy or retention preference is not automatically a security vulnerability. Require an explicit data-access boundary or deletion/revocation guarantee and an unauthorized reader or later operation. +- Use `confirmed` for complete source-visible lineage and bounded dummy-tenant tests. Use `needs_validation` when external storage policy, retention, CDN behavior, replica lag, or backup access is unavailable. +``` + +## Tenant and object-isolation attack classes (subagent_type: `general`) + +**Missing tenant or owner enforcement** +A read, update, delete, list, count, or bulk query identifies an object without binding it to the authenticated tenant/owner, or trusts body fields to supply that identity. Compare direct lookup, nested relationship, background, admin, import, and legacy paths. + +**Composite-key and namespace collision** +Cache keys, object paths, database uniqueness, search document IDs, temporary files, or deduplication keys omit tenant or environment. Two principals can overwrite or retrieve the same logical key even though application records carry separate owners. + +**Policy and query disagreement** +Row-level policy, ORM default scopes, authorization filters, and raw/bypass clients apply different predicates. Check joins, aggregates, aliases, views, transactions, `unscoped` or service clients, and error paths where context is missing. + +**Blob and signed-reference overreach** +Object keys, attachment IDs, version IDs, shared links, or signed URLs permit operations or namespaces beyond the issuing principal's access, or remain valid after the underlying ACL changes. Bind operation, exact object/version, audience, expiry, and tenant. + +## Derived-data and disclosure attack classes (subagent_type: `general`) + +**Search, cache, and index ACL drift** +A primary record's ACL or lifecycle changes without invalidating a searchable, cached, embedded, thumbnail, RSS, preview, or index copy. Validate filtering at retrieval time as well as document ingestion and invalidation. + +**Analytics, logs, traces, and diagnostics as alternate readers** +Private content or credentials are emitted into systems with broader access, longer retention, or tenant mixing. Confirm the data class and realistic reader; field names, public identifiers, and operator-only content under intended policy are not enough. + +**Enumeration and aggregate oracles** +Counts, filters, ordering, errors, unique constraints, timings, notification behavior, or existence checks disclose protected object or account state. Require a concrete confidential predicate and observable distinction, not general response variance. + +## Export, backup, restore, and migration attack classes (subagent_type: `general`) + +**Export and backup scope expansion** +An export, snapshot, portability package, report, or backup includes other tenants, inaccessible object fields, soft-deleted data, secret values, or history above the requester's access. Check per-item authorization after selection and authorization to download the final artifact. + +**Import and restore authority expansion** +Restore/import bypasses owner, schema, ACL, uniqueness, or validation rules, overwrites existing resources, or recreates records in a tenant the requester cannot write. Validate archive contents as untrusted and authorize the resulting operation rather than trusting prior provenance. + +**Migration default and ownership confusion** +Old records lack tenant/ACL/lifecycle fields, incompatible IDs collide, or partial rollout makes new and old readers apply different defaults. Review backfill, dual-read/write, compatibility, rollback, and resumed-migration paths. + +**Backup and replication boundary drift** +Encryption keys, storage accounts, cross-region replicas, restoration environments, or support snapshots have broader identity or tenant scope than primary data. Source can confirm only in-repo policy; hosted access and retention require `needs_validation`. + +## Deletion, revocation, and lifecycle attack classes (subagent_type: `general`) + +**Soft-delete and tombstone bypass** +Direct lookup, search, relation traversal, object link, background processor, or restore ignores the lifecycle predicate and returns or acts on a deleted/revoked record. Check whether soft-deleted identifiers can be re-registered before all references are gone. + +**Stale authorization and derived copy use** +Membership removal, ACL update, consent withdrawal, secret revocation, or role downgrade does not invalidate sessions, caches, subscriptions, jobs, or materialized data that continue to authorize future operations. + +**Retention and queued-work overrun** +Deletion completes in primary storage while queued processors, retries, exports, analytics, or generated artifacts recreate or retain the data beyond the promised boundary. Find idempotent deletion and tombstone propagation. + +**Restore reintroduces invalid state** +Backup, undo, undelete, or replica recovery restores data, credentials, memberships, or permissions that current policy no longer allows. Re-authorize restored state and reapply lifecycle changes made after the snapshot. + +## Universal moves (apply across the above) + +- Pick one protected record and draw primary write, query, cache, index, event, export, backup, deletion, and restore paths. Mark principal and tenant at every edge. +- Compare two dummy tenants through the same local service methods, then repeat after ACL change, deletion, account switch, and restore. Do not use real user data. +- Start at bypass clients, background jobs, migrations, global uniqueness, and cache keys. These paths commonly omit request-scoped identity that interactive endpoints carry. + +## Validation rules (apply before reporting ANY finding here) + +1. Name attacker or lower-trust principal, protected data/state, affected owner/tenant, alternate copy or operation, and unauthorized disclosure or mutation. +2. Cite both intended source-of-truth policy and the path that omits or disagrees with it. Confirm another layer does not enforce the same tenant/lifecycle condition. +3. Use local dummy tenants and non-sensitive fixtures to prove cross-scope access or stale lifecycle behavior. Stop at the minimum observable record or operation. +4. If external cache, object storage, replicas, analytics, backup, or retention policy is required, classify `needs_validation` and state the owner-observed check. +5. Return `confirmed` only with complete lineage and concrete boundary impact. Return `needs_validation` with the exact unresolved storage, ACL, invalidation, retention, or restore fact. diff --git a/skills/thirdshift-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md b/skills/thirdshift-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md new file mode 100644 index 00000000..d6230098 --- /dev/null +++ b/skills/thirdshift-security-audit/DESKTOP-MOBILE-AND-LOCAL-IPC.md @@ -0,0 +1,89 @@ +# Desktop, Mobile, and Local IPC Hunting + +#### When to use this file + +Reach for this file when the target is a desktop or mobile app, privileged helper, updater, local daemon, webview host, deep-link handler, browser native-messaging host, or local IPC client/server. Relevant untrusted actors may be a downloaded document, remote web content, another local app, another OS user, a sandboxed process, or a lower-privilege account. State that starting capability instead of treating all local users as equivalent. + +Use `CLIENT-SIDE.md` for browser-side webview behavior, `MEMORY-SAFETY-AND-BINARY.md` for native memory and loader safety, and `SUPPLY-CHAIN-AND-RELEASE.md` for update authenticity. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Establish the realistic local or remote-content attacker: another app, another OS user, a sandboxed child, an untrusted document, or a remote origin. Self-harm within the same account and authority is not a boundary violation. +- Paths, process names, bundle/package IDs, and claimed sender fields are not peer authentication. Use OS peer credentials, code identity, capability handles, or protected channel state. +- The native bridge or helper must authorize each operation and final resource after parsing. A trusted UI or broker does not make attacker-influenceable arguments trusted. +- OS sandbox, signing, entitlements, permissions, keychain ACLs, exported-component policy, and prompt behavior are real controls when pinned and visible. +- Use `confirmed` for source evidence plus bounded local/emulator tests. Use `needs_validation` when signing, manifest merge, OS version, device policy, installer ACL, or packaging is required but not observable. +``` + +## Deep-link, callback, and navigation attack classes (subagent_type: `general`) + +**Custom-scheme and deep-link ambiguity** +Another app or page can invoke a route that mutates state, imports data, completes authentication, or selects an account without a current-session and one-time callback binding. Review URI normalization, duplicate query fields, scheme/host/path matching, exported activity/handler policy, and stale/replayed links. + +**App and account handoff confusion** +OAuth, SSO, magic-link, invite, device pairing, passwordless, or payment callbacks return to the wrong installed app, profile, tenant, or pending transaction. Bind state to the initiating app identity, current session, account, provider, operation, and expiry. + +**File-open and intent authority confusion** +An associated file, share intent, drag/drop item, pasteboard/clipboard record, notification action, or open-file event triggers a privileged operation without confirming content type, sender trust where applicable, current user intent, and final target. + +## Webview and native-bridge attack classes (subagent_type: `general`) + +**Navigation-origin to bridge confusion** +Remote or attacker-controlled frames can reach a JavaScript/native bridge intended only for packaged content. Validate origin at call time and after every navigation, redirect, subframe creation, popup, and error/fallback page. URL-prefix checks and initial-load checks are insufficient. + +**Over-broad native bridge capabilities** +Web content can select arbitrary files, commands, IPC methods, credentials, or system actions through a generic bridge. Check method allowlists, normalized arguments, user/tenant authority, gesture/confirmation requirements, and return-value disclosure. + +**Webview file and universal access** +Remote content can read app-local files, privileged custom schemes, or internal origins because file access, universal access, mixed content, debug interfaces, or custom protocol handlers join origins unexpectedly. Missing a restrictive setting without reachable protected content is hardening. + +## Local IPC and exported-component attack classes (subagent_type: `general`) + +**IPC peer-authentication gaps** +Unix sockets, named pipes, XPC, Binder, D-Bus, native messaging, RPC, shared memory, or loopback listeners accept a lower-trust peer without checking OS credentials, code identity, sandbox token, or channel ownership. Require a meaningful method or disclosure behind the channel. + +**Claimed principal versus channel identity** +The authenticated process/channel belongs to one app or user, but request fields select another user, tenant, profile, or capability. Bind each method and resource to the peer credential rather than a caller-declared identifier. + +**Exported service, activity, receiver, or provider overreach** +A mobile component or local automation endpoint is externally invokable and performs an operation intended for the app itself. Review final merged manifests, intent filters, permission/signature level, path grants, and alternate aliases. Manifest status unknown after packaging requires `needs_validation`. + +**IPC lifecycle and correlation confusion** +Predictable request IDs, reused handles, stale channels, inherited descriptors, world-writable socket paths, or restart behavior lets one peer answer, cancel, or reuse another peer's operation. Review creation permissions and cleanup of socket files, locks, ports, and shared mappings. + +## Privileged-helper and local-file attack classes (subagent_type: `general`) + +**Privileged helper as confused deputy** +A low-privilege caller can select a privileged command, file, service, user, or system setting without per-operation authorization. Review sudo/polkit/UAC/XPC helper rules and ensure the helper independently validates normalized arguments. + +**Install, update, and repair path trust** +A privileged installer/helper reads manifests, scripts, packages, symlinks, working directories, or repair state writable by a lower-trust actor after authorization. Bind authorization to immutable content and safe destination paths. + +**Local file ownership and TOCTOU** +The app checks a file/path then follows replacement, symlink, mount, or case/normalization changes during a privileged read/write. Use descriptor-relative operations and verify final ownership. Focus `MEMORY-SAFETY-AND-BINARY.md` on parsing after the file is opened. + +**Credential-store and local-secret boundary mismatch** +A keychain/keystore item, token file, backup, log, clipboard, notification preview, or local config is readable by another app/profile/user with less authority. Plaintext readable only by the same intended OS account is not automatically a vulnerability; state the lower-trust reader and credential power. + +## Application-state and device-lifecycle attack classes (subagent_type: `general`) + +**Account switch, logout, and device restore leakage** +Cached data, background tasks, widgets, notifications, local databases, webview storage, or biometric approvals survive logout/account change and appear under a later account. Review backup/restore and multi-profile behavior. + +**Pending-action and user-presence confusion** +Notification, widget, shortcut, share sheet, biometric prompt, or deferred operation authorizes a different action than displayed, executes after expiry, or uses another profile's pending state. Bind confirmation to normalized action, resource, account, and current foreground state. + +## Universal moves (apply across the above) + +- Enumerate every process, app component, local endpoint, URI scheme, file association, webview origin, and helper. Record OS identity, runtime privilege, caller, and callable operation. +- Read final packaging inputs: merged manifest, entitlements, installer rules, native-messaging registration, protocol handlers, and ACL creation. Source declarations can be overwritten downstream. +- Validate with dummy profiles and non-sensitive local fixtures on an isolated machine/emulator. Do not interact with other users' apps, credentials, or production services. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the attacker starting capability, OS/app principal crossed, entry channel, accepted argument or state, and unauthorized operation or disclosure. +2. Confirm OS sandbox, peer credential, signing, entitlement, permission, user-consent, and installer controls that apply. Unknown packaging/runtime facts require `needs_validation`. +3. For webview bridges, cite both navigation/origin control and privileged native sink. For IPC, cite peer authentication and per-resource authorization. For helpers, verify final normalized destination. +4. Keep local tests bounded and use dummy content/accounts. Stop after proving the boundary result; do not extend proof into persistence or broader system modification. +5. Return `confirmed` only with a complete source and local evidence chain. Return `needs_validation` with the exact OS, manifest, signing, ACL, or device-lifecycle fact required. diff --git a/skills/thirdshift-security-audit/HUNTING.md b/skills/thirdshift-security-audit/HUNTING.md new file mode 100644 index 00000000..377d4dcf --- /dev/null +++ b/skills/thirdshift-security-audit/HUNTING.md @@ -0,0 +1,251 @@ +# Vulnerability Hunting + +### Phase 2: Run coverage-led hunting waves + +The parent assigns `planned` ledger units to `general` agents. Use enough focused hunters to cover the units without combining unrelated boundaries. One hunter may own closely related units in one subsystem; no unit may be silently unassigned because of an agent-count limit — a unit the budget cannot reach is explicitly `deferred` with reason `budget_cannot_reserve_critics_and_validation`. + +When a budget or profile caps hunter count, assign units in priority order and record the ordering rationale in the ledger. Rank by: (1) unauthenticated or lowest-trust entry surfaces before authenticated ones; (2) boundaries protecting the most valuable resources (credentials, cross-tenant data, code execution, release authority); (3) prior-run gaps, revalidation targets, and changed source before same-source re-passes; (4) units whose class historically yields confirmed findings for this target type over speculative ones. Ties break lexicographically by `coverage_id` so runs stay deterministic. + +Before launch, the parent changes assigned units to `in_progress`, sets a canonical lowercase `agent_id`, and creates that agent's `scratch/` and parent-owned `artifacts/`. Hunters read source and parent-provided context, write only to their unique `scratch/`, and return one structured result through the Task tool. They never write retained artifacts or edit target source, `architecture.md`, `coverage-ledger.json`, `findings.json`, or another agent's files. + +## Required hunter prompt + +Every hunter prompt contains these parts in this order: + +1. A two-sentence role preamble: the hunter's goal is to find source-grounded security invariant failures in its assigned units, and it must return exactly one JSON object matching the structured-result contract at the end of this prompt. +2. `architecture.md` verbatim. +3. Assigned coverage IDs, subsystem, boundary, repository-relative starting paths, and each unit's assignment block map from `coverage-ledger.json`. +4. The exact selected blocks, copied verbatim: each selected ordinary attack-class block from `ATTACK-CLASSES.md`, and from each selected companion its `Core discipline`, each chosen attack-class subsection, `Universal moves`, and `Validation rules`. Ordinary blocks are self-contained and carry no companion-style `Core discipline`, `Universal moves`, or `Validation rules` sections. Do not send block or companion names alone. +5. Explicit excluded ordinary and companion blocks with a reason for each exclusion. +6. The core hunting method below, followed by the promotion procedure block. +7. The core validation rules below. +8. Carried same-source prior confirmed exclusions, each limited to fingerprint, title, and root cause, plus peer-owned current coverage IDs that this hunter must not duplicate. +9. The unique scratch/artifact paths, safe agent ID, predeclared promotion allowlist and byte limits, and the structured-result contract, including the Structured hunter result block below and the `confirmed` and `needs_validation` branches of `report-schema.json` copied verbatim. + +A prompt may select several companion blocks when the same path crosses several domains. Keep their constraints together. Scope is the hunter's coverage obligation, not permission to duplicate excluded work. If an unexpected different boundary appears, return it under `uncovered` so the parent creates a stable ledger unit and assigns it in the next wave. + +#### Core hunting method — include in every hunter prompt + +```text +## Defensive vulnerability-finding method + +Your goal is to find source-grounded security invariant failures and the smallest fix, +not to expand harm beyond the boundary result. Stay within source review and bounded local execution. +Do not contact deployed endpoints, provider APIs, registries, identity systems, +message brokers, shared services, or other users. Use local dummy data only. + +READ THE CODE AT DEPTH. Follow each assigned input through parsing, identity, +authorization, normalization, state, derived copies, and the final sink. Read sibling, +legacy, batch, retry, cancellation, migration, and error paths that produce the same +effect. Compare sibling controls for equivalence, not only presence, and compare what +one component guarantees with what the next component assumes. + +WORK FROM A CONCRETE INVARIANT: +1. Name the lower-trust principal and starting capability. +2. Name the accepted value, action, state transition, or resource selector. +3. Locate the control that should reject, bind, isolate, limit, or revoke it. +4. Trace the exact source path after that decision. +5. Stop at the smallest affected dummy record, wrong return value, process-integrity + effect, or locally observable shared-resource effect. +6. State a source-level change and regression case that enforce the invariant. + +DEPTH BOUND: trace only paths that can reach your assigned boundary or whose +guarantees that boundary relies on. Stop a line of investigation as soon as the +invariant is settled either way, and record the result in your structured output — +a covered, candidate, or blocked disposition, or an `uncovered` entry — instead of +continuing to search. + +TEST SAD PATHS AND DISAGREEMENTS. Check absent, empty, zero, negative, maximum, +over-limit, duplicate, mixed encoding, stale, revoked, reordered, concurrent, +partially migrated, failed dependency, and rollback state only where the interface +accepts them. Compare canonicalization and units at every parser or policy handoff. +For multi-step issues, treat each output as a prerequisite and do not assume a later +boundary. If any prerequisite is not established, record a blocker. + +When a proposed high or critical candidate reveals a reusable root cause, search paths +owned by the assigned coverage IDs for lexical, structural, and logical variants. +Consolidate the same root cause, but establish each variant's conditions and impact +independently. Do not investigate peer-owned units. Return a variant with no current +coverage unit as `uncovered`. + +USE THE NARROWEST LOCAL CHECK THAT SETTLES THE CLAIM. Target-controlled builds, +tests, processes, browsers, emulators, fuzzers, and fixture processing may run only +inside the parent-approved OS-enforced sandbox. It must disable external networking, +start from an empty allowlisted environment, expose target and tools read-only, permit +writes only to your scratch directory, and apply low CPU, memory, process, file-size, +disk, and wall-clock limits. Isolated loopback is allowed only for a local fixture. +If any control is unavailable, do not execute: return needs_validation with that exact +blocker. Prefer an existing unit test, minimal function harness, dummy-tenant service +call, small malformed fixture, deterministic race schedule, or locally rendered policy. +Do not install or fetch tools. + +Record the exact input, command, limits, and minimum result. For the environment, +record only allowlisted variable names and safe non-secret values needed to reproduce +the check. Never capture the ambient environment, inherited variables, credentials, +authentication state, or unrelated host paths. The target-controlled process writes +only in scratch. After the sandbox and all its processes terminate, only trusted +parent-side code may promote predeclared scratch-relative files, following the +promotion procedure block included verbatim in this prompt. You and target code never +write retained artifacts. If promotion is unavailable or fails for decisive evidence, +return needs_validation with the exact promotion blocker. +Never stress availability, invoke a live target, use a real credential, publish an +artifact, or continue past the minimum observed effect. + +A deployment, browser, provider, broker, OS, proxy, package, secret, or identity fact +outside source is not proof either way. If one such fact is decisive, return a +needs_validation record with the exact missing observation and safe owner-observed check. +``` + +#### Promotion procedure — copy this promotion procedure verbatim into every hunter prompt + +```text +Artifact promotion procedure (trusted parent-side code only): +Reference only for you: the parent performs these steps; you never perform them. + +Before execution, the parent opens and retains trusted, non-inheritable directory +descriptors for the agent's scratch/ and artifacts/ roots, and records an allowlist +of expected scratch-relative artifact files plus explicit per-file and cumulative +byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox +and all its processes terminate, trusted parent-side code promotes each allowlisted +file separately: + +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or + symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with + no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and + within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, + type, link count, or size. +7. For the destination, walk every parent component from the retained + artifacts-root descriptor with no-follow directory-relative operations; require + each existing component to be a real directory, and create any missing directory + exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened + destination is a regular file with link count exactly one, and copy from the + verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or + open or promote a symlink, FIFO, socket, device, directory, hard-linked file, + changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch + entry; if it is decisive evidence, retain `needs_validation` with the exact + promotion blocker. +``` + +#### Core validation rules — include in every hunter prompt + +```text +## Candidate gate + +1. A candidate needs a complete repository-relative source trace and evidence for the + claimed root cause, including the strongest source-visible control. +2. A proposed confirmed record needs a bounded local observed result, meaningful impact + across a stated boundary, complete conditions, and no visible preventing layer. +3. Do not strengthen a crash into code execution, ordinary work into shared availability, + or a same-principal action into privilege gain. +4. If a required fact is not source-visible or locally observable, use + needs_validation. Name exact blockers; do not give it severity or speculative completion. +5. A missing best practice with no affected principal/resource is excluded or hardening, + not a finding. A candidate disproved by source is not needs_validation. +6. Use the same source-derived fingerprint for the same root cause in every state. + It must match `^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$` and must not include a line, + wave, agent, severity, or verdict. +7. Return an empty candidate array when nothing survives these gates. +``` + +## Local validation boundaries + +Local execution is for confirmation, not impact expansion: + +- **Allowed only in the required OS sandbox:** offline builds with present dependencies; isolated-loopback processes using dummy state; unit and integration tests; small fixture processing; sanitizers; bounded fuzz/regression tests; deterministic concurrency checks; local browser/emulator tests with dummy accounts; rendered manifests and policy evaluation with dummy identities; mocked external or paid calls. +- **Disallowed:** live or deployed traffic; requests to services not started for this isolated check; network dependency installation; real accounts or credentials; production data; shared queues, cloud resources, runners, registries, signing or release services; publishing; stress, saturation, or cost generation; any work after the minimum dummy-data boundary result. + +The sandbox starts with an empty environment, gives target code no external network or host writable path, and enforces explicit low resource and time limits for every check, not only checks expected to be expensive. Scratch output remains target-controlled after exit. Promote it only with the no-follow, path-confined, regular-file, bounded-size host procedure in `SKILL.md`. Missing any sandbox or promotion capability does not erase a source-grounded candidate; represent the exact blocker in `needs_validation`. + +## Structured hunter result + +Return exactly one JSON object, with no surrounding prose: + +```json +{ + "units": [ + { + "coverage_id": "one assigned ID", + "disposition": "covered|candidate|blocked", + "reviewed_paths": ["repo/relative/path"], + "checks": [ + { + "agent_id": "canonical owner of this check", + "reviewed_paths": ["repo/relative/path owned by this check"], + "invariant": "specific control checked for this unit", + "method": "source|local", + "result": "what source or the bounded check established", + "artifact": "agents/<agent-id>/artifacts/file for local, null for source" + } + ], + "candidate_fingerprints": [], + "unresolved": [] + } + ], + "candidates": [], + "hardening": ["concrete non-finding note"], + "uncovered": [ + { + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "reason": "why this needs its own deterministic coverage unit" + } + ] +} +``` + +Each `candidates` entry is schema-shaped except that it uses `proposed_verdict` in place of `verdict`: + +- `proposed_verdict: "confirmed"`: include every field required by the `confirmed` branch of `report-schema.json` other than `verdict`: `fingerprint`, title, description, `root_cause`, `intended_behavior`, ordered `trace`, `evidence`, `conditions`, target-neutral `execution`, `remediation`, `severity`, and `confidence`. The execution instructions describe only the bounded local check already performed. `payloads` holds the minimum test input, fixture, or native invocation. `observed_result` records actual local output. Overall severity must not exceed observed impact. +- `proposed_verdict: "needs_validation"`: include every field required by that schema branch other than `verdict`: `fingerprint`, title, description, `claimed_root_cause`, ordered `trace`, `evidence`, nonempty `blockers`, and `validation_plan` with at least one applicable `local` or `deployment` step. Do not invent an inapplicable context. Do not include severity, execution, remediation, reason, or a confirmed `root_cause`. `deployment` is an owner-observed check, not a request to probe a live target. + +Every assigned coverage ID appears exactly once in `units`. A `covered` unit needs an owner, nonempty `reviewed_paths` and `checks`, no unresolved fact, and no candidate. A `candidate` unit has the same owned evidence and is the only state that carries linked fingerprints. A `blocked` unit is an owned partial review with nonempty paths, checks, and unresolved facts but no fingerprint. All source paths are repository-relative, never absolute or traversal paths. A trace with several entries begins at `entrypoint`, ends at `sink`, and labels intermediate steps `propagation`. Every check has its own canonical lowercase `agent_id` and nonempty `reviewed_paths`; the unit-level list is exactly the union of those owned paths. A `source` check uses `artifact: null`. A `local` check uses one successfully parent-promoted regular file beneath `agents/<check.agent_id>/artifacts/`; this permits a verifier to add independently owned evidence without taking ownership from the hunter. Never link scratch, an output-root file, or another check owner's artifact. + +## Parent consolidation and ledger update + +The parent validates each unit result, maps it to exactly one assigned `coverage_id`, and updates only that ledger unit. Reject duplicate or absent IDs, unsafe unit or check agent IDs, source checks with artifacts, and local artifacts that trusted parent-side code did not promote into the check owner's artifacts subtree. Copy the unit's `reviewed_paths`, its `checks` into the unit's `local_checks`, linked artifact paths, candidate fingerprints, and unresolved facts into the ledger. Retain each hunter's `hardening` list in a parent bookkeeping field on the relevant units (outside the semantic fields) so Phase 6 can report it. A failed, malformed, or unsupported conclusion leaves that unit `planned` for reassignment. Untouched budget/profile units become unassigned `deferred` units with empty evidence and a reason; do not hide partial evidence in `deferred`. Run `validate-coverage-ledger.cjs` after the update; an invalid ledger cannot drive another assignment. This per-unit contract allows one hunter to close one unit while returning a candidate or blocker for another. + +Consolidate candidate entries by fingerprint and then by root cause. One root cause that exposes several entry paths or effects is one candidate with the strongest complete trace. Related but independent missing controls use separate fingerprints. Record duplicate fingerprints in the relevant ledger unit and do not send duplicate candidates to validation. + +## Coverage-critic waves + +Immediately after each hunter wave, spend the reserved invocation on one fresh `research` post-wave coverage critic. It receives `architecture.md`, the full coverage ledger including each assignment block map, current candidate fingerprints and states, and the prior-ledger gap summary. It reads source but does not write or run targets. Require exactly this JSON: + +```json +{ + "missing_units": [ + { + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "selected_companion_blocks": ["FILE.md#section"], + "excluded_blocks": [{"block": "FILE.md#section", "reason": "..."}], + "reason": "source-backed coverage gap" + } + ], + "reassign_ids": ["existing-id-that-did-not-close"], + "resolved_prior_leads": ["fingerprint"], + "stop": false +} +``` + +The critic checks for unmapped entry points, unchecked parallel paths, missing lifecycle modes, selected companion classes without a unit, unjustified exclusions, units closed without paths/checks, and prior `needs_validation` or changed-source gaps that no unit addresses. It proposes coverage, not findings. `stop` is the critic's own assessment: `true` only when it accepts no `missing_units` and no `reassign_ids`; the parent's loop condition below, not `stop` alone, decides whether another wave runs. For each fingerprint in `resolved_prior_leads`, the parent marks the linked unit or prior-lead entry resolved and records the critic's source-backed reason. + +The parent rejects proposed units outside the review scope or source/local boundary, derives canonical IDs for accepted units, and deduplicates them against current units. A prior same-source completed unit may supply evidence; prior `deferred`, `blocked`, `out_of_scope`, or changed-source units become current work and never suppress an accepted unit. Fail rather than merge a canonical ID collision. For each legitimate `reassign_id` with live `blocked`, `covered`, or `candidate` evidence, append that exact terminal record to the unit's `attempts` with the critic's source-backed `reassignment_reason`. Preserve its owner, checks, artifacts, fingerprints, and unresolved facts in that archive. Increment the live `wave`; the next hunter must be a fresh owner and receives an `in_progress` unit with empty live evidence. The hunter's terminal result writes only its new evidence into the live fields. Never copy an archived owner's checks or artifacts into the new live attempt. Sort IDs and validate the ledger before another assignment. In `standard` and `deep`, when the post-wave critic reports no accepted `missing_units` or legitimate `reassign_ids` and no `planned` units remain, spend the separately reserved invocation on a distinct final-clean critic. Complete coverage only when that critic also returns no accepted work. If it finds work, queue it and repeat the wave, post-wave critic, and final-clean process. If time or resources force an early stop, mark every untouched unit `deferred`, preserve the critic's reason, and disclose the gap in the report. Never use a silent wave or agent cap as evidence of complete coverage. + +The run profile bounds this loop. A `quick` run has exactly one hunter wave followed by exactly one final critic pass. Add each accepted `missing_unit` to the current ledger and mark it `deferred` with reason `quick_profile_final_critic`. For each legitimate evidence-bearing `reassign_id`, archive the live terminal state in `attempts`, increment `wave`, and set the live state to unassigned `deferred` with empty evidence and reason `quick_profile_final_critic`. Do not launch a second hunter wave or another critic. In a scoped run, the critic still reports out-of-scope gaps it notices, but the parent records them as `out_of_scope` with the critic's reason instead of assigning them. The early-stop rule above is the same mechanism: `quick` is a pre-declared early stop, not evidence of complete coverage. + +A budget bounds it the same way. Before assigning each wave, compare remaining budget against its hunter count, the validation reserve, the immediate post-wave critic, and the retained final-clean critic (`quick` reserves only its single final post-wave critic). Shrink the hunter wave to fit, taking units in priority order. If those mandatory reserves do not fit, launch no hunter from that wave and mark its planned units `deferred` with reason `budget_cannot_reserve_critics_and_validation`. Critic-proposed units enter the same ranked queue rather than extending the budget. If surviving candidates exceed the validation reserve, follow the incomplete-run rule in `SKILL.md`: stop hunting, validate in fingerprint order, retain unvalidated units as unresolved candidates, and never present them as findings. diff --git a/skills/thirdshift-security-audit/LICENSE b/skills/thirdshift-security-audit/LICENSE new file mode 100644 index 00000000..6dbc9ecb --- /dev/null +++ b/skills/thirdshift-security-audit/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025-2026 Cloudflare, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/skills/thirdshift-security-audit/MEMORY-SAFETY-AND-BINARY.md b/skills/thirdshift-security-audit/MEMORY-SAFETY-AND-BINARY.md new file mode 100644 index 00000000..86ff4920 --- /dev/null +++ b/skills/thirdshift-security-audit/MEMORY-SAFETY-AND-BINARY.md @@ -0,0 +1,101 @@ +# Memory Safety, Binary, and Kernel Hunting + +#### When to use this file + +Reach for this file when the target processes untrusted bytes in a memory-unsafe or privileged context: C/C++/Objective-C, Rust `unsafe`, FFI, kernel modules and drivers, parsers and decoders, network daemons, firmware, binary loaders, language runtimes, and JITs. Use `PROTOCOLS-RPC-AND-MESSAGING.md` for protocol authorization and state-machine logic, and this file for process integrity, memory safety, ABI boundaries, and loader behavior. + +Pick relevant classes from Phase 1 and split large targets by parser, allocator/lifetime, FFI, concurrency, loader, runtime, or privileged interface. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Re-derive every bound and lifetime from attacker-controlled inputs and all callers. Validate against the worst accepted case, not a typical test vector. +- A panic, sanitizer finding, or crash proves a defect only when a realistic untrusted input reaches it. Do not infer memory corruption, code execution, or shared availability impact from a label alone. +- Validate in a local harness with sanitizers, deterministic concurrency tests, existing fuzz targets, and debugger-assisted fault classification. Stop after proving the violated invariant and observable impact; do not develop post-corruption techniques. +- Assembly, JIT code, custom allocators, intra-object accesses, and foreign libraries can escape sanitizer coverage. Identify which relevant instructions are instrumented. +- Classify as `confirmed` only after source evidence and bounded local validation establish the defect and effect. Use `needs_validation` when ABI, allocator, architecture, feature, deployment, or reachability facts remain unknown. +``` + +## Bounds, integer, and representation attack classes (subagent_type: `general`) + +**Out-of-bounds read or write** +A length, offset, index, or terminator reaches a fixed or allocated buffer without a correct bound. Recalculate available headroom after prefixes, alignment, padding, and terminators. Check both source and destination capacity, and whether a short input is read before its declared length is trusted. + +**Integer overflow, underflow, truncation, and signedness** +Review attacker-controlled arithmetic before allocation, copy, loop, indexing, and pointer operations. High-hit patterns include `a - b` with `b > a`, `count * element_size`, additions near the type maximum, negative values converted to unsigned, 64-bit lengths narrowed to 32-bit fields, and sentinel values such as `-1` becoming a large size. Confirm which checked representation is later used. + +**Unit and pointer-depth confusion** +Code mixes bytes, elements, code units, pages, words, wire units, or nested pointer element sizes. Compare the unit at parse, validation, allocation, API boundary, and copy. A bounds check using the same wrong unit as the allocation is still wrong. + +**Uninitialized or partially initialized data** +A buffer, padding, struct field, or vector capacity is returned, compared, hashed, serialized, or passed across a trust boundary before initialization. Require an observable consumer and realistic output length; stack allocation by itself is not disclosure. + +## Lifetime, type, and concurrency attack classes (subagent_type: `general`) + +**Use-after-free, stale view, and double free** +Owners are released while callbacks, wait queues, timers, iterators, borrowed slices, or cached raw pointers can still use them. Review every error, cancellation, close, and realloc path. For embedded notification anchors, each free path must drain or detach all observers. + +**Type confusion and invalid downcast** +A tag, vtable, union discriminator, object kind, or foreign handle is checked differently from the representation later read. Look for unchecked dynamic casts, stale tags after reuse, and serialized types whose validated element differs from the element consumed. Confirm a wrong-type read or write locally without extending the test beyond the violated invariant. + +**Reference-count and ownership races** +Non-atomic retain/release, a check followed by an unlocked use, or inconsistent ownership across threads can free or mutate an object during access. Compare fast, error, shutdown, and compatibility paths for the same lock and ownership rules. + +**Shared-state races and TOCTOU** +Concurrent parser streams, global caches, lazy initialization, signal handlers, and resource teardown can invalidate bounds, policy, or pointers established earlier. Verify the race with a repeatable local schedule, barrier, or thread sanitizer; a hypothetical interleaving without a security-relevant state transition remains `needs_validation`. + +**Lock-order, deadlock, and starvation** +Externally reachable operations acquire locks in inconsistent order or hold them across callbacks and blocking I/O. Report under availability only when bounded input can stop shared progress; otherwise record it for fixing as a concurrency defect. + +## FFI and ABI attack classes (subagent_type: `general`) + +**Pointer-length and ownership contract mismatch** +Caller and callee disagree on who allocates, frees, pins, or mutates a buffer, how long a pointer remains valid, or whether a length is bytes or elements. Trace both sides of every `extern`, CGo/JNI/Python/native binding, and generated wrapper. Check null, zero length, aliasing, and callback retention. + +**Layout, alignment, and enum disagreement** +Foreign code receives a struct, bitfield, packed record, callback signature, integer width, enum, or calling convention that differs by architecture or build flag. Verify `repr`, packing, alignment, endianness, and ABI-specific types. An in-repo declaration mismatch can be confirmed locally; an opaque foreign implementation requires `needs_validation`. + +**Unwind, exception, and thread-affinity violations** +Exceptions or panics cross an ABI that forbids unwinding, callbacks run after teardown, or APIs requiring one runtime thread are invoked elsewhere. Review error conversion and cancellation. Confirm whether the process aborts or state is corrupted before assigning impact. + +## Binary loading and runtime attack classes (subagent_type: `general`) + +**Library, plugin, and executable search-order trust** +A privileged process loads a library, plugin, runtime image, or helper from a path writable by a less-trusted principal, or resolves a bare name through an attacker-influenceable working directory or environment. Compare intended installation ownership with each fallback and compatibility search path. A user loading their own plugin into their own process is not a boundary violation. + +**Missing artifact identity or signature binding** +A loader verifies one file or metadata record but maps a different image because path resolution, file replacement, architecture slices, or embedded resources are not bound to the check. Supply-channel authenticity belongs in `SUPPLY-CHAIN-AND-RELEASE.md`; this class covers the local verification-to-map gap. + +**Malformed binary metadata and relocation handling** +Offsets, counts, sections, relocations, symbols, bytecode, or debug metadata are trusted before range, overlap, and representation checks. Test parsers with bounded local fixtures and sanitizers. Separate memory corruption from a safely rejected malformed file. + +**JIT and generated-code consistency** +Validator, interpreter, optimizer, and generated code disagree about types, bounds, side effects, or lifetime. Diff optimized and unoptimized paths using the same local input. Confirm a process-integrity effect; output variance that stays within language semantics is not a finding. + +**Unload, reload, and teardown safety** +Live function pointers, callbacks, worker threads, or data views survive module unload or runtime reset. Review shutdown and failed-load cleanup as closely as startup. + +## Kernel and privileged-interface attack classes (subagent_type: `general`) + +**User-copy bounds and repeated reads** +A syscall, ioctl, driver, or kernel parser derives a trusted fact from user memory then reads the same mutable address again. Copy the full request once or revalidate the later copy. Also audit size, direction, and access checks at each user-copy primitive. + +**Privileged object lifecycle and dispatch consistency** +Externally reachable objects have unbalanced retain/release, teardown without observer drain, unchecked selector/table indices, or duplicated compatibility paths that omit a guard. Diff each dispatch and free path side by side. + +**Under-authorized powerful interfaces** +A device node, admin socket, helper, or management API validates shape but not the caller's authority over the resource. Establish actual interface ownership and reachability; permissions or sandbox policy outside the repository make this `needs_validation`. + +## Universal moves (apply across the above) + +- Audit fixes and duplicated paths for the same source-to-sink shape. A check in one caller, architecture, protocol role, feature flag, or compatibility path does not protect its siblings. +- Build a table for every parser or FFI boundary: accepted length/type, checked representation, allocation owner, consumer, thread, and teardown. Most native findings are one disagreement in that table. +- Use existing corpora and small locally generated boundary fixtures. Save exact sanitizer/runtime output and the input property that triggers it; avoid large resource consumption and any live target. + +## Validation rules (apply before reporting ANY finding here) + +1. Establish a realistic untrusted entry and exact operation that violates a bounds, type, lifetime, ABI, concurrency, loader, or authority invariant. +2. Classify the observable effect: invalid read, invalid write, stale alias, wrong object, uninitialized output, unauthorized image load, deadlock, or safe process termination. Do not claim a stronger effect than observed. +3. Run the narrowest local harness, existing test, sanitizer, or fuzzer needed to reproduce the effect. Verify sanitizer coverage of the faulting operation and record architecture/build conditions. +4. For concurrency, use a deterministic schedule or sanitizer trace. For binary loading, prove the checked identity differs from the mapped identity and name the lower-trust writer. +5. Return `confirmed` findings only with exact input, source trace, and observed result. Return `needs_validation` for a specific unresolved reachability, ABI, build, deployment, or runtime fact and state the bounded check needed. diff --git a/skills/thirdshift-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md b/skills/thirdshift-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md new file mode 100644 index 00000000..b04ab88f --- /dev/null +++ b/skills/thirdshift-security-audit/PROTOCOLS-RPC-AND-MESSAGING.md @@ -0,0 +1,81 @@ +# Protocols, RPC, and Messaging Hunting + +#### When to use this file + +Reach for this file when the target uses gRPC, GraphQL transports, Cap'n Proto, Thrift, Protobuf, custom binary protocols, streaming RPC, webhooks, brokers, queues, pub/sub, or event buses. It covers peer identity, logical message interpretation, routing, replay, ordering, and delivery semantics. Use `MEMORY-SAFETY-AND-BINARY.md` for parser memory safety, `WEB-PROTOCOL-AND-AUTH.md` for HTTP framing, and `RESOURCE-EXHAUSTION-AND-AVAILABILITY.md` for availability impact. + +Split large systems by producer/consumer pair, external/internal peer role, synchronous RPC, streaming, and asynchronous message path. + +## Core discipline (include in every agent prompt for this domain) + +``` +- "Internal" is not authentication. Name the peer identity at every hop and show how it becomes the application principal used for authorization. +- Schema validation proves message shape, not provenance, resource authority, ordering, or safe values. Follow decoded fields to policy and side effects. +- Broker guarantees and application guarantees differ. Write down retry, ordering, acknowledgement, deduplication, and transaction behavior before evaluating state changes. +- Parser disagreement requires two concrete consumers, schema versions, or wire representations and one security-relevant divergent value. +- Use `confirmed` for source-complete paths plus bounded local producer/consumer tests. Use `needs_validation` for broker ACL, service-mesh identity, topic attachment, or compatibility behavior outside the repository. +``` + +## Framing, schema, and interpretation attack classes (subagent_type: `general`) + +**Message boundary and canonicalization disagreement** +Components disagree on length, compression, duplicate fields, unknown fields, encoding, numeric width, normalization, or envelope/body precedence. Compare generated and custom parsers, gateways, language bindings, and version converters. Confirm which principal, resource, or operation differs after decoding. + +**Union, enum, and default confusion** +Unknown variants, missing discriminators, zero values, default privileges, or compatibility mappings reach code that assumes a validated case. Review exhaustive dispatch, default branches, and how old consumers interpret newly added fields. + +**Envelope and payload identity mismatch** +Authorization uses trusted-looking routing or envelope metadata while the handler acts on a conflicting tenant, account, subject, object, or sender in the body. Identify which source is authoritative and ensure clients cannot override it. + +## RPC identity and authorization attack classes (subagent_type: `general`) + +**Interceptor and method-path inconsistency** +An authn/authz interceptor applies to unary methods but not streams, reflection, health, gateway-transcoded paths, compatibility services, or individual stream messages. Compare every registration and route to the same operation. + +**Peer identity to application-principal confusion** +mTLS, workload identity, bearer metadata, forwarded identity, or broker credentials authenticate a channel, but a caller-controlled field selects the user or tenant. The channel identity and claimed principal must be bound by deterministic policy. + +**Per-item and streaming authorization gaps** +A stream, subscription, batch, or bulk message is authorized once, then later items name different resources or continue after role, membership, or token revocation. Re-check where scope can change and bind subscriptions to their original principal. + +**Callback and reply-correlation confusion** +Predictable, reused, or cross-tenant correlation IDs let a response, webhook, cancellation, or acknowledgment satisfy another caller's pending operation. Bind each outstanding request to authenticated peer, tenant, operation, and lifecycle. + +## Broker and queue isolation attack classes (subagent_type: `general`) + +**Topic, routing-key, and subscription scope gaps** +A publisher or subscriber can select another tenant's topic, wildcard, consumer group, partition, reply queue, or dead-letter route. Check broker-enforced ACLs where visible and application-side namespace construction. Tenant text inside a payload is not isolation. + +**Dead-letter, retry, and diagnostic disclosure** +Messages routed to dead-letter queues, error topics, tracing, or operator views contain secrets or cross-tenant payloads accessible to a lower-trust consumer. Review policy and redaction at the failure path, not just normal delivery. + +**Untrusted producer treated as control plane** +A message body can declare itself an admin event, provider callback, replication record, or migration instruction without an independently authenticated producer and event type. Verify signatures and source/account/audience binding before privileged handling. + +## Replay, ordering, and transaction attack classes (subagent_type: `general`) + +**Duplicate delivery and idempotency gaps** +Retries or redelivery repeat a side effect because deduplication is absent, occurs after mutation, or uses a key that collides across tenants or operations. Confirm the broker's delivery model and the side effect that is not naturally idempotent. + +**Out-of-order and stale message acceptance** +Older state, revoked membership, canceled work, or pre-step-up authorization arrives after newer state and overwrites it. Review sequence/version checks, tombstones, partition changes, and restore/replay workflows. + +**Acknowledgment/commit ordering defects** +Acknowledgment occurs before durable commit and loses security-relevant work, or commit happens before an unreliable acknowledgment and duplicates a mutation. Evaluate transactional outbox/inbox behavior and failure recovery. + +**Partial multi-consumer transitions** +Several consumers jointly implement one authorization or business transition, but retries and partial failure leave only a subset committed. Identify invariants that must become durable atomically or compensate with current authorization. + +## Universal moves (apply across the above) + +- Draw producer → broker/transport → gateway → consumer → storage for each message family. At each hop record authenticated peer, authoritative tenant/resource fields, validation, and side effect. +- Feed the same small fixture to every in-repo schema version or language binding. Test duplicate, missing, unknown, boundary, replayed, and reordered messages without producing load. +- Compare normal, retry, dead-letter, replay, migration, reflection, stream, and gateway-transcoded routes. Security policy must survive transport changes. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the realistic producer or peer, accepted message, authenticated channel identity, affected principal/resource, and unauthorized mutation or disclosure. +2. For disagreement claims, cite both parsers/consumers and the divergent decoded value. Safe rejection by either side prevents confirmation. +3. For replay/order claims, establish actual delivery guarantees and reproduce the invariant failure with a bounded local/in-memory transport. +4. For authorization and isolation, verify all interceptor, broker ACL, gateway, and consumer layers visible in source. External attachments make the candidate `needs_validation`. +5. Return `confirmed` only with the complete message lifecycle and observed meaningful result. Return `needs_validation` with the exact broker, service identity, route, or delivery fact required. diff --git a/skills/thirdshift-security-audit/RECONNAISSANCE.md b/skills/thirdshift-security-audit/RECONNAISSANCE.md new file mode 100644 index 00000000..1a170a93 --- /dev/null +++ b/skills/thirdshift-security-audit/RECONNAISSANCE.md @@ -0,0 +1,156 @@ +# Reconnaissance + +### Phase 1: Map the source and plan coverage + +The parent initializes `run-metadata.json`, applies the strict pre-reconnaissance budget gate in `SKILL.md`, then creates agent scratch roots and the shared ledger before hunting. If the gate fails, record the incomplete status in metadata and launch no reconnaissance agent. Reconnaissance reads the target and locally available build/configuration state only. It does not contact deployed endpoints, external identity providers, registries, brokers, cloud APIs, or other shared services. + +Launch several `research` agents in parallel. They return structured facts to the parent and do not write files. + +**Agent 1a: Product, stack, and local operation** + +```text +Read the target at <target>. Do not use network access. Return: +1. Product type, users, operators, and ordinary trust-sensitive actions. +2. Languages, frameworks, build system, runtimes, and locally visible deployment models. +3. Repository-relative entry points and subsystem boundaries. +4. Exact build and test commands that could run offline with local dependencies, their expected write locations, and the target-controlled inputs they process. Do not run them during reconnaissance. +5. Comparable software or protocol visible from local documentation and dependencies. If no useful comparison is source-grounded, say so. +6. Missing local toolchains or runtime facts that limit bounded execution. +Return only source facts with repository-relative file:line references. +``` + +**Agent 1b: Principals, authority, and controls** + +```text +Read all source that establishes identity, authorization, isolation, and privilege. Map: +1. Each lower-trust principal and the actions it has by design. +2. Authentication or peer identity at each entry surface. +3. Per-resource authorization and tenant/owner scope. +4. Process, browser, workload, CI, plugin, model/tool, device, or local-IPC authority. +5. Privilege changes, confirmation, revocation, recovery, and fallback paths. +6. Which controls are source-visible and which depend on an unobserved deployment fact. +Return trust boundaries and control locations with repository-relative file:line references. Do not infer live reachability. +``` + +**Agent 1c: Entry surfaces, copies, and sinks** + +```text +Inventory every source-visible place external or lower-trust input enters: +- HTTP/browser, RPC/message/protocol, files/archive/document, CLI/env/config, plugins/dependencies/CI, cloud events/IAM selectors, model context/tool arguments, mobile/deep-link/webview, and local IPC. +For each surface, follow major transformations, stored or derived copies, and security-relevant sinks. Record source-visible limits and parallel paths to the same effect. +Return repository-relative paths and line numbers. Be complete, but do not execute or send inputs. +``` + +**Agent 1d: Local execution and deployment visibility** + +```text +Read tests, build definitions, manifests, packaging, and maintained environment overlays. Return: +1. Small offline tests or existing fixtures that could validate trust boundaries with dummy data inside the required OS-enforced sandbox. +2. Processes that could use an isolated loopback network namespace without external or shared dependencies. +3. Commands that would fetch dependencies, publish artifacts, contact paid/provider APIs, or affect shared state; mark them prohibited for this run. +4. Deployed controls and attachments that source cannot establish and therefore require needs_validation if decisive. +5. The final active source path for each deployment mode only where the repository selects it deterministically. +6. Whether the local platform can enforce an empty allowlisted environment, no external network, read-only target/toolchain mounts, scratch-only writes, and explicit CPU, memory, process, file-size, disk, and wall-clock limits. Missing controls block target-controlled execution. +7. Whether trusted parent-side code can promote predeclared scratch files with path-confined no-follow descriptor traversal, nonblocking regular-file checks, no-follow traversal of every destination parent, exclusive regular-file destination creation, and explicit per-file and cumulative size bounds. Missing promotion controls block use of scratch files as evidence. +``` + +Add focused reconnaissance agents for materially distinct deployment modes or subsystems that these four do not map. Do not silently omit them: if the budget gate in `SKILL.md` blocks a focused agent, launch nothing for it, seed the unmapped area as a `deferred` ledger unit with reason `budget_cannot_reserve_critics_and_validation`, and disclose the gap in the report. + +## Prior-run input + +Before selecting work, the parent reads every available prior `coverage-ledger.json` and `findings.json` for the same repo: + +- Compare the source locations, controls, conditions, and source-derived identity for every prior record and unit against the current source. +- Carry an unchanged prior `confirmed` record into the current candidate set, with the same fingerprint, only when its relevant source, conditions, and qualifying evidence still apply. Link it to a current `planned` unit with `prior_status: "prior_confirmed_same_source"` and put only that root cause on the hunter exclusion list. The Phase 3 verifier that re-verifies the carried record becomes that unit's assignment owner; its source re-check is the unit's first check and moves the unit to `candidate` with the carried fingerprint. +- Build a current planned `prior_confirmed_changed_source` revalidation unit when any relevant source or condition changed. Do not exclude that root cause from hunting or assume the prior verdict still applies. +- Build current work units for every prior `needs_validation`, `deferred`, `blocked`, `out_of_scope`, and changed-source unit. These states are priority input, never deduplication or suppression keys. +- Carry a still-blocked prior `needs_validation` record with the same fingerprint only after current source supports its trace. Link it to a current `planned` unit with `prior_status: "prior_needs_validation"`; the record keeps the unresolved blocker. The Phase 3 verifier that re-checks the carried record becomes that unit's assignment owner; its re-check is the unit's first check and moves the unit to `candidate` with the carried fingerprint. Include the record in final verification. +- Treat prior rejected records as stale claims unless current evidence changes the failed trace or missing condition. An unchanged rejection suppresses only that exact claim, not review of the coverage unit. +- Record missing or incompatible ledgers instead of treating them as empty coverage. + +State paths and source refs used in `run-metadata.json`. Summarize only the coverage consequences in `architecture.md`. + +## Architecture summary and companion selection + +The parent synthesizes `<output-dir>/architecture.md`, with a hard cap of about 1,000 words. Include: + +1. Product, principals, normal authority, and protected resources. +2. The comparable-software baseline from Agent 1a, when one is source-grounded: what security trade-offs the comparable accepts. Use it to calibrate effort and severity, never to dismiss a demonstrated finding; if the comparable shares a defect pattern that has mattered in practice, that strengthens the finding. Omit this line when no meaningful comparable exists. +3. Tech stack, source-visible deployment paths, and offline build/test limits. +4. Entry surfaces and the important source-to-sink or lifecycle paths. +5. Trust boundaries and the strongest source-visible control on each. +6. Repository-relative starting paths. +7. Prior coverage gaps, changed-source and blocked revalidation targets, and same-source confirmed exclusions. +8. A short companion-selection summary derived from [ATTACK-CLASSES.md](ATTACK-CLASSES.md): selected files and the source-visible boundaries that require them. + +Keep the assignment-level ordinary block, selected companion blocks, and excluded blocks with reasons in each ledger unit, not in `architecture.md`. This keeps the architecture cap valid for large runs and makes the exact hunter prompt map machine-checkable. + +Do not select a companion file merely because the language or dependency name appears. Select it because reconnaissance found the trust-sensitive boundary described by its `When to use this file` section. Do not exclude a visible boundary just because another agent will review a related class. + +## Deterministic coverage ledger + +The parent writes `<output-dir>/coverage-ledger.json` as a top-level JSON array. Derive one unit for every material combination of entry surface, trust boundary, subsystem, and applicable ordinary or companion attack class at the granularity the run profile sets (`quick` uses one all-in-scope subsystem identity; `deep` adds lifecycle modes). For a scoped run, seed in-scope surfaces for assignment and retain discovered excluded surfaces as `out_of_scope` units so later full runs can turn them into current work. + +Each dimension has a human label and a stable source-derived value in `canonical_refs`. Use the same canonical reference for the same source object across runs even if its display label changes. Suitable references include a repository-relative entry path plus exported scope, a route or message identity defined in source, the source control that defines a boundary, a repository package path, and the exact attack-class block reference. A block reference is `FILE.md#` plus the exact class name as written in bold or as a heading in that file — a stable identifier matched against the file text, not a rendered HTML anchor. For companion section blocks, use the heading text before any parenthetical qualifier (for example `Core discipline`). Do not derive references by lowercasing or slugging display labels. + +Derive `coverage_id` without lossy slugs: + +1. Require every reference to be Unicode NFC with valid scalar values, visible content, no control, format, line/paragraph separator, or default-ignorable code point, and no surrounding whitespace. +2. Encode its UTF-8 bytes with RFC 3986 percent encoding: leave only `A-Z a-z 0-9 - . _ ~` unescaped and use uppercase `%HH` for every other byte. +3. Join encoded `surface`, `boundary`, `subsystem`, and `attack_class` references with `::`; append encoded `lifecycle` when present. + +Use the fixed canonical value `profile/quick/all-in-scope-subsystems` for the quick profile's coarsened subsystem dimension. Do not include wave number, agent, verdict, severity, or line number in a reference or ID. Sort units lexicographically by `coverage_id` before each assignment. Fail on every duplicate ID. If duplicate IDs have different semantic fields, treat that as a canonical identity collision; never merge or silently overwrite them. The validator also rejects one semantic tuple represented by different canonical references. + +Each unit records: + +```json +{ + "coverage_id": "...", + "canonical_refs": { + "surface": "src/router.ts#POST /users/:id", + "boundary": "src/authz.ts#requireOwner", + "subsystem": "packages/api", + "attack_class": "ATTACK-CLASSES.md#Access control" + }, + "surface": "...", + "boundary": "...", + "subsystem": "...", + "attack_class": "...", + "starting_paths": ["repo/relative/path"], + "ordinary_attack_class_block": "ATTACK-CLASSES.md#Access control", + "selected_companion_blocks": ["FILE.md#section"], + "excluded_blocks": [{"block": "FILE.md#section", "reason": "..."}], + "prior_status": "new|prior_confirmed_same_source|prior_confirmed_changed_source|prior_needs_validation|prior_deferred|prior_blocked|prior_out_of_scope|prior_covered_same_source|prior_covered_changed_source|prior_rejected_claim_changed|none", + "attempts": [], + "wave": 1, + "status": "planned", + "agent_id": null, + "reviewed_paths": [], + "local_checks": [], + "result_fingerprints": [], + "unresolved": [] +} +``` + +When `lifecycle` is material, add both `canonical_refs.lifecycle` and a human `lifecycle` field. `ordinary_attack_class_block` is null only when no ordinary block applies. The selected companion list includes each applicable class plus its companion `Core discipline`, `Universal moves`, and `Validation rules`; `excluded_blocks` records every considered but unselected block and the source fact that excludes it. + +The parent may add bookkeeping fields but keeps the semantic fields above stable. In `prior_status`, `new` marks a surface first seen in this run when compatible prior ledgers exist; `none` marks a unit seeded when no compatible prior ledger is available. Prior `deferred`, `blocked`, `out_of_scope`, and changed-source units initialize as current `planned` work when now in scope. A prior same-source covered unit remains visible in the current ledger; assign changed source, important lifecycle paths, and exact conflicts first, then use the coverage critic to decide whether it needs another pass. + +`attempts` is an append-only archive for evidence-bearing assignments that a coverage critic reopens. Before reassignment, append the prior unit's exact `wave`, `status`, `agent_id`, `reviewed_paths`, `local_checks`, `result_fingerprints`, and `unresolved`, plus the critic's source-backed `reassignment_reason`. Only `blocked`, `covered`, and `candidate` states can be archived. Archived attempts retain the same state and evidence invariants as live units, use strictly increasing waves below the current wave, and retain their producing owners and artifacts. The next assignment increments `wave`, uses a fresh owner, and starts with empty live evidence. If the profile or budget prevents another assignment, increment `wave` and use live `deferred` state with null owner, empty evidence, and the stop reason. Never copy an archived owner's checks or artifacts into the live state. A later live terminal state contains only the new attempt's evidence; the archive remains unchanged. + +Enforce this state table exactly: + +| Status | Unit `agent_id` | `reviewed_paths` / `local_checks` | `result_fingerprints` | `unresolved` | +|---|---|---|---|---| +| `planned` | null | empty | empty | empty | +| `not_applicable`, `out_of_scope`, `deferred` | null | empty | empty | nonempty reason | +| `in_progress` | canonical owner | empty | empty | empty | +| `blocked` | canonical owner | both nonempty owned partial evidence | empty | nonempty blocker | +| `covered` | canonical owner | both nonempty | empty | empty | +| `candidate` | canonical owner | both nonempty | nonempty | optional | + +Canonical agent IDs match `^[a-z0-9][a-z0-9_-]{0,63}$` and are not Windows device names. Lowercase is mandatory, so one ledger cannot contain case-fold aliases. The unit `agent_id` records the assignment owner. Every check records its own `agent_id` and nonempty `reviewed_paths`; the unit-level `reviewed_paths` is exactly their union. A source-only check uses `artifact: null`. A local check requires a regular file promoted only by trusted parent-side code under exactly `agents/<check.agent_id>/artifacts/`. This lets hunter and verifier checks coexist in one unit. Scratch paths, output-root files, symlinks, special files, and another check owner's artifacts are not evidence. + +The ledger is the coverage claim. An architecture summary, agent count, or generic "auth reviewed" sentence is not coverage evidence. Phase 2 closes units only from the paths and checks in a hunter's structured result. + +Run `node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json` after seeding, after every parent update, and before Phase 6. The validator rejects input beyond 5 MiB, 64 nesting levels, 10,000 units, 1,000 entries in a nested collection, or 500,000 traversed values, and caps reported validation errors at 100. In practice the 5 MiB byte limit holds roughly 2,000-5,000 realistic units, so it binds before the 10,000-unit cap. Fix every error before assigning work or making a coverage claim. diff --git a/skills/thirdshift-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md b/skills/thirdshift-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md new file mode 100644 index 00000000..6cbbd1e2 --- /dev/null +++ b/skills/thirdshift-security-audit/RESOURCE-EXHAUSTION-AND-AVAILABILITY.md @@ -0,0 +1,78 @@ +# Resource Exhaustion and Availability Hunting + +#### When to use this file + +Reach for this file when untrusted requests, messages, files, tenant state, or agent work can consume CPU, memory, disk, connections, worker slots, paid APIs, or queue capacity, or can deadlock/crash a shared service. This domain distinguishes a source-reviewable availability vulnerability from a general performance issue. Never validate by stressing a shared or live service. + +Use `MEMORY-SAFETY-AND-BINARY.md` for memory-integrity defects and `PROTOCOLS-RPC-AND-MESSAGING.md` for broker delivery logic. A reachable fatal error belongs here for shared impact even when the underlying parser is covered elsewhere. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Require an input-to-cost path, a missing effective bound, and impact on another user, shared service, safety function, or operator-owned spend. Self-limiting work in the requester's own process is not a service vulnerability. +- A missing rate limit is not enough. Check body/message/file caps, concurrency, queues, deadlines, database constraints, upstream gateways, and per-tenant quotas before calling a path unbounded. +- Do not run stress, saturation, or production tests. Use asymptotic analysis, small boundary fixtures, mocked paid calls, strict local resource limits, and deterministic cancellation tests. +- State attacker cost, service work, persistence, scope, and recovery. One bounded input with superlinear or persistent shared effect is materially different from sustained volume. +- Use `confirmed` for source-visible bounds failures demonstrated safely. Use `needs_validation` when upstream caps, deployed topology, autoscaling, paid quota, or recovery behavior is outside the repository. +``` + +## Computational amplification attack classes (subagent_type: `general`) + +**Superlinear parsing, matching, or evaluation** +Small accepted input drives catastrophic regex backtracking, nested parsing, recursive validation, symbolic evaluation, graph traversal, template expansion, or adversarial sort/hash behavior. Derive accepted depth/cardinality and complexity, then demonstrate a bounded growth curve locally. + +**Decompression and representation amplification** +Compressed, sparse, nested, aliased, or encoded input expands far beyond the checked transfer or file size. Verify limits after every expansion and across parser stages, including archives, images, fonts, structured documents, and protocol compression tables. + +**Database and downstream query amplification** +A small request creates broad scans, pathological joins, fan-out, unbounded sort/aggregation, or many downstream calls because query depth, filter cardinality, pagination, or expansion fields are not bounded. Confirm authorization does not intentionally permit the same resource scope. + +## Resource accumulation attack classes (subagent_type: `general`) + +**Unbounded buffering and cardinality** +Bodies, out-of-order streams, uploads, sessions, unique cache keys, metrics labels, log fields, subscriptions, or pending jobs accumulate without per-item and aggregate limits. Find cleanup and expiration on disconnect, timeout, cancellation, and partial parse. + +**File descriptor, handle, and temporary-resource leaks** +Malformed or canceled work misses cleanup and retains sockets, files, database cursors, timers, subprocesses, temporary files, or object references. Confirm the leak repeats through bounded local iterations and affects a shared pool. + +**Detached work after cancellation** +Client timeout, disconnect, canceled job, or failed authorization returns control but leaves database, model, network, or worker work running. Trace cancellation and deadline propagation through every layer. + +## Quota and scheduling attack classes (subagent_type: `general`) + +**Pre-authentication work imbalance** +Expensive parsing, key lookup, cryptography, decompression, or external requests happen before authentication and the earliest size/rate gate. Compare minimal requester effort to shared service cost and check upstream limits. + +**Quota-accounting scope and reset gaps** +Accounting uses attacker-influenceable IP, route, tenant, key prefix, task ID, or other dimension, allowing one principal's work to escape its intended budget or consume another principal's allocation. Review integer overflow, distributed races, retries, reconnects, and account switching. + +**Worker, pool, and priority starvation** +Low-priority or attacker-controlled jobs hold shared locks, workers, database pools, event-loop turns, or scheduler priority needed by unrelated users. Require a path that bypasses queue/concurrency fairness or retains a slot beyond its deadline. + +## Failure and recovery attack classes (subagent_type: `general`) + +**Reachable fatal error or deadlock** +An untrusted input reaches `panic`, abort, fatal assertion, unhandled exception, process exit, lock cycle, or infinite loop in a shared process. Confirm supervisor scope and whether one worker or the whole service becomes unavailable. A restarted isolated worker may reduce impact but does not erase the defect. + +**Retry storm and fail-open amplification** +Timeouts, dependency errors, partially processed messages, or health-check failures trigger synchronized or unbounded retries without jitter, ceilings, circuit breaking, or deduplication. Verify one bounded failure source can create persistent aggregate work. + +**Poison-record and head-of-line blocking** +One malformed record or message repeatedly fails at the front of a shared queue, partition, startup scan, migration, or recovery loop. Review skip/quarantine policy, offsets, and whether other tenants share the blocked unit. + +**Unsafe recovery and capacity rollback** +A restart, restore, fallback, or cleanup path rebuilds unbounded state, ignores current quotas, or restores the input that immediately repeats failure. Recovery correctness is part of availability. + +## Universal moves (apply across the above) + +- Build an input-to-resource table: earliest accepted size/cardinality, work before auth, downstream fan-out, persistence, shared pool, limit and cleanup owner, recovery. +- Compare aggregate limits with per-object limits. Ten thousand valid one-byte items may evade a per-message cap while exhausting tenant-wide or process-wide state. +- Validate only in an isolated fixture with strict CPU/memory/time limits and small growth points. Mock external and paid calls and stop once the missing bound or cancellation is observable. + +## Validation rules (apply before reporting ANY finding here) + +1. Name untrusted input, requester work, service amplification or retained resource, shared blast radius, and recovery. Missing limits without concrete shared impact are hardening. +2. Confirm no source-visible upstream, parser, queue, tenant, or framework bound prevents the path. Unknown deployed controls require `needs_validation`. +3. For superlinear behavior, establish the accepted complexity and bounded local growth. For leaks, show repeatable retention after cleanup should occur. For fatal paths, identify process/supervisor isolation. +4. Prioritize by low requester work, unauthenticated reachability, cross-tenant scope, persistence, and poor recovery; do not validate with availability impact. +5. Return `confirmed` only with safe local proof and meaningful shared effect. Return `needs_validation` with the exact upstream limit, topology, quota, or recovery observation an owner must check. diff --git a/skills/thirdshift-security-audit/SKILL.md b/skills/thirdshift-security-audit/SKILL.md new file mode 100644 index 00000000..b4522fef --- /dev/null +++ b/skills/thirdshift-security-audit/SKILL.md @@ -0,0 +1,192 @@ +--- +name: thirdshift-security-audit +description: Security guidance and vulnerability review for codebases, APIs, services, CLI tools, libraries, and daemons. Use for security questions, focused reviews, vulnerability research, security audits, or pen tests. Run the complete workflow only for explicit codebase audit or pen-test requests, full/comprehensive/end-to-end reviews, or requested report artifacts. +--- + +# Security Audit + +Find vulnerabilities that violate a real trust boundary, then give owners the source evidence, safe reproduction, priority, and smallest effective fix. This is a defensive, source-first workflow. A candidate without a concrete affected principal, resource, or security outcome is not a confirmed finding. + +## Operating modes + +This skill is guidance by default. Loading it does not authorize the complete audit workflow or file creation. + +- **Guidance mode**: For security questions, focused reviews, methodology, triage, or investigation of specific findings, use only the relevant parts of this skill. Do not automatically run all six phases, create an output directory, or write audit artifacts. You may launch focused agents when useful; they return results to the current task. +- **Full audit mode**: Use the complete workflow when the user explicitly asks to audit or pen-test a codebase, asks for a full, comprehensive, or end-to-end security review, or requests report artifacts. Run all six phases and write the files defined below. + +If the request could mean either mode, ask one focused question before creating files or starting the complete workflow. + +## Platform terminology + +This skill is agent-neutral: + +- **Parent** is the agent that coordinates the run and owns shared state. +- **Task tool** is the platform's delegation or sub-agent mechanism. +- **`research` agent** is a delegated agent for focused source exploration and factual verification. +- **`general` agent** is a delegated agent for broad investigation and bounded local execution. +- **`subagent_type:`** in a heading names which of these two delegated agent roles runs that work. + +Use equivalent platform capabilities while preserving role, write-isolation, prompt, and independence boundaries. + +## Universal execution safety + +These rules apply in both operating modes. Source inspection is read-only. Run target-controlled builds, tests, processes, browsers, emulators, fuzzers, and fixture processing only inside an OS-enforced sandbox that provides all of these controls: + +- no external network; use only an isolated loopback namespace when the check needs local client/server traffic; +- an empty environment populated from an explicit allowlist with safe values, with scratch-local `HOME`, temporary directories, and caches; +- a read-only target and toolchain, with the target-controlled process able to write only inside its assigned `scratch/` directory; and +- explicit low CPU, memory, process, file-size, disk, and wall-clock limits. + +The agent, outside the target-controlled process, may make a disposable source copy in an assigned `scratch/` directory when a build must write beside source. In guidance mode, do not retain target-controlled files. In full audit mode, only trusted parent-side code may promote the minimum non-secret result to retained `artifacts/` using the procedure under Write isolation. Never expose a retained output directory (other than the agent's own assigned `scratch/`), another agent's directory, the host home directory, credentials, sockets, or shared services to target code. Do not install dependencies or let builds fetch them. Use only tools and dependencies already available locally. If every control cannot be enforced, do not execute target code: report the missing sandbox capability as a needs-validation blocker and give a safe validation plan. + +Use dummy principals, fixtures, and secrets. Do not probe deployed endpoints, external services, shared infrastructure, production identities, other users' data, or live control planes. Do not test availability against a live or shared process, publish artifacts, alter releases, spend paid API quota, or continue beyond the minimum local effect needed to establish a defect. If the decisive fact is outside source or the sandboxed fixture, report it as needing validation. + +## Full audit setup + +In full audit mode, resolve these values before reconnaissance: + +- **Skill directory**: the absolute directory containing this `SKILL.md`. +- **Target**: the absolute repository root under review. +- **Repo name**: a stable repository identifier from the directory or local Git remote. +- **Output directory**: a new writable directory outside the target, defaulting to `~/security-audit-skill/<repo-name>/run-<N>`, where `<N>` is the next unused integer. Use a directory inside the target only when the user explicitly selects it and the parent verifies that version control ignores the whole directory. Otherwise stop and request an external path. +- **Source ref**: the reviewed commit and whether the worktree is dirty. Do not treat unreviewed generated or modified files as another revision. + +### Write isolation + +The parent creates and is the only writer of shared run files: + +- `run-metadata.json` +- `architecture.md` +- `coverage-ledger.json` +- `findings.json` +- `REPORT.md` +- `FINDINGS-DETAIL.md` +- `NEEDS-VALIDATION.md` + +Each hunter or verifier receives a unique root under `<output-dir>/agents/<agent-id>/`, with separate `scratch/` and `artifacts/` directories. Canonical agent IDs match `^[a-z0-9][a-z0-9_-]{0,63}$` and must not equal a Windows device name such as `con`, `prn`, `aux`, `nul`, `com1` through `com9`, or `lpt1` through `lpt9`. Lowercase IDs prevent case-fold collisions. The agent and every target-controlled process may write only to `scratch/`; retained `artifacts/` is parent-owned, is never exposed to the sandbox, and is writable only by trusted parent-side promotion code. Agents may not change shared files, target source, retained artifacts, or another agent's directory. Do not use `/tmp` or the host home directory as a writable fallback. + +Before execution, the parent opens and retains trusted, non-inheritable directory descriptors for the agent's `scratch/` and `artifacts/` roots, and records an allowlist of expected scratch-relative artifact files plus explicit per-file and cumulative byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox and all its processes terminate, trusted parent-side code promotes each allowlisted file separately: + +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, type, link count, or size. +7. For the destination, walk every parent component from the retained artifacts-root descriptor with no-follow directory-relative operations; require each existing component to be a real directory, and create any missing directory exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened destination is a regular file with link count exactly one, and copy from the verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or open or promote a symlink, FIFO, socket, device, directory, hard-linked file, changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch entry; if it is decisive evidence, retain `needs_validation` with the exact promotion blocker. + +[HUNTING.md](HUNTING.md) and [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md) carry this procedure as one identical fenced block for hunter and verifier prompts; it states the same rules in the same order as this list. + +For a reproduced check, record the command, exact test input, sandbox limits, and only the allowlisted environment variable names plus safe non-secret values needed to reproduce it. Never capture or copy the ambient environment, inherited variables, credential values, authentication state, or unrelated host paths. Launch from an empty environment rather than trying to redact one after execution. + +Before delegation, the parent writes `run-metadata.json` with at least `run_id`, `repo`, `target`, `source_ref`, `profile`, `scope_paths`, `budget` (null if unset), `execution_policy: "sandboxed-source-and-local-only"`, selected companion files, prior-run paths, shared-file owners, and `run_status: "in_progress"`. Update metadata only when those facts change; candidate state belongs in the coverage ledger and `findings.json`. + +## Full audit planning + +The coverage, prior-run, profile, and budget requirements in this section apply only in full audit mode. + +### Coverage and prior runs + +No one pass is complete. Build a deterministic coverage plan before hunting and update it after every agent result. [RECONNAISSANCE.md](RECONNAISSANCE.md) defines the stable coverage units and [HUNTING.md](HUNTING.md) defines coverage-critic waves. The parent alone updates the ledger. + +If prior runs exist, read every compatible `coverage-ledger.json` and `findings.json` before planning the current run: + +1. Compare the relevant current source with each prior record and unit. A prior source ref alone is not evidence that a path is unchanged. +2. Carry a prior `confirmed` record into the current candidate set only when its relevant source and conditions are unchanged and its evidence still meets the current contract. Link it to a current ledger unit seeded `planned`, preserve its fingerprint, exclude only that carried root cause from hunters, and send the carried record through the current final verification path; the Phase 3 verifier that re-checks it becomes that unit's assignment owner and moves it to `candidate`. +3. When relevant source for a prior `confirmed` record changed, create a current planned revalidation unit. Do not put that record on the hunter exclusion list. It remains confirmed only if current independent validation establishes the current path and result. +4. Make prior `needs_validation`, `deferred`, `blocked`, `out_of_scope`, and any changed-source unit current work. A still-external `needs_validation` record may be carried only after the current source trace is checked and linked by fingerprint to a current `planned` unit whose verifier re-check supplies its owner and evidence; the record keeps the unresolved blocker. These prior states never suppress a current unit. +5. A prior same-source covered unit may inform priority, but it remains visible in the current ledger. A prior `rejected` record suppresses only the unchanged failed claim, not coverage of its unit; changed evidence creates current work. +6. Read the prior profile and scope. A prior `quick` or scoped ledger contributes only its recorded evidence and gaps, never an implied "rest is fine." + +If no prior ledger exists, say so in the final coverage statement. Never imply that one run exhausts the target. + +### Run profiles and scope + +During full audit setup, pick a profile from the user's request or propose one from the target's size and stakes. Record it in `run-metadata.json` (`profile`, `scope_paths`) and state it in the report. The default is `standard`. + +- **`quick`** — a bounded pass for small targets, re-runs, or a fast first look. Coarsen ledger units to surface × boundary × attack class (subsystem uses the fixed canonical `profile/quick/all-in-scope-subsystems` identifier), run exactly one hunter wave followed by exactly one final coverage-critic pass, and use one fresh verifier per candidate for both candidate validation and final record verification. Do not launch a follow-up hunter wave: record the critic's accepted discoveries and reassignments as `deferred`. +- **`standard`** — the workflow as written. +- **`deep`** — for high-stakes or large targets. Split ledger units per subsystem and lifecycle mode, run critic waves to a clean pass, keep candidate validation and final record verification as separate fresh agents, and give `prior_covered_same_source` units an independent second pass. + +A **scoped run** audits a subset: named paths, one subsystem, one companion domain, or the diff between two source refs. Seed ledger units only for in-scope surfaces and record everything else as `out_of_scope` — never as `covered`. A scoped or `quick` run must present itself as partial coverage. + +Profiles change breadth and redundancy, never the evidence bar. Do not scale away the candidate gate, the source/local execution boundary, `needs_validation` discipline, schema validation, or independent verification of `confirmed` records. + +#### Cost budget + +The ledger makes spend countable: one unit is roughly one hunter assignment, and one surviving candidate is one or two verifier assignments depending on profile. When the user sets a budget — or the parent proposes one for a large target — record `budget` in `run-metadata.json` as a maximum number of agent invocations across all phases. + +Apply the strict budget gate before launching any reconnaissance agent. Reserve the four baseline reconnaissance calls, one final post-wave critic for `quick` or one post-wave plus one distinct final-clean critic for `standard`/`deep`, and at least one verifier call. Add focused reconnaissance only after repeating this gate for each extra call. If the requested budget cannot fund that minimum, launch no agent: ask for a larger budget, narrower scope, or different profile. If the request remains unchanged, set `run_status: "incomplete"` with `incomplete_reason: "budget_cannot_fund_reconnaissance_and_reserves"` and report that no audit pass ran. + +Spend it in this order: + +1. Count reconnaissance, every post-wave critic, and the separate final-clean critic as agent invocations. +2. **Reserve critics and validation before hunting.** For `quick`, reserve its one post-wave final critic. Before every `standard` or `deep` hunter wave, reserve one immediate post-wave critic plus one distinct final-clean critic. Also reserve verifier cost from the profile (about 1 or 2 agents per expected candidate; when in doubt reserve 30% of the balance after critic reservation). Never assign hunters into either reserve. +3. Assign hunters to units in priority order until the hunting allowance is spent. Spend the reserved post-wave critic immediately after that wave; keep the final-clean and validation reserves intact. +4. Before a later wave, reserve its new post-wave critic again. If the remaining budget cannot cover the required critic calls and validation reserve, launch no hunters from that wave, mark its planned units `deferred` with reason `budget_cannot_reserve_critics_and_validation`, and use the retained final-clean critic to record the resulting gap. + +Before wave 1, update the pre-recon estimate with seeded units, implied hunter count, mandatory critic calls, validation reserve, and whether the remaining budget covers the plan. If it clearly cannot, say so and propose either a tighter scope or a coarser profile instead of silently thinning evidence. If later facts consume the required final-critic reserve, launch no hunters, mark all planned work deferred, set the run incomplete with reason `critic_budget_exhausted`, and make no complete-coverage claim. + +A strict total-agent budget can still be exceeded by an unexpectedly large candidate set or by a material Phase 5 replacement that needs another independent verifier. If the remaining budget cannot validate every candidate, stop hunting, validate candidates in fingerprint order while the budget permits, and set `run_status: "incomplete"` plus `incomplete_reason: "validation_budget_exhausted"`. Keep each unvalidated fingerprint linked to a `candidate` ledger unit with that unresolved reason. Do not put an unvalidated candidate in `findings.json`, relabel it `needs_validation`, or report the run as complete. Phase 6 may produce a partial report only if its first section states that candidate validation is incomplete and lists the affected fingerprints and units. Never exceed a user-set strict budget silently. + +## Core principles + +### Require a boundary and result + +For every candidate, name the lower-trust principal, accepted input or action, intended control, crossed boundary, affected principal or resource, and concrete observed or owner-observable result. Do not elevate a missing best practice, guessed deployment behavior, generic parser crash, or self-impact into a security finding. + +### Use bounded local evidence + +Static analysis establishes the source path. Sandboxed local tests resolve behavior when all execution controls are available: a minimal function harness, existing unit test, small parser fixture, dummy-tenant integration test, locally rendered configuration, or bounded isolated-loopback client. Stop at a wrong return value, unauthorized dummy record, sanitizer finding, policy difference, or other minimum effect. Do not extend the local check beyond the minimum boundary result or produce persistence, post-fault, or concealment material. + +### Respect source visibility + +Deployment controls, proxy behavior, provider settings, browser headers, identity policy, broker ACLs, packaging, and topology are real controls. If they are required and absent from the repository, do not assume either presence or absence. Use `needs_validation` with the exact missing fact and a safe owner-observed or local plan. + +### Separate priority from certainty + +Only `confirmed` records receive severity. Likelihood and impact must reflect the demonstrated conditions and result; overall severity cannot exceed demonstrated impact. `needs_validation` means a specific source-grounded boundary hypothesis is blocked, not a low-confidence confirmed vulnerability, and it has no severity. + +Calibrate overall severity with these anchors: + +- **critical** — an unauthenticated actor gains code execution, full data-store access, or takeover of arbitrary accounts. +- **high** — an actor fully defeats an explicit security control with real consequences: authentication bypass, cross-tenant read or write, stored script execution affecting other users, authenticated code execution, or an unauthenticated remote stop of a shared service. +- **medium** — a real boundary violation with limited blast radius, uncommon preconditions, or consequences confined to a narrow resource set. +- **low** — disclosure of non-secret internals, or an effect requiring sustained effort for minimal gain. +- **informational** — a confirmed but minimal-impact observation, useful mainly as a prerequisite inside a larger finding. + +The high/medium discriminator: does the demonstrated result fully defeat an explicit control for an action with real consequences, or only weaken it? If you cannot state the concrete damage, the severity is lower than it feels. + +### Recommend the smallest effective source fix + +For each confirmed finding, identify the invariant the code must enforce and the narrowest source change that enforces it at the last trusted decision point. Prefer specific repository-relative changes and regression tests over generic hardening advice. The audit describes fixes; it does not modify target source. + +## Full audit workflow + +In full audit mode, follow all six phases in order: + +1. **Reconnaissance** — map the source, trust boundaries, local build paths, companion selections, prior evidence, and initial deterministic coverage ledger with [RECONNAISSANCE.md](RECONNAISSANCE.md). +2. **Coverage-led hunting waves** — assign isolated hunters from the ledger and collect structured candidate results with [HUNTING.md](HUNTING.md), [ATTACK-CLASSES.md](ATTACK-CLASSES.md), and the selected domain companions. +3. **Candidate validation** — consolidate fingerprints and give every candidate to a fresh source verifier as defined in [VALIDATION-AND-REPORTING.md](VALIDATION-AND-REPORTING.md). +4. **Structured output** — write all final `confirmed`, `needs_validation`, and `rejected` records to `findings.json`; validate it with `report-schema.json` and `validate-findings.cjs`, and validate the coverage claim with `validate-coverage-ledger.cjs`. +5. **Independent record verification** — use fresh agents to verify final source claims and reconcile corrections or state changes. +6. **Target-neutral report** — derive `REPORT.md`, `FINDINGS-DETAIL.md`, and `NEEDS-VALIDATION.md` from the final records, with no live-probe instructions. + +Do not end the run before one of exactly two terminal states: (a) all Phase 6 artifacts are written and both validators pass, or (b) `run_status: "incomplete"` is recorded with its exact reason and the gap is disclosed in the report. Never stop mid-phase. + +## Anti-patterns + +1. Checklist deviations presented as vulnerabilities. +2. Defense-in-depth advice with no reachable boundary violation. +3. Live or shared-environment testing where bounded local evidence is insufficient. +4. Guessing provider, proxy, browser, identity, or deployment behavior not present in source. +5. Treating intended same-principal authority or self-impact as a cross-boundary result. +6. Reporting a parser or runtime effect stronger than the observed effect. +7. Emitting prose-only hunter results that cannot be deduplicated or verified. +8. Re-reporting carried same-source prior confirmed records or using them as exemplars that anchor the hunt. +9. Assigning severity to `needs_validation` records. +10. Writing the report before independent verification or letting prose and JSON disagree. diff --git a/skills/thirdshift-security-audit/SUPPLY-CHAIN-AND-RELEASE.md b/skills/thirdshift-security-audit/SUPPLY-CHAIN-AND-RELEASE.md new file mode 100644 index 00000000..bf96aed0 --- /dev/null +++ b/skills/thirdshift-security-audit/SUPPLY-CHAIN-AND-RELEASE.md @@ -0,0 +1,73 @@ +# Supply Chain and Release Hunting + +#### When to use this file + +Reach for this file when the target resolves dependencies, builds from untrusted contributions, runs CI, creates release artifacts, signs or promotes builds, loads plugins, or updates deployed software. This domain covers trust handoffs from source and dependency to the artifact a user runs. Use `MEMORY-SAFETY-AND-BINARY.md` for flaws inside a local binary loader and `CLOUD-AND-DEPLOYMENT.md` for runtime workload authority. + +Split large targets into dependency resolution, CI isolation, artifact provenance, release authorization, and updater/plugin trust. + +## Core discipline (include in every agent prompt for this domain) + +``` +- A mutable or known-vulnerable dependency is not a finding by itself. Show who can influence resolution, which build consumes it, and what execution or release boundary follows. +- Follow integrity across every handoff: source identity, resolved inputs, build worker, artifact identity, test result, signature/attestation, promotion, and update consumer. +- CI configuration is authorization code. Establish which event triggered a workflow, whose code runs, which secrets and tokens exist, and what it may publish or mutate. +- A checksum fetched from the same untrusted location as the artifact does not establish independent integrity. Identify the trusted root and failure behavior. +- Use `confirmed` for in-repo control-flow failures with bounded local validation. Use `needs_validation` for branch protection, hosted-runner, registry, signing-service, or production promotion facts that are not observable. +``` + +## Dependency and build-input attack classes (subagent_type: `general`) + +**Dependency source and namespace confusion** +Resolver configuration can select an unintended public/private namespace, fallback registry, mirror, repository, or source URL. Review package names, source priority, lockfile and checksum use, alternate build files, platform-specific resolution, and first-install versus update behavior. + +**Mutable and unbound build inputs** +Builds consume branches, tags, unverified submodules, downloaded tools, generated assets, remote includes, floating CI actions, or container tags whose content can change without source review. Require a lower-trust writer and a path into trusted build output; reproducibility by itself does not prove authenticity. + +**Generated-source and codegen provenance gaps** +Schemas, vendored archives, generated clients, localization, documentation examples, or binary blobs produce executable or shipped content without the same review and integrity gate as source. Compare local regeneration with committed output and verify who controls input and generator. + +**Build-context inclusion** +Secrets, local configuration, repository metadata, test fixtures, or developer artifacts enter a package or image because the build context and ignore rules exceed intended release inputs. Confirm that the resulting artifact exposes a real credential, private data, or privileged configuration. + +## CI and automation attack classes (subagent_type: `general`) + +**Untrusted code in a privileged workflow** +A pull request, issue comment, fork, dependency update, or external event runs contributor-controlled code with protected secrets, write tokens, deployment authority, or a trusted runner. Compare trigger type, checkout ref, approval gate, environment protection, and permission narrowing. Do not assume repository-host defaults that are not in source. + +**Workflow command and expression confusion** +Attacker-controlled branch names, commit messages, issue fields, artifact names, matrix values, or generated output enter shell commands, template expressions, paths, or privileged workflow inputs without canonical validation. + +**Cache, artifact, and workspace trust mixing** +A lower-trust job can populate a cache, artifact, shared workspace, or output that a higher-trust job later restores and executes or releases. Review cache keys and namespaces, artifact producer identity, digest binding, retention, and whether promotion re-resolves by mutable name. + +**Automation identity overreach** +CI jobs receive permissions beyond the operation, repository, environment, or duration needed, and untrusted job inputs can select the affected resource. Missing least privilege alone is hardening; require a reachable privileged action. + +## Release and update attack classes (subagent_type: `general`) + +**Build-to-promotion substitution** +Tests, review, signature, and publication refer to mutable tags, filenames, channels, or artifact IDs rather than the same immutable digest. Check every copy, repack, architecture merge, and provenance step between build and release. + +**Release authorization and signing-policy gaps** +A release or signature is accepted from the wrong workflow, repository, branch, environment, key role, or threshold. Review identity claims inside attestations and verify the consumer validates them, not just a valid signature. Rotation, expiry, and revocation must fail closed where policy requires. + +**Update metadata and rollback confusion** +An updater authenticates payload bytes but not version, product, platform, channel, target path, expiry, or rollback state, or it accepts metadata and payload from different authorized transactions. Verify atomic installation and recovery behavior. A signature API call without policy binding is incomplete. + +**Plugin and extension trust expansion** +An extension package gains host authority beyond its declared scope, a lower-trust publisher can replace another publisher's identity, or install/update hooks run before authenticity and capability checks. Intended installation of arbitrary same-user plugins is not a privilege boundary. + +## Universal moves (apply across the above) + +- Walk backward from a released digest or installed update to every source, generated input, credential, worker, cache, test result, and authorization decision. +- Compare untrusted and protected workflow events side by side. Mark each persisted channel crossing between them and require an immutable identity plus producer trust. +- Review revoked key, failed download, missing attestation, partial platform release, rollback, and registry outage paths. The failure policy is part of release integrity. + +## Validation rules (apply before reporting ANY finding here) + +1. Name the lower-trust actor, controllable source/cache/artifact/metadata, consuming trusted job or updater, and resulting unauthorized publication, code inclusion, secret disclosure, or privileged execution. +2. Prove artifact identity across the broken handoff. A different mutable name or unbound digest must reach a real consumer. +3. Verify built-in package-manager, repository-host, registry, and signing defaults for the pinned version. Unknown hosted controls require `needs_validation`. +4. Keep local validation bounded: use a harmless fixture repository, dummy credential marker, local registry/config, and non-production artifact namespace. Do not publish or alter a real release. +5. Return `confirmed` only with a complete source-visible handoff and meaningful result. Return `needs_validation` with the precise branch, runner, registry, signing, or deployment fact an owner must observe. diff --git a/skills/thirdshift-security-audit/VALIDATION-AND-REPORTING.md b/skills/thirdshift-security-audit/VALIDATION-AND-REPORTING.md new file mode 100644 index 00000000..5e200d73 --- /dev/null +++ b/skills/thirdshift-security-audit/VALIDATION-AND-REPORTING.md @@ -0,0 +1,186 @@ +# Validation, Structured Output, Verification, and Reporting + +### Phase 3: Independently validate every candidate + +After the clean coverage-critic pass or an explicitly recorded early stop, consolidate Phase 2 candidates and carried same-source prior confirmations by stable fingerprint and root cause. Give every unique proposed `confirmed` and `needs_validation` candidate to a fresh `general` verifier that did not hunt it. A carried prior confirmation follows the same current verification path even though hunters exclude that unchanged root cause. A verifier may read hunter or prior artifacts but must re-read every cited current source location and independently run any decisive check it can reproduce safely. + +Assign each verifier a canonical lowercase unique ID and `<output-dir>/agents/<verifier-id>/scratch/` plus parent-owned `artifacts/`. The verifier writes only to `scratch/` and never writes retained artifacts. It receives only the candidate, its linked coverage-unit checks and artifact paths, architecture facts needed to interpret the path, exact relevant companion validation blocks, the promotion procedure block below, the source/local execution boundary, the `confirmed`, `needs_validation`, and `rejected` branches of `report-schema.json` copied verbatim, and prior records with the same fingerprint. It must not receive another verifier's conclusion. + +#### Candidate-verifier prompt + +```text +You did not write this candidate. Try to refute it from repository source and bounded +local evidence. Do not contact deployed endpoints or external/shared services. Run +target-controlled code only inside the approved OS-enforced sandbox: no external +network, empty allowlisted environment, read-only target and tools, scratch-only +writes, and explicit low resource and wall-clock limits. If any control is unavailable, +do not execute; retain the exact missing capability as a needs_validation blocker. +Treat every scratch entry as target-controlled after execution. After the sandbox and +all its processes terminate, only trusted parent-side code may promote a predeclared +scratch-relative file, following the promotion procedure block included verbatim in +this prompt. You and target code never write retained artifacts. If promotion is +unavailable or fails, do not use that file as evidence. + +1. Verify every trace and evidence file, positive line number, scope, and description. + Confirm the first entry is a real lower-trust entrypoint and the last is the + claimed sink or boundary effect. +2. Reconstruct the strongest source-visible validation, identity, authorization, + normalization, lifecycle, framework, and containment controls on the path. + Where the architecture summary names a comparable baseline, note whether it + shares the pattern — as calibration, never as grounds to dismiss. +3. For a proposed confirmed candidate, independently reproduce the minimum observed + result when possible. Verify inputs, interface shape, conditions, and affected + dummy principal/resource. Do not infer a stronger result or continue after it. +4. Verify that likelihood, impact, confidence, and the proposed source fix match only + what the evidence establishes. +5. For a proposed needs_validation candidate, decide whether the blocker is genuinely + outside source/local observation. If source refutes the trace, reject it. If the + missing fact remains decisive, keep needs_validation and make the local and + owner-observed plans exact and non-destructive. +6. Preserve the fingerprint for the same source-derived root cause across every state. + +Return exactly one JSON object and no surrounding prose: +{"decision": "confirmed|needs_validation|rejected", "record": { ... }} +where record exactly matches the decision's verdict branch of the schema included +in this prompt. A corrected record replaces the hunter's wording. +``` + +Copy this promotion procedure verbatim into every candidate-verifier prompt: + +```text +Artifact promotion procedure (trusted parent-side code only): +Reference only for you: the parent performs these steps; you never perform them. + +Before execution, the parent opens and retains trusted, non-inheritable directory +descriptors for the agent's scratch/ and artifacts/ roots, and records an allowlist +of expected scratch-relative artifact files plus explicit per-file and cumulative +byte limits. Never pass those descriptors to the agent or sandbox. After the sandbox +and all its processes terminate, trusted parent-side code promotes each allowlisted +file separately: + +1. Validate the declared relative path: reject absolute, empty, `.`, `..`, or + symlinked components. +2. Walk each parent component from the retained scratch-root descriptor with + no-follow directory-relative operations; never reopen by path. +3. Open the leaf no-follow and nonblocking. +4. Verify with `fstat` that it is a regular file with link count exactly one and + within the recorded per-file and cumulative byte limits. +5. Enforce those limits again while reading from that descriptor. +6. Copy exactly the verified size, repeat `fstat`, and reject a changed identity, + type, link count, or size. +7. For the destination, walk every parent component from the retained + artifacts-root descriptor with no-follow directory-relative operations; require + each existing component to be a real directory, and create any missing directory + exclusively before reopening and verifying it no-follow. +8. Create the leaf exclusively without following links, verify that the opened + destination is a regular file with link count exactly one, and copy from the + verified source descriptor without reopening either path. +9. Use equivalent race-safe APIs on non-POSIX systems. +10. Never recursively copy or glob scratch, extract an archive into artifacts, or + open or promote a symlink, FIFO, socket, device, directory, hard-linked file, + changing file, or file that exceeds its bound. +11. If any check is unavailable, cannot be enforced, or fails, discard the scratch + entry; if it is decisive evidence, retain `needs_validation` with the exact + promotion blocker. +``` + +A verifier can promote `needs_validation` to `confirmed` only after independently establishing the complete path and bounded observed result. Demote proposed confirmation to `needs_validation` when a specific deployment or runtime fact remains unknown. Use `rejected` when source, local behavior, a visible control, missing meaningful impact, or an impossible prerequisite refutes the claim. `needs_validation` is never a parking place for a speculative idea. + +The parent checks that each verifier returned the same fingerprint unless it identified a genuinely different root cause. Merge corrections, record the decision in every linked coverage unit, and ensure there is one final record per fingerprint. Discard a malformed or prose-wrapped verifier result without repairing it; re-run that candidate with a fresh verifier when the budget permits, otherwise it remains an unvalidated ledger candidate under the incomplete-run rule. + +When verifier evidence updates a ledger check, set that check's `agent_id` to the verifier's canonical ID and list its nonempty repository-relative `reviewed_paths`. Keep the unit-level `reviewed_paths` equal to the union across checks. Use `method: "source"` with `artifact: null` for source-only review. Use `method: "local"` only with a file successfully promoted by trusted parent-side code below `agents/<check.agent_id>/artifacts/`. The unit retains its original assignment owner, so independently owned hunter and verifier checks can coexist. For a carried prior record's seeded `planned` unit there is no prior owner: the verifier that re-checks it becomes the unit's assignment owner, and its re-check is the unit's first check, moving the unit to `candidate` with the carried fingerprint. + +If a strict total-agent budget cannot cover every candidate, set the run status to incomplete and follow the deterministic budget rule in `SKILL.md`. An unvalidated candidate remains only in the ledger. It does not enter `findings.json` under any verdict. + +### Phase 4: Write and validate `findings.json` + +The parent writes all independently decided records to `<output-dir>/findings.json`, sorted by fingerprint. Include: + +- `confirmed`: source-grounded vulnerabilities with complete local execution evidence, conditions, specific remediation, likelihood/impact/overall severity, and confidence. +- `needs_validation`: source-grounded candidates with an exact unresolved blocker and at least one applicable local or owner-observed deployment plan. +- `rejected`: source-grounded candidates disproved during validation, retained so future runs do not repeat the unsupported claim without changed evidence. + +Read `report-schema.json` immediately before writing. It uses `additionalProperties: false`; do not carry hunter wrapper fields into a record. Keep these verdict contracts distinct: + +- A `confirmed` record uses `root_cause`, `intended_behavior`, `conditions`, `execution`, `remediation`, `severity`, and `confidence`. It must not use `claimed_root_cause`, `blockers`, `validation_plan`, or `reason`. `execution` is target-neutral and uses the target's native interface: API/HTTP input, CLI call, library call, message, file fixture, browser action, rendered policy, or local harness as applicable. `observed_result` is nonempty and factual. +- A `needs_validation` record uses `claimed_root_cause`, `trace`, `evidence`, `blockers`, and at least one nonempty `validation_plan.local` or `validation_plan.deployment` field. Include both only when both contexts can resolve distinct facts. It must not use severity, execution, remediation, reason, or confirmed root cause. +- A `rejected` record uses `claimed_root_cause`, `trace`, `evidence`, and `reason`. It must not use severity, execution, remediation, blockers, validation plan, or confirmed root cause. + +Every record has a stable fingerprint, title, description, and repository-relative source paths. A multi-step trace begins with `entrypoint`, ends with `sink`, and uses `propagation` only between them. One-entry traces use `entrypoint` or `sink`. Overall severity cannot exceed demonstrated impact. + +Run: + +```sh +node <skill-dir>/validate-findings.cjs <output-dir>/findings.json +node <skill-dir>/validate-coverage-ledger.cjs <output-dir>/coverage-ledger.json +``` + +Fix every structural and semantic error before continuing. The findings validator rejects input beyond 5 MiB, 1,000 top-level findings, or 64 nesting levels, and caps reported error output at 100 messages. Validator success proves format and ledger consistency only. + +### Phase 5: Verify the final records with fresh eyes + +Launch one fresh `research` verifier per final `confirmed` and `needs_validation` record, in parallel. This verifier checks the structured record, not the hunter write-up, and remains inside source/local boundaries. + +In a `quick` run, Phase 3 and Phase 5 merge: the Phase 3 verifier also performs these record checks and returns the final schema-shaped record, so each candidate gets one fresh independent reviewer instead of two. Every other profile keeps the two passes separate. Never skip independent review of a `confirmed` record in any profile. + +For `confirmed`, require it to check: + +1. Every repository-relative trace/evidence path, line, scope, and described operation. +2. Real entry interface and exact local input shape. +3. Every condition, parser/policy step, source-visible preventing layer, and observed local result. +4. Affected principal/resource and demonstrated impact. +5. Severity separation: realistic likelihood, demonstrated impact, overall no greater than impact. +6. Remediation strategy and any `code_changes`, including whether the fix enforces the invariant without merely moving trust. + +For `needs_validation`, require it to check: + +1. The source path is real and supports only the `claimed_root_cause` stated. +2. Every listed blocker is decisive and not already answerable locally. +3. The candidate names a boundary and a possible concrete result rather than a generic concern. +4. At least one validation-plan field is present and exact. `local` uses a bounded fixture; `deployment` asks an owner to observe a configuration, identity, route, policy, or runtime fact. Do not invent a plan for an inapplicable context, and never send audit traffic to a deployment. +5. The fingerprint matches prior/current records for the same root cause. + +Each verifier returns exactly one JSON object: `{"decision":"verified","fingerprint":"..."}` or `{"decision":"replace","reason":"...","record":{...}}`, with no surrounding prose. A replacement record must match its `confirmed`, `needs_validation`, or `rejected` schema branch. Treat a malformed or prose-wrapped Phase 5 result the same way as in Phase 3: discard it without repairing it and re-run with a fresh verifier when the budget permits. + +Do not apply a Phase 5 replacement as final when it promotes a record to a stronger verdict, including any promotion to `confirmed`, or materially changes the root cause, trace, execution input or observed result, demonstrated impact, or severity. Give that complete replacement to a new independent verifier that did not hunt, perform Phase 3 validation, or propose the Phase 5 replacement. The new verifier rechecks the current source and independently reproduces any decisive local result under the execution boundary, then returns `verified` or another replacement. Apply a material replacement only after this fresh verification. If another material replacement results, repeat with a fresh verifier. If budget or independence is unavailable, remove the disputed record from `findings.json`, keep its ledger unit as an unresolved candidate, and set `run_status: "incomplete"` with an exact `incomplete_reason`. Non-material wording or repository-line corrections may be applied directly when they do not change meaning or evidence. + +After every applied replacement, rerun both validators and update linked ledger decisions. If a final verifier identifies a separate root cause, assign a new fingerprint and send it through independent candidate validation before inclusion. Set `run_status: "complete"` only when every ledger candidate has an independent final disposition and every retained record passes Phase 5. + +Do not verify only `confirmed` records. A misleading `needs_validation` handoff wastes owner time and can preserve a false premise. + +### Phase 6: Produce target-neutral reports from final records + +Only after Phase 5 passes for every record retained in `findings.json`, derive prose from the final records, the ledger, and the hunter `hardening` notes retained in ledger bookkeeping. An incomplete run may report independently verified records, but it must identify each unresolved ledger candidate and must not present it as a finding. The prose files never change a verdict, severity, blocker, or demonstrated impact. + +#### `REPORT.md` + +Write: + +1. Run profile, scope, budget (if set) with agents spent versus planned, source ref, sandboxed source-and-local-only execution statement, prior-run use, and explicit deferred and out-of-scope coverage. Name carried same-source confirmations and changed-source revalidations. A `quick`, scoped, budget-limited, or incomplete run states plainly that it is a partial pass. If candidate validation exhausted a strict budget, state that the run is incomplete and list every unvalidated fingerprint and linked unit; do not describe those candidates as findings. If the budget prevented a mandatory critic, state which critic did not run and make no clean-coverage claim. +2. One short security posture summary. +3. A confirmed-findings table: severity, title, affected boundary, and one-line observed result. +4. Each confirmed finding: repository source location, lower-trust principal, target-native bounded reproduction, conditions, actual result, impact, priority rationale, and smallest source fix. +5. A separate `NEEDS VALIDATION` table. Give each lead's title, repository trace, exact blocker, bounded local next step, and safe owner-observed deployment check. Do not assign severity or call it a confirmed vulnerability. +6. Separate hardening notes and positive source patterns. +7. Coverage summary from the ledger: covered, candidate, blocked, and deferred counts, plus important exclusions and the final critic result. + +Do not describe rejected records as findings. Mention their fingerprints only when they explain a prior disagreement or coverage decision. + +#### `FINDINGS-DETAIL.md` + +For each confirmed `medium`, `high`, or `critical` record, copy the complete source path and target-neutral local reproduction: + +- ordered repository-relative trace and evidence; +- dummy attacker/principal and affected dummy resource; +- native input, invocation, or fixture and exact bounded instructions; +- observed output and the security invariant it proves; +- conditions and containment; +- source-level remediation and regression case. + +#### `NEEDS-VALIDATION.md` + +For every unresolved record, copy the source trace, verified evidence, exact blocker, affected boundary, and each applicable bounded local or owner-observed resolution plan. Keep these as prioritized leads without severity. Do not turn them into live test guidance or assume the missing deployment fact. + +HTTP is one possible native interface, not the default. A library finding may use a function call, a parser a fixture, a CLI a command, a desktop app an IPC or file action, and infrastructure a locally rendered policy. Do not require an endpoint, external account, or live environment that the target does not have. + +Keep the report proportional to the evidence. A clean run may have zero confirmed records. State that result and the remaining coverage/validation limits without inventing LOW findings. diff --git a/skills/thirdshift-security-audit/WEB-PROTOCOL-AND-AUTH.md b/skills/thirdshift-security-audit/WEB-PROTOCOL-AND-AUTH.md new file mode 100644 index 00000000..1a099fa5 --- /dev/null +++ b/skills/thirdshift-security-audit/WEB-PROTOCOL-AND-AUTH.md @@ -0,0 +1,105 @@ +# HTTP-Protocol and Authentication Hunting + +#### When to use this file + +Reach for this file when the target speaks HTTP at a parsing, caching, browser-authentication, or identity boundary: web applications, APIs, reverse proxies, CDNs, gateways, custom HTTP servers, and services implementing sessions, JWT, OAuth/OIDC, SAML, password recovery, MFA, passkeys, API keys, or mTLS. Use this with `ATTACK-CLASSES.md`: access-control review asks whether a principal may perform an operation; this file asks whether the HTTP or identity layer can confuse which principal, request, assurance level, or token the operation belongs to. + +Pick classes from Phase 1. Split a large target into request framing and cache policy, browser authentication, federated identity, strong authentication and recovery, service credentials, and session lifecycle. A single server behind an unobserved managed proxy has little source-confirmable smuggling surface; a proxy or custom parser has much more. + +## Core discipline (include in every agent prompt for this domain) + +``` +- Framing and cache findings require two interpretations of the same request, response, or key. Name both components and the exact normalized value on each side. +- For every credential, find the signature or secret verification and every binding required for its role: issuer, audience, origin, RP, client, session, principal, resource, assurance, expiry, and one-time state. +- Host, Forwarded, X-Forwarded-*, Origin, Referer, redirect targets, callback state, and request-derived URLs are trust decisions. Trace each to the affected identity or response. +- A missing header, cookie attribute, MFA prompt, or rate limit is not a finding alone. Require an accepted invalid request, cross-principal impact, assurance downgrade, or credential disclosure. +- Classify `confirmed` only from complete source evidence and bounded local request/token tests. Use `needs_validation` when proxy, IdP, browser, certificate, secret, or deployed configuration is required but not visible. +``` + +## HTTP framing and cache attack classes (subagent_type: `general`) + +**Request framing and desynchronization** +Front end and back end disagree on request length or header normalization. Review multiple `Content-Length` values, `Transfer-Encoding`, HTTP/2 or HTTP/3 downgrade, header-name normalization, forbidden connection headers, and CR/LF conversion. Confirm which bytes one component assigns to a request and which bytes its peer assigns to the next request. + +**Web cache poisoning through unkeyed input** +A request value changes cached content or security-relevant headers but is absent from the cache key. Compare cache key construction with every response variant, including forwarded host/scheme, selected cookies, query normalization, language/device headers, and authorization state. + +**Cache deception and private-response caching** +Cache routing treats a private dynamic path as a public static asset, or caches a response whose identity and authorization inputs are missing from policy. Compare edge cacheability with application route parsing, suffix/path-parameter normalization, and response cache directives. + +**Host and forwarded-header trust** +Untrusted host/proxy metadata determines absolute URLs, tenant routing, callbacks, reset links, cache keys, or the client address used by authorization. Confirm who can supply the header and whether trusted ingress removes client-provided copies. + +**Response-header injection** +Untrusted data reaches `Location`, `Set-Cookie`, CSP, or another response header with unsafe control characters or normalization. Verify framework rejection before reporting and require a security-relevant response change. + +## Browser-session attack classes (subagent_type: `general`) + +**Ordinary CSRF** +A browser sends ambient credentials to a state-changing endpoint that accepts a cross-site request without an effective anti-CSRF token, same-site request binding, or strict Origin/Referer validation. Inventory every cookie-authenticated mutation, including form, JSON-like, multipart, method-override, and legacy routes. SameSite is effective only for the cookie and browser contexts actually used; login CSRF and cross-site subresource requests can have different requirements. + +**Session fixation and invalidation** +Session identifiers are not rotated on login, account switch, MFA completion, impersonation, or other privilege changes, or remain valid after logout, password change, revocation, and account disable. Check server sessions, refresh tokens, signed cookies, websocket state, cache copies, and fallback endpoints. + +**Cookie scope and transport** +A sensitive cookie has an over-broad `Domain` or `Path`, can cross an insecure transport, or conflicts with a sibling cookie that another component selects differently. Bare missing flags remain hardening notes unless a realistic less-trusted origin, network position, or browser path can gain or replace the credential. + +## Federated-identity attack classes (subagent_type: `general`) + +First establish role. Authorization-server controls such as redirect allowlisting and code issuance do not belong to a relying-party client. Verification and binding defects belong to the component consuming the artifact. + +**JWT verification and claim binding** +Check signature verification, server-pinned algorithm and key source, then `exp`, `nbf`, `aud`, and `iss`. Review `kid`, `jku`, and `x5u` as untrusted key selectors, duplicate/header normalization, and decode-without-verify paths. A valid token for another service is invalid here even when signed by a trusted issuer. + +**OAuth/OIDC request and callback binding** +Validate exact `redirect_uri` ownership where the target is the authorization server; session-bound `state`; PKCE and authorization-code binding where applicable; ID-token issuer/audience/signature/nonce; and selected-IdP binding in multi-provider flows. Compare initial callback, retry, mobile/deep-link, and account-link routes. + +**SAML signed-object and assertion binding** +Ensure the element whose signature is validated is the element used as identity. Review unsigned/fallback paths, safe XML parser configuration, canonicalization differences, and freshness/binding fields such as validity windows, audience/recipient, request correlation, and replay state. + +## MFA, passkey, and account-transition attack classes (subagent_type: `general`) + +**MFA enrollment and assurance downgrade** +Enrollment, replacement, disablement, recovery-code generation, trusted-device creation, and fallback login require the intended prior assurance. Check that a valid first factor cannot enroll or replace the second factor without policy-required fresh authentication, and that disabled or stale factors stop authorizing sessions. + +**Step-up binding and bypass** +A successful challenge upgrades the wrong session, account, tenant, action, or API request, or an alternate route omits the assurance check. Bind the challenge to principal, current session, assurance target, operation or resource when required, expiry, and one-time completion. Compare UI, API, batch, recovery, and resumed-flow paths. + +**WebAuthn and passkey verification** +At registration, bind challenge, RP ID, expected origin, credential, user/userHandle, algorithm, and policy-required user verification to the initiating session. At authentication, verify challenge, RP/origin, credential membership, signature, and intended user presence/verification. Check account-discovery and linking flows for userHandle or credential-to-account confusion. Signature-counter handling is meaningful only when the product treats regressions as a clone signal. + +**Account linking and identity collision** +Adding an IdP, passkey, email, phone, device, or external account to an existing account must require a current authenticated session, verified ownership of the new identity, policy-required step-up, and callback state bound to the account that initiated linking. Review unlink/relink and invite-acceptance paths for verified-identifier or tenant collisions. + +**Password reset and broader recovery** +Recovery tokens, support/admin recovery, backup codes, device migration, and email or phone change often become the weakest authentication path. Verify token randomness, user/action binding, expiry, one-time state, rate/accounting controls, delivery URL trust, and invalidation of prior tokens and sessions. Different responses that only reveal public account existence are not automatically security findings. + +## API-key and mTLS attack classes (subagent_type: `general`) + +**API-key scope and resource binding** +A key authenticates to broader tenants, resources, actions, or environments than its server-side record grants, or request parameters override those bindings. Review key lookup, prefix/full-secret verification, type confusion between publishable and secret keys, scope checks, rotation, revocation caches, and bulk endpoints. + +**API-key exposure and unsafe transport** +Keys appear in client bundles, URLs, redirects, logs, error paths, build artifacts, or responses accessible to a lower-trust principal. A public identifier called a key is not a secret. Confirm key type and the authority gained by disclosure. + +**mTLS peer and application-identity confusion** +A process trusts client-certificate identity headers from any network peer, verifies a chain but maps attacker-influenceable subject text to an account incorrectly, or accepts a certificate for the wrong trust domain, extended usage, audience, or validity policy. Where a trusted proxy terminates mTLS, verify only that proxy can connect, it removes incoming identity headers, and the backend binds the sanitized identity to the request. + +**Certificate lifecycle fallback** +Expired, revoked, missing, or renewal-failed certificates cause silent fallback to bearer-only or anonymous operation, or long-lived pooled connections retain authorization after revocation. Missing deployment revocation data makes the result `needs_validation`; an in-repo fail-open branch is source-confirmable. + +## Universal moves (apply across the above) + +- Walk issue → store → transmit → consume → refresh → revoke for every credential and challenge. Compare normal, error, retry, migration, legacy, and account-switch paths. +- Enumerate every door to the same identity and every route to the same sensitive operation. The effective policy is the weakest parallel path, not the most polished UI. +- Diff parser, proxy, router, cache, and application normalization side by side. For local validation, feed identical bounded request fixtures into each component rather than sending traffic to a live deployment. +- For recovery and linking, draw the account before/after graph. Each edge must name the current principal, proof of the new identity, required assurance, callback/session binding, and revocation effect. + +## Validation rules (apply before reporting ANY finding here) + +1. Apply a source-visibility gate. Proxy chains, edge cache keys, IdP policy, certificate trust, browser cookie behavior, secrets, and deployed auth modes may be outside the repository. Record a precise `needs_validation` candidate instead of asserting missing infrastructure behavior. +2. For framing and cache findings, name both components and the divergent parse/key. Confirm cross-request, cross-user, or private-response impact with bounded local fixtures. +3. For token, MFA, passkey, account-link, recovery, API-key, and mTLS findings, cite the verification line and missing principal/session/resource/origin/audience/action/assurance binding. Prove the server accepts the invalid transition or credential. +4. For CSRF, name the ambient credential, state-changing route, accepted cross-site request shape, browser cookie policy, and missing effective check. Read-only actions and routes requiring a non-ambient bearer token do not qualify. +5. Verify framework and library defaults. If version or configuration is unknown, use `needs_validation`; do not turn an unverified critical claim into a lower-severity confirmed finding. +6. Return `confirmed` only with a complete source trace and observable unauthorized identity, state, or disclosure. For `needs_validation`, name the missing fact and safe local or owner-observed check that resolves it. diff --git a/skills/thirdshift-security-audit/report-schema.json b/skills/thirdshift-security-audit/report-schema.json new file mode 100644 index 00000000..55815a36 --- /dev/null +++ b/skills/thirdshift-security-audit/report-schema.json @@ -0,0 +1,461 @@ +{ + "$comment": "Top-level contract for findings.json. validate-findings.cjs interprets and checks this schema directly.", + "type": "array", + "items": { + "oneOf": [ + { + "type": "object", + "description": "A source-grounded vulnerability that was independently demonstrated.", + "properties": { + "verdict": { + "type": "string", + "const": "confirmed" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$", + "description": "A stable source-derived identifier that does not change between validation states." + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "intended_behavior": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "conditions": { + "type": "array", + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["authentication_level", "authorization_role", "user_interaction", "system_configuration", "network_routing", "environmental_dependency", "data_state", "timing_dependency", "third_party_dependency"] + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "description"], + "additionalProperties": false + } + }, + "execution": { + "type": "object", + "description": "Target-neutral reproduction in the target's native interface.", + "properties": { + "attacker_perspective": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "payloads": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string" + } + }, + "instructions": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "observed_result": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["attacker_perspective", "payloads", "instructions", "observed_result"], + "additionalProperties": false + }, + "remediation": { + "type": "object", + "properties": { + "strategy": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "code_changes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "file_name": { + "type": "string", + "minLength": 1 + }, + "fixed_code": { + "type": "string" + } + }, + "required": ["file_name", "fixed_code"], + "additionalProperties": false + } + } + }, + "required": ["strategy"], + "additionalProperties": false + }, + "severity": { + "type": "object", + "properties": { + "likelihood": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + }, + "impact": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + }, + "overall_severity": { + "type": "string", + "enum": ["informational", "low", "medium", "high", "critical"] + } + }, + "required": ["likelihood", "impact", "overall_severity"], + "additionalProperties": false + }, + "confidence": { + "type": "object", + "properties": { + "score": { + "type": "string", + "enum": ["low", "medium", "high"] + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["score", "reason"], + "additionalProperties": false + } + }, + "required": ["verdict", "fingerprint", "title", "description", "root_cause", "intended_behavior", "trace", "evidence", "conditions", "execution", "remediation", "severity", "confidence"], + "additionalProperties": false + }, + { + "type": "object", + "description": "A source-grounded candidate whose decisive validation is blocked.", + "properties": { + "verdict": { + "type": "string", + "const": "needs_validation" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$" + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "claimed_root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "blockers": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "validation_plan": { + "type": "object", + "properties": { + "local": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "deployment": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "additionalProperties": false + } + }, + "required": ["verdict", "fingerprint", "title", "description", "claimed_root_cause", "trace", "evidence", "blockers", "validation_plan"], + "additionalProperties": false + }, + { + "type": "object", + "description": "A source-grounded candidate refuted during validation.", + "properties": { + "verdict": { + "type": "string", + "const": "rejected" + }, + "fingerprint": { + "type": "string", + "minLength": 1, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$" + }, + "title": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "claimed_root_cause": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "trace": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": ["entrypoint", "propagation", "sink"] + }, + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "scope": { + "type": "string", + "minLength": 1, + "visibleContent": true + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["kind", "file", "line", "scope", "description"], + "additionalProperties": false + } + }, + "evidence": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["file", "line", "description"], + "additionalProperties": false + } + }, + "reason": { + "type": "string", + "minLength": 1, + "visibleContent": true + } + }, + "required": ["verdict", "fingerprint", "title", "description", "claimed_root_cause", "trace", "evidence", "reason"], + "additionalProperties": false + } + ] + } +} diff --git a/skills/thirdshift-security-audit/validate-coverage-ledger.cjs b/skills/thirdshift-security-audit/validate-coverage-ledger.cjs new file mode 100755 index 00000000..0ef58657 --- /dev/null +++ b/skills/thirdshift-security-audit/validate-coverage-ledger.cjs @@ -0,0 +1,872 @@ +#!/usr/bin/env node + +/** + * Validates coverage-ledger.json and its canonical coverage IDs. + * Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json> + */ + +const fs = require("node:fs"); +const path = require("node:path"); +const { TextDecoder } = require("node:util"); + +// Conservative bounds apply before JSON.parse and again to the parsed document. +const LIMITS = Object.freeze({ + inputBytes: 5 * 1024 * 1024, + units: 10000, + collectionItems: 1000, + objectFields: 1000, + nestingDepth: 64, + preflightValues: 500000, + validationErrors: 100, +}); +const MAX_INPUT_BYTES = LIMITS.inputBytes; +const MAX_UNITS = LIMITS.units; +const MAX_LIST_ITEMS = LIMITS.collectionItems; +const MAX_TEXT_LENGTH = 4096; +const REQUIRED_FIELDS = [ + "coverage_id", + "canonical_refs", + "surface", + "boundary", + "subsystem", + "attack_class", + "starting_paths", + "ordinary_attack_class_block", + "selected_companion_blocks", + "excluded_blocks", + "prior_status", + "attempts", + "wave", + "status", + "agent_id", + "reviewed_paths", + "local_checks", + "result_fingerprints", + "unresolved", +]; +const REF_FIELDS = ["surface", "boundary", "subsystem", "attack_class"]; +const STATUSES = new Set([ + "planned", + "not_applicable", + "out_of_scope", + "in_progress", + "covered", + "candidate", + "blocked", + "deferred", +]); +const ATTEMPT_STATUSES = new Set(["covered", "candidate", "blocked"]); +const ATTEMPT_FIELDS = [ + "wave", + "status", + "agent_id", + "reviewed_paths", + "local_checks", + "result_fingerprints", + "unresolved", + "reassignment_reason", +]; +const PRIOR_STATUSES = new Set([ + "new", + "prior_confirmed_same_source", + "prior_confirmed_changed_source", + "prior_needs_validation", + "prior_deferred", + "prior_blocked", + "prior_out_of_scope", + "prior_covered_same_source", + "prior_covered_changed_source", + "prior_rejected_claim_changed", + "none", +]); +const FINGERPRINT_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:/@+-]*$/; +const AGENT_ID_PATTERN = /^[a-z0-9][a-z0-9_-]{0,63}$/; +const WINDOWS_RESERVED_AGENT_ID = /^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])$/; +const VISIBLE_CONTENT = /[^\p{White_Space}\p{Cc}\p{Cf}\p{Default_Ignorable_Code_Point}]/u; +const PATH_FORBIDDEN_CHARACTER = /[\p{Cc}\p{Cf}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/u; +const WINDOWS_RESERVED_COMPONENT = /^(?:con|prn|aux|nul|clock\$|conin\$|conout\$|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\.|$)/iu; +const UTF8_DECODER = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); +const UNSAFE_DIAGNOSTIC_CHARACTER = /[\p{Cc}\p{Cf}\p{Cs}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/gu; +const MAX_DIAGNOSTIC_STRING_LENGTH = 256; + +const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value, key); + +class JsonStructureError extends Error {} +class SafeInputError extends Error {} + +function escapeUnsafeDiagnosticCharacters(value) { + return String(value).replace(UNSAFE_DIAGNOSTIC_CHARACTER, (character) => { + const codePoint = character.codePointAt(0); + return codePoint <= 0xffff + ? `\\u${codePoint.toString(16).padStart(4, "0")}` + : `\\u{${codePoint.toString(16)}}`; + }); +} + +function safeQuote(value) { + let serialized; + if (typeof value === "string") { + const clipped = value.length > MAX_DIAGNOSTIC_STRING_LENGTH + ? `${value.slice(0, MAX_DIAGNOSTIC_STRING_LENGTH)}...` + : value; + serialized = JSON.stringify(clipped); + } else if (value === null || typeof value === "boolean") { + serialized = String(value); + } else if (typeof value === "number" && Number.isFinite(value)) { + serialized = String(value); + } else { + serialized = `"<${Array.isArray(value) ? "array" : typeof value}>"`; + } + return escapeUnsafeDiagnosticCharacters(serialized); +} + +function createErrorList() { + const errors = []; + Object.defineProperty(errors, "push", { + value(...messages) { + const remaining = LIMITS.validationErrors - this.length; + if (remaining > 0) { + Array.prototype.push.apply(this, messages.slice(0, remaining).map(escapeUnsafeDiagnosticCharacters)); + } + return this.length; + }, + }); + return errors; +} + +function isObject(value) { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function preflightJsonText(contents) { + const containers = []; + let rootState = "value"; + let inString = false; + let escaped = false; + let totalValues = 0; + + function fail(message) { + throw new JsonStructureError(`input ${message}`); + } + + function currentContainer() { + return containers[containers.length - 1]; + } + + function countValue() { + totalValues++; + if (totalValues > LIMITS.preflightValues) { + fail(`exceeds ${LIMITS.preflightValues} total value limit`); + } + } + + function beginValue() { + const container = currentContainer(); + if (!container) { + if (rootState !== "value") fail("has malformed JSON structure"); + rootState = "end"; + } else if (container.type === "array") { + if (container.state !== "firstValueOrEnd" && container.state !== "value") { + fail("has malformed JSON array structure"); + } + container.items++; + const limit = container.topLevel ? MAX_UNITS : LIMITS.collectionItems; + if (container.items > limit) { + fail(container.topLevel + ? `exceeds ${limit} top-level unit limit` + : `exceeds ${limit} item array limit`); + } + container.state = "commaOrEnd"; + } else { + if (container.state !== "value") fail("has malformed JSON object structure"); + container.state = "commaOrEnd"; + } + countValue(); + } + + function beginString() { + const container = currentContainer(); + if (container && container.type === "object" && + (container.state === "firstKeyOrEnd" || container.state === "key")) { + container.fields++; + if (container.fields > LIMITS.objectFields) { + fail(`exceeds ${LIMITS.objectFields} field object limit`); + } + container.state = "colon"; + } else { + beginValue(); + } + inString = true; + } + + function beginContainer(type) { + beginValue(); + if (containers.length >= LIMITS.nestingDepth) { + fail(`exceeds nesting depth limit ${LIMITS.nestingDepth}`); + } + containers.push(type === "array" + ? { type, state: "firstValueOrEnd", items: 0, topLevel: containers.length === 0 } + : { type, state: "firstKeyOrEnd", fields: 0 }); + } + + function closeContainer(type) { + const container = currentContainer(); + if (!container || container.type !== type) fail("has mismatched JSON containers"); + const canClose = type === "array" + ? container.state === "firstValueOrEnd" || container.state === "commaOrEnd" + : container.state === "firstKeyOrEnd" || container.state === "commaOrEnd"; + if (!canClose) fail(`has malformed JSON ${type} structure`); + containers.pop(); + } + + function isWhitespace(character) { + return character === " " || character === "\t" || character === "\r" || character === "\n"; + } + + function isTokenDelimiter(character) { + return isWhitespace(character) || character === "," || character === ":" || + character === "[" || character === "]" || character === "{" || + character === "}" || character === "\""; + } + + for (let index = 0; index < contents.length; index++) { + const character = contents[index]; + if (inString) { + if (escaped) { + escaped = false; + } else if (character === "\\") { + escaped = true; + } else if (character === "\"") { + inString = false; + } + continue; + } + if (isWhitespace(character)) continue; + + if (character === "\"") { + beginString(); + } else if (character === "[") { + beginContainer("array"); + } else if (character === "{") { + beginContainer("object"); + } else if (character === "]") { + closeContainer("array"); + } else if (character === "}") { + closeContainer("object"); + } else if (character === ":") { + const container = currentContainer(); + if (!container || container.type !== "object" || container.state !== "colon") { + fail("has malformed JSON object structure"); + } + container.state = "value"; + } else if (character === ",") { + const container = currentContainer(); + if (!container || container.state !== "commaOrEnd") fail("has malformed JSON collection structure"); + container.state = container.type === "array" ? "value" : "key"; + } else { + beginValue(); + while (index + 1 < contents.length && !isTokenDelimiter(contents[index + 1])) index++; + } + } + + if (inString) fail("has an unterminated JSON string"); + if (containers.length > 0) fail("has truncated JSON structure"); + if (rootState !== "end") fail("has no JSON value"); +} + +function preflightDocument(root) { + const errors = createErrorList(); + const stack = [{ value: root, depth: 0, location: "$" }]; + let visited = 0; + + while (stack.length > 0) { + const { value, depth, location } = stack.pop(); + visited++; + if (visited > LIMITS.preflightValues) { + errors.push(`$: exceeds ${LIMITS.preflightValues} total values`); + return errors; + } + if (value === null || typeof value !== "object") continue; + if (depth >= LIMITS.nestingDepth) { + errors.push(`${location}: exceeds nesting depth limit ${LIMITS.nestingDepth}`); + return errors; + } + + if (Array.isArray(value)) { + const limit = location === "$" ? MAX_UNITS : MAX_LIST_ITEMS; + if (value.length > limit) { + errors.push(`${location}: exceeds ${limit} entries`); + return errors; + } + for (let index = value.length - 1; index >= 0; index--) { + stack.push({ value: value[index], depth: depth + 1, location: `${location}[${index}]` }); + } + continue; + } + + const keys = Object.keys(value); + if (keys.length > LIMITS.objectFields) { + errors.push(`${location}: exceeds ${LIMITS.objectFields} object fields`); + return errors; + } + for (let index = keys.length - 1; index >= 0; index--) { + const key = keys[index]; + stack.push({ value: value[key], depth: depth + 1, location: `${location}{${index}}` }); + } + } + + return errors; +} + +function hasValidUnicodeScalarValues(value) { + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff) { + if (index >= value.length) return false; + const second = value.charCodeAt(index++); + if (second < 0xdc00 || second > 0xdfff) return false; + } else if (first >= 0xdc00 && first <= 0xdfff) { + return false; + } + } + return true; +} + +function hasVisibleProse(value) { + return hasValidUnicodeScalarValues(value) && VISIBLE_CONTENT.test(value); +} + +function isVisibleText(value, maxLength = MAX_TEXT_LENGTH) { + return typeof value === "string" && + value.length > 0 && + value.length <= maxLength && + value.trim() === value && + hasVisibleProse(value); +} + +function isCanonicalRef(value) { + return isVisibleText(value, 1024) && + !PATH_FORBIDDEN_CHARACTER.test(value) && + value.normalize("NFC") === value; +} + +function encodeCanonicalRef(value) { + if (!isCanonicalRef(value)) throw new TypeError("invalid canonical reference"); + let encoded = ""; + for (const byte of Buffer.from(value, "utf8")) { + const unreserved = + (byte >= 0x41 && byte <= 0x5a) || + (byte >= 0x61 && byte <= 0x7a) || + (byte >= 0x30 && byte <= 0x39) || + byte === 0x2d || byte === 0x2e || byte === 0x5f || byte === 0x7e; + encoded += unreserved ? String.fromCharCode(byte) : `%${byte.toString(16).toUpperCase().padStart(2, "0")}`; + } + return encoded; +} + +function canonicalCoverageId(refs) { + if (!isObject(refs)) throw new TypeError("canonical_refs must be an object"); + const fields = hasOwn(refs, "lifecycle") ? [...REF_FIELDS, "lifecycle"] : REF_FIELDS; + if (Object.keys(refs).length !== fields.length || fields.some((field) => !hasOwn(refs, field))) { + throw new TypeError("canonical_refs has missing or unexpected fields"); + } + return fields.map((field) => encodeCanonicalRef(refs[field])).join("::"); +} + +function isSafeRelativePath(value) { + if (typeof value !== "string" || value.length === 0 || !hasValidUnicodeScalarValues(value) || value.trim() !== value || PATH_FORBIDDEN_CHARACTER.test(value) || value.includes("\\") || value.includes(":")) return false; + if (path.posix.isAbsolute(value) || path.win32.isAbsolute(value) || /^[A-Za-z]:/.test(value) || value.startsWith("~")) return false; + return value.split("/").every((segment) => + segment !== "" && + segment !== "." && + segment !== ".." && + !/[ .]$/u.test(segment) && + !WINDOWS_RESERVED_COMPONENT.test(segment)); +} + +function isSafeAgentId(value) { + return typeof value === "string" && + AGENT_ID_PATTERN.test(value) && + !WINDOWS_RESERVED_AGENT_ID.test(value); +} + +function isOwnedArtifactPath(value, agentId) { + if (!isSafeAgentId(agentId) || !isSafeRelativePath(value)) return false; + const prefix = `agents/${agentId}/artifacts/`; + return value.startsWith(prefix) && value.length > prefix.length; +} + +function validateStringArray(value, location, errors, options = {}) { + const { allowEmpty = true, fingerprint = false, pathValue = false } = options; + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (!allowEmpty && value.length === 0) errors.push(`${location}: must not be empty`); + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + const seen = new Set(); + value.slice(0, MAX_LIST_ITEMS).forEach((entry, index) => { + const entryLocation = `${location}[${index}]`; + const valid = pathValue ? isSafeRelativePath(entry) : isVisibleText(entry); + if (!valid) errors.push(`${entryLocation}: invalid ${pathValue ? "repository-relative path" : "text"}`); + if (fingerprint && typeof entry === "string" && !FINGERPRINT_PATTERN.test(entry)) { + errors.push(`${entryLocation}: invalid fingerprint`); + } + if (typeof entry === "string" && seen.has(entry)) errors.push(`${entryLocation}: duplicate entry`); + if (typeof entry === "string") seen.add(entry); + }); +} + +function validateChecks(value, location, errors) { + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + value.slice(0, MAX_LIST_ITEMS).forEach((check, index) => { + const base = `${location}[${index}]`; + if (!isObject(check)) { + errors.push(`${base}: expected object`); + return; + } + for (const field of ["agent_id", "reviewed_paths", "invariant", "method", "result", "artifact"]) { + if (!hasOwn(check, field)) errors.push(`${base}: missing required field ${safeQuote(field)}`); + } + if (!isSafeAgentId(check.agent_id)) errors.push(`${base}.agent_id: expected a canonical lowercase agent ID`); + validateStringArray(check.reviewed_paths, `${base}.reviewed_paths`, errors, { allowEmpty: false, pathValue: true }); + if (!isVisibleText(check.invariant)) errors.push(`${base}.invariant: invalid text`); + if (check.method !== "source" && check.method !== "local") errors.push(`${base}.method: expected "source" or "local"`); + if (!isVisibleText(check.result)) errors.push(`${base}.result: invalid text`); + if (check.method === "source" && check.artifact !== null) { + errors.push(`${base}.artifact: source-only check must use null`); + } else if (check.method === "local") { + if (!isOwnedArtifactPath(check.artifact, check.agent_id)) { + errors.push(`${base}.artifact: local check requires an artifact owned by agent ${safeQuote(isSafeAgentId(check.agent_id) ? check.agent_id : "<agent-id>")}`); + } + } else if (check.method !== "source" && check.method !== "local" && check.artifact !== null && !isSafeRelativePath(check.artifact)) { + errors.push(`${base}.artifact: expected null or a safe output-relative path`); + } + }); +} + +function validateReviewedPathOwnership(unit, base, errors) { + if (!Array.isArray(unit.reviewed_paths) || !Array.isArray(unit.local_checks)) return; + const aggregatePaths = new Set(unit.reviewed_paths.filter((value) => typeof value === "string")); + const ownedPaths = new Set(); + for (const check of unit.local_checks) { + if (!isObject(check) || !Array.isArray(check.reviewed_paths)) continue; + for (const reviewedPath of check.reviewed_paths) { + if (typeof reviewedPath === "string") ownedPaths.add(reviewedPath); + } + } + for (const reviewedPath of aggregatePaths) { + if (!ownedPaths.has(reviewedPath)) errors.push(`${base}.reviewed_paths: ${safeQuote(reviewedPath)} has no check owner`); + } + for (const reviewedPath of ownedPaths) { + if (!aggregatePaths.has(reviewedPath)) errors.push(`${base}.local_checks: owned path ${safeQuote(reviewedPath)} is absent from aggregate reviewed_paths`); + } +} + +function validateExcludedBlocks(value, location, errors) { + if (!Array.isArray(value)) { + errors.push(`${location}: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${location}: exceeds ${MAX_LIST_ITEMS} entries`); + const seen = new Set(); + value.slice(0, MAX_LIST_ITEMS).forEach((entry, index) => { + const base = `${location}[${index}]`; + if (!isObject(entry)) { + errors.push(`${base}: expected object`); + return; + } + if (!isVisibleText(entry.block)) errors.push(`${base}.block: invalid text`); + if (!isVisibleText(entry.reason)) errors.push(`${base}.reason: invalid text`); + if (typeof entry.block === "string" && seen.has(entry.block)) errors.push(`${base}.block: duplicate entry`); + if (typeof entry.block === "string") seen.add(entry.block); + }); +} + +function semanticKey(unit) { + return JSON.stringify([ + unit.surface, + unit.boundary, + unit.subsystem, + unit.attack_class, + hasOwn(unit, "lifecycle") ? unit.lifecycle : null, + ]); +} + +function hasValidSemanticFields(unit) { + return ["surface", "boundary", "subsystem", "attack_class"].every((field) => isVisibleText(unit[field])) && + (!hasOwn(unit, "lifecycle") || isVisibleText(unit.lifecycle)); +} + +function requireEmptyArray(unit, field, base, errors) { + if (Array.isArray(unit[field]) && unit[field].length > 0) { + errors.push(`${base}.${field}: unit with status ${safeQuote(unit.status)} must keep this array empty`); + } +} + +function requireNonemptyArray(unit, field, base, errors) { + if (!Array.isArray(unit[field]) || unit[field].length === 0) { + errors.push(`${base}.${field}: unit with status ${safeQuote(unit.status)} requires entries`); + } +} + +function validateStateInvariants(unit, base, errors) { + const emptyEvidence = () => { + requireEmptyArray(unit, "reviewed_paths", base, errors); + requireEmptyArray(unit, "local_checks", base, errors); + }; + const requireOwner = () => { + if (!isSafeAgentId(unit.agent_id)) errors.push(`${base}.agent_id: unit with status ${safeQuote(unit.status)} requires a canonical lowercase agent ID`); + }; + + if (unit.status !== "candidate") requireEmptyArray(unit, "result_fingerprints", base, errors); + + switch (unit.status) { + case "planned": + if (unit.agent_id !== null) errors.push(`${base}.agent_id: planned unit must be unassigned`); + emptyEvidence(); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "not_applicable": + case "out_of_scope": + case "deferred": + if (unit.agent_id !== null) errors.push(`${base}.agent_id: unit with status ${safeQuote(unit.status)} must be unassigned`); + emptyEvidence(); + requireNonemptyArray(unit, "unresolved", base, errors); + break; + case "in_progress": + requireOwner(); + emptyEvidence(); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "blocked": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireNonemptyArray(unit, "unresolved", base, errors); + break; + case "covered": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireEmptyArray(unit, "unresolved", base, errors); + break; + case "candidate": + requireOwner(); + requireNonemptyArray(unit, "reviewed_paths", base, errors); + requireNonemptyArray(unit, "local_checks", base, errors); + requireNonemptyArray(unit, "result_fingerprints", base, errors); + break; + } +} + +function validateAttempts(value, unit, base, errors) { + if (!Array.isArray(value)) { + errors.push(`${base}.attempts: expected array`); + return; + } + if (value.length > MAX_LIST_ITEMS) errors.push(`${base}.attempts: exceeds ${MAX_LIST_ITEMS} entries`); + + const priorOwners = new Set(); + const priorArtifacts = new Set(); + let previousWave = 0; + value.slice(0, MAX_LIST_ITEMS).forEach((attempt, index) => { + const attemptBase = `${base}.attempts[${index}]`; + if (!isObject(attempt)) { + errors.push(`${attemptBase}: expected object`); + return; + } + for (const field of ATTEMPT_FIELDS) { + if (!hasOwn(attempt, field)) errors.push(`${attemptBase}: missing required field ${safeQuote(field)}`); + } + if (!Number.isInteger(attempt.wave) || attempt.wave < 1) { + errors.push(`${attemptBase}.wave: expected a positive integer`); + } else { + if (attempt.wave <= previousWave) errors.push(`${attemptBase}.wave: archived attempt waves must be strictly increasing`); + if (Number.isInteger(unit.wave) && attempt.wave >= unit.wave) { + errors.push(`${attemptBase}.wave: archived attempt wave must precede current wave ${safeQuote(unit.wave)}`); + } + previousWave = attempt.wave; + } + if (!ATTEMPT_STATUSES.has(attempt.status)) { + errors.push(`${attemptBase}.status: expected "covered", "candidate", or "blocked"`); + } + let hasFreshOwner = false; + if (!isSafeAgentId(attempt.agent_id)) { + errors.push(`${attemptBase}.agent_id: archived attempt requires a canonical lowercase agent ID`); + } else if (priorOwners.has(attempt.agent_id)) { + errors.push(`${attemptBase}.agent_id: assignment owner must be fresh for each attempt`); + } else { + hasFreshOwner = true; + } + validateStringArray(attempt.reviewed_paths, `${attemptBase}.reviewed_paths`, errors, { pathValue: true }); + validateChecks(attempt.local_checks, `${attemptBase}.local_checks`, errors); + validateReviewedPathOwnership(attempt, attemptBase, errors); + validateStringArray(attempt.result_fingerprints, `${attemptBase}.result_fingerprints`, errors, { fingerprint: true }); + validateStringArray(attempt.unresolved, `${attemptBase}.unresolved`, errors); + if (!isVisibleText(attempt.reassignment_reason)) errors.push(`${attemptBase}.reassignment_reason: invalid text`); + validateStateInvariants(attempt, attemptBase, errors); + + if (Array.isArray(attempt.local_checks)) { + attempt.local_checks.forEach((check, checkIndex) => { + if (!isObject(check)) return; + if (priorOwners.has(check.agent_id)) { + errors.push(`${attemptBase}.local_checks[${checkIndex}].agent_id: prior assignment owner evidence must remain in its earlier attempt`); + } + if (typeof check.artifact === "string" && priorArtifacts.has(check.artifact)) { + errors.push(`${attemptBase}.local_checks[${checkIndex}].artifact: artifact from an earlier attempt cannot be reused`); + } + if (check.method === "local" && typeof check.artifact === "string") priorArtifacts.add(check.artifact); + }); + } + if (hasFreshOwner) priorOwners.add(attempt.agent_id); + }); + + if (isSafeAgentId(unit.agent_id) && priorOwners.has(unit.agent_id)) { + errors.push(`${base}.agent_id: current assignment owner must be fresh after reassignment`); + } + if (Array.isArray(unit.local_checks)) { + unit.local_checks.forEach((check, index) => { + if (!isObject(check)) return; + if (priorOwners.has(check.agent_id)) { + errors.push(`${base}.local_checks[${index}].agent_id: prior assignment owner evidence must remain in its archived attempt`); + } + if (typeof check.artifact === "string" && priorArtifacts.has(check.artifact)) { + errors.push(`${base}.local_checks[${index}].artifact: artifact from an archived attempt cannot be reused`); + } + }); + } +} + +function collectUnitErrors(unit, index) { + const errors = createErrorList(); + const base = `$[${index}]`; + if (!isObject(unit)) return [`${base}: expected object`]; + + for (const field of REQUIRED_FIELDS) { + if (!hasOwn(unit, field)) errors.push(`${base}: missing required field ${safeQuote(field)}`); + } + for (const field of ["surface", "boundary", "subsystem", "attack_class"]) { + if (!isVisibleText(unit[field])) errors.push(`${base}.${field}: invalid text`); + } + if (hasOwn(unit, "lifecycle") && !isVisibleText(unit.lifecycle)) errors.push(`${base}.lifecycle: invalid text`); + + let expectedId = null; + if (!isObject(unit.canonical_refs)) { + errors.push(`${base}.canonical_refs: expected object`); + } else { + const expectedFields = hasOwn(unit.canonical_refs, "lifecycle") ? [...REF_FIELDS, "lifecycle"] : REF_FIELDS; + for (const field of expectedFields) { + if (!hasOwn(unit.canonical_refs, field)) { + errors.push(`${base}.canonical_refs: missing required field ${safeQuote(field)}`); + } else if (!isCanonicalRef(unit.canonical_refs[field])) { + errors.push(`${base}.canonical_refs.${field}: invalid canonical reference`); + } + } + if (Object.keys(unit.canonical_refs).some((field) => !expectedFields.includes(field))) { + errors.push(`${base}.canonical_refs: contains unexpected fields`); + } + if (hasOwn(unit, "lifecycle") !== hasOwn(unit.canonical_refs, "lifecycle")) { + errors.push(`${base}: lifecycle and canonical_refs.lifecycle must appear together`); + } + try { + expectedId = canonicalCoverageId(unit.canonical_refs); + } catch { + // The specific reference errors above are more useful. + } + } + if (!isVisibleText(unit.coverage_id, 65536)) { + errors.push(`${base}.coverage_id: invalid text`); + } else if (expectedId !== null && unit.coverage_id !== expectedId) { + errors.push(`${base}.coverage_id: expected canonical ID ${safeQuote(expectedId)}`); + } + + validateStringArray(unit.starting_paths, `${base}.starting_paths`, errors, { allowEmpty: false, pathValue: true }); + if (unit.ordinary_attack_class_block !== null && !isVisibleText(unit.ordinary_attack_class_block)) { + errors.push(`${base}.ordinary_attack_class_block: expected null or non-empty text`); + } + validateStringArray(unit.selected_companion_blocks, `${base}.selected_companion_blocks`, errors); + validateExcludedBlocks(unit.excluded_blocks, `${base}.excluded_blocks`, errors); + if (Array.isArray(unit.selected_companion_blocks) && Array.isArray(unit.excluded_blocks)) { + const selected = new Set(unit.selected_companion_blocks); + unit.excluded_blocks.forEach((entry, blockIndex) => { + if (isObject(entry) && selected.has(entry.block)) { + errors.push(`${base}.excluded_blocks[${blockIndex}].block: block is also selected`); + } + }); + } + + if (!PRIOR_STATUSES.has(unit.prior_status)) errors.push(`${base}.prior_status: invalid value ${safeQuote(unit.prior_status)}`); + if (!STATUSES.has(unit.status)) errors.push(`${base}.status: invalid value ${safeQuote(unit.status)}`); + if (!Number.isInteger(unit.wave) || unit.wave < 1) errors.push(`${base}.wave: expected a positive integer`); + if (unit.agent_id !== null && !isSafeAgentId(unit.agent_id)) errors.push(`${base}.agent_id: expected null or a safe agent ID`); + + validateAttempts(unit.attempts, unit, base, errors); + validateStringArray(unit.reviewed_paths, `${base}.reviewed_paths`, errors, { pathValue: true }); + validateChecks(unit.local_checks, `${base}.local_checks`, errors); + validateReviewedPathOwnership(unit, base, errors); + validateStringArray(unit.result_fingerprints, `${base}.result_fingerprints`, errors, { fingerprint: true }); + validateStringArray(unit.unresolved, `${base}.unresolved`, errors); + + validateStateInvariants(unit, base, errors); + + return errors; +} + +function readFileWithinLimit(file) { + const noFollow = fs.constants.O_NOFOLLOW; + const nonBlock = fs.constants.O_NONBLOCK; + if (!Number.isInteger(noFollow) || noFollow === 0 || !Number.isInteger(nonBlock) || nonBlock === 0) { + // Node exposes no race-safe fallback on these platforms, so reject all inputs. + throw new SafeInputError("OS no-follow and nonblocking input protection is unavailable"); + } + + let descriptor; + try { + descriptor = fs.openSync(file, fs.constants.O_RDONLY | noFollow | nonBlock); + } catch (error) { + if (error && (error.code === "ELOOP" || error.code === "EMLINK")) throw new SafeInputError("input must not be a symlink"); + throw error; + } + try { + const stat = fs.fstatSync(descriptor); + if (!stat.isFile()) throw new SafeInputError("input must be a regular file"); + if (stat.size > MAX_INPUT_BYTES) throw new SafeInputError(`input exceeds ${MAX_INPUT_BYTES} byte limit`); + + const chunks = []; + const buffer = Buffer.allocUnsafe(64 * 1024); + let bytesRead = 0; + while (true) { + const count = fs.readSync(descriptor, buffer, 0, buffer.length, null); + if (count === 0) break; + bytesRead += count; + if (bytesRead > MAX_INPUT_BYTES) throw new SafeInputError(`input exceeds ${MAX_INPUT_BYTES} byte limit`); + chunks.push(Buffer.from(buffer.subarray(0, count))); + } + try { + return UTF8_DECODER.decode(Buffer.concat(chunks, bytesRead)); + } catch { + throw new SafeInputError("input is not valid UTF-8"); + } + } finally { + fs.closeSync(descriptor); + } +} + +function validateDocument(ledger) { + const errors = createErrorList(); + if (!Array.isArray(ledger)) { + errors.push("$: expected a top-level array"); + return errors; + } + if (ledger.length > MAX_UNITS) { + errors.push(`$: exceeds ${MAX_UNITS} coverage units`); + return errors; + } + + errors.push(...preflightDocument(ledger)); + if (errors.length > 0) return errors; + + const ids = new Map(); + const semantics = new Map(); + let previousId = null; + for (let index = 0; index < ledger.length && errors.length < LIMITS.validationErrors; index++) { + const unit = ledger[index]; + errors.push(...collectUnitErrors(unit, index)); + if (errors.length >= LIMITS.validationErrors) break; + if (!isObject(unit) || typeof unit.coverage_id !== "string") continue; + + const key = hasValidSemanticFields(unit) ? semanticKey(unit) : null; + if (ids.has(unit.coverage_id)) { + const previous = ids.get(unit.coverage_id); + const qualifier = key !== null && previous.key !== null && previous.key !== key + ? "canonical identity collision with different semantic fields" + : "duplicate coverage ID"; + errors.push(`$[${index}].coverage_id: ${qualifier} at $[${previous.index}]`); + } else { + ids.set(unit.coverage_id, { index, key }); + } + if (key !== null && semantics.has(key) && semantics.get(key).id !== unit.coverage_id) { + const previous = semantics.get(key); + errors.push(`$[${index}].canonical_refs: semantic tuple already uses coverage ID ${safeQuote(previous.id)} at $[${previous.index}]`); + } else if (key !== null) { + semantics.set(key, { id: unit.coverage_id, index }); + } + if (previousId !== null && previousId > unit.coverage_id) { + errors.push(`$[${index}].coverage_id: units must be sorted lexicographically`); + } + previousId = unit.coverage_id; + } + return errors; +} + +function run(file) { + if (!file) { + console.error("Usage: node validate-coverage-ledger.cjs <path-to-coverage-ledger.json>"); + return 1; + } + + let contents; + try { + contents = readFileWithinLimit(file); + } catch (error) { + const reason = error instanceof SafeInputError ? error.message : "input could not be opened or read safely"; + console.error(`Failed to read coverage ledger: ${reason}`); + return 1; + } + + let ledger; + try { + preflightJsonText(contents); + } catch (error) { + const reason = error instanceof JsonStructureError ? error.message : "invalid JSON structure"; + console.error(`Failed to parse coverage ledger: ${reason}`); + return 1; + } + try { + ledger = JSON.parse(contents); + } catch { + console.error("Failed to parse coverage ledger: invalid JSON syntax"); + return 1; + } + + let errors; + try { + errors = validateDocument(ledger); + } catch { + console.error("Failed to validate coverage ledger: unexpected validation error"); + return 1; + } + for (const message of errors) console.error("ERROR:", message); + if (errors.length > 0) { + const cap = errors.length === LIMITS.validationErrors ? `; output capped at ${LIMITS.validationErrors}` : ""; + console.error(`FAIL: ${errors.length} validation error(s)${cap}`); + return 1; + } + console.log(`PASS: ${ledger.length} coverage units valid`); + return 0; +} + +module.exports = { + LIMITS, + PATH_FORBIDDEN_CHARACTER, + UNSAFE_DIAGNOSTIC_CHARACTER, + VISIBLE_CONTENT, + WINDOWS_RESERVED_COMPONENT, + canonicalCoverageId, + encodeCanonicalRef, + hasVisibleProse, + isSafeAgentId, + isSafeRelativePath, + preflightJsonText, + readFileWithinLimit, + safeQuote, + validateDocument, +}; + +if (require.main === module) process.exit(run(process.argv[2])); diff --git a/skills/thirdshift-security-audit/validate-coverage-ledger.test.cjs b/skills/thirdshift-security-audit/validate-coverage-ledger.test.cjs new file mode 100644 index 00000000..b1444f57 --- /dev/null +++ b/skills/thirdshift-security-audit/validate-coverage-ledger.test.cjs @@ -0,0 +1,740 @@ +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const { spawnSync } = require("node:child_process"); +const test = require("node:test"); +const { + LIMITS, + canonicalCoverageId, + encodeCanonicalRef, + isSafeAgentId, + isSafeRelativePath, + preflightJsonText, + validateDocument, +} = require("./validate-coverage-ledger.cjs"); + +const validatorPath = path.join(__dirname, "validate-coverage-ledger.cjs"); +const CLI_TIMEOUT_MS = 5000; +const HOSTILE_CLI_TIMEOUT_MS = 15000; +const HAS_SAFE_INPUT_OPEN = Number.isInteger(fs.constants.O_NOFOLLOW) && + fs.constants.O_NOFOLLOW !== 0 && + Number.isInteger(fs.constants.O_NONBLOCK) && + fs.constants.O_NONBLOCK !== 0; + +function unit(overrides = {}) { + const canonicalRefs = overrides.canonical_refs || { + surface: "src/router.ts#POST /users/:id", + boundary: "src/authz.ts#requireOwner", + subsystem: "packages/api", + attack_class: "ATTACK-CLASSES.md#Access control", + }; + const value = { + coverage_id: canonicalCoverageId(canonicalRefs), + canonical_refs: canonicalRefs, + surface: "Update-user route", + boundary: "Object ownership", + subsystem: "API", + attack_class: "Access control", + starting_paths: ["src/router.ts", "src/authz.ts"], + ordinary_attack_class_block: "ATTACK-CLASSES.md#Access control", + selected_companion_blocks: [], + excluded_blocks: [{ block: "WEB-PROTOCOL-AND-AUTH.md#Cache behavior", reason: "The route is not cached." }], + prior_status: "new", + attempts: [], + wave: 1, + status: "planned", + agent_id: null, + reviewed_paths: [], + local_checks: [], + result_fingerprints: [], + unresolved: [], + }; + return Object.assign(value, overrides, { canonical_refs: canonicalRefs }); +} + +function errorsFor(value) { + return validateDocument(value); +} + +function runCli(contents, options = {}) { + const { nodeArgs = [], timeout = CLI_TIMEOUT_MS } = options; + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-")); + const ledgerPath = path.join(directory, "coverage-ledger.json"); + try { + fs.writeFileSync(ledgerPath, contents); + return spawnSync(process.execPath, [...nodeArgs, validatorPath, ledgerPath], { + encoding: "utf8", + timeout, + }); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +function cliOutput(result) { + return `${result.stdout}${result.stderr}`; +} + +const TERMINAL_CONTROL_PAYLOAD = "\u001b\u0007\u0085\u202e"; +const TERMINAL_CONTROL_BYTES = [ + Buffer.from([0x1b]), + Buffer.from([0x07]), + Buffer.from("\u0085"), + Buffer.from("\u202e"), +]; + +function assertNoInjectedControlBytes(output) { + const bytes = Buffer.isBuffer(output) ? output : Buffer.from(output, "utf8"); + for (const marker of TERMINAL_CONTROL_BYTES) { + assert.equal(bytes.indexOf(marker), -1, `found raw control bytes ${marker.toString("hex")}`); + } +} + +function sourceCheck(agentId = "hunter-1", overrides = {}) { + return { + agent_id: agentId, + reviewed_paths: ["src/router.ts"], + invariant: "The route checks object ownership.", + method: "source", + result: "The owner check applies before the update.", + artifact: null, + ...overrides, + }; +} + +function localCheck(agentId = "hunter-1", overrides = {}) { + return sourceCheck(agentId, { + method: "local", + result: "The bounded fixture accepted the other owner's object.", + artifact: `agents/${agentId}/artifacts/result.txt`, + ...overrides, + }); +} + +function archivedAttempt(overrides = {}) { + const value = { + wave: 1, + status: "blocked", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: [], + unresolved: ["The deployed policy is unavailable."], + reassignment_reason: "The critic found an unchecked parallel path.", + }; + return Object.assign(value, overrides); +} + +test("accepts an empty ledger and complete units", () => { + assert.deepEqual(errorsFor([]), []); + assert.deepEqual(errorsFor([unit()]), []); + + const missingAttempts = unit(); + delete missingAttempts.attempts; + assert(errorsFor([missingAttempts]).some((error) => error.includes('missing required field "attempts"'))); + + const covered = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert.deepEqual(errorsFor([covered]), []); +}); + +test("accepts a complete ledger through the CLI", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const result = runCli(JSON.stringify([unit()])); + assert.equal(result.status, 0, cliOutput(result)); + assert.match(result.stdout, /PASS: 1 coverage units valid/); +}); + +test("text preflight ignores structural characters and escapes inside strings", () => { + const value = unit({ + surface: "Route \\ slash [list] {object}, colon: quoted \"value\"", + excluded_blocks: [{ + block: "COMPANION.md#Literal [brackets] {braces}", + reason: "The text contains a backslash \\ before an escaped \"quote\".", + }], + }); + const contents = JSON.stringify([value]); + assert.doesNotThrow(() => preflightJsonText(contents)); + assert.deepEqual(JSON.parse(contents), [value]); + if (HAS_SAFE_INPUT_OPEN) { + const result = runCli(contents); + assert.equal(result.status, 0, cliOutput(result)); + } +}); + +test("text preflight enforces structural cardinality limits", () => { + const depthLimit = LIMITS.nestingDepth; + assert.doesNotThrow(() => preflightJsonText(`${"[".repeat(depthLimit)}0${"]".repeat(depthLimit)}`)); + assert.throws( + () => preflightJsonText(`${"[".repeat(depthLimit + 1)}0${"]".repeat(depthLimit + 1)}`), + /exceeds nesting depth limit 64/, + ); + + const tooManyUnits = `[${"null,".repeat(LIMITS.units)}null]`; + assert.throws(() => preflightJsonText(tooManyUnits), /exceeds 10000 top-level unit limit/); + + const tooManyItems = `[[${"null,".repeat(LIMITS.collectionItems)}null]]`; + assert.throws(() => preflightJsonText(tooManyItems), /exceeds 1000 item array limit/); + + const objectFields = Array.from( + { length: LIMITS.objectFields + 1 }, + (_, index) => `"field${index}":null`, + ).join(","); + assert.throws(() => preflightJsonText(`[{${objectFields}}]`), /exceeds 1000 field object limit/); + + const fullArray = `[${"null,".repeat(LIMITS.collectionItems - 1)}null]`; + const arraysNeeded = Math.floor(LIMITS.preflightValues / (LIMITS.collectionItems + 1)) + 1; + const tooManyValues = `[${Array.from({ length: arraysNeeded }, () => fullArray).join(",")}]`; + assert.throws(() => preflightJsonText(tooManyValues), /exceeds 500000 total value limit/); +}); + +test("text preflight rejects malformed structural truncation cleanly", () => { + assert.throws(() => preflightJsonText("["), /truncated JSON structure/); + assert.throws(() => preflightJsonText("[\"unterminated"), /unterminated JSON string/); + assert.throws(() => preflightJsonText("[{\"field\":1]"), /mismatched JSON containers/); +}); + +test("derives collision-free canonical IDs from exact UTF-8 references", () => { + assert.equal(encodeCanonicalRef("route:POST /users"), "route%3APOST%20%2Fusers"); + assert.notEqual(encodeCanonicalRef("route name"), encodeCanonicalRef("route-name")); + assert.equal( + canonicalCoverageId({ surface: "a", boundary: "b", subsystem: "c", attack_class: "d", lifecycle: "retry" }), + "a::b::c::d::retry", + ); + assert.throws(() => encodeCanonicalRef("e\u0301"), /invalid canonical reference/); + assert.throws(() => encodeCanonicalRef("bad\u0000ref"), /invalid canonical reference/); + assert.throws(() => encodeCanonicalRef("hidden\u200bref"), /invalid canonical reference/); +}); + +test("rejects noncanonical, duplicate, and colliding IDs", () => { + const wrong = unit({ coverage_id: "display-label-slug" }); + assert(errorsFor([wrong]).some((error) => error.includes("expected canonical ID"))); + + const duplicate = unit(); + assert(errorsFor([duplicate, unit()]).some((error) => error.includes("duplicate coverage ID"))); + + const collision = unit(); + const differentMeaning = unit({ surface: "Delete-user route" }); + assert(errorsFor([collision, differentMeaning]).some((error) => error.includes("canonical identity collision"))); +}); + +test("requires canonical references to be own properties", () => { + const inherited = Object.create(unit().canonical_refs); + const value = unit(); + value.canonical_refs = inherited; + assert(errorsFor([value]).some((error) => error.includes("missing required field"))); +}); + +test("rejects aliases for one semantic tuple", () => { + const first = unit(); + const refs = { ...first.canonical_refs, surface: "src/alias.ts#updateUser" }; + const alias = unit({ canonical_refs: refs }); + const ledger = [first, alias].sort((left, right) => left.coverage_id.localeCompare(right.coverage_id)); + assert(errorsFor(ledger).some((error) => error.includes("semantic tuple already uses coverage ID"))); +}); + +test("requires lexicographic order", () => { + const secondRefs = { + surface: "zzz", + boundary: "src/authz.ts#requireOwner", + subsystem: "packages/api", + attack_class: "ATTACK-CLASSES.md#Access control", + }; + assert(errorsFor([unit({ canonical_refs: secondRefs }), unit()]) + .some((error) => error.includes("sorted lexicographically"))); +}); + +test("validates assignment block maps", () => { + const overlap = unit({ + selected_companion_blocks: ["AI-AND-LLM.md#Tool calls"], + excluded_blocks: [{ block: "AI-AND-LLM.md#Tool calls", reason: "Claimed irrelevant." }], + }); + assert(errorsFor([overlap]).some((error) => error.includes("also selected"))); + + const noReason = unit({ excluded_blocks: [{ block: "AI-AND-LLM.md#Tool calls", reason: "" }] }); + assert(errorsFor([noReason]).some((error) => error.includes("reason"))); +}); + +test("requires owned artifacts for local checks and null artifacts for source checks", () => { + const local = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck()], + }); + assert.deepEqual(errorsFor([local]), []); + + const independentlyVerified = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts", "src/authz.ts"], + local_checks: [sourceCheck(), localCheck("verifier-1", { reviewed_paths: ["src/authz.ts"] })], + }); + assert.deepEqual(errorsFor([independentlyVerified]), []); + + const unownedPath = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts", "src/authz.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([unownedPath]).some((error) => error.includes("has no check owner"))); + + for (const [checkAgentId, artifact] of [ + [null, "agents/hunter-1/artifacts/result.txt"], + ["hunter-1", null], + ["hunter-1", "result.txt"], + ["hunter-1", "agents/hunter-2/artifacts/result.txt"], + ["../hunter", "agents/../hunter/artifacts/result.txt"], + ]) { + const value = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck(checkAgentId, { artifact })], + }); + assert.notEqual(errorsFor([value]).length, 0, `${checkAgentId}: ${artifact}`); + } + + const unownedSource = unit({ + status: "blocked", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["The boundary behavior is not source-visible."], + }); + assert(errorsFor([unownedSource]).some((error) => error.includes("unit with status \"blocked\" requires a canonical lowercase agent ID"))); + + const sourceWithArtifact = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { artifact: "agents/hunter-1/artifacts/source.txt" })], + }); + assert(errorsFor([sourceWithArtifact]).some((error) => error.includes("source-only check must use null"))); +}); + +test("requires canonical lowercase filesystem-safe agent IDs", () => { + for (const value of ["hunter-1", "verifier_2", "a0"]) assert.equal(isSafeAgentId(value), true, value); + for (const value of ["Hunter-1", "hunter.1", "hunter-1.", "hunter ", "con", "prn", "aux", "nul", "com1", "lpt9", "../hunter"]) { + assert.equal(isSafeAgentId(value), false, value); + } + + const caseAlias = unit({ status: "in_progress", agent_id: "Hunter-1" }); + assert(errorsFor([caseAlias]).some((error) => error.includes("canonical lowercase agent ID"))); +}); + +test("enforces state evidence", () => { + assert(errorsFor([unit({ status: "in_progress" })]).some((error) => error.includes("unit with status \"in_progress\" requires"))); + assert(errorsFor([unit({ status: "blocked" })]).some((error) => error.includes("unresolved"))); + assert(errorsFor([unit({ status: "candidate" })]).some((error) => error.includes("reviewed_paths"))); + + assert.deepEqual(errorsFor([unit({ status: "in_progress", agent_id: "hunter-1" })]), []); + const inProgressEvidence = unit({ + status: "in_progress", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([inProgressEvidence]).some((error) => error.includes("must keep this array empty"))); + + const assignedPlanned = unit({ + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + }); + assert(errorsFor([assignedPlanned]).some((error) => error.includes("planned unit must be unassigned"))); + + const candidate = unit({ + status: "candidate", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { invariant: "Ownership is required.", result: "No check exists." })], + result_fingerprints: ["src-router-missing-owner-check"], + unresolved: ["validation_budget_exhausted"], + }); + assert.deepEqual(errorsFor([candidate]), []); + + const blocked = unit({ + status: "blocked", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["The deployed policy is unavailable."], + }); + assert.deepEqual(errorsFor([blocked]), []); + const blockedFingerprint = { ...blocked, result_fingerprints: ["forbidden-fingerprint"] }; + assert(errorsFor([blockedFingerprint]).some((error) => error.includes("result_fingerprints"))); + + for (const status of ["not_applicable", "out_of_scope", "deferred"]) { + assert.deepEqual(errorsFor([unit({ status, unresolved: ["Reason recorded."] })]), []); + const invalid = unit({ + status, + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: ["forbidden-fingerprint"], + unresolved: ["Reason recorded."], + }); + const errors = errorsFor([invalid]); + assert(errors.some((error) => error.includes("must be unassigned")), status); + assert(errors.some((error) => error.includes("reviewed_paths")), status); + assert(errors.some((error) => error.includes("result_fingerprints")), status); + } + + const coveredFingerprint = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + result_fingerprints: ["forbidden-fingerprint"], + }); + assert(errorsFor([coveredFingerprint]).some((error) => error.includes("result_fingerprints"))); + + const coveredUnresolved = unit({ + status: "covered", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck()], + unresolved: ["Unexpected unresolved claim."], + }); + assert(errorsFor([coveredUnresolved]).some((error) => error.includes("unresolved"))); +}); + +test("archives prior evidence when a critic assigns a fresh owner", () => { + const reassigned = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "in_progress", + agent_id: "hunter-2", + }); + assert.deepEqual(errorsFor([reassigned]), []); + + const finalClosure = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "covered", + agent_id: "hunter-2", + reviewed_paths: ["src/authz.ts"], + local_checks: [sourceCheck("hunter-2", { reviewed_paths: ["src/authz.ts"] })], + }); + assert.deepEqual(errorsFor([finalClosure]), []); +}); + +test("preserves candidate provenance when reassignment must be deferred", () => { + const candidateAttempt = archivedAttempt({ + status: "candidate", + result_fingerprints: ["src-router-missing-owner-check"], + unresolved: ["validation_budget_exhausted"], + }); + const deferred = unit({ + attempts: [candidateAttempt], + wave: 2, + status: "deferred", + unresolved: ["quick_profile_final_critic"], + }); + assert.deepEqual(errorsFor([deferred]), []); +}); + +test("rejects reassignment owner reuse and evidence mixing", () => { + const reusedOwner = unit({ + attempts: [archivedAttempt()], + wave: 2, + status: "in_progress", + agent_id: "hunter-1", + }); + assert(errorsFor([reusedOwner]).some((error) => error.includes("current assignment owner must be fresh"))); + + const mixedEvidence = unit({ + attempts: [archivedAttempt({ local_checks: [localCheck()] })], + wave: 2, + status: "covered", + agent_id: "hunter-2", + reviewed_paths: ["src/router.ts"], + local_checks: [localCheck()], + }); + const errors = errorsFor([mixedEvidence]); + assert(errors.some((error) => error.includes("prior assignment owner evidence must remain"))); + assert(errors.some((error) => error.includes("artifact from an archived attempt cannot be reused"))); + + const mixedHistory = unit({ + attempts: [ + archivedAttempt({ local_checks: [localCheck()] }), + archivedAttempt({ + wave: 2, + agent_id: "hunter-2", + local_checks: [localCheck()], + }), + ], + wave: 3, + status: "in_progress", + agent_id: "hunter-3", + }); + const historyErrors = errorsFor([mixedHistory]); + assert(historyErrors.some((error) => error.includes("prior assignment owner evidence must remain in its earlier attempt"))); + assert(historyErrors.some((error) => error.includes("artifact from an earlier attempt cannot be reused"))); + + const unordered = unit({ + attempts: [archivedAttempt(), archivedAttempt({ + wave: 1, + agent_id: "hunter-2", + local_checks: [sourceCheck("hunter-2")], + })], + wave: 3, + status: "in_progress", + agent_id: "hunter-3", + }); + assert(errorsFor([unordered]).some((error) => error.includes("strictly increasing"))); +}); + +test("rejects unsafe paths and malformed fingerprints", () => { + for (const value of [ + "/etc/passwd", + "../src/file.js", + "src/../file.js", + "src/con.txt", + "src/PRN", + "src/AUX.c", + "src/NUL", + "src/CLOCK$.txt", + "src/conin$.txt", + "src/conout$", + "src/COM1.log", + "src/lpt9", + "src/COM\u00b9.log", + "src/COM\u00b2.log", + "src/COM\u00b3.log", + "src/lpt\u00b9", + "src/lpt\u00b2", + "src/lpt\u00b3", + "src/file.js.", + "C:/src/file.js", + "src/file\n.js", + "src/file\u0085.js", + "src/file\u2028.js", + "src/file\u200b.js", + "src/file\u034f.js", + "src/file\ufe0f.js", + ]) { + assert.equal(isSafeRelativePath(value), false, value); + } + assert.equal(isSafeRelativePath("src/handler.js"), true); + assert.equal(isSafeRelativePath("src/caf\u00e9/handler.js"), true); + assert(errorsFor([unit({ starting_paths: ["../src/router.ts"] })]).some((error) => error.includes("repository-relative path"))); + assert(errorsFor([unit({ + status: "candidate", + agent_id: "hunter-1", + reviewed_paths: ["src/router.ts"], + local_checks: [sourceCheck("hunter-1", { invariant: "Ownership is required.", result: "No check exists." })], + result_fingerprints: ["not stable"], + })]).some((error) => error.includes("invalid fingerprint"))); +}); + +test("rejects format, default-ignorable, and invalid-scalar prose", () => { + for (const invisible of ["\u200b", "\u034f", "\ufe0f", "\ud800"]) { + assert(errorsFor([unit({ surface: invisible })]).some((error) => error.includes("surface")), JSON.stringify(invisible)); + assert(errorsFor([unit({ + excluded_blocks: [{ block: "ATTACK-CLASSES.md#Access control", reason: invisible }], + })]).some((error) => error.includes("reason")), JSON.stringify(invisible)); + } +}); + +test("quotes input-derived controls in direct validation errors", () => { + const invalidStatus = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + const invalidPath = `src/${TERMINAL_CONTROL_PAYLOAD}.js`; + const value = unit({ + status: invalidStatus, + agent_id: "hunter-1", + reviewed_paths: [invalidPath], + local_checks: [sourceCheck()], + result_fingerprints: ["force-state-error"], + }); + const output = errorsFor([value]).join("\n"); + + assert.match(output, /\$\[0\]\.status/); + assert.match(output, /\$\[0\]\.reviewed_paths/); + assert.match(output, /\\u001b/); + assert.match(output, /\\u0007/); + assert.match(output, /\\u0085/); + assert.match(output, /\\u202e/); + assertNoInjectedControlBytes(output); +}); + +test("quotes input-derived controls in CLI validation errors", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const value = unit({ + status: `invalid-${TERMINAL_CONTROL_PAYLOAD}`, + result_fingerprints: ["force-state-error"], + }); + const result = runCli(JSON.stringify([value])); + + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /\$\[0\]\.status/); + assert.match(result.stderr, /\\u001b/); + assertNoInjectedControlBytes(result.stderr); +}); + +test("returns a generic syntax error without parser-supplied controls", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const malformed = Buffer.concat([ + Buffer.from("["), + Buffer.from(TERMINAL_CONTROL_PAYLOAD), + Buffer.from("]"), + ]); + const result = runCli(malformed); + + assert.equal(result.status, 1, cliOutput(result)); + assert.equal(result.stderr, "Failed to parse coverage ledger: invalid JSON syntax\n"); + assertNoInjectedControlBytes(result.stderr); +}); + +test("does not reflect controls from a failed CLI input path", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-path-")); + const missingPath = path.join(directory, `missing-${TERMINAL_CONTROL_PAYLOAD}.json`); + try { + const result = spawnSync(process.execPath, [validatorPath, missingPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /Failed to read coverage ledger:/); + assertNoInjectedControlBytes(result.stderr); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects invalid UTF-8 through the CLI", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const encoded = Buffer.from(JSON.stringify([unit()])); + const marker = Buffer.from("Update-user route"); + const markerOffset = encoded.indexOf(marker); + assert.notEqual(markerOffset, -1); + const malformed = Buffer.concat([ + encoded.subarray(0, markerOffset), + Buffer.from([0x80]), + encoded.subarray(markerOffset + marker.length), + ]); + + const result = runCli(malformed); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /input is not valid UTF-8/); + assert.doesNotMatch(output, /TypeError|stack|at validate-coverage-ledger/i); +}); + +test("rejects a FIFO through the CLI without blocking", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-fifo-")); + const fifoPath = path.join(directory, "coverage-ledger.json"); + try { + const created = spawnSync("mkfifo", [fifoPath], { encoding: "utf8", timeout: CLI_TIMEOUT_MS }); + assert.equal(created.status, 0, cliOutput(created)); + + const result = spawnSync(process.execPath, [validatorPath, fifoPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must be a regular file/); + assert.doesNotMatch(output, /stack|at validate-coverage-ledger/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects a symlink through the CLI", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-coverage-ledger-symlink-")); + const targetPath = path.join(directory, "target.json"); + const symlinkPath = path.join(directory, "coverage-ledger.json"); + try { + fs.writeFileSync(targetPath, JSON.stringify([unit()])); + fs.symlinkSync(targetPath, symlinkPath); + const result = spawnSync(process.execPath, [validatorPath, symlinkPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must not be a symlink/); + assert.doesNotMatch(output, /stack|at validate-coverage-ledger/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("rejects deeply nested input without recursion failure", () => { + let nested = 0; + for (let depth = 0; depth < 20000; depth++) nested = [nested]; + assert(errorsFor(nested).some((error) => error.includes("exceeds nesting depth limit 64"))); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(`${"[".repeat(20000)}0${"]".repeat(20000)}`); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /exceeds nesting depth limit 64/); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); +}); + +test("rejects multi-megabyte nesting under a constrained Node heap", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const openContainers = "[".repeat(2000000); + const cases = [ + openContainers, + `${openContainers}0${"]".repeat(2000000)}`, + ]; + for (const contents of cases) { + const result = runCli(contents, { + nodeArgs: ["--max-old-space-size=64"], + timeout: HOSTILE_CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /exceeds nesting depth limit 64/); + assert.doesNotMatch(output, /heap out of memory|allocation failed|RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); + } +}); + +test("caps malformed 10000-unit validation output", () => { + assert.equal(errorsFor(Array.from({ length: LIMITS.units }, () => null)).length, LIMITS.validationErrors); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(JSON.stringify(Array.from({ length: LIMITS.units }, () => null))); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-coverage-ledger/i); +}); + +test("rejects malformed top-level data and excessive unit counts", () => { + assert.deepEqual(errorsFor({ units: [] }), ["$: expected a top-level array"]); + const tooMany = Array.from({ length: 10001 }, () => null); + const errors = errorsFor(tooMany); + assert.deepEqual(errors, ["$: exceeds 10000 coverage units"]); + + const oversizedCollection = unit({ extra: Array.from({ length: LIMITS.collectionItems + 1 }, () => null) }); + assert(errorsFor([oversizedCollection]).some((error) => error.includes("exceeds 1000 entries"))); +}); + +test("accepts a canonical ID derived from near-maximum multibyte references", () => { + const canonicalRefs = { + surface: "\u6f22".repeat(1024), + boundary: "\u00e9".repeat(1024), + subsystem: "packages/api", + attack_class: "\u6f22".repeat(1023) + "\u00e9", + }; + const value = unit({ canonical_refs: canonicalRefs }); + assert(value.coverage_id.length > 16384, `coverage_id length ${value.coverage_id.length}`); + assert(value.coverage_id.length <= 65536, `coverage_id length ${value.coverage_id.length}`); + assert.deepEqual(errorsFor([value]), []); + + if (HAS_SAFE_INPUT_OPEN) { + const result = runCli(JSON.stringify([value])); + assert.equal(result.status, 0, cliOutput(result)); + } +}); diff --git a/skills/thirdshift-security-audit/validate-findings.cjs b/skills/thirdshift-security-audit/validate-findings.cjs new file mode 100755 index 00000000..2843fece --- /dev/null +++ b/skills/thirdshift-security-audit/validate-findings.cjs @@ -0,0 +1,773 @@ +#!/usr/bin/env node + +/** + * Validates findings.json against report-schema.json. + * Usage: node validate-findings.cjs <path-to-findings.json> + * + * This is a dependency-free interpreter for the JSON Schema keywords used by + * report-schema.json, plus finding-specific checks that are clearer in code. + */ + +const fs = require("fs"); +const path = require("path"); +const { TextDecoder } = require("util"); + +const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value, key); +const SUPPORTED_TYPES = new Set(["object", "array", "string", "integer", "number", "boolean", "null"]); +const SUPPORTED_KEYWORDS = new Set([ + "$comment", + "additionalProperties", + "const", + "description", + "enum", + "items", + "minimum", + "minItems", + "minLength", + "oneOf", + "pattern", + "properties", + "required", + "type", + "uniqueItems", + "visibleContent", +]); +const SEVERITY_RANK = new Map([ + ["informational", 0], + ["low", 1], + ["medium", 2], + ["high", 3], + ["critical", 4], +]); +const LIMITS = Object.freeze({ + inputBytes: 5 * 1024 * 1024, + nestingDepth: 64, + arrayItems: 1000, + canonicalKeyBytes: 1024 * 1024, + uniqueSetBytes: 5 * 1024 * 1024, + validationErrors: 100, +}); +const VISIBLE_CONTENT = /[^\p{White_Space}\p{Cc}\p{Cf}\p{Default_Ignorable_Code_Point}]/u; +const PATH_FORBIDDEN_CHARACTER = /[\p{Cc}\p{Cf}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/u; +const WINDOWS_RESERVED_COMPONENT = /^(?:con|prn|aux|nul|clock\$|conin\$|conout\$|com[1-9\u00b9\u00b2\u00b3]|lpt[1-9\u00b9\u00b2\u00b3])(?:\.|$)/iu; +const UTF8_DECODER = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); +const UNSAFE_DIAGNOSTIC_CHARACTER = /[\p{Cc}\p{Cf}\p{Cs}\p{Zl}\p{Zp}\p{Default_Ignorable_Code_Point}]/gu; +const MAX_DIAGNOSTIC_STRING_LENGTH = 256; + +class JsonStructureError extends Error {} +class SafeInputError extends Error {} + +function escapeUnsafeDiagnosticCharacters(value) { + return String(value).replace(UNSAFE_DIAGNOSTIC_CHARACTER, (character) => { + const codePoint = character.codePointAt(0); + return codePoint <= 0xffff + ? `\\u${codePoint.toString(16).padStart(4, "0")}` + : `\\u{${codePoint.toString(16)}}`; + }); +} + +function safeQuote(value) { + let serialized; + if (typeof value === "string") { + const clipped = value.length > MAX_DIAGNOSTIC_STRING_LENGTH + ? `${value.slice(0, MAX_DIAGNOSTIC_STRING_LENGTH)}...` + : value; + serialized = JSON.stringify(clipped); + } else if (value === null || typeof value === "boolean") { + serialized = String(value); + } else if (typeof value === "number" && Number.isFinite(value)) { + serialized = String(value); + } else { + serialized = `"<${Array.isArray(value) ? "array" : typeof value}>"`; + } + return escapeUnsafeDiagnosticCharacters(serialized); +} + +function propertyPath(base, key) { + return /^[A-Za-z_][A-Za-z0-9_]*$/.test(key) + ? `${base}.${key}` + : `${base}[${safeQuote(key)}]`; +} + +function createErrorList() { + const errors = []; + Object.defineProperty(errors, "push", { + value(...messages) { + const remaining = LIMITS.validationErrors - this.length; + if (remaining > 0) { + Array.prototype.push.apply(this, messages.slice(0, remaining).map(escapeUnsafeDiagnosticCharacters)); + } + return this.length; + }, + }); + return errors; +} + +function typeOf(value) { + if (Array.isArray(value)) return "array"; + if (value === null) return "null"; + return typeof value; +} + +function deepEqual(left, right) { + if (left === right) return true; + if (typeOf(left) !== typeOf(right)) return false; + if (Array.isArray(left)) { + return left.length === right.length && left.every((value, index) => deepEqual(value, right[index])); + } + if (left !== null && typeof left === "object") { + const leftKeys = Object.keys(left); + const rightKeys = Object.keys(right); + return leftKeys.length === rightKeys.length && + leftKeys.every((key) => hasOwn(right, key) && deepEqual(left[key], right[key])); + } + return false; +} + +function codePointLength(value) { + let length = 0; + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff && index < value.length) { + const second = value.charCodeAt(index); + if (second >= 0xdc00 && second <= 0xdfff) index++; + } + length++; + } + return length; +} + +function hasValidUnicodeScalarValues(value) { + let index = 0; + while (index < value.length) { + const first = value.charCodeAt(index++); + if (first >= 0xd800 && first <= 0xdbff) { + if (index >= value.length) return false; + const second = value.charCodeAt(index++); + if (second < 0xdc00 || second > 0xdfff) return false; + } else if (first >= 0xdc00 && first <= 0xdfff) { + return false; + } + } + return true; +} + +function hasVisibleProse(value) { + return hasValidUnicodeScalarValues(value) && VISIBLE_CONTENT.test(value); +} + +function canonicalKey(value) { + const chunks = []; + let bytes = 0; + + function append(chunk) { + bytes += Buffer.byteLength(chunk); + if (bytes > LIMITS.canonicalKeyBytes) { + throw new Error(`canonical key exceeds ${LIMITS.canonicalKeyBytes} byte limit`); + } + chunks.push(chunk); + } + + function encode(item) { + const type = typeOf(item); + if (type === "null") { + append("null"); + } else if (type === "string") { + append(`string:${JSON.stringify(item)}`); + } else if (type === "number") { + append(`number:${Object.is(item, -0) ? "0" : String(item)}`); + } else if (type === "boolean") { + append(`boolean:${item ? "true" : "false"}`); + } else if (type === "array") { + append("array:["); + item.forEach((entry, index) => { + if (index > 0) append(","); + encode(entry); + }); + append("]"); + } else if (type === "object") { + append("object:{"); + Object.keys(item).sort().forEach((key, index) => { + if (index > 0) append(","); + append(JSON.stringify(key)); + append(":"); + encode(item[key]); + }); + append("}"); + } else { + append(`${type}:${String(item)}`); + } + } + + encode(value); + return { key: chunks.join(""), bytes }; +} + +function collectDataLimitErrors(value, location = "$data") { + const stack = [{ value, location, depth: value !== null && typeof value === "object" ? 1 : 0 }]; + const seen = new WeakSet(); + + while (stack.length > 0) { + const current = stack.pop(); + if (current.value === null || typeof current.value !== "object") continue; + if (current.depth > LIMITS.nestingDepth) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: exceeds ${LIMITS.nestingDepth} level nesting depth limit`)]; + } + if (seen.has(current.value)) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: input must not contain repeated or cyclic object references`)]; + } + seen.add(current.value); + + if (Array.isArray(current.value)) { + if (current.value.length > LIMITS.arrayItems) { + return [escapeUnsafeDiagnosticCharacters(`${current.location}: exceeds ${LIMITS.arrayItems} item array limit`)]; + } + for (let index = current.value.length - 1; index >= 0; index--) { + const child = current.value[index]; + if (child !== null && typeof child === "object") { + stack.push({ value: child, location: `${current.location}[${index}]`, depth: current.depth + 1 }); + } + } + } else { + const keys = Object.keys(current.value); + for (let index = keys.length - 1; index >= 0; index--) { + const key = keys[index]; + const child = current.value[key]; + if (child !== null && typeof child === "object") { + stack.push({ value: child, location: propertyPath(current.location, key), depth: current.depth + 1 }); + } + } + } + } + + return []; +} + +function collectSchemaErrors(schema, location = "schema") { + const errors = createErrorList(); + + function check(node, p) { + if (node === null || typeof node !== "object" || Array.isArray(node)) { + errors.push(`${p}: schema must be an object`); + return; + } + + for (const key of Object.keys(node)) { + if (!SUPPORTED_KEYWORDS.has(key)) errors.push(`${p}: unsupported schema keyword ${safeQuote(key)}`); + } + + if (hasOwn(node, "$comment") && typeof node.$comment !== "string") { + errors.push(`${p}.$comment: expected string`); + } + if (hasOwn(node, "description") && typeof node.description !== "string") { + errors.push(`${p}.description: expected string`); + } + if (hasOwn(node, "type") && (!SUPPORTED_TYPES.has(node.type))) { + errors.push(`${p}.type: unsupported type ${safeQuote(node.type)}`); + } + if (hasOwn(node, "properties")) { + if (node.properties === null || typeof node.properties !== "object" || Array.isArray(node.properties)) { + errors.push(`${p}.properties: expected object`); + } else { + for (const key of Object.keys(node.properties)) check(node.properties[key], propertyPath(`${p}.properties`, key)); + } + } + if (hasOwn(node, "required")) { + if (!Array.isArray(node.required) || node.required.some((key) => typeof key !== "string")) { + errors.push(`${p}.required: expected an array of strings`); + } else if (new Set(node.required).size !== node.required.length) { + errors.push(`${p}.required: entries must be unique`); + } + } + if (hasOwn(node, "additionalProperties") && typeof node.additionalProperties !== "boolean") { + errors.push(`${p}.additionalProperties: only boolean values are supported`); + } + if (hasOwn(node, "enum")) { + if (!Array.isArray(node.enum) || node.enum.length === 0) { + errors.push(`${p}.enum: expected a non-empty array`); + } else { + const seen = new Set(); + for (const value of node.enum) { + let key; + try { + key = canonicalKey(value).key; + } catch (error) { + errors.push(`${p}.enum: ${error.message}`); + break; + } + if (seen.has(key)) { + errors.push(`${p}.enum: entries must be unique`); + break; + } + seen.add(key); + } + } + } + if (hasOwn(node, "items")) check(node.items, `${p}.items`); + for (const keyword of ["minItems", "minLength"]) { + if (hasOwn(node, keyword) && (!Number.isInteger(node[keyword]) || node[keyword] < 0)) { + errors.push(`${p}.${keyword}: expected a non-negative integer`); + } + } + if (hasOwn(node, "minimum") && (typeof node.minimum !== "number" || !Number.isFinite(node.minimum))) { + errors.push(`${p}.minimum: expected a finite number`); + } + if (hasOwn(node, "pattern")) { + if (typeof node.pattern !== "string") { + errors.push(`${p}.pattern: expected string`); + } else { + try { + new RegExp(node.pattern); + } catch (error) { + errors.push(`${p}.pattern: invalid regular expression`); + } + } + } + if (hasOwn(node, "uniqueItems") && typeof node.uniqueItems !== "boolean") { + errors.push(`${p}.uniqueItems: expected boolean`); + } + if (hasOwn(node, "visibleContent")) { + if (typeof node.visibleContent !== "boolean") { + errors.push(`${p}.visibleContent: expected boolean`); + } else if (node.visibleContent === true && node.type !== "string") { + errors.push(`${p}.visibleContent: requires type "string"`); + } + } + if (hasOwn(node, "oneOf")) { + if (!Array.isArray(node.oneOf) || node.oneOf.length === 0) { + errors.push(`${p}.oneOf: expected a non-empty array`); + } else { + node.oneOf.forEach((branch, index) => check(branch, `${p}.oneOf[${index}]`)); + } + } + } + + check(schema, location); + return errors; +} + +function findDiscriminator(schema) { + if (!hasOwn(schema, "properties") || typeof schema.properties !== "object") return null; + for (const key of Object.keys(schema.properties)) { + const subSchema = schema.properties[key]; + if (subSchema && typeof subSchema === "object" && hasOwn(subSchema, "const")) { + return { key, value: subSchema.const }; + } + } + return null; +} + +function validate(value, schema, p, errors) { + if (errors.length >= LIMITS.validationErrors) return; + if (hasOwn(schema, "oneOf")) { + const results = schema.oneOf.map((branch) => collectUnchecked(value, branch, p)); + const passingIndexes = results + .map((branchErrors, index) => branchErrors.length === 0 ? index : -1) + .filter((index) => index !== -1); + + if (passingIndexes.length !== 1) { + errors.push(`${p}: must match exactly one schema in oneOf; matched ${passingIndexes.length}`); + if (passingIndexes.length === 0 && value !== null && typeof value === "object" && !Array.isArray(value)) { + const matchingDiscriminators = schema.oneOf + .map((branch, index) => ({ discriminator: findDiscriminator(branch), index })) + .filter(({ discriminator }) => discriminator && hasOwn(value, discriminator.key) && deepEqual(value[discriminator.key], discriminator.value)); + if (matchingDiscriminators.length === 1) { + errors.push(...results[matchingDiscriminators[0].index]); + } + } + } + } + + if (hasOwn(schema, "const") && !deepEqual(value, schema.const)) { + errors.push(`${p}: must equal ${safeQuote(schema.const)}, got ${safeQuote(value)}`); + } + if (hasOwn(schema, "enum") && !schema.enum.some((allowed) => deepEqual(value, allowed))) { + const allowed = schema.enum.map(safeQuote).join(", "); + errors.push(`${p}: invalid value ${safeQuote(value)} (expected one of ${allowed})`); + } + + if (hasOwn(schema, "type") && typeOf(value) !== schema.type && !(schema.type === "integer" && typeOf(value) === "number" && Number.isInteger(value))) { + errors.push(`${p}: expected ${schema.type}, got ${typeOf(value)}`); + return; + } + + if (typeOf(value) === "object") { + for (const req of hasOwn(schema, "required") ? schema.required : []) { + if (!hasOwn(value, req)) errors.push(`${p}: missing required field ${safeQuote(req)}`); + } + for (const key of Object.keys(value)) { + if (hasOwn(schema, "properties") && hasOwn(schema.properties, key)) { + validate(value[key], schema.properties[key], propertyPath(p, key), errors); + } else if (hasOwn(schema, "additionalProperties") && schema.additionalProperties === false) { + errors.push(`${p}: unexpected field ${safeQuote(key)}`); + } + } + } + + if (Array.isArray(value)) { + if (hasOwn(schema, "minItems") && value.length < schema.minItems) { + errors.push(`${p}: must have at least ${schema.minItems} item(s), got ${value.length}`); + } + if (hasOwn(schema, "uniqueItems") && schema.uniqueItems === true) { + const seen = new Set(); + let setBytes = 0; + for (let i = 0; i < value.length; i++) { + let canonical; + try { + canonical = canonicalKey(value[i]); + } catch (error) { + errors.push(`${p}[${i}]: ${error.message}`); + break; + } + if (seen.has(canonical.key)) { + errors.push(`${p}: items must be unique; duplicate at index ${i}`); + continue; + } + setBytes += canonical.bytes; + if (setBytes > LIMITS.uniqueSetBytes) { + errors.push(`${p}: canonical uniqueness set exceeds ${LIMITS.uniqueSetBytes} byte limit`); + break; + } + seen.add(canonical.key); + } + } + if (hasOwn(schema, "items")) { + value.forEach((item, index) => validate(item, schema.items, `${p}[${index}]`, errors)); + } + } + + if (typeof value === "string") { + if (hasOwn(schema, "minLength") && codePointLength(value) < schema.minLength) { + errors.push(`${p}: must have at least ${schema.minLength} character(s)`); + } + if (schema.visibleContent === true) { + if (!hasValidUnicodeScalarValues(value)) { + errors.push(`${p}: must contain only valid Unicode scalar values`); + } else if (!VISIBLE_CONTENT.test(value)) { + errors.push(`${p}: must contain a visible character`); + } + } + if (hasOwn(schema, "pattern") && !(new RegExp(schema.pattern).test(value))) { + errors.push(`${p}: must match pattern ${JSON.stringify(schema.pattern)}`); + } + } + + if (typeof value === "number" && hasOwn(schema, "minimum") && value < schema.minimum) { + errors.push(`${p}: must be at least ${schema.minimum}, got ${value}`); + } +} + +function collectUnchecked(value, schema, p) { + const errors = createErrorList(); + validate(value, schema, p, errors); + return errors; +} + +function collect(value, schema, p = "$data") { + const limitErrors = collectDataLimitErrors(value, p); + if (limitErrors.length > 0) return limitErrors; + return collectUnchecked(value, schema, p); +} + +function isSafeRelativeSourcePath(value) { + if (typeof value !== "string" || value.length === 0 || !hasValidUnicodeScalarValues(value) || value.trim() !== value || PATH_FORBIDDEN_CHARACTER.test(value) || value.includes("\\") || value.includes(":")) return false; + if (path.posix.isAbsolute(value) || path.win32.isAbsolute(value) || /^[A-Za-z]:/.test(value) || value.startsWith("~")) return false; + const segments = value.split("/"); + return segments.every((segment) => + segment !== "" && + segment !== "." && + segment !== ".." && + !/[ .]$/u.test(segment) && + !WINDOWS_RESERVED_COMPONENT.test(segment)); +} + +function collectFindingSemanticErrors(findings) { + const errors = createErrorList(); + if (!Array.isArray(findings)) return errors; + + const fingerprints = new Map(); + let previousFingerprint = null; + findings.forEach((finding, index) => { + if (errors.length >= LIMITS.validationErrors) return; + if (!finding || typeof finding !== "object" || Array.isArray(finding)) return; + const base = `$[${index}]`; + + if (hasOwn(finding, "fingerprint") && typeof finding.fingerprint === "string") { + if (fingerprints.has(finding.fingerprint)) { + errors.push(`${base}.fingerprint: duplicate of $[${fingerprints.get(finding.fingerprint)}].fingerprint`); + } else { + fingerprints.set(finding.fingerprint, index); + } + if (previousFingerprint !== null && previousFingerprint > finding.fingerprint) { + errors.push(`${base}.fingerprint: findings must be sorted lexicographically`); + } + previousFingerprint = finding.fingerprint; + } + + for (const field of ["trace", "evidence"]) { + if (!hasOwn(finding, field) || !Array.isArray(finding[field])) continue; + finding[field].forEach((entry, entryIndex) => { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) return; + if (hasOwn(entry, "line") && (!Number.isInteger(entry.line) || entry.line < 1)) { + errors.push(`${base}.${field}[${entryIndex}].line: must be a positive integer`); + } + if (hasOwn(entry, "file") && !isSafeRelativeSourcePath(entry.file)) { + errors.push(`${base}.${field}[${entryIndex}].file: must be a safe repository-relative source path`); + } + }); + } + if (finding.remediation && Array.isArray(finding.remediation.code_changes)) { + finding.remediation.code_changes.forEach((change, changeIndex) => { + if (change && hasOwn(change, "file_name") && !isSafeRelativeSourcePath(change.file_name)) { + errors.push(`${base}.remediation.code_changes[${changeIndex}].file_name: must be a safe repository-relative source path`); + } + }); + } + + if (Array.isArray(finding.trace) && finding.trace.length === 1) { + const kind = finding.trace[0] && finding.trace[0].kind; + if (kind !== "entrypoint" && kind !== "sink") { + errors.push(`${base}.trace[0].kind: a one-line trace must be "entrypoint" or "sink"`); + } + } else if (Array.isArray(finding.trace) && finding.trace.length > 1) { + const last = finding.trace.length - 1; + if (finding.trace[0] && finding.trace[0].kind !== "entrypoint") { + errors.push(`${base}.trace[0].kind: must be "entrypoint", got ${safeQuote(finding.trace[0].kind)}`); + } + if (finding.trace[last] && finding.trace[last].kind !== "sink") { + errors.push(`${base}.trace[${last}].kind: must be "sink", got ${safeQuote(finding.trace[last].kind)}`); + } + for (let traceIndex = 1; traceIndex < last; traceIndex++) { + if (finding.trace[traceIndex] && finding.trace[traceIndex].kind !== "propagation") { + errors.push(`${base}.trace[${traceIndex}].kind: must be "propagation", got ${safeQuote(finding.trace[traceIndex].kind)}`); + } + } + } + + const verdict = finding.verdict; + if (verdict === "confirmed") { + for (const forbidden of ["claimed_root_cause", "blockers", "validation_plan", "reason"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: confirmed finding must not contain ${safeQuote(forbidden)}`); + } + if (!finding.execution || typeof finding.execution !== "object" || typeof finding.execution.observed_result !== "string" || !hasVisibleProse(finding.execution.observed_result)) { + errors.push(`${base}: confirmed finding requires a visible execution observed_result`); + } + if (!finding.remediation || typeof finding.remediation !== "object" || typeof finding.remediation.strategy !== "string" || !hasVisibleProse(finding.remediation.strategy)) { + errors.push(`${base}: confirmed finding requires visible remediation`); + } + const overall = finding.severity && finding.severity.overall_severity; + const impact = finding.severity && finding.severity.impact && finding.severity.impact.score; + if (SEVERITY_RANK.has(overall) && SEVERITY_RANK.has(impact) && SEVERITY_RANK.get(overall) > SEVERITY_RANK.get(impact)) { + errors.push(`${base}.severity.overall_severity: cannot exceed demonstrated impact ${safeQuote(impact)}`); + } + } else if (verdict === "needs_validation") { + if (hasOwn(finding, "severity")) errors.push(`${base}: needs_validation finding must not contain "severity"`); + for (const forbidden of ["execution", "remediation", "reason", "root_cause"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: needs_validation finding must not contain ${safeQuote(forbidden)}`); + } + const plan = finding.validation_plan; + const hasLocalPlan = plan && typeof plan.local === "string" && hasVisibleProse(plan.local); + const hasDeploymentPlan = plan && typeof plan.deployment === "string" && hasVisibleProse(plan.deployment); + if (!hasLocalPlan && !hasDeploymentPlan) { + errors.push(`${base}.validation_plan: requires at least one visible local or deployment plan`); + } + } else if (verdict === "rejected") { + for (const forbidden of ["severity", "execution", "remediation", "blockers", "validation_plan", "root_cause"]) { + if (hasOwn(finding, forbidden)) errors.push(`${base}: rejected finding must not contain ${safeQuote(forbidden)}`); + } + } + }); + + return errors; +} + +function validateDocument(findings, schema) { + const schemaErrors = collectSchemaErrors(schema); + if (schemaErrors.length > 0) return schemaErrors; + const limitErrors = collectDataLimitErrors(findings, "$"); + if (limitErrors.length > 0) return limitErrors; + const errors = collectUnchecked(findings, schema, "$"); + if (errors.length < LIMITS.validationErrors) { + errors.push(...collectFindingSemanticErrors(findings)); + } + return errors; +} + +function loadSchema(schemaPath) { + const schema = JSON.parse(fs.readFileSync(schemaPath, "utf8")); + const errors = collectSchemaErrors(schema); + if (errors.length > 0) throw new Error(`unsupported or invalid report schema:\n${errors.join("\n")}`); + return schema; +} + +function readFileWithinLimit(file) { + const noFollow = fs.constants.O_NOFOLLOW; + const nonBlock = fs.constants.O_NONBLOCK; + if (!Number.isInteger(noFollow) || noFollow === 0 || !Number.isInteger(nonBlock) || nonBlock === 0) { + throw new SafeInputError("OS no-follow and nonblocking input protection is unavailable"); + } + + let descriptor; + try { + descriptor = fs.openSync(file, fs.constants.O_RDONLY | noFollow | nonBlock); + } catch (error) { + if (error && (error.code === "ELOOP" || error.code === "EMLINK")) { + throw new SafeInputError("input must not be a symlink"); + } + throw error; + } + try { + const stat = fs.fstatSync(descriptor); + if (!stat.isFile()) { + throw new SafeInputError("input must be a regular file"); + } + if (stat.size > LIMITS.inputBytes) { + throw new SafeInputError(`input exceeds ${LIMITS.inputBytes} byte limit`); + } + + const chunks = []; + const buffer = Buffer.allocUnsafe(64 * 1024); + let bytesRead = 0; + while (true) { + const count = fs.readSync(descriptor, buffer, 0, buffer.length, null); + if (count === 0) break; + bytesRead += count; + if (bytesRead > LIMITS.inputBytes) { + throw new SafeInputError(`input exceeds ${LIMITS.inputBytes} byte limit`); + } + chunks.push(Buffer.from(buffer.subarray(0, count))); + } + try { + return UTF8_DECODER.decode(Buffer.concat(chunks, bytesRead)); + } catch { + throw new SafeInputError("input is not valid UTF-8"); + } + } finally { + fs.closeSync(descriptor); + } +} + +function enforceJsonTextLimits(contents) { + const containers = []; + let inString = false; + let escaped = false; + + function markArrayItem() { + const container = containers[containers.length - 1]; + if (!container || container.type !== "array" || !container.expectsItem) return; + container.expectsItem = false; + container.items++; + if (container.items > LIMITS.arrayItems) { + throw new JsonStructureError(`input exceeds ${LIMITS.arrayItems} item array limit`); + } + } + + for (let index = 0; index < contents.length; index++) { + const character = contents[index]; + if (inString) { + if (escaped) { + escaped = false; + } else if (character === "\\") { + escaped = true; + } else if (character === "\"") { + inString = false; + } + continue; + } + + if (character === "\"") { + markArrayItem(); + inString = true; + } else if (character === "[" || character === "{") { + markArrayItem(); + if (containers.length >= LIMITS.nestingDepth) { + throw new JsonStructureError(`input exceeds ${LIMITS.nestingDepth} level nesting depth limit`); + } + containers.push({ + type: character === "[" ? "array" : "object", + expectsItem: character === "[", + items: 0, + }); + } else if (character === "]" || character === "}") { + containers.pop(); + } else if (character === ",") { + const container = containers[containers.length - 1]; + if (container && container.type === "array") container.expectsItem = true; + } else if (!/\s/.test(character)) { + markArrayItem(); + } + } +} + +function run(file) { + if (!file) { + console.error("Usage: node validate-findings.cjs <path-to-findings.json>"); + return 1; + } + + let schema; + try { + schema = loadSchema(path.join(__dirname, "report-schema.json")); + } catch (error) { + console.error("Failed to load report-schema.json:", error.message); + return 1; + } + + let contents; + try { + contents = readFileWithinLimit(file); + } catch (error) { + const reason = error instanceof SafeInputError ? error.message : "input could not be opened or read safely"; + console.error(`Failed to read findings JSON: ${reason}`); + return 1; + } + + try { + enforceJsonTextLimits(contents); + } catch (error) { + const reason = error instanceof JsonStructureError ? error.message : "invalid JSON structure"; + console.error(`Failed to parse findings JSON: ${reason}`); + return 1; + } + + let findings; + try { + findings = JSON.parse(contents); + } catch { + console.error("Failed to parse findings JSON: invalid JSON syntax"); + return 1; + } + + let errors; + try { + errors = validateDocument(findings, schema); + } catch { + console.error("Failed to validate findings JSON: unexpected validation error"); + return 1; + } + for (const message of errors) console.error("ERROR:", escapeUnsafeDiagnosticCharacters(message)); + if (errors.length > 0) { + const cap = errors.length === LIMITS.validationErrors ? `; output capped at ${LIMITS.validationErrors}` : ""; + console.error(`FAIL: ${errors.length} validation error(s)${cap}`); + return 1; + } + console.log(`PASS: ${findings.length} findings valid`); + return 0; +} + +module.exports = { + LIMITS, + PATH_FORBIDDEN_CHARACTER, + UNSAFE_DIAGNOSTIC_CHARACTER, + VISIBLE_CONTENT, + WINDOWS_RESERVED_COMPONENT, + collect, + collectFindingSemanticErrors, + collectSchemaErrors, + hasVisibleProse, + isSafeRelativeSourcePath, + validateDocument, +}; + +if (require.main === module) process.exit(run(process.argv[2])); diff --git a/skills/thirdshift-security-audit/validate-findings.test.cjs b/skills/thirdshift-security-audit/validate-findings.test.cjs new file mode 100644 index 00000000..8d245b22 --- /dev/null +++ b/skills/thirdshift-security-audit/validate-findings.test.cjs @@ -0,0 +1,652 @@ +const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const os = require("node:os"); +const path = require("node:path"); +const { spawnSync } = require("node:child_process"); +const test = require("node:test"); +const schema = require("./report-schema.json"); +const { + LIMITS, + collect, + collectSchemaErrors, + validateDocument, +} = require("./validate-findings.cjs"); + +const validatorPath = path.join(__dirname, "validate-findings.cjs"); +const CLI_TIMEOUT_MS = 5000; +const HOSTILE_CLI_TIMEOUT_MS = 15000; +const HAS_SAFE_INPUT_OPEN = Number.isInteger(fs.constants.O_NOFOLLOW) && + fs.constants.O_NOFOLLOW !== 0 && + Number.isInteger(fs.constants.O_NONBLOCK) && + fs.constants.O_NONBLOCK !== 0; +const TERMINAL_CONTROL_PAYLOAD = "\u001b\u0007\u0085\u202e\u034f\ufe0f"; +const TERMINAL_CONTROL_BYTES = [ + Buffer.from([0x1b]), + Buffer.from([0x07]), + Buffer.from("\u0085"), + Buffer.from("\u202e"), + Buffer.from("\u034f"), + Buffer.from("\ufe0f"), +]; + +function source(kind = "entrypoint", file = "src/handler.c", line = 10) { + return { kind, file, line, scope: "handle", description: "Attacker data reaches the operation." }; +} + +function evidence(file = "src/handler.c", line = 10) { + return { file, line, description: "The source performs the operation without the required check." }; +} + +function confirmed() { + return { + verdict: "confirmed", + fingerprint: "src-handler-missing-check", + title: "Missing ownership check", + description: "An attacker can reach an operation without the intended ownership check.", + root_cause: "handle omits the ownership check before changing the object.", + intended_behavior: "Only the object's owner can change it.", + trace: [source("entrypoint"), source("propagation", "src/model.c", 20), source("sink", "src/store.c", 30)], + evidence: [evidence()], + conditions: [], + execution: { + attacker_perspective: "An unprivileged remote user with their own account.", + payloads: ["An object identifier owned by another user."], + instructions: ["Submit the identifier through the public operation."], + observed_result: "The other user's object changes.", + }, + remediation: { strategy: "Check ownership before the state change." }, + severity: { + likelihood: { score: "medium", reason: "The operation is directly reachable." }, + impact: { score: "medium", reason: "The attacker changes one protected object." }, + overall_severity: "medium", + }, + confidence: { score: "high", reason: "The source path and result were reproduced." }, + }; +} + +function needsValidation() { + return { + verdict: "needs_validation", + fingerprint: "src-parser-size-hypothesis", + title: "Unchecked parsed size", + description: "A parsed size may reach an allocation without a limit.", + claimed_root_cause: "parse_size may pass an unbounded value to allocate.", + trace: [source()], + evidence: [evidence()], + blockers: ["The generated parser source is absent from this checkout."], + validation_plan: { + local: "Generate the parser and submit the smallest input that exceeds the documented limit.", + deployment: "In an approved test deployment, confirm the request reaches the generated parser and record the bounded observable result.", + }, + }; +} + +function rejected() { + return { + verdict: "rejected", + fingerprint: "src-router-auth-bypass", + title: "Authorization bypass in router", + description: "The candidate claimed a route bypassed authorization.", + claimed_root_cause: "dispatch was claimed to skip the authorization wrapper.", + trace: [source("sink")], + evidence: [evidence()], + reason: "All routes pass through the authorization wrapper before dispatch.", + }; +} + +function errorsFor(value) { + return validateDocument(value, schema); +} + +function runCli(contents, options = {}) { + const { nodeArgs = [], timeout = CLI_TIMEOUT_MS } = options; + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-")); + const findingsPath = path.join(directory, "findings.json"); + try { + fs.writeFileSync(findingsPath, contents); + return spawnSync(process.execPath, [...nodeArgs, validatorPath, findingsPath], { + encoding: "utf8", + timeout, + }); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +function cliOutput(result) { + return `${result.stdout}${result.stderr}`; +} + +function assertNoInjectedControlBytes(output) { + const bytes = Buffer.isBuffer(output) ? output : Buffer.from(output, "utf8"); + for (const marker of TERMINAL_CONTROL_BYTES) { + assert.equal(bytes.indexOf(marker), -1, `found raw control bytes ${marker.toString("hex")}`); + } +} + +function producerShapedFindings() { + const demonstrated = confirmed(); + demonstrated.conditions = [{ + kind: "authentication_level", + description: "The attacker needs a normal account.", + }]; + demonstrated.execution.payloads = ["", " \t\r\n", "\u0000\u001f\u007f", "\u034f", "\ufe0f", "\ud800", "\udc00", "[{,}]\\\""]; + demonstrated.remediation.code_changes = [{ + file_name: "src/handler.c", + fixed_code: "", + }]; + + const blocked = needsValidation(); + delete blocked.validation_plan.deployment; + + return [demonstrated, blocked, rejected()]; +} + +function rejectMutation(factory, mutate) { + const value = factory(); + mutate(value); + assert.notEqual(errorsFor([value]).length, 0); +} + +test("schema is an actual top-level array with exactly three branches", () => { + assert.equal(schema.type, "array"); + assert.equal(schema.items.oneOf.length, 3); + assert.deepEqual(schema.items.oneOf.map((branch) => branch.properties.verdict.const), [ + "confirmed", "needs_validation", "rejected", + ]); + const confirmedSchema = schema.items.oneOf[0].properties; + assert.equal(confirmedSchema.title.visibleContent, true); + assert.equal(confirmedSchema.execution.properties.payloads.items.minLength, undefined); + assert.equal(confirmedSchema.execution.properties.payloads.items.visibleContent, undefined); + assert.equal(confirmedSchema.remediation.properties.code_changes.items.properties.fixed_code.minLength, undefined); +}); + +test("accepts a producer-shaped findings document through the CLI", () => { + const result = runCli(JSON.stringify(producerShapedFindings())); + assert.equal(result.status, 0, cliOutput(result)); + assert.match(result.stdout, /PASS: 3 findings valid/); +}); + +test("accepts empty output and each complete branch", () => { + assert.deepEqual(errorsFor([]), []); + assert.deepEqual(errorsFor([confirmed(), needsValidation(), rejected()]), []); + const localOnly = needsValidation(); + delete localOnly.validation_plan.deployment; + assert.deepEqual(errorsFor([localOnly]), []); + const deploymentOnly = needsValidation(); + delete deploymentOnly.validation_plan.local; + assert.deepEqual(errorsFor([deploymentOnly]), []); +}); + +test("allows a one-line finding trace", () => { + const finding = confirmed(); + finding.trace = [source("entrypoint")]; + assert.deepEqual(errorsFor([finding]), []); +}); + +test("rejects empty required content", () => { + const cases = [ + [confirmed, (finding) => { finding.title = ""; }], + [confirmed, (finding) => { finding.title = " "; }], + [confirmed, (finding) => { finding.evidence = []; }], + [confirmed, (finding) => { finding.execution.payloads = []; }], + [confirmed, (finding) => { finding.execution.instructions = []; }], + [confirmed, (finding) => { finding.execution.observed_result = ""; }], + [confirmed, (finding) => { finding.remediation.strategy = ""; }], + [needsValidation, (finding) => { finding.blockers = []; }], + [needsValidation, (finding) => { finding.validation_plan = {}; }], + [needsValidation, (finding) => { finding.validation_plan = { local: " " }; }], + [rejected, (finding) => { finding.claimed_root_cause = ""; }], + ]; + for (const [factory, mutate] of cases) rejectMutation(factory, mutate); +}); + +test("preserves exact payload and replacement-code strings", () => { + const finding = confirmed(); + const payloads = ["", " \t\r\n", "\u0000\u001f\u007f", "\u034f", "\ufe0f", "\ud800", "\udc00"]; + const fixedCode = "\u0000 \t\r\n\u001f\u007f\u034f\ufe0f\ud800x\udc00"; + finding.execution.payloads = payloads.slice(); + finding.remediation.code_changes = [{ file_name: "src/handler.c", fixed_code: fixedCode }]; + + assert.deepEqual(errorsFor([finding]), []); + assert.deepEqual(finding.execution.payloads, payloads); + assert.equal(finding.remediation.code_changes[0].fixed_code, fixedCode); +}); + +test("rejects invalid scalars and whitespace, control, format, or default-ignorable prose", () => { + for (const invisible of ["\u0000\t\r\n\u001f\u007f\u200b", "\u034f", "\ufe0f", "\ud800", "\udc00", "visible\ud800"]) { + const cases = [ + [confirmed, (finding) => { finding.title = invisible; }], + [confirmed, (finding) => { finding.trace[0].scope = invisible; }], + [confirmed, (finding) => { finding.evidence[0].description = invisible; }], + [confirmed, (finding) => { finding.execution.instructions = [invisible]; }], + [confirmed, (finding) => { finding.remediation.strategy = invisible; }], + [confirmed, (finding) => { finding.severity.impact.reason = invisible; }], + [confirmed, (finding) => { finding.confidence.reason = invisible; }], + [needsValidation, (finding) => { finding.blockers = [invisible]; }], + [needsValidation, (finding) => { finding.validation_plan = { local: invisible }; }], + [rejected, (finding) => { finding.reason = invisible; }], + ]; + for (const [factory, mutate] of cases) rejectMutation(factory, mutate); + } +}); + +test("quotes input-derived controls in direct validation values and paths", () => { + const finding = confirmed(); + finding.trace[0].kind = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + finding.execution[`extra-${TERMINAL_CONTROL_PAYLOAD}`] = "value"; + + const cyclic = {}; + cyclic[`path-${TERMINAL_CONTROL_PAYLOAD}`] = cyclic; + const output = [ + ...errorsFor([finding]), + ...collect(cyclic, { type: "object" }, "$input"), + ].join("\n"); + + for (const escaped of ["\\u001b", "\\u0007", "\\u0085", "\\u202e", "\\u034f", "\\ufe0f"]) { + assert(output.includes(escaped), `missing escaped diagnostic ${escaped}`); + } + assertNoInjectedControlBytes(output); +}); + +test("rejects line zero", () => { + rejectMutation(confirmed, (finding) => { finding.trace[0].line = 0; }); + rejectMutation(rejected, (finding) => { finding.evidence[0].line = 0; }); +}); + +test("does not treat inherited or Object-prototype properties as schema properties", () => { + rejectMutation(confirmed, (finding) => { finding.constructor = "not allowed"; }); + const inherited = Object.create({ verdict: "confirmed" }); + assert(errorsFor([inherited]).some((error) => error.includes("exactly one"))); + assert(collect(Object.create({ constructor: "inherited" }), { + type: "object", + properties: { constructor: { type: "string" } }, + required: ["constructor"], + additionalProperties: false, + }).some((error) => error.includes("missing required"))); +}); + +test("oneOf requires exactly one passing branch", () => { + assert(collect("value", { oneOf: [{ type: "string" }, { minLength: 1 }] }, "$test") + .some((error) => error.includes("matched 2"))); + assert(collect(7, { oneOf: [{ type: "string" }, { minimum: 10 }] }, "$test") + .some((error) => error.includes("matched 0"))); +}); + +test("rejects duplicate fingerprints and unique array entries", () => { + const first = confirmed(); + const second = rejected(); + second.fingerprint = first.fingerprint; + assert(errorsFor([first, second]).some((error) => error.includes("duplicate of"))); + rejectMutation(needsValidation, (finding) => { finding.blockers = [finding.blockers[0], finding.blockers[0]]; }); +}); + +test("uses canonical Set uniqueness for structured entries at the array limit", () => { + const entries = Array.from({ length: LIMITS.arrayItems }, (_, id) => ({ id, label: String(id) })); + assert.deepEqual(collect(entries, { type: "array", uniqueItems: true }), []); + + const duplicate = entries.slice(0, -1); + duplicate.push({ label: "0", id: 0 }); + assert(collect(duplicate, { type: "array", uniqueItems: true }) + .some((error) => error.includes(`duplicate at index ${LIMITS.arrayItems - 1}`))); +}); + +test("bounds canonical uniqueness keys and Set storage", () => { + const oversizedKey = "x".repeat(LIMITS.canonicalKeyBytes + 1); + assert(collect([oversizedKey], { type: "array", uniqueItems: true }) + .some((error) => error.includes("canonical key exceeds"))); + + const itemLength = Math.floor(LIMITS.uniqueSetBytes / 6); + const largeUniqueItems = Array.from({ length: 6 }, (_, index) => `${index}${"x".repeat(itemLength)}`); + assert(collect(largeUniqueItems, { type: "array", uniqueItems: true }) + .some((error) => error.includes("canonical uniqueness set exceeds"))); +}); + +test("requires findings to be sorted by fingerprint", () => { + const first = confirmed(); + const second = rejected(); + assert(errorsFor([second, first]).some((error) => error.includes("sorted lexicographically"))); +}); + +test("rejects severity above demonstrated impact", () => { + rejectMutation(confirmed, (finding) => { + finding.severity.overall_severity = "high"; + finding.severity.impact.score = "medium"; + }); +}); + +test("rejects unsafe source paths", () => { + const badPaths = [ + "/etc/passwd", + "../src/file.c", + "src/../file.c", + "src//file.c", + "C:\\src\\file.c", + "src/file:name.c", + "src/file\nname.c", + "src/file\u0001name.c", + "src/file\u0085name.c", + "src/file\u2028name.c", + "src/file\u202ename.c", + "src/file\u2066name.c", + "src/file\u200dname.c", + "src/file\u034fname.c", + "src/file\ufe0fname.c", + "src/file\ud800name.c", + "src/file\udc00name.c", + "CON", + "src/con.txt", + "src/PRN", + "src/AUX.c", + "src/NUL", + "src/COM1.log", + "src/lpt9", + "src/CONIN$", + "src/CONOUT$.txt", + "src/CLOCK$.txt", + "src/COM\u00b9.log", + "src/LPT\u00b2.log", + "src /file.c", + "src./file.c", + "src/file.c ", + "src/file.c.", + ]; + for (const badPath of badPaths) { + rejectMutation(confirmed, (finding) => { finding.trace[0].file = badPath; }); + } + rejectMutation(rejected, (finding) => { finding.evidence[0].file = "NUL.txt"; }); + rejectMutation(confirmed, (finding) => { + finding.remediation.code_changes = [{ file_name: "src/file:name.c", fixed_code: "replacement" }]; + }); +}); + +test("accepts legitimate Unicode source paths and prose", () => { + const finding = confirmed(); + finding.title = "Finding \ud83d\ude00 cafe\u0301"; + finding.trace[0].file = "src/日本語/cafe\u0301-\ud83d\ude00.ts"; + finding.evidence[0].file = "src/mañana/файл.ts"; + finding.remediation.code_changes = [{ + file_name: "src/修正/éxito.ts", + fixed_code: "replacement", + }]; + assert.deepEqual(errorsFor([finding]), []); +}); + +test("CLI rejects input above the byte limit without an exception trace", () => { + const result = runCli(Buffer.alloc(LIMITS.inputBytes + 1, 0x20)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`input exceeds ${LIMITS.inputBytes} byte limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("CLI rejects invalid UTF-8 without replacement or an exception trace", () => { + const findings = producerShapedFindings(); + findings[0].execution.payloads = ["INVALID_UTF8"]; + const encoded = Buffer.from(JSON.stringify(findings)); + const marker = Buffer.from("INVALID_UTF8"); + const markerOffset = encoded.indexOf(marker); + assert.notEqual(markerOffset, -1); + const malformed = Buffer.concat([ + encoded.subarray(0, markerOffset), + Buffer.from([0x80]), + encoded.subarray(markerOffset + marker.length), + ]); + + const result = runCli(malformed); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /input is not valid UTF-8/); + assert.doesNotMatch(output, /TypeError|stack|at validate-findings/i); +}); + +test("quotes input-derived controls in CLI validation errors", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const finding = confirmed(); + finding.trace[0].kind = `invalid-${TERMINAL_CONTROL_PAYLOAD}`; + finding.execution[`extra-${TERMINAL_CONTROL_PAYLOAD}`] = "value"; + const result = runCli(JSON.stringify([finding])); + + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /\$\[0\]\.trace\[0\]\.kind/); + assert.match(result.stderr, /\\u001b/); + assert.match(result.stderr, /\\u202e/); + assertNoInjectedControlBytes(result.stderr); +}); + +test("returns a generic syntax error without parser-supplied controls", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const malformed = Buffer.concat([ + Buffer.from("["), + Buffer.from(TERMINAL_CONTROL_PAYLOAD), + Buffer.from("]"), + ]); + const result = runCli(malformed); + + assert.equal(result.status, 1, cliOutput(result)); + assert.equal(result.stderr, "Failed to parse findings JSON: invalid JSON syntax\n"); + assertNoInjectedControlBytes(result.stderr); +}); + +test("does not reflect controls from a failed CLI input path", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-path-")); + const missingPath = path.join(directory, `missing-${TERMINAL_CONTROL_PAYLOAD}.json`); + try { + const result = spawnSync(process.execPath, [validatorPath, missingPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + assert.equal(result.status, 1, cliOutput(result)); + assert.match(result.stderr, /Failed to read findings JSON:/); + assertNoInjectedControlBytes(result.stderr); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects lone-surrogate prose without changing payload semantics", () => { + const findings = producerShapedFindings(); + findings[0].title = "\ud800"; + const result = runCli(JSON.stringify(findings)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /must contain only valid Unicode scalar values/); + assert.doesNotMatch(output, /stack|at validate-findings/i); +}); + +test("CLI rejects Unicode format controls in source paths", () => { + const findings = producerShapedFindings(); + findings[0].trace[0].file = "src/file\u202ename.c"; + const result = runCli(JSON.stringify(findings)); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, /must be a safe repository-relative source path/); + assert.doesNotMatch(output, /stack|at validate-findings/i); +}); + +test("CLI rejects a FIFO without blocking", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-fifo-")); + const fifoPath = path.join(directory, "findings.json"); + try { + const created = spawnSync("mkfifo", [fifoPath], { encoding: "utf8", timeout: CLI_TIMEOUT_MS }); + assert.equal(created.status, 0, cliOutput(created)); + + const result = spawnSync(process.execPath, [validatorPath, fifoPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must be a regular file/); + assert.doesNotMatch(output, /stack|at validate-findings/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects a symlink without following it", { skip: process.platform === "win32" || !HAS_SAFE_INPUT_OPEN }, () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "validate-findings-symlink-")); + const targetPath = path.join(directory, "target.json"); + const symlinkPath = path.join(directory, "findings.json"); + try { + fs.writeFileSync(targetPath, JSON.stringify(producerShapedFindings())); + fs.symlinkSync(targetPath, symlinkPath); + const result = spawnSync(process.execPath, [validatorPath, symlinkPath], { + encoding: "utf8", + timeout: CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /input must not be a symlink/); + assert.doesNotMatch(output, /stack|at validate-findings/i); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("CLI rejects input above the nesting-depth limit without an exception trace", () => { + const levels = LIMITS.nestingDepth + 1; + const result = runCli(`${"[".repeat(levels)}0${"]".repeat(levels)}`); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`${LIMITS.nestingDepth} level nesting depth limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("CLI rejects an oversized array without an exception trace", () => { + const result = runCli(JSON.stringify(Array(LIMITS.arrayItems + 1).fill(null))); + const output = cliOutput(result); + assert.equal(result.status, 1, output); + assert.match(output, new RegExp(`${LIMITS.arrayItems} item array limit`)); + assert.doesNotMatch(output, /RangeError|Maximum call stack|heap out of memory/i); +}); + +test("checks pattern and branch invariants", () => { + rejectMutation(rejected, (finding) => { finding.fingerprint = "not stable"; }); + rejectMutation(needsValidation, (finding) => { + finding.severity = { impact: { score: "low" }, overall_severity: "low" }; + }); +}); + +test("rejects unsupported and malformed schema keywords", () => { + assert(collectSchemaErrors({ type: "string", format: "uuid" }).some((error) => error.includes("format"))); + assert(collectSchemaErrors({ type: "string", pattern: "[" }).some((error) => error.includes("regular expression"))); + assert(collectSchemaErrors({ type: "string", visibleContent: "yes" }).some((error) => error.includes("expected boolean"))); + assert(collectSchemaErrors({ type: "array", visibleContent: true }).some((error) => error.includes("requires type"))); + assert.notEqual(validateDocument([], { type: "array", maxItems: 1 }).length, 0); +}); + +test("caps malformed 1000-finding validation output", () => { + assert.equal(errorsFor(Array.from({ length: LIMITS.arrayItems }, () => null)).length, LIMITS.validationErrors); + if (!HAS_SAFE_INPUT_OPEN) return; + + const result = runCli(JSON.stringify(Array.from({ length: LIMITS.arrayItems }, () => null))); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /RangeError|Maximum call stack|stack|at validate-findings/i); +}); + +test("caps amplified in-limit findings output under a constrained Node heap", { skip: !HAS_SAFE_INPUT_OPEN }, () => { + const findings = Array.from({ length: 750 }, () => ({ + verdict: "confirmed", + trace: Array.from({ length: LIMITS.arrayItems }, () => 0), + evidence: Array.from({ length: LIMITS.arrayItems }, () => 0), + })); + const contents = JSON.stringify(findings); + assert(contents.length > 3 * 1000 * 1000, `hostile input too small: ${contents.length}`); + assert(contents.length <= LIMITS.inputBytes, `hostile input over limit: ${contents.length}`); + + const result = runCli(contents, { + nodeArgs: ["--max-old-space-size=64"], + timeout: HOSTILE_CLI_TIMEOUT_MS, + }); + const output = cliOutput(result); + assert.notEqual(result.error && result.error.code, "ETIMEDOUT", output); + assert.equal(result.status, 1, output); + assert.match(output, /output capped at 100/); + assert(output.length < 20000, `unexpected output length ${output.length}`); + assert.doesNotMatch(output, /heap out of memory|allocation failed|RangeError|Maximum call stack/i); +}); + +test("keeps shared helpers aligned with the coverage-ledger validator", () => { + const findingsModule = require("./validate-findings.cjs"); + const ledgerModule = require("./validate-coverage-ledger.cjs"); + + for (const name of [ + "VISIBLE_CONTENT", + "PATH_FORBIDDEN_CHARACTER", + "WINDOWS_RESERVED_COMPONENT", + "UNSAFE_DIAGNOSTIC_CHARACTER", + ]) { + assert.equal(findingsModule[name].source, ledgerModule[name].source, `${name} source`); + assert.equal(findingsModule[name].flags, ledgerModule[name].flags, `${name} flags`); + } + + const sharedLimitKeys = Object.keys(findingsModule.LIMITS) + .filter((key) => Object.prototype.hasOwnProperty.call(ledgerModule.LIMITS, key)) + .sort(); + assert.deepEqual(sharedLimitKeys, ["inputBytes", "nestingDepth", "validationErrors"]); + for (const key of sharedLimitKeys) { + assert.equal(findingsModule.LIMITS[key], ledgerModule.LIMITS[key], `LIMITS.${key}`); + } + + const pathCorpus = [ + "src/handler.js", + "src/caf\u00e9/handler.js", + "src/\u65e5\u672c\u8a9e/\u0444\u0430\u0439\u043b.ts", + "src/cloc\u212a$.txt", + "src/CLOCK$.txt", + "src/con.txt", + "CON", + "src/COM\u00b9.log", + "src/lpt\u00b3", + "/etc/passwd", + "../src/file.c", + "src/../file.c", + "src//file.c", + "src\\file.c", + "src/file:name.c", + "~home/file.c", + "C:/file.c", + "src/file.c ", + "src/file.c.", + "src/file\u202ename.c", + "src/file\u200b.js", + "src/file\u034f.js", + "src/file\ufe0f.js", + "src/file\ud800name.c", + "src/file\udc00name.c", + ]; + for (const value of pathCorpus) { + assert.equal( + findingsModule.isSafeRelativeSourcePath(value), + ledgerModule.isSafeRelativePath(value), + `path verdict diverges for ${JSON.stringify(value)}`, + ); + } + assert.equal(findingsModule.isSafeRelativeSourcePath("src/cloc\u212a$.txt"), false); + assert.equal(ledgerModule.isSafeRelativePath("src/cloc\u212a$.txt"), false); + + const proseCorpus = [ + "Valid prose.", + "caf\u00e9", + "", + " \t\r\n", + "\u200b", + "\u034f", + "\ufe0f", + "\ud800", + "\udc00", + "visible\ud800", + ]; + for (const value of proseCorpus) { + assert.equal( + findingsModule.hasVisibleProse(value), + ledgerModule.hasVisibleProse(value), + `prose verdict diverges for ${JSON.stringify(value)}`, + ); + } +}); diff --git a/skills/thirdshift-to-spec/SKILL.md b/skills/thirdshift-to-spec/SKILL.md index 56dc42ed..5990f1a8 100644 --- a/skills/thirdshift-to-spec/SKILL.md +++ b/skills/thirdshift-to-spec/SKILL.md @@ -1,6 +1,6 @@ --- name: thirdshift-to-spec -description: "Turn the current session into a spec and publish it to the project issue tracker: no interview, just synthesis of what you've already settled. Only for an Architecture review." +description: "Turn the current session into a spec and publish it to the project issue tracker: no interview, just synthesis of what you've already settled. For an Architecture review or Security fix publishing." disable-model-invocation: false --- @@ -10,6 +10,8 @@ The issue tracker and triage label vocabulary should have been provided to you. The spec is all this skill writes. Write nothing to the repository: no spec file, commits or branches. A `CONTEXT.md` or ADR change the spec needs is work for one of its tickets. +For **Security fix publishing**, the Session prompt overrides the template: keep the Spec terse, saying only what the fix changes and linking the private record. Keep all finding evidence in that record. On a private repository, use the finding's existing issue as the Spec and preserve its body and evidence; publish its Tickets with `thirdshift-to-tickets`. + ## Process 1. Explore the repo to understand the current state of the codebase, if you haven't already. Use the project's domain glossary vocabulary throughout the spec, and respect any ADRs in the area you're touching. diff --git a/skills/thirdshift-to-tickets/SKILL.md b/skills/thirdshift-to-tickets/SKILL.md index 2ed473f3..d9568644 100644 --- a/skills/thirdshift-to-tickets/SKILL.md +++ b/skills/thirdshift-to-tickets/SKILL.md @@ -1,6 +1,6 @@ --- name: thirdshift-to-tickets -description: Break a plan, spec, or the current session into a set of tracer-bullet tickets, each declaring its blocking edges, published to the issue tracker with native sub-issue and blocking links. Only for an Architecture review. +description: Break a plan, spec, or the current session into a set of tracer-bullet tickets, each declaring its blocking edges, published to the issue tracker with native sub-issue and blocking links. For an Architecture review or Security fix publishing. disable-model-invocation: false --- @@ -12,6 +12,8 @@ The issue tracker and triage label vocabulary should have been provided to you. The tickets are all this skill writes. Write nothing to the repository: no ticket files, commits or branches. A `CONTEXT.md` or ADR change the work needs is part of a ticket's work, named in its acceptance criteria. +For **Security fix publishing**, keep every Ticket terse: say only what the fix changes and link the private record. Keep all finding evidence in that record. A private finding's issue is the parent Spec, and its existing body and evidence stay unchanged. + ## Process ### 1. Gather context diff --git a/src/architect.rs b/src/architect.rs index df86aa18..724f9161 100644 --- a/src/architect.rs +++ b/src/architect.rs @@ -7,9 +7,9 @@ //! candidate published no plan, and the Architect run ends on the issue it //! named instead, labelled an //! Architect idea: the idea issue it filed for its top recommendation, or the -//! open issue that already covers it. Only one Architect run or Pickup run -//! per repository runs at a time on a machine: an Architect run started while -//! another of either is still running is skipped, before any review. So is +//! open issue that already covers it. Only one Pass per repository runs at a +//! time on a machine: an Architect run started while another Pass is still +//! running is skipped, before any review. So is //! one that finds an Architect plan still open on the repository: it never //! retries or dispatches an Architect plan that is already there. And so is //! one that finds an Architect idea there waiting for triage: the factory has @@ -67,7 +67,7 @@ pub enum Outcome { /// skipped run's progress line gives it. #[derive(Debug)] pub enum Skipped { - /// Another Architect run on this repository, or a Pickup run, is still + /// Another Pass on this repository is still /// running on this machine, the Spec run or Run it dispatched included. AlreadyRunning(AlreadyRunning), /// These Architect plans are still open on this repository: at least @@ -227,7 +227,7 @@ pub fn run( } = match launch::start(base) { Ok(Start::Clear(launch)) => launch, Ok(Start::AlreadyRunning(running)) => { - logs::skipped(Pass::ArchitectRun, &running.0, &running); + logs::skipped(Pass::Architect, &running.0, &running); return Outcome::Skipped(Skipped::AlreadyRunning(running)); } Err(error) => return failed(error.into()), @@ -285,7 +285,7 @@ fn run_through(outside: &mut impl Outside, architect_run: &ArchitectRun) -> Outc match gates(outside) { Ok(Decision::GoAhead) => {} Ok(Decision::Skip(skipped)) => { - outside.skipped(Pass::ArchitectRun, &skipped); + outside.skipped(Pass::Architect, &skipped); return Outcome::Skipped(skipped); } Err(error) => return failed(error.into()), diff --git a/src/args.rs b/src/args.rs index f84d668d..e99b9bde 100644 --- a/src/args.rs +++ b/src/args.rs @@ -1,5 +1,5 @@ //! The command line: which command, and for a Run, its Issue URL and flags, -//! for an Architect run, its focus and flags, or for a Pickup run, its flags. +//! or for a Pass, its flags and any arguments its own kind takes. use std::iter::Peekable; use std::mem::discriminant; @@ -24,7 +24,8 @@ pub enum Command { /// `email-test`, with the address it was given, if any. EmailTest(Option<String>), Architect(ArchitectArgs), - Pickup(PickupArgs), + Pickup(PassArgs), + Secure(PassArgs), Run(RunArgs), } @@ -49,15 +50,15 @@ pub struct ArchitectArgs { pub plan_only: bool, } -/// A Pickup run's arguments. +/// A Pickup run or Security run's shared Pass arguments. #[derive(Debug, PartialEq, Eq)] -pub struct PickupArgs { +pub struct PassArgs { /// The Base branch `base <branch>` named, if given; without it, the /// branch checked out in the Launch directory is the Base branch. pub base: Option<String>, /// The flags it shares with a Run: `email` and `no-email` for its own - /// Run notification, the rest for the Spec run or Run the Ready issue is - /// dispatched as. + /// Run notification, the rest for a run the Pass dispatches. The + /// Security run also uses `security-fix` for its own choice of work. pub flags: Flags, } @@ -77,7 +78,7 @@ pub struct RunArgs { } /// Parse the arguments after the program name. `help`, `version`, `update`, -/// `setup`, `email-test`, `architect` and `pickup` are commands only as the +/// `setup`, `email-test`, `architect`, `pickup` and `secure` are commands only as the /// first argument. Otherwise it is a Run: one Issue URL, with each Run flag at /// most once, before or after it. /// `email` may be followed by the address to send the Run notification to, @@ -105,7 +106,8 @@ pub fn parse(args: &[String]) -> Result<Command> { }; } Some("architect") => return parse_architect(&args[1..]).map(Command::Architect), - Some("pickup") => return parse_pickup(&args[1..]).map(Command::Pickup), + Some("pickup") => return parse_pass(&args[1..], "pickup").map(Command::Pickup), + Some("secure") => return parse_pass(&args[1..], "secure").map(Command::Secure), _ => {} } let mut issue = None; @@ -148,7 +150,7 @@ fn parse_architect(args: &[String]) -> Result<ArchitectArgs> { let mut flags = Flags::default(); let mut args = args.iter().peekable(); while let Some(arg) = args.next() { - if take_flag(&mut flags, arg, &mut args)? { + if take_pass_flag(&mut base, &mut flags, arg, &mut args)? { continue; } match arg.as_str() { @@ -158,7 +160,6 @@ fn parse_architect(args: &[String]) -> Result<ArchitectArgs> { } plan_only = true; } - "base" | "--base" => ask_word(&mut base, arg, args.next(), "a branch")?, _ if arg.starts_with('-') => bail!("unexpected argument after architect: {arg}"), _ if focus.is_some() => bail!("unexpected argument after the focus: {arg}"), _ if arg.trim().is_empty() => bail!("the focus is empty"), @@ -179,26 +180,40 @@ fn parse_architect(args: &[String]) -> Result<ArchitectArgs> { }) } -/// Parse the arguments after `pickup`: its flags, each at most once, in any +/// Parse the arguments after `pickup` or `secure`: shared flags, at most once, in any /// order, and nothing else. `base` must be followed by the Base branch. -fn parse_pickup(args: &[String]) -> Result<PickupArgs> { +fn parse_pass(args: &[String], command: &str) -> Result<PassArgs> { let mut base = None; let mut flags = Flags::default(); let mut args = args.iter().peekable(); while let Some(arg) = args.next() { - if take_flag(&mut flags, arg, &mut args)? { - continue; + if !take_pass_flag(&mut base, &mut flags, arg, &mut args)? { + bail!("unexpected argument after {command}: {arg}"); } - match arg.as_str() { - "base" | "--base" => ask_word(&mut base, arg, args.next(), "a branch")?, - _ => bail!("unexpected argument after pickup: {arg}"), + } + Ok(PassArgs { base, flags }) +} + +/// Take a Pass's shared words: its Base branch and the flags for its +/// dispatched run and its own Run notification. Each kind's parser handles +/// only the arguments specific to that kind. +fn take_pass_flag<'a>( + base: &mut Option<String>, + flags: &mut Flags, + arg: &str, + rest: &mut Peekable<impl Iterator<Item = &'a String>>, +) -> Result<bool> { + match arg { + "base" | "--base" => { + ask_word(base, arg, rest.next(), "a branch")?; + Ok(true) } + _ => take_flag(flags, arg, rest), } - Ok(PickupArgs { base, flags }) } -/// Record in `flags` what `arg` asks for, if it is one of the flags a Run, -/// an Architect run and a Pickup run all take, with or without its dashes, +/// Record in `flags` what `arg` asks for, if it is one of the flags a Run +/// and a Pass both take, with or without its dashes, /// taking from `rest` the address after `email`, if one is there, the /// number after `parallel`, and the name or level after `harness`, `model` /// and `effort`. False, taking nothing, if `arg` is none of them. @@ -231,6 +246,30 @@ fn take_flag<'a>( "no-base-fix" | "--no-base-fix" => { ask_once(&mut flags.base_fix, BaseFixAsk::Forbid, arg, BASE_FIX_FLAGS)? } + "security-fix" | "--security-fix" => ask_once( + &mut flags.security_fix, + crate::security::FixAsk::Allow, + arg, + SECURITY_FIX_FLAGS, + )?, + "no-security-fix" | "--no-security-fix" => ask_once( + &mut flags.security_fix, + crate::security::FixAsk::Forbid, + arg, + SECURITY_FIX_FLAGS, + )?, + "security-review" | "--security-review" => ask_once( + &mut flags.security_review, + crate::security::review::Ask::Allow, + arg, + SECURITY_REVIEW_FLAGS, + )?, + "no-security-review" | "--no-security-review" => ask_once( + &mut flags.security_review, + crate::security::review::Ask::Forbid, + arg, + SECURITY_REVIEW_FLAGS, + )?, "harness" | "--harness" => ask_harness(&mut flags.harness.harness, arg, rest.next())?, "model" | "--model" => { let model = &mut flags.harness.model_and_effort.model; @@ -248,6 +287,8 @@ fn take_flag<'a>( const MERGE_FLAGS: &str = "merge and no-merge"; const EMAIL_FLAGS: &str = "email and no-email"; const BASE_FIX_FLAGS: &str = "base-fix and no-base-fix"; +const SECURITY_REVIEW_FLAGS: &str = "security-review and no-security-review"; +const SECURITY_FIX_FLAGS: &str = "security-fix and no-security-fix"; /// Record in `given` what the flag `arg` asked for: the same kind of ask /// twice is a repeated argument, and a different one contradicts the first, @@ -327,6 +368,22 @@ mod tests { const URL: &str = "https://github.com/acme/widgets/issues/7"; + #[test] + fn security_review_words_are_accepted_on_commands_that_start_runs() { + for command in [URL, "pickup", "architect", "secure"] { + for word in [ + "security-review", + "--security-review", + "no-security-review", + "--no-security-review", + ] { + assert!(parse_strs(&[command, word]).is_ok(), "{command} {word}"); + } + } + assert!(parse_strs(&[URL, "security-review", "no-security-review"]).is_err()); + assert!(parse_strs(&[URL, "security-review", "--security-review"]).is_err()); + } + fn parse_strs(args: &[&str]) -> Result<Command> { let args: Vec<String> = args.iter().map(|arg| arg.to_string()).collect(); parse(&args) @@ -691,7 +748,7 @@ mod tests { } /// The Pickup run `args` parse to. - fn pickup_args(args: &[&str]) -> PickupArgs { + fn pickup_args(args: &[&str]) -> PassArgs { match parse_strs(args) { Ok(Command::Pickup(pickup_args)) => pickup_args, Ok(_) => panic!("{args:?}: not a Pickup run"), @@ -701,14 +758,16 @@ mod tests { #[test] fn pickup_takes_each_of_its_flags_with_or_without_dashes_in_any_order() { - let none = PickupArgs { + let none = PassArgs { base: None, flags: Flags::default(), }; assert_eq!(pickup_args(&["pickup"]), none); - let all = PickupArgs { + let all = PassArgs { base: Some("develop".to_string()), flags: Flags { + security_fix: None, + security_review: None, goal: Some(Goal::Merged), email: Some(NotificationAsk::Send(Some("me@example.com".to_string()))), parallel: NonZeroUsize::new(2), @@ -760,9 +819,11 @@ mod tests { ] { assert_eq!(pickup_args(&args), all, "{args:?}"); } - let cautious = PickupArgs { + let cautious = PassArgs { base: None, flags: Flags { + security_fix: None, + security_review: None, goal: Some(Goal::ReadyForReview), email: Some(NotificationAsk::Skip), parallel: None, diff --git a/src/asks.rs b/src/asks.rs index d70c0c1e..0b398917 100644 --- a/src/asks.rs +++ b/src/asks.rs @@ -14,7 +14,7 @@ use crate::issue::IssueUrl; use crate::notification::NotificationAsk; use crate::run::Goal; -/// The flags a Run, an Architect run and a Pickup run share, as the command +/// The flags a Run and a Pass share, as the command /// gave them: each is none if the command said nothing about it. #[derive(Debug, Default, Clone, PartialEq, Eq)] pub struct Flags { @@ -26,6 +26,10 @@ pub struct Flags { pub parallel: Option<NonZeroUsize>, /// What `base-fix` or `no-base-fix` asked for. pub base_fix: Option<BaseFixAsk>, + /// What `security-fix` or `no-security-fix` asked for. + pub security_fix: Option<crate::security::FixAsk>, + /// What security-review or no-security-review asked for. + pub security_review: Option<crate::security::review::Ask>, /// What `harness <name>`, `model <name>` and `effort <level>` asked for. pub harness: harness::Asked, } @@ -44,6 +48,10 @@ pub struct Asks { pub parallel_asked: bool, /// What it is asked about a Base fix. pub base_fix: BaseFixAsk, + /// Whether a Security run may fix reproduced findings, passed to child Runs. + pub security_fix: bool, + /// Whether Delivery runs the optional guidance Security review. + pub security_review: bool, /// Whether it first brings the Launch directory's checkout of the Base /// branch up to date with origin. pub launch_pull: bool, @@ -77,6 +85,11 @@ impl Asks { tickets_at_once: flags.parallel.unwrap_or(config.spec_parallel), parallel_asked: flags.parallel.is_some(), base_fix, + security_fix: flags.security_fix_allowed(config), + security_review: flags + .security_review + .map(|ask| ask == crate::security::review::Ask::Allow) + .unwrap_or(config.security_review), launch_pull: config.launch_pull, harness: flags.harness(config), } @@ -96,6 +109,8 @@ impl Asks { tickets_at_once: config.spec_parallel, parallel_asked: false, base_fix: given.base_fix.clone(), + security_fix: given.security.fix, + security_review: given.security.review, launch_pull: false, harness: given.harness.clone(), } @@ -130,7 +145,7 @@ impl Asks { Asks::dispatched(issue, &flags, config) } - /// What the Spec run or Run that an Architect run or a Pickup run + /// What the Spec run or Run that a Pass /// dispatches `issue` as is asked: what [`Asks::of_run`] would ask a Run /// started on the Issue URL with `flags`, except that it sends no Run /// notification of its own. The run that dispatched it sends the one, as @@ -144,8 +159,15 @@ impl Asks { } impl Flags { - /// Whether any flag here is for the Spec run or Run that an Architect - /// run or a Pickup run dispatches: every one but `email` and `no-email`, + pub fn security_fix_allowed(&self, config: &UserConfig) -> bool { + match self.security_fix { + Some(crate::security::FixAsk::Allow) => true, + Some(crate::security::FixAsk::Forbid) => false, + None => config.security_fix.unwrap_or(false), + } + } + /// Whether any flag here is for the Spec run or Run that a Pass + /// dispatches: every one but `email` and `no-email`, /// which ask that run for its own Run notification. pub fn any_for_dispatched_run(&self) -> bool { let for_dispatched_run = Flags { @@ -157,7 +179,7 @@ impl Flags { /// What a run with these flags is asked about its own Run notification: /// what `email` or `no-email` asked for, else what `config` says. It is - /// a Run's ask, and an Architect run's or a Pickup run's for the one + /// a Run's ask, and a Pass's for the one /// notification it sends itself. pub fn notification(&self, config: &UserConfig) -> NotificationAsk { self.email.clone().unwrap_or(if config.email.always { @@ -169,11 +191,25 @@ impl Flags { /// The Harness, Model and Effort a command with these flags runs its /// sessions on: each as its flag asked, else as `config` sets it, else - /// the default. It is a Run's, and an Architect run's or a Pickup run's + /// the default. It is a Run's, and a Pass's /// for its own sessions as for the run it dispatches. pub fn harness(&self, config: &UserConfig) -> Choice { Choice::of(&self.harness, &config.harness) } + + /// The Harness of a Security run: command, Security setting, default + /// Harness, then Claude Code. Model and Effort follow the chosen Harness. + pub fn security_harness(&self, config: &UserConfig) -> Choice { + let settings = harness::Settings { + default: config.security_harness.or(config.harness.default), + ..config.harness.clone() + }; + let mut choice = Choice::of(&self.harness, &settings); + if self.harness.harness.is_none() && config.security_harness.is_some() { + choice.chosen_by = harness::ChosenBy::SecurityConfig; + } + choice + } } /// The command that starts the Run on `issue` again as `flags` asked for it, @@ -194,6 +230,16 @@ fn retry_with_base_fix(issue: &IssueUrl, flags: &Flags) -> String { if let Some(parallel) = flags.parallel { command += &format!(" parallel {parallel}"); } + match flags.security_fix { + Some(crate::security::FixAsk::Allow) => command += " security-fix", + Some(crate::security::FixAsk::Forbid) => command += " no-security-fix", + None => {} + } + match flags.security_review { + Some(crate::security::review::Ask::Allow) => command += " security-review", + Some(crate::security::review::Ask::Forbid) => command += " no-security-review", + None => {} + } let asked = &flags.harness; if let Some(harness) = asked.harness { command += &format!(" harness {}", harness.name()); @@ -244,6 +290,9 @@ mod tests { pickup_limit: n(3), pickup_wait: chrono::TimeDelta::minutes(30), harness: harness::Settings::default(), + security_harness: None, + security_fix: None, + security_review: false, } } @@ -323,6 +372,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::ReadyForReview, notification: NotificationAsk::Skip, tickets_at_once: n(3), @@ -341,6 +392,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, notification: NotificationAsk::Send(None), tickets_at_once: n(5), @@ -355,6 +408,8 @@ mod tests { #[test] fn each_flag_decides_its_ask_whatever_the_user_config_says() { let against = Flags { + security_fix: None, + security_review: None, goal: Some(Goal::ReadyForReview), email: Some(NotificationAsk::Skip), parallel: Some(n(2)), @@ -365,6 +420,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::ReadyForReview, notification: NotificationAsk::Skip, tickets_at_once: n(2), @@ -377,6 +434,8 @@ mod tests { ); let for_it = Flags { + security_fix: None, + security_review: None, goal: Some(Goal::Merged), email: Some(to("flag@example.com")), parallel: Some(n(2)), @@ -387,6 +446,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, notification: to("flag@example.com"), tickets_at_once: n(2), @@ -402,6 +463,7 @@ mod tests { /// fix, its sessions on Claude's `sonnet`. fn given(kind: Kind, base_fix: BaseFixAsk) -> Given { Given { + security: crate::security::Options::default(), kind, stamp: "20261003T120000-0400".to_string(), base_fix, @@ -461,10 +523,12 @@ mod tests { } } - /// Every flag an Architect run or a Pickup run takes for the run it + /// Every flag a Pass takes for the run it /// dispatches, with `email` for its own Run notification. fn dispatch_flags() -> Flags { Flags { + security_fix: None, + security_review: None, goal: Some(Goal::Merged), email: Some(to("flag@example.com")), parallel: Some(n(2)), @@ -479,6 +543,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, notification: NotificationAsk::Skip, tickets_at_once: n(2), @@ -495,6 +561,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, // Whatever `email.always` says: the Architect run sends it. notification: NotificationAsk::Skip, @@ -513,6 +581,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, notification: NotificationAsk::Skip, tickets_at_once: n(2), @@ -529,6 +599,8 @@ mod tests { assert_eq!( asks, Asks { + security_fix: false, + security_review: false, goal: Goal::Merged, // Whatever `email.always` says: the Pickup run sends it. notification: NotificationAsk::Skip, @@ -613,6 +685,8 @@ mod tests { #[test] fn the_retry_command_is_the_issue_url_and_the_flags_as_given_with_base_fix_added() { let flags = |goal, email, parallel| Flags { + security_fix: None, + security_review: None, goal, email, parallel, diff --git a/src/base_fix.rs b/src/base_fix.rs index 61e9d801..6225fc6f 100644 --- a/src/base_fix.rs +++ b/src/base_fix.rs @@ -23,6 +23,7 @@ use crate::issue::IssueUrl; use crate::labels::{Label, Labels, READY_FOR_AGENT}; use crate::poll; use crate::progress; +use crate::security::Options; /// The label that marks a Base fix issue, which a Run finds an open one by. pub const BASE_FIX: Label = Label::new("base-fix", "A Base fix: CI is red on a Base branch"); @@ -61,11 +62,10 @@ const RETRY_WITH: &str = "Retry with"; /// What starts the line of [`Advice`] naming the User config's `base.fix`. const OR_SET: &str = "Or set"; -/// A line of the advice a Run gives after its cause when Inherited failures -/// fail it with no Base fix taken: each check where it fails on the Base -/// branch, then, if nobody decided against a Base fix, how to allow one. It -/// reads `<label>: <value>` on stderr, and the Run notification lays it out -/// like its other lines. +/// A line of advice shared by stderr and the Run notification. A Base fix's +/// advice names the failing checks and how to allow fixing; a Security run's +/// offers fixing when the operator left it undecided. Each line reads +/// `<label>: <value>` on stderr; the notification lays it out like its other lines. #[derive(Debug, Clone, PartialEq, Eq)] pub struct Advice { /// What the line starts with, saying what its value is. @@ -116,6 +116,7 @@ pub struct BaseFix { /// The Harness, Model and Effort a Base fix it starts runs its sessions /// on: the Run's. harness: Choice, + security: Options, } /// The Base fix a Run took as its one: one it started, or one it found @@ -141,7 +142,7 @@ impl BaseFix { /// that asked `ask` about a Base fix, and whose sessions run on /// `harness`, as a Base fix it starts does. A Base fix starts none of /// its own, whatever it asked. - pub fn new(child: Option<&Kind>, ask: BaseFixAsk, harness: Choice) -> Self { + pub fn new(child: Option<&Kind>, ask: BaseFixAsk, harness: Choice, security: Options) -> Self { let (on_inherited_failures, retry) = match (child, ask) { (Some(Kind::BaseFix { .. }), _) => (OnInheritedFailures::IsBaseFix, None), (_, BaseFixAsk::Allow) => (OnInheritedFailures::StartBaseFix, None), @@ -154,6 +155,7 @@ impl BaseFix { taken: None, advice: Vec::new(), harness, + security, } } @@ -218,6 +220,7 @@ impl BaseFix { launch, issue, harness: &harness, + security: self.security, }; self.fix_through(&mut outside, issue, pr_url, base, base_commit, failed) } @@ -470,6 +473,7 @@ struct LaunchAndGitHub<'a> { launch: &'a Git, issue: &'a IssueUrl, harness: &'a Choice, + security: Options, } impl Outside for LaunchAndGitHub<'_> { @@ -527,7 +531,7 @@ impl Outside for LaunchAndGitHub<'_> { let kind = Kind::BaseFix { base: base.to_string(), }; - child_run::start(fix, kind, BaseFixAsk::Forbid, self.harness)?.wait() + child_run::start(fix, kind, BaseFixAsk::Forbid, self.security, self.harness)?.wait() } fn pause(&mut self) -> Result<()> { @@ -875,7 +879,7 @@ mod tests { /// A Run's Base fix as asked `ask`, not itself a Base fix. fn asked(ask: BaseFixAsk) -> BaseFix { - BaseFix::new(None, ask, Choice::default()) + BaseFix::new(None, ask, Choice::default(), Options::default()) } fn undecided() -> BaseFixAsk { @@ -1322,7 +1326,7 @@ mod tests { base: "main".to_string(), }; for ask in [BaseFixAsk::Allow, BaseFixAsk::Forbid, undecided()] { - let base_fix = BaseFix::new(Some(&child), ask, Choice::default()); + let base_fix = BaseFix::new(Some(&child), ask, Choice::default(), Options::default()); assert!(!base_fix.sees_inherited_failures()); assert_eq!(base_fix.ask_of_tickets(), BaseFixAsk::Forbid); } @@ -1330,8 +1334,13 @@ mod tests { spec_branch: "spec-3".to_string(), }; assert!( - BaseFix::new(Some(&ticket), BaseFixAsk::Allow, Choice::default()) - .sees_inherited_failures() + BaseFix::new( + Some(&ticket), + BaseFixAsk::Allow, + Choice::default(), + Options::default() + ) + .sees_inherited_failures() ); assert!(asked(BaseFixAsk::Forbid).sees_inherited_failures()); } diff --git a/src/child_run.rs b/src/child_run.rs index 8a9d0b7b..4c5267db 100644 --- a/src/child_run.rs +++ b/src/child_run.rs @@ -24,6 +24,7 @@ use crate::issue::IssueUrl; use crate::logs; use crate::progress; use crate::run_ending; +use crate::security::Options; #[cfg(test)] use execution_tests::Fault; @@ -71,6 +72,7 @@ pub struct Given { pub stamp: String, /// What it is asked about a Base fix. pub base_fix: BaseFixAsk, + pub security: Options, /// The Harness, Model and Effort its sessions run on: the command's /// that started it. pub harness: Choice, @@ -91,6 +93,8 @@ const STAMP: &str = "--stamp"; /// The hidden argument that asks a child Run [`BaseFixAsk::Allow`]. const ALLOW_BASE_FIX: &str = "--allow-base-fix"; +const SECURITY_REVIEW: &str = "--run-security-review"; +const ALLOW_SECURITY_FIX: &str = "--allow-security-fix"; /// The hidden argument followed by the command that starts the Spec run /// again with a Base fix allowed, for the child Run to offer: it asks @@ -125,6 +129,12 @@ impl Given { BaseFixAsk::Forbid => {} BaseFixAsk::Undecided { retry } => args.extend([OFFER_BASE_FIX, retry]), } + if self.security.fix { + args.push(ALLOW_SECURITY_FIX); + } + if self.security.review { + args.push(SECURITY_REVIEW); + } args.extend([SESSIONS_HARNESS, self.harness.harness.name()]); if let Some(model) = &self.harness.model { args.extend([SESSIONS_MODEL, model]); @@ -144,6 +154,7 @@ pub struct Reader { kind: Option<Kind>, stamp: Option<String>, base_fix: Option<BaseFixAsk>, + security: Options, harness: Option<Harness>, model: Option<String>, effort: Option<String>, @@ -163,6 +174,18 @@ impl Reader { None => bail!("missing {missing}"), }; match arg { + SECURITY_REVIEW => { + if self.security.review { + bail!("repeated argument: {arg}"); + } + self.security.review = true; + } + ALLOW_SECURITY_FIX => { + if self.security.fix { + bail!("repeated argument: {arg}"); + } + self.security.fix = true; + } SPEC_BRANCH | BASE_FIX_INTO => { not_yet_given(&self.kind, arg)?; let base = value("Base branch")?; @@ -218,6 +241,8 @@ impl Reader { let Some(kind) = self.kind else { if self.stamp.is_some() || self.base_fix.is_some() + || self.security.review + || self.security.fix || self.harness.is_some() || self.model.is_some() || self.effort.is_some() @@ -237,6 +262,7 @@ impl Reader { kind, stamp, base_fix: self.base_fix.unwrap_or(BaseFixAsk::Forbid), + security: self.security, harness: Choice { harness, model: self.model, @@ -298,12 +324,14 @@ pub fn start( issue: &IssueUrl, kind: Kind, base_fix: BaseFixAsk, + security: Options, harness: &Choice, ) -> Result<Handle> { let given = Given { kind, stamp: logs::stamp(), base_fix, + security, harness: harness.clone(), }; start_from(&own_executable()?, issue, &given) @@ -681,6 +709,7 @@ mod tests { kind: ticket(), stamp: STAMPED.to_string(), base_fix: BaseFixAsk::Forbid, + security: Options::default(), harness: Choice::default(), }; let Err(error) = start_from(executable, &issue, &given) else { @@ -738,6 +767,7 @@ mod tests { kind: kind.clone(), stamp: STAMPED.to_string(), base_fix, + security: Options::default(), harness: harness.clone(), }; let written = given.to_args(); diff --git a/src/child_run/execution_tests.rs b/src/child_run/execution_tests.rs index eb1996a8..4681924a 100644 --- a/src/child_run/execution_tests.rs +++ b/src/child_run/execution_tests.rs @@ -167,6 +167,7 @@ impl Fixture { }, stamp: "20261003T120000-0400".to_string(), base_fix: BaseFixAsk::Forbid, + security: crate::security::Options::default(), harness: Choice::default(), }, startup, diff --git a/src/child_run/runs.rs b/src/child_run/runs.rs index da58eb5d..e7c3d420 100644 --- a/src/child_run/runs.rs +++ b/src/child_run/runs.rs @@ -9,6 +9,7 @@ use super::{Ended, Handle, Kind, POLL}; use crate::base_fix::BaseFixAsk; use crate::harness::Choice; use crate::issue::IssueUrl; +use crate::security::Options; /// Own child Runs until their numbered endings are delivered or cleanup finishes. #[derive(Default)] @@ -26,10 +27,11 @@ impl Runs { issue: &IssueUrl, kind: Kind, base_fix: BaseFixAsk, + security: Options, harness: &Choice, ) -> Result<()> { self.start_using(issue.number, || { - super::start(issue, kind, base_fix, harness) + super::start(issue, kind, base_fix, security, harness) }) } diff --git a/src/claim.rs b/src/claim.rs index 351eb0ed..050e6274 100644 --- a/src/claim.rs +++ b/src/claim.rs @@ -3,7 +3,8 @@ //! Run or a Spec run is working on. It is made before the run's work, and //! ended once with how the run ended: removed once its Self-merge has left the //! issue closed, kept while its pull request waits for review, and released -//! when the run failed with nothing on origin to take over. +//! when the run failed with nothing on origin to take over, except a failed +//! Security fix, which stays with the Day shift. //! //! What it reads and changes of GitHub and origin, whether the run is //! interrupted, and its progress lines all go through [`Outside`]: @@ -63,7 +64,7 @@ impl Claim<'_> { /// closed. /// - Ready for review: the Claim stays while the pull request waits. /// - Failed: the Claim is released if nothing of the run is on origin to - /// take over. + /// take over, except a Security fix, whose Claim stays. /// /// The run has ended as it has, so this never fails: a failure is a /// warning naming what to run by hand, and an interrupt doesn't stop it. @@ -156,6 +157,8 @@ struct Claimed<'a> { added_in_progress: bool, /// Whether making it took `ready-for-agent` off the issue. removed_ready_for_agent: bool, + /// A failed Security fix stays with the Day shift even without a push. + security_fix: bool, } impl<'a> Claimed<'a> { @@ -177,6 +180,7 @@ impl<'a> Claimed<'a> { issue, added_in_progress: edit.puts_on(IN_PROGRESS), removed_ready_for_agent: edit.takes_off(READY_FOR_AGENT), + security_fix: edit.labels().has(crate::security::fixing::SECURITY_FIX), }; if !claimed.changed_any() { return Ok(claimed); @@ -198,6 +202,7 @@ impl<'a> Claimed<'a> { match reached { Some(Goal::Merged) => self.remove_if_closed(outside), Some(Goal::ReadyForReview) => {} + None if self.security_fix => {} None => self.release_if_nothing_on_origin(outside), } } diff --git a/src/command.rs b/src/command.rs index 2492c226..bce4ba1c 100644 --- a/src/command.rs +++ b/src/command.rs @@ -1,6 +1,6 @@ //! A command's lifecycle, from its start to its exit code, for every command -//! that does factory work: a Run or Spec run started on an Issue URL, an -//! Architect run or a Pickup run. A child Run goes through it too, as part +//! that does factory work: a Run or Spec run started on an Issue URL, +//! or a Pass. A child Run goes through it too, as part //! of the command that started it. //! //! A command [`start`]s: it begins its record in the logs, loads the User @@ -40,8 +40,13 @@ pub enum Ending { review: Result<Reviewed, FailedRun>, dispatched: Option<Ended>, }, - /// A pass, an Architect run or a Pickup run, was skipped before any - /// work. + /// A Security run completed its audit, or failed. + Security(crate::security::Ended), + SecurityFix { + ended: Ended, + findings: Vec<crate::security::RecordedFinding>, + }, + /// A Pass was skipped before any work. Skipped(Skip), } diff --git a/src/config.rs b/src/config.rs index 4f978ec6..9815acbf 100644 --- a/src/config.rs +++ b/src/config.rs @@ -29,7 +29,7 @@ pub struct UserConfig { /// each repository's in `<owner>/<repo>/`, by default /// `~/.thirdshift/logs`. pub logs_dir: PathBuf, - /// `activity.quiet_skips`: a skipped Architect run or Pickup run prints + /// `activity.quiet_skips`: a skipped Pass prints /// nothing, its Activity log line its only trace. pub quiet_skips: bool, /// The `[email]` section. @@ -47,6 +47,12 @@ pub struct UserConfig { /// The `[harness]` section: the Harness every Command runs its sessions /// on, and a Model and Effort for each Harness. pub harness: harness::Settings, + /// `security.harness`: the Harness a Security run uses ahead of + /// `harness.default`, unless its command names one. + pub security_harness: Option<Harness>, + /// `security.fix`: off by default; None leaves the decision unmade. + pub security_fix: Option<bool>, + pub security_review: bool, } /// The `[email]` section: where email goes and who it comes from. The Resend @@ -89,6 +95,9 @@ impl UserConfig { pickup_limit: NonZeroUsize::new(3).unwrap(), pickup_wait: TimeDelta::minutes(30), harness: harness::Settings::default(), + security_harness: None, + security_fix: None, + security_review: false, } } @@ -114,6 +123,7 @@ impl UserConfig { | "spec" | "pickup" | "harness" + | "security" ); let settings = match value { Value::Table(settings) if known => settings, @@ -169,6 +179,27 @@ impl UserConfig { ) })?; } + ("security", "review", Value::Boolean(review)) => { + config.security_review = *review + } + ("security", "review", _) => { + bail!("security.review must be true or false in {file}") + } + ("security", "fix", Value::Boolean(fix)) => config.security_fix = Some(*fix), + ("security", "fix", _) => bail!("security.fix must be true or false in {file}"), + ("security", "harness", Value::String(name)) => { + let name = name.trim(); + if !name.is_empty() { + config.security_harness = Some(Harness::named(name).with_context(|| { + format!("security.harness must be {}, or blank, not {name:?}, in {file}", harness::names()) + })?); + } + } + ("security", "harness", _) => { + bail!( + "security.harness must be a quoted name, blank for the default Harness, in {file}" + ) + } ("harness", "default", Value::String(name)) => match Harness::named(name) { Some(harness) => config.harness.default = Some(harness), None => bail!( @@ -207,6 +238,8 @@ pub struct UserConfigChanges { pub merge_always: bool, pub base_fix: bool, pub launch_pull: bool, + pub security_fix: bool, + pub security_review: bool, /// None disables Run notifications while retaining their addresses. pub notifications: Option<NotificationAddresses>, /// None retains every Harness setting. Unset Model/Effort writes blank. @@ -255,6 +288,8 @@ impl UserConfigDocument { set(&mut document, "merge", "always", changes.merge_always); set(&mut document, "base", "fix", changes.base_fix); set(&mut document, "launch", "pull", changes.launch_pull); + set(&mut document, "security", "fix", changes.security_fix); + set(&mut document, "security", "review", changes.security_review); set( &mut document, "email", @@ -660,7 +695,7 @@ from = "onboarding@resend.dev" # the sender; default onboarding@resend.dev, whi dir = "~/.thirdshift/logs" # the root of the logs, each repository's in <owner>/<repo>/; default ~/.thirdshift/logs [activity] -quiet_skips = false # a skipped Architect run or Pickup run prints nothing, leaving only its Activity log line; default false +quiet_skips = false # a skipped Pass prints nothing, leaving only its Activity log line; default false [spec] parallel = 3 # how many Tickets a Spec run runs at once; default 3 @@ -669,6 +704,11 @@ parallel = 3 # how many Tickets a Spec run runs at once; default 3 limit = 3 # how many open issues labelled in-progress stop a Pickup run taking another; default 3 wait_minutes = 30 # minutes since the latest shaping event before an issue is Ready; 0 disables waiting; default 30 +[security] +fix = false # Security runs may fix reproduced findings; default false +review = false # Runs review their change for Security findings; default false +harness = "" # the Harness a Security run uses unless its command names one; default blank, for harness.default or Claude Code + [harness] default = "claude" # the Harness every Run's sessions run on, claude, codex, agy, grok, muse or opencode; default claude @@ -790,6 +830,8 @@ mod tests { config.email.from = None; assert_eq!(config.harness.default, Some(Harness::Claude)); config.harness.default = None; + assert_eq!(config.security_fix, Some(false)); + config.security_fix = None; assert_eq!(config, UserConfig::defaults(Path::new("/home/me"))); } @@ -823,6 +865,9 @@ mod tests { "spec.parallel", "pickup.limit", "pickup.wait_minutes", + "security.fix", + "security.review", + "security.harness", "harness.default", "harness.claude.model", "harness.claude.effort", @@ -1174,6 +1219,8 @@ mod tests { merge_always: true, base_fix: true, launch_pull: true, + security_fix: false, + security_review: false, notifications: Some(NotificationAddresses { to: "me@example.com".to_string(), from: "ts@example.com".to_string(), @@ -1217,6 +1264,10 @@ mod tests { assert_eq!(after.harness.default, Some(Harness::Claude)); after.harness.default = None; } + if before.security_fix.is_none() { + assert_eq!(after.security_fix, Some(false)); + after.security_fix = None; + } assert_eq!(after, before, "{completed}"); let again = document(&completed).render(None); assert_eq!(again, completed, "completing twice changed it"); @@ -1323,6 +1374,8 @@ mod tests { merge_always: false, base_fix: false, launch_pull: false, + security_fix: false, + security_review: false, notifications: Some(NotificationAddresses { to: to.to_string(), from: crate::email::DEFAULT_FROM.to_string(), @@ -1415,6 +1468,8 @@ mod tests { merge_always: true, base_fix: false, launch_pull: false, + security_fix: false, + security_review: false, notifications: None, harness: Some(( Harness::Codex, @@ -1458,6 +1513,8 @@ mod tests { merge_always: true, base_fix: true, launch_pull: true, + security_fix: false, + security_review: false, notifications: Some(NotificationAddresses { to: "o\"brien@example.com".to_string(), from: "ts@example.com".to_string(), @@ -1542,6 +1599,8 @@ mod tests { merge_always: true, base_fix: false, launch_pull: true, + security_fix: false, + security_review: false, notifications: None, harness: None, }; diff --git a/src/delivery.rs b/src/delivery.rs index e1c0a05f..534ad960 100644 --- a/src/delivery.rs +++ b/src/delivery.rs @@ -1,5 +1,6 @@ //! Delivery: what a Run, or a Spec run for its Spec PR, does from its -//! worktree once its work begins. The opening session, the push after it, +//! worktree once its work begins. The opening session, the optional guidance +//! Security review, the push after it, //! marking the pull request ready, the Repair loop that keeps it mergeable //! and green, the Self-merge in a Merge run and its steps after the merge, //! and the Failed run path when any of these fails. @@ -29,6 +30,7 @@ use crate::progress; use crate::prompt; use crate::run::{Goal, Reached}; use crate::run_ending::Cause; +use crate::security::review::{Hold as SecurityHold, Outcome as SecurityOutcome, Review}; use crate::session::{Logs, Sessions}; use crate::worktree::{ForeignCommits, Merge, PendingMerge, Worktree}; @@ -36,6 +38,9 @@ use repair_loop::{Repair, Upstream}; /// A Delivery of the pull request for `issue`, the Run's issue or the Spec, /// into the Base branch `base`, to `goal`. pub struct Delivery<'a> { + pub security_fix: bool, + /// Whether the opening session is followed by a guidance Security review. + pub security_review: bool, pub issue: &'a IssueUrl, pub base: &'a str, pub goal: Goal, @@ -63,7 +68,8 @@ pub struct Opening<'a> { impl Delivery<'_> { /// Take the pull request from the branch checked out in `worktree` to the /// goal. With `opening.catch_up_from_origin`, first fast-forward the - /// branch to origin. Then run the opening session, push the branch, for + /// branch to origin. Then run the opening session and optional Security + /// review, push the branch, for /// any commit the session left unpushed, write the line that says what it /// was built with in the pull request's body, only warning if that fails, /// restore the optional Tickets checklist, and mark the pull request ready, @@ -82,6 +88,7 @@ impl Delivery<'_> { checklist: Option<&str>, ) -> Result<Reached, FailedRun> { let route = Route { + security_review: self.security_review, issue: self.issue, base: self.base, branch: worktree.branch(), @@ -123,6 +130,7 @@ impl Delivery<'_> { /// The steps of a Delivery of the pull request for `issue`, from `branch` /// into the Base branch `base`, to `goal`, its sessions run on `harness`. struct Route<'a> { + security_review: bool, issue: &'a IssueUrl, base: &'a str, branch: &'a str, @@ -146,10 +154,40 @@ impl Route<'_> { outside.catch_up()?; } outside.session(opening.kind, &opening.prompt)?; + let review = self.security_review.then(|| { + SecurityOutcome::of(outside.security_review(&prompt::security_review( + self.issue, + self.base, + self.branch, + ))) + }); outside.push()?; self.write_built_with(outside); let pr = outside.mark_pr_ready(checklist)?; - outside.take_to_goal(&pr.url, self.goal)?; + let hold = review.as_ref().and_then(SecurityOutcome::hold); + if let Some(review) = &review + && let Err(error) = outside.record_security_review(review) + { + outside.warn( + &error, + "could not write the Security review outcome in the pull request's body" + .to_string(), + ); + } + // A Security hold prevents Self-merge, while ordinary readiness still + // includes repairing conflicts and watching the branch's CI. + let goal = if hold.is_some() { + Goal::ReadyForReview + } else { + self.goal + }; + outside.take_to_goal(&pr.url, goal)?; + if let Some(hold) = hold { + outside.step(hold.to_string()); + if self.goal == Goal::Merged { + return Err(hold.into()); + } + } if self.goal == Goal::Merged { self.after_merge(outside, &pr); } @@ -236,7 +274,7 @@ fn retry_if_interrupted<O: Outside>( /// sessions have ended with `error`: commit and push the work, and send an /// open PR back to draft. An interrupt, if one was requested, is the error /// instead: it can surface as some other error, such as a killed git. A -/// `PolicyRefusal` neither pushes nor converts, so the PR stays ready on the +/// `PolicyRefusal` or `SecurityHold` neither pushes nor converts, so the PR stays ready on the /// head whose CI was watched. Problems along the way are reported, not /// raised, so the error is what the Run fails with. Worktree owns preservation /// and retains work that may not have reached origin. `log` is the most recent @@ -248,7 +286,7 @@ fn fail(outside: &mut impl FailedOutside, log: Option<PathBuf>, error: anyhow::E // to its first line: the reason goes in the failure commit's subject. let reason = Cause::of(&error).first_line().to_string(); // Everything is already on origin: the Repair loop pushed the head. - let keep_ready = error.is::<PolicyRefusal>(); + let keep_ready = error.is::<PolicyRefusal>() || error.is::<SecurityHold>(); if !keep_ready && let Err(problem) = outside.preserve_failed_run(&reason) { outside.step(format!( "could not push the failed run's work, so it may exist only locally: {problem:#}" @@ -277,6 +315,8 @@ trait Outside { fn catch_up(&mut self) -> Result<()>; /// Run the session `kind` given `prompt`. fn session(&mut self, kind: &str, prompt: &str) -> Result<()>; + fn security_review(&mut self, prompt: &str) -> Result<Review>; + fn record_security_review(&mut self, review: &SecurityOutcome) -> Result<()>; /// Push the Issue branch. fn push(&mut self) -> Result<()>; /// Write the annotation through the captured pull request module. @@ -335,6 +375,14 @@ impl Outside for InWorktree<'_> { self.sessions.run(kind, prompt) } + fn security_review(&mut self, prompt: &str) -> Result<Review> { + crate::security::review::run(self.sessions, self.worktree, self.issue, self.base, prompt) + } + + fn record_security_review(&mut self, review: &SecurityOutcome) -> Result<()> { + self.pull_request.record_security_review(&review.entry()) + } + fn push(&mut self) -> Result<()> { self.worktree.push() } @@ -576,6 +624,14 @@ mod tests { } impl Outside for Scripted { + fn record_security_review(&mut self, _: &SecurityOutcome) -> Result<()> { + Ok(()) + } + fn security_review(&mut self, _: &str) -> Result<Review> { + Review::from_final_message(Some( + "Security review: {\"unaddressed_count\":0,\"findings\":[],\"pre_existing_count\":0}", + )) + } fn catch_up(&mut self) -> Result<()> { self.calls.push(Call::CatchUp); self.check(Fails::CatchUp) @@ -655,6 +711,7 @@ mod tests { fn deliver(outside: &mut Scripted, goal: Goal, spec: bool) -> Result<String, FailedRun> { let issue = IssueUrl::parse("https://github.com/acme/widgets/issues/7").unwrap(); let route = Route { + security_review: false, issue: &issue, base: "main", branch: "issue-7", diff --git a/src/github.rs b/src/github.rs index e8af0e71..bcd543b2 100644 --- a/src/github.rs +++ b/src/github.rs @@ -12,6 +12,10 @@ use crate::issue::IssueUrl; use crate::labels::{Label, Labels}; use crate::process::{self, Control, Interruption}; +mod advisories; +pub(crate) use advisories::FIX_TICKET_MARKER; +pub use advisories::{DraftAdvisory, Package, SecurityRecord, SecurityRecords}; + #[cfg(test)] mod execution_tests; @@ -614,9 +618,13 @@ impl GitHub { .with_context(|| format!("{command} returned invalid JSON")) } fn output(&self, args: &[&str]) -> Result<Output> { + self.output_with_input(args, None) + } + + fn output_with_input(&self, args: &[&str], input: Option<&[u8]>) -> Result<Output> { process::output( Command::new("gh").args(args), - None, + input, Control { name: "gh", interruption: self.interruption, diff --git a/src/github/advisories.rs b/src/github/advisories.rs new file mode 100644 index 00000000..19d114e0 --- /dev/null +++ b/src/github/advisories.rs @@ -0,0 +1,558 @@ +//! Private Security finding records. Public repositories use draft advisories; +//! private repositories whose advisory endpoint is unavailable use issues. + +use anyhow::{Context, Result, bail}; +use serde::Serialize; +use serde_json::{Value, json}; + +use super::GitHub; +use crate::issue::IssueUrl; +use crate::labels::{Label, NEEDS_TRIAGE}; +use crate::security::reproduction::{FixSize, Reproduction, Severity}; + +const SECURITY_FINDING: Label = Label::new( + "security-finding", + "A Security finding recorded privately for the Day shift", +); + +pub(crate) const FIX_TICKET_MARKER: &str = "\n<!-- thirdshift:security-fix -->\nFix Ticket: "; + +const FIX_PENDING: &str = "Fix Run: pending"; +const FIX_FAILED: &str = "Fix Run: failed"; +const FIX_SUCCEEDED: &str = "Fix Run: succeeded"; + +pub enum SecurityRecords { + Advisories(Vec<Value>), + Issues(Vec<Value>), +} + +impl SecurityRecords { + /// The record layout belongs to its storage, not each rule that reads it. + fn entries(&self) -> (&[Value], &'static str, &'static str) { + match self { + Self::Advisories(records) => (records, "description", "draft"), + Self::Issues(records) => (records, "body", "open"), + } + } + + fn values_mut(&mut self) -> &mut Vec<Value> { + match self { + Self::Advisories(records) | Self::Issues(records) => records, + } + } + + /// An unreproduced, untriaged finding still needs the Day shift's call. + pub fn waiting_for_day_shift(&self, fixing: bool) -> bool { + match self { + Self::Advisories(records) => records.iter().any(|record| { + record["state"] == "draft" + && (reproduced_severity(&record["description"]).is_some() + && !fixing + && record["security_fix_closed"] != true + || reproduced_severity(&record["description"]).is_none() + && record["severity"].is_null()) + }), + Self::Issues(records) => records.iter().any(|record| { + record["state"] + .as_str() + .is_some_and(|state| state.eq_ignore_ascii_case("open")) + && (reproduced_severity(&record["body"]).is_some() + && !fixing + && record["security_fix_closed"] != true + || reproduced_severity(&record["body"]).is_none() + && record["labels"].as_array().is_some_and(|labels| { + labels.iter().any(|label| { + label["name"] + .as_str() + .is_some_and(|name| NEEDS_TRIAGE.is_named(name)) + }) + })) + }), + } + } + + /// A dispatched fix that failed and whose issue the Day shift has not closed. + pub fn failed_fix(&self) -> Result<Option<IssueUrl>> { + let (records, field, _) = self.entries(); + for record in records { + if record["security_fix_closed"] == true { + continue; + } + if let Some((_, fix)) = record[field] + .as_str() + .and_then(|text| text.rsplit_once(FIX_TICKET_MARKER)) + // The Pass lock rules out an active dispatch here. A pending + // ending is also a failed/incomplete fix until success is recorded. + && fix.lines().rev().find(|line| line.starts_with("Fix Run: ")) + .is_some_and(|line| line == FIX_PENDING || line == FIX_FAILED) + { + return Ok(Some(IssueUrl::parse( + fix.lines().next().unwrap_or_default(), + )?)); + } + } + Ok(None) + } + + /// Most severe reproduced finding still awaiting a fix, ties in record order. + /// A recorded Ticket has already been dispatched; its Run owns that fix. + pub fn next_fix(&self) -> Result<Option<(SecurityRecord, crate::security::RecordedFinding)>> { + let (records, description, state) = self.entries(); + let mut next = None; + for value in records { + if !value["state"] + .as_str() + .is_some_and(|s| s.eq_ignore_ascii_case(state)) + || has_fix_ticket(&value[description]) + { + continue; + } + if let Some((severity, _)) = + reproduced_outcome(value[description].as_str().unwrap_or_default()) + { + // The Day shift's current advisory grade takes precedence + // over the historical proof-of-concept's score. + let severity = value["severity"] + .as_str() + .and_then(Severity::parse) + .unwrap_or(severity); + if next + .as_ref() + .is_none_or(|(most_severe, _)| severity < *most_severe) + { + next = Some((severity, value)); + } + } + } + next.map(|(severity, value)| { + let mut metadata = crate::security::RecordedFinding::of_record(value)?; + metadata.severity = Some(severity.name().into()); + Ok((self.record(value)?, metadata)) + }) + .transpose() + } + + pub fn remember(&mut self, record: Value) { + self.values_mut().push(record); + } + + pub fn finding(&self, draft: &DraftAdvisory) -> Option<&Value> { + let (records, field, _) = self.entries(); + let marker = format!("Fingerprint: `{}`", draft.fingerprint); + records.iter().find(|record| { + record[field] + .as_str() + .is_some_and(|text| text.lines().any(|line| line == marker)) + }) + } + + pub fn record(&self, value: &Value) -> Result<SecurityRecord> { + Ok(match self { + Self::Advisories(_) => SecurityRecord::Advisory { + id: value["ghsa_id"] + .as_str() + .context("Security finding record has no advisory ID")? + .to_string(), + description: value["description"] + .as_str() + .context("Security finding record has no description")? + .to_string(), + untriaged: value["state"] == "draft" && value["severity"].is_null(), + }, + Self::Issues(_) => SecurityRecord::Issue { + number: value["number"] + .as_u64() + .context("Security finding record has no issue number")?, + description: value["body"] + .as_str() + .context("Security finding record has no body")? + .to_string(), + untriaged: value["state"] + .as_str() + .is_some_and(|state| state.eq_ignore_ascii_case("open")) + && value["labels"].as_array().is_some_and(|labels| { + labels.iter().any(|label| { + label["name"] + .as_str() + .is_some_and(|name| NEEDS_TRIAGE.is_named(name)) + }) + }), + }, + }) + } +} + +fn reproduced_severity(description: &Value) -> Option<Severity> { + reproduced_outcome(description.as_str()?).map(|(severity, _)| severity) +} + +fn reproduced_outcome(description: &str) -> Option<(Severity, FixSize)> { + let (_, reproduction) = + description.split_once("\n<!-- thirdshift:security-reproduction -->\n")?; + let outcome = reproduction + .lines() + .find_map(|line| line.strip_prefix("Outcome: "))?; + match outcome.split_whitespace().collect::<Vec<_>>().as_slice() { + ["reproduced", severity, "single"] => Some((Severity::parse(severity)?, FixSize::Single)), + ["reproduced", severity, "spec"] => Some((Severity::parse(severity)?, FixSize::Spec)), + _ => None, + } +} + +fn has_fix_ticket(description: &Value) -> bool { + description + .as_str() + .is_some_and(|text| text.contains(FIX_TICKET_MARKER)) +} + +/// A finding read from the private storage selected for this repository. +pub enum SecurityRecord { + Advisory { + id: String, + description: String, + untriaged: bool, + }, + Issue { + number: u64, + description: String, + untriaged: bool, + }, +} + +impl SecurityRecord { + /// The size judged by the completed reproduction, never the candidate write-up. + pub fn fix_size(&self) -> Result<FixSize> { + reproduced_outcome(self.description()) + .map(|(_, size)| size) + .context("the Security finding has no reproduced fix size") + } + + /// A Day-shift decision is the finding's grade; a repeated fingerprint + /// must not publish new proof-of-concept evidence or replace that grade. + pub fn untriaged(&self) -> bool { + match self { + Self::Advisory { untriaged, .. } | Self::Issue { untriaged, .. } => *untriaged, + } + } + + pub fn description(&self) -> &str { + match self { + Self::Advisory { description, .. } | Self::Issue { description, .. } => description, + } + } + + pub fn name(&self) -> String { + match self { + Self::Advisory { id, .. } => id.clone(), + Self::Issue { number, .. } => format!("issue #{number}"), + } + } +} + +#[derive(Clone, Debug, Serialize)] +pub struct Package { + pub ecosystem: String, + pub name: Option<String>, +} + +#[derive(Clone, Debug)] +pub struct DraftAdvisory { + pub fingerprint: String, + pub summary: String, + pub description: String, + pub package: Package, +} + +impl GitHub { + /// Link the dispatched fix issue without changing the private record's grade. + pub fn link_security_fix( + &self, + repo: &str, + record: &SecurityRecord, + issue: &IssueUrl, + ) -> Result<()> { + self.write_security_fix(repo, record, issue, FIX_PENDING) + } + + /// Record the dispatch's ending, preserving any private edits made during it. + pub fn record_security_fix_ending( + &self, + repo: &str, + record: &SecurityRecord, + issue: &IssueUrl, + succeeded: bool, + ) -> Result<()> { + self.write_security_fix( + repo, + record, + issue, + if succeeded { FIX_SUCCEEDED } else { FIX_FAILED }, + ) + } + + fn write_security_fix( + &self, + repo: &str, + record: &SecurityRecord, + issue: &IssueUrl, + ending: &str, + ) -> Result<()> { + let (path, field) = match record { + SecurityRecord::Advisory { id, .. } => ( + format!("repos/{repo}/security-advisories/{id}"), + "description", + ), + SecurityRecord::Issue { number, .. } => { + (format!("repos/{repo}/issues/{number}"), "body") + } + }; + let output = self.output_with_input(&["api", &path], None)?; + if !output.status.success() { + bail!( + "reading the private record before linking its fix failed ({})", + output.status + ); + } + let current: Value = + serde_json::from_slice(&output.stdout).context("private record is invalid JSON")?; + let current = current[field] + .as_str() + .context("the private record has no description")?; + let description = if ending == FIX_PENDING { + if current != record.description() { + bail!("the private record changed while publishing its fix; leaving it unchanged"); + } + // Persist before dispatch: a later write failure cannot reopen Fencing. + format!( + "{}{FIX_TICKET_MARKER}{}\n{ending}\n", + current.trim_end(), + issue.url + ) + } else { + let link = current + .rsplit_once(FIX_TICKET_MARKER) + .and_then(|(_, fix)| fix.lines().next()); + if link != Some(issue.url.as_str()) { + bail!( + "the private record's fix Ticket changed during its Run; leaving it unchanged" + ); + } + format!("{}\n{ending}\n", current.trim_end()) + }; + let body = serde_json::to_vec(&json!({field: description}))?; + let output = self.output_with_input( + &["api", "--method", "PATCH", &path, "--input", "-"], + Some(&body), + )?; + if !output.status.success() { + bail!( + "linking the private record's fix failed ({})", + output.status + ); + } + Ok(()) + } + + pub fn security_fix_text(&self, issue: &IssueUrl) -> Result<String> { + let value = self.issue_view(issue, "title,body")?; + let title = value["title"] + .as_str() + .context("the Security fix Ticket has no title")?; + let body = value["body"] + .as_str() + .context("the Security fix Ticket has no body")?; + Ok(format!("{title}\n\n{body}")) + } + /// Every state and every page. A 404 selects issues only when repository + /// metadata confirms they will be private. + pub fn security_records(&self, repo: &str) -> Result<SecurityRecords> { + let records = match self.gh_api_items( + &format!("repos/{repo}/security-advisories?per_page=100"), + "", + ) { + Ok(records) => Ok(SecurityRecords::Advisories(records)), + Err(error) if format!("{error:#}").contains("(HTTP 404)") => { + let repository = self.gh_json(&["api", &format!("repos/{repo}")])?; + if repository["private"].as_bool() != Some(true) { + bail!( + "security advisories unavailable; refusing to record findings in public issues" + ); + } + let records = self.gh_api_items( + &format!("repos/{repo}/issues?state=all&labels=security-finding&per_page=100"), + "", + )?; + Ok(SecurityRecords::Issues( + records + .into_iter() + .filter(|record| record.get("pull_request").is_none()) + .collect(), + )) + } + Err(error) => Err(error), + }; + let mut records = records?; + let (_, field, _) = records.entries(); + for record in records.values_mut() { + if let Some((_, link)) = record[field] + .as_str() + .and_then(|text| text.rsplit_once(FIX_TICKET_MARKER)) + { + let ticket = IssueUrl::parse(link.lines().next().unwrap_or_default()) + .context("the private record has an invalid fix Ticket link")?; + if !ticket.repo_slug().eq_ignore_ascii_case(repo) { + bail!("the private record's fix Ticket belongs to another repository"); + } + record["security_fix_closed"] = json!(!self.issue_is_open(&ticket)?); + } + } + Ok(records) + } + + /// Use the storage selected while reading the repository's records. + pub fn create_security_record( + &self, + repo: &str, + records: &SecurityRecords, + draft: &DraftAdvisory, + ) -> Result<Value> { + match records { + SecurityRecords::Advisories(_) => self.create_security_advisory(repo, draft), + SecurityRecords::Issues(_) => { + self.ensure_labels(repo, &[SECURITY_FINDING, NEEDS_TRIAGE])?; + let output = self.output_with_input( + &[ + "issue", + "create", + "--repo", + repo, + "--title", + &draft.summary, + "--body-file", + "-", + "--label", + "security-finding,needs-triage", + ], + Some(draft.description.as_bytes()), + )?; + if !output.status.success() { + // API errors can echo private evidence, so do not relay them. + bail!( + "creating a private Security finding issue failed ({})", + output.status + ); + } + let url = std::str::from_utf8(&output.stdout) + .context("creating a private Security finding issue returned invalid UTF-8")?; + let issue = IssueUrl::parse(url.trim()).map_err(|_| { + anyhow::anyhow!( + "creating a private Security finding issue returned no issue URL" + ) + })?; + let mut record = self.issue_view(&issue, "number,body,state,labels,title")?; + record["html_url"] = json!(url.trim()); + Ok(record) + } + } + } + + /// Only the reproduction fields change; state, labels and disclosure stay + /// with the Day shift. API diagnostics may contain private test evidence. + pub fn update_security_record( + &self, + repo: &str, + record: &SecurityRecord, + reproduction: &Reproduction, + ) -> Result<()> { + if !record.untriaged() { + bail!("refusing to replace a triaged Security finding record"); + } + let (path, storage) = match record { + SecurityRecord::Advisory { id, .. } => ( + format!("repos/{repo}/security-advisories/{id}"), + SecurityRecords::Advisories(Vec::new()), + ), + SecurityRecord::Issue { number, .. } => ( + format!("repos/{repo}/issues/{number}"), + SecurityRecords::Issues(Vec::new()), + ), + }; + let observed = self.output_with_input(&["api", &path], None)?; + if !observed.status.success() { + bail!( + "reading a private Security finding record before its update failed ({})", + observed.status + ); + } + let observed: Value = serde_json::from_slice(&observed.stdout) + .context("reading a private Security finding record returned invalid JSON")?; + let current = storage.record(&observed)?; + if !current.untriaged() { + bail!( + "the Security finding was triaged during its reproduction; leaving the record unchanged" + ); + } + if current.description() != record.description() { + bail!( + "the Security finding changed during its reproduction; leaving the record unchanged" + ); + } + if matches!(record, SecurityRecord::Advisory { .. }) + && matches!(reproduction.severity(), Some(Severity::Informational)) + { + bail!( + "informational severity has no GitHub advisory field; the Day shift must decide its representation; leaving the record unchanged" + ); + } + let description = reproduction.description(record.description()); + let body = match record { + SecurityRecord::Advisory { .. } => { + json!({"description": description, "severity": reproduction.severity()}) + } + SecurityRecord::Issue { .. } => json!({"body": description}), + }; + let body = serde_json::to_vec(&body)?; + let output = self.output_with_input( + &["api", "--method", "PATCH", &path, "--input", "-"], + Some(&body), + )?; + if !output.status.success() { + bail!( + "updating a private Security finding record failed ({})", + output.status + ); + } + Ok(()) + } + + /// The create endpoint creates a draft. No severity or version claim. + fn create_security_advisory(&self, repo: &str, draft: &DraftAdvisory) -> Result<Value> { + let body = serde_json::to_vec(&json!({ + "summary": draft.summary, + "description": draft.description, + "severity": null, + "cwe_ids": [], + "vulnerabilities": [{"package": draft.package, "vulnerable_version_range": null}] + }))?; + let output = self.output_with_input( + &[ + "api", + "--method", + "POST", + &format!("repos/{repo}/security-advisories"), + "--input", + "-", + ], + Some(&body), + )?; + if !output.status.success() { + // API errors can echo private evidence, so do not relay them. + bail!( + "creating a draft repository security advisory failed ({})", + output.status + ); + } + serde_json::from_slice(&output.stdout) + .context("creating a draft advisory returned invalid JSON") + } +} diff --git a/src/harness.rs b/src/harness.rs index dc7c81c0..c93efe7d 100644 --- a/src/harness.rs +++ b/src/harness.rs @@ -143,6 +143,8 @@ pub enum ChosenBy { Command, /// The User config's `harness.default`. UserConfig, + /// The User config's `security.harness`. + SecurityConfig, /// Neither: Claude Code, the default. Default, } @@ -257,6 +259,7 @@ impl fmt::Display for ChosenBy { f.write_str(match self { ChosenBy::Command => "the harness flag", ChosenBy::UserConfig => "harness.default in the User config", + ChosenBy::SecurityConfig => "security.harness in the User config", ChosenBy::Default => { "the default, as neither the harness flag nor harness.default in the User \ config names one" diff --git a/src/harness/adapter.rs b/src/harness/adapter.rs index 04468b60..9cfbe34b 100644 --- a/src/harness/adapter.rs +++ b/src/harness/adapter.rs @@ -23,6 +23,18 @@ pub trait Adapter: Sync { /// Arguments for a new session or Resume, and how its prompt reaches /// the CLI. The adapter applies its skill-loading style here. fn session(&self, choice: &Choice, resume: Option<&str>, prompt: &str) -> Invocation; + + /// A Security session or its Resume. Guidance reviews do not permit + /// delegation; other security sessions request fresh sub-agents. + fn security_session( + &self, + choice: &Choice, + resume: Option<&str>, + prompt: &str, + _fresh_sub_agents: bool, + ) -> Invocation { + self.session(choice, resume, prompt) + } fn check(&self, choice: &mut Choice) -> Result<()>; fn ask_settings( &self, diff --git a/src/harness/agy/stream.rs b/src/harness/agy/stream.rs index eee6337f..0206808b 100644 --- a/src/harness/agy/stream.rs +++ b/src/harness/agy/stream.rs @@ -75,6 +75,7 @@ impl Decoder for AgyProgress { report: Report { warnings: Vec::new(), summary, + ..Report::default() }, ended: Ended { session_id: self.session_id.clone(), diff --git a/src/harness/codex.rs b/src/harness/codex.rs index 551b4140..94e02d8e 100644 --- a/src/harness/codex.rs +++ b/src/harness/codex.rs @@ -45,6 +45,31 @@ impl Adapter for Codex { stdin: None, } } + fn security_session( + &self, + choice: &Choice, + resume: Option<&str>, + prompt: &str, + fresh_sub_agents: bool, + ) -> Invocation { + let prompt = if fresh_sub_agents { + format!( + "{prompt}\nStart fresh sub-agents with `fork_turns: \"none\"`, giving each only its own task and necessary evidence, so the skill's verifiers stay independent.\n" + ) + } else { + prompt.to_string() + }; + let mut invocation = self.session(choice, resume, &prompt); + // After `exec`, before any Resume subcommand: repeated on each launch. + invocation.args.splice( + 1..1, + [ + "-c".to_string(), + "agents.max_concurrent_threads_per_session=8".to_string(), + ], + ); + invocation + } fn check(&self, choice: &mut Choice) -> Result<()> { check_model_and_effort(choice) } diff --git a/src/harness/codex/stream.rs b/src/harness/codex/stream.rs index 9d64f017..4ac91fcf 100644 --- a/src/harness/codex/stream.rs +++ b/src/harness/codex/stream.rs @@ -187,6 +187,7 @@ impl Decoder for CodexProgress { report: Report { warnings: Vec::new(), summary, + ..Report::default() }, ended: Ended { session_id: self.session_id, diff --git a/src/harness/interpretation.rs b/src/harness/interpretation.rs index 03bef9a1..1e25933a 100644 --- a/src/harness/interpretation.rs +++ b/src/harness/interpretation.rs @@ -10,6 +10,9 @@ use crate::interrupt; mod stream; pub(super) use stream::Stream; +mod security; +pub use security::SafeguardRefusal; +use security::Security; /// Live interpretation has only line condensation. Consume it after the /// process owner has stopped or waited for the child and joined its workers. @@ -17,6 +20,7 @@ pub struct Interpretation { cli: &'static str, decoder: Box<dyn Decoder>, retained: Retained, + security: Option<Security>, } /// Reporting survives ordinary failures; only success supplies Resume facts. @@ -25,9 +29,12 @@ pub struct Completion { pub outcome: Result<Ended>, } +#[derive(Default)] pub struct Report { pub warnings: Vec<String>, pub summary: Option<String>, + /// Answering Models recovered from retained records, in response order. + pub models: Vec<String>, } #[derive(Debug)] @@ -93,6 +100,7 @@ impl Retained { if log.message.is_some() { facts.ended.final_message = log.message; } + facts.report.models = log.models; } } Self::OpenCode(worktree) => { @@ -102,6 +110,7 @@ impl Retained { // A readable export is authoritative, even without text. facts.ended.final_message = export.message; facts.report.summary = export.summary; + facts.report.models = export.models; if export.failure.is_some() { facts.outcome = TurnOutcome::Failed; facts.diagnostic = facts.diagnostic.take().or(export.failure); @@ -118,6 +127,7 @@ enum Failure { Rejected { status: ExitStatus, diagnostic: Option<String>, + refusal: Option<SafeguardRefusal>, }, } @@ -125,7 +135,11 @@ impl Failure { fn session_error(self, cli: &str) -> anyhow::Error { match self { Self::Execution(error) => error, - Self::Rejected { status, diagnostic } => { + Self::Rejected { + status, + diagnostic, + refusal, + } => { let ended = if status.success() { "'s turn failed".to_string() } else { @@ -136,9 +150,13 @@ impl Failure { .map_or("by signal".to_string(), |code| code.to_string()) ) }; - match diagnostic { + let error = match diagnostic { Some(error) => anyhow!("{cli}{ended}: {error}"), None => anyhow!("{cli}{ended}"), + }; + match refusal { + Some(refusal) => error.context(refusal), + None => error, } } } @@ -151,12 +169,25 @@ impl Interpretation { cli, decoder, retained, + security: None, } } + /// Apply refusal and Model reporting rules independently of the session's log label. + pub fn for_security(mut self, requested_model: Option<&str>) -> Self { + self.security = Some(Security::new(self.cli, requested_model)); + self + } + /// Unknown or malformed lines produce no progress, never an error. pub fn condense(&mut self, raw: &str) -> Vec<String> { - self.decoder.condense(raw) + let mut lines = self.decoder.condense(raw); + if let Some(security) = &mut self.security + && let Some(line) = security.observe(raw) + { + lines.push(line); + } + lines } pub fn finish(self, execution: Result<ExitStatus>) -> Completion { @@ -187,8 +218,16 @@ impl Interpretation { if let Err(error) = interrupt::check() { return (None, Err(Failure::Execution(error))); } + let security_session = self.security.is_some(); let mut facts = self.decoder.complete(); + let refusal = self.security.and_then(|security| security.refusal); + if refusal.is_some() { + facts.outcome = TurnOutcome::Failed; + } let recovered = self.retained.reconcile(&mut facts); + if !security_session { + facts.report.models.clear(); + } if let Err(error) = interrupt::check() { return (None, Err(Failure::Execution(error))); } @@ -200,6 +239,7 @@ impl Interpretation { Err(Failure::Rejected { status, diagnostic: facts.diagnostic, + refusal, }) } else { Ok(facts.ended) diff --git a/src/harness/interpretation/security.rs b/src/harness/interpretation/security.rs new file mode 100644 index 00000000..2ed61ea9 --- /dev/null +++ b/src/harness/interpretation/security.rs @@ -0,0 +1,116 @@ +//! Security session evidence: safeguard refusals are failures even when a +//! Harness reports a successful turn. Raw diagnostics stay in local logs. + +use std::fmt; + +use serde_json::Value; + +use crate::harness::Harness; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SafeguardRefusal { + ClaudeCyber, + CodexCyber, +} + +impl SafeguardRefusal { + pub fn description(self) -> &'static str { + match self { + Self::ClaudeCyber => "Claude Code's [cyber] safeguard refusal", + Self::CodexCyber => "Codex's cybersecurity safeguard refusal", + } + } +} + +impl fmt::Display for SafeguardRefusal { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.description()) + } +} + +impl std::error::Error for SafeguardRefusal {} + +pub(super) struct Security { + harness: Option<Harness>, + pub refusal: Option<SafeguardRefusal>, + last_model: Option<String>, + requested_model: Option<String>, +} + +impl Security { + pub fn new(cli: &str, requested_model: Option<&str>) -> Self { + Self { + harness: Harness::named(cli), + refusal: None, + last_model: None, + requested_model: requested_model.map(String::from), + } + } + + pub fn observe(&mut self, raw: &str) -> Option<String> { + let event = serde_json::from_str::<Value>(raw).ok()?; + let model = match self.harness { + Some(Harness::Claude) => { + let refused = match event["type"].as_str() { + Some("system") => { + event["subtype"] == "model_refusal_no_fallback" + && event["api_refusal_category"] == "cyber" + } + Some("result") => event["result"].as_str().is_some_and(|text| { + let text = text.trim_start(); + text.starts_with("[cyber]") + || (text.starts_with("API Error:") && text.contains("[cyber]")) + }), + _ => false, + }; + if refused { + self.refusal = Some(SafeguardRefusal::ClaudeCyber); + } + Self::answer_model(&event) + } + Some(Harness::Grok) => Self::answer_model(&event), + Some(Harness::Codex) => { + if event["type"] == "turn.failed" + && event["error"]["message"] + .as_str() + .is_some_and(|text| text.to_ascii_lowercase().contains("cybersecurity")) + { + self.refusal = Some(SafeguardRefusal::CodexCyber); + } + if event["type"] == "thread.started" && self.last_model.is_none() { + let model = self.requested_model.as_deref().unwrap_or("Harness default"); + let basis = if self.requested_model.is_some() { + "requested" + } else { + "no Model requested" + }; + let line = format!("Model: {model} ({basis})"); + self.last_model = Some(model.to_string()); + return Some(line); + } + None + } + Some(Harness::Agy) if event["event"] == "init" => event["init"]["model"].as_str(), + _ => None, + }?; + if model.is_empty() || self.last_model.as_deref() == Some(model) { + return None; + } + self.last_model = Some(model.to_string()); + Some(format!("Model: {model}")) + } + + // Init names the requested Model; result.modelUsage includes sub-agents. + // Only main-loop assistant messages identify the Model that answered. + fn answer_model(event: &Value) -> Option<&str> { + if event["type"] != "assistant" + || !event["parent_tool_use_id"].is_null() + || event["is_api_error_message"] == true + { + return None; + } + event["message"]["model"] + .as_str() + .filter(|model| *model != "<synthetic>") + } +} diff --git a/src/harness/interpretation/stream.rs b/src/harness/interpretation/stream.rs index 70269541..3c69fa63 100644 --- a/src/harness/interpretation/stream.rs +++ b/src/harness/interpretation/stream.rs @@ -234,6 +234,7 @@ impl Decoder for Stream { report: Report { warnings: Vec::new(), summary, + ..Report::default() }, ended, outcome: TurnOutcome::from_failed(failed), diff --git a/src/harness/interpretation_tests.rs b/src/harness/interpretation_tests.rs index 5ce9e157..e467e2df 100644 --- a/src/harness/interpretation_tests.rs +++ b/src/harness/interpretation_tests.rs @@ -1,6 +1,79 @@ use super::*; use std::os::unix::process::ExitStatusExt; +#[test] +fn agy_security_progress_names_the_resolved_model() { + let mut interpretation = stream(Harness::Agy, "").for_security(Some("gemini-3.8-flash")); + let lines = interpretation.condense( + r#"{"event":"init","conversation_id":"s1","init":{"model":"gemini-3.8-flash-high"}}"#, + ); + assert!( + lines.contains(&"Model: gemini-3.8-flash-high".to_string()), + "{lines:?}" + ); +} + +#[test] +fn a_security_session_can_discuss_the_cyber_marker_without_being_refused() { + let mut interpretation = stream(Harness::Claude, "").for_security(None); + interpretation.condense( + r#"{"type":"result","subtype":"success","result":"The [cyber] refusal test passed.\nSecurity audit: complete"}"#, + ); + let ended = finish(interpretation).outcome.unwrap(); + assert_eq!( + ended.final_message.as_deref(), + Some("The [cyber] refusal test passed.\nSecurity audit: complete") + ); +} + +#[test] +fn a_terminal_claude_cyber_refusal_overrides_a_successful_security_result() { + let event = r#"{"type":"system","subtype":"model_refusal_no_fallback","api_refusal_category":"cyber","content":"Private safeguard explanation."}"#; + for security in [false, true] { + let mut interpretation = stream(Harness::Claude, ""); + if security { + interpretation = interpretation.for_security(None); + } + interpretation.condense(event); + interpretation.condense( + r#"{"type":"result","subtype":"success","result":"Security audit: complete"}"#, + ); + let completion = finish(interpretation); + if security { + let error = completion.outcome.unwrap_err(); + assert_eq!( + error.downcast_ref::<interpretation::SafeguardRefusal>(), + Some(&interpretation::SafeguardRefusal::ClaudeCyber) + ); + } else { + assert!(completion.outcome.is_ok()); + } + } +} + +#[test] +fn security_model_progress_names_only_new_main_loop_answers() { + let mut interpretation = stream(Harness::Claude, "").for_security(Some("opus")); + for ignored in [ + r#"{"type":"system","subtype":"init","model":"requested-model"}"#, + r#"{"type":"assistant","parent_tool_use_id":"child","message":{"model":"child-model","content":[]}}"#, + r#"{"type":"result","subtype":"success","modelUsage":{"child-model":{"inputTokens":100}}}"#, + ] { + assert!( + interpretation + .condense(ignored) + .iter() + .all(|line| !line.starts_with("Model:")) + ); + } + let answer = r#"{"type":"assistant","message":{"model":"claude-opus-4-8","content":[]}}"#; + assert_eq!( + interpretation.condense(answer), + vec!["Model: claude-opus-4-8"] + ); + assert!(interpretation.condense(answer).is_empty()); +} + #[test] fn claude_completion_keeps_the_last_result_and_the_last_complete_usage() { let mut interpretation = Harness::Claude diff --git a/src/harness/muse/log.rs b/src/harness/muse/log.rs index 46c00322..7d997fcb 100644 --- a/src/harness/muse/log.rs +++ b/src/harness/muse/log.rs @@ -8,6 +8,7 @@ use std::path::{Path, PathBuf}; #[derive(Default)] pub(in crate::harness) struct SessionLog { pub(in crate::harness) message: Option<String>, + pub(in crate::harness) models: Vec<String>, tokens: Option<Tokens>, } @@ -46,6 +47,11 @@ impl SessionLog { self.message = event["text"].as_str().map(String::from); } Some("model_completed") => { + if let Some(model) = event["model"].as_str().filter(|model| !model.is_empty()) + && self.models.last().map(String::as_str) != Some(model) + { + self.models.push(model.to_string()); + } if let Some(usage) = event["usage"].as_object() { let tokens = self.tokens.get_or_insert_default(); tokens.input += usage diff --git a/src/harness/muse/stream.rs b/src/harness/muse/stream.rs index 4ed46f95..09faba82 100644 --- a/src/harness/muse/stream.rs +++ b/src/harness/muse/stream.rs @@ -135,6 +135,7 @@ impl Decoder for MuseProgress { report: Report { warnings: self.skill_load.warnings(), summary: None, + ..Report::default() }, ended: Ended { session_id: self.session_id, diff --git a/src/harness/opencode/export.rs b/src/harness/opencode/export.rs index f3628df1..50d76ad6 100644 --- a/src/harness/opencode/export.rs +++ b/src/harness/opencode/export.rs @@ -11,6 +11,7 @@ pub(in crate::harness) struct SessionExport { pub(in crate::harness) message: Option<String>, pub(in crate::harness) summary: Option<String>, pub(in crate::harness) failure: Option<String>, + pub(in crate::harness) models: Vec<String>, } /// OpenCode records uncached input and visible output separately from cache @@ -64,12 +65,22 @@ impl SessionExport { message: None, summary: None, failure: None, + models: Vec::new(), }; let mut usage: Option<Usage> = None; for message in messages .iter() .filter(|message| message["type"] == "assistant") { + if let Some(id) = message["model"]["id"].as_str().filter(|id| !id.is_empty()) { + let model = message["model"]["providerID"] + .as_str() + .filter(|provider| !provider.is_empty()) + .map_or_else(|| id.to_string(), |provider| format!("{provider}/{id}")); + if result.models.last() != Some(&model) { + result.models.push(model); + } + } result.message = message["content"] .as_array() .and_then(|parts| parts.iter().rev().find(|part| part["type"] == "text")) diff --git a/src/harness/opencode/stream.rs b/src/harness/opencode/stream.rs index 9c9f0e4f..0d6fd4e1 100644 --- a/src/harness/opencode/stream.rs +++ b/src/harness/opencode/stream.rs @@ -85,6 +85,7 @@ impl Decoder for OpenCodeProgress { report: Report { warnings: self.skill_load.warnings(), summary: None, + ..Report::default() }, ended: Ended { session_id: self.session_id, diff --git a/src/launch.rs b/src/launch.rs index 1f7ad05b..45d59d26 100644 --- a/src/launch.rs +++ b/src/launch.rs @@ -1,6 +1,6 @@ //! The Launch directory, as every command opens it: a Run, from an Issue -//! URL, or a run with no Issue URL, an Architect run or a Pickup run. Its -//! `origin` names the repository, git has the identity an agent commits +//! URL, or a Pass, with no Issue URL. Its `origin` names the repository, +//! git has the identity an agent commits //! with, and a branch is checked out there, or none on a detached HEAD. The //! Base branch is the one named for the run, or else the one checked out //! there, and must be on origin with no local copy ahead of it. With @@ -51,7 +51,7 @@ pub struct BaseBranch { enum Opening { /// A Run, from an Issue URL. Run, - /// A run with no Issue URL, an Architect run or a Pickup run, whose + /// A Pass, with no Issue URL, whose /// command can name its Base branch with `base <branch>`. Pass, } @@ -194,6 +194,11 @@ impl BaseBranch { &self.name } + /// The head on origin sampled while preparing this Base branch. + pub fn origin_commit(&self) -> &str { + &self.origin_commit + } + /// The `launch.pull` fast-forward toward the sampled origin commit, /// only while the Base branch is still checked out as at preparation. /// No run depends on this, so a failure is only a warning and local @@ -279,20 +284,15 @@ pub enum Start { AlreadyRunning(AlreadyRunning), } -/// Why a run with no Issue URL is skipped at its start: another on the -/// repository, an Architect run or a Pickup run, is still running on this -/// machine, the Spec run or Run it dispatched included. Its `Display` is the -/// reason, as the skipped run gives it. +/// Why a Pass is skipped at its start: another Pass on the repository is +/// still running on this machine, the Spec run or Run it dispatched included. +/// Its `Display` is the reason, as the skipped run gives it. #[derive(Debug)] pub struct AlreadyRunning(pub Repo); impl fmt::Display for AlreadyRunning { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!( - f, - "an Architect run or a Pickup run is already running on {}", - self.0.slug() - ) + write!(f, "another Pass is already running on {}", self.0.slug()) } } @@ -310,7 +310,7 @@ pub fn start(base: Option<&str>) -> Result<Start> { let repo = directory.repo()?; directory.check_identity()?; let base = directory.prepare_base_branch(base)?; - let Some(lock) = try_run_lock(&repo)? else { + let Some(lock) = try_pass_lock(&repo)? else { return Ok(Start::AlreadyRunning(AlreadyRunning(repo))); }; // Never closed, so the lock is held for as long as this process lives, @@ -330,16 +330,15 @@ pub fn repo() -> Result<Repo> { LaunchDirectory::open(current_dir()?, Opening::Pass)?.repo() } -/// Try, without waiting, for the lock that makes its holder the one Architect -/// run or Pickup run on `repo` on this machine: an advisory lock on a file -/// under `~/.thirdshift` named for the repository, as GitHub compares names, -/// whatever their case. It is held until the file returned is closed, or the +/// Try, without waiting, for the lock that makes its holder the one Pass on +/// `repo` on this machine: an advisory lock on a file under `~/.thirdshift` +/// named for the repository, as GitHub compares names, whatever their case. It is held until the file returned is closed, or the /// process ends. `None` if another process holds it: one of them on `repo` is /// still running. The lock file itself is never deleted, and means nothing /// unless locked. Its directory keeps the name it had when only an Architect /// run took the lock, so that a thirdshift from before the Pickup run and /// one from after still exclude each other. -fn try_run_lock(repo: &Repo) -> Result<Option<File>> { +fn try_pass_lock(repo: &Repo) -> Result<Option<File>> { let dir = config::home()? .join(".thirdshift/architect-locks") .join(repo.owner.to_ascii_lowercase()); diff --git a/src/logs.rs b/src/logs.rs index fffe9cd6..9b541922 100644 --- a/src/logs.rs +++ b/src/logs.rs @@ -16,7 +16,7 @@ mod command_log; mod effects; mod recording; -use std::fmt::Display; +use std::fmt::{self, Display}; use std::path::{Path, PathBuf}; use std::sync::{LazyLock, Mutex, MutexGuard, PoisonError}; @@ -40,6 +40,7 @@ pub enum Begin<'a> { ArchitectRun, /// `thirdshift pickup`, a pass. PickupRun, + SecurityRun, } /// The work a command starts. @@ -53,13 +54,26 @@ pub enum Work<'a> { PickupRun(&'a IssueUrl), /// An Architect run, on its repository. ArchitectRun(&'a Repo), + /// A Security run, on its repository. + SecurityRun(&'a Repo), } -/// A pass, a command that may be skipped before it starts work. -#[derive(Clone, Copy)] +/// A Pass, a command that may be skipped before it starts work. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Pass { - ArchitectRun, - PickupRun, + Architect, + Pickup, + Security, +} + +impl Display for Pass { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.write_str(match self { + Pass::Architect => "Architect run", + Pass::Pickup => "Pickup run", + Pass::Security => "Security run", + }) + } } /// The production command's complete record, shared with reader threads. @@ -89,7 +103,7 @@ pub fn configured(config: &UserConfig) { /// Command log, showing any lines held from the terminal, with a line naming /// the Harness, Model and Effort, then write the Activity log's line that it /// started, naming that log and those. Only the outermost command's first -/// call does anything: the work a Pickup run or an Architect run started +/// call does anything: the work a Pass started /// covers the Run it dispatches, and a child Run records nothing, as the /// command that started it does. pub fn started(work: Work, harness: &Choice) { @@ -161,7 +175,7 @@ impl Work<'_> { fn repo(self) -> Repo { match self { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => issue.repo(), - Work::ArchitectRun(repo) => repo.clone(), + Work::ArchitectRun(repo) | Work::SecurityRun(repo) => repo.clone(), } } @@ -170,8 +184,9 @@ impl Work<'_> { match self { Work::Run(_) => Kind::Run, Work::SpecRun(_) => Kind::SpecRun, - Work::PickupRun(_) => Kind::PickupRun, - Work::ArchitectRun(_) => Kind::ArchitectRun, + Work::PickupRun(_) => Kind::Pass(Pass::Pickup), + Work::ArchitectRun(_) => Kind::Pass(Pass::Architect), + Work::SecurityRun(_) => Kind::Pass(Pass::Security), } } @@ -179,7 +194,7 @@ impl Work<'_> { fn issue(self) -> Option<u64> { match self { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => Some(issue.number), - Work::ArchitectRun(_) => None, + Work::ArchitectRun(_) | Work::SecurityRun(_) => None, } } @@ -195,6 +210,7 @@ impl Work<'_> { Work::Run(issue) | Work::SpecRun(issue) => ("issue", format!("{}-", issue.number)), Work::PickupRun(issue) => ("pickup", format!("{}-", issue.number)), Work::ArchitectRun(_) => ("architect", String::new()), + Work::SecurityRun(_) => ("secure", String::new()), }; root.join("commands") .join(folder) diff --git a/src/logs/activity.rs b/src/logs/activity.rs index c34c287e..2c2f6e39 100644 --- a/src/logs/activity.rs +++ b/src/logs/activity.rs @@ -8,6 +8,7 @@ use std::path::Path; use anyhow::{Context, Result}; +use super::Pass; use super::effects::Tail; pub(super) const FILE: &str = "activity.log"; @@ -19,18 +20,16 @@ const TIME_FORMAT_LENGTH: usize = "2026-10-03 10:41:01".len(); pub(super) enum Kind { Run, SpecRun, - ArchitectRun, - PickupRun, + Pass(Pass), } impl Display for Kind { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - f.write_str(match self { - Kind::Run => "Run", - Kind::SpecRun => "Spec run", - Kind::ArchitectRun => "Architect run", - Kind::PickupRun => "Pickup run", - }) + match self { + Kind::Run => f.write_str("Run"), + Kind::SpecRun => f.write_str("Spec run"), + Kind::Pass(pass) => pass.fmt(f), + } } } diff --git a/src/logs/recording.rs b/src/logs/recording.rs index 094d94dd..4adeb294 100644 --- a/src/logs/recording.rs +++ b/src/logs/recording.rs @@ -83,6 +83,7 @@ impl<O: Outside> Record<O> { Begin::Run(issue) => (format!("starting on {}", issue.url), false), Begin::ArchitectRun => ("Architect run starting".into(), true), Begin::PickupRun => ("Pickup run starting".into(), true), + Begin::SecurityRun => ("Security run starting".into(), true), }; let now = self.outside.now(); self.stamp @@ -132,10 +133,7 @@ impl<O: Outside> Record<O> { } pub(super) fn skipped(&mut self, pass: Pass, repo: &Repo, reason: impl Display) { - let kind = match pass { - Pass::ArchitectRun => Kind::ArchitectRun, - Pass::PickupRun => Kind::PickupRun, - }; + let kind = Kind::Pass(pass); self.activity_write( &self.root(repo), &format!("{kind} skipped: {reason}"), diff --git a/src/logs/recording/tests.rs b/src/logs/recording/tests.rs index 1949060d..f0d90543 100644 --- a/src/logs/recording/tests.rs +++ b/src/logs/recording/tests.rs @@ -431,8 +431,9 @@ fn output_and_start_without_a_factory_beginning_keep_no_command_log() { fn skip(memory: &Scripted, pass: Pass, reason: &str) { let mut record = Record::new(memory.clone()); record.begin(match pass { - Pass::PickupRun => Begin::PickupRun, - Pass::ArchitectRun => Begin::ArchitectRun, + Pass::Pickup => Begin::PickupRun, + Pass::Architect => Begin::ArchitectRun, + Pass::Security => Begin::SecurityRun, }); record.configured(&config(true)); record.eprint(reason); @@ -445,28 +446,28 @@ fn skip(memory: &Scripted, pass: Pass, reason: &str) { #[test] fn separate_recordings_suppress_skips_from_shared_history_until_their_kind_changes() { let memory = Scripted::default(); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::ArchitectRun, "a plan is open"); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Architect, "a plan is open"); + skip(&memory, Pass::Pickup, "idle"); // A Run's lines do not reset the Pickup run's skip. let mut run = Record::new(memory.clone()); run.begin(Begin::Run(&issue())); run.configured(&config(false)); run.started(Work::Run(&issue()), &harness()); run.ended("ready"); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::PickupRun, "at the Claim limit"); - skip(&memory, Pass::PickupRun, "at the Claim limit"); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Pickup, "at the Claim limit"); + skip(&memory, Pass::Pickup, "at the Claim limit"); + skip(&memory, Pass::Pickup, "idle"); let mut pickup = Record::new(memory.clone()); pickup.begin(Begin::PickupRun); pickup.configured(&config(true)); pickup.started(Work::PickupRun(&issue()), &harness()); // Even just an intervening start permits the same skip again. - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); pickup.ended("ready"); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::ArchitectRun, "a plan is open"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Architect, "a plan is open"); assert_eq!( memory.file(ACTIVITY), "2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Architect run skipped: a plan is open\n2026-10-03 12:00:00 Run #7 started: commands/issue/7-20261003T120000-0400.log, on claude · opus · high\n2026-10-03 12:00:00 Run #7 ended: ready\n2026-10-03 12:00:00 Pickup run skipped: at the Claim limit\n2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Pickup run #7 started: commands/pickup/7-20261003T120000-0400.log, on claude · opus · high\n2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Pickup run #7 ended: ready\n2026-10-03 12:00:00 Pickup run skipped: idle\n" @@ -487,14 +488,14 @@ fn skip_history_uses_the_last_64_kib_and_discards_a_cut_first_line() { .unwrap() .files .insert(PathBuf::from(ACTIVITY), seed.clone()); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); let expected = if prefix.is_empty() { seed } else { format!("{seed}{line}") }; assert_eq!(memory.file(ACTIVITY), expected); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); assert_eq!(memory.file(ACTIVITY), expected); } let memory = Scripted::default(); @@ -505,7 +506,7 @@ fn skip_history_uses_the_last_64_kib_and_discards_a_cut_first_line() { .unwrap() .files .insert(PathBuf::from(ACTIVITY), seed.into()); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); assert_eq!(memory.file(ACTIVITY), format!("{seed}{line}")); } @@ -642,8 +643,8 @@ fn activity_read_and_append_warnings_reveal_quiet_output_and_reset_in_fresh_inst for read_failure in [false, true] { let memory = Scripted::default(); for (begin, pass, kind) in [ - (Begin::PickupRun, Pass::PickupRun, "Pickup run"), - (Begin::ArchitectRun, Pass::ArchitectRun, "Architect run"), + (Begin::PickupRun, Pass::Pickup, "Pickup run"), + (Begin::ArchitectRun, Pass::Architect, "Architect run"), ] { { let mut faults = memory.0.lock().unwrap(); diff --git a/src/main.rs b/src/main.rs index 85308ba9..7b4021e2 100644 --- a/src/main.rs +++ b/src/main.rs @@ -34,6 +34,7 @@ mod ready; mod resend_key; mod run; mod run_ending; +mod security; mod session; mod setup; mod skills; @@ -46,7 +47,7 @@ mod worktree; use std::process::ExitCode; use architect::Outcome; -use args::{ArchitectArgs, Command, PickupArgs, RunArgs}; +use args::{ArchitectArgs, Command, PassArgs, RunArgs}; use asks::Asks; use command::{Ending, failure}; use config::UserConfig; @@ -67,6 +68,8 @@ usage: thirdshift <Issue URL> Run the factory on the iss thirdshift architect base <branch> [<focus>] Do either with <branch> as the Base branch, from a clone on any branch thirdshift pickup Take the lowest-numbered Ready issue in the repository and run it thirdshift pickup base <branch> Do that with <branch> as the Base branch, from a clone on any branch + thirdshift secure Audit the Base branch and record findings privately + thirdshift secure base <branch> Do that with <branch> as the Base branch thirdshift email-test [<address>] Send a test email through Resend, to check the email setup thirdshift setup Choose your defaults, then write the User config with every setting thirdshift update Update thirdshift to the latest release @@ -143,7 +146,8 @@ labels and creating the label if the repository lacks it. A Ticket's Run in a Sp Base fix make none, and an issue already in-progress is left as it is. A Run whose Claim can't be made stops there, naming the cause. The Claim is released, the issue's labels put back as they were, when the Run or the Spec run fails with nothing on origin to take over: -no Issue branch or Spec branch and no pull request. It is removed once a Self-merge has left +no Issue branch or Spec branch and no pull request. A failed Security fix keeps its Claim +even with nothing on origin, for the Day shift. It is removed once a Self-merge has left the issue closed, and otherwise stays: a failure to release or remove it is a warning naming the command to run by hand. @@ -198,7 +202,7 @@ The review starts at <branch>'s head on origin, and the run the plan is dispatch off <branch> and targets it with its pull request. <branch> must exist on origin, with no local copy of it ahead, and launch.pull updates the clone only when <branch> is the branch checked out. base goes with --plan-only too, before or after the focus and the other flags. base is -for architect and pickup only: thirdshift <Issue URL> doesn't take it. Without base, the Base +for a Pass only: thirdshift <Issue URL> doesn't take it. Without base, the Base branch is the branch checked out. A review that finds no Strong candidate publishes no plan. It files its top recommendation as @@ -213,12 +217,12 @@ A review that fails, is interrupted, or ends without naming one of these issues Architect run and leaves any plan it published labelled needs-triage. One that finds no deepening opportunity at all has no issue to name, so it fails the Architect run too. -Only one Architect run or Pickup run per repository runs at a time on a machine. An Architect -run started while another on the same repository, or a Pickup run, is still running, the Spec -run or Run it dispatched included, is skipped: it prints an Architect run or a Pickup run is -already running on <owner>/<repo>, does nothing else and exits 0. Nothing is left to clear -once that other run ends, however it ends. Runs started on an Issue URL are never skipped -this way. +A Pass is a command started with no Issue URL that decides before any work whether it is skipped. +Only one Pass per repository runs at a time on a machine. A Pass started while another Pass +on the same repository is still running, the Spec run or Run it dispatched included, is skipped: +it prints another Pass is already running on <owner>/<repo>, does nothing else and exits 0. +Nothing is left to clear once that other run ends, however it ends. Runs started on an Issue +URL are never skipped this way. An Architect run that finds an open issue labelled architect-plan is skipped too, before any review, with or without --plan-only: the last Architect plan is not finished. It names @@ -244,7 +248,8 @@ pickup starts a Pickup run from the clone, with no Issue URL: one pass, which ta lowest-numbered Ready issue in the repository and dispatches it as thirdshift <Issue URL> would: a Spec run on a Spec, a Run otherwise. A Ready issue is an open issue labelled ready-for-agent that has none of ready-for-human, needs-info, wontfix and needs-triage, is -not in-progress, is not a sub-issue, is not labelled base-fix, has no open blocker, and was +not in-progress, is not a sub-issue, is not labelled base-fix or security-fix, has no open +blocker, and was never started: no Issue branch for it is on origin, and no pull request from one exists, open, merged or closed. A Spec whose Tickets are all closed is not one either: a Spec run would find nothing to do. It must also be settled: by default, thirty minutes have passed since @@ -272,7 +277,7 @@ pickup takes nothing else: no focus and no --plan-only. A Pickup run is skipped, exiting 0 with nothing on stdout and one line on stderr saying why, after any lines on issues it passed over, when the repository has no Ready issue, when the -repository is at its Claim limit, and while an Architect run or another Pickup run on the same +repository is at its Claim limit, and while another Pass on the same repository is still running on this machine. A Pickup run takes nothing while as many open issues are labelled in-progress, whoever @@ -310,6 +315,40 @@ none, even when asked, so a scheduler can start one every few minutes. The notif checks, an address and a Resend API key, are made before any other work on every run, so one that would be skipped fails on them too, with exit 1. +--email, --email <address> and --no-email ask a Security run for its Run notification as +they do a Run, and email.always sets the default. It sends one when its audit ends, whether +it succeeded, failed or was interrupted, listing each recorded finding's severity when +known, title and private link, never its write-up. A skipped Security run sends none. +The address and Resend API key checks come before skip checks or work; a failed send is +only a warning and never changes the run's outcome. + +security-review enables one guidance review after a Run's opening session, before Delivery +pushes and enters the Repair loop. [security] review = true also enables it; +no-security-review overrides the setting. Both words are accepted on commands that start +Runs and follow dispatched Runs and Base fixes. Setup asks; the default is off. +Only reproduced findings are fixed, with their failing tests kept. Unaddressed introduced +findings, a refused review or an incomplete review hold Self-merge, leaving the PR ready +for review and naming the reason. Other Runs record the Security outcome and continue. + +security-fix allows a Security run to fix one reproduced finding: the most severe first, +ties in private-record order, before auditing or after an audit reproduces a finding. +The publishing session follows the reproduction's fix size: one terse Ticket, or a Spec +with Tickets. Every new issue says what the fix changes and links the private record. +On a private repository the finding's own issue is the fix's Ticket or Spec; a single +fix needs no publishing session. thirdshift checks the issues, swaps the top issue's +needs-triage for ready-for-agent, adds security-fix to every fix issue and dispatches its +Run or Spec run. The Security run ends as that run ends, including merge when requested. +Fixing is off by default. [security] fix = true also allows it; +no-security-fix overrides the setting. Both words, with or without dashes, are accepted +on Run, Spec run, Architect run, Pickup run and Security run commands, and passed to their +Runs. Without permission, a reproduced finding waits until its record is closed or +published, or its fix Ticket is closed. A failed fix keeps its Claim even without a push, +and Security runs pause while its issue is open. Pickup never retries a Security fix, +even if its Claim could not be made. Closing the issue lets Security runs go on. +When neither the command nor the User config decided against fixing, a run that left a +reproduced finding unfixed offers security-fix and fix under [security] on stderr and in +its notification. The README's Fencing section shows the scheduled command. + The User config, ~/.thirdshift/config.toml, sets defaults for every Run on this machine; thirdshift setup asks for your defaults and writes one listing every setting, to edit. With merge.always set, every Run is a Merge run unless given --no-merge: @@ -332,21 +371,21 @@ logs.dir sets the root of the logs instead of ~/.thirdshift/logs: an absolute pa Each repository's logs go in <owner>/<repo>/ under it, named for the GitHub repository, in folders thirdshift creates as it needs them. Session logs go in sessions/, as <n>-<stamp>-<kind>.jsonl for a Run and architect-<stamp>-<kind>.jsonl for an Architect run. -Command logs, everything a Run, a Spec run, an Architect run or a Pickup run printed, go in +Command logs, everything a Run, a Spec run or a Pass printed, go in commands/issue/<n>-<stamp>.log, commands/pickup/<n>-<stamp>.log, named for the issue taken, -and commands/architect/<stamp>.log. A Pickup run or Architect run skipped before any work +and commands/architect/<stamp>.log. A Pass skipped before any work keeps no Command log, nor does a Run that fails before any work, as on the Origin match. [logs] dir = \"~/elsewhere/logs\" Each repository's Activity log, activity.log, is a short record of what the factory did there: -a line when a Run, a Spec run, an Architect run or a Pickup run starts work, naming its Command +a line when a Run, a Spec run or a Pass starts work, naming its Command log, and one when it ends, with its outcome, each starting with the local date and time. A -skipped Pickup run or Architect run writes a line only when its reason differs from the last +skipped Pass writes a line only when its reason differs from the last line of its own kind, so a repository that sits idle shows one line, not one per pass. -With activity.quiet_skips set, a skipped Pickup run or Architect run prints nothing on stdout +With activity.quiet_skips set, a skipped Pass prints nothing on stdout or stderr, its starting line included, leaving only its Activity log line; a pass that does work prints as ever. A crontab line can then send its output to one file, which catches only what failed: @@ -396,6 +435,7 @@ fn main() -> ExitCode { } Ok(Command::Architect(architect_args)) => return architect(architect_args), Ok(Command::Pickup(pickup_args)) => return pickup(pickup_args), + Ok(Command::Secure(args)) => return secure(args), Ok(Command::Run(run_args)) => run_args, Err(error) => return argument_error(format_args!("{error:#}")), }; @@ -431,8 +471,8 @@ fn main() -> ExitCode { /// it ended on on stdout: the plan, the idea issue the review filed, or the /// issue that already covers its top recommendation. One whose review or /// plan fails puts the cause and the session log on stderr. One that is -/// skipped says why on stderr, and is no failure: as another on its -/// repository, or a Pickup run, is still running, it puts nothing on stdout, +/// skipped says why on stderr, and is no failure: as another Pass on its +/// repository is still running, it puts nothing on stdout, /// and as Architect plans are still open there, or Architect ideas wait for /// triage there, the URL of each, or as it has a Ready issue, that issue's /// URL. If asked, by the command or the User config, it sends one Run @@ -481,7 +521,7 @@ fn architect(args: ArchitectArgs) -> ExitCode { /// The repository's Activity log records the skip, if it differs from the /// last, or the start and end of its work. With `activity.quiet_skips` set, a /// skipped pass prints nothing at all. -fn pickup(args: PickupArgs) -> ExitCode { +fn pickup(args: PassArgs) -> ExitCode { let ask = |config: &UserConfig| args.flags.notification(config); let (config, mut started) = match command::start(Begin::PickupRun, ask, About::PickupRun) { Ok(started) => started, @@ -504,6 +544,29 @@ fn pickup(args: PickupArgs) -> ExitCode { } } +/// A Security run, with one notification when asked, unless skipped. +fn secure(args: PassArgs) -> ExitCode { + let (config, mut started) = match command::start( + Begin::SecurityRun, + |config| args.flags.notification(config), + About::SecurityRun, + ) { + Ok(started) => started, + Err(failure) => return failure, + }; + let mut harness = args.flags.security_harness(&config); + let outcome = security::run(args.base.as_deref(), &args.flags, &config, &mut harness); + started.built_with(&harness); + started.finish(match outcome { + security::Outcome::Skipped(skipped) => Ending::Skipped(command::Skip { + reason: skipped.to_string(), + urls: Vec::new(), + }), + security::Outcome::Audited(audited) => Ending::Security(audited), + security::Outcome::Fixed { ended, findings } => Ending::SecurityFix { ended, findings }, + }) +} + /// The end of a command other than a Run: the line that says how it went, /// or its error, on stderr. fn outcome(result: anyhow::Result<impl std::fmt::Display>) -> ExitCode { diff --git a/src/notification.rs b/src/notification.rs index 8f2e4838..385979ce 100644 --- a/src/notification.rs +++ b/src/notification.rs @@ -1,5 +1,5 @@ -//! The Run notification: one email when a Run, a Spec run or an Architect run -//! ends, whatever its outcome, or when a Pickup run that took an issue does, +//! The Run notification: one email when a Run, a Spec run or a Pass ends, +//! whatever its outcome, unless the Pass was skipped, //! through the same checks and the same send as `email-test`. use std::path::Path; @@ -18,7 +18,7 @@ use crate::logs; use crate::progress; use crate::run_ending::Account; -/// What a Run, an Architect run or a Pickup run asks about its Run +/// What a Run or a Pass asks about its Run /// notification, by its command or, without `email` or `no-email`, by the /// User config. #[derive(Debug, Clone, PartialEq, Eq)] @@ -55,6 +55,8 @@ pub enum About<'a> { ArchitectRun, /// A Pickup run, which is about the issue it takes, once it takes one. PickupRun, + /// A Security run, from the Launch directory. + SecurityRun, } /// What a Run notification's subject names. @@ -69,7 +71,9 @@ enum Subject { /// An Architect run, with the repository the Launch directory's `origin` /// names, if it names one. ArchitectRun(Option<Repo>), - /// A Pickup run that has taken no issue yet, and so has nothing to tell. + /// A Security run, with its repository if origin names one. + SecurityRun(Option<Repo>), + /// A Pass whose notification subject has not been established yet. Pending, } @@ -102,6 +106,7 @@ impl RunNotification { // run's own preflight reports why. About::ArchitectRun => Subject::ArchitectRun(launch::repo().ok()), About::PickupRun => Subject::Pending, + About::SecurityRun => Subject::SecurityRun(launch::repo().ok()), }; Ok(RunNotification { checked, @@ -159,6 +164,13 @@ fn subject_line(subject: &Subject, outcome: &str) -> Option<String> { match subject { Subject::Issue { issue, title } => Some(issue_subject(issue, title.as_deref(), outcome)), Subject::ArchitectRun(repo) => Some(architect_subject(repo.as_ref(), outcome)), + Subject::SecurityRun(repo) => { + let repo = repo + .as_ref() + .map(|repo| format!(" {}", repo.slug())) + .unwrap_or_default(); + Some(format!("[thirdshift]{repo} Security run: {outcome}")) + } Subject::Pending => None, } } @@ -198,6 +210,25 @@ fn body( took: Duration, ) -> String { let mut text = format!("Result: {}\n", account.outcome); + if let Some(findings) = account.security_findings { + let audit = match &account.ended { + Ok(line) => line.as_str(), + Err(_) => account.outcome, + }; + text += &format!("Audit: {audit}\n"); + if !findings.is_empty() { + text += "\nSecurity findings:\n"; + for finding in findings { + let severity = finding + .severity + .as_ref() + .map(|severity| format!("{severity}: ")) + .unwrap_or_default(); + text += &format!("- {severity}{}: {}\n", finding.title, finding.url); + } + text += "\n"; + } + } if let Some(review) = &account.review { text += &format!("Review: {}\n", review.line); if review.dispatched.is_some() { @@ -207,10 +238,19 @@ fn body( if let Some(pr_url) = account.pr_url { text += &format!("Pull request: {pr_url}\n"); } + // Security failure causes can quote private verifier work or Harness + // diagnostics. Only the fixed safeguard category is safe to send. if let Err(cause) = &account.ended && !account.interrupted { - text += &format!("Cause: {}\n", cause.full()); + let cause = if account.security_findings.is_some() { + cause.safeguard_refusal() + } else { + Some(cause.full()) + }; + if let Some(cause) = cause { + text += &format!("Cause: {cause}\n"); + } } for line in account.advice { text += &format!("{:<14}{}\n", format!("{}:", line.label), line.value); @@ -290,6 +330,7 @@ mod tests { log: Some(Path::new(LOG)), ticket_lines: &[], review: None, + security_findings: None, urls: vec![PR], } } @@ -307,6 +348,7 @@ mod tests { log: None, ticket_lines: &[], review: None, + security_findings: None, urls: Vec::new(), } } diff --git a/src/pass.rs b/src/pass.rs index 71a7af6d..aeba7bac 100644 --- a/src/pass.rs +++ b/src/pass.rs @@ -1,4 +1,4 @@ -//! The Pass seam: what an Architect run's or a Pickup run's gates, which +//! The Pass seam: what a Pass's gates, which //! decide before any work whether it is skipped, read and change of GitHub, //! what an Architect run's conclusion after its Architecture review does, //! viewing and labelling the issue the review ended on, and what a pass @@ -14,7 +14,10 @@ use std::fmt::Display; use std::path::PathBuf; +use crate::github::{DraftAdvisory, SecurityRecord, SecurityRecords}; +use crate::security::reproduction::Reproduction; use anyhow::Result; +use serde_json::Value; use crate::asks::{Asks, Flags}; use crate::config::UserConfig; @@ -28,7 +31,7 @@ use crate::logs::{self, Pass, Work}; use crate::progress; use crate::ready::{self, ReadyIssue}; use crate::run::{self, Ended, StartedBy}; -use crate::session::{Logs, Sessions}; +use crate::session::{Logs, Purpose, Sessions}; use crate::worktree::ReviewWorktree; /// The Architecture review session's kind, in its progress lines and log @@ -39,6 +42,11 @@ const REVIEW: &str = "architecture-review"; /// the pass's Base branch `base`, whatever the Launch directory has checked /// out. pub enum Dispatch<'a> { + SecurityFix { + issue: &'a IssueUrl, + is_spec: bool, + base: &'a str, + }, /// An Architect run's Architect plan. ArchitectPlan { plan: &'a IssueUrl, base: &'a str }, /// The Ready issue a Pickup run took, a Spec or not, as `is_spec` says. @@ -94,6 +102,46 @@ pub trait Outside { prompt: &str, conclude: impl FnOnce(&mut Self, Option<&str>) -> Result<T>, ) -> (Result<T>, Option<PathBuf>); + /// Check Node.js before a Security audit needs its validators. + fn check_node(&mut self) -> Result<()>; + /// Audit the Base branch in a disposable checkout and validate its artifacts. + fn audit(&mut self, base: &str) -> (Result<crate::security::audit::Audited>, Option<PathBuf>); + /// Private advisory or issue records in every state. + fn security_records(&mut self) -> Result<SecurityRecords>; + /// Whether origin's prepared Base branch is the last completed audit's commit. + fn base_unchanged_since_security_audit(&mut self) -> Result<bool>; + /// Record one finding in the repository's private storage. + fn create_security_record( + &mut self, + records: &SecurityRecords, + finding: &DraftAdvisory, + ) -> Result<Value>; + /// Reproduce the finding read from this record, in a fresh checkout. + fn reproduce( + &mut self, + record: &SecurityRecord, + number: usize, + ) -> (Result<Reproduction>, Option<PathBuf>); + /// Write only a completed reproduction to its private record. + fn update_security_record( + &mut self, + record: &SecurityRecord, + reproduction: &Reproduction, + ) -> Result<()>; + fn publish_security_fix( + &mut self, + base: &str, + record: &SecurityRecord, + url: &str, + ) -> (Result<IssueUrl>, Option<PathBuf>); + fn link_security_fix(&mut self, record: &SecurityRecord, issue: &IssueUrl) -> Result<()>; + /// Record the dispatch's ending in its private record, even after interruption. + fn record_security_fix_ending( + &mut self, + record: &SecurityRecord, + issue: &IssueUrl, + succeeded: bool, + ) -> Result<()>; /// Run `dispatch` to its end, on the Harness the pass checked. fn dispatch(&mut self, dispatch: Dispatch) -> Ended; } @@ -175,7 +223,7 @@ impl Outside for LaunchAndGitHub<'_> { let harness = self.harness.clone(); let concluded = Sessions::within(&logs, worktree.path(), &harness, |sessions| { progress::step(starting); - let final_message = sessions.run_to_final_message(REVIEW, prompt)?; + let final_message = sessions.run_to_final_message(Purpose::Ordinary, REVIEW, prompt)?; conclude(self, final_message.as_deref()) }); // Removed once the review is concluded. @@ -183,16 +231,65 @@ impl Outside for LaunchAndGitHub<'_> { concluded } + fn check_node(&mut self) -> Result<()> { + crate::security::audit::check_node() + } + + fn audit(&mut self, base: &str) -> (Result<crate::security::audit::Audited>, Option<PathBuf>) { + crate::security::audit::run(self.launch.git(), self.repo, base, self.harness) + } + + fn security_records(&mut self) -> Result<SecurityRecords> { + GitHub::new().security_records(&self.repo.slug()) + } + + fn base_unchanged_since_security_audit(&mut self) -> Result<bool> { + Ok( + crate::security::audit::last_commit(self.repo, self.base.name())?.as_deref() + == Some(self.base.origin_commit()), + ) + } + + fn create_security_record( + &mut self, + records: &SecurityRecords, + finding: &DraftAdvisory, + ) -> Result<Value> { + GitHub::new().create_security_record(&self.repo.slug(), records, finding) + } + + fn reproduce( + &mut self, + record: &SecurityRecord, + number: usize, + ) -> (Result<Reproduction>, Option<PathBuf>) { + crate::security::reproduction::run( + self.launch.git(), + self.repo, + record, + number, + self.harness, + ) + } + + fn update_security_record( + &mut self, + record: &SecurityRecord, + reproduction: &Reproduction, + ) -> Result<()> { + GitHub::new().update_security_record(&self.repo.slug(), record, reproduction) + } + /// The run's asks are the Architect plan's or the Ready issue's, from /// the command's flags and the User config, on the checked Harness. fn dispatch(&mut self, dispatch: Dispatch) -> Ended { let (issue, asks, base) = match dispatch { - Dispatch::ArchitectPlan { plan, base } => ( - plan, - Asks::of_architect_plan(plan, self.flags, self.config), + Dispatch::SecurityFix { + issue, + is_spec, base, - ), - Dispatch::ReadyIssue { + } + | Dispatch::ReadyIssue { issue, is_spec, base, @@ -201,6 +298,11 @@ impl Outside for LaunchAndGitHub<'_> { Asks::of_ready_issue(issue, is_spec, self.flags, self.config), base, ), + Dispatch::ArchitectPlan { plan, base } => ( + plan, + Asks::of_architect_plan(plan, self.flags, self.config), + base, + ), }; let mut asks = Asks { harness: self.harness.clone(), @@ -208,6 +310,40 @@ impl Outside for LaunchAndGitHub<'_> { }; run::run_to_end(issue, &mut asks, StartedBy::Dispatch { base }) } + + fn publish_security_fix( + &mut self, + base: &str, + record: &SecurityRecord, + url: &str, + ) -> (Result<IssueUrl>, Option<PathBuf>) { + crate::security::fixing::publish( + self.launch.git(), + self.repo, + base, + record, + url, + self.harness, + ) + } + + fn link_security_fix(&mut self, record: &SecurityRecord, issue: &IssueUrl) -> Result<()> { + GitHub::new().link_security_fix(&self.repo.slug(), record, issue) + } + + fn record_security_fix_ending( + &mut self, + record: &SecurityRecord, + issue: &IssueUrl, + succeeded: bool, + ) -> Result<()> { + GitHub::new().completion().record_security_fix_ending( + &self.repo.slug(), + record, + issue, + succeeded, + ) + } } #[cfg(test)] @@ -221,12 +357,15 @@ mod in_memory { use anyhow::{Result, anyhow, bail}; use super::{Dispatch, Outside}; + use crate::github::{DraftAdvisory, SecurityRecord, SecurityRecords}; use crate::github::{Issue, ListedIssue}; use crate::issue::{IssueUrl, Repo}; use crate::labels::{Edit, Label, Labels}; use crate::logs::{Pass, Work}; use crate::ready::ReadyIssue; use crate::run::{Ended, Goal, Reached}; + use crate::security::reproduction::{Outcome, Reproduction}; + use serde_json::{Value, json}; /// A call a pass made outside itself, in the order it made it. #[derive(Debug, PartialEq, Eq)] @@ -253,6 +392,31 @@ mod in_memory { Skipped(String), /// It checked its Harness, Model and Effort. HarnessCheck, + /// It checked Node.js for the Security audit validators. + NodeCheck, + /// It ran and validated the Security audit of this Base branch. + SecurityAudit(String), + /// It listed private advisories in every state. + AdvisoryList, + /// It compared the Base branch with the completed audit history. + AuditHistory, + /// It recorded this fingerprint privately. + CreateAdvisory(String), + /// It tried to reproduce this private record, in sequence. + Reproduce(String), + /// It wrote a completed reproduction to this private record. + UpdateSecurityRecord(String), + PublishSecurityFix(String), + LinkSecurityFix(u64), + SecurityFixEnding { + issue: u64, + succeeded: bool, + }, + DispatchSecurityFix { + issue: u64, + is_spec: bool, + base: String, + }, /// It recorded that it started work: on this issue, for a Pickup /// run, or on its repository, for an Architect run. Started(Option<u64>), @@ -260,7 +424,10 @@ mod in_memory { Pull, /// It ran the Architecture review session of this Base branch on /// this prompt, handing on its starting line next. - Review { base: String, prompt: String }, + Review { + base: String, + prompt: String, + }, /// It dispatched this Ready issue, a Spec or not, on this Base /// branch. DispatchReadyIssue { @@ -269,7 +436,10 @@ mod in_memory { base: String, }, /// It dispatched this Architect plan on this Base branch. - DispatchPlan { plan: u64, base: String }, + DispatchPlan { + plan: u64, + base: String, + }, } /// A repository in memory: its issues, filed by label, open or @@ -293,6 +463,12 @@ mod in_memory { ready: Option<(ListedIssue, bool)>, /// Whether the Harness check fails. harness_failing: bool, + node_failing: bool, + audit_findings: Vec<DraftAdvisory>, + audit_error: Option<String>, + advisories: Vec<Value>, + finding_issues: Option<Vec<Value>>, + unchanged_base: bool, /// The Architecture review session's final message, if it has one, /// or the cause it fails with. review: Result<Option<String>, String>, @@ -317,6 +493,12 @@ mod in_memory { failing_edits: Vec::new(), ready: None, harness_failing: false, + node_failing: false, + audit_findings: Vec::new(), + audit_error: None, + advisories: Vec::new(), + finding_issues: None, + unchanged_base: false, review: Ok(None), session_log: None, ending: None, @@ -391,6 +573,36 @@ mod in_memory { self } + pub fn node_failing(mut self) -> Self { + self.node_failing = true; + self + } + + pub fn audited(mut self, findings: Vec<DraftAdvisory>) -> Self { + self.audit_findings = findings; + self + } + + pub fn audit_failing(mut self, cause: &str) -> Self { + self.audit_error = Some(cause.to_string()); + self + } + + pub fn advisories(mut self, advisories: Vec<Value>) -> Self { + self.advisories = advisories; + self + } + + pub fn finding_issues(mut self, issues: Vec<Value>) -> Self { + self.finding_issues = Some(issues); + self + } + + pub fn unchanged_base(mut self) -> Self { + self.unchanged_base = true; + self + } + /// Have the Architecture review session end with `final_message`. pub fn reviewed(mut self, final_message: &str) -> Self { self.review = Ok(Some(final_message.to_string())); @@ -502,7 +714,7 @@ mod in_memory { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => { Some(issue.number) } - Work::ArchitectRun(_) => None, + Work::ArchitectRun(_) | Work::SecurityRun(_) => None, }; self.calls.push(Call::Started(issue)); } @@ -530,8 +742,95 @@ mod in_memory { } } + fn check_node(&mut self) -> Result<()> { + self.calls.push(Call::NodeCheck); + if self.node_failing { + bail!("Node.js is required"); + } + Ok(()) + } + + fn audit( + &mut self, + base: &str, + ) -> (Result<crate::security::audit::Audited>, Option<PathBuf>) { + self.calls.push(Call::SecurityAudit(base.to_string())); + let audited = match self.audit_error.take() { + Some(cause) => Err(anyhow!(cause)), + None => Ok(crate::security::audit::Audited { + findings: std::mem::take(&mut self.audit_findings), + }), + }; + (audited, self.session_log.clone()) + } + + fn security_records(&mut self) -> Result<SecurityRecords> { + self.calls.push(Call::AdvisoryList); + Ok(match &self.finding_issues { + Some(issues) => SecurityRecords::Issues(issues.clone()), + None => SecurityRecords::Advisories(self.advisories.clone()), + }) + } + + fn base_unchanged_since_security_audit(&mut self) -> Result<bool> { + self.calls.push(Call::AuditHistory); + Ok(self.unchanged_base) + } + + fn create_security_record( + &mut self, + _records: &SecurityRecords, + finding: &DraftAdvisory, + ) -> Result<Value> { + self.calls + .push(Call::CreateAdvisory(finding.fingerprint.clone())); + let advisory = json!({ + "ghsa_id": finding.fingerprint, + "description": finding.description, "state": "draft", + "summary": finding.summary, + "html_url": format!("https://github.com/acme/widgets/security/advisories/{}", finding.fingerprint), + "severity": null + }); + self.advisories.push(advisory.clone()); + Ok(advisory) + } + + fn reproduce( + &mut self, + record: &SecurityRecord, + _number: usize, + ) -> (Result<Reproduction>, Option<PathBuf>) { + self.calls.push(Call::Reproduce(record.name())); + ( + Ok(Reproduction { + outcome: Outcome::NotReproduced, + notes: "Not reproduced with a local fixture".into(), + test: "bounded_fixture()".into(), + }), + self.session_log.clone(), + ) + } + + fn update_security_record( + &mut self, + record: &SecurityRecord, + _reproduction: &Reproduction, + ) -> Result<()> { + self.calls.push(Call::UpdateSecurityRecord(record.name())); + Ok(()) + } + fn dispatch(&mut self, dispatch: Dispatch) -> Ended { self.calls.push(match dispatch { + Dispatch::SecurityFix { + issue, + is_spec, + base, + } => Call::DispatchSecurityFix { + issue: issue.number, + is_spec, + base: base.into(), + }, Dispatch::ArchitectPlan { plan, base } => Call::DispatchPlan { plan: plan.number, base: base.to_string(), @@ -550,6 +849,37 @@ mod in_memory { .take() .expect("a dispatch, with no ending scripted for it") } + + fn publish_security_fix( + &mut self, + _base: &str, + record: &SecurityRecord, + _url: &str, + ) -> (Result<IssueUrl>, Option<PathBuf>) { + self.calls.push(Call::PublishSecurityFix(record.name())); + ( + Ok(IssueUrl::parse("https://github.com/acme/widgets/issues/8").unwrap()), + self.session_log.clone(), + ) + } + + fn link_security_fix(&mut self, _record: &SecurityRecord, issue: &IssueUrl) -> Result<()> { + self.calls.push(Call::LinkSecurityFix(issue.number)); + Ok(()) + } + + fn record_security_fix_ending( + &mut self, + _record: &SecurityRecord, + issue: &IssueUrl, + succeeded: bool, + ) -> Result<()> { + self.calls.push(Call::SecurityFixEnding { + issue: issue.number, + succeeded, + }); + Ok(()) + } } /// Issue `number` in `acme/widgets` as a listing gives it, labelled diff --git a/src/pickup.rs b/src/pickup.rs index 7a8e997c..1e244aac 100644 --- a/src/pickup.rs +++ b/src/pickup.rs @@ -1,9 +1,9 @@ //! A Pickup run: from the Launch directory with no Issue URL, the search //! for the lowest-numbered Ready issue in the repository, saying why it //! passed over each issue labelled `ready-for-agent` before it, then that -//! issue dispatched as a Spec run or a Run. Only one Pickup run or Architect -//! run per repository runs at a time on a machine: one started while another -//! is still running is skipped, before any search. One that runs first makes +//! issue dispatched as a Spec run or a Run. Only one Pass per repository +//! runs at a time on a machine: one started while another Pass is still +//! running is skipped, before any search. One that runs first makes //! the Sweep, and is then skipped if it finds the repository at its Claim //! limit, or with no Ready issue. @@ -45,7 +45,7 @@ pub struct Took { /// Why a Pickup run was skipped. Its `Display` is the reason, as the skipped /// run's one line gives it. pub enum Skipped { - /// An Architect run or another Pickup run on this repository is still + /// Another Pass on this repository is still /// running on this machine, the Spec run or Run it dispatched included. AlreadyRunning(AlreadyRunning), /// The repository is at its Claim limit: `claimed` open issues carry a @@ -90,7 +90,7 @@ impl fmt::Display for Skipped { /// Claim. /// /// Once the preflight checks pass, and before any search, the Pickup run is -/// skipped if an Architect run or another Pickup run on the same repository +/// skipped if another Pass on the same repository /// is still running on this machine. Otherwise this process is that /// repository's one such run until it exits, through whatever it dispatches. /// A skip is recorded in the repository's Activity log. The rest, from the @@ -111,7 +111,7 @@ pub fn run( } = match launch::start(base)? { Start::Clear(launch) => launch, Start::AlreadyRunning(running) => { - logs::skipped(Pass::PickupRun, &running.0, &running); + logs::skipped(Pass::Pickup, &running.0, &running); return Ok(Outcome::Skipped(Skipped::AlreadyRunning(running))); } }; @@ -142,7 +142,7 @@ fn run_through( let ReadyIssue { listed, is_spec } = match gates(outside, repo, limit)? { Decision::Take(ready) => ready, Decision::Skip(skipped) => { - outside.skipped(Pass::PickupRun, &skipped); + outside.skipped(Pass::Pickup, &skipped); return Ok(Outcome::Skipped(skipped)); } }; diff --git a/src/prompt.rs b/src/prompt.rs index b5f8e71f..be6e566e 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -145,7 +145,7 @@ pub fn spec_review(spec: &IssueUrl, base: &str, branch: &str, pr_url: &str) -> S \n\ {address_findings}\n\ \n\ - Push branch {branch}. Do not rebase or force-push.\n\ + Do not push: thirdshift pushes branch {branch} after the optional Security review. Do not rebase or force-push.\n\ \n\ Update PR {pr_url} using the `thirdshift-pr` skill, rewriting its body to cover the whole Spec. Leave out its Tickets checklist, or keep it between its markers as it is: thirdshift puts it back. Leave the PR a draft: thirdshift marks it ready once you are done.\n\ \n\ @@ -292,3 +292,82 @@ pub fn resume(killed: &[&str]) -> String { killed = killed.join("; "), ) } + +/// The Security audit's terminal protocol: only these final lines are read. +pub const AUDIT_COMPLETE_LINE: &str = "Security audit: complete"; +pub const AUDIT_INCOMPLETE_LINE: &str = "Security audit: incomplete"; + +pub fn security_audit( + base: &str, + commit: &str, + root: &std::path::Path, + output: &std::path::Path, + threat_model: Option<&str>, +) -> String { + let threat_model = threat_model + .map(|file| format!("Read the repository's threat-model document `{file}`.\n")) + .unwrap_or_default(); + format!( + "/thirdshift-security-audit\nUse the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile.\nAudit the whole repository at commit `{commit}`, the head of Base branch `{base}` on origin. All vendored and third-party code are out of scope.\n{threat_model}Output directory: `{output}`.\nAudit root: `{root}`.\nRead compatible earlier runs under the audit root before planning this audit.\nThis session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing.\nIf a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking.\nWrite the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing.\nEnd your final message with exactly `{AUDIT_COMPLETE_LINE}` after writing valid artifacts, or `{AUDIT_INCOMPLETE_LINE}` when incomplete.\n\n{HEADLESS}", + output = output.display(), + root = root.display() + ) +} + +/// One guidance session on the Run's change, before Delivery pushes and +/// enters the Repair loop. The separate Security audit remains a full audit. +pub const SECURITY_REVIEW_REPORT_FILE: &str = "<private report file>"; +pub const SECURITY_REVIEW_MERGE_BASE: &str = "<merge base commit>"; + +pub fn security_review(issue: &IssueUrl, base: &str, branch: &str) -> String { + format!( + "Use the `thirdshift-security-audit` skill in guidance mode.\n\ + Review the change for {url} on branch {branch} against Base branch {base} with `git diff {SECURITY_REVIEW_MERGE_BASE}...HEAD`, including supporting code.\n\ + Read the relevant security attack-class guidance and the repository's SECURITY.md or threat model when present.\n\ + Use one session. Do not delegate auditors or run the full six-phase audit, validators, coverage ledger or audit artifacts.\n\ + Fix only a Security finding you can show with a failing proof-of-concept test; run it before fixing, keep it as a regression test, and rerun it afterward. Use harmless local payloads; touch no deployed site or real third-party service. Commit each fix.\n\ + Merge base commit: `{SECURITY_REVIEW_MERGE_BASE}`.\n\ + Classify introduced findings against this pinned merge base: run each proof-of-concept on HEAD and the untouched merge base; a test that also fails there is pre-existing. Never include pre-existing vulnerability details, titles, fingerprints, tests or private-record links in the pull request or final message, and do not fix those here.\n\ + Private report file: `{SECURITY_REVIEW_REPORT_FILE}`.\n\ + Write a JSON array to that file, even when empty ([]). Each entry is an old finding with exactly these fields: fingerprint (stable across reviews and audits, no newlines or backticks), title (one line), description (the private write-up and evidence), proof_of_concept (test: full test text, command: exact command run at both commits, head_exit_code: positive failing exit code, merge_base_exit_code: positive failing exit code, notes: reproduction evidence and scoring rationale, severity: critical/high/medium/low/informational scored with the skill's likelihood-and-impact rubric, fix_size: single/spec judged from whether one session can hold the fix). All text fields are nonempty. Do not commit or copy this report into the worktree. thirdshift records it privately with the Security run's fingerprint matching and reproduced-outcome format so a later Security run can fix it when allowed; existing records and the Day shift's grades are preserved. Fixing old findings belongs to a separate Security run.\n\ + Update this branch's pull request: list each unaddressed introduced finding under Security in its Unaddressed findings, with evidence and why it is unaddressed. Preserve Standards and Spec entries and the rest of the body. Do not merge. Delivery pushes any new commits and marks the PR ready after this session.\n\ + If refused, incomplete, or missing a prerequisite, say so and do not claim completion.\n\ + After a complete review, end your final message with exactly one line of the form:\n\ + Security review: {{\"unaddressed_count\":0,\"findings\":[],\"pre_existing_count\":0}}\n\ + unaddressed_count counts only unaddressed findings introduced by this change; findings is an array of their short titles, with one title per finding. pre_existing_count is the number of entries in the private report. No old-finding details belong in this line.\n\n\ + {HEADLESS}", + url = issue.url, + ) +} + +pub fn security_reproduction(commit: &str, finding: &str, test: &std::path::Path) -> String { + format!( + "Reproduce this recorded Security finding at audited commit `{commit}`.\n\ + Read the `thirdshift-security-audit` skill's likelihood-and-impact severity rubric.\n\ + Write a proof-of-concept test from the finding's validation plan, then run it against the untouched code in this throwaway worktree. Use harmless payloads only, never a deployed site or a real third-party service. Do not fix or change repository source.\n\ + If the test reproduces the finding, score its severity with the skill's rubric and judge whether one session can hold the fix (single) or it needs a Spec (spec).\n\ + Test file: `{test}`.\n\ + Write the test's full text to this file, even when not reproduced. In your final message, give reproduction notes: the exact command, its result, and, when reproduced, likelihood, impact and the fix-size reasoning.\n\ + End your final message with exactly `Security reproduction: reproduced <severity> <size>`, where severity is critical, high, medium, low or informational and size is single or spec, or `Security reproduction: not reproduced`.\n\ + This session commits, pushes, opens and publishes nothing.\n\n\ + Recorded finding:\n{finding}\n\n{HEADLESS}", + test = test.display(), + ) +} + +pub const SECURITY_FIX_LINE: &str = "Security fix Ticket: "; +pub const SECURITY_FIX_SPEC_LINE: &str = "Security fix Spec: "; + +pub fn security_fix(base: &str, url: &str, finding: &str) -> String { + format!( + "Publish the fix for this reproduced Security finding on Base branch `{base}`.\n\ + Read the private record at {url} and the record below.\n\ + Follow the completed reproduction's fix size: single publishes one Ticket; spec publishes a Spec with Tickets using `thirdshift-to-spec` and `thirdshift-to-tickets`.\n\ + On a public repository, publish one new top issue labelled `needs-triage`. Create that label if missing. On a private repository, reuse the finding's issue as the top issue and preserve its body and evidence; add the bigger fix's Tickets as its native sub-issues.\n\ + Every new issue is terse: it says only what the fix changes and links the private record. It carries none of the write-up, trace, evidence, reproduction notes, proof-of-concept test or exploit details, even paraphrased. Keep those in the private record. This overrides the skills' templates.\n\ + A Spec's Tickets must be new, open, labelled `ready-for-agent`, linked as native sub-issues with their native blocking links, and have no sub-issues of their own. Read the links back before finishing.\n\ + This session changes no repository source, commits and pushes nothing, opens no pull request, and does not implement the fix. thirdshift checks every issue, marks the top issue ready, labels all fix issues security-fix and dispatches its Run or Spec run.\n\ + End your final message with exactly `{SECURITY_FIX_LINE}<Issue URL>` for single or `{SECURITY_FIX_SPEC_LINE}<Issue URL>` for spec, naming the top issue.\n\n\ + Private record:\n{finding}\n\n{HEADLESS}" + ) +} diff --git a/src/prompts_page.rs b/src/prompts_page.rs index 684d5b26..37b29adf 100644 --- a/src/prompts_page.rs +++ b/src/prompts_page.rs @@ -19,7 +19,7 @@ use crate::prompt; use crate::session::{claude_args, codex_args}; use crate::skills::SKILLS; -use Sender::{ArchitectRun, Units}; +use Sender::{ArchitectRun, SecurityRun, Units}; /// A press unit on the home page, which each prompt and skill links back to. struct Unit { @@ -60,7 +60,7 @@ const SESSION_ID: &str = "<session id>"; const PROMPT: &str = "<prompt>"; const MODEL: &str = "<model>"; const EFFORT: &str = "<effort>"; -const PLACEHOLDERS: [&str; 18] = [ +const PLACEHOLDERS: [&str; 20] = [ ISSUE_URL, SPEC_URL, NUMBER, @@ -79,6 +79,8 @@ const PLACEHOLDERS: [&str; 18] = [ MODEL, EFFORT, prompt::REVIEW_REPORTS_DIRECTORY, + prompt::SECURITY_REVIEW_REPORT_FILE, + prompt::SECURITY_REVIEW_MERGE_BASE, ]; /// Who sends a prompt. @@ -87,6 +89,8 @@ enum Sender { Units(&'static [Unit]), /// An Architect run, which the press units, a Run's, don't cover. ArchitectRun, + /// A Security run, before any dispatched Run. + SecurityRun, } /// A prompt as the page shows it: when it is sent, by whom, and its text @@ -162,6 +166,37 @@ fn prompts() -> Vec<Prompt> { sender: ArchitectRun, text: prompt::architecture_review(BASE, Some(FOCUS)), }, + Prompt { + id: "prompt-security-review", + title: "Security review", + when: "When enabled, runs one guidance review after a Run's opening session or a Spec run's Spec review, before Delivery pushes and enters the Repair loop. Ticket Runs get none; Base fixes get one too. Introduced findings, refusals or incomplete reviews hold Self-merge.", + sender: Units(&[REVIEW]), + text: prompt::security_review(&issue, BASE, BRANCH), + }, + Prompt { + id: "prompt-security-audit", + title: "Security audit", + when: "Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root.", + sender: SecurityRun, + text: prompt::security_audit( + BASE, + "<audited commit>", + std::path::Path::new("<audit root>"), + std::path::Path::new("<output directory>"), + Some("SECURITY.md"), + ), + }, + Prompt { + id: "prompt-security-reproduction", + title: "Security reproduction", + when: "After a Security audit, tries to reproduce one recorded finding in a fresh throwaway worktree at its audited commit. The test is copied into the private record only after a complete outcome.", + sender: SecurityRun, + text: prompt::security_reproduction( + "<audited commit>", + "<recorded Security finding>", + std::path::Path::new("<test file>"), + ), + }, Prompt { id: "prompt-conflict-repair", title: "Conflict Repair", @@ -169,6 +204,13 @@ fn prompts() -> Vec<Prompt> { sender: Units(&[FINISH]), text: prompt::conflict_repair(&issue, BASE, BRANCH, PR_URL), }, + Prompt { + id: "prompt-security-fix", + title: "Security fix publishing", + when: "With fixing allowed, publishes a terse Ticket or Spec with Tickets for the most severe reproduced finding, reusing a private finding's issue. thirdshift checks and marks it ready before dispatching its Run or Spec run.", + sender: SecurityRun, + text: prompt::security_fix(BASE, "<private record URL>", "<recorded Security finding>"), + }, Prompt { id: "prompt-foreign-conflict-repair", title: "Conflict Repair, on Foreign commits", @@ -334,12 +376,15 @@ fn skill_used_by(skill: &str) -> String { "thirdshift-resolving-merge-conflicts" => { format!("{}, in a Repair", unit_links(&[FINISH])) } - "thirdshift-improve-codebase-architecture" - | "thirdshift-to-spec" - | "thirdshift-to-tickets" - | "thirdshift-codebase-design" => { + "thirdshift-improve-codebase-architecture" | "thirdshift-codebase-design" => { "the Architecture review, in an Architect run".to_string() } + "thirdshift-to-spec" | "thirdshift-to-tickets" => { + "the Architecture review, in an Architect run, or Security fix publishing, in a Security run".to_string() + } + "thirdshift-security-audit" => { + "the Security audit, in a Security run, or a Security review".to_string() + } _ => panic!("the Factory skill {skill} has no user: add it to skill_used_by"), } } @@ -481,7 +526,7 @@ fn licence_section(html: &mut String) { r##"<section class="sec" aria-labelledby="licence-title"> <div class="wrap"> <div class="sec-head"><p class="eyebrow">Licence and credits</p><h2 id="licence-title">Whose skills these are</h2></div> - <p class="lede">The Factory skills are adapted from Matt Pocock's skills, under this licence. The <a href="#skill-thirdshift-pr">thirdshift-pr</a> skill credits Dex Horthy's <code>show-me</code> skill in its <a href="#file-thirdshift-pr-credits-md">CREDITS.md</a>.</p> + <p class="lede">Most Factory skills are adapted from Matt Pocock's skills, under the licence below. The <a href="#skill-thirdshift-pr">thirdshift-pr</a> skill credits Dex Horthy's <code>show-me</code> skill in its <a href="#file-thirdshift-pr-credits-md">CREDITS.md</a>. The <a href="#skill-thirdshift-security-audit">thirdshift-security-audit</a> skill is copied from <a href="https://github.com/cloudflare/security-audit-skill">Cloudflare's security-audit-skill</a>, under its own <a href="#file-thirdshift-security-audit-license">MIT licence</a>, with its upstream repository and commit recorded in <a href="#file-thirdshift-security-audit-credits-md">CREDITS.md</a>.</p> "##, ); let mut files: Vec<&File> = SKILLS.files().collect(); @@ -529,6 +574,7 @@ fn sent_by(sender: &Sender) -> String { match sender { Units(units) => unit_links(units), ArchitectRun => "an Architect run, before any unit".to_string(), + SecurityRun => "a Security run, before any unit".to_string(), } } @@ -620,6 +666,38 @@ mod tests { ); } + #[test] + fn publishing_skills_show_security_fix_publishing_as_a_user() { + let html = render(); + for name in ["thirdshift-to-spec", "thirdshift-to-tickets"] { + let (_, card) = html + .split_once(&format!("id=\"skill-{name}\"")) + .expect("the page lists the publishing skill"); + let (card, _) = card.split_once("</article>").unwrap(); + let (_, usage) = card + .split_once("Used by ") + .expect("the card names its users"); + let (usage, _) = usage.split_once("</p>").unwrap(); + assert!(usage.contains("Security fix publishing"), "{name}: {usage}"); + assert!(usage.contains("Architecture review"), "{name}: {usage}"); + } + } + + #[test] + fn prompts_page_shows_cloudflares_security_audit_and_its_mit_license() { + let html = render(); + let (_, skill) = html + .split_once("id=\"skill-thirdshift-security-audit\"") + .expect("the page lists thirdshift-security-audit"); + let (skill, _) = skill.split_once("</article>").unwrap(); + assert!(skill.contains("Security run") && skill.contains("Security review")); + assert!(skill.contains("MIT License")); + assert!(skill.contains("Copyright (c) 2025-2026 Cloudflare, Inc.")); + assert!(skill.contains("https://github.com/cloudflare/security-audit-skill")); + assert!(skill.contains("c1c8a8c1471069fb0e188eeaff69b8e8db6564a8")); + assert!(html.contains("href=\"#file-thirdshift-security-audit-license\"")); + } + /// The golden-file test for `prompts/`: a file for each prompt and the /// README.md, as rendered now, and no other. The crates.io package leaves /// `prompts/` out, so it skips there. diff --git a/src/pull_request.rs b/src/pull_request.rs index 882183b4..61d1d570 100644 --- a/src/pull_request.rs +++ b/src/pull_request.rs @@ -179,6 +179,37 @@ impl<A: Adapter> PullRequest<A> { Ok(()) } + /// Retain a deterministic review outcome even if the session refused + /// before it could edit the body. Keep the agent's detailed evidence. + pub fn record_security_review(&mut self, entry: &str) -> Result<()> { + let snapshot = self.observe(false)?.context("no PR found")?; + self.validate(&snapshot, true)?; + let body = self.adapter.body(&self.issue, snapshot.pr.number, false)?; + let section = "## Unaddressed findings"; + let start = "<!-- thirdshift:security-review -->"; + let end = "<!-- /thirdshift:security-review -->"; + let summary = format!("{start}\n### Security review outcome\n\n{entry}\n{end}\n"); + let previous = body + .find(start) + .and_then(|at| body[at..].find(end).map(|last| (at, at + last + end.len()))); + let updated = if let Some((first, last)) = previous { + format!("{}{}{}", &body[..first], summary.trim_end(), &body[last..]) + } else if let Some(at) = body.find(section) { + let after = at + section.len(); + let end = body[after..] + .find("\n## ") + .map_or(body.len(), |next| after + next); + format!("{}\n\n{summary}\n{}", body[..end].trim_end(), &body[end..]) + } else { + format!("{}\n\n{section}\n\n{summary}", body.trim_end()) + }; + if updated != body { + self.adapter + .set_body(&self.issue, snapshot.pr.number, &updated, false)?; + } + Ok(()) + } + /// Observe afresh and validate the expected branches and open state /// before marking this number ready. Already-ready PRs need no request. pub fn mark_ready(&mut self, checklist: Option<&str>) -> Result<Identified> { diff --git a/src/ready.rs b/src/ready.rs index f4a257a2..48383ff5 100644 --- a/src/ready.rs +++ b/src/ready.rs @@ -90,6 +90,8 @@ enum Reason { Ticket(IssueUrl), /// It is a Base fix issue, which the Run that opened it owns. BaseFix, + /// It belongs to a Security run, even if Claim creation failed. + SecurityFix, /// It is blocked by these open issues. Blocked(Vec<u64>), /// It was started, as this shows. @@ -177,6 +179,7 @@ impl<'a, R: Reads, P: FnMut(String)> Search<'a, R, P> { let why = match reason { Reason::Unready(label) => format!("labelled {label}"), Reason::Claimed => format!("labelled {}", claim::IN_PROGRESS), + Reason::SecurityFix => "a Security fix: its Security run owns it".to_string(), Reason::Ticket(spec) => { let listed = self.candidates.iter().position(|candidate| { candidate.issue.number == spec.number && candidate.issue.in_same_repo(spec) @@ -225,9 +228,9 @@ impl<'a, R: Reads, P: FnMut(String)> Search<'a, R, P> { /// Where `candidate`, an open issue labelled `ready-for-agent`, stands at /// `now`, reading what its listing doesn't give through `reads`. It is a /// Ready issue when it has no label that makes an Unready Ticket and no -/// Claim, is not a sub-issue, has no `base-fix` label and no open blocker, -/// was never started, is not a Spec whose Tickets are all closed, and is -/// settled: `wait` has passed since it was last labelled +/// Claim, is not a sub-issue, has no `base-fix` or `security-fix` label and no +/// open blocker, was never started, is not a Spec whose Tickets are all +/// closed, and is settled: `wait` has passed since it was last labelled /// `ready-for-agent` and since a sub-issue or a "blocked by" link of its was /// last added or removed. fn standing_of( @@ -243,6 +246,9 @@ fn standing_of( if claim::is_on(&candidate.labels) { return passed_over(Reason::Claimed); } + if candidate.labels.has(crate::security::fixing::SECURITY_FIX) { + return passed_over(Reason::SecurityFix); + } let read = reads.candidate(&candidate.issue)?; if let Some(spec) = read.parent { return passed_over(Reason::Ticket(spec)); diff --git a/src/run.rs b/src/run.rs index 4861d29b..89d7b9a3 100644 --- a/src/run.rs +++ b/src/run.rs @@ -75,8 +75,7 @@ pub enum StartedBy<'a> { /// `thirdshift <Issue URL>`: the Base branch is the branch checked out /// in the Launch directory. Command, - /// An Architect run, dispatching its plan, or a Pickup run, dispatching - /// the Ready issue it took: the Base branch is that run's, whatever the + /// A Pass, dispatching an issue: the Base branch is that Pass's, whatever the /// Launch directory has checked out. Dispatch { base: &'a str }, /// Another thirdshift, as this child Run, a Ticket's Run in a Spec run or @@ -119,6 +118,10 @@ pub fn run_to_end(issue: &IssueUrl, asks: &mut Asks, started_by: StartedBy) -> E started_by.child(), asks.base_fix.clone(), asks.harness.clone(), + crate::security::Options { + fix: asks.security_fix, + review: asks.security_review, + }, ); let outcome = run(issue, asks, started_by, &mut base_fix); Ended { @@ -164,6 +167,9 @@ fn run( directory: &directory, issue, goal: asks.goal, + security_fix: asks.security_fix, + security_review: asks.security_review + && !matches!(started_by, StartedBy::Child(Kind::Ticket { .. })), base_fix, logs: Logs::of_run(issue), harness: &asks.harness, @@ -184,7 +190,7 @@ fn run( /// The Base branch `started_by` gave the Run, if it gave one, stands in for /// the checked-out branch as the Base branch: the Spec branch or the Base /// branch of the Run that started a child Run, or the Base branch of the -/// Architect run or the Pickup run that dispatched this one. A Continuation's +/// Pass that dispatched this one. A Continuation's /// open pull request's base beats either. Unless the Run is a child Run, an /// issue with sub-issues is a Spec, taken on by a Spec run instead, whose /// Spec branch is picked like an Issue branch, running as many Tickets at @@ -356,6 +362,8 @@ struct LaunchAndGitHub<'a> { directory: &'a LaunchDirectory, issue: &'a IssueUrl, goal: Goal, + security_fix: bool, + security_review: bool, base_fix: &'a mut BaseFix, logs: Logs, harness: &'a Choice, @@ -366,6 +374,8 @@ impl LaunchAndGitHub<'_> { /// The Delivery of the issue into the Base branch `base`. fn delivery<'d>(&'d mut self, base: &'d str) -> Delivery<'d> { Delivery { + security_fix: self.security_fix, + security_review: self.security_review, issue: self.issue, base, goal: self.goal, @@ -561,7 +571,9 @@ mod tests { let spec_run = match work { Work::Run(_) => false, Work::SpecRun(_) => true, - Work::PickupRun(_) | Work::ArchitectRun(_) => panic!("not a Run's work"), + Work::PickupRun(_) | Work::ArchitectRun(_) | Work::SecurityRun(_) => { + panic!("not a Run's work") + } }; self.calls.push(Call::Started { spec_run }); } @@ -689,6 +701,8 @@ mod tests { /// What a Run started by its command is asked, by default. fn asks() -> Asks { Asks { + security_fix: false, + security_review: false, goal: Goal::ReadyForReview, notification: NotificationAsk::Skip, tickets_at_once: NonZeroUsize::new(1).unwrap(), diff --git a/src/run_ending.rs b/src/run_ending.rs index efc5974f..6222f362 100644 --- a/src/run_ending.rs +++ b/src/run_ending.rs @@ -33,9 +33,36 @@ const BASE_FIX: &str = "Base fix: "; pub fn read(ending: &Ending) -> Result<Account<'_>, &Skip> { match ending { Ending::Run(ended) => Ok(Account::of_run(ended)), + Ending::SecurityFix { ended, findings } => Ok(Account { + security_findings: Some(findings), + ..Account::of_run(ended) + }), Ending::Architect { review, dispatched } => { Ok(Account::of_architect(review, dispatched.as_ref())) } + Ending::Security(audited) => { + let account = match &audited.outcome { + Ok(recorded) => Account { + outcome: "findings recorded", + ended: Ok(recorded.to_string()), + interrupted: false, + pr_url: None, + advice: &[], + base_fix: None, + log: None, + ticket_lines: &[], + review: None, + security_findings: None, + urls: Vec::new(), + }, + Err(failed) => Account::of_failure(failed, "audit failed"), + }; + Ok(Account { + security_findings: Some(&audited.findings), + advice: &audited.advice, + ..account + }) + } Ending::Skipped(skip) => Err(skip), } } @@ -75,6 +102,8 @@ pub struct Account<'a> { pub ticket_lines: &'a [String], /// In an Architect run, how its Architecture review ended. pub review: Option<Review<'a>>, + /// Safe metadata from a Security run's private records, even if the audit failed. + pub security_findings: Option<&'a [crate::security::RecordedFinding]>, /// The URLs on stdout, a line each: the pull request's, or that of the /// issue an Architect run's review ended on. pub urls: Vec<&'a str>, @@ -93,22 +122,33 @@ pub struct Review<'a> { /// A failure's cause, as stderr gives it, down to what a session left /// running. #[derive(Debug, PartialEq, Eq)] -pub struct Cause(String); +pub struct Cause { + full: String, + safeguard_refusal: Option<crate::harness::interpretation::SafeguardRefusal>, +} impl Cause { /// The cause of a failure that failed with `error`. pub fn of(error: &anyhow::Error) -> Self { - Cause(format!("{error:#}")) + Self { + full: format!("{error:#}"), + safeguard_refusal: error.downcast_ref().copied(), + } } /// The whole cause, of as many lines as it has. pub fn full(&self) -> &str { - &self.0 + &self.full } /// Its first line, all of it that a short account gives. pub fn first_line(&self) -> &str { - self.0.lines().next().unwrap_or_default() + self.full.lines().next().unwrap_or_default() + } + + /// A fixed refusal category safe to send without the Harness's private diagnostic. + pub fn safeguard_refusal(&self) -> Option<&'static str> { + self.safeguard_refusal.map(|refusal| refusal.description()) } } @@ -131,6 +171,7 @@ impl<'a> Account<'a> { log: reached.log.as_deref(), ticket_lines: &reached.ticket_lines, review: None, + security_findings: None, urls: vec![&reached.pr_url], }, Err(failed) => Account { @@ -154,6 +195,7 @@ impl<'a> Account<'a> { log: failed.log.as_deref(), ticket_lines: &failed.ticket_lines, review: None, + security_findings: None, urls: failed.pr_url.as_deref().into_iter().collect(), } } @@ -181,6 +223,7 @@ impl<'a> Account<'a> { log: None, ticket_lines: &[], review: None, + security_findings: None, urls: vec![reviewed.url()], }, (Err(failed), None) => Account::of_failure(failed, "review failed"), @@ -249,18 +292,17 @@ impl Shown { steps.push(format!("{BASE_FIX}{report}")); } match &account.ended { - // Also on stderr, so the outcome shows even when stdout is - // captured. + // Also on stderr, so the outcome shows even when stdout is captured. Ok(line) => steps.push(line.clone()), - Err(cause) => { - steps.push(cause.full().to_string()); - steps.extend(account.advice.iter().map(Advice::to_string)); - if let Some(log) = account.log { - steps.push(format!("{SESSION_LOG}{}", log.display())); - } - if let Some(log) = logs::command_log_path() { - steps.push(format!("{COMMAND_LOG}{}", log.display())); - } + Err(cause) => steps.push(cause.full().to_string()), + } + steps.extend(account.advice.iter().map(Advice::to_string)); + if account.ended.is_err() { + if let Some(log) = account.log { + steps.push(format!("{SESSION_LOG}{}", log.display())); + } + if let Some(log) = logs::command_log_path() { + steps.push(format!("{COMMAND_LOG}{}", log.display())); } } Shown { @@ -400,7 +442,7 @@ mod tests { } fn cause(cause: &str) -> Cause { - Cause(cause.to_string()) + Cause::of(&anyhow!("{cause}")) } fn strings(lines: &[&str]) -> Vec<String> { @@ -433,6 +475,7 @@ mod tests { log: Some(Path::new(LOG)), ticket_lines: &[], review: None, + security_findings: None, urls: vec![PR], } } @@ -442,7 +485,7 @@ mod tests { fn failed_account(cause: &str) -> Account<'static> { Account { outcome: "failed", - ended: Err(Cause(cause.to_string())), + ended: Err(Cause::of(&anyhow!("{cause}"))), interrupted: false, pr_url: None, advice: &[], @@ -450,6 +493,7 @@ mod tests { log: Some(Path::new(LOG)), ticket_lines: &[], review: None, + security_findings: None, urls: Vec::new(), } } @@ -625,6 +669,7 @@ mod tests { line: format!("{outcome}: {PLAN}"), dispatched: None, }), + security_findings: None, urls: vec![PLAN], } ); diff --git a/src/security.rs b/src/security.rs new file mode 100644 index 00000000..73673f2e --- /dev/null +++ b/src/security.rs @@ -0,0 +1,800 @@ +//! Security runs: gate before work, audit origin's Base branch, then record +//! and reproduce the findings privately for the Day shift. + +use std::fmt; + +use anyhow::{Context, Result}; +use serde_json::Value; + +use crate::asks::Flags; +use crate::base_fix::Advice; +use crate::config::UserConfig; +use crate::failed_run::FailedRun; +use crate::github::ListedIssue; +use crate::harness::Choice; +use crate::issue::Repo; +use crate::launch::{self, AlreadyRunning, Launch, Start}; +use crate::logs::{self, Pass, Work}; +use crate::pass::{LaunchAndGitHub, Outside}; + +pub mod audit; +pub(crate) mod fixing; +pub mod reproduction; +pub(crate) mod review; + +/// Resolved security choices passed together to child Runs and Base fixes. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub struct Options { + pub fix: bool, + pub review: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FixAsk { + Allow, + Forbid, +} + +pub enum Outcome { + Skipped(Skipped), + Audited(Ended), + Fixed { + ended: crate::run::Ended, + findings: Vec<RecordedFinding>, + }, +} + +/// The audit's outcome and the private records it reached, including before a failure. +pub struct Ended { + pub outcome: Result<Recorded, FailedRun>, + pub findings: Vec<RecordedFinding>, + pub advice: Vec<Advice>, +} + +impl From<anyhow::Error> for Ended { + fn from(error: anyhow::Error) -> Self { + Self { + outcome: Err(error.into()), + findings: Vec::new(), + advice: Vec::new(), + } + } +} + +/// Preserve the typed command, quoting its words so the offered command can be run. +fn command_with_fixing() -> String { + let words = std::env::args() + .skip(1) + .map(|word| { + if word + .chars() + .all(|c| c.is_ascii_alphanumeric() || "-_/.:@=".contains(c)) + && !word.is_empty() + { + word + } else { + format!("'{}'", word.replace('\'', "'\\''")) + } + }) + .collect::<Vec<_>>(); + format!("thirdshift {} security-fix", words.join(" ")) +} + +/// Only the metadata allowed in a Run notification; no private write-up. +#[derive(Debug, PartialEq, Eq)] +pub struct RecordedFinding { + pub severity: Option<String>, + pub title: String, + pub url: String, +} + +impl RecordedFinding { + pub(crate) fn of_record(record: &Value) -> Result<Self> { + Ok(Self { + severity: record["severity"].as_str().map(String::from), + title: record["summary"] + .as_str() + .or_else(|| record["title"].as_str()) + .context("the Security finding's private record has no title")? + .to_string(), + url: record["html_url"] + .as_str() + .context("the Security finding's private record has no link")? + .to_string(), + }) + } +} + +pub enum Skipped { + AlreadyRunning(AlreadyRunning), + ReadyIssue(ListedIssue), + FindingWaiting, + FailedFix(crate::issue::IssueUrl), + UnchangedBase(String), +} + +impl fmt::Display for Skipped { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + Self::AlreadyRunning(running) => running.fmt(f), + Self::ReadyIssue(listed) => write!( + f, + "Ready issue #{} \"{}\" goes first: {}", + listed.issue.number, listed.title, listed.issue.url + ), + Self::FailedFix(ticket) => write!( + f, + "failed Security fix #{} is still open for the Day shift: {}", + ticket.number, ticket.url + ), + Self::FindingWaiting => f.write_str("a Security finding is waiting for the Day shift"), + Self::UnchangedBase(base) => write!( + f, + "Base branch {base} hasn't changed since the last completed Security audit" + ), + } + } +} + +pub struct Recorded { + pub created: usize, + pub existing: usize, +} + +impl fmt::Display for Recorded { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!( + f, + "Security audit recorded {} new finding(s), {} already recorded", + self.created, self.existing + ) + } +} + +pub fn run( + base: Option<&str>, + flags: &Flags, + config: &UserConfig, + harness: &mut Choice, +) -> Outcome { + let Launch { + directory, + repo, + base, + } = match launch::start(base) { + Ok(Start::Clear(launch)) => launch, + Ok(Start::AlreadyRunning(running)) => { + logs::skipped(Pass::Security, &running.0, &running); + return Outcome::Skipped(Skipped::AlreadyRunning(running)); + } + Err(error) => return Outcome::Audited(error.into()), + }; + let offer_command = + (flags.security_fix.is_none() && config.security_fix.is_none()).then(command_with_fixing); + run_through( + &mut LaunchAndGitHub { + launch: &directory, + base: &base, + repo: &repo, + harness, + flags, + config, + }, + &repo, + base.name(), + flags.security_fix_allowed(config), + offer_command.as_deref(), + ) +} + +fn run_through( + outside: &mut impl Outside, + repo: &Repo, + base: &str, + fixing: bool, + offer_command: Option<&str>, +) -> Outcome { + match outside.ready_issue() { + Ok(Some(ready)) => { + let skipped = Skipped::ReadyIssue(ready.listed); + outside.skipped(Pass::Security, &skipped); + return Outcome::Skipped(skipped); + } + Ok(None) => {} + Err(error) => return Outcome::Audited(error.into()), + } + let records = match outside.security_records() { + Ok(records) => records, + Err(error) => return Outcome::Audited(error.into()), + }; + match records.failed_fix() { + Ok(Some(ticket)) => { + let skipped = Skipped::FailedFix(ticket); + outside.skipped(Pass::Security, &skipped); + return Outcome::Skipped(skipped); + } + Ok(None) => {} + Err(error) => return Outcome::Audited(error.into()), + } + if fixing { + match records.next_fix() { + Ok(Some((record, metadata))) => return fix(outside, repo, base, record, metadata), + Ok(None) => {} + Err(error) => return Outcome::Audited(error.into()), + } + } + if records.waiting_for_day_shift(fixing) { + let skipped = Skipped::FindingWaiting; + outside.skipped(Pass::Security, &skipped); + return Outcome::Skipped(skipped); + } + match outside.base_unchanged_since_security_audit() { + Ok(true) => { + let skipped = Skipped::UnchangedBase(base.to_string()); + outside.skipped(Pass::Security, &skipped); + return Outcome::Skipped(skipped); + } + Ok(false) => {} + Err(error) => return Outcome::Audited(error.into()), + } + let audited = audit_and_record(outside, repo, base, offer_command); + if fixing && audited.outcome.is_ok() { + match outside + .security_records() + .and_then(|records| records.next_fix()) + { + Ok(Some((record, metadata))) => { + let outcome = fix(outside, repo, base, record, metadata); + if let Outcome::Fixed { ended, .. } = outcome { + return Outcome::Fixed { + ended, + findings: audited.findings, + }; + } + return outcome; + } + Ok(None) => {} + Err(error) => { + return Outcome::Audited(Ended { + outcome: Err(error.into()), + ..audited + }); + } + } + } + Outcome::Audited(audited) +} + +fn fix( + outside: &mut impl Outside, + repo: &Repo, + base: &str, + record: crate::github::SecurityRecord, + metadata: RecordedFinding, +) -> Outcome { + let mut log = None; + let ended = (|| -> Result<crate::run::Ended> { + outside.check_harness()?; + outside.started(Work::SecurityRun(repo)); + let (published, session_log) = outside.publish_security_fix(base, &record, &metadata.url); + log = session_log; + let issue = published?; + outside.link_security_fix(&record, &issue)?; + let mut ended = outside.dispatch(crate::pass::Dispatch::SecurityFix { + issue: &issue, + is_spec: record.fix_size()? == reproduction::FixSize::Spec, + base, + }); + if let Err(error) = + outside.record_security_fix_ending(&record, &issue, ended.outcome.is_ok()) + { + ended.outcome = Err(match ended.outcome { + Err(mut failed) => { + let cause = + format!("could not record the failed Security fix's ending: {error:#}"); + failed.error = std::mem::replace(&mut failed.error, error).context(cause); + failed + } + Ok(reached) => FailedRun { + log: reached.log, + pr_url: Some(reached.pr_url), + ticket_lines: reached.ticket_lines, + ..error + .context("could not record the successful Security fix's ending") + .into() + }, + }); + } + Ok(ended) + })(); + Outcome::Fixed { + ended: ended.unwrap_or_else(|error| crate::run::Ended { + outcome: Err(FailedRun { + log, + ..error.into() + }), + base_fix: None, + advice: Vec::new(), + }), + findings: vec![metadata], + } +} + +fn audit_and_record( + outside: &mut impl Outside, + repo: &Repo, + base: &str, + offer_command: Option<&str>, +) -> Ended { + let mut advice = Vec::new(); + let mut findings = Vec::new(); + let mut log = None; + let recorded = (|| -> Result<Recorded> { + outside.check_harness()?; + outside.check_node()?; + outside.started(Work::SecurityRun(repo)); + let (audited, session_log) = outside.audit(base); + log = session_log; + let audited = audited?; + let mut recorded = Recorded { + created: 0, + existing: 0, + }; + if audited.findings.is_empty() { + return Ok(recorded); + } + let mut known = outside.security_records()?; + let mut records = Vec::new(); + let mut seen = std::collections::HashSet::new(); + for finding in audited.findings { + let (record, metadata) = if let Some(value) = known.finding(&finding) { + recorded.existing += 1; + (known.record(value)?, RecordedFinding::of_record(value)?) + } else { + let value = outside.create_security_record(&known, &finding)?; + let record = known.record(&value)?; + let metadata = RecordedFinding::of_record(&value)?; + known.remember(value); + recorded.created += 1; + outside.step("recorded a Security finding privately".to_string()); + (record, metadata) + }; + let url = metadata.url.clone(); + findings.push(metadata); + if record.untriaged() && seen.insert(finding.fingerprint) { + records.push((record, url)); + } else if !record.untriaged() { + outside.step(format!( + "keeping the Day shift's grade for {}", + record.name() + )); + } + } + for (index, (record, url)) in records.iter().enumerate() { + let number = index + 1; + outside.step(format!( + "starting Security reproduction {number} of {}", + record.name() + )); + let (reproduced, session_log) = outside.reproduce(record, number); + if session_log.is_some() { + log = session_log; + } + let reproduced = reproduced?; + outside.update_security_record(record, &reproduced)?; + if reproduced.severity().is_some() + && let Some(command) = offer_command + { + advice = vec![ + Advice { + label: "Allow fixing", + value: command.to_string(), + }, + Advice { + label: "Or set", + value: "fix = true under [security] in ~/.thirdshift/config.toml" + .to_string(), + }, + ]; + } + for finding in findings.iter_mut().filter(|finding| finding.url == *url) { + finding.severity = reproduced + .severity() + .map(|severity| severity.name().to_string()); + } + outside.step(format!( + "Security reproduction {number}: {}", + reproduced.outcome + )); + } + Ok(recorded) + })(); + Ended { + outcome: recorded.map_err(|error| FailedRun { + log, + ..error.into() + }), + findings, + advice, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pass::{Call, InMemory, widgets}; + + #[test] + fn a_reproduced_fix_precedes_waiting_and_unchanged_but_yields_to_a_ready_issue() { + for ready in [false, true] { + let mut outside = InMemory::default().advisories(vec![ + serde_json::json!({"state":"draft", "severity":null}), + serde_json::json!({ + "state":"draft", "severity":"critical", "ghsa_id":"GHSA-first", + "summary":"Bound input", "html_url":"https://github.com/acme/widgets/security/advisories/GHSA-first", + "description":"Private record\n<!-- thirdshift:security-reproduction -->\nOutcome: reproduced critical single\n" + }), + serde_json::json!({ + "state":"draft", "severity":"critical", "ghsa_id":"GHSA-second", + "summary":"Other bound", "html_url":"https://github.com/acme/widgets/security/advisories/GHSA-second", + "description":"Private record\n<!-- thirdshift:security-reproduction -->\nOutcome: reproduced critical single\n" + }), + ]).unchanged_base().dispatched_ending(crate::pass::ready_for_review()); + if ready { + outside = outside.ready(7, false); + } + let outcome = run_through(&mut outside, &widgets(), "main", true, None); + if ready { + assert!(matches!(outcome, Outcome::Skipped(Skipped::ReadyIssue(_)))); + assert!(matches!( + &outside.calls[..], + [Call::ReadySearch, Call::Skipped(_)] + )); + } else { + assert!(matches!( + outcome, + Outcome::Fixed { + ended: crate::run::Ended { outcome: Ok(_), .. }, + .. + } + )); + assert!( + outside + .calls + .contains(&Call::PublishSecurityFix("GHSA-first".into())) + ); + assert!(outside.calls.contains(&Call::DispatchSecurityFix { + issue: 8, + is_spec: false, + base: "main".into() + })); + assert!(!outside.calls.contains(&Call::AuditHistory)); + assert!(!outside.calls.contains(&Call::NodeCheck)); + } + } + } + + #[test] + fn a_draft_without_severity_waits_before_prerequisites_or_audit() { + let mut outside = InMemory::default() + .advisories(vec![serde_json::json!({ + "state": "draft", "severity": null, + "ghsa_id": "GHSA-test", "summary": "Unchecked input size" + })]) + .harness_failing(); + let outcome = run_through(&mut outside, &widgets(), "main", false, None); + assert!(matches!(outcome, Outcome::Skipped(_))); + assert_eq!( + outside.calls, + vec![ + Call::ReadySearch, + Call::AdvisoryList, + Call::Skipped("a Security finding is waiting for the Day shift".into()), + ] + ); + } + + #[test] + fn advisory_triage_ends_the_wait() { + for (state, severity, waits) in [ + ("draft", None, true), + ("draft", Some("low"), false), + ("draft", Some("high"), false), + ("published", None, false), + ("closed", None, false), + ("triage", None, false), + ] { + let mut outside = InMemory::default().advisories(vec![serde_json::json!({ + "state": state, "severity": severity + })]); + let outcome = run_through(&mut outside, &widgets(), "main", false, None); + assert_eq!( + matches!(outcome, Outcome::Skipped(_)), + waits, + "{state}: {severity:?}" + ); + assert_eq!( + outside.calls.contains(&Call::SecurityAudit("main".into())), + !waits + ); + } + } + + #[test] + fn only_open_private_findings_still_needing_triage_wait() { + for (state, labels, waits) in [ + ("open", vec!["security-finding", "needs-triage"], true), + ("OPEN", vec!["security-finding", "Needs-Triage"], true), + ("closed", vec!["security-finding", "needs-triage"], false), + ("open", vec!["security-finding"], false), + ("open", vec!["security-finding", "wontfix"], false), + ] { + let mut outside = InMemory::default().finding_issues(vec![serde_json::json!({ + "state": state, + "labels": labels.iter().map(|name| serde_json::json!({"name":name})).collect::<Vec<_>>() + })]); + let outcome = run_through(&mut outside, &widgets(), "main", false, None); + assert_eq!( + matches!(outcome, Outcome::Skipped(_)), + waits, + "{state}: {labels:?}" + ); + assert_eq!( + outside.calls.contains(&Call::SecurityAudit("main".into())), + !waits + ); + } + } + + #[test] + fn checks_prerequisites_then_audits_without_pulling() { + let mut outside = InMemory::default(); + let outcome = run_through(&mut outside, &widgets(), "main", false, None); + assert!(matches!( + outcome, + Outcome::Audited(Ended { outcome: Ok(_), .. }) + )); + assert_eq!( + outside.calls, + vec![ + Call::ReadySearch, + Call::AdvisoryList, + Call::AuditHistory, + Call::HarnessCheck, + Call::NodeCheck, + Call::Started(None), + Call::SecurityAudit("main".to_string()) + ] + ); + } + #[test] + fn a_ready_issue_precedes_every_prerequisite_and_audit() { + let mut outside = InMemory::default() + .ready(7, false) + .advisories(vec![serde_json::json!({"state":"draft", "severity":null})]) + .unchanged_base() + .node_failing() + .harness_failing(); + assert!(matches!( + run_through(&mut outside, &widgets(), "main", false, None), + Outcome::Skipped(Skipped::ReadyIssue(_)) + )); + assert!(matches!( + &outside.calls[..], + [Call::ReadySearch, Call::Skipped(_)] + )); + } + + #[test] + fn security_gates_choose_ready_then_waiting_then_unchanged_before_work() { + for ready in [false, true] { + for waiting in [false, true] { + for unchanged in [false, true] { + let mut outside = InMemory::default(); + if ready { + outside = outside.ready(7, false); + } + if waiting { + outside = outside.advisories(vec![ + serde_json::json!({"state":"draft", "severity":null}), + ]); + } + if unchanged { + outside = outside.unchanged_base(); + } + let outcome = run_through(&mut outside, &widgets(), "main", false, None); + let expected = if ready { + "Ready issue #7" + } else if waiting { + "a Security finding is waiting for the Day shift" + } else if unchanged { + "Base branch main hasn't changed since the last completed Security audit" + } else { + assert!(matches!( + outcome, + Outcome::Audited(Ended { outcome: Ok(_), .. }) + )); + assert!(outside.calls.contains(&Call::SecurityAudit("main".into()))); + continue; + }; + let Outcome::Skipped(skipped) = outcome else { + panic!("Security run should skip"); + }; + assert!(skipped.to_string().starts_with(expected)); + assert!(!outside.calls.contains(&Call::HarnessCheck)); + assert_eq!(outside.calls.contains(&Call::AdvisoryList), !ready); + assert_eq!( + outside.calls.contains(&Call::AuditHistory), + !ready && !waiting + ); + } + } + } + } + + #[test] + fn a_failed_fix_yields_to_ready_work_but_precedes_waiting_and_audit_history() { + for private in [false, true] { + for ready in [false, true] { + for closed in [false, true] { + for fixing in [false, true] { + let body = "Private record\n<!-- thirdshift:security-fix -->\nFix Ticket: https://github.com/acme/widgets/issues/8\nFix Run: failed\n"; + let values = vec![ + serde_json::json!({ + "state": if private { "open" } else { "draft" }, + "severity": null, "description": body, "body": body, + "security_fix_closed": closed, + }), + serde_json::json!({ + "state": if private { "open" } else { "draft" }, + "severity": null, "labels": [{"name":"needs-triage"}], + }), + ]; + let mut outside = if private { + InMemory::default().finding_issues(values) + } else { + InMemory::default().advisories(values) + } + .unchanged_base(); + if ready { + outside = outside.ready(7, false); + } + let Outcome::Skipped(skipped) = + run_through(&mut outside, &widgets(), "main", fixing, None) + else { + panic!("Security run should skip"); + }; + let expected = if ready { + "Ready issue #7" + } else if !closed { + "failed Security fix #8 is still open" + } else { + "a Security finding is waiting for the Day shift" + }; + assert!(skipped.to_string().starts_with(expected), "{skipped}"); + assert!(!outside.calls.contains(&Call::HarnessCheck)); + assert!(!outside.calls.contains(&Call::AuditHistory)); + assert_eq!(outside.calls.contains(&Call::AdvisoryList), !ready); + } + } + } + } + } + + #[test] + fn each_failed_prerequisite_prevents_later_operations() { + for (mut outside, expected) in [ + ( + InMemory::default().harness_failing(), + vec![ + Call::ReadySearch, + Call::AdvisoryList, + Call::AuditHistory, + Call::HarnessCheck, + ], + ), + ( + InMemory::default().node_failing(), + vec![ + Call::ReadySearch, + Call::AdvisoryList, + Call::AuditHistory, + Call::HarnessCheck, + Call::NodeCheck, + ], + ), + ] { + assert!(matches!( + run_through(&mut outside, &widgets(), "main", false, None), + Outcome::Audited(Ended { + outcome: Err(_), + .. + }) + )); + assert_eq!(outside.calls, expected); + } + } + + #[test] + fn a_failed_audit_records_nothing_and_keeps_its_session_log() { + let mut outside = InMemory::default() + .audit_failing("invalid report") + .session_log("audit.jsonl"); + let Outcome::Audited(Ended { + outcome: Err(failed), + .. + }) = run_through(&mut outside, &widgets(), "main", false, None) + else { + panic!("audit should fail"); + }; + assert_eq!(failed.log, Some(std::path::PathBuf::from("audit.jsonl"))); + assert_eq!( + outside + .calls + .iter() + .filter(|call| **call == Call::AdvisoryList) + .count(), + 1 + ); + assert!( + !outside + .calls + .iter() + .any(|call| matches!(call, Call::CreateAdvisory(_))) + ); + } + + #[test] + fn only_new_fingerprints_get_a_private_record() { + use crate::github::{DraftAdvisory, Package}; + use serde_json::json; + let draft = |fingerprint: &str| DraftAdvisory { + fingerprint: fingerprint.into(), + summary: "Candidate".into(), + description: format!("Fingerprint: `{fingerprint}`"), + package: Package { + ecosystem: "other".into(), + name: None, + }, + }; + let mut outside = InMemory::default() + .audited(vec![draft("old"), draft("new")]) + .advisories(vec![ + json!({"ghsa_id":"old", "state":"closed", "description":"Fingerprint: `old`", "summary":"Candidate", "html_url":"https://github.com/acme/widgets/security/advisories/GHSA-old"}), + ]); + let Outcome::Audited(Ended { + outcome: Ok(recorded), + .. + }) = run_through(&mut outside, &widgets(), "main", false, None) + else { + panic!("audit should succeed"); + }; + assert_eq!((recorded.created, recorded.existing), (1, 1)); + assert_eq!( + outside + .calls + .iter() + .filter_map(|call| match call { + Call::CreateAdvisory(fingerprint) => Some(fingerprint.as_str()), + _ => None, + }) + .collect::<Vec<_>>(), + vec!["new"] + ); + assert_eq!( + outside + .calls + .iter() + .filter(|call| matches!( + call, + Call::CreateAdvisory(_) | Call::Reproduce(_) | Call::UpdateSecurityRecord(_) + )) + .collect::<Vec<_>>(), + vec![ + &Call::CreateAdvisory("new".into()), + &Call::Reproduce("new".into()), + &Call::UpdateSecurityRecord("new".into()), + ] + ); + } +} diff --git a/src/security/audit.rs b/src/security/audit.rs new file mode 100644 index 00000000..539b52c5 --- /dev/null +++ b/src/security/audit.rs @@ -0,0 +1,274 @@ +//! The Security audit session and its validated report artifacts. Kept under +//! the repository's logs, independently of the disposable audit checkout. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use anyhow::{Context, Result, bail}; +use serde_json::{Value, json}; + +use crate::git::Git; +use crate::github::{DraftAdvisory, Package}; +use crate::harness::Choice; +use crate::issue::Repo; +use crate::logs; +use crate::process::{self, Control, Interruption}; +use crate::progress; +use crate::prompt; +use crate::session::{Logs, Purpose, Sessions}; +use crate::skills; +use crate::worktree::ReviewWorktree; + +pub struct Audited { + pub findings: Vec<DraftAdvisory>, +} + +/// Read the skill's own run records, ignoring unfinished or unreadable records +/// so a failed audit can be retried. Completion time, rather than the directory's +/// random suffix, orders audits that started within the same second. +pub fn last_commit(repo: &Repo, base: &str) -> Result<Option<String>> { + let root = logs::root(repo).join("audits"); + let entries = match fs::read_dir(&root) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(error).context("could not read the Security audit history"), + }; + let mut last = None; + for entry in entries { + let entry = entry?; + if !entry.file_type()?.is_dir() { + continue; + } + let path = entry.path().join("run-metadata.json"); + let Some(record) = fs::read(&path) + .ok() + .and_then(|bytes| serde_json::from_slice::<Value>(&bytes).ok()) + else { + continue; + }; + if record["run_status"] != "complete" || record["source_ref"]["branch"] != base { + continue; + } + let completed = fs::metadata(&path)?.modified()?; + if last.as_ref().is_none_or(|(time, _)| completed > *time) { + last = Some(( + completed, + record["source_ref"]["commit"].as_str().map(String::from), + )); + } + } + Ok(last.and_then(|(_, commit)| commit)) +} + +pub fn check_node() -> Result<()> { + node(&["--version"]) + .context("Node.js is required for the Security audit validators; install node on PATH")?; + Ok(()) +} + +pub fn run( + launch: &Git, + repo: &Repo, + base: &str, + harness: &Choice, +) -> (Result<Audited>, Option<PathBuf>) { + let prepared = (|| -> Result<_> { + let worktree = ReviewWorktree::create(launch, &repo.name, base)?; + let commit = Git::new(worktree.path()).run(&["rev-parse", "HEAD"])?; + let root = logs::root(repo).join("audits"); + fs::create_dir_all(&root) + .with_context(|| format!("could not create {}", root.display()))?; + let output = tempfile::Builder::new() + .prefix(&format!("{}-", logs::stamp())) + .tempdir_in(&root)? + .keep(); + // Case-insensitive filesystems can resolve a candidate with different + // casing. Name the document as it is tracked in the audited checkout. + let tracked = Git::new(worktree.path()).run(&["ls-files", "-z"])?; + let threat_model = [ + "SECURITY.md", + "THREAT_MODEL.md", + "THREAT-MODEL.md", + "docs/threat-model.md", + "docs/THREAT-MODEL.md", + "docs/THREAT_MODEL.md", + ] + .into_iter() + .find(|file| { + tracked.split('\0').any(|tracked| tracked == *file) + && worktree.path().join(file).is_file() + }); + let prompt = prompt::security_audit(base, &commit, &root, &output, threat_model); + Ok((worktree, output, prompt, commit)) + })(); + let (worktree, output, prompt, commit) = match prepared { + Ok(prepared) => prepared, + Err(error) => return (Err(error), None), + }; + let logs = Logs::of_security_run(repo); + let (audited, log) = Sessions::within(&logs, worktree.path(), harness, |sessions| { + progress::step(format!("starting the Security audit of {base}")); + let message = + sessions.run_to_final_message(Purpose::Security, "security-audit", &prompt)?; + let line = message.as_deref().and_then(|message| { + message + .lines() + .map(str::trim) + .rfind(|line| !line.is_empty()) + }); + match line { + Some(prompt::AUDIT_COMPLETE_LINE) => {} + Some(prompt::AUDIT_INCOMPLETE_LINE) => bail!( + "the Security audit ended incomplete; artifacts: {}", + output.display() + ), + _ => bail!("the Security audit ended without the final line its prompt asks for"), + } + let metadata: Value = serde_json::from_slice( + &fs::read(output.join("run-metadata.json")) + .context("could not read the Security audit's run-metadata.json")?, + ) + .context("the Security audit's run-metadata.json is invalid JSON")?; + if metadata["run_status"] != "complete" { + bail!( + "the Security audit's run-metadata.json does not mark the audit complete; artifacts: {}", + output.display() + ); + } + let skill = skills::written_out()?.join("thirdshift-security-audit"); + for (validator, report) in [ + ("validate-findings.cjs", "findings.json"), + ("validate-coverage-ledger.cjs", "coverage-ledger.json"), + ] { + node(&[ + skill + .join(validator) + .to_str() + .context("non-UTF-8 skill path")?, + output + .join(report) + .to_str() + .context("non-UTF-8 report path")?, + ]) + .with_context(|| { + format!("the Security audit report failed the skill's validator {validator}") + })?; + } + let findings: Vec<Value> = + serde_json::from_slice(&fs::read(output.join("findings.json"))?)?; + let package = package_in(worktree.path()); + let findings = findings.into_iter().filter(|finding| finding["verdict"] != "rejected").map(|finding| { + let fingerprint = finding["fingerprint"].as_str().context("finding has no fingerprint")?.to_string(); + Ok(DraftAdvisory { + summary: finding["title"].as_str().context("finding has no title")?.to_string(), + description: format!("Found by thirdshift's Security run.\n\nFingerprint: `{fingerprint}`\nAudited commit: `{commit}`\n\n{}\n\n```json\n{}\n```\n", finding["description"].as_str().unwrap_or_default(), serde_json::to_string_pretty(&finding)?), + fingerprint, + package: package.clone(), + }) + }).collect::<Result<_>>()?; + Ok(Audited { findings }) + }); + // The session may claim completion before a validator or its final line + // fails. Keep that attempt incomplete in the same skill record. The source + // ref comes from the acquired checkout, even if the session wrote it wrong. + let recorded = finish_record(&output, &commit, base, audited.is_ok()); + let audited = match (audited, recorded) { + (Ok(audited), Ok(())) => Ok(audited), + (Err(error), _) | (_, Err(error)) => Err(error), + }; + (audited, log) +} + +fn finish_record(output: &Path, commit: &str, base: &str, complete: bool) -> Result<()> { + let path = output.join("run-metadata.json"); + let mut metadata: Value = match fs::read(&path) + .ok() + .and_then(|bytes| serde_json::from_slice(&bytes).ok()) + { + Some(Value::Object(metadata)) => Value::Object(metadata), + _ if !complete => return Ok(()), + _ => bail!("could not read the completed Security audit's run record"), + }; + metadata["source_ref"] = json!({"commit": commit, "branch": base, "dirty": false}); + metadata["run_status"] = Value::String(if complete { "complete" } else { "incomplete" }.into()); + if !complete { + metadata["incomplete_reason"] = + Value::String("thirdshift did not accept the Security audit".into()); + } + fs::write(&path, serde_json::to_vec_pretty(&metadata)?) + .context("could not finish the Security audit's run record") +} + +fn node(args: &[&str]) -> Result<()> { + let output = process::output( + Command::new("node").args(args), + None, + Control { + name: "Node.js", + interruption: Interruption::Ordinary, + stop: &|child| process::stop(child, &[libc::SIGTERM]), + }, + )?; + if !output.status.success() { + // The validators' diagnostics can contain private evidence. Leave + // details in the audit artifacts, out of progress and Command logs. + bail!("Node.js exited {}", output.status); + } + Ok(()) +} + +/// Manifest identity only: no affected or patched versions are inferred. +pub(super) fn package_in(worktree: &Path) -> Package { + for (file, section, ecosystem) in [ + ("Cargo.toml", "package", "rust"), + ("pyproject.toml", "project", "pip"), + ] { + let name = fs::read_to_string(worktree.join(file)) + .ok() + .and_then(|text| toml::from_str::<toml::Value>(&text).ok()) + .and_then(|manifest| { + manifest + .get(section)? + .get("name")? + .as_str() + .map(String::from) + }); + if let Some(name) = name { + return Package { + ecosystem: ecosystem.to_string(), + name: Some(name), + }; + } + } + let npm = fs::read(worktree.join("package.json")) + .ok() + .and_then(|text| serde_json::from_slice::<Value>(&text).ok()) + .and_then(|manifest| manifest["name"].as_str().map(String::from)); + if let Some(name) = npm { + return Package { + ecosystem: "npm".to_string(), + name: Some(name), + }; + } + let go = fs::read_to_string(worktree.join("go.mod")) + .ok() + .and_then(|text| { + text.lines().find_map(|line| { + let mut words = line.split_whitespace(); + (words.next()? == "module") + .then(|| words.next().map(|name| name.trim_matches('"').to_string())) + .flatten() + }) + }); + match go { + Some(name) => Package { + ecosystem: "go".to_string(), + name: Some(name), + }, + None => Package { + ecosystem: "other".to_string(), + name: None, + }, + } +} diff --git a/src/security/fixing.rs b/src/security/fixing.rs new file mode 100644 index 00000000..a0f8940a --- /dev/null +++ b/src/security/fixing.rs @@ -0,0 +1,188 @@ +//! Publish and check a terse fix Ticket or Spec, without exposing the private write-up. + +use std::path::PathBuf; + +use anyhow::{Context, Result, bail}; +use chrono::Utc; + +use crate::git::Git; +use crate::github::{GitHub, SecurityRecord}; +use crate::harness::Choice; +use crate::issue::{IssueUrl, Repo}; +use crate::labels::{Edit, Label, NEEDS_TRIAGE, READY_FOR_AGENT}; +use crate::prompt; +use crate::security::reproduction::FixSize; +use crate::session::{Logs, Purpose, Sessions}; +use crate::worktree::ReviewWorktree; + +pub const SECURITY_FIX: Label = + Label::new("security-fix", "A fix for a reproduced Security finding"); + +pub fn publish( + launch: &Git, + repo: &Repo, + base: &str, + record: &SecurityRecord, + url: &str, + harness: &Choice, +) -> (Result<IssueUrl>, Option<PathBuf>) { + let size = match record.fix_size() { + Ok(size) => size, + Err(error) => return (Err(error), None), + }; + if let SecurityRecord::Issue { number, .. } = record + && size == FixSize::Single + { + let ready = (|| -> Result<IssueUrl> { + let issue = IssueUrl::parse(url)?; + if issue.number != *number || !issue.repo_slug().eq_ignore_ascii_case(&repo.slug()) { + bail!("the private Security finding's issue does not match its record"); + } + let github = GitHub::new(); + let viewed = github.issue(&issue)?; + if !viewed.is_open + || viewed + .labels + .swapped(&[NEEDS_TRIAGE], &[]) + .unready() + .is_some() + || !github.tickets(&issue)?.is_empty() + { + bail!("the private Security finding must be an open, ready single Ticket"); + } + if crate::interrupt::requested() { + bail!("interrupted"); + } + Edit::of( + &issue, + viewed.labels, + &[NEEDS_TRIAGE], + &[READY_FOR_AGENT, SECURITY_FIX], + ) + .apply(&github)?; + Ok(issue) + })(); + return (ready, None); + } + let worktree = match ReviewWorktree::create(launch, &repo.name, base) { + Ok(worktree) => worktree, + Err(error) => return (Err(error), None), + }; + let logs = Logs::of_security_run(repo); + let started = Utc::now(); + let prompt = prompt::security_fix(base, url, record.description()); + Sessions::within(&logs, worktree.path(), harness, |sessions| { + let message = + sessions.run_to_final_message(Purpose::Security, "security-fix-publishing", &prompt)?; + let line = message + .as_deref() + .unwrap_or_default() + .lines() + .map(str::trim) + .rfind(|line| !line.is_empty()) + .unwrap_or_default(); + let prefix = match size { + FixSize::Single => prompt::SECURITY_FIX_LINE, + FixSize::Spec => prompt::SECURITY_FIX_SPEC_LINE, + }; + let issue = line.strip_prefix(prefix).and_then(|url| IssueUrl::parse(url).ok()).context("the Security fix publishing session ended without the final line its prompt asks for")?; + if !issue.repo_slug().eq_ignore_ascii_case(&repo.slug()) { + bail!("the Security fix Ticket is not in this repository"); + } + let private = matches!(record, SecurityRecord::Issue { .. }); + if let SecurityRecord::Issue { number, .. } = record + && issue.number != *number + { + bail!("a private Security fix must reuse the finding's own issue"); + } + let github = GitHub::new(); + let viewed = github.issue(&issue)?; + if !viewed.is_open { + bail!("the Security fix Ticket is closed"); + } + if !private && viewed.created.timestamp() < started.timestamp() { + bail!("the Security fix Ticket was created before this session"); + } + if !private && !viewed.labels.has(NEEDS_TRIAGE) + || viewed + .labels + .swapped(&[NEEDS_TRIAGE], &[]) + .unready() + .is_some() + { + bail!( + "the Security fix Ticket must be labelled needs-triage with no other Unready Ticket label" + ); + } + let tickets = github.tickets(&issue)?; + if (size == FixSize::Spec) == tickets.is_empty() { + bail!("the Security fix's sub-issues do not match the reproduced fix size"); + } + if !private { + check_issue_text(&github, &issue, record, url)?; + } + for child in &tickets { + let issue = issue.sibling(child.number); + let viewed = github.issue(&issue)?; + if !viewed.is_open || viewed.created.timestamp() < started.timestamp() { + bail!("a Security fix Spec's Ticket must be new and open"); + } + if child.has_sub_issues + || !viewed.labels.has(READY_FOR_AGENT) + || viewed.labels.unready().is_some() + { + bail!("a Security fix Spec's Ticket must be ready-for-agent with no sub-issues"); + } + check_issue_text(&github, &issue, record, url)?; + } + for child in tickets { + let issue = issue.sibling(child.number); + Edit::of(&issue, github.issue_labels(&issue)?, &[], &[SECURITY_FIX]).apply(&github)?; + } + if crate::interrupt::requested() { + bail!("interrupted"); + } + Edit::of( + &issue, + viewed.labels, + &[NEEDS_TRIAGE], + &[READY_FOR_AGENT, SECURITY_FIX], + ) + .apply(&github)?; + Ok(issue) + }) +} + +fn check_issue_text( + github: &GitHub, + issue: &IssueUrl, + record: &SecurityRecord, + url: &str, +) -> Result<()> { + let body = github.security_fix_text(issue)?; + if !body.contains(url) { + bail!("the Security fix Ticket does not link the private record"); + } + // Reject copied write-up lines and test text. The session is also + // instructed to publish only what the fix changes, never a paraphrase. + for line in record.description().lines().filter(|line| { + let line = line.trim(); + // A bare identifier can also be an ordinary word in fix prose. + // Complete short statements such as bypass_login(); stay checked. + line.chars().any(char::is_alphanumeric) + && !line.chars().all(|ch| ch.is_alphanumeric() || ch == '_') + && !line.starts_with("```") + && !line.starts_with("Fingerprint:") + && !line.starts_with("Audited commit:") + && !line.starts_with("Outcome:") + && !line.starts_with("Severity:") + && !line.starts_with("Fix size:") + && !line.starts_with('#') + && !line.starts_with("<!--") + }) { + if body.contains(line.trim()) { + bail!("the Security fix Ticket includes private write-up text"); + } + } + Ok(()) +} diff --git a/src/security/reproduction.rs b/src/security/reproduction.rs new file mode 100644 index 00000000..f93f06e8 --- /dev/null +++ b/src/security/reproduction.rs @@ -0,0 +1,212 @@ +//! One proof-of-concept session in an owned disposable checkout. Its test +//! and final message are collected before the checkout is removed. + +use std::fmt; +use std::fs; +use std::path::PathBuf; + +use anyhow::{Context, Result, bail}; +use serde::Serialize; + +use crate::git::Git; +use crate::github::{FIX_TICKET_MARKER, SecurityRecord}; +use crate::harness::Choice; +use crate::issue::Repo; +use crate::logs; +use crate::prompt; +use crate::session::{Logs, Purpose, Sessions}; +use crate::worktree::ReviewWorktree; + +// Declaration order is severity order, most severe first. +#[derive(Clone, Copy, Serialize, PartialEq, Eq, PartialOrd, Ord)] +#[serde(rename_all = "lowercase")] +pub enum Severity { + Critical, + High, + Medium, + Low, + Informational, +} + +impl Severity { + pub(crate) fn parse(name: &str) -> Option<Self> { + match name { + "critical" => Some(Self::Critical), + "high" => Some(Self::High), + "medium" => Some(Self::Medium), + "low" => Some(Self::Low), + "informational" => Some(Self::Informational), + _ => None, + } + } + + pub(crate) fn name(self) -> &'static str { + match self { + Self::Critical => "critical", + Self::High => "high", + Self::Medium => "medium", + Self::Low => "low", + Self::Informational => "informational", + } + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +pub enum FixSize { + Single, + Spec, +} + +impl FixSize { + fn name(&self) -> &'static str { + match self { + Self::Single => "single", + Self::Spec => "spec", + } + } +} + +pub enum Outcome { + Reproduced { severity: Severity, size: FixSize }, + NotReproduced, +} + +impl fmt::Display for Outcome { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Reproduced { severity, size } => { + write!(f, "reproduced {} {}", severity.name(), size.name()) + } + Self::NotReproduced => write!(f, "not reproduced"), + } + } +} + +pub struct Reproduction { + pub outcome: Outcome, + pub notes: String, + pub test: String, +} + +impl Reproduction { + pub fn severity(&self) -> Option<Severity> { + match self.outcome { + Outcome::Reproduced { severity, .. } => Some(severity), + Outcome::NotReproduced => None, + } + } + + pub fn description(&self, original: &str) -> String { + // Replacing reproduction evidence must keep the dispatched fix's link. + let (original, fix) = original + .rsplit_once(FIX_TICKET_MARKER) + .map_or((original, String::new()), |(original, ticket)| { + (original, format!("{FIX_TICKET_MARKER}{ticket}")) + }); + let original = original + .split_once("\n<!-- thirdshift:security-reproduction -->\n") + .map_or(original, |(original, _)| original); + let severity = self + .severity() + .map(|severity| format!("Severity: {}\n", severity.name())) + .unwrap_or_default(); + let size = match &self.outcome { + Outcome::Reproduced { size, .. } => format!("Fix size: {}\n", size.name()), + Outcome::NotReproduced => String::new(), + }; + // A test may itself contain Markdown fences. Preserve its text without + // allowing one of those fences to close the record's code block. + let fence = "`".repeat( + self.test + .lines() + .map(|line| line.chars().take_while(|c| *c == '`').count()) + .max() + .unwrap_or(0) + .max(2) + + 1, + ); + format!( + "{}\n\n<!-- thirdshift:security-reproduction -->\n## Reproduction\n\nOutcome: {}\n{severity}{size}\n{}\n\n### Proof-of-concept test\n\n{fence}\n{}{fence}\n{fix}", + original.trim_end(), + self.outcome, + self.notes, + if self.test.ends_with('\n') { + self.test.clone() + } else { + format!("{}\n", self.test) + } + ) + } +} + +pub fn run( + launch: &Git, + repo: &Repo, + record: &SecurityRecord, + number: usize, + harness: &Choice, +) -> (Result<Reproduction>, Option<PathBuf>) { + let prepared = (|| -> Result<_> { + let commit = record + .description() + .lines() + .find_map(|line| { + line.strip_prefix("Audited commit: `") + .and_then(|commit| commit.strip_suffix('`')) + }) + .context("Security finding record has no audited commit")?; + if ![40, 64].contains(&commit.len()) || !commit.bytes().all(|byte| byte.is_ascii_hexdigit()) + { + bail!("Security finding record has an invalid audited commit"); + } + let worktree = ReviewWorktree::at_commit(launch, &repo.name, commit)?; + let root = logs::root(repo).join("reproductions"); + fs::create_dir_all(&root)?; + let output = tempfile::Builder::new() + .prefix("reproduction-") + .tempdir_in(root)?; + let test = output.path().join("test.txt"); + let prompt = prompt::security_reproduction(commit, record.description(), &test); + Ok((worktree, output, test, prompt)) + })(); + let (worktree, _output, test, prompt) = match prepared { + Ok(prepared) => prepared, + Err(error) => return (Err(error), None), + }; + let logs = Logs::of_security_run(repo); + Sessions::within(&logs, worktree.path(), harness, |sessions| { + let message = sessions.run_to_final_message( + Purpose::Security, + &format!("security-reproduction-{number}"), + &prompt, + )?; + let message = message.as_deref().unwrap_or_default().trim_end(); + let (notes, line) = message.rsplit_once('\n').unwrap_or(("", message)); + let outcome = match line.split_whitespace().collect::<Vec<_>>().as_slice() { + ["Security", "reproduction:", "not", "reproduced"] => Outcome::NotReproduced, + ["Security", "reproduction:", "reproduced", severity, size] => { + let severity = Severity::parse(severity) + .context("the Security reproduction ended with an invalid severity")?; + let size = match *size { + "single" => FixSize::Single, + "spec" => FixSize::Spec, + _ => bail!("the Security reproduction ended with an invalid fix size"), + }; + Outcome::Reproduced { severity, size } + } + _ => { + bail!("the Security reproduction ended without the final line its prompt asks for") + } + }; + let test = fs::read_to_string(test) + .context("could not read the Security reproduction's test file")?; + if notes.trim().is_empty() || test.trim().is_empty() { + bail!("the Security reproduction ended without reproduction notes or test text"); + } + Ok(Reproduction { + outcome, + notes: notes.trim().to_string(), + test, + }) + }) +} diff --git a/src/security/review.rs b/src/security/review.rs new file mode 100644 index 00000000..03ee441d --- /dev/null +++ b/src/security/review.rs @@ -0,0 +1,240 @@ +//! The guidance review's final-line contract. Only introduced findings may +//! appear here; pre-existing vulnerability details stay out of public text. + +use anyhow::{Context, Result, bail}; +use serde::{Deserialize, Serialize}; +use std::{fmt, fs}; + +use super::reproduction::{FixSize, Outcome as ReproductionOutcome, Reproduction, Severity}; +use crate::github::{DraftAdvisory, GitHub}; +use crate::harness::interpretation::SafeguardRefusal; +use crate::issue::IssueUrl; +use crate::session::{Purpose, Sessions}; +use crate::worktree::Worktree; +use crate::{logs, progress, prompt}; + +/// A fresh report outside the worktree, retained even when the session or +/// private recording fails. Only the introduced titles reach Delivery. +pub(crate) fn run( + sessions: &Sessions<'_>, + worktree: &Worktree, + issue: &IssueUrl, + base: &str, + text: &str, +) -> Result<Review> { + let merge_base = worktree.merge_base_commit(base)?; + let root = logs::root(&issue.repo()).join("security-reviews"); + fs::create_dir_all(&root)?; + let directory = tempfile::Builder::new() + .prefix(&format!("{}-{}-", issue.number, logs::stamp())) + .tempdir_in(root)? + .keep(); + let report = directory.join("pre-existing.json"); + let text = text + .replace( + prompt::SECURITY_REVIEW_REPORT_FILE, + &report.to_string_lossy(), + ) + .replace(prompt::SECURITY_REVIEW_MERGE_BASE, &merge_base); + progress::step(format!( + "Security review private report: {}", + report.display() + )); + let message = + sessions.run_to_final_message(Purpose::SecurityReview, "security-review", &text)?; + // A session may change commits on the Issue branch, but it cannot hand + // us a replacement checkout to observe or publish findings from. + worktree.head()?; + let review = Review::from_final_message(message.as_deref())?; + let findings: Vec<OldFinding> = serde_json::from_slice(&fs::read(report)?)?; + if findings.len() != review.pre_existing_count { + bail!("Security review's private report does not match its final count"); + } + // Validate the entire report before creating any record. Malformed report + // diagnostics are never passed into the public outcome. + let findings = findings + .into_iter() + .map(|finding| { + let reproduction = finding.reproduction()?; + Ok((finding, reproduction)) + }) + .collect::<Result<Vec<_>>>()?; + if !findings.is_empty() { + let github = GitHub::new(); + let repo = issue.repo_slug(); + let mut known = github.security_records(&repo)?; + let package = super::audit::package_in(worktree.path()); + for (finding, reproduction) in findings { + let draft = DraftAdvisory { + fingerprint: finding.fingerprint.clone(), + summary: finding.title.clone(), + description: format!( + "Found by thirdshift's Security review.\n\nFingerprint: `{}`\nAudited commit: `{merge_base}`\nReview issue: {}\n\n{}\n\n```json\n{}\n```\n", + finding.fingerprint, + issue.url, + finding.description, + serde_json::to_string_pretty(&finding)? + ), + package: package.clone(), + }; + // Reuse every record state and preserve the Day shift's grade. + if known.finding(&draft).is_none() { + let record = github.create_security_record(&repo, &known, &draft)?; + github.update_security_record(&repo, &known.record(&record)?, &reproduction)?; + known.remember(record); + progress::step("recorded a pre-existing Security finding privately"); + } + } + } + Ok(review) +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct OldFinding { + fingerprint: String, + title: String, + description: String, + proof_of_concept: ProofOfConcept, +} + +#[derive(Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct ProofOfConcept { + test: String, + command: String, + head_exit_code: i32, + merge_base_exit_code: i32, + notes: String, + severity: String, + fix_size: String, +} + +impl OldFinding { + fn reproduction(&self) -> Result<Reproduction> { + let proof = &self.proof_of_concept; + if [ + &self.fingerprint, + &self.title, + &self.description, + &proof.test, + &proof.command, + &proof.notes, + ] + .iter() + .any(|text| text.trim().is_empty()) + || self.fingerprint.contains(['\r', '\n', '`']) + || self.title.contains(['\r', '\n']) + || proof.head_exit_code <= 0 + || proof.merge_base_exit_code <= 0 + { + bail!( + "Security review's private report lacks a valid old finding and failing proof-of-concept runs" + ); + } + let severity = Severity::parse(&proof.severity) + .context("Security review's private report has an invalid reproduced severity")?; + let size = match proof.fix_size.as_str() { + "single" => FixSize::Single, + "spec" => FixSize::Spec, + _ => bail!("Security review's private report has an invalid reproduced fix size"), + }; + Ok(Reproduction { + outcome: ReproductionOutcome::Reproduced { severity, size }, + notes: format!( + "Command: {}\nHEAD exit code: {}\nMerge base exit code: {}\n\n{}", + proof.command, proof.head_exit_code, proof.merge_base_exit_code, proof.notes + ), + test: proof.test.clone(), + }) + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Ask { + Allow, + Forbid, +} + +pub(crate) enum Outcome { + Complete(Review), + Incomplete(String), +} + +impl Outcome { + pub fn of(result: Result<Review>) -> Self { + match result { + Ok(review) => Self::Complete(review), + Err(error) => Self::Incomplete(match error.downcast_ref::<SafeguardRefusal>() { + Some(refusal) => format!("Security review refused: {refusal}"), + None => "Security review incomplete: session failed, ended early or omitted a valid final line; see Session log".to_string(), + }), + } + } + + pub fn hold(&self) -> Option<Hold> { + match self { + Self::Complete(review) if review.findings.is_empty() => None, + Self::Complete(review) => Some(Hold(format!( + "Security review left {} unaddressed introduced finding(s): {}", + review.findings.len(), + review.findings.join("; ") + ))), + Self::Incomplete(reason) => Some(Hold(reason.clone())), + } + } + + pub fn entry(&self) -> String { + match self { + Self::Complete(review) if review.findings.is_empty() => { + "No unaddressed introduced findings.".to_string() + } + Self::Complete(review) => review + .findings + .iter() + .map(|title| format!("- {title}\n")) + .collect(), + Self::Incomplete(reason) => format!("- {reason}\n"), + } + } +} + +/// Delivery has pushed and marked the PR ready before raising this hold. +#[derive(Debug)] +pub(crate) struct Hold(pub String); + +impl fmt::Display for Hold { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(&self.0) + } +} + +impl std::error::Error for Hold {} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct Review { + unaddressed_count: usize, + pub findings: Vec<String>, + pre_existing_count: usize, +} + +impl Review { + pub fn from_final_message(message: Option<&str>) -> Result<Self> { + let json = message + .and_then(|message| message.lines().rfind(|line| !line.trim().is_empty())) + .and_then(|line| line.strip_prefix("Security review: ")) + .context("Security review ended without its required final line")?; + let review: Self = serde_json::from_str(json) + .context("Security review ended with an invalid final line")?; + if review.unaddressed_count != review.findings.len() + || review + .findings + .iter() + .any(|title| title.trim().is_empty() || title.contains(['\r', '\n'])) + { + bail!("Security review ended with an inconsistent finding count or title"); + } + Ok(review) + } +} diff --git a/src/session.rs b/src/session.rs index 88430bde..d69c7bab 100644 --- a/src/session.rs +++ b/src/session.rs @@ -9,7 +9,7 @@ use std::time::{Duration, Instant}; #[cfg(test)] use anyhow::bail; -use anyhow::{Context, Result}; +use anyhow::{Context, Result, anyhow}; #[cfg(test)] pub(crate) use crate::harness::claude::claude_args; @@ -29,6 +29,16 @@ use review_reports::ReviewReports; #[cfg(test)] mod execution_tests; +/// The session's launch policy, retained by its Resume independently of +/// the label used for logs and progress. +#[derive(Clone, Copy)] +pub enum Purpose { + Ordinary, + Security, + /// A guidance review retains security checks without delegating auditors. + SecurityReview, +} + /// Where a Run's or an Architect run's Session logs go: in `sessions/` under /// the root of its repository's logs, under the User config's `logs.dir` or /// `~/.thirdshift/logs`, each named for what is run and stamped with the @@ -58,6 +68,14 @@ impl Logs { } } + /// The logs of the Security run on `repo`. + pub fn of_security_run(repo: &Repo) -> Self { + Logs { + name: "secure".to_string(), + dir: logs::root(repo).join("sessions"), + } + } + /// Where a session's stream is logged: /// `<root>/sessions/<name>-<stamp>-<kind>.jsonl`. pub fn path(&self, kind: &str) -> PathBuf { @@ -149,13 +167,20 @@ impl<'a> Sessions<'a> { /// abandoned rather than awaited: a progress line names it and this /// succeeds, leaving the steps that follow to decide the outcome. pub fn run(&self, kind: &str, prompt: &str) -> Result<()> { - self.run_to_final_message(kind, prompt).map(drop) + self.run_to_final_message(Purpose::Ordinary, kind, prompt) + .map(drop) } /// Run a session as [`Sessions::run`] does, and return its final message: /// what the agent said as it ended its last turn, its Resume's if it got - /// one, if anything. - pub fn run_to_final_message(&self, kind: &str, prompt: &str) -> Result<Option<String>> { + /// one, if anything. `purpose` selects the launch policy for both the + /// session and its Resume; `kind` only labels their logs and progress. + pub fn run_to_final_message( + &self, + purpose: Purpose, + kind: &str, + prompt: &str, + ) -> Result<Option<String>> { let reports = if prompt.contains(prompt::REVIEW_REPORTS_DIRECTORY) { match ReviewReports::new(self.worktree) { Ok(reports) => Some(reports), @@ -169,7 +194,7 @@ impl<'a> Sessions<'a> { } else { None }; - let mut ended = self.start(kind, None, prompt, reports.as_ref())?; + let mut ended = self.start(purpose, kind, None, prompt, reports.as_ref())?; // What ended last, as the progress line on its killed work calls it. let mut ended_last = "session"; if let (false, Some(session_id)) = (ended.killed.is_empty(), &ended.session_id) { @@ -181,6 +206,7 @@ impl<'a> Sessions<'a> { "{kind}: background work was killed as the session ended; resuming it once" )); ended = self.start( + purpose, &format!("{kind}-resume"), Some(session_id), &resume_prompt, @@ -190,6 +216,11 @@ impl<'a> Sessions<'a> { } if !ended.killed.is_empty() { let ending = ending_with(&ended.killed_work()); + if matches!(purpose, Purpose::Security | Purpose::SecurityReview) { + return Err(anyhow!( + "{kind}: the {ended_last} {ending}; Security session incomplete" + )); + } self.step(format!( "{kind}: the {ended_last} {ending}; carrying on, as it may have been abandoned" )); @@ -204,6 +235,7 @@ impl<'a> Sessions<'a> { /// the last log. fn start( &self, + purpose: Purpose, kind: &str, resume: Option<&str>, prompt: &str, @@ -219,7 +251,7 @@ impl<'a> Sessions<'a> { let ended = self .outside .borrow_mut() - .run_session(kind, resume, &prompt, &log); + .run_session(purpose, kind, resume, &prompt, &log); if let Some(reports) = reports { for line in reports.keep(&log) { self.step(format!("{kind}: {line}")); @@ -241,6 +273,7 @@ trait Outside { /// what it ended with once it has exited cleanly, its turn not failed. fn run_session( &mut self, + purpose: Purpose, kind: &str, resume: Option<&str>, prompt: &str, @@ -261,13 +294,28 @@ struct OnMachine { impl Outside for OnMachine { fn run_session( &mut self, + purpose: Purpose, kind: &str, resume: Option<&str>, prompt: &str, log: &Path, ) -> Result<Ended> { - let invocation = self.adapter.session(&self.harness, resume, prompt); let interpretation = self.adapter.interpretation(&self.worktree, prompt); + let (invocation, interpretation) = match purpose { + Purpose::Security | Purpose::SecurityReview => ( + self.adapter.security_session( + &self.harness, + resume, + prompt, + matches!(purpose, Purpose::Security), + ), + interpretation.for_security(self.harness.model.as_deref()), + ), + Purpose::Ordinary => ( + self.adapter.session(&self.harness, resume, prompt), + interpretation, + ), + }; run( kind, self.adapter, @@ -330,6 +378,9 @@ fn run( let completion = executed.state.finish(executed.execution); if let Some(report) = completion.report { + for model in report.models { + progress::step(format_args!("{kind}: Model: {model}")); + } for warning in report.warnings { progress::step(format_args!("{kind}: {warning}")); } @@ -415,6 +466,7 @@ mod tests { impl Outside for Scripted { fn run_session( &mut self, + _purpose: Purpose, kind: &str, resume: Option<&str>, prompt: &str, @@ -504,7 +556,7 @@ mod tests { #[test] fn a_clean_session_runs_once_and_returns_its_final_message() { let (taken, log, calls) = take(vec![ended(Some("s1"), &[], "done")], |sessions| { - sessions.run_to_final_message("implement", "do it") + sessions.run_to_final_message(Purpose::Ordinary, "implement", "do it") }); assert_eq!(taken.unwrap(), Some("done".to_string())); @@ -522,7 +574,7 @@ mod tests { ended(Some("s1"), &["cargo test"], "waiting"), ended(Some("s1"), &[], "done"), ], - |sessions| sessions.run_to_final_message("implement", "do it"), + |sessions| sessions.run_to_final_message(Purpose::Ordinary, "implement", "do it"), ); assert_eq!(taken.unwrap(), Some("done".to_string())); @@ -546,6 +598,31 @@ mod tests { assert_eq!(log, Some(logs().path("implement-resume"))); } + #[test] + fn a_security_session_cannot_claim_completion_with_killed_work() { + for id in [None, Some("s1")] { + let mut endings = vec![ended( + id, + &["cargo test"], + "Security review: {\"unaddressed_count\":0,\"findings\":[]}", + )]; + if id.is_some() { + endings.push(ended( + id, + &["cargo test"], + "Security review: {\"unaddressed_count\":0,\"findings\":[]}", + )); + } + let (taken, _, _) = take(endings, |sessions| { + sessions.run_to_final_message(Purpose::Security, "security-review", "review") + }); + assert!( + taken.is_err(), + "unfinished security work must hold Self-merge" + ); + } + } + #[test] fn a_resume_that_ends_the_same_way_gets_no_second_and_a_later_failure_names_the_work() { let (taken, log, calls) = take( @@ -555,7 +632,9 @@ mod tests { ], |sessions| -> Result<()> { assert_eq!( - sessions.run_to_final_message("implement", "do it").unwrap(), + sessions + .run_to_final_message(Purpose::Ordinary, "implement", "do it") + .unwrap(), Some("still waiting".to_string()) ); bail!("no PR found") diff --git a/src/session/execution_tests.rs b/src/session/execution_tests.rs index 36cc0192..f0fc8668 100644 --- a/src/session/execution_tests.rs +++ b/src/session/execution_tests.rs @@ -113,7 +113,7 @@ fn execute_session(root: &Path) -> anyhow::Error { }; let prompt = format!("/thirdshift-implement {}", "p".repeat(1024 * 1024)); let (result, log) = Sessions::within(&logs, &worktree, &choice, |sessions| { - sessions.run_to_final_message("implement", &prompt) + sessions.run_to_final_message(Purpose::Ordinary, "implement", &prompt) }); assert!(log.unwrap().exists()); result.unwrap_err() diff --git a/src/setup.rs b/src/setup.rs index 1819bb0e..96d61d4c 100644 --- a/src/setup.rs +++ b/src/setup.rs @@ -719,6 +719,8 @@ mod tests { const MERGE: &str = "Every Run a Merge run?"; const BASE_FIX: &str = "Every Run may start a Base fix when the Base branch's CI is red?"; const PULL: &str = "fast-forward"; + const SECURITY_REVIEW: &str = "Runs review their change for Security findings?"; + const SECURITY_FIX: &str = "Security runs may fix reproduced findings?"; const NOTIFY: &str = "Run notifications, an email"; const TO: &str = "Send Run notifications to"; const FROM: &str = "Send them from"; @@ -733,7 +735,13 @@ mod tests { const KEY: &str = "re_secret_123"; /// The answers that take every default and turn nothing on. - const ENTER_THROUGHOUT: [(&str, &str); 3] = [(MERGE, ""), (PULL, ""), (NOTIFY, "")]; + const ENTER_THROUGHOUT: [(&str, &str); 5] = [ + (MERGE, ""), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]; /// The answers that turn Run notifications on, to `me@example.com` from /// the default sender, then `rest`. @@ -743,6 +751,8 @@ mod tests { let mut answers = vec![ (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, ""), @@ -1044,6 +1054,8 @@ to = \"me@example.com\" # my inbox [ "Every Run a Merge run? [y/N] ", "Every Run first fast-forwards your checkout of the Base branch? [y/N] ", + "Security runs may fix reproduced findings? [y/N] ", + "Runs review their change for Security findings? [y/N] ", "Run notifications, an email as each Run ends? [y/N] ", ] ); @@ -1057,6 +1069,8 @@ to = \"me@example.com\" # my inbox (MERGE, "y"), (BASE_FIX, "y"), (PULL, "yes"), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, "ts@acme.dev"), @@ -1126,6 +1140,11 @@ parallel = 5 limit = 5 wait_minutes = 60 +[security] +harness = \"codex\" +fix = false +review = false + [harness] default = \"claude\" @@ -1161,6 +1180,8 @@ effort = \"\" (MERGE, ""), (BASE_FIX, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, ""), (TO, ""), (FROM, ""), @@ -1181,6 +1202,8 @@ effort = \"\" "Every Run a Merge run? [Y/n] ", "Every Run may start a Base fix when the Base branch's CI is red? [Y/n] ", "Every Run first fast-forwards your checkout of the Base branch? [Y/n] ", + "Security runs may fix reproduced findings? [y/N] ", + "Runs review their change for Security findings? [y/N] ", "Run notifications, an email as each Run ends? [Y/n] ", "Send Run notifications to [mine@example.net]: ", "Send them from [ts@acme.dev]: ", @@ -1193,7 +1216,13 @@ effort = \"\" fn on_a_terminal_changed_answers_are_written_over_the_user_config() { let mut outside = Scripted { user_config: Some(DEFAULTS.to_string()), - ..Scripted::answering(&[(MERGE, ""), (PULL, "y"), (NOTIFY, "")]) + ..Scripted::answering(&[ + (MERGE, ""), + (PULL, "y"), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]) }; let done = setup(&mut outside).unwrap(); @@ -1224,6 +1253,8 @@ always = false # quiet, please (MERGE, "y"), (BASE_FIX, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, ""), @@ -1325,8 +1356,14 @@ always = false # quiet, please #[test] fn with_merging_on_setup_asks_about_base_fixes_and_writes_the_answer() { for (answer, allowed) in [("y", true), ("", false), ("n", false)] { - let mut outside = - Scripted::answering(&[(MERGE, "y"), (BASE_FIX, answer), (PULL, ""), (NOTIFY, "")]); + let mut outside = Scripted::answering(&[ + (MERGE, "y"), + (BASE_FIX, answer), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]); setup(&mut outside).unwrap(); @@ -1346,7 +1383,13 @@ always = false # quiet, please #[test] fn with_merging_off_setup_asks_nothing_about_base_fixes_and_writes_the_default() { for answer in ["", "n"] { - let mut outside = Scripted::answering(&[(MERGE, answer), (PULL, ""), (NOTIFY, "")]); + let mut outside = Scripted::answering(&[ + (MERGE, answer), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]); setup(&mut outside).unwrap(); @@ -1360,7 +1403,14 @@ always = false # quiet, please fn the_base_fix_question_defaults_to_the_user_configs_base_fix() { let mut outside = Scripted { user_config: Some("[merge]\nalways = true\n\n[base]\nfix = true # mine\n".to_string()), - ..Scripted::answering(&[(MERGE, ""), (BASE_FIX, ""), (PULL, ""), (NOTIFY, "")]) + ..Scripted::answering(&[ + (MERGE, ""), + (BASE_FIX, ""), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]) }; setup(&mut outside).unwrap(); @@ -1380,7 +1430,13 @@ always = false # quiet, please fn turning_merging_off_writes_base_fix_at_its_default() { let mut outside = Scripted { user_config: Some("[merge]\nalways = true\n\n[base]\nfix = true # mine\n".to_string()), - ..Scripted::answering(&[(MERGE, "n"), (PULL, ""), (NOTIFY, "")]) + ..Scripted::answering(&[ + (MERGE, "n"), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ]) }; setup(&mut outside).unwrap(); @@ -1397,6 +1453,8 @@ always = false # quiet, please (MERGE, "YES"), (BASE_FIX, "No"), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, ""), ]); @@ -1410,6 +1468,8 @@ always = false # quiet, please let mut outside = Scripted::answering(&[ (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, ""), (TO, "me.example.com"), @@ -1438,7 +1498,13 @@ always = false # quiet, please let mut outside = Scripted { key: Ok(Some(Source::Credentials(CREDENTIALS.into()))), github_email: Ok(Some("octo@example.com")), - ..Scripted::answering(&[(MERGE, ""), (PULL, ""), (NOTIFY, "n")]) + ..Scripted::answering(&[ + (MERGE, ""), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), + (NOTIFY, "n"), + ]) }; setup(&mut outside).unwrap(); @@ -1680,6 +1746,8 @@ always = false # quiet, please ..Scripted::answering(&[ (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, ""), (FROM, ""), @@ -1780,6 +1848,8 @@ always = false # quiet, please (MERGE, "y"), (BASE_FIX, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, ""), ]); @@ -1798,6 +1868,8 @@ always = false # quiet, please (OFFER, "y"), (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, ""), @@ -1895,6 +1967,8 @@ always = false # quiet, please (OFFER, "y"), (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, ""), diff --git a/src/setup/questions.rs b/src/setup/questions.rs index 33398e42..0fb5c21b 100644 --- a/src/setup/questions.rs +++ b/src/setup/questions.rs @@ -26,6 +26,9 @@ pub struct Answers { pub base_fix: bool, /// `launch.pull`. pub launch_pull: bool, + /// `security.fix`: Security runs may fix reproduced findings. + pub security_fix: bool, + pub security_review: bool, /// With Run notifications on, `email.always`, their settings; with them /// off, `None`, and the email settings stay as they were. pub notifications: Option<Notifications>, @@ -38,6 +41,8 @@ impl Answers { merge_always: self.merge_always, base_fix: self.base_fix, launch_pull: self.launch_pull, + security_fix: self.security_fix, + security_review: self.security_review, notifications: self .notifications .as_ref() @@ -118,6 +123,16 @@ pub fn ask( "Every Run first fast-forwards your checkout of the Base branch?", current.launch_pull, )?; + let security_fix = yes_or_no( + outside, + "Security runs may fix reproduced findings?", + current.security_fix.unwrap_or(false), + )?; + let security_review = yes_or_no( + outside, + "Runs review their change for Security findings?", + current.security_review, + )?; if !yes_or_no( outside, "Run notifications, an email as each Run ends?", @@ -128,6 +143,8 @@ pub fn ask( merge_always, base_fix, launch_pull, + security_fix, + security_review, notifications: None, }); } @@ -169,6 +186,8 @@ pub fn ask( merge_always, base_fix, launch_pull, + security_fix, + security_review, notifications: Some(Notifications { to, from, diff --git a/src/skills.rs b/src/skills.rs index 109c03ea..a186cbbb 100644 --- a/src/skills.rs +++ b/src/skills.rs @@ -61,7 +61,7 @@ pub fn remove_written() { /// Where the Command wrote the Factory skills out, writing them now if it /// has not yet. -fn written_out() -> Result<PathBuf> { +pub(crate) fn written_out() -> Result<PathBuf> { let mut written = WRITTEN.lock().unwrap_or_else(PoisonError::into_inner); if let Some(dir) = &*written { return Ok(dir.path().to_owned()); diff --git a/src/spec_run.rs b/src/spec_run.rs index bdf76671..f71db799 100644 --- a/src/spec_run.rs +++ b/src/spec_run.rs @@ -1,6 +1,6 @@ //! A Spec run: each of a Spec's Tickets, in dependency order and several at //! once, taken by a Ticket's Run, a Merge run into the Spec branch in a child `thirdshift` -//! (ADR-0006), then the Spec review and the Spec PR from the Spec branch into +//! (ADR-0006), then the Spec review, optional Security review and the Spec PR from the Spec branch into //! the Base branch, kept mergeable and green like a Run's PR, and Self-merged //! when the Spec run was asked to merge. //! @@ -195,6 +195,8 @@ struct ChildRunsAndGitHub<'a> { /// Held until it starts. delivery: Option<Delivery<'a>>, base_fix: BaseFixAsk, + security_fix: bool, + security_review: bool, harness: &'a Choice, spec_pr: PullRequest, } @@ -203,11 +205,14 @@ impl<'a> ChildRunsAndGitHub<'a> { /// The outside world of a Spec run, as the struct says, with no Ticket's /// Run started yet. fn new(worktree: Worktree, delivery: Delivery<'a>, spec_pr: PullRequest) -> Self { + let security_review = delivery.security_review; Self { spec: delivery.issue, children: Runs::default(), worktree: Some(worktree), base_fix: delivery.base_fix.ask_of_tickets(), + security_fix: delivery.security_fix, + security_review, harness: delivery.harness, delivery: Some(delivery), spec_pr, @@ -232,8 +237,16 @@ impl Outside for ChildRunsAndGitHub<'_> { spec_branch: self.worktree().branch().to_string(), }; let ticket = self.spec.sibling(number); - self.children - .start(&ticket, kind, self.base_fix.clone(), self.harness) + self.children.start( + &ticket, + kind, + self.base_fix.clone(), + crate::security::Options { + fix: self.security_fix, + review: self.security_review, + }, + self.harness, + ) } fn next_ending(&mut self) -> (u64, Result<Ended>) { diff --git a/src/worktree.rs b/src/worktree.rs index 3ddb00c7..610face7 100644 --- a/src/worktree.rs +++ b/src/worktree.rs @@ -204,6 +204,14 @@ impl Worktree { .ordinary(&self.launch, |operation| operation.head()) } + /// Observe the merge base against the published Base branch only while + /// this is still the acquired checkout on the selected Issue branch. + pub fn merge_base_commit(&self, base: &str) -> Result<String> { + self.checkout.ordinary(&self.launch, |operation| { + operation.run(&["merge-base", &format!("refs/remotes/origin/{base}"), "HEAD"]) + }) + } + /// Fetch and sample `origin/<base>`, then merge that commit into the /// Issue branch. A clean outcome includes the selected commit for CI /// comparison, including when the merge had nothing to do. @@ -356,6 +364,23 @@ pub struct ReviewWorktree { } impl ReviewWorktree { + /// Reproduce one finding at its audited commit, without fetching a newer + /// Base branch. Uses the review checkout's acquisition and disposal rules. + pub fn at_commit(launch: &Git, repo: &str, commit: &str) -> Result<Self> { + let _lock = lock_launch(launch)?; + let (root, path) = sibling(launch, &format!("{repo}-security-reproduce"))?; + ownership::recover_review(launch, &path)?; + progress::step(format_args!( + "creating worktree {} detached at {commit}", + path.display() + )); + let checkout = acquisition::add(launch, None, &path, commit)?; + Ok(Self { + launch: Git::new(root), + checkout, + }) + } + /// Check out `origin/<base>`, detached, in a new worktree next to the /// launch repository's root, named `<repo>-architect`. A worktree left /// there by a process that ended before cleanup is removed only with diff --git a/tests/activity_log.rs b/tests/activity_log.rs index 68d22aa1..6ab151d6 100644 --- a/tests/activity_log.rs +++ b/tests/activity_log.rs @@ -1,9 +1,9 @@ //! The Activity log: `<logs.dir>/<owner>/<repo>/activity.log`, one running //! record per repository of what the factory did there. A Run, a Spec run, -//! an Architect run or a Pickup run writes a line when it starts work, +//! or a Pass writes a line when it starts work, //! naming its Command log, and one when it ends, with its outcome; the run a -//! command dispatched or started writes none of its own. A skipped Pickup -//! run or Architect run writes a line only when its reason differs from the +//! command dispatched or started writes none of its own. A skipped Pass +//! writes a line only when its reason differs from the //! last line of its own kind. With `activity.quiet_skips`, a skipped pass //! prints nothing at all, its Activity log line its only trace. @@ -241,7 +241,7 @@ while [ -d "$root" ] && [ ! -e "$root/release" ]; do sleep 0.05; done assert_eq!(result.code, Some(0), "stderr: {}", result.stderr); } let lines = activity(&scenario); - let running = "skipped: an Architect run or a Pickup run is already running on acme/widgets"; + let running = "skipped: another Pass is already running on acme/widgets"; assert_eq!(lines.len(), 4, "{lines:?}"); assert!(lines[0].starts_with("Pickup run #7 started: "), "{lines:?}"); assert_eq!(lines[1], format!("Pickup run {running}")); diff --git a/tests/architect_run.rs b/tests/architect_run.rs index 36cf1e62..c6b64ceb 100644 --- a/tests/architect_run.rs +++ b/tests/architect_run.rs @@ -1716,8 +1716,7 @@ fn a_dispatched_spec_run_that_fails_sends_one_notification_with_each_tickets_out /// What a skipped Architect run says when another is running on the /// scenario's repository. -const ALREADY_RUNNING: &str = - "thirdshift: an Architect run or a Pickup run is already running on acme/widgets\n"; +const ALREADY_RUNNING: &str = "thirdshift: another Pass is already running on acme/widgets\n"; /// Assert the Architect run was skipped as one is already running: exit 0, /// that line alone on stderr, and nothing on stdout. diff --git a/tests/codex_sessions.rs b/tests/codex_sessions.rs index 16f5e5ee..82e87072 100644 --- a/tests/codex_sessions.rs +++ b/tests/codex_sessions.rs @@ -479,6 +479,28 @@ fn a_merge_run_on_codex_merges_the_pr() { assert!(scenario.claude_calls().is_empty()); } +#[test] +fn ordinary_runs_ignore_the_security_harness_and_keep_codexs_ordinary_session_protocol() { + let scenario = Scenario::new(); + scenario.user_config_is( + "[security]\nharness = \"claude\"\n\ + [harness]\ndefault = \"codex\"\n\ + [harness.codex]\nmodel = \"gpt-6-luna\"\neffort = \"high\"\n", + ); + scenario.agent_does_for(7, AGENT_OPENS_PR); + let result = scenario.run(&[&scenario.issue_url(7)]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_sessions_on_codex(&scenario, "gpt-6-luna", "high"); + for call in scenario.codex_calls() { + assert!( + !argv(&call) + .iter() + .any(|arg| arg.contains("max_concurrent_threads")) + ); + assert!(!call["prompt"].as_str().unwrap().contains("fork_turns")); + } +} + /// Assert the Run failed with `error` before any work: no Claim, no /// worktree, no session and no Command log. fn assert_failed_before_any_work(scenario: &Scenario, result: &RunResult, error: &str) { diff --git a/tests/command_surface.rs b/tests/command_surface.rs index d3bb2d14..7a688fcf 100644 --- a/tests/command_surface.rs +++ b/tests/command_surface.rs @@ -17,6 +17,8 @@ fn assert_help_text(text: &str) { "thirdshift architect base <branch> [<focus>]", "thirdshift pickup", "thirdshift pickup base <branch>", + "thirdshift secure", + "thirdshift secure base <branch>", "thirdshift email-test [<address>]", "thirdshift setup", "thirdshift update", @@ -99,8 +101,8 @@ fn help_documents_architect_its_focus_the_dispatch_plan_only_and_when_an_archite "Any triage decision lets the next Architect run go ahead", "Last, an Architect run is skipped while the repository has a Ready issue, by the search a Pickup run makes", "one with no Pickup run to take it keeps Architect runs from starting", - "Only one Architect run or Pickup run per repository runs at a time on a machine", - "is skipped: it prints an Architect run or a Pickup run is already running on <owner>/<repo>, does nothing else and exits 0", + "Only one Pass per repository runs at a time on a machine", + "is skipped: it prints another Pass is already running on <owner>/<repo>, does nothing else and exits 0", "A skipped run sends none, even when asked", "swaps its needs-triage label for ready-for-agent, labels it architect-plan", "creating the label if the repository lacks it", @@ -202,7 +204,7 @@ fn help_documents_base_for_an_architect_run() { "<branch> must exist on origin, with no local copy of it ahead", "launch.pull updates the clone only when <branch> is the branch checked out", "base goes with --plan-only too", - "base is for architect and pickup only", + "base is for a Pass only", "Without base, the Base branch is the branch checked out", "thirdshift architect base main", ] { @@ -222,7 +224,7 @@ fn help_documents_pickup_the_ready_issue_the_dispatch_the_skips_and_the_run_noti "A Ready issue is an open issue labelled ready-for-agent", "none of ready-for-human, needs-info, wontfix and needs-triage", "is not in-progress", - "is not a sub-issue, is not labelled base-fix, has no open blocker", + "is not a sub-issue, is not labelled base-fix or security-fix, has no open blocker", "was never started: no Issue branch for it is on origin, and no pull request from one exists, open, merged or closed", "A Spec whose Tickets are all closed is not one either: a Spec run would find nothing to do", "It must also be settled: by default, thirty minutes have passed since ready-for-agent was applied to it, and since a sub-issue or a \"blocked by\" link of its was last added or removed", @@ -238,7 +240,7 @@ fn help_documents_pickup_the_ready_issue_the_dispatch_the_skips_and_the_run_noti "pickup takes nothing else: no focus and no --plan-only", "A Pickup run is skipped, exiting 0 with nothing on stdout and one line on stderr saying why, after any lines on issues it passed over", "when the repository has no Ready issue", - "while an Architect run or another Pickup run on the same repository is still running on this machine", + "while another Pass on the same repository is still running on this machine", "--email, --email <address> and --no-email ask a Pickup run for its Run notification as they do a Run, and email.always sets the default", "A pass that took an issue sends one: the notification the run it dispatched would send by hand, with that run's subject, outcome and body", "The dispatched run sends none of its own", diff --git a/tests/fakes/claude.rs b/tests/fakes/claude.rs index 169dafff..6312c045 100644 --- a/tests/fakes/claude.rs +++ b/tests/fakes/claude.rs @@ -226,6 +226,7 @@ pub fn main(argv: Vec<String>) { let after_result = format!("{}.after-result.{session}", record_path.display()); let final_message = format!("{}.final-message.{session}", record_path.display()); let status = bash() + .env("FAKE_CLAUDE_PROMPT", argv.last().unwrap()) .env("FAKE_CLAUDE_AFTER_RESULT", &after_result) .env("FAKE_CLAUDE_FINAL_MESSAGE", &final_message) .status() diff --git a/tests/fakes/codex.rs b/tests/fakes/codex.rs index e2d10b01..5df55418 100644 --- a/tests/fakes/codex.rs +++ b/tests/fakes/codex.rs @@ -106,6 +106,7 @@ pub fn main(argv: Vec<String>) { let status = Command::new("bash") .arg("-e") .arg(&script) + .env("FAKE_CLAUDE_PROMPT", argv.last().unwrap()) .env("FAKE_CLAUDE_FINAL_MESSAGE", &final_message) .env("FAKE_CODEX_ERROR", &error) .status() diff --git a/tests/fakes/gh.rs b/tests/fakes/gh.rs index 4afbe8c4..80032ba9 100644 --- a/tests/fakes/gh.rs +++ b/tests/fakes/gh.rs @@ -4,6 +4,8 @@ //! //! ```text //! {"repo": "owner/repo", +//! "private"?: true, +//! "advisory_error"?: "<stderr>", //! "issues": {"<number>": "OPEN" | "CLOSED"}, //! "titles"?: {"<number>": "<title>"}, //! "created"?: {"<number>": "<ISO 8601 time>"}, @@ -24,6 +26,7 @@ //! "refuse_merges"?: {"times", "error"}, //! "after_merge"?: "<script>", //! "failing"?: ["<command> <subcommand>", ...], +//! "advisory_create_fails_after"?: <number of records>, //! "user_email"?: "<address>" | null, //! "sub_issues"?: {"<number>": [<number>, ...]}, //! "labels"?: {"<number>": ["<label>", ...]}, @@ -651,6 +654,14 @@ fn issue_fields(state: &Json, n: &str, issue_state: &Json) -> Json { ("url", string(url)), ("labels", Array(labels)), ("createdAt", created), + ( + "body", + state + .get("bodies") + .and_then(|bodies| bodies.get(n)) + .cloned() + .unwrap_or_else(|| string("")), + ), ]) } @@ -808,6 +819,7 @@ fn issue_close(state: &mut Json, positional: &[String], flags: &Flags) { /// Open an issue with `--title`, `--body` and the comma-separated `--label`s, /// numbered one past the highest issue, and print its URL. fn issue_create(state: &mut Json, flags: &Flags) { + use std::io::Read; let labels: Vec<&str> = flag(flags, "label") .map(|labels| labels.split(',').collect()) .unwrap_or_default(); @@ -827,7 +839,13 @@ fn issue_create(state: &mut Json, flags: &Flags) { state.at_mut("issues").set(&n, string("OPEN")); let title = flag(flags, "title").expect("no --title"); state.entry("titles", object([])).set(&n, string(title)); - let body = flag(flags, "body").expect("no --body"); + let mut input = String::new(); + let body = if flag(flags, "body-file") == Some("-") { + std::io::stdin().read_to_string(&mut input).unwrap(); + &input + } else { + flag(flags, "body").expect("no --body") + }; state.entry("bodies", object([])).set(&n, string(body)); let labels = labels.into_iter().map(string).collect(); state.entry("labels", object([])).set(&n, Array(labels)); @@ -943,8 +961,174 @@ fn repo_prefix(path: &str) -> Option<(&str, &str)> { /// per run with `--jq` set to `WORKFLOW_RUN_LINES`, and for the PRs one /// `<merge commit> <number>` line per PR with `--jq` set to /// `COMMIT_PR_LINES`. +/// Advisory create/list/update endpoints, backed by private JSON state. +fn advisory_api(state: &mut Json, positional: &[String], flags: &Flags) { + use std::io::Read; + let (path, query) = positional[0] + .split_once('?') + .unwrap_or((&positional[0], "")); + let (repo, rest) = repo_prefix(path).unwrap(); + check_repo_is(state, Some(repo)); + if let Some(error) = state.get("advisory_error") { + die(error.str(), 1); + } + if state.get("private").is_some_and(Json::truthy) { + die("gh: Not Found (HTTP 404)", 1); + } + let method = flag(flags, "method").unwrap_or("GET"); + if method == "GET" { + if let Some(id) = rest.strip_prefix("security-advisories/") { + let advisory = state + .at("advisories") + .items() + .iter() + .find(|advisory| advisory.at("ghsa_id").as_str() == Some(id)) + .unwrap(); + println!("{advisory}"); + return; + } + let wanted = query + .split('&') + .find_map(|pair| pair.strip_prefix("state=")); + let advisories = state.get("advisories").map(Json::items).unwrap_or_default(); + let listed: Vec<Json> = advisories + .iter() + .filter(|record| { + wanted.is_none_or(|wanted| record.at("state").as_str() == Some(wanted)) + }) + .cloned() + .collect(); + if flag(flags, "jq") == Some(".[]") { + for record in listed { + println!("{record}"); + } + } else { + println!("{}", Array(listed)); + } + return; + } + let mut input = String::new(); + let mut body = if flag(flags, "input") == Some("-") { + std::io::stdin().read_to_string(&mut input).unwrap(); + parse_json(&input) + } else { + object([]) + }; + if method == "POST" + && state + .get("advisory_create_fails_after") + .and_then(Json::as_i64) + .is_some_and(|limit| { + state + .get("advisories") + .map_or(0, |records| records.items().len()) + >= limit as usize + }) + { + die("HTTP 422: Private candidate write-up.", 1); + } + let advisories = state.entry("advisories", Array(Vec::new())).items_mut(); + if method == "POST" && rest == "security-advisories" { + if !body.has("summary") || !body.has("description") || !body.has("vulnerabilities") { + die("HTTP 422: required advisory fields missing", 1); + } + let id = format!("GHSA-test-test-{:04}", advisories.len() + 1); + body.set("ghsa_id", string(&id)); + body.set( + "html_url", + string(format!( + "https://github.com/{repo}/security/advisories/{id}" + )), + ); + body.set("state", string("draft")); + advisories.push(body.clone()); + } else if method == "PATCH" { + let id = rest.strip_prefix("security-advisories/").unwrap(); + let advisory = advisories + .iter_mut() + .find(|advisory| advisory.at("ghsa_id").as_str() == Some(id)) + .unwrap(); + let Json::Object(fields) = body else { + panic!("advisory update must be an object"); + }; + for (key, value) in fields { + advisory.set(&key, value); + } + body = advisory.clone(); + } else { + die("fake gh: unsupported advisory operation", 2); + } + save(state); + println!("{body}"); +} + +/// Update a finding issue's body through the private-record operation. +fn finding_issue_patch(state: &mut Json, rest: &[&str]) { + use std::io::Read; + let (positional, flags) = parse(rest); + let (repo, path) = repo_prefix(&positional[0]).unwrap(); + check_repo_is(state, Some(repo)); + let n = path.strip_prefix("issues/").unwrap(); + assert_eq!(flag(&flags, "input"), Some("-")); + let mut input = String::new(); + std::io::stdin().read_to_string(&mut input).unwrap(); + let body = parse_json(&input); + let issue_state = state.at("issues").at(n).clone(); + state + .entry("bodies", object([])) + .set(n, body.at("body").clone()); + save(state); + println!("{}", issue_fields(state, n, &issue_state)); +} + fn api(state: &mut Json, positional: &[String], flags: &Flags) { let path = &positional[0]; + if let Some((repo, rest)) = repo_prefix(path) + && let Some(n) = rest.strip_prefix("issues/") + && n.parse::<u64>().is_ok() + { + check_repo_is(state, Some(repo)); + println!("{}", issue_fields(state, n, state.at("issues").at(n))); + return; + } + if path == &format!("repos/{}", state.at("repo").str()) { + println!( + "{}", + object([( + "private", + Bool(state.get("private").is_some_and(Json::truthy)) + )]) + ); + return; + } + if let Some((repo, rest)) = repo_prefix(path) + && rest.starts_with("issues?") + { + check_repo_is(state, Some(repo)); + assert!(rest.contains("state=all")); + assert!(rest.contains("labels=security-finding")); + assert!(rest.contains("per_page=100")); + assert!(flags.contains_key("paginate")); + assert_eq!(flag(flags, "jq"), Some(".[]")); + let Json::Object(issues) = state.at("issues") else { + panic!("issues is an object") + }; + for (n, issue_state) in issues { + if issue_labels(state, n) + .iter() + .any(|label| label.str().eq_ignore_ascii_case("security-finding")) + { + let mut issue = issue_fields(state, n, issue_state); + issue.set("html_url", issue.at("url").clone()); + println!("{issue}"); + } + } + return; + } + if path.contains("/security-advisories") { + advisory_api(state, positional, flags); + return; + } if path == "user" && flags.is_empty() { let email = state.get("user_email").cloned().unwrap_or(Null); println!( @@ -1770,7 +1954,23 @@ pub fn main(args: Vec<String>) { release_view(&state, &positional, &flags); } ["api", "graphql", rest @ ..] => graphql(&state, rest), - ["api", "--method", "PATCH", rest @ ..] => pr_patch(&mut state, rest), + ["api", "--method", "PATCH", rest @ ..] + if rest.first().is_some_and(|path| { + repo_prefix(path).is_some_and(|(_, path)| { + path.strip_prefix("issues/") + .is_some_and(|number| number.parse::<u64>().is_ok()) + }) + }) => + { + finding_issue_patch(&mut state, rest) + } + ["api", "--method", "PATCH", rest @ ..] + if !rest + .first() + .is_some_and(|path| path.contains("/security-advisories")) => + { + pr_patch(&mut state, rest) + } ["api", "--method", "PUT", rest @ ..] => issue_labels_put(&mut state, rest), ["api", "--method", "DELETE", rest @ ..] => issue_label_delete(&mut state, rest), ["api", path, ..] if generate_notes_repo(path).is_some() => { diff --git a/tests/fresh_run.rs b/tests/fresh_run.rs index 4cc00471..b7168afc 100644 --- a/tests/fresh_run.rs +++ b/tests/fresh_run.rs @@ -174,6 +174,53 @@ fn finds_every_factory_skill_in_the_worktree_as_thirdshift_skill() { } } +#[test] +fn ships_cloudflares_security_audit_with_its_license_and_provenance_on_both_harnesses() { + for harness in ["claude", "codex"] { + let scenario = Scenario::new(); + let status = scenario.path("status.txt"); + scenario.agent_does(&format!( + "git status --porcelain --untracked-files=all > '{}'\n{AGENT_COMMITS_AND_OPENS_PR}", + status.display() + )); + + let result = scenario.run(&[&scenario.issue_url(7), "harness", harness]); + + assert_eq!(result.code, Some(0), "stderr: {}", result.stderr); + assert_eq!(std::fs::read_to_string(&status).unwrap(), ""); + let calls = if harness == "claude" { + scenario.assert_every_session_found_the_factory_skills(); + scenario.claude_calls() + } else { + scenario.assert_every_codex_session_found_the_factory_skills(); + scenario.codex_calls() + }; + let files = &calls[0]["skill_files"]; + let license = files["thirdshift-security-audit/LICENSE"].as_str().unwrap(); + assert!(license.contains("MIT License")); + assert!(license.contains("Copyright (c) 2025-2026 Cloudflare, Inc.")); + let credits = files["thirdshift-security-audit/CREDITS.md"] + .as_str() + .unwrap(); + assert!(credits.contains("https://github.com/cloudflare/security-audit-skill")); + assert!(credits.contains("c1c8a8c1471069fb0e188eeaff69b8e8db6564a8")); + for supporting in [ + "HUNTING.md", + "VALIDATION-AND-REPORTING.md", + "report-schema.json", + "validate-coverage-ledger.cjs", + "validate-coverage-ledger.test.cjs", + "validate-findings.cjs", + "validate-findings.test.cjs", + ] { + assert!( + files[format!("thirdshift-security-audit/{supporting}")].is_string(), + "{harness} found no {supporting}" + ); + } + } +} + #[test] fn ships_the_mattpocock_skills_mit_notice_with_the_factory_skills() { let scenario = Scenario::new(); diff --git a/tests/muse_sessions.rs b/tests/muse_sessions.rs index a4e74f1b..f30e6eb1 100644 --- a/tests/muse_sessions.rs +++ b/tests/muse_sessions.rs @@ -297,6 +297,8 @@ fn setup_checks_muses_proposed_model_with_the_same_flags_and_environment() { ("Effort for muse", "low"), ("Every Run a Merge run", ""), ("Every Run first fast-forwards", ""), + ("Security runs may fix reproduced findings?", ""), + ("Runs review their change for Security findings?", ""), ("Run notifications", ""), ], ); diff --git a/tests/opencode_sessions.rs b/tests/opencode_sessions.rs index 5a9922af..756d497b 100644 --- a/tests/opencode_sessions.rs +++ b/tests/opencode_sessions.rs @@ -406,6 +406,8 @@ fn setup_proposes_no_model_and_checks_the_answer_standalone() { ("Effort for opencode", "high"), ("Every Run a Merge run", ""), ("Every Run first fast-forwards", ""), + ("Security runs may fix reproduced findings?", ""), + ("Runs review their change for Security findings?", ""), ("Run notifications", ""), ], ); @@ -482,6 +484,8 @@ fn setup_retries_a_refused_model_and_effort_with_the_same_preflight_check() { ("Effort for opencode", "low"), ("Every Run a Merge run", ""), ("Every Run first fast-forwards", ""), + ("Security runs may fix reproduced findings?", ""), + ("Runs review their change for Security findings?", ""), ("Run notifications", ""), ], ); diff --git a/tests/packaging.rs b/tests/packaging.rs index 1b9315e6..821cc637 100644 --- a/tests/packaging.rs +++ b/tests/packaging.rs @@ -84,6 +84,14 @@ fn package_carries_the_factory_skills_with_their_license_and_credits() { "skills/thirdshift-codebase-design/SKILL.md", "skills/thirdshift-codebase-design/DEEPENING.md", "skills/thirdshift-codebase-design/DESIGN-IT-TWICE.md", + "skills/thirdshift-security-audit/SKILL.md", + "skills/thirdshift-security-audit/LICENSE", + "skills/thirdshift-security-audit/CREDITS.md", + "skills/thirdshift-security-audit/report-schema.json", + "skills/thirdshift-security-audit/validate-findings.cjs", + "skills/thirdshift-security-audit/validate-findings.test.cjs", + "skills/thirdshift-security-audit/validate-coverage-ledger.cjs", + "skills/thirdshift-security-audit/validate-coverage-ledger.test.cjs", "src/main.rs", "README.md", "LICENSE", diff --git a/tests/pickup_run.rs b/tests/pickup_run.rs index 7383e944..af6f9504 100644 --- a/tests/pickup_run.rs +++ b/tests/pickup_run.rs @@ -788,8 +788,7 @@ fn an_origin_that_is_not_on_github_stops_the_pass_before_any_label_changes() { /// What a skipped Pickup run or Architect run says when another of either is /// running on the scenario's repository. -const ALREADY_RUNNING: &str = - "thirdshift: an Architect run or a Pickup run is already running on acme/widgets\n"; +const ALREADY_RUNNING: &str = "thirdshift: another Pass is already running on acme/widgets\n"; /// A script in which the agent touches `started` in the scenario root, then /// waits there until the test touches `release`, or until the scenario is diff --git a/tests/process_tree.rs b/tests/process_tree.rs index 9e3d60a2..faac247d 100644 --- a/tests/process_tree.rs +++ b/tests/process_tree.rs @@ -100,12 +100,16 @@ impl SessionCommand { unsafe { libc::kill(self.0, 0) == 0 } } - fn assert_stopped(&self, message: &str) { + fn wait_for_stop(&self) -> bool { let deadline = Instant::now() + Duration::from_secs(2); while self.exists() && Instant::now() < deadline { thread::sleep(Duration::from_millis(10)); } - assert!(!self.exists(), "{message}"); + !self.exists() + } + + fn assert_stopped(&self, message: &str) { + assert!(self.wait_for_stop(), "{message}"); } } @@ -169,6 +173,12 @@ fn interrupting_a_live_session_stops_its_stdout_holder_before_slow_cleanup() { fn assert_live_session_stdout_holder_stopped(slow_cleanup: bool) { let scenario = Scenario::new(); + // Failed run preservation may be slow even when the session stops promptly. + scenario.repo_has_hook( + &scenario.origin_dir(), + "pre-receive", + "#!/bin/sh\nsleep 3\n", + ); let pid = scenario.path("stdout-holder-pid"); let cleanup_started = scenario.path("cleanup-started"); let holder_at_cleanup = scenario.path("stdout-holder-at-cleanup"); @@ -201,7 +211,8 @@ PATH="${{PATH#*:}}" exec git "$@" // Bash waits on its own FIFO so a child sleep cannot outlive the holder // and keep stdout open without writing the natural-expiry marker. scenario.agent_does(&format!( - r#"bash -c 'while kill -0 "$1" 2>/dev/null; do sleep 0.01; done + r#"printf 'interrupted work\n' > interrupted-work.txt +bash -c 'while kill -0 "$1" 2>/dev/null; do sleep 0.01; done mkfifo "$5" exec 9<>"$5" echo $$ > "$2" @@ -219,6 +230,9 @@ touch "$4"' holder "$PPID" "{pid}" "{started}" "{expired}" "{wait_pipe}" & let command = SessionCommand(fs::read_to_string(pid).unwrap().trim().parse().unwrap()); assert!(command.exists(), "the stdout holder never started"); held.signal("INT"); + // Observe stopping before waiting for the Run's preservation and cleanup. + // Finish the exact Run before asserting, so a failure leaves no Run behind. + let stopped = command.wait_for_stop(); let result = held.finish(); assert_eq!(result.code, Some(1), "{}", result.stderr); @@ -226,10 +240,17 @@ touch "$4"' holder "$PPID" "{pid}" "{started}" "{expired}" "{wait_pipe}" & // Observe whether the holder expired: timing the whole Run also counts // finishing Git work and Claim release, which can be slow under CI load. assert!( - !expired.exists(), + stopped && !expired.exists(), "interruption waited for the stdout holder's natural expiry" ); command.assert_stopped("the stdout holder survived interruption"); + assert_eq!( + scenario + .origin_file("issue-7", "interrupted-work.txt") + .as_deref(), + Some("interrupted work\n"), + "the slow push did not preserve the interrupted work" + ); if slow_cleanup { assert!(cleanup_started.exists(), "the delayed cleanup never ran"); assert!( diff --git a/tests/security_review.rs b/tests/security_review.rs new file mode 100644 index 00000000..636be0ac --- /dev/null +++ b/tests/security_review.rs @@ -0,0 +1,766 @@ +mod support; + +use serde_json::json; +use support::Scenario; + +const OPENS_PR: &str = r#" +echo feature > feature.txt +git add feature.txt +git commit -q -m 'Add feature' +gh pr create --base main --head issue-7 --title 'Add feature' --body 'Closes #7' +"#; + +fn old_finding() -> serde_json::Value { + json!({ + "fingerprint": "old-input-bound", + "title": "Old unchecked bound", + "description": "Private old vulnerability evidence.", + "proof_of_concept": { + "test": "assert_bounded_input();", + "command": "cargo test bounded_input", + "head_exit_code": 101, + "merge_base_exit_code": 101, + "notes": "The bounded local fixture fails at both commits. Low likelihood and low impact; one session can add a bound.", + "severity": "low", + "fix_size": "single" + } + }) +} + +fn review_script(old: &serde_json::Value, introduced: &[&str]) -> String { + format!( + r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > review-prompt.txt +report=$(sed -n 's/^Private report file: `\(.*\)`\.$/\1/p' review-prompt.txt) +test -n "$report" +printf '%s\n' '{old}' > "$report" +printf '%s\n' 'Security review: {outcome}' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + outcome = json!({ + "unaddressed_count": introduced.len(), + "findings": introduced, + "pre_existing_count": old.as_array().unwrap().len() + }) + ) +} + +#[test] +fn an_old_finding_is_recorded_privately_without_holding_self_merge() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([old_finding()]), &[])); + let result = scenario.run(&[ + &scenario.issue_url(7), + "security-review", + "merge", + "security-fix", + ]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let records = state["advisories"] + .as_array() + .expect("private draft advisory"); + assert_eq!(records.len(), 1); + assert_eq!(records[0]["state"], "draft"); + assert_eq!(records[0]["severity"], "low"); + let description = records[0]["description"].as_str().unwrap(); + assert!(description.contains("Fingerprint: `old-input-bound`")); + assert!(description.contains("Private old vulnerability evidence.")); + assert!(description.contains("assert_bounded_input();")); + assert!(description.contains(scenario.launch_git(&["rev-parse", "main"]).trim())); + assert_eq!(state["prs"][0]["state"], "MERGED"); + assert_eq!( + scenario.claude_calls().len(), + 2, + "old findings belong to a Security run" + ); + for private in [ + "Old unchecked bound", + "old-input-bound", + "Private old vulnerability evidence.", + "assert_bounded_input();", + ] { + assert!(!state["prs"][0]["body"].as_str().unwrap().contains(private)); + assert!(!result.stderr.contains(private)); + assert!(!result.stdout.contains(private)); + } + let reports = scenario.entries("home/.thirdshift/logs/acme/widgets/security-reviews"); + assert_eq!(reports.len(), 1); + let kept: serde_json::Value = serde_json::from_slice( + &std::fs::read(scenario.path(&format!( + "home/.thirdshift/logs/acme/widgets/security-reviews/{}/pre-existing.json", + reports[0] + ))) + .unwrap(), + ) + .unwrap(); + assert_eq!(kept, json!([old_finding()])); +} + +#[test] +fn mixed_old_and_introduced_findings_keep_the_introduced_hold_and_old_details_private() { + for harness in ["claude", "codex"] { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session( + 2, + &review_script(&json!([old_finding()]), &["Introduced cross-tenant read"]), + ); + let result = scenario.run(&[ + &scenario.issue_url(7), + "security-review", + "merge", + "harness", + harness, + ]); + assert_eq!(result.code, Some(1), "{harness}: {}", result.stderr); + let state = scenario.gh_state(); + assert_eq!(state["advisories"].as_array().unwrap().len(), 1); + assert_eq!(state["prs"][0]["state"], "OPEN"); + assert_eq!(state["prs"][0]["isDraft"], false); + let body = state["prs"][0]["body"].as_str().unwrap(); + assert!(body.contains("Introduced cross-tenant read")); + assert!(result.stderr.contains("Introduced cross-tenant read")); + for private in [ + "Old unchecked bound", + "old-input-bound", + "Private old vulnerability evidence.", + "assert_bounded_input();", + ] { + assert!(!body.contains(private)); + assert!(!result.stderr.contains(private)); + } + } +} + +#[test] +fn repeated_fingerprints_reuse_private_records_and_preserve_day_shift_grades() { + for (private, record_state) in [ + (false, "draft"), + (false, "published"), + (false, "closed"), + (true, "OPEN"), + (true, "CLOSED"), + ] { + let scenario = Scenario::new(); + let mut state = scenario.gh_state(); + state["private"] = json!(private); + scenario.write_gh_state(&state); + scenario.agent_does_in_session(1, OPENS_PR); + // Duplicates within one report must match just as later reviews do. + scenario.agent_does_in_session( + 2, + &review_script(&json!([old_finding(), old_finding()]), &[]), + ); + let first = scenario.run(&[&scenario.issue_url(7), "security-review", "no-merge"]); + assert_eq!( + first.code, + Some(0), + "{private} {record_state}: {}", + first.stderr + ); + let mut state = scenario.gh_state(); + if private { + assert!(state["advisories"].is_null()); + assert_eq!(state["issues"].as_object().unwrap().len(), 2); + assert_eq!( + state["labels"]["8"], + json!(["security-finding", "needs-triage"]) + ); + assert!( + state["bodies"]["8"] + .as_str() + .unwrap() + .contains("Private old vulnerability evidence.") + ); + state["issues"]["8"] = json!(record_state); + state["labels"]["8"] = json!(["security-finding"]); + } else { + assert_eq!(state["advisories"].as_array().unwrap().len(), 1); + state["advisories"][0]["state"] = json!(record_state); + state["advisories"][0]["severity"] = json!("low"); + } + scenario.write_gh_state(&state); + scenario.agent_does_in_session(3, "true"); + let mut repeated = old_finding(); + repeated["description"] = json!("A later review must not replace the grade or write-up."); + scenario.agent_does_in_session(4, &review_script(&json!([repeated]), &[])); + let second = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!( + second.code, + Some(0), + "{private} {record_state}: {}", + second.stderr + ); + let after = scenario.gh_state(); + assert_eq!(after["prs"][0]["state"], "MERGED"); + if private { + assert_eq!(after["issues"]["8"], state["issues"]["8"]); + assert_eq!(after["labels"]["8"], state["labels"]["8"]); + assert_eq!(after["bodies"]["8"], state["bodies"]["8"]); + assert_eq!(after["issues"].as_object().unwrap().len(), 2); + } else { + assert_eq!(after["advisories"], state["advisories"]); + } + let body = after["prs"][0]["body"].as_str().unwrap(); + assert!(!body.contains("Old unchecked bound")); + assert!(!body.contains("old-input-bound")); + } +} + +#[test] +fn incomplete_private_reports_hold_self_merge_without_publishing_report_diagnostics() { + let valid = review_script(&json!([old_finding()]), &[]); + let mut passes_at_base = old_finding(); + passes_at_base["proof_of_concept"]["merge_base_exit_code"] = json!(0); + let mut no_test = old_finding(); + no_test["proof_of_concept"]["test"] = json!(""); + for script in [ + valid.replace("\"pre_existing_count\":1", "\"pre_existing_count\":0"), + format!("{valid}\nrm \"$report\"\n"), + format!("{valid}\nprintf '%s' 'Private old vulnerability evidence.' > \"$report\"\n"), + review_script(&json!([passes_at_base]), &[]), + review_script(&json!([no_test]), &[]), + ] { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &script); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let state = scenario.gh_state(); + assert!(state["advisories"].is_null()); + assert_eq!(state["prs"][0]["state"], "OPEN"); + assert_eq!(state["prs"][0]["isDraft"], false); + let body = state["prs"][0]["body"].as_str().unwrap(); + assert!(body.contains("Security review incomplete")); + assert!(!body.contains("Private old vulnerability evidence.")); + assert!( + !result + .stderr + .contains("Private old vulnerability evidence.") + ); + } +} + +#[test] +fn review_uses_the_published_base_when_the_local_base_is_stale() { + let scenario = Scenario::new(); + scenario.origin_has_commit( + "main", + "base-new.txt", + "base change", + "Advance the Base branch", + ); + let base = scenario.origin_git(&["rev-parse", "main"]); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session( + 2, + &format!( + r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > review-prompt.txt +merge_base=$(sed -n 's/^Merge base commit: `\(.*\)`\.$/\1/p' review-prompt.txt) +test "$merge_base" = "$(git rev-parse origin/main)" +{} +"#, + review_script(&json!([old_finding()]), &[]) + ), + ); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert!( + state["advisories"][0]["description"] + .as_str() + .unwrap() + .contains(base.trim()) + ); + assert_eq!(state["prs"][0]["state"], "MERGED"); +} + +#[test] +fn command_words_override_the_config_and_review_is_off_by_default() { + for (config, word, enabled) in [ + ("", None, false), + ("[security]\nreview = true\n", None, true), + ( + "[security]\nreview = true\n", + Some("no-security-review"), + false, + ), + ( + "[security]\nreview = false\n", + Some("security-review"), + true, + ), + ] { + let scenario = Scenario::new(); + scenario.user_config_is(config); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([]), &[])); + let url = scenario.issue_url(7); + let mut args = vec![url.as_str(), "merge"]; + args.extend(word); + let result = scenario.run(&args); + assert_eq!(result.code, Some(0), "{config} {word:?}: {}", result.stderr); + assert_eq!(scenario.claude_calls().len(), if enabled { 2 } else { 1 }); + assert_eq!(scenario.gh_state()["prs"][0]["state"], "MERGED"); + } +} + +#[test] +fn invalid_or_missing_final_lines_hold_self_merge_but_report_only_runs_continue() { + for final_message in [ + "Review incomplete", + "Security review: {}", + r#"Security review: {"unaddressed_count":0,"findings":["Cross-tenant read"],"pre_existing_count":0}"#, + r#"Security review: {"unaddressed_count":1,"findings":[""],"pre_existing_count":0}"#, + r#"Security review: {"unaddressed_count":0,"findings":[],"pre_existing_count":0}\nMore text"#, + ] { + for merge in [true, false] { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + // Write literal data as a file through the existing fixture. + std::fs::write(scenario.path("review-final.txt"), final_message).unwrap(); + scenario.agent_does_in_session(2, r#"cat "$(dirname "$FAKE_CLAUDE_SCRIPT")/review-final.txt" > "$FAKE_CLAUDE_FINAL_MESSAGE""#); + let result = scenario.run(&[ + &scenario.issue_url(7), + "security-review", + if merge { "merge" } else { "no-merge" }, + ]); + assert_eq!( + result.code, + Some(if merge { 1 } else { 0 }), + "{final_message}: {}", + result.stderr + ); + assert!( + result.stderr.contains("Security review incomplete"), + "{}", + result.stderr + ); + let pr = &scenario.gh_state()["prs"][0]; + assert_eq!(pr["state"], "OPEN"); + assert_eq!(pr["isDraft"], false); + assert!( + pr["body"] + .as_str() + .unwrap() + .contains("Security review incomplete") + ); + } + } +} + +#[test] +fn safeguard_refusals_hold_self_merge_without_publishing_private_diagnostics() { + for (harness, script, cause) in [ + ( + "claude", + r#"printf '%s\n' 'API Error: [cyber] Private refusal evidence' > "$FAKE_CLAUDE_FINAL_MESSAGE""#, + "Claude Code's [cyber] safeguard refusal", + ), + ( + "codex", + r#"printf '%s\n' 'Cybersecurity safeguard refused: Private refusal evidence' > "$FAKE_CODEX_ERROR" +exit 1"#, + "Codex's cybersecurity safeguard refusal", + ), + ] { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, script); + let result = scenario.run(&[ + &scenario.issue_url(7), + "security-review", + "merge", + "harness", + harness, + ]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!(result.stderr.contains(cause), "{}", result.stderr); + let pr = &scenario.gh_state()["prs"][0]; + assert_eq!(pr["state"], "OPEN"); + assert_eq!(pr["isDraft"], false); + let body = pr["body"].as_str().unwrap(); + assert!(body.contains(cause), "{body}"); + assert!(!body.contains("Private refusal evidence")); + } +} + +#[test] +fn base_fixes_inherit_the_security_review_choice() { + let scenario = Scenario::new(); + scenario.agent_does_for_in_session( + 7, + 1, + &format!( + r#"{OPENS_PR} +gh fake checks "$(git rev-parse HEAD)" '[{{"name":"test","conclusion":"failure"}}]' +gh fake checks "$(git rev-parse origin/main)" '[{{"name":"test","conclusion":"failure"}}]' +"# + ), + ); + scenario.agent_does_for_in_session(7, 2, &review_script(&json!([]), &[])); + scenario.agent_does_for_in_session( + 8, + 1, + r#" +echo fixed > ci-fix.txt +git add ci-fix.txt +git commit -q -m 'Fix base CI' +gh pr create --base main --head issue-8 --title 'Fix base CI' --body 'Closes #8' +gh fake checks "$(git rev-parse HEAD)" '[{"name":"test","conclusion":"success"}]' +"#, + ); + scenario.agent_does_for_in_session(8, 2, &review_script(&json!([]), &[])); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "base-fix"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let calls = scenario.claude_calls(); + assert_eq!(calls.len(), 4); + assert!( + calls[3]["prompt"] + .as_str() + .unwrap() + .contains("against Base branch main") + ); + assert_eq!(scenario.gh_state()["prs"][1]["state"], "MERGED"); +} + +#[test] +fn an_unaddressed_introduced_finding_holds_self_merge_with_the_pr_ready() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([]), &["Cross-tenant read"])); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("Cross-tenant read"), + "{}", + result.stderr + ); + let pr = &scenario.gh_state()["prs"][0]; + assert_eq!(pr["state"], "OPEN"); + assert_eq!(pr["isDraft"], false); + assert!(pr["body"].as_str().unwrap().contains("### Security")); + assert!(pr["body"].as_str().unwrap().contains("Cross-tenant read")); +} + +#[test] +fn enabled_review_fixes_reach_origin_before_delivery_finishes() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session( + 2, + &format!( + r#" +echo checked > security-fix.txt +git add security-fix.txt +git commit -q -m 'Fix reproduced finding' +{clean} +"#, + clean = review_script(&json!([]), &[]) + ), + ); + let result = scenario.run(&[&scenario.issue_url(7), "security-review"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario + .origin_file("issue-7", "security-fix.txt") + .as_deref(), + Some("checked\n") + ); + let calls = scenario.claude_calls(); + assert_eq!(calls.len(), 2); + let prompt = calls[1]["prompt"].as_str().unwrap(); + assert!(prompt.contains("guidance mode")); + assert!(prompt.contains("Do not delegate")); + assert!(prompt.contains(&format!( + "git diff {}...HEAD", + scenario.launch_git(&["rev-parse", "main"]).trim() + ))); +} + +// Regression A. +#[test] +fn review_regression_a_security_hold_still_repairs_red_ci() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &format!( + "{OPENS_PR}\n{}", + r#" +gh fake checks "$(git rev-parse HEAD)" '[{"name":"test","conclusion":"failure"}]' +gh fake checks "$(git rev-parse origin/main)" '[{"name":"test","conclusion":"success"}]' +"# + ), + ); + scenario.agent_does_in_session(2, &review_script(&json!([]), &["Cross-tenant read"])); + scenario.agent_does_in_session( + 3, + r#" +echo fixed > ci-fixed.txt +git add ci-fixed.txt +git commit -q -m 'Fix branch CI' +gh fake checks "$(git rev-parse HEAD)" '[{"name":"test","conclusion":"success"}]' +"#, + ); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("Cross-tenant read"), + "{}", + result.stderr + ); + assert_eq!( + scenario.claude_calls().len(), + 3, + "The held Merge run should repair red CI before handing over its PR: {}", + result.stderr + ); + assert_eq!( + scenario.origin_file("issue-7", "ci-fixed.txt").as_deref(), + Some("fixed\n") + ); + let state = scenario.gh_state(); + assert_eq!(state["prs"][0]["state"], "OPEN"); + assert_eq!(state["prs"][0]["isDraft"], false); +} + +// Regression B. +#[test] +fn review_regression_b_body_write_failure_preserves_refusal_and_readiness() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session( + 2, + r#" +gh fake fails 'api --method PATCH repos/acme/widgets/pulls/1' +printf '%s\n' 'API Error: [cyber] Private refusal evidence' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + ); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let state = scenario.gh_state(); + assert_eq!(state["prs"][0]["state"], "OPEN"); + assert_eq!( + state["prs"][0]["isDraft"], false, + "A refused review must leave the already-ready PR ready when only its summary PATCH fails: {}", + result.stderr + ); + assert!( + result.stderr.contains("Security review refused"), + "The failure must retain the Security refusal cause: {}", + result.stderr + ); +} + +// Regression C. +#[test] +fn review_regression_c_continuation_replaces_the_previous_security_outcome() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([]), &["Cross-tenant read"])); + let first = scenario.run(&[&scenario.issue_url(7), "security-review", "no-merge"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + assert!( + scenario.gh_state()["prs"][0]["body"] + .as_str() + .unwrap() + .contains("Cross-tenant read") + ); + scenario.agent_does_in_session( + 3, + r#" +echo fixed > feature.txt +git add feature.txt +git commit -q -m 'Fix the previously reported finding' +"#, + ); + scenario.agent_does_in_session(4, &review_script(&json!([]), &[])); + let second = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(second.code, Some(0), "{}", second.stderr); + let state = scenario.gh_state(); + assert_eq!(state["prs"][0]["state"], "MERGED"); + let body = state["prs"][0]["body"].as_str().unwrap(); + assert!( + !body.contains("Cross-tenant read"), + "The fixed finding remains falsely listed as unaddressed after a clean review: {body}" + ); + assert_eq!( + body.matches("### Security review outcome").count(), + 1, + "{body}" + ); +} + +#[test] +fn review_old_finding_can_be_taken_by_next_security_run() { + for private in [false, true] { + let scenario = Scenario::new(); + let mut state = scenario.gh_state(); + state["private"] = json!(private); + scenario.write_gh_state(&state); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([old_finding()]), &[])); + let first = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + let before = scenario.claude_calls().len(); + if !private { + let record = scenario.gh_state()["advisories"][0]["html_url"] + .as_str() + .unwrap() + .to_string(); + scenario.agent_does_in_session(3, &format!(r#" +gh label create needs-triage --description 'Needs triage' +issue=$(gh issue create --title 'Bound input' --body 'Add a bound. Private record: {record}' --label needs-triage) +printf '%s\n' "Security fix Ticket: $issue" > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#)); + } + scenario.agent_does_for( + 8, + r#" +echo bounded > bounded.txt +git add bounded.txt +git commit -q -m 'Bound input' +gh pr create --base main --head issue-8 --title 'Bound input' --body 'Closes #8' +"#, + ); + let next = scenario.run(&["secure", "base", "main", "security-fix"]); + assert!( + scenario.claude_calls().len() > before, + "Security run must take the reproduced review finding when fixing is allowed; private={private}, stderr={}", + next.stderr + ); + assert_eq!(next.code, Some(0), "{private}: {}", next.stderr); + let state = scenario.gh_state(); + assert_eq!(state["prs"][1]["head"], "issue-8"); + assert_eq!(state["prs"][1]["isDraft"], false); + assert_eq!( + scenario.origin_file("issue-8", "bounded.txt").as_deref(), + Some("bounded\n") + ); + assert!( + !state["prs"][1]["body"] + .as_str() + .unwrap() + .contains("Private old vulnerability evidence.") + ); + } +} + +#[test] +fn switched_checkout_is_rejected_before_private_recording() { + for before_review in [true, false] { + let scenario = Scenario::new(); + let switch = "\ngit switch -c replacement-branch\n"; + scenario.agent_does_in_session( + 1, + &format!("{OPENS_PR}{}", if before_review { switch } else { "" }), + ); + scenario.agent_does_in_session( + 2, + &format!( + "{}{}", + review_script(&json!([old_finding()]), &[]), + if before_review { "" } else { switch } + ), + ); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let records = scenario.gh_state()["advisories"].clone(); + assert!( + records.is_null() || records.as_array().is_some_and(|records| records.is_empty()), + "A changed checkout must be rejected before private recording: {records}; {}", + result.stderr + ); + assert_eq!( + scenario.claude_calls().len(), + if before_review { 1 } else { 2 }, + "A checkout replaced by the opening session must not start a Security review" + ); + } +} + +#[test] +fn review_merge_base_uses_remote_ref_when_local_tag_shadows_it() { + let scenario = Scenario::new(); + scenario.launch_git(&["tag", "origin/main"]); + scenario.origin_has_commit("main", "base-new.txt", "base change", "Advance Base branch"); + let published_base = scenario.origin_git(&["rev-parse", "refs/heads/main"]); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([]), &[])); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "no-merge"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let calls = scenario.claude_calls(); + let prompt = calls[1]["prompt"].as_str().unwrap(); + let expected = format!("Merge base commit: `{}`.", published_base.trim()); + assert!( + prompt.contains(&expected), + "Security review used a local tag instead of published Base branch merge base; expected {expected}, got {prompt}" + ); +} + +#[test] +fn guidance_security_review_on_codex_does_not_request_delegated_auditors() { + for resumed in [false, true] { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + let mut review = review_script(&json!([]), &[]); + if resumed { + review.push_str( + r#"printf '%s\n' '{"type":"item.started","item":{"id":"pending","type":"command_execution","command":"bounded-fixture","status":"in_progress"}}' +"#, + ); + scenario.agent_does_in_session( + 3, + r#"printf '%s\n' 'Security review: {"unaddressed_count":0,"findings":[],"pre_existing_count":0}' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + ); + } + scenario.agent_does_in_session(2, &review); + let result = scenario.run(&[ + &scenario.issue_url(7), + "security-review", + "harness", + "codex", + ]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let calls = scenario.codex_calls(); + assert_eq!(calls.len(), if resumed { 3 } else { 2 }); + assert!( + calls[1]["prompt"] + .as_str() + .unwrap() + .contains("Use one session. Do not delegate auditors") + ); + for call in &calls[1..] { + let prompt = call["prompt"].as_str().unwrap(); + assert!( + !prompt.contains("Start fresh sub-agents"), + "The chosen guidance review and its Resume must not request delegated auditors: {prompt}" + ); + let args = call["argv"].as_array().unwrap(); + assert!(args.windows(2).any(|pair| pair + == [ + json!("-c"), + json!("agents.max_concurrent_threads_per_session=8") + ])); + } + } +} + +#[test] +fn reproduced_old_review_finding_sets_the_private_advisory_severity() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, OPENS_PR); + scenario.agent_does_in_session(2, &review_script(&json!([old_finding()]), &[])); + let result = scenario.run(&[&scenario.issue_url(7), "security-review", "merge"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert_eq!(state["prs"][0]["state"], "MERGED"); + assert_eq!(state["advisories"][0]["state"], "draft"); + assert_eq!( + state["advisories"][0]["severity"], "low", + "The validated old finding already has a failing proof of concept and scored severity; record it as a Security run does" + ); +} diff --git a/tests/security_run.rs b/tests/security_run.rs new file mode 100644 index 00000000..b62864ae --- /dev/null +++ b/tests/security_run.rs @@ -0,0 +1,3216 @@ +mod support; + +use std::fs; + +use serde_json::{Value, json}; +use support::resend::ResendStandIn; +use support::{REPO, Scenario}; + +fn resend_env(resend: &ResendStandIn) -> [(&str, &str); 2] { + [ + ("THIRDSHIFT_RESEND_URL", resend.url()), + ("RESEND_API_KEY", "re_security_test"), + ] +} + +fn the_one_notification(resend: &ResendStandIn) -> (String, String) { + let requests = resend.requests(); + assert_eq!(requests.len(), 1, "{requests:?}"); + let body = &requests[0].body; + ( + body["subject"].as_str().unwrap().to_string(), + body["text"].as_str().unwrap().to_string(), + ) +} + +fn finding(fingerprint: &str) -> Value { + json!({ + "verdict": "needs_validation", "fingerprint": fingerprint, + "title": "Unchecked input size", "description": "Private candidate write-up.", + "claimed_root_cause": "Input reaches storage without a bound.", + "trace": [{"kind":"entrypoint", "file":"README.md", "line":1, "scope":"input", "description":"Private trace."}], + "evidence": [{"file":"README.md", "line":1, "description":"Private evidence."}], + "blockers": ["No sandbox available."], + "validation_plan": {"local":"Exercise a bounded fixture."} + }) +} + +fn safeguard_refusals() -> [(&'static str, &'static str, &'static str); 2] { + [ + ( + "claude", + r#"printf '%s\n' 'API Error: [cyber] Private refusal evidence.' > "$FAKE_CLAUDE_FINAL_MESSAGE""#, + "Claude Code's [cyber] safeguard refusal", + ), + ( + "codex", + r#"printf '%s\n' 'Cybersecurity safeguard refused: Private refusal evidence.' > "$FAKE_CODEX_ERROR" +exit 1"#, + "Codex's cybersecurity safeguard refusal", + ), + ] +} + +fn security_command_log(scenario: &Scenario) -> String { + let logs = scenario.entries("home/.thirdshift/logs/acme/widgets/commands/secure"); + fs::read_to_string(scenario.path(&format!( + "home/.thirdshift/logs/acme/widgets/commands/secure/{}", + logs[0] + ))) + .unwrap() +} + +#[test] +fn a_claude_cyber_refusal_fails_the_security_audit_with_its_cause() { + let scenario = Scenario::new(); + scenario.agent_does( + r#" +printf '%s\n' 'API Error: [cyber] This request was refused by the safeguard.' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + ); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result + .stderr + .contains("Claude Code's [cyber] safeguard refusal"), + "{}", + result.stderr + ); + assert!(scenario.gh_state()["advisories"].is_null()); + assert_eq!(scenario.claude_calls().len(), 1); + assert_eq!(scenario.entries("work"), vec![REPO]); +} + +#[test] +fn safeguard_refusals_notify_the_cause_without_private_diagnostics() { + for (harness, script, cause) in safeguard_refusals() { + let scenario = Scenario::new(); + scenario.agent_does(script); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = scenario.run_with_env( + &["secure", "harness", harness, "email", "me@example.com"], + &resend_env(&resend), + ); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!(result.stderr.contains(cause), "{}", result.stderr); + let (subject, text) = the_one_notification(&resend); + assert_eq!( + subject, + "[thirdshift] acme/widgets Security run: audit failed" + ); + assert!(text.contains(&format!("Cause: {cause}")), "{text}"); + assert!(!text.contains("Private refusal evidence."), "{text}"); + } +} + +#[test] +fn security_sessions_log_claudes_answering_models_in_progress_and_the_command_log() { + let scenario = Scenario::new(); + scenario.agent_does_in_session(1, "true\n"); // Requested Model check. + scenario.agent_does_in_session( + 2, + &format!( + r#" +printf '%s\n' '{{"type":"assistant","message":{{"model":"claude-opus-5-5","content":[]}}}}' +printf '%s\n' '{{"type":"assistant","parent_tool_use_id":"child","message":{{"model":"claude-sonnet-5","content":[]}}}}' +printf '%s\n' '{{"type":"system","subtype":"model_refusal_fallback","original_model":"claude-opus-5-5","fallback_model":"claude-opus-4-8","api_refusal_category":"cyber"}}' +printf '%s\n' '{{"type":"assistant","message":{{"model":"claude-opus-4-8","content":[]}}}}' +{} +"#, + audit_script(&json!([finding("model-switch")]).to_string()) + ), + ); + scenario.agent_does_in_session( + 3, + &format!( + "printf '%s\\n' '{{\"type\":\"assistant\",\"message\":{{\"model\":\"claude-opus-4-8\",\"content\":[]}}}}'\n{}", + reproduction_script("not reproduced") + ), + ); + let result = scenario.run(&["secure", "model", "opus"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let command_log = security_command_log(&scenario); + for line in [ + "security-audit: Model: claude-opus-5-5", + "security-audit: Model: claude-opus-4-8", + "security-reproduction-1: Model: claude-opus-4-8", + ] { + assert!(result.stderr.contains(line), "{}", result.stderr); + assert!(command_log.contains(line), "{command_log}"); + } + assert!( + !command_log.contains("Model: claude-sonnet-5"), + "{command_log}" + ); +} + +#[test] +fn codex_logs_the_requested_model_for_each_security_session_and_resume() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &format!( + "{}\nprintf '%s\\n' '{{\"type\":\"item.started\",\"item\":{{\"id\":\"child\",\"type\":\"collab_tool_call\",\"tool\":\"spawn_agent\",\"status\":\"in_progress\"}}}}'\n", + audit_script(&json!([finding("codex-model")]).to_string()) + ), + ); + scenario.agent_does_in_session( + 2, + r#"printf '%s\n' 'Security audit: complete' > "$FAKE_CLAUDE_FINAL_MESSAGE""#, + ); + scenario.agent_does_in_session(3, &reproduction_script("not reproduced")); + let result = scenario.run(&["secure", "harness", "codex", "model", "GPT-6.1-Sol"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let command_log = security_command_log(&scenario); + for kind in [ + "security-audit", + "security-audit-resume", + "security-reproduction-1", + ] { + let line = format!("{kind}: Model: gpt-6.1-sol (requested)"); + assert!(result.stderr.contains(&line), "{}", result.stderr); + assert!(command_log.contains(&line), "{command_log}"); + } +} + +#[test] +fn refused_security_reproductions_keep_the_record_and_stop_before_the_next_finding() { + for (harness, script, cause) in safeguard_refusals() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("first"), finding("second")]).to_string()), + ); + scenario.agent_does_in_session(2, script); + scenario.agent_does_in_session(3, "exit 99\n"); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = scenario.run_with_env( + &["secure", "harness", harness, "email", "me@example.com"], + &resend_env(&resend), + ); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!(result.stderr.contains(cause), "{}", result.stderr); + let (_, text) = the_one_notification(&resend); + assert!(text.contains(cause), "{text}"); + assert!(!text.contains("Private refusal evidence."), "{text}"); + let state = scenario.gh_state(); + for record in state["advisories"].as_array().unwrap() { + assert!(record["severity"].is_null()); + assert!( + !record["description"] + .as_str() + .unwrap() + .contains("## Reproduction") + ); + } + assert_eq!( + scenario.claude_calls().len() + scenario.codex_calls().len(), + 2 + ); + assert_eq!(scenario.entries("work"), vec![REPO]); + } +} + +#[test] +fn a_reproduction_scores_the_finding_and_keeps_its_test_in_the_private_record() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("input-size")]).to_string()), + ); + scenario.agent_does_in_session(2, r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > prompt.txt +test_file=$(sed -n 's/^Test file: `\(.*\)`\.$/\1/p' prompt.txt) +test -n "$test_file" +printf '%s\n' 'assert_eq!(bounded_fixture(), "overflow");' > "$test_file" +printf '%s\n' 'A harmless local fixture crossed the storage bound. Likelihood high, impact high; one session can hold the fix.' 'Security reproduction: reproduced high single' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let record = &state["advisories"][0]; + assert_eq!(record["severity"], "high"); + let body = record["description"].as_str().unwrap(); + assert!(body.contains("Likelihood high, impact high"), "{body}"); + assert!( + body.contains("assert_eq!(bounded_fixture(), \"overflow\");"), + "{body}" + ); + assert!(body.contains("Fix size: single"), "{body}"); + assert!( + result + .stderr + .contains("reproduction 1: reproduced high single"), + "{}", + result.stderr + ); + assert_eq!(scenario.claude_calls().len(), 2); + scenario.assert_every_session_found_the_factory_skills(); +} + +#[test] +fn a_private_reproduction_writes_severity_size_notes_and_test_into_the_finding_issue() { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + scenario.write_gh_state(&github); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("input-size")]).to_string()), + ); + scenario.agent_does_in_session(2, &reproduction_script("reproduced medium spec")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let body = state["bodies"]["8"].as_str().unwrap(); + for required in [ + "Fingerprint: `input-size`", + "Private candidate write-up.", + "Severity: medium", + "Fix size: spec", + "Local command: bounded-fixture", + "bounded_fixture();", + ] { + assert!(body.contains(required), "missing {required}: {body}"); + } + assert_eq!( + state["labels"]["8"], + json!(["security-finding", "needs-triage"]) + ); + assert_eq!(state["issues"]["8"], "OPEN"); + assert!(result.stderr.contains("issue #8"), "{}", result.stderr); + assert!(!result.stderr.contains("Local command: bounded-fixture")); +} + +fn reproduction_script(outcome: &str) -> String { + format!( + r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > prompt.txt +test_file=$(sed -n 's/^Test file: `\(.*\)`\.$/\1/p' prompt.txt) +test -n "$test_file" +printf '%s\n' 'bounded_fixture();' > "$test_file" +printf '%s\n' 'Local command: bounded-fixture; harmless local evidence and likelihood/impact reasoning.' 'Security reproduction: {outcome}' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"# + ) +} + +fn with_reproduced_findings(severities: &[&str]) -> Scenario { + let scenario = Scenario::new(); + scenario.repo_has_labels(&["needs-triage", "ready-for-agent"]); + let commit = scenario.origin_git(&["rev-parse", "main"]); + let commit = commit.trim_end(); + let mut state = scenario.gh_state(); + state["advisories"] = json!(severities.iter().enumerate().map(|(index, severity)| { + json!({ + "ghsa_id": format!("GHSA-finding-{index}"), "state": "draft", + "summary": format!("Finding {index}"), "severity": severity, + "html_url": format!("https://github.com/acme/widgets/security/advisories/GHSA-finding-{index}"), + "description": format!("Fingerprint: `finding-{index}`\nAudited commit: `{commit}`\nPrivate candidate write-up.\n\n<!-- thirdshift:security-reproduction -->\n## Reproduction\n\nOutcome: reproduced {severity} single\nSeverity: {severity}\nFix size: single\n\nPrivate reproduction notes.\n\n### Proof-of-concept test\n\n```\nbounded_fixture();\n```\n") + }) + }).collect::<Vec<_>>()); + scenario.write_gh_state(&state); + scenario +} + +fn publish_fix(record: &str) -> String { + format!( + r#" +url=$(gh issue create --title "Bound accepted input" --body "Reject oversized input. Private record: https://github.com/acme/widgets/security/advisories/{record}" --label needs-triage) +printf 'Security fix Ticket: %s\n' "$url" > "$FAKE_CLAUDE_FINAL_MESSAGE" +"# + ) +} + +fn implement_fix() -> &'static str { + r#" +echo bounded > bounded.txt +git add bounded.txt +git commit -q -m 'Bound accepted input' +gh pr create --base main --head issue-8 --title 'Bound accepted input' --body 'Closes #8' +"# +} + +fn publish_spec(record: &str) -> String { + format!( + r#" +url=$(gh issue create --title "Bound accepted input" --body "Bound input across storage and transport. Private record: {record}" --label needs-triage) +gh issue create --title "Bound storage input" --body "Bound storage input. Private record: {record}" --label ready-for-agent +gh issue create --title "Bound transport input" --body "Bound transport input. Private record: {record}" --label ready-for-agent +gh fake sub-issues 8 '[9,10]' +printf 'Security fix Spec: %s\n' "$url" > "$FAKE_CLAUDE_FINAL_MESSAGE" +"# + ) +} + +fn implement_spec_ticket(ticket: u32, spec: u32) -> String { + format!( + r#" +echo bounded > bounded-{ticket}.txt +git add bounded-{ticket}.txt +git commit -q -m 'Bound accepted input' +gh pr create --base issue-{spec} --head issue-{ticket} --title 'Bound accepted input' --body 'Closes #{ticket}' +"# + ) +} + +#[test] +fn a_bigger_public_fix_publishes_tickets_and_ends_as_the_spec_run() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + state["advisories"][0]["description"] = json!( + state["advisories"][0]["description"] + .as_str() + .unwrap() + .replace("high single", "high spec") + .replace("Fix size: single", "Fix size: spec") + ); + state["blocked_by"]["10"] = json!([9]); + scenario.write_gh_state(&state); + scenario.agent_does_in_session( + 1, + &publish_spec("https://github.com/acme/widgets/security/advisories/GHSA-finding-0"), + ); + scenario.agent_does_for(9, &implement_spec_ticket(9, 8)); + scenario.agent_does_for( + 10, + &format!("test -f bounded-9.txt\n{}", implement_spec_ticket(10, 8)), + ); + let result = scenario.run(&["secure", "security-fix", "parallel", "1"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let spec_pr = state["prs"] + .as_array() + .unwrap() + .iter() + .find(|pr| pr["head"] == "issue-8") + .unwrap(); + assert_eq!( + result.stdout, + format!("{}\n", spec_pr["url"].as_str().unwrap()) + ); + assert_eq!(spec_pr["base"], "main"); + assert_eq!(spec_pr["state"], "OPEN"); + assert_eq!(spec_pr["isDraft"], false); + for ticket in [9, 10] { + assert_eq!(state["issues"][ticket.to_string()], "CLOSED"); + assert_eq!( + state["labels"][ticket.to_string()], + json!(["ready-for-agent", "security-fix"]) + ); + } + assert_eq!(state["labels"]["8"], json!(["security-fix", "in-progress"])); + let calls = scenario.claude_calls(); + assert!( + calls[0]["prompt"] + .as_str() + .unwrap() + .contains("thirdshift-to-spec") + ); + assert!( + calls[0]["prompt"] + .as_str() + .unwrap() + .contains("thirdshift-to-tickets") + ); + assert_eq!(calls.len(), 4); +} + +#[test] +fn invalid_security_fix_specs_are_not_marked_ready_or_dispatched() { + let record = "https://github.com/acme/widgets/security/advisories/GHSA-finding-0"; + for script in [ + format!("{}gh fake sub-issues 8 '[]'\n", publish_spec(record)), + publish_spec(record).replace("Security fix Spec:", "Security fix Ticket:"), + publish_spec(record).replace("Bound storage input. Private record:", "Private candidate write-up. Private record:"), + publish_spec(record).replace("Bound storage input. Private record: https://github.com/acme/widgets/security/advisories/GHSA-finding-0", "Bound storage input."), + format!("{}gh fake created 9 '2020-01-01T00:00:00Z'\n", publish_spec(record)), + format!("{}gh issue close 9\n", publish_spec(record)), + format!("{}gh fake labels 9 '[\"ready-for-human\"]'\n", publish_spec(record)), + format!("{}gh fake sub-issues 9 '[7]'\n", publish_spec(record)), + ] { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + state["advisories"][0]["description"] = json!(state["advisories"][0]["description"].as_str().unwrap().replace("high single", "high spec")); + scenario.write_gh_state(&state); + scenario.agent_does_in_session(1, &script); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(1), "{script}: {}", result.stderr); + assert_eq!(scenario.gh_state()["labels"]["8"], json!(["needs-triage"])); + assert!(scenario.gh_state()["prs"].as_array().unwrap().is_empty()); + assert_eq!(scenario.claude_calls().len(), 1); + } +} + +#[test] +fn fixing_selects_the_most_severe_record_then_dispatches_one_ticket_before_auditing() { + let scenario = with_reproduced_findings(&["low", "critical", "critical", "high"]); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-1")); + scenario.agent_does_for(8, implement_fix()); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(result.stdout, "https://github.com/acme/widgets/pull/1\n"); + assert!( + result + .stderr + .ends_with("PR https://github.com/acme/widgets/pull/1 is ready for review\n"), + "{}", + result.stderr + ); + let state = scenario.gh_state(); + assert_eq!(state["labels"]["8"], json!(["security-fix", "in-progress"])); + let calls = scenario.claude_calls(); + assert_eq!(calls.len(), 2); + let prompt = calls[0]["prompt"].as_str().unwrap(); + assert!(prompt.contains("GHSA-finding-1"), "{prompt}"); + assert!(!prompt.contains("GHSA-finding-2")); + assert!(!prompt.contains("Security audit: complete")); + let ticket = state["bodies"]["8"].as_str().unwrap(); + assert!(ticket.contains("GHSA-finding-1")); + assert!(!ticket.contains("Private candidate write-up")); + assert!( + state["advisories"][1]["description"] + .as_str() + .unwrap() + .contains("https://github.com/acme/widgets/issues/8") + ); +} + +#[test] +fn a_fix_ticket_title_cannot_copy_the_private_write_up() { + let scenario = with_reproduced_findings(&["high"]); + scenario.agent_does_in_session( + 1, + &publish_fix("GHSA-finding-0") + .replace("Bound accepted input", "Private candidate write-up."), + ); + scenario.agent_does_for(8, implement_fix()); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("includes private write-up text"), + "{}", + result.stderr + ); + assert_eq!(scenario.gh_state()["labels"]["8"], json!(["needs-triage"])); + assert_eq!(scenario.claude_calls().len(), 1); +} + +#[test] +fn a_dispatched_fix_still_waits_without_permission_until_its_ticket_is_closed() { + let scenario = with_reproduced_findings(&["high"]); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for(8, implement_fix()); + let fixed = scenario.run(&["secure", "security-fix"]); + assert_eq!(fixed.code, Some(0), "{}", fixed.stderr); + let waiting = scenario.run(&["secure"]); + assert_eq!(waiting.code, Some(0), "{}", waiting.stderr); + assert!( + waiting + .stderr + .contains("a Security finding is waiting for the Day shift"), + "{}", + waiting.stderr + ); + assert_eq!(scenario.claude_calls().len(), 2); + let mut state = scenario.gh_state(); + state["issues"]["8"] = json!("CLOSED"); + scenario.write_gh_state(&state); + // Once fixed, the next run may audit the Base branch again. + scenario.agent_does(&audit_script("[]")); + let after = scenario.run(&["secure"]); + assert_eq!(after.code, Some(0), "{}", after.stderr); + assert!( + !after + .stderr + .contains("a Security finding is waiting for the Day shift") + ); +} + +#[test] +fn reproduced_findings_wait_without_permission_even_after_severity_is_written() { + for config in ["", "[security]\nfix = false\n", "[security]\nfix = true\n"] { + let scenario = with_reproduced_findings(&["high"]); + scenario.user_config_is(config); + let args = if config.contains("true") { + vec!["secure", "no-security-fix"] + } else { + vec!["secure"] + }; + let result = scenario.run(&args); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result + .stderr + .contains("a Security finding is waiting for the Day shift"), + "{}", + result.stderr + ); + assert!(scenario.claude_calls().is_empty()); + assert_eq!(scenario.gh_state()["issues"].as_object().unwrap().len(), 1); + } +} + +#[test] +fn a_private_one_session_fix_reuses_the_findings_issue_and_preserves_its_evidence() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + state["private"] = json!(true); + state["bodies"]["7"] = state["advisories"][0]["description"].clone(); + state["labels"]["7"] = json!(["security-finding", "needs-triage", "bug"]); + let evidence = state["bodies"]["7"].as_str().unwrap().to_string(); + scenario.write_gh_state(&state); + scenario.agent_does_for( + 7, + &implement_fix() + .replace("issue-8", "issue-7") + .replace("#8", "#7"), + ); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert_eq!(state["issues"].as_object().unwrap().len(), 1); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .starts_with(&evidence) + ); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .contains("Fix Ticket: https://github.com/acme/widgets/issues/7") + ); + assert_eq!( + state["labels"]["7"], + json!(["security-finding", "bug", "security-fix", "in-progress"]) + ); + assert_eq!(state["prs"][0]["head"], "issue-7"); + assert_eq!(scenario.claude_calls().len(), 1); +} + +#[test] +fn a_bigger_private_fix_adds_tickets_to_the_findings_issue() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + state["private"] = json!(true); + state["bodies"]["7"] = json!( + state["advisories"][0]["description"] + .as_str() + .unwrap() + .replace("high single", "high spec") + .replace("Fix size: single", "Fix size: spec") + ); + state["labels"]["7"] = json!(["security-finding", "needs-triage"]); + state["blocked_by"]["9"] = json!([8]); + let evidence = state["bodies"]["7"].as_str().unwrap().to_string(); + scenario.write_gh_state(&state); + scenario.agent_does_for_in_session(7, 1, r#" +gh issue create --title 'Bound storage input' --body 'Bound storage input. Private record: https://github.com/acme/widgets/issues/7' --label ready-for-agent +gh issue create --title 'Bound transport input' --body 'Bound transport input. Private record: https://github.com/acme/widgets/issues/7' --label ready-for-agent +gh fake sub-issues 7 '[8,9]' +printf '%s\n' 'Security fix Spec: https://github.com/acme/widgets/issues/7' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#); + scenario.agent_does_for_in_session(7, 2, "true"); + scenario.agent_does_for(8, &implement_spec_ticket(8, 7)); + scenario.agent_does_for( + 9, + &format!("test -f bounded-8.txt\n{}", implement_spec_ticket(9, 7)), + ); + let result = scenario.run(&["secure", "security-fix", "merge", "parallel", "1"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert_eq!(state["issues"].as_object().unwrap().len(), 3); + assert_eq!(state["sub_issues"]["7"], json!([8, 9])); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .starts_with(&evidence) + ); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .contains("Fix Ticket: https://github.com/acme/widgets/issues/7") + ); + assert_eq!(state["issues"]["7"], "CLOSED"); + let spec_pr = state["prs"] + .as_array() + .unwrap() + .iter() + .find(|pr| pr["head"] == "issue-7") + .unwrap(); + assert_eq!(spec_pr["state"], "MERGED"); + assert_eq!(spec_pr["base"], "main"); + assert_eq!( + result.stdout, + format!("{}\n", spec_pr["url"].as_str().unwrap()) + ); + for ticket in [8, 9] { + assert!( + state["bodies"][ticket.to_string()] + .as_str() + .unwrap() + .contains("https://github.com/acme/widgets/issues/7") + ); + assert_eq!( + state["labels"][ticket.to_string()], + json!(["ready-for-agent", "security-fix"]) + ); + } +} + +#[test] +fn the_setting_allows_a_merge_fix_and_the_command_can_override_either_setting() { + for (config, word, merged) in [ + ( + "[security]\nfix = true\n[merge]\nalways = true\n", + None, + true, + ), + ("[security]\nfix = false\n", Some("security-fix"), false), + ( + "[security]\nfix = true\n[merge]\nalways = true\n", + Some("no-merge"), + false, + ), + ] { + let scenario = with_reproduced_findings(&["medium"]); + scenario.user_config_is(config); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for(8, implement_fix()); + let mut args = vec!["secure"]; + args.extend(word); + let result = scenario.run(&args); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["prs"][0]["state"], + if merged { "MERGED" } else { "OPEN" } + ); + assert_eq!(scenario.claude_calls().len(), 2); + } +} + +#[test] +fn an_audit_goes_on_to_one_reproduced_fix_when_fixing_is_allowed() { + let scenario = Scenario::new(); + scenario.repo_has_labels(&["needs-triage", "ready-for-agent"]); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("first"), finding("second")]).to_string()), + ); + scenario.agent_does_in_session(2, &reproduction_script("reproduced high single")); + scenario.agent_does_in_session(3, &reproduction_script("reproduced critical spec")); + scenario.agent_does_in_session( + 4, + &publish_spec("https://github.com/acme/widgets/security/advisories/GHSA-test-test-0002"), + ); + scenario.agent_does_for(9, &implement_spec_ticket(9, 8)); + scenario.agent_does_for(10, &implement_spec_ticket(10, 8)); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let spec_pr = state["prs"] + .as_array() + .unwrap() + .iter() + .find(|pr| pr["head"] == "issue-8") + .unwrap(); + assert_eq!( + result.stdout, + format!("{}\n", spec_pr["url"].as_str().unwrap()) + ); + assert_eq!(scenario.claude_calls().len(), 7); + assert_eq!(scenario.gh_state()["issues"].as_object().unwrap().len(), 4); + assert!( + scenario.gh_state()["bodies"]["8"] + .as_str() + .unwrap() + .contains("GHSA-test-test-0002") + ); +} + +#[test] +fn the_security_run_ends_as_a_failed_fix_and_sends_one_private_metadata_notification() { + let scenario = with_reproduced_findings(&["critical"]); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for(8, &format!("{}exit 1\n", implement_fix())); + let resend = ResendStandIn::replying(200, r#"{"id":"sent"}"#); + let result = scenario.run_with_env( + &["secure", "security-fix", "email", "day@example.com"], + &resend_env(&resend), + ); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("claude exited 1"), + "{}", + result.stderr + ); + assert_eq!( + scenario.gh_state()["labels"]["8"], + json!(["security-fix", "in-progress"]) + ); + let (subject, body) = the_one_notification(&resend); + assert_eq!(subject, "[thirdshift] acme/widgets Security run: failed"); + assert!(body.contains("critical")); + assert!(body.contains("GHSA-finding-0")); + assert!(!body.contains("Private candidate write-up")); + assert!(!body.contains("bounded_fixture")); +} + +#[test] +fn a_failed_fix_keeps_its_claim_and_pauses_security_and_pickup_until_closed() { + for private in [false, true] { + for pushed in [false, true] { + let scenario = with_reproduced_findings(&["critical"]); + let issue = if private { 7 } else { 8 }; + if private { + let mut state = scenario.gh_state(); + state["private"] = json!(true); + state["bodies"]["7"] = state["advisories"][0]["description"].clone(); + state["labels"]["7"] = json!(["security-finding", "needs-triage"]); + scenario.write_gh_state(&state); + } else { + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + } + scenario.agent_does_for( + issue, + &format!( + "{}exit 1\n", + if pushed { implement_fix() } else { "" } + .replace("issue-8", &format!("issue-{issue}")) + .replace("#8", &format!("#{issue}")) + ), + ); + let failed = scenario.run(&["secure", "security-fix"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + assert_eq!( + scenario.issue_labels(issue), + if private { + vec!["security-finding", "security-fix", "in-progress"] + } else { + vec!["security-fix", "in-progress"] + } + ); + // Even contradictory ready labelling cannot make Pickup retry a Claim. + scenario.issue_labelled( + issue, + if private { + &[ + "security-finding", + "security-fix", + "in-progress", + "ready-for-agent", + ] + } else { + &["security-fix", "in-progress", "ready-for-agent"] + }, + ); + let reason = format!("failed Security fix #{issue} is still open"); + let resend = ResendStandIn::replying(200, r#"{"id":"sent"}"#); + for args in [ + vec!["secure", "security-fix", "email", "day@example.com"], + vec!["secure", "no-security-fix", "email", "day@example.com"], + vec!["pickup"], + ] { + let skipped = scenario.run_with_env(&args, &resend_env(&resend)); + assert_eq!(skipped.code, Some(0), "{}", skipped.stderr); + assert!(skipped.stdout.is_empty()); + if args[0] == "secure" { + assert!(skipped.stderr.contains(&reason), "{}", skipped.stderr); + } + } + let fix_sessions = if private { 1 } else { 2 }; + assert_eq!(scenario.claude_calls().len(), fix_sessions); + assert!(resend.requests().is_empty()); + assert_eq!( + scenario + .entries("home/.thirdshift/logs/acme/widgets/commands/secure") + .len(), + 1 + ); + let activity = fs::read_to_string( + scenario.path("home/.thirdshift/logs/acme/widgets/activity.log"), + ) + .unwrap(); + assert!( + activity + .matches(&format!("Security run skipped: {reason}")) + .count() + == 1, + "{activity}" + ); + let mut state = scenario.gh_state(); + state["issues"][issue.to_string()] = json!("CLOSED"); + scenario.write_gh_state(&state); + scenario.agent_does(&audit_script("[]")); + let after = scenario.run(&["secure", "security-fix"]); + assert_eq!(after.code, Some(0), "{}", after.stderr); + assert!( + after + .stderr + .contains("Security audit recorded 0 new finding(s)"), + "{}", + after.stderr + ); + assert_eq!(scenario.claude_calls().len(), fix_sessions + 1); + } + } +} + +#[test] +fn a_failed_spec_fix_keeps_its_claim_and_pauses_security_and_pickup_until_closed() { + for private in [false, true] { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + let evidence = state["advisories"][0]["description"] + .as_str() + .unwrap() + .replace("high single", "high spec") + .replace("Fix size: single", "Fix size: spec"); + let (spec, ticket) = if private { (7, 8) } else { (8, 9) }; + if private { + state["private"] = json!(true); + state["bodies"]["7"] = json!(evidence); + state["labels"]["7"] = json!(["security-finding", "needs-triage"]); + scenario.agent_does_for_in_session(7, 1, r#" +gh issue create --title 'Bound storage input' --body 'Bound storage input. Private record: https://github.com/acme/widgets/issues/7' --label ready-for-agent +gh issue create --title 'Bound transport input' --body 'Bound transport input. Private record: https://github.com/acme/widgets/issues/7' --label ready-for-agent +gh fake sub-issues 7 '[8,9]' +printf '%s\n' 'Security fix Spec: https://github.com/acme/widgets/issues/7' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#); + } else { + state["advisories"][0]["description"] = json!(evidence); + scenario.agent_does_in_session( + 1, + &publish_spec("https://github.com/acme/widgets/security/advisories/GHSA-finding-0"), + ); + } + state["blocked_by"][(ticket + 1).to_string()] = json!([ticket]); + scenario.write_gh_state(&state); + scenario.agent_does_for(ticket, "exit 1\n"); + let failed = scenario.run(&["secure", "security-fix", "parallel", "1"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + assert!( + failed.stderr.contains("claude exited 1"), + "{}", + failed.stderr + ); + assert_eq!( + scenario.issue_labels(spec), + if private { + vec!["security-finding", "security-fix", "in-progress"] + } else { + vec!["security-fix", "in-progress"] + } + ); + let state = scenario.gh_state(); + assert_eq!(state["issues"][spec.to_string()], "OPEN"); + assert_eq!(state["issues"][ticket.to_string()], "OPEN"); + let description = if private { + &state["bodies"]["7"] + } else { + &state["advisories"][0]["description"] + } + .as_str() + .unwrap(); + assert!(description.starts_with(&evidence), "{description}"); + assert!( + description.contains(&format!( + "Fix Ticket: https://github.com/acme/widgets/issues/{spec}" + )), + "{description}" + ); + assert!(description.ends_with("Fix Run: failed\n"), "{description}"); + assert_eq!(scenario.claude_calls().len(), 2); + let reason = format!("failed Security fix #{spec} is still open"); + for permission in ["security-fix", "no-security-fix"] { + let skipped = scenario.run(&["secure", permission]); + assert_eq!(skipped.code, Some(0), "{}", skipped.stderr); + assert!(skipped.stderr.contains(&reason), "{}", skipped.stderr); + } + let pickup = scenario.run(&["pickup"]); + assert_eq!(pickup.code, Some(0), "{}", pickup.stderr); + assert!( + !pickup.stderr.contains("taking Ready issue"), + "{}", + pickup.stderr + ); + assert_eq!(scenario.claude_calls().len(), 2); + let mut state = scenario.gh_state(); + state["issues"][spec.to_string()] = json!("CLOSED"); + scenario.write_gh_state(&state); + scenario.agent_does(&audit_script("[]")); + let after = scenario.run(&["secure", "security-fix"]); + assert_eq!(after.code, Some(0), "{}", after.stderr); + assert!( + after + .stderr + .contains("Security audit recorded 0 new finding(s)"), + "{}", + after.stderr + ); + assert_eq!(scenario.claude_calls().len(), 3); + } +} + +#[test] +fn a_successful_fix_awaiting_review_does_not_pause_allowed_security_work() { + let scenario = with_reproduced_findings(&["high"]); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for(8, implement_fix()); + let fixed = scenario.run(&["secure", "security-fix"]); + assert_eq!(fixed.code, Some(0), "{}", fixed.stderr); + scenario.agent_does(&audit_script("[]")); + let audited = scenario.run(&["secure", "security-fix"]); + assert_eq!(audited.code, Some(0), "{}", audited.stderr); + assert!( + audited + .stderr + .contains("Security audit recorded 0 new finding(s)"), + "{}", + audited.stderr + ); + assert_eq!(scenario.claude_calls().len(), 3); +} + +#[test] +fn an_undecided_run_offers_both_ways_to_allow_a_reproduced_fix() { + for (config, permission, reproduced, offers) in [ + ("", None, true, true), + ("", Some("no-security-fix"), true, false), + ("[security]\nfix = false\n", None, true, false), + ( + "[security]\nfix = true\n", + Some("no-security-fix"), + true, + false, + ), + ("", None, false, false), + ] { + let scenario = Scenario::new(); + scenario.user_config_is(config); + scenario.agent_does_in_session(1, &audit_script(&json!([finding("offer")]).to_string())); + scenario.agent_does_in_session( + 2, + &reproduction_script(if reproduced { + "reproduced high single" + } else { + "not reproduced" + }), + ); + let resend = ResendStandIn::replying(200, r#"{"id":"sent"}"#); + let mut args = vec![ + "secure", + "base", + "main", + "harness", + "claude", + "email", + "day@example.com", + ]; + if let Some(permission) = permission { + args.push(permission); + } + let result = scenario.run_with_env(&args, &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let (_, body) = the_one_notification(&resend); + for text in [&result.stderr, &body] { + assert_eq!( + text.contains( + "thirdshift secure base main harness claude email day@example.com security-fix" + ), + offers, + "{text}" + ); + assert_eq!( + text.contains("fix = true under [security]"), + offers, + "{text}" + ); + assert!(!text.contains("Private candidate write-up"), "{text}"); + } + assert_eq!(scenario.claude_calls().len(), 2); + } +} + +#[test] +fn invalid_fix_tickets_are_not_marked_ready_or_dispatched() { + for script in [ + "printf '%s\\n' 'No protocol line' > \"$FAKE_CLAUDE_FINAL_MESSAGE\"\n".to_string(), + publish_fix("GHSA-finding-0") + .replace("\"$url\"", "\"https://github.com/other/widgets/issues/8\""), + publish_fix("GHSA-finding-0") + .replace("Reject oversized input.", "Private candidate write-up."), + publish_fix("GHSA-finding-0").replace("GHSA-finding-0", "GHSA-other"), + format!( + "{}gh fake sub-issues 8 '[7]'\n", + publish_fix("GHSA-finding-0") + ), + format!("{}gh issue close 8\n", publish_fix("GHSA-finding-0")), + format!( + "{}gh fake created 8 '2020-01-01T00:00:00Z'\n", + publish_fix("GHSA-finding-0") + ), + ] { + let scenario = with_reproduced_findings(&["high"]); + scenario.agent_does_in_session(1, &script); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(1), "{script}: {}", result.stderr); + assert_eq!(scenario.claude_calls().len(), 1); + assert!( + !scenario.gh_state()["labels"]["8"] + .as_array() + .is_some_and(|labels| labels.contains(&json!("ready-for-agent"))) + ); + assert!(scenario.gh_state()["prs"].as_array().unwrap().is_empty()); + } +} + +#[test] +fn security_fix_words_are_accepted_once_on_every_command_that_starts_runs() { + for prefix in [ + vec!["https://github.com/acme/widgets/issues/7"], + vec!["architect"], + vec!["pickup"], + vec!["secure"], + ] { + for word in [ + "security-fix", + "--security-fix", + "no-security-fix", + "--no-security-fix", + ] { + let scenario = Scenario::new(); + scenario.user_config_is("[security]\nfix = 'invalid'\n"); + let mut args = prefix.clone(); + args.push(word); + let result = scenario.run(&args); + assert_eq!(result.code, Some(1), "{args:?}: {}", result.stderr); + assert!( + result.stderr.contains("security.fix must be true or false"), + "{}", + result.stderr + ); + } + for (words, error) in [ + ( + vec!["security-fix", "no-security-fix"], + "can't be used together", + ), + (vec!["security-fix", "--security-fix"], "repeated argument"), + ] { + let scenario = Scenario::new(); + let mut args = prefix.clone(); + args.extend(words); + let result = scenario.run(&args); + assert_eq!(result.code, Some(2), "{args:?}: {}", result.stderr); + assert!(result.stderr.contains(error)); + } + } +} + +#[test] +fn reproductions_are_sequential_fresh_and_pinned_to_the_audited_commit() { + let scenario = Scenario::new(); + let audited = scenario.origin_git(&["rev-parse", "main"]); + scenario.agent_does_in_session( + 1, + &format!( + r#"{} +touch audit-only +new=$(git commit-tree "$(git rev-parse 'HEAD^{{tree}}')" -p HEAD -m 'Base advanced') +git push origin "$new:refs/heads/main" +"#, + audit_script(&json!([finding("first"), finding("second")]).to_string()) + ), + ); + scenario.agent_does_in_session( + 2, + &format!( + r#" +test -z "$(git branch --show-current)" +test ! -e audit-only +git rev-parse HEAD >> "$HOME/../reproductions" +printf '%s\n' first >> "$HOME/../sequence" +touch previous-poc +{} +"#, + reproduction_script("reproduced low single") + ), + ); + scenario.agent_does_in_session( + 3, + &format!( + r#" +test ! -e previous-poc +test ! -e audit-only +test "$(cat "$HOME/../sequence")" = first +git rev-parse HEAD >> "$HOME/../reproductions" +printf '%s\n' second >> "$HOME/../sequence" +{} +"#, + reproduction_script("not reproduced") + ), + ); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_ne!(scenario.origin_git(&["rev-parse", "main"]), audited); + assert_eq!( + fs::read_to_string(scenario.path("reproductions")).unwrap(), + format!("{audited}{audited}") + ); + assert_eq!( + fs::read_to_string(scenario.path("sequence")).unwrap(), + "first\nsecond\n" + ); + let calls = scenario.claude_calls(); + assert_eq!(calls.len(), 3); + for (index, fingerprint) in [(1, "first"), (2, "second")] { + let prompt = calls[index]["prompt"].as_str().unwrap(); + for required in [ + "validation plan", + "untouched code", + "harmless payloads only", + "never a deployed site or a real third-party service", + "likelihood-and-impact", + "one session", + "Test file:", + "Security reproduction:", + "You run headless", + ] { + assert!(prompt.contains(required), "missing {required}: {prompt}"); + } + assert!( + prompt.contains(&format!("Fingerprint: `{fingerprint}`")), + "{prompt}" + ); + assert_eq!(calls[index]["branch"], ""); + } + assert_eq!(scenario.entries("work"), vec![REPO]); + scenario.assert_every_session_found_the_factory_skills(); +} + +#[test] +fn incomplete_reproductions_preserve_the_record_and_stop_before_the_next_session() { + for private in [false, true] { + for (outcome, cause) in [ + ("No final line", "without the final line"), + ("reproduced invalid single", "invalid severity"), + ("reproduced high invalid", "invalid fix size"), + ] { + let scenario = Scenario::new(); + let commit = scenario.origin_git(&["rev-parse", "main"]); + let original = format!( + "Fingerprint: `first`\nAudited commit: `{}`\nExisting private evidence.\n", + commit.trim() + ); + let mut github = scenario.gh_state(); + github["private"] = json!(private); + scenario.write_gh_state(&github); + if private { + github["issues"]["8"] = json!("OPEN"); + github["bodies"] = json!({"8": original}); + github["labels"] = json!({"8": ["security-finding", "needs-triage"]}); + } else { + github["advisories"] = json!([{ + "ghsa_id": "GHSA-existing", "description": original, "severity": null, "state": "draft", + "summary": "Existing finding", "html_url": "https://github.com/acme/widgets/security/advisories/GHSA-existing" + }]); + } + scenario.agent_does_in_session( + 1, + &audit_script_with_records( + &scenario, + &json!([finding("first"), finding("second")]).to_string(), + &github, + ), + ); + scenario.agent_does_in_session(2, &reproduction_script(outcome)); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!(result.stderr.contains(cause), "{}", result.stderr); + assert!( + result.stderr.contains("security-reproduction-1.jsonl"), + "{}", + result.stderr + ); + let state = scenario.gh_state(); + if private { + assert_eq!(state["bodies"]["8"], original); + assert_eq!(state["labels"]["8"], github["labels"]["8"]); + assert!( + state["bodies"]["9"] + .as_str() + .unwrap() + .contains("Fingerprint: `second`") + ); + } else { + assert_eq!(state["advisories"][0], github["advisories"][0]); + assert_eq!(state["advisories"].as_array().unwrap().len(), 2); + } + assert_eq!(scenario.claude_calls().len(), 2); + assert!( + scenario + .gh_calls_of("api", "--method") + .iter() + .all(|call| call[2] != "PATCH") + ); + assert_eq!(scenario.entries("work"), vec![REPO]); + } + } +} + +#[test] +fn an_unreproduced_finding_keeps_no_severity_and_keeps_the_test_and_notes() { + let scenario = Scenario::new(); + let commit = scenario.origin_git(&["rev-parse", "main"]); + let original = format!( + "Fingerprint: `input-size`\nAudited commit: `{}`\nExisting private evidence.\n", + commit.trim() + ); + let mut github = scenario.gh_state(); + github["advisories"] = json!([{ + "ghsa_id": "GHSA-existing", "description": original, "severity": null, "state": "draft", + "summary": "Existing finding", "html_url": "https://github.com/acme/widgets/security/advisories/GHSA-existing" + }]); + scenario.agent_does_in_session( + 1, + &audit_script_with_records( + &scenario, + &json!([finding("input-size")]).to_string(), + &github, + ), + ); + scenario.agent_does_in_session(2, &reproduction_script("not reproduced")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let record = &state["advisories"][0]; + assert_eq!(record["severity"], Value::Null); + let body = record["description"].as_str().unwrap(); + assert!(body.starts_with(original.trim_end()), "{body}"); + assert!(body.contains("Outcome: not reproduced"), "{body}"); + assert!(body.contains("Local command: bounded-fixture"), "{body}"); + assert!(body.contains("bounded_fixture();"), "{body}"); + assert!(!body.contains("Severity:"), "{body}"); + assert!(!body.contains("Fix size:"), "{body}"); + assert!( + result.stderr.contains("reproduction 1: not reproduced"), + "{}", + result.stderr + ); +} + +#[test] +fn records_a_finding_privately_as_a_draft_without_severity_or_versions() { + let scenario = Scenario::new(); + scenario.origin_has_commit( + "main", + "Cargo.toml", + "[package]\nname = \"widgets\"\nversion = \"1.2.3\"\n", + "Add manifest", + ); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let advisory = &state["advisories"][0]; + assert_eq!(advisory["state"], "draft"); + assert_eq!(advisory["summary"], "Unchecked input size"); + assert_eq!(advisory["severity"], Value::Null); + assert_eq!(advisory["cwe_ids"], json!([])); + assert_eq!( + advisory["vulnerabilities"][0]["package"], + json!({"ecosystem":"rust", "name":"widgets"}) + ); + assert_eq!( + advisory["vulnerabilities"][0]["vulnerable_version_range"], + Value::Null + ); + let description = advisory["description"] + .as_str() + .expect("no private advisory was recorded"); + for required in [ + "Private candidate write-up.", + "Private trace.", + "Private evidence.", + "Exercise a bounded fixture.", + "input-size", + "thirdshift's Security run", + scenario.origin_git(&["rev-parse", "main"]).trim(), + ] { + assert!( + description.contains(required), + "missing {required}: {description}" + ); + } + assert!(!result.stderr.contains("Private candidate write-up.")); + assert!(state["prs"].as_array().unwrap().is_empty()); + assert_eq!(state["issues"].as_object().unwrap().len(), 1); + assert_eq!( + scenario.origin_git(&["log", "--format=%s", "main"]), + "Add manifest\nInitial commit\n" + ); +} + +#[test] +fn a_private_repository_records_each_finding_as_a_labelled_issue_with_the_advisory_body() { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), finding("storage-bound")]).to_string(), + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert!(state["advisories"].is_null()); + assert_eq!(state["issues"].as_object().unwrap().len(), 3); + let commit = scenario.origin_git(&["rev-parse", "main"]); + for (number, fingerprint) in [("8", "input-size"), ("9", "storage-bound")] { + assert_eq!(state["titles"][number], "Unchecked input size"); + assert_eq!( + state["labels"][number], + json!(["security-finding", "needs-triage"]) + ); + let expected = format!( + "Found by thirdshift's Security run.\n\nFingerprint: `{fingerprint}`\nAudited commit: `{}`\n\nPrivate candidate write-up.\n\n```json\n{}\n```\n", + commit.trim(), + serde_json::to_string_pretty(&finding(fingerprint)).unwrap() + ); + let body = state["bodies"][number].as_str().unwrap(); + assert!(body.starts_with(&expected), "{body}"); + assert!(body.contains("Outcome: not reproduced"), "{body}"); + assert!(!body.contains("Severity:"), "{body}"); + } + assert!(!result.stderr.contains("Private candidate write-up.")); +} + +#[test] +fn repeat_private_audits_match_open_and_closed_findings_after_triage_without_duplicates() { + for state in ["OPEN", "CLOSED"] { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), finding("input-size-extra")]).to_string(), + )); + let first = scenario.run(&["secure"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + assert!( + first + .stderr + .contains("2 new finding(s), 0 already recorded"), + "{}", + first.stderr + ); + let mut github = scenario.gh_state(); + for number in ["8", "9"] { + github["issues"][number] = json!(state); + github["labels"][number] = json!(["Security-Finding"]); + } + scenario.write_gh_state(&github); + scenario.origin_has_commit("main", "new-work.txt", "new work", "Move Base branch"); + let second = scenario.run(&["secure"]); + assert_eq!(second.code, Some(0), "{}", second.stderr); + assert!( + second + .stderr + .contains("0 new finding(s), 2 already recorded"), + "{}", + second.stderr + ); + assert_eq!(scenario.gh_state(), github); + assert_eq!(scenario.gh_calls_of("issue", "create").len(), 2); + } +} + +#[test] +fn private_finding_labels_are_created_with_descriptions_and_existing_labels_are_kept() { + for existing in [ + vec!["bug"], + vec!["bug", "Security-Finding"], + vec!["bug", "NEEDS-TRIAGE"], + vec!["bug", "Security-Finding", "NEEDS-TRIAGE"], + ] { + let scenario = Scenario::new(); + scenario.repo_has_labels(&existing); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let created = scenario.gh_calls_of("label", "create"); + let mut expected_labels: Vec<String> = + existing.iter().map(|name| name.to_string()).collect(); + let mut missing = 0; + for (label, description) in [ + ( + "security-finding", + "A Security finding recorded privately for the Day shift", + ), + ("needs-triage", "Maintainer needs to evaluate this issue"), + ] { + if !existing.iter().any(|name| name.eq_ignore_ascii_case(label)) { + missing += 1; + assert!( + created.iter().any(|call| call[2] == label + && call + .windows(2) + .any(|args| args == ["--description", description])), + "{created:?}" + ); + expected_labels.push(label.to_string()); + } + } + assert_eq!(created.len(), missing); + assert_eq!(scenario.repo_labels(), expected_labels); + assert_eq!( + scenario.issue_labels(8), + ["security-finding", "needs-triage"] + ); + } +} + +#[test] +fn advisory_errors_never_record_findings_in_a_public_repository_or_on_non_404_failures() { + for (private, error) in [ + (false, "gh: Not Found (HTTP 404)"), + (true, "gh: Forbidden (HTTP 403)"), + (true, "gh: Bad Gateway (HTTP 502)"), + ] { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(private); + github["advisory_error"] = json!(error); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_ne!(result.code, Some(0), "{}", result.stderr); + assert!(scenario.gh_calls_of("issue", "create").is_empty()); + assert!(scenario.gh_calls_of("label", "create").is_empty()); + assert_eq!(scenario.gh_state()["issues"].as_object().unwrap().len(), 1); + assert!(!result.stderr.contains("Private candidate write-up.")); + } +} + +#[test] +fn reads_the_package_from_an_npm_manifest() { + let scenario = Scenario::new(); + scenario.origin_has_commit( + "main", + "package.json", + r#"{"name":"@acme/widgets","version":"1.0.0"}"#, + "Add npm manifest", + ); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + json!({"ecosystem":"npm", "name":"@acme/widgets"}) + ); +} + +fn audit_script(findings: &str) -> String { + format!( + r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > prompt.txt +test_file=$(sed -n 's/^Test file: `\(.*\)`\.$/\1/p' prompt.txt) +if [ -n "$test_file" ]; then + printf '%s\n' 'bounded_fixture();' > "$test_file" + printf '%s\n' 'The harmless bounded fixture did not reproduce the claim.' 'Security reproduction: not reproduced' > "$FAKE_CLAUDE_FINAL_MESSAGE" + exit 0 +fi +output=$(sed -n 's/^Output directory: `\(.*\)`\.$/\1/p' prompt.txt) +test -n "$output" +printf '%s\n' '{findings}' > "$output/findings.json" +printf '%s\n' '[]' > "$output/coverage-ledger.json" +printf '%s\n' '{{"run_status":"complete"}}' > "$output/run-metadata.json" +printf '%s\n' 'Security audit: complete' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"# + ) +} + +/// A finding recorded during the audit is first seen after the waiting gate. +fn audit_script_with_records(scenario: &Scenario, findings: &str, records: &Value) -> String { + fs::write( + scenario.path("audit-records.json"), + serde_json::to_vec(records).unwrap(), + ) + .unwrap(); + format!( + "cp \"$HOME/../audit-records.json\" \"$FAKE_GH_STATE\"\n{}", + audit_script(findings) + ) +} + +fn assert_two_quiet_skips_send_no_email(scenario: &Scenario) { + let resend = support::resend::ResendStandIn::replying(200, r#"{"id":"unused"}"#); + for _ in 0..2 { + let skipped = scenario.run_with_env( + &["secure", "email", "me@example.com"], + &[ + ("THIRDSHIFT_RESEND_URL", resend.url()), + ("RESEND_API_KEY", "re_test"), + ], + ); + assert_eq!(skipped.code, Some(0), "{}", skipped.stderr); + assert_eq!(skipped.stdout, ""); + assert_eq!(skipped.stderr, ""); + } + assert!(resend.requests().is_empty()); +} + +#[test] +fn an_unchanged_base_skips_after_a_completed_audit_and_resumes_when_origin_moves() { + let scenario = Scenario::new(); + scenario.user_config_is("[activity]\nquiet_skips = true\n"); + scenario.agent_does(&audit_script("[]")); + let first = scenario.run(&["secure"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + let before = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + let commands = scenario.entries("home/.thirdshift/logs/acme/widgets/commands/secure"); + assert_two_quiet_skips_send_no_email(&scenario); + assert_eq!(scenario.claude_calls().len(), 1); + assert_eq!( + scenario.entries("home/.thirdshift/logs/acme/widgets/commands/secure"), + commands + ); + let after = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert_eq!(after.lines().count(), before.lines().count() + 1); + assert!( + after.contains("Base branch main hasn't changed since the last completed Security audit"), + "{after}" + ); + scenario.origin_has_commit("main", "change.txt", "new work", "Change Base branch"); + let next = scenario.run(&["secure"]); + assert_eq!(next.code, Some(0), "{}", next.stderr); + assert_eq!(scenario.claude_calls().len(), 2); +} + +#[test] +fn waiting_findings_skip_quietly_once_and_triage_allows_the_next_audit() { + for triage in [ + "closed", + "published", + "severity", + "private-closed", + "private-labelled", + ] { + let scenario = Scenario::new(); + scenario.user_config_is("[activity]\nquiet_skips = true\n"); + scenario.agent_does(&audit_script("[]")); + let mut github = scenario.gh_state(); + let private = triage.starts_with("private-"); + if private { + github["private"] = json!(true); + github["labels"]["7"] = json!(["security-finding", "Needs-Triage"]); + } else { + github["advisories"] = json!([{"state":"draft", "severity":null, "summary":"Private title", "description":"Private evidence"}]); + } + scenario.write_gh_state(&github); + assert_two_quiet_skips_send_no_email(&scenario); + assert!(scenario.claude_calls().is_empty()); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/commands") + .exists() + ); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/audits") + .exists() + ); + let log = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert_eq!(log.lines().count(), 1, "{log}"); + assert!( + log.contains("Security run skipped: a Security finding is waiting for the Day shift") + ); + assert!(!log.contains("Private")); + assert_eq!(scenario.gh_state(), github); + match triage { + "private-closed" => github["issues"]["7"] = json!("CLOSED"), + "private-labelled" => github["labels"]["7"] = json!(["security-finding"]), + "severity" => github["advisories"][0]["severity"] = json!("high"), + state => github["advisories"][0]["state"] = json!(state), + } + scenario.write_gh_state(&github); + let next = scenario.run(&["secure"]); + assert_eq!(next.code, Some(0), "{triage}: {}", next.stderr); + assert_eq!(scenario.claude_calls().len(), 1, "{triage}"); + } +} + +#[test] +fn incomplete_or_rejected_audit_artifacts_do_not_hold_the_next_attempt() { + for script in [ + audit_script("[{}]"), + audit_script("[]").replace("Security audit: complete", "Missing protocol line"), + audit_script("[]").replace("Security audit: complete", "Security audit: incomplete"), + audit_script("[]").replace( + "\"run_status\":\"complete\"", + "\"run_status\":\"incomplete\"", + ), + format!("{}\nexit 1\n", audit_script("[]")), + ] { + let scenario = Scenario::new(); + scenario.agent_does(&script); + let failed = scenario.run(&["secure"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + let runs = scenario.entries("home/.thirdshift/logs/acme/widgets/audits"); + let record: Value = serde_json::from_slice( + &fs::read(scenario.path(&format!( + "home/.thirdshift/logs/acme/widgets/audits/{}/run-metadata.json", + runs[0] + ))) + .unwrap(), + ) + .unwrap(); + assert_eq!(record["run_status"], "incomplete"); + scenario.agent_does(&audit_script("[]")); + let retry = scenario.run(&["secure"]); + assert_eq!(retry.code, Some(0), "{}", retry.stderr); + assert_eq!(scenario.claude_calls().len(), 2); + } +} + +#[test] +fn keeps_fingerprints_in_every_advisory_state_and_creates_each_new_finding_once() { + for state in ["draft", "published", "closed", "triage"] { + let scenario = Scenario::new(); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), finding("storage-bound")]).to_string(), + )); + let first = scenario.run(&["secure"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + let mut github = scenario.gh_state(); + assert_eq!(github["advisories"].as_array().unwrap().len(), 2); + for advisory in github["advisories"].as_array_mut().unwrap() { + advisory["state"] = json!(state); + if state == "draft" { + advisory["severity"] = json!("low"); + } + } + scenario.write_gh_state(&github); + scenario.origin_has_commit("main", "new-work.txt", "new work", "Move Base branch"); + let second = scenario.run(&["secure"]); + assert_eq!(second.code, Some(0), "{}", second.stderr); + assert!( + second + .stderr + .contains("0 new finding(s), 2 already recorded"), + "{}", + second.stderr + ); + assert_eq!(scenario.gh_state()["advisories"], github["advisories"]); + let api_calls = scenario.gh_calls_of("api", "--method"); + assert_eq!(api_calls.iter().filter(|call| call[2] == "POST").count(), 2); + assert_eq!( + api_calls.iter().filter(|call| call[2] == "PATCH").count(), + 2 + ); + assert_eq!( + scenario + .entries("home/.thirdshift/logs/acme/widgets/audits") + .len(), + 2 + ); + } +} + +#[test] +fn an_explicit_base_and_harness_audit_origin_without_pulling_local_work() { + let scenario = Scenario::new(); + scenario.origin_has_branch("stable", "main", &["Stable"]); + scenario.user_config_is("[launch]\npull = true\n"); + scenario.agent_does(&audit_script("[]")); + let head = scenario.launch_git(&["rev-parse", "HEAD"]); + let result = scenario.run(&[ + "secure", "base", "stable", "harness", "codex", "model", "gpt-5.5", "effort", "high", + "merge", "base-fix", "parallel", "2", + ]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(scenario.launch_git(&["rev-parse", "HEAD"]), head); + assert_eq!(scenario.launch_git(&["branch", "--show-current"]), "main\n"); + assert!(scenario.claude_calls().is_empty()); + let calls = scenario.codex_calls(); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0]["branch"], ""); + assert!( + calls[0]["prompt"] + .as_str() + .unwrap() + .contains(scenario.origin_git(&["rev-parse", "stable"]).trim()) + ); + scenario.assert_every_codex_session_found_the_factory_skills(); +} + +#[test] +fn the_security_harness_overrides_the_default_and_uses_its_own_model_and_effort() { + let scenario = Scenario::new(); + scenario.user_config_is( + "[security]\nharness = \"codex\"\n\ + [harness]\ndefault = \"claude\"\n\ + [harness.claude]\nmodel = \"opus\"\neffort = \"low\"\n\ + [harness.codex]\nmodel = \"gpt-6.1-sol\"\neffort = \"xhigh\"\n", + ); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!(scenario.claude_calls().is_empty()); + let calls = scenario.codex_calls(); + assert_eq!(calls.len(), 1); + let args = calls[0]["argv"].as_array().unwrap(); + assert!( + args.windows(2) + .any(|pair| pair == [json!("-m"), json!("gpt-6.1-sol")]) + ); + assert!( + args.windows(2) + .any(|pair| pair == [json!("-c"), json!("model_reasoning_effort=\"xhigh\"")]) + ); +} + +#[test] +fn codex_security_sessions_and_their_resumes_raise_the_thread_cap_and_request_fresh_sub_agents() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &format!( + "{}\necho '{{\"type\":\"item.started\",\"item\":{{\"id\":\"pending\",\"type\":\"collab_tool_call\",\"tool\":\"spawn_agent\",\"prompt\":\"Verify finding\",\"status\":\"in_progress\"}}}}'\n", + audit_script(&json!([finding("input-size")]).to_string()) + ), + ); + scenario + .agent_does("printf '%s\\n' 'Security audit: complete' > \"$FAKE_CLAUDE_FINAL_MESSAGE\"\n"); + scenario.agent_does_in_session( + 3, + &format!( + "{}\necho '{{\"type\":\"item.started\",\"item\":{{\"id\":\"pending\",\"type\":\"command_execution\",\"command\":\"bounded-fixture\",\"status\":\"in_progress\"}}}}'\n", + reproduction_script("reproduced high single") + ), + ); + scenario.agent_does_in_session( + 4, + "printf '%s\\n' 'The resumed local fixture confirmed the finding.' 'Security reproduction: reproduced high single' > \"$FAKE_CLAUDE_FINAL_MESSAGE\"\n", + ); + let result = scenario.run(&["secure", "harness", "codex"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let calls = scenario.codex_calls(); + assert_eq!(calls.len(), 4); + for call in &calls { + let args = call["argv"].as_array().unwrap(); + assert!( + args.windows(2).any(|pair| pair + == [ + json!("-c"), + json!("agents.max_concurrent_threads_per_session=8") + ]), + "missing Security thread cap: {args:?}" + ); + let prompt = call["prompt"].as_str().unwrap(); + assert!(prompt.contains("fork_turns: \"none\""), "{prompt}"); + assert!(prompt.contains("fresh sub-agents"), "{prompt}"); + } + for (index, thread) in [(1, "fake-thread-1"), (3, "fake-thread-3")] { + let resumed = calls[index]["argv"].as_array().unwrap(); + assert!( + resumed + .windows(2) + .any(|pair| pair == [json!("resume"), json!(thread)]), + "{resumed:?}" + ); + } + assert_eq!(scenario.gh_state()["advisories"][0]["severity"], "high"); +} + +#[test] +fn the_command_harness_wins_over_security_and_default_settings_and_uses_claudes_settings() { + let scenario = Scenario::new(); + scenario.user_config_is( + "[security]\nharness = \"codex\"\n\ + [harness]\ndefault = \"agy\"\n\ + [harness.claude]\nmodel = \"opus\"\neffort = \"high\"\n\ + [harness.codex]\nmodel = \"gpt-6.1-sol\"\neffort = \"xhigh\"\n", + ); + // Claude's Model check is its first call; the second runs the audit. + scenario.agent_does_in_session(1, "true\n"); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure", "harness", "claude"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!(scenario.codex_calls().is_empty()); + let calls = scenario.claude_calls(); + assert_eq!(calls.len(), 2); + let args = calls[1]["argv"].as_array().unwrap(); + for pair in [ + [json!("--model"), json!("opus")], + [json!("--effort"), json!("high")], + ] { + assert!(args.windows(2).any(|actual| actual == pair), "{args:?}"); + } + let prompt = calls[1]["prompt"].as_str().unwrap(); + assert!(!prompt.contains("fork_turns"), "{prompt}"); + assert!( + !args + .iter() + .any(|arg| arg.as_str().unwrap().contains("max_concurrent_threads")) + ); +} + +#[test] +fn a_blank_or_missing_security_harness_preserves_the_default_harness_and_its_settings() { + for security in [ + "", + "[security]\n", + "[security]\nharness = \"\"\n", + "[security]\nharness = \" \"\n", + ] { + let scenario = Scenario::new(); + scenario.user_config_is(&format!( + "{security}[harness]\ndefault = \"codex\"\n\ + [harness.codex]\nmodel = \"gpt-6-luna\"\neffort = \"high\"\n" + )); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{security}: {}", result.stderr); + assert!(scenario.claude_calls().is_empty()); + let calls = scenario.codex_calls(); + let args = calls[0]["argv"].as_array().unwrap(); + assert!( + args.windows(2) + .any(|pair| pair == [json!("-m"), json!("gpt-6-luna")]) + ); + assert!( + args.windows(2) + .any(|pair| pair == [json!("-c"), json!("model_reasoning_effort=\"high\"")]) + ); + } + let scenario = Scenario::new(); + scenario.user_config_is("[security]\nharness = \"\"\n"); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(scenario.claude_calls().len(), 1); + assert!(scenario.codex_calls().is_empty()); +} + +#[test] +fn invalid_security_harness_settings_name_the_key_and_stop_before_any_work() { + for config in [ + "[security]\nharness = \"unknown\"\n", + "[security]\nharness = true\n", + ] { + let scenario = Scenario::new(); + scenario.user_config_is(config); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("security.harness"), + "{}", + result.stderr + ); + assert!(scenario.gh_calls().is_empty()); + assert!(scenario.claude_calls().is_empty()); + assert!(scenario.codex_calls().is_empty()); + } +} + +#[test] +fn a_ready_issue_skips_before_any_session_and_keeps_only_the_activity_log() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["ready-for-agent"]); + scenario.user_config_is("[activity]\nquiet_skips = true\n"); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(result.stdout, ""); + assert_eq!(result.stderr, ""); + assert!(scenario.claude_calls().is_empty()); + let log = fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!( + log.contains("Security run skipped: Ready issue #7"), + "{log}" + ); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/commands") + .exists() + ); +} + +#[test] +fn another_pass_holds_the_security_run_and_its_skip_is_logged() { + let scenario = Scenario::new(); + let waiting = format!( + "touch \"$HOME/../started\"\nwhile [ ! -f \"$HOME/../release\" ]; do sleep 0.05; done\n{}", + audit_script("[]") + ); + scenario.agent_does(&waiting); + let first = scenario.run_until(&["secure"], &[], "started"); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let second = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + let pickup = scenario.run(&["pickup"]); + fs::write(scenario.path("release"), "").unwrap(); + let first = first.finish(); + assert!(resend.requests().is_empty()); + assert_eq!(first.code, Some(0), "{}", first.stderr); + for skipped in [second, pickup] { + assert_eq!(skipped.code, Some(0), "{}", skipped.stderr); + assert!( + skipped.stderr.contains("another Pass is already running"), + "{}", + skipped.stderr + ); + } + assert_eq!(scenario.claude_calls().len(), 1); + assert_eq!( + scenario + .entries("home/.thirdshift/logs/acme/widgets/commands/secure") + .len(), + 1 + ); + let log = fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!(log.contains("Security run skipped: another Pass is already running")); +} + +#[test] +fn missing_node_fails_before_any_session_and_names_the_prerequisite() { + let scenario = Scenario::new(); + let git = std::process::Command::new("which") + .arg("git") + .output() + .unwrap(); + let git = String::from_utf8(git.stdout).unwrap(); + std::os::unix::fs::symlink(git.trim(), scenario.path("bin/git")).unwrap(); + let path = scenario.path("bin"); + let result = scenario.run_with_env(&["secure"], &[("PATH", path.to_str().unwrap())]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("Node.js is required"), + "{}", + result.stderr + ); + assert!(scenario.claude_calls().is_empty()); + assert_eq!(scenario.entries("work"), vec![REPO]); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/commands") + .exists() + ); +} + +#[test] +fn missing_final_line_or_invalid_reports_fail_without_recording() { + for (script, cause) in [ + ( + audit_script("[]").replace("Security audit: complete", "No protocol line"), + "without the final line", + ), + ( + audit_script("[]").replace("Security audit: complete", "Security audit: incomplete"), + "ended incomplete", + ), + (audit_script("[{}]"), "validator validate-findings.cjs"), + ( + audit_script("[]").replace( + "'[]' > \"$output/coverage-ledger.json\"", + "'[{}]' > \"$output/coverage-ledger.json\"", + ), + "validator validate-coverage-ledger.cjs", + ), + ] { + let scenario = Scenario::new(); + scenario.agent_does(&script); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains(cause), + "expected {cause}: {}", + result.stderr + ); + assert!(result.stderr.contains("session log:")); + assert!(scenario.gh_state()["advisories"].is_null()); + assert_eq!(scenario.entries("work"), vec![REPO]); + let log = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!(log.contains("Security run started")); + assert!(log.contains("Security run ended: failed:"), "{log}"); + let logs = scenario.entries("home/.thirdshift/logs/acme/widgets/commands/secure"); + let command_log = fs::read_to_string(scenario.path(&format!( + "home/.thirdshift/logs/acme/widgets/commands/secure/{}", + logs[0] + ))) + .unwrap(); + assert!(command_log.contains(cause)); + } +} + +#[test] +fn rejected_candidates_stay_in_the_report_without_a_private_advisory() { + let scenario = Scenario::new(); + let mut rejected = finding("refuted-claim"); + rejected.as_object_mut().unwrap().remove("blockers"); + rejected.as_object_mut().unwrap().remove("validation_plan"); + rejected["verdict"] = json!("rejected"); + rejected["reason"] = json!("The source already checks the bound."); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), rejected]).to_string(), + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"].as_array().unwrap().len(), + 1 + ); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + json!({"ecosystem":"other", "name":null}) + ); +} + +#[test] +fn an_incomplete_run_record_cannot_be_overridden_by_a_complete_final_line() { + let scenario = Scenario::new(); + scenario.agent_does(&audit_script("[]").replace( + "\"run_status\":\"complete\"", + "\"run_status\":\"incomplete\"", + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("run-metadata.json"), + "{}", + result.stderr + ); + assert!(scenario.gh_state()["advisories"].is_null()); +} + +#[test] +fn audits_origins_base_without_changing_the_launch_directory() { + let scenario = Scenario::new(); + scenario.origin_has_commit("main", "SECURITY.md", "Trust model", "Document trust model"); + scenario.agent_does(&format!( + "{}\ntest -z \"$(git branch --show-current)\"\ntest -f SECURITY.md\n", + audit_script("[]") + )); + let head = scenario.launch_git(&["rev-parse", "HEAD"]); + fs::write(scenario.launch_dir().join("local.txt"), "local work").unwrap(); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result + .stderr + .contains("Security audit recorded 0 new finding(s)"), + "{}", + result.stderr + ); + assert_eq!(scenario.launch_git(&["rev-parse", "HEAD"]), head); + assert_eq!( + fs::read_to_string(scenario.launch_dir().join("local.txt")).unwrap(), + "local work" + ); + assert_eq!(scenario.entries("work"), vec![REPO]); + assert_eq!( + scenario.origin_git(&["log", "--format=%s", "main"]), + "Document trust model\nInitial commit\n" + ); + assert!(scenario.gh_state()["prs"].as_array().unwrap().is_empty()); + let prompt = scenario.first_prompt(); + for required in [ + "thirdshift-security-audit", + "full audit mode", + "quick", + "vendored", + "third-party", + "SECURITY.md", + "earlier runs", + "incomplete", + "Security audit: complete", + "You run headless", + "commits, pushes, opens and publishes nothing", + ] { + assert!(prompt.contains(required), "missing {required}: {prompt}"); + } + scenario.assert_every_session_found_the_factory_skills(); +} + +#[test] +fn security_advisories_preserve_python_and_go_manifest_identity() { + for (manifest, contents, package) in [ + ( + "pyproject.toml", + "[project]\nname = \"widgets\"\nversion = \"1.0.0\"\n", + json!({"ecosystem":"pip", "name":"widgets"}), + ), + ( + "go.mod", + "module example.com/acme/widgets\n\ngo 1.24\n", + json!({"ecosystem":"go", "name":"example.com/acme/widgets"}), + ), + ] { + let scenario = Scenario::new(); + scenario.origin_has_commit("main", manifest, contents, "Add manifest"); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + package, + "package identity was lost for {manifest}" + ); + } +} + +#[test] +fn security_audit_names_the_threat_model_under_docs() { + let scenario = Scenario::new(); + fs::create_dir_all(scenario.launch_dir().join("docs")).unwrap(); + fs::write( + scenario.launch_dir().join("docs/THREAT-MODEL.md"), + "# Threat model\nAll callers are authenticated; tenant isolation is the boundary.\n", + ) + .unwrap(); + scenario.launch_git(&["add", "docs/THREAT-MODEL.md"]); + scenario.launch_git(&["commit", "-m", "Document threat model"]); + scenario.launch_git(&["push", "origin", "main"]); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains("Read the repository's threat-model document `docs/THREAT-MODEL.md`."), + "the existing threat-model document was not named: {prompt}" + ); +} + +#[test] +fn reproduction_preserves_an_original_finding_reproduction_heading() { + for private in [false, true] { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(private); + scenario.write_gh_state(&github); + let mut candidate = finding("markdown-evidence"); + let original_writeup = "Candidate notes.\n\n## Reproduction\n\nOriginal local validation evidence must survive."; + candidate["description"] = json!(original_writeup); + scenario.agent_does_in_session(1, &audit_script(&json!([candidate]).to_string())); + scenario.agent_does_in_session(2, &reproduction_script("reproduced medium single")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let body = if private { + state["bodies"]["8"].as_str().unwrap() + } else { + state["advisories"][0]["description"].as_str().unwrap() + }; + assert!( + body.contains(original_writeup), + "original finding was deleted: {body}" + ); + assert!( + body.contains("\"validation_plan\""), + "validation plan was deleted: {body}" + ); + assert!(body.contains("Severity: medium"), "{body}"); + } +} + +#[test] +fn a_published_finding_keeps_new_reproduction_evidence_private_and_its_grade() { + let scenario = Scenario::new(); + let commit = scenario.origin_git(&["rev-parse", "main"]); + let original = format!( + "Fingerprint: `published-finding`\nAudited commit: `{}`\nDay-shift-approved published description.\n", + commit.trim() + ); + let mut github = scenario.gh_state(); + github["advisories"] = json!([{ + "ghsa_id": "GHSA-existing", "description": original, + "severity": "high", "state": "published", + "summary": "Existing finding", "html_url": "https://github.com/acme/widgets/security/advisories/GHSA-existing" + }]); + scenario.write_gh_state(&github); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("published-finding")]).to_string()), + ); + scenario.agent_does_in_session(2, &reproduction_script("reproduced critical single")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + assert_eq!( + state["advisories"][0]["description"], original, + "new reproduction evidence became part of a published advisory" + ); + assert_eq!( + state["advisories"][0]["severity"], "high", + "the Day shift grade changed" + ); +} + +#[test] +fn a_private_reproduction_accepts_the_rubrics_informational_severity() { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + scenario.write_gh_state(&github); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("minimal-impact")]).to_string()), + ); + scenario.agent_does_in_session(2, &reproduction_script("reproduced informational single")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let body = state["bodies"]["8"].as_str().unwrap(); + assert!(body.contains("Severity: informational"), "{body}"); + assert!(body.contains("bounded_fixture();"), "{body}"); + assert!(body.contains("Local command: bounded-fixture"), "{body}"); +} + +#[test] +fn a_finding_triaged_during_reproduction_is_left_unchanged() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("input-size")]).to_string()), + ); + scenario.agent_does_in_session(2, &format!(r#" +printf '%s\n' '{{"state":"published","severity":"high"}}' | gh api --method PATCH repos/acme/widgets/security-advisories/GHSA-test-test-0001 --input - >/dev/null +{} +"#, reproduction_script("reproduced critical single"))); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("triaged during its reproduction"), + "{}", + result.stderr + ); + let state = scenario.gh_state(); + let record = &state["advisories"][0]; + assert_eq!(record["state"], "published"); + assert_eq!(record["severity"], "high"); + assert!( + !record["description"] + .as_str() + .unwrap() + .contains("bounded_fixture();") + ); +} + +#[test] +fn an_informational_advisory_requires_a_day_shift_decision_without_an_invented_grade() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &audit_script(&json!([finding("minimal-impact")]).to_string()), + ); + scenario.agent_does_in_session(2, &reproduction_script("reproduced informational single")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result + .stderr + .contains("the Day shift must decide its representation"), + "{}", + result.stderr + ); + let state = scenario.gh_state(); + let record = &state["advisories"][0]; + assert_eq!(record["severity"], Value::Null); + assert!( + !record["description"] + .as_str() + .unwrap() + .contains("<!-- thirdshift:security-reproduction -->") + ); + assert!( + scenario + .gh_calls_of("api", "--method") + .iter() + .all(|call| call[2] != "PATCH") + ); +} + +#[test] +fn a_completed_security_audit_sends_one_notification_when_asked() { + let scenario = Scenario::new(); + scenario.agent_does(&audit_script("[]")); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let (subject, text) = the_one_notification(&resend); + assert_eq!( + subject, + "[thirdshift] acme/widgets Security run: findings recorded" + ); + assert_eq!(resend.requests()[0].body["to"], "me@example.com"); + assert!(text.contains("Built with claude"), "{text}"); +} + +#[test] +fn the_notification_lists_each_recorded_finding_without_its_private_write_up() { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["advisories"] = json!([{ + "ghsa_id": "GHSA-existing", + "state": "draft", "severity": "high", "summary": "Existing finding", + "html_url": "https://github.com/acme/widgets/security/advisories/GHSA-existing", + "description": "Fingerprint: `existing`\nPrivate existing write-up." + }]); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script( + &json!([finding("existing"), finding("new")]).to_string(), + )); + scenario.agent_does_in_session(2, &reproduction_script("reproduced medium single")); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let (_, text) = the_one_notification(&resend); + assert!( + text.contains("Security audit recorded 1 new finding(s), 1 already recorded"), + "{text}" + ); + assert!(text.contains("high: Existing finding"), "{text}"); + assert!( + text.contains("https://github.com/acme/widgets/security/advisories/GHSA-existing"), + "{text}" + ); + assert!(text.contains("Unchecked input size"), "{text}"); + assert!(text.contains("medium: Unchecked input size"), "{text}"); + assert_eq!(scenario.claude_calls().len(), 2); + assert!( + text.contains("https://github.com/acme/widgets/security/advisories/GHSA-test-test-0002"), + "{text}" + ); + for private in [ + "Private candidate write-up.", + "Private existing write-up.", + "Input reaches storage without a bound.", + "Private trace.", + "Private evidence.", + "No sandbox available.", + "Exercise a bounded fixture.", + "Fingerprint:", + "Local command: bounded-fixture", + "bounded_fixture();", + ] { + assert!( + !resend.requests()[0].body.to_string().contains(private), + "leaked {private}: {text}" + ); + } +} + +#[test] +fn a_private_security_notification_lists_existing_and_new_issue_links_without_write_ups() { + for state in ["OPEN", "CLOSED"] { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(true); + github["issues"]["8"] = json!(state); + github["labels"]["8"] = json!(["security-finding"]); + github["titles"]["8"] = json!("Existing finding"); + github["bodies"]["8"] = json!("Fingerprint: `existing`\nPrivate existing write-up."); + scenario.write_gh_state(&github); + scenario.agent_does(&audit_script( + &json!([finding("existing"), finding("new")]).to_string(), + )); + scenario.agent_does_in_session(2, &reproduction_script("reproduced high single")); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let (_, text) = the_one_notification(&resend); + for metadata in [ + "Security audit recorded 1 new finding(s), 1 already recorded", + "Existing finding", + "https://github.com/acme/widgets/issues/8", + "high: Unchecked input size", + "https://github.com/acme/widgets/issues/9", + ] { + assert!(text.contains(metadata), "missing {metadata}: {text}"); + } + for private in [ + "Private candidate write-up.", + "Private existing write-up.", + "Input reaches storage without a bound.", + "Private trace.", + "Private evidence.", + "No sandbox available.", + "Exercise a bounded fixture.", + "Fingerprint:", + "Local command: bounded-fixture", + "bounded_fixture();", + ] { + assert!( + !resend.requests()[0].body.to_string().contains(private), + "leaked {private}: {text}" + ); + } + assert_eq!(scenario.gh_calls_of("issue", "create").len(), 1); + assert_eq!(scenario.gh_state()["issues"].as_object().unwrap().len(), 3); + } +} + +#[test] +fn security_notifications_follow_email_defaults_and_command_overrides() { + for (config, args, expected_to) in [ + ("", vec!["secure"], None), + ( + "[email]\nalways = true\nto = \"default@example.com\"\n", + vec!["secure"], + Some("default@example.com"), + ), + ( + "[email]\nto = \"default@example.com\"\n", + vec!["secure", "--email"], + Some("default@example.com"), + ), + ( + "[email]\nalways = true\nto = \"default@example.com\"\n", + vec!["secure", "no-email"], + None, + ), + ( + "[email]\nto = \"default@example.com\"\n", + vec!["secure", "--email", "override@example.com"], + Some("override@example.com"), + ), + ] { + let scenario = Scenario::new(); + scenario.user_config_is(config); + scenario.agent_does(&audit_script("[]")); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = scenario.run_with_env(&args, &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + match expected_to { + Some(to) => { + the_one_notification(&resend); + assert_eq!(resend.requests()[0].body["to"], to); + } + None => assert!(resend.requests().is_empty()), + } + } +} + +#[test] +fn a_recording_failure_still_notifies_each_finding_already_recorded() { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["advisory_create_fails_after"] = json!(1); + scenario.write_gh_state(&github); + let mut first = finding("first"); + first["title"] = json!("First recorded finding"); + let mut second = finding("second"); + second["title"] = json!("Unrecorded finding"); + scenario.agent_does(&audit_script(&json!([first, second]).to_string())); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"].as_array().unwrap().len(), + 1 + ); + let (subject, text) = the_one_notification(&resend); + assert!(subject.ends_with(": audit failed"), "{subject}"); + assert!(text.contains("First recorded finding"), "{text}"); + assert!( + text.contains("https://github.com/acme/widgets/security/advisories/GHSA-test-test-0001"), + "{text}" + ); + assert!(!text.contains("Unrecorded finding"), "{text}"); + assert!(!text.contains("Private candidate write-up."), "{text}"); +} + +#[test] +fn a_skipped_security_run_sends_no_notification_requested_by_word_or_config() { + for args in [vec!["secure"], vec!["secure", "email", "me@example.com"]] { + let scenario = Scenario::new(); + scenario.user_config_is("[email]\nalways = true\nto = \"me@example.com\"\n"); + scenario.issue_labelled(7, &["ready-for-agent"]); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = scenario.run_with_env(&args, &resend_env(&resend)); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result.stderr.contains("Ready issue #7"), + "{}", + result.stderr + ); + assert!(scenario.claude_calls().is_empty()); + assert!(resend.requests().is_empty()); + } +} + +#[test] +fn failed_security_audits_send_one_notification_with_a_safe_status() { + for (script, failing_call, cause) in [ + ( + audit_script("[]").replace("Security audit: complete", "Security audit: incomplete"), + None, + "ended incomplete", + ), + ( + audit_script("[{}]"), + None, + "validator validate-findings.cjs", + ), + ( + audit_script(&json!([finding("new")]).to_string()), + Some("api --method POST"), + "creating a draft repository security advisory failed", + ), + (audit_script("[]"), Some("issue list"), "gh issue list"), + ] { + let scenario = Scenario::new(); + scenario.agent_does(&script); + if let Some(call) = failing_call { + if call == "api --method POST" { + // The API consumes the request before rejecting it, avoiding + // a race between writing stdin and an early fake gh exit. + let mut github = scenario.gh_state(); + github["advisory_create_fails_after"] = json!(0); + scenario.write_gh_state(&github); + } else { + scenario.gh_fails(call); + } + } + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let (subject, text) = the_one_notification(&resend); + assert_eq!( + subject, + "[thirdshift] acme/widgets Security run: audit failed" + ); + assert!(text.contains("Audit: audit failed"), "{text}"); + assert!( + result.stderr.contains(cause), + "expected {cause}: {}", + result.stderr + ); + assert!(!text.contains("Cause:"), "{text}"); + assert!(!text.contains("Private candidate write-up."), "{text}"); + } +} + +#[test] +fn an_interrupted_security_audit_sends_one_notification() { + let scenario = Scenario::new(); + scenario.agent_does("touch \"$HOME/../started\"\nsleep 60\n"); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = scenario.run_and_signal_with_env( + &["secure", "email", "me@example.com"], + &resend_env(&resend), + "started", + "TERM", + ); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let (subject, text) = the_one_notification(&resend); + assert_eq!( + subject, + "[thirdshift] acme/widgets Security run: interrupted" + ); + assert!(text.contains("Audit: interrupted"), "{text}"); + assert!(!text.contains("Cause:"), "{text}"); +} + +#[test] +fn a_failed_send_keeps_the_security_runs_success_or_failure() { + for script in [audit_script("[]"), "exit 3".to_string()] { + let scenario = Scenario::new(); + scenario.agent_does(&script); + let without_email = scenario.run(&["secure"]); + scenario.origin_has_commit("main", "new-work.txt", "new work", "Move Base branch"); + let resend = ResendStandIn::replying(500, "upstream exploded"); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, without_email.code, "{}", result.stderr); + assert_eq!(result.stdout, without_email.stdout); + the_one_notification(&resend); + assert!( + result + .stderr + .contains("warning: could not send the Run notification"), + "{}", + result.stderr + ); + } +} + +#[test] +fn notification_checks_precede_security_skip_checks_and_work() { + for (args, env, cause) in [ + (vec!["secure", "email"], true, "no email address"), + ( + vec!["secure", "email", "me@example.com"], + false, + "no Resend API key", + ), + ] { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["ready-for-agent"]); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let env = if env { + resend_env(&resend).to_vec() + } else { + vec![("THIRDSHIFT_RESEND_URL", resend.url())] + }; + let result = scenario.run_with_env(&args, &env); + scenario.assert_rejected_before_any_work(&result, cause); + assert!(scenario.gh_calls().is_empty()); + assert!(resend.requests().is_empty()); + } +} + +#[test] +fn unchanged_base_is_remembered_separately_for_each_branch() { + let scenario = Scenario::new(); + scenario.origin_has_branch("stable", "main", &["Stable branch"]); + scenario.agent_does(&audit_script("[]")); + + let main_audit = scenario.run(&["secure", "base", "main"]); + assert_eq!(main_audit.code, Some(0), "{}", main_audit.stderr); + std::thread::sleep(std::time::Duration::from_millis(20)); + + let stable_audit = scenario.run(&["secure", "base", "stable"]); + assert_eq!(stable_audit.code, Some(0), "{}", stable_audit.stderr); + assert_eq!(scenario.claude_calls().len(), 2); + + let unchanged_main = scenario.run(&["secure", "base", "main"]); + assert_eq!(unchanged_main.code, Some(0), "{}", unchanged_main.stderr); + assert_eq!( + scenario.claude_calls().len(), + 2, + "An unchanged main must skip after stable is audited: {}", + unchanged_main.stderr + ); + let activity = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!(activity.contains( + "Security run skipped: Base branch main hasn't changed since the last completed Security audit" + )); +} + +#[test] +fn failed_security_notification_excludes_description_from_killed_verifier() { + let scenario = Scenario::new(); + scenario.agent_does_in_session( + 1, + &format!( + "{}{}", + audit_script(&json!([finding("private-candidate")]).to_string()), + support::leaves_running("Private candidate write-up.") + ), + ); + scenario.agent_does_in_session( + 2, + &format!( + "{}printf '%s\\n' 'Security audit: incomplete' > \"$FAKE_CLAUDE_FINAL_MESSAGE\"\n", + support::leaves_running("Private candidate write-up.") + ), + ); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert_eq!(scenario.claude_calls().len(), 2); + let (subject, text) = the_one_notification(&resend); + assert!(subject.ends_with(": audit failed"), "{subject}"); + assert!(text.contains("Audit: audit failed"), "{text}"); + assert!( + !resend.requests()[0] + .body + .to_string() + .contains("Private candidate write-up."), + "finding description leaked to Resend: {text}" + ); +} + +#[test] +fn security_notification_omits_private_background_trace() { + let scenario = Scenario::new(); + scenario.agent_does(&support::leaves_running( + "Private exploit trace: POST /admin/debug with token", + )); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(1), "{}", result.stderr); + let (_, text) = the_one_notification(&resend); + assert!(!text.contains("Private exploit trace"), "{text}"); +} + +#[test] +fn a_reproduction_failure_notifies_recorded_metadata_without_private_evidence() { + for private in [false, true] { + let scenario = Scenario::new(); + let mut github = scenario.gh_state(); + github["private"] = json!(private); + scenario.write_gh_state(&github); + let mut first = finding("first"); + first["title"] = json!("First recorded finding"); + let mut second = finding("second"); + second["title"] = json!("Second recorded finding"); + scenario.agent_does_in_session(1, &audit_script(&json!([first, second]).to_string())); + scenario.agent_does_in_session(2, &reproduction_script("reproduced high single")); + scenario.agent_does_in_session(3, &reproduction_script("incomplete")); + let resend = ResendStandIn::replying(200, r#"{"id":"1"}"#); + let result = + scenario.run_with_env(&["secure", "email", "me@example.com"], &resend_env(&resend)); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result + .stderr + .contains("Security reproduction ended without the final line"), + "{}", + result.stderr + ); + let (subject, text) = the_one_notification(&resend); + assert!(subject.ends_with(": audit failed"), "{subject}"); + assert!(text.contains("high: First recorded finding"), "{text}"); + assert!(text.contains("Second recorded finding"), "{text}"); + assert!(!text.contains("high: Second recorded finding"), "{text}"); + let url = if private { + "https://github.com/acme/widgets/issues/" + } else { + "https://github.com/acme/widgets/security/advisories/" + }; + assert_eq!(text.matches(url).count(), 2, "{text}"); + for evidence in [ + "Private candidate write-up.", + "Private trace.", + "Private evidence.", + "Local command: bounded-fixture", + "bounded_fixture();", + "Cause:", + ] { + assert!( + !resend.requests()[0].body.to_string().contains(evidence), + "leaked {evidence}: {text}" + ); + } + assert_eq!(scenario.claude_calls().len(), 3); + } +} + +#[test] +fn spec_review_short_private_poc_must_not_be_dispatched() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + let description = state["advisories"][0]["description"] + .as_str() + .unwrap() + .replace("bounded_fixture();", "bypass_login();"); + state["advisories"][0]["description"] = json!(description); + scenario.write_gh_state(&state); + scenario.agent_does_in_session( + 1, + &publish_fix("GHSA-finding-0").replace( + "Reject oversized input.", + "Reject oversized input. Proof-of-concept: bypass_login();", + ), + ); + scenario.agent_does_for(8, implement_fix()); + let result = scenario.run(&["secure", "security-fix"]); + let state = scenario.gh_state(); + assert!( + state["bodies"]["8"] + .as_str() + .unwrap() + .contains("bypass_login();") + ); + assert_eq!( + result.code, + Some(1), + "a Ticket copying the private proof-of-concept was dispatched: {}", + result.stderr, + ); + assert_eq!(scenario.claude_calls().len(), 1); + assert!(state["prs"].as_array().unwrap().is_empty()); +} + +#[test] +fn spec_review_current_day_shift_severity_must_decide_fix_priority() { + let scenario = with_reproduced_findings(&["critical", "high"]); + let mut state = scenario.gh_state(); + // This models changing severity in GitHub without rewriting the historical + // reproduction outcome, which must remain useful private evidence. + state["advisories"][0]["severity"] = json!("low"); + scenario.write_gh_state(&state); + scenario.agent_does_in_session(1, r#" +case "$FAKE_CLAUDE_PROMPT" in + *GHSA-finding-1*) record=GHSA-finding-1 ;; + *GHSA-finding-0*) record=GHSA-finding-0 ;; + *) exit 1 ;; +esac +url=$(gh issue create --title "Bound accepted input" --body "Reject oversized input. Private record: https://github.com/acme/widgets/security/advisories/$record" --label needs-triage) +printf 'Security fix Ticket: %s\n' "$url" > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#); + scenario.agent_does_for(8, implement_fix()); + let result = scenario.run(&["secure", "security-fix"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let calls = scenario.claude_calls(); + let prompt = calls[0]["prompt"].as_str().unwrap(); + assert!( + prompt.contains("GHSA-finding-1"), + "the current high finding must precede the finding regraded low: {prompt}", + ); +} + +#[test] +fn a_later_audit_keeps_the_link_to_a_successfully_dispatched_private_fix() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + state["private"] = json!(true); + state["bodies"]["7"] = state["advisories"][0]["description"].clone(); + state["labels"]["7"] = json!(["security-finding", "needs-triage"]); + scenario.write_gh_state(&state); + scenario.agent_does_for( + 7, + &implement_fix() + .replace("issue-8", "issue-7") + .replace("#8", "#7"), + ); + let first = scenario.run(&["secure", "security-fix"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + let state = scenario.gh_state(); + assert_eq!(state["prs"][0]["state"], "OPEN"); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .contains("Fix Ticket: https://github.com/acme/widgets/issues/7") + ); + assert_eq!(state["issues"].as_object().unwrap().len(), 1); + + scenario.origin_has_commit( + "main", + "other-work.txt", + "other work", + "Advance Base branch", + ); + scenario.agent_does_in_session(2, &audit_script(&json!([finding("finding-0")]).to_string())); + scenario.agent_does_for_in_session(7, 2, &reproduction_script("reproduced high single")); + let second = scenario.run(&["secure", "security-fix"]); + assert_eq!(second.code, Some(0), "{}", second.stderr); + let state = scenario.gh_state(); + assert_eq!( + state["issues"].as_object().unwrap().len(), + 1, + "another fix Ticket was published for a finding whose first fix succeeded: {state}" + ); + assert!( + state["bodies"]["7"] + .as_str() + .unwrap() + .contains("Fix Ticket: https://github.com/acme/widgets/issues/7") + ); +} + +#[test] +fn spec_followup_common_poc_token_must_not_block_a_terse_fix_ticket() { + let scenario = with_reproduced_findings(&["high"]); + let mut state = scenario.gh_state(); + let description = state["advisories"][0]["description"] + .as_str() + .unwrap() + .replace("bounded_fixture();", "assert (\n input\n) == 'overflow'"); + state["advisories"][0]["description"] = json!(description); + scenario.write_gh_state(&state); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for(8, implement_fix()); + let result = scenario.run(&["secure", "security-fix"]); + let state = scenario.gh_state(); + let ticket = state["bodies"]["8"].as_str().unwrap(); + assert_eq!( + ticket, + "Reject oversized input. Private record: https://github.com/acme/widgets/security/advisories/GHSA-finding-0", + ); + assert!(!ticket.contains("assert (") && !ticket.contains("overflow")); + assert_eq!( + result.code, + Some(0), + "a valid terse Ticket was rejected because it shares an ordinary token with the private test: {}", + result.stderr, + ); + assert_eq!(scenario.claude_calls().len(), 2); + assert_eq!(state["prs"].as_array().unwrap().len(), 1); +} + +#[test] +fn claude_security_progress_ignores_synthetic_api_errors() { + let scenario = Scenario::new(); + scenario.agent_does(&format!( + r#" +printf '%s\n' '{{"type":"assistant","message":{{"model":"claude-opus-4-8","content":[]}}}}' +printf '%s\n' '{{"type":"assistant","parent_tool_use_id":null,"is_api_error_message":true,"error":"rate_limit","message":{{"model":"<synthetic>","role":"assistant","content":[{{"type":"text","text":"API Error: Rate limit reached"}}]}}}}' +printf '%s\n' '{{"type":"assistant","message":{{"model":"claude-opus-4-8","content":[]}}}}' +{} +"#, + audit_script("[]") + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let command_log = security_command_log(&scenario); + for text in [&result.stderr, &command_log] { + assert!(!text.contains("Model: <synthetic>"), "{text}"); + assert_eq!( + text.matches("security-audit: Model: claude-opus-4-8") + .count(), + 1, + "{text}" + ); + } +} + +#[test] +fn grok_security_audits_log_the_model_that_answered() { + let scenario = Scenario::new(); + scenario.agent_does( + r#" +node -e 'const fs = require("fs"); const calls = JSON.parse(fs.readFileSync(process.env.FAKE_GROK_RECORD, "utf8")); process.stdout.write(calls[calls.length - 1].prompt);' > prompt.txt +output=$(sed -n 's/^Output directory: `\(.*\)`\.$/\1/p' prompt.txt) +test -n "$output" +printf '%s\n' '[]' > "$output/findings.json" +printf '%s\n' '[]' > "$output/coverage-ledger.json" +printf '%s\n' '{"run_status":"complete"}' > "$output/run-metadata.json" +printf '%s\n' '{"type":"assistant","message":{"model":"grok-4.7","content":[]}}' +printf '%s\n' 'Security audit: complete' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + ); + let result = scenario.run(&["secure", "harness", "grok", "model", "grok-4.7"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let command_log = security_command_log(&scenario); + for text in [&result.stderr, &command_log] { + assert!(text.contains("security-audit: Model: grok-4.7"), "{text}"); + } +} + +#[test] +fn muse_security_audits_log_the_answering_model_from_the_session_record() { + let scenario = Scenario::new(); + scenario.agent_does(&format!( + r#" +export FAKE_CLAUDE_PROMPT="$(node -e 'const fs = require("fs"); const calls = JSON.parse(fs.readFileSync(process.env.FAKE_MUSE_RECORD, "utf8")); process.stdout.write(calls[calls.length - 1].prompt);')" +{} +root="$HOME/.local/share/muse/sessions/2026/10/06/fake-muse-1" +mkdir -p "$root" +printf '%s\n' '{{"stream":{{"id":"child"}},"payload":{{"kind":"run","event":{{"kind":"model_completed","model":"child-model"}}}}}}' '{{"stream":{{"id":"fake-muse-1"}},"payload":{{"kind":"run","event":{{"kind":"model_completed","model":"muse-spark-1.3"}}}}}}' > "$root/session.jsonl" +"#, + audit_script("[]") + )); + let result = + scenario.run_with_env(&["secure", "harness", "muse"], &[("FAKE_MUSE_NO_LOG", "1")]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result + .stderr + .contains("security-audit: Model: muse-spark-1.3"), + "{}", + result.stderr + ); + assert!( + !result.stderr.contains("Model: child-model"), + "{}", + result.stderr + ); +} + +#[test] +fn opencode_security_audits_log_the_answering_model_from_the_export() { + let scenario = Scenario::new(); + scenario.agent_does(&format!( + r#" +export FAKE_CLAUDE_PROMPT="$(node -e 'const fs = require("fs"); const calls = JSON.parse(fs.readFileSync(process.env.FAKE_OPENCODE_RECORD, "utf8")); process.stdout.write(calls[calls.length - 1].prompt);')" +{} +"#, + audit_script("[]") + )); + let script = scenario.path("export-model.sh"); + fs::write(&script, format!( + r#"node -e 'const fs = require("fs"); const p = process.argv[1]; const record = JSON.parse(fs.readFileSync(p, "utf8")); record.info.model = {{id:"title-model",providerID:"title"}}; for (const message of record.messages) {{ if (message.type === "assistant") message.model = {{id:"MiMo-V2.6-Pro",providerID:"primalabs"}}; }} fs.writeFileSync(p, JSON.stringify(record));' "{}" +"#, + scenario.path("opencode-calls.fake-opencode-1.export.json").display() + )).unwrap(); + let result = scenario.run_with_env( + &["secure", "harness", "opencode"], + &[("FAKE_OPENCODE_EXPORT_SCRIPT", script.to_str().unwrap())], + ); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result + .stderr + .contains("security-audit: Model: primalabs/MiMo-V2.6-Pro"), + "{}", + result.stderr + ); + assert!( + !result.stderr.contains("Model: title/"), + "{}", + result.stderr + ); +} + +#[test] +fn review_failed_fix_before_claim_is_not_retried_by_pickup() { + let scenario = with_reproduced_findings(&["high"]); + scenario.agent_does_in_session( + 1, + &format!( + "{}gh fake fails 'label create in-progress'\n", + publish_fix("GHSA-finding-0") + ), + ); + scenario.agent_does_for(8, implement_fix()); + + let failed = scenario.run(&["secure", "security-fix"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + assert!( + failed.stderr.contains("could not make the Claim on #8"), + "{}", + failed.stderr + ); + + let mut state = scenario.gh_state(); + state["failing"] = json!([]); + scenario.write_gh_state(&state); + scenario.issue_timeline( + 8, + &[(support::TimelineEvent::Labelled("ready-for-agent"), 20)], + ); + + let pickup = scenario.run(&["pickup"]); + assert_eq!(pickup.code, Some(0), "{}", pickup.stderr); + assert!( + !pickup.stderr.contains("taking Ready issue #8"), + "Pickup retried a failed Security fix: {}", + pickup.stderr + ); + assert_eq!(scenario.claude_calls().len(), 1); +} + +#[test] +fn review_failed_fix_marker_write_error_still_pauses_security() { + assert_a_failed_fix_record_outage_pauses_security("high"); +} + +#[test] +fn a_failed_fix_still_pauses_after_its_failure_record_patch_is_rejected() { + assert_a_failed_fix_record_outage_pauses_security("critical"); +} + +#[test] +fn a_failed_fix_preserves_private_edits_made_during_its_run_and_still_pauses() { + for private in [false, true] { + let scenario = with_reproduced_findings(&["high"]); + let issue = if private { 7 } else { 8 }; + let mut state = scenario.gh_state(); + let original = state["advisories"][0]["description"] + .as_str() + .unwrap() + .to_string(); + if private { + state["private"] = json!(true); + state["bodies"]["7"] = json!(original); + state["labels"]["7"] = json!(["security-finding", "needs-triage"]); + scenario.write_gh_state(&state); + } else { + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + } + let description = format!( + "{original}\n<!-- thirdshift:security-fix -->\nFix Ticket: https://github.com/acme/widgets/issues/{issue}\nFix Run: pending\nDay shift note: keep the input contract.\n" + ); + let (path, field) = if private { + ("issues/7", "body") + } else { + ("security-advisories/GHSA-finding-0", "description") + }; + scenario.agent_does_for(issue, &format!( + "gh api --method PATCH repos/acme/widgets/{path} --input - <<'RECORD'\n{}\nRECORD\nexit 1\n", + json!({field: description}) + )); + let failed = scenario.run(&["secure", "security-fix"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + let state = scenario.gh_state(); + let description = if private { + &state["bodies"]["7"] + } else { + &state["advisories"][0]["description"] + } + .as_str() + .unwrap(); + assert!( + description.contains("Day shift note: keep the input contract."), + "{description}" + ); + let next = scenario.run(&["secure", "security-fix"]); + assert_eq!(next.code, Some(0), "{}", next.stderr); + assert!( + next.stderr + .contains(&format!("failed Security fix #{issue} is still open")), + "{}", + next.stderr + ); + assert_eq!(scenario.claude_calls().len(), if private { 1 } else { 2 }); + } +} + +fn assert_a_failed_fix_record_outage_pauses_security(severity: &str) { + let scenario = with_reproduced_findings(&[severity]); + scenario.agent_does_in_session(1, &publish_fix("GHSA-finding-0")); + scenario.agent_does_for( + 8, + "gh fake fails 'api --method PATCH repos/acme/widgets/security-advisories/GHSA-finding-0'\nexit 1\n", + ); + + let failed = scenario.run(&["secure", "security-fix"]); + assert_eq!(failed.code, Some(1), "{}", failed.stderr); + assert!( + failed + .stderr + .contains("could not record the failed Security fix's ending"), + "{}", + failed.stderr + ); + assert_eq!(scenario.gh_state()["issues"]["8"], "OPEN"); + assert_eq!( + scenario.issue_labels(8), + vec!["security-fix", "in-progress"] + ); + + let mut state = scenario.gh_state(); + state["failing"] = json!([]); + scenario.write_gh_state(&state); + scenario.agent_does(&audit_script("[]")); + + let next = scenario.run(&["secure", "security-fix"]); + assert_eq!(next.code, Some(0), "{}", next.stderr); + assert!( + next.stderr.contains("failed Security fix #8 is still open"), + "Security resumed with an open failed fix: {}", + next.stderr + ); + assert_eq!(scenario.claude_calls().len(), 2); +} diff --git a/tests/setup.rs b/tests/setup.rs index 91548450..d65a2988 100644 --- a/tests/setup.rs +++ b/tests/setup.rs @@ -51,6 +51,7 @@ fn with_no_terminal_and_no_user_config_setup_writes_the_defaults() { "[activity]", "[spec]", "[pickup]", + "[security]", "[harness]", "[harness.claude]", "[harness.codex]", @@ -75,6 +76,8 @@ fn with_no_terminal_and_no_user_config_setup_writes_the_defaults() { assert_eq!(config["spec"]["parallel"].as_integer(), Some(3)); assert_eq!(config["pickup"]["limit"].as_integer(), Some(3)); assert_eq!(config["pickup"]["wait_minutes"].as_integer(), Some(30)); + assert_eq!(config["security"]["fix"].as_bool(), Some(false)); + assert_eq!(config["security"]["harness"].as_str(), Some("")); assert_eq!(config["harness"]["default"].as_str(), Some("claude")); for harness in ["claude", "codex", "agy", "grok", "muse", "opencode"] { for key in ["model", "effort"] { @@ -188,6 +191,8 @@ const EFFORT: &str = "Effort for claude"; const MERGE: &str = "Merge run?"; const BASE_FIX: &str = "Every Run may start a Base fix when the Base branch's CI is red?"; const PULL: &str = "fast-forward"; +const SECURITY_REVIEW: &str = "Runs review their change for Security findings?"; +const SECURITY_FIX: &str = "Security runs may fix reproduced findings?"; const NOTIFY: &str = "Run notifications, an email"; const TO: &str = "Send Run notifications to"; const FROM: &str = "Send them from"; @@ -197,6 +202,97 @@ const KEPT: &str = "Resend API key (input hidden, Enter keeps the saved one):"; const WROTE_CREDENTIALS: &str = "wrote the Credentials"; const KEY: &str = "re_secret_123"; +#[test] +fn setup_can_enable_security_review_and_keeps_it_as_the_next_default() { + let scenario = Scenario::new(); + for (answer, choices) in [("y", "[y/N]"), ("", "[Y/n]")] { + let result = setup_on_terminal( + &scenario, + &[], + &[ + (HARNESS, ""), + (MODEL, ""), + (EFFORT, ""), + (MERGE, ""), + (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, answer), + (NOTIFY, ""), + ], + ); + assert!( + result + .stderr + .contains(&format!("{SECURITY_REVIEW} {choices}")) + ); + let config: toml::Table = result.user_config.unwrap().parse().unwrap(); + assert_eq!(config["security"]["review"].as_bool(), Some(true)); + } +} + +#[test] +fn setup_asks_once_about_security_fixing_with_off_as_the_default_and_writes_yes() { + let scenario = Scenario::new(); + let result = setup_on_terminal( + &scenario, + &[], + &[ + (HARNESS, ""), + (MODEL, ""), + (EFFORT, ""), + (MERGE, ""), + (PULL, ""), + (SECURITY_FIX, "y"), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ], + ); + assert_eq!(result.stderr.matches(SECURITY_FIX).count(), 1); + assert!(result.stderr.contains(&format!("{SECURITY_FIX} [y/N]"))); + let text = result.user_config.unwrap(); + let config: toml::Table = text.parse().unwrap(); + assert_eq!(config["security"]["fix"].as_bool(), Some(true)); + assert_eq!(config["security"]["harness"].as_str(), Some("")); + assert!(text.contains("# the Harness a Security run uses unless its command names one")); +} + +#[test] +fn rerunning_setup_keeps_security_answers_as_defaults_and_can_turn_fixing_off() { + let scenario = Scenario::new(); + let saved = "# my Security settings\n[security]\nfix = true # keep this choice\nharness = 'codex' # use Codex\n"; + scenario.user_config_is(saved); + + for (answer, enabled, choices) in [ + ("", true, "[Y/n]"), + ("n", false, "[Y/n]"), + ("", false, "[y/N]"), + ] { + let result = setup_on_terminal( + &scenario, + &[], + &[ + (HARNESS, ""), + (MODEL, ""), + (EFFORT, ""), + (MERGE, ""), + (PULL, ""), + (SECURITY_FIX, answer), + (SECURITY_REVIEW, ""), + (NOTIFY, ""), + ], + ); + assert!(result.stderr.contains(&format!("{SECURITY_FIX} {choices}"))); + assert_eq!(result.stderr.matches(SECURITY_FIX).count(), 1); + let text = result.user_config.unwrap(); + let config: toml::Table = text.parse().unwrap(); + assert_eq!(config["security"]["fix"].as_bool(), Some(enabled)); + assert_eq!(config["security"]["harness"].as_str(), Some("codex")); + assert!(text.starts_with("# my Security settings\n[security]\n")); + assert!(text.contains("# keep this choice\n")); + assert!(text.contains("harness = 'codex' # use Codex\n")); + } +} + #[test] fn cancelling_a_claude_setup_check_cleans_up_without_writing_or_retrying() { use std::time::{Duration, Instant}; @@ -344,6 +440,8 @@ fn cancelling_later_questions_after_successful_or_repeated_checks_restores_echo_ keys.extend([ TerminalStep::line(MERGE, ""), TerminalStep::line(PULL, ""), + TerminalStep::line(SECURITY_FIX, ""), + TerminalStep::line(SECURITY_REVIEW, ""), TerminalStep::line(NOTIFY, "y"), TerminalStep::line(TO, "me@example.com"), TerminalStep::line(FROM, ""), @@ -385,7 +483,10 @@ fn pasted_answers_are_left_available_to_later_terminal_questions() { let result = setup_on_terminal( &scenario, &[], - &[(HARNESS, " claude \n\n\n\n\ny\n café@example.com \n\n")], + &[( + HARNESS, + " claude \n\n\n\n\n\n\ny\n café@example.com \n\n", + )], ); let config: toml::Table = result.user_config.unwrap().parse().unwrap(); assert_eq!(config["harness"]["default"].as_str(), Some("claude")); @@ -426,6 +527,8 @@ fn a_partial_final_terminal_answer_is_trimmed_and_accepted_before_eof() { TerminalStep::line(EFFORT, ""), TerminalStep::line(MERGE, ""), TerminalStep::line(PULL, ""), + TerminalStep::line(SECURITY_FIX, ""), + TerminalStep::line(SECURITY_REVIEW, ""), // Queue EOF for both the partial answer and the next question. // macOS can finish both reads before another input action runs. TerminalStep::bytes(NOTIFY, b" y \x04\x04\x04"), @@ -465,6 +568,8 @@ fn notifications_on<'a>(rest: &[Keystrokes<'a>]) -> Vec<Keystrokes<'a>> { (EFFORT, ""), (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, "me@example.com"), (FROM, ""), @@ -490,18 +595,24 @@ fn on_a_terminal_pressing_enter_throughout_writes_what_setup_with_no_terminal_wr (EFFORT, ""), (MERGE, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, ""), ], ); assert_eq!(result.user_config, user_config(&unattended)); - for prompt in [MERGE, PULL, NOTIFY] { + for prompt in [MERGE, PULL, SECURITY_FIX, NOTIFY] { assert!( result.stderr.contains(prompt), "terminal: {}", result.stderr ); } + assert_eq!(result.stderr.matches(SECURITY_FIX).count(), 1); + assert!(result.stderr.contains(&format!("{SECURITY_FIX} [y/N]"))); + let config: toml::Table = result.user_config.unwrap().parse().unwrap(); + assert_eq!(config["security"]["fix"].as_bool(), Some(false)); } #[test] @@ -574,7 +685,8 @@ fn ctrl_c_during_the_questions_writes_no_user_config() { (EFFORT, ""), (MERGE, "y"), (BASE_FIX, ""), - (PULL, CTRL_C), + (PULL, ""), + (SECURITY_FIX, CTRL_C), ], ); @@ -599,6 +711,8 @@ fn ctrl_c_during_the_questions_leaves_an_existing_user_config_unchanged() { (EFFORT, ""), (MERGE, "n"), (PULL, "y"), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, "y"), (TO, CTRL_C), ], diff --git a/tests/setup_offer.rs b/tests/setup_offer.rs index 2974a968..e9c7bacb 100644 --- a/tests/setup_offer.rs +++ b/tests/setup_offer.rs @@ -28,6 +28,8 @@ const EFFORT: &str = "Effort for claude"; const MERGE: &str = "Merge run?"; const BASE_FIX: &str = "Every Run may start a Base fix when the Base branch's CI is red?"; const PULL: &str = "fast-forward"; +const SECURITY_REVIEW: &str = "Runs review their change for Security findings?"; +const SECURITY_FIX: &str = "Security runs may fix reproduced findings?"; const NOTIFY: &str = "Run notifications, an email"; const KEY: &str = "re_secret_123"; @@ -82,6 +84,8 @@ fn accepting_and_choosing_merge_always_makes_that_run_a_merge_run() { (MERGE, "y"), (BASE_FIX, ""), (PULL, ""), + (SECURITY_FIX, ""), + (SECURITY_REVIEW, ""), (NOTIFY, ""), ], ); diff --git a/tests/spec_run.rs b/tests/spec_run.rs index f99770bf..95096cd3 100644 --- a/tests/spec_run.rs +++ b/tests/spec_run.rs @@ -109,6 +109,24 @@ fn assert_contains(text: &str, part: &str) { assert!(text.contains(part), "expected {part:?} in: {text}"); } +/// A complete Security review with no old findings writes its empty private +/// report as well as the public introduced-finding outcome. +fn security_review_script(introduced: &[&str]) -> String { + format!( + r#" +report=$(printf '%s' "$FAKE_CLAUDE_PROMPT" | sed -n 's/^Private report file: `\(.*\)`\.$/\1/p') +test -n "$report" +printf '%s\n' '[]' > "$report" +printf '%s\n' 'Security review: {outcome}' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"#, + outcome = serde_json::json!({ + "unaddressed_count": introduced.len(), + "findings": introduced, + "pre_existing_count": 0 + }) + ) +} + #[test] fn a_linear_spec_lands_each_ticket_in_order_then_opens_a_ready_spec_pr() { let scenario = linear_spec(); @@ -151,6 +169,226 @@ fn a_linear_spec_lands_each_ticket_in_order_then_opens_a_ready_spec_pr() { scenario.assert_cleaned_up("issue-20"); } +#[test] +fn security_review_runs_once_after_spec_review_over_the_whole_spec_branch() { + let scenario = linear_spec(); + let base_commit = scenario.launch_git(&["rev-parse", "main"]); + scenario.agent_does_for_in_session( + SPEC, + 1, + r#" +echo reviewed > spec-reviewed.txt +git add spec-reviewed.txt +git commit -q -m 'Complete Spec review' +"#, + ); + scenario.agent_does_for_in_session( + SPEC, + 2, + &format!( + r#" +test -f first.txt +test -f second.txt +test -f spec-reviewed.txt +gh pr view issue-20 --json isDraft | grep -q '"isDraft": true' +! git cat-file -e origin/issue-20:spec-reviewed.txt +merge_base=$(printf '%s' "$FAKE_CLAUDE_PROMPT" | sed -n 's/^Merge base commit: `\(.*\)`\.$/\1/p') +test "$merge_base" = "$(git rev-parse refs/remotes/origin/main)" +git diff "$merge_base"...HEAD --name-only > changed.txt +grep -qx first.txt changed.txt +grep -qx second.txt changed.txt +grep -qx spec-reviewed.txt changed.txt +rm changed.txt +echo fixed > security-fixed.txt +git add security-fixed.txt +git commit -q -m 'Fix reproduced Security finding' +{review} +"#, + review = security_review_script(&[]) + ), + ); + + let result = scenario.run(&[&spec_url(&scenario), "security-review", "merge"]); + + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(sessions_by_issue(&scenario), ["21", "22", "20", "20"]); + let calls = scenario.claude_calls(); + assert_contains( + calls[2]["prompt"].as_str().unwrap(), + "/thirdshift-code-review main", + ); + let prompt = calls[3]["prompt"].as_str().unwrap(); + assert_contains(prompt, "guidance mode"); + assert_contains(prompt, "Do not delegate auditors"); + assert_contains(prompt, "https://github.com/acme/widgets/issues/20"); + assert_contains( + prompt, + &format!( + "branch issue-20 against Base branch main with `git diff {}...HEAD`", + base_commit.trim() + ), + ); + assert_eq!(calls[3]["branch"], "issue-20"); + assert_eq!(spec_pr(&scenario)["state"], "MERGED"); + for file in [ + "first.txt", + "second.txt", + "spec-reviewed.txt", + "security-fixed.txt", + ] { + assert!( + scenario.origin_file("main", file).is_some(), + "missing {file}" + ); + } + let reviewed = result.stderr.find("spec-review: session ended").unwrap(); + let security = result + .stderr + .find("security-review: session started") + .unwrap(); + assert!(reviewed < security, "{}", result.stderr); +} + +#[test] +fn spec_review_leaves_the_push_to_delivery_after_the_optional_security_review() { + let scenario = linear_spec(); + let result = scenario.run(&[&spec_url(&scenario)]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let call = spec_review_call(&scenario); + assert_contains( + call["prompt"].as_str().unwrap(), + "Do not push: thirdshift pushes branch issue-20 after the optional Security review.", + ); +} + +#[test] +fn security_review_on_a_spec_obeys_config_and_command_overrides() { + for (config, word, enabled) in [ + ("", None, false), + ("[security]\nreview = true\n", None, true), + ( + "[security]\nreview = true\n", + Some("no-security-review"), + false, + ), + ( + "[security]\nreview = false\n", + Some("security-review"), + true, + ), + ] { + let scenario = linear_spec(); + scenario.user_config_is(config); + scenario.agent_does_for_in_session(SPEC, 2, &security_review_script(&[])); + let url = spec_url(&scenario); + let mut args = vec![url.as_str(), "merge"]; + args.extend(word); + + let result = scenario.run(&args); + + assert_eq!(result.code, Some(0), "{config} {word:?}: {}", result.stderr); + assert_eq!( + sessions_by_issue(&scenario), + if enabled { + vec!["21", "22", "20", "20"] + } else { + vec!["21", "22", "20"] + } + ); + assert_eq!(spec_pr(&scenario)["state"], "MERGED"); + } +} + +#[test] +fn security_findings_refusals_and_incomplete_reviews_hold_only_the_spec_self_merge() { + let introduced = security_review_script(&["Cross-tenant read"]); + for (script, cause) in [ + ( + introduced.as_str(), + "Security review left 1 unaddressed introduced finding(s): Cross-tenant read", + ), + ( + r#"printf '%s\n' 'API Error: [cyber] Private refusal evidence' > "$FAKE_CLAUDE_FINAL_MESSAGE""#, + "Security review refused: Claude Code's [cyber] safeguard refusal", + ), + ( + r#"printf '%s\n' 'Review incomplete' > "$FAKE_CLAUDE_FINAL_MESSAGE""#, + "Security review incomplete", + ), + ("exit 1\n", "Security review incomplete"), + ] { + for merge in [true, false] { + let scenario = linear_spec(); + scenario.agent_does_for_in_session(SPEC, 2, script); + + let result = scenario.run(&[ + &spec_url(&scenario), + "security-review", + if merge { "merge" } else { "no-merge" }, + ]); + + assert_eq!( + result.code, + Some(i32::from(merge)), + "{cause}: {}", + result.stderr + ); + assert_contains(&result.stderr, cause); + assert_eq!(sessions_by_issue(&scenario), ["21", "22", "20", "20"]); + let pr = spec_pr(&scenario); + assert_eq!(pr["state"], "OPEN"); + assert_eq!(pr["isDraft"], false); + let body = pr["body"].as_str().unwrap(); + assert_contains(body, "### Security"); + // Introduced finding titles and sanitized refusal/incomplete causes. + assert_contains( + body, + if cause.contains("Cross-tenant read") { + "Cross-tenant read" + } else { + cause + }, + ); + assert!(!body.contains("Private refusal evidence")); + assert_contains(body, "#21 landed"); + assert_contains(body, "#22 landed"); + assert_eq!(scenario.gh_state()["issues"]["20"], "OPEN"); + assert_eq!(scenario.origin_file("main", "first.txt"), None); + for ticket in ["issue-21", "issue-22"] { + assert_eq!(pr_from(&scenario, ticket).unwrap()["state"], "MERGED"); + } + } + } +} + +#[test] +fn a_spec_security_hold_still_repairs_ci_without_repeating_the_review() { + let scenario = linear_spec(); + scenario.agent_does_for_in_session(SPEC, 1, &checks_on_head(RED)); + scenario.agent_does_for_in_session(SPEC, 2, &security_review_script(&["Cross-tenant read"])); + scenario.agent_does_for_in_session( + SPEC, + 3, + &format!("{}{}", commits_fix(1), checks_on_head(GREEN)), + ); + + let result = scenario.run(&[&spec_url(&scenario), "security-review", "merge"]); + + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert_contains(&result.stderr, "Cross-tenant read"); + let prompts = spec_prompts(&scenario); + assert_eq!(prompts.len(), 3); + assert_contains(&prompts[0], "/thirdshift-code-review main"); + assert_contains(&prompts[1], "guidance mode"); + assert_contains(&prompts[2], "CI"); + assert_eq!( + scenario.origin_file("issue-20", "fix-1.txt").as_deref(), + Some("fix\n") + ); + assert_eq!(spec_pr(&scenario)["state"], "OPEN"); + assert_eq!(spec_pr(&scenario)["isDraft"], false); +} + #[test] fn a_spec_run_claims_the_spec_and_its_tickets_runs_change_no_label() { let scenario = linear_spec(); diff --git a/tests/support/mod.rs b/tests/support/mod.rs index 8e068de5..a4313a2f 100644 --- a/tests/support/mod.rs +++ b/tests/support/mod.rs @@ -78,7 +78,7 @@ pub const OWNER: &str = "acme"; pub const REPO: &str = "widgets"; /// Every Factory skill, by the name a session finds it under. -pub const FACTORY_SKILLS: [&str; 9] = [ +pub const FACTORY_SKILLS: [&str; 10] = [ "thirdshift-implement", "thirdshift-code-review", "thirdshift-pr", @@ -88,6 +88,7 @@ pub const FACTORY_SKILLS: [&str; 9] = [ "thirdshift-to-spec", "thirdshift-to-tickets", "thirdshift-codebase-design", + "thirdshift-security-audit", ]; /// How long a test waits for a Run to reach a point, such as starting the