Skip to content

Reproduce recorded Security findings - #566

Merged
JacobStephens2 merged 7 commits into
issue-427from
issue-542
Oct 8, 2026
Merged

JacobStephens2 merged 7 commits into
issue-427from
issue-542

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

after(Security audit)
  record findings privately
  preserve findings already triaged by the Day shift
  for each untriaged record, in order
    reproduce in a fresh Worktree at the recorded audited commit
    require a final outcome, reproduction notes and test text
    update the private record with the outcome and reproduced severity

Incomplete sessions leave the finding unchanged. The prompt asks for harmless local tests against untouched code, the skill's severity rubric and the fix size. Factory skills are linked into each Worktree; progress names each reproduction and outcome. A record edited or triaged during reproduction is preserved.

Closes #542

Evidence

  • CI repair — Before: the base merge added a required session::Purpose, while reproduction still called the old two-argument API. All four reported jobs failed with E0061.
    After: reproduction selects Purpose::Security; the Codex integration scenario now checks the audit, reproduction and both Resumes retain the Security launch policy. cargo fmt --check, cargo clippy --all-targets -- -D warnings, full cargo test, full cargo nextest run (1,541 passed), and dist build --print=linkage --output-format=json --artifacts=local --target=x86_64-unknown-linux-musl pass locally. The static musl binary also passes thirdshift version. Native macOS validation runs in CI.

  • Before: cargo test --test security_run a_reproduction_scores_the_finding_and_keeps_its_test_in_the_private_record -- --exact failed: severity was null instead of high, with no reproduction session.
    After: passes; the record carries severity, notes, fix size and test text.

  • Before: each Spec review regression failed at its asserted behavior: an ordinary Markdown heading deleted evidence, a published advisory received fresh test evidence and a replacement grade, and a private issue rejected informational.
    After: all three supplied commands pass, and all three tests are retained:

    cargo test --test security_run reproduction_preserves_an_original_finding_reproduction_heading -- --exact
    cargo test --test security_run a_published_finding_keeps_new_reproduction_evidence_private_and_its_grade -- --exact
    cargo test --test security_run a_private_reproduction_accepts_the_rubrics_informational_severity -- --exact
  • Validation: all 32 Security run scenarios pass; cargo test passes; cargo clippy --all-targets -- -D warnings and cargo fmt --check pass. Generated prompts and the Prompts and skills page were regenerated with UPDATE_PROMPTS=1 cargo test prompts_page (9 tests pass).

Seams under test, from #427:

  • The end-to-end scenario harness: the real binary, real git, fake GitHub and scripted agent CLIs.
  • The Pass seam's in-memory double: Security run operation ordering.

Merge Danger

Door: two-way

Blast Radius: Security

Security runs launch additional sessions and update untriaged private records. Informational findings on public repositories stop with an explicit cause while the Day shift decides how their grade should be represented.

Unaddressed findings

Standards

None. Standards reported 0 findings.

Spec

  • S3, public advisory representation of informational: the private-issue bug is fixed. The remaining call belongs to the Day shift and is tracked in the policy issue. The skill's rubric includes informational, while GitHub's advisory update API accepts critical/high/medium/low/null. Neither the Spec nor an ADR authorizes substituting another grade. The guard preserves the advisory and fails with an explicit cause. cargo test --test security_run an_informational_advisory_requires_a_day_shift_decision_without_an_invented_grade -- --exact passes and exercises that case.

Spec reported 3 findings: S1 and S2 are addressed; S3 is fixed for private issues and its public-advisory policy remains above. Both axes reviewed through b495ebc, including every changed file. Unread changed files: none on either axis.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 marked this pull request as ready for review October 8, 2026 11:31
The base branch added a required session Purpose, but reproduction still called the old two-argument API. This caused E0061 in both CI test jobs and both release builds.

Pass Purpose::Security and extend the Codex integration scenario to cover reproduction and its Resume with the Security launch policy.

Validated with cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test, cargo nextest run (1541 passed), and the Linux musl dist artifact build plus its version command.
Preserve private Security notifications alongside sequential finding reproduction. Retain safe record metadata through failures, report successfully stored reproduction severity, and preserve Day shift triage and private evidence.

Keep both branches' Security tests and cover notifications for completed and failed reproductions in advisory and private-issue storage. Validated with cargo check --locked --all-targets, cargo clippy --locked --all-targets -- -D warnings, cargo fmt --check, and the full cargo test --locked suite (1554 passed, 1 ignored).
Preserve the waiting-finding and unchanged-Base audit gates from #541 alongside #542's sequential reproduction sessions and post-audit record refresh. Retain both branches' tests and adapt reproduction fixtures and repeat audits to the new gates.

Validation: cargo check --all-targets; cargo clippy --all-targets -- -D warnings; cargo test --quiet (1,562 passed, 1 ignored); cargo fmt --check.
@JacobStephens2
JacobStephens2 merged commit f397e26 into issue-427 Oct 8, 2026
13 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-542 branch October 8, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant