fix(ci): the hardening audit files no issue for a tested fix #3479

Problem

.github/workflows/defect-pr-hardening-audit.yml files an issue for each merged defect pull request that adds no line to a declared document. A regression test or a new gate does not count. Each issue has the same text, with no root cause and no lesson, so a person or a job cannot plan from it. Each one also uses an intake round.

The family now has 8 open issues and 8 merged pull requests, from 2026-11-02T00:08Z to 2026-11-04T15:29Z. The last pair is after the window 2026-11-03T15:05Z to 2026-11-04T15:05Z.

The GitHub file lists show that each of the 8 pull requests added tests and an openspec/specs delta. #3381 and #3399 also added stage gates.

The cause: audit_pr_hardening (src/devops_cli/github/defect_classifier.py:155) sums additions only on CONST_HARDENING_SURFACES (:174-179, src/devops_cli/config/constants.py:3948).

The audit also goes against the document that owns the rule. docs/SELF_IMPROVEMENT.md:165-168 says that a check that fails when the problem returns guards it, and that a rule goes into a document only when no check is possible. The issue text cites AGENTS.md, and a search of AGENTS.md and CONTRIBUTING.md finds no such rule.

Two older items cannot pass for the same reason, because the audit counts only the additions of the defect pull request itself. These are the scenario hardening_report_passes of #2526 and the refine scenario of #2660 (the audit reports present and exits 0).

Expected

The audit counts additions under tests/ and openspec/specs/ as a guardrail. A defect pull request with no test, no spec and no surface line appears in the job summary, and the workflow files no issue. The workflow text stops citing AGENTS.md.

Deletion of the audit is an option, but only the owner can choose it, because AGENTS.md rule 5 gives only the owner the right to lower a quality threshold.

Acceptance criteria

• A test of audit_pr_hardening: a defect pull request with additions under tests/ is hardened.
• A test of audit_pr_hardening: a defect pull request with additions under openspec/specs/ is hardened.
• A test of audit_pr_hardening: a defect pull request with no test, no spec and no surface line is not hardened.
• uv run devops pr hardening-report --pr 3467 reports it as hardened.
• A pull request that is not hardened appears in the job summary of the workflow.
• The workflow has no step that creates an issue, and it does not pass --fail-on-missing.
• The workflow text does not cite AGENTS.md, and it agrees with docs/SELF_IMPROVEMENT.md:165-168.
• The 8 family issues above end as not planned, each with the reason in a comment.
• The same pull request decides fix(hardening): add guardrails to hardening surfaces #2526 and docs(architecture): add the guardrail of #2504 to a declared hardening surface #2660, and a comment on each states the decision.

Existing tool considered

The GitHub job summary and the existing devops pr hardening-report command (src/devops_cli/commands/pr.py:2083-2122), which is the only user of the classifier. No new tool.

Related

Related: #2526, #2660, #1979

Proposed design

Problem statement

The hardening audit files a generic issue for each merged defect pull request that adds no line to a declared document. A regression test or a living spec does not count. The audit reports a tested fix as unhardened, and each filed issue has no root cause. The workflow also cites AGENTS.md, which holds no such rule. The change counts additions under tests/ and openspec/specs/, writes an unhardened pull request to the job summary, and files no issue.

Acceptance criteria

Requirement: A test counts as a hardening surface

audit_pr_hardening SHALL count an addition under tests/ as a hardening guardrail.

Scenario: A defect pull request adds a regression test

• WHEN A defect pull request adds lines under tests/ and adds no line under another hardening surface
• THEN The audit reports the pull request as hardened and names tests/.

Requirement: A living spec counts as a hardening surface

audit_pr_hardening SHALL count an addition under openspec/specs/ as a hardening guardrail.

Scenario: A defect pull request adds a living spec requirement

• WHEN A defect pull request adds lines under openspec/specs/ and adds no line under another hardening surface
• THEN The audit reports the pull request as hardened and names openspec/specs/.

Requirement: No guardrail leaves the pull request unhardened

audit_pr_hardening SHALL report a defect pull request with no addition under a hardening surface as not hardened.

Scenario: A defect pull request adds only product code

• WHEN A defect pull request adds lines under src/ only
• THEN The audit reports the pull request as not hardened and names no surface.

Requirement: Pull request 3467 reports hardened

The audit SHALL report the recorded file list of pull request 3467 as hardened.

Scenario: The recorded file list of pull request 3467 is audited

• WHEN A test audits the file list recorded for pull request 3467
• THEN The audit reports the pull request as hardened through its tests/ additions.

Requirement: The job summary shows the report

The workflow SHALL append the hardening report to $GITHUB_STEP_SUMMARY.

Scenario: The workflow file is read

• WHEN A test reads .github/workflows/defect-pr-hardening-audit.yml
• THEN The report step writes its output to $GITHUB_STEP_SUMMARY.

Requirement: The workflow files no issue

The workflow MUST NOT create an issue and MUST NOT pass --fail-on-missing to the report.

Scenario: The workflow file is read

• WHEN A test reads .github/workflows/defect-pr-hardening-audit.yml
• THEN The file holds no issues create command and no --fail-on-missing argument.

Requirement: The workflow holds no issue write permission

The workflow MUST NOT request the issues: write permission.

Scenario: The permissions block is read

• WHEN A test reads the permissions block of .github/workflows/defect-pr-hardening-audit.yml
• THEN The block holds no issues: write entry.

Requirement: The workflow cites the owning document

The workflow text MUST NOT cite AGENTS.md and SHALL cite docs/SELF_IMPROVEMENT.md.

Scenario: The workflow text is read

• WHEN A test reads the text of .github/workflows/defect-pr-hardening-audit.yml
• THEN The text holds no AGENTS.md and holds docs/SELF_IMPROVEMENT.md.

Existing Tool Considered

No alternatives check has run. The GitHub Actions job summary and the existing devops pr hardening-report command cover the need. The change adds no new tool. The command lacks the tests/ and openspec/specs/ surfaces. The job summary cannot enforce a gate or file an issue. The change adds two path prefixes to CONST_HARDENING_SURFACES and edits one workflow.

Key questions

• Which additions does audit_pr_hardening count today?: It sums additions only on CONST_HARDENING_SURFACES. [sources: src/devops_cli/github/defect_classifier.py:155, src/devops_cli/config/constants.py:3948]
• Which additions count as a hardening guardrail?: The audit counts additions under the declared surfaces, tests/ and openspec/specs/. It counts no addition under openspec/changes/. (best practice) [sources: docs/SELF_IMPROVEMENT.md:165-168, CONTEXT.md, src/devops_cli/config/constants.py:3948, src/devops_cli/github/defect_classifier.py:155]
  • Chosen: Add tests/ and openspec/specs/ to CONST_HARDENING_SURFACES. A regression test and a living spec fail when the defect returns, and docs/SELF_IMPROVEMENT.md names that check as the guard.
  • Rejected: Count openspec/changes/ as well. Every item pull request writes a change folder, so every defect pull request would count as hardened.
  • Rejected: Delete the audit. Only the owner can lower a quality threshold, and the body's Expected keeps the audit.
• How does the workflow report an unhardened pull request?: The workflow appends the report to the job summary and exits 0. It files no issue and passes no --fail-on-missing. (best practice) [sources: .github/workflows/defect-pr-hardening-audit.yml, docs/SELF_IMPROVEMENT.md:165-168]
  • Chosen: Write the job summary. The merge already happened, so a failed post-merge check cannot block it and marks the default branch red.
  • Rejected: File an intake issue. The text is generic, it has no root cause and it uses an intake round.
  • Rejected: Fail the report step. The body's Expected asks for no --fail-on-missing, and a post-merge failure cannot block the merge.
• Which document does the workflow cite for the hardening rule?: The workflow cites docs/SELF_IMPROVEMENT.md, which owns self-hardening. It cites no AGENTS.md. (best practice) [sources: docs/SELF_IMPROVEMENT.md:165-168, AGENTS.md]
  • Chosen: Cite docs/SELF_IMPROVEMENT.md. The document states that a check guards a rule and that a rule goes into a document only when no check is possible.
  • Rejected: Keep AGENTS.md. AGENTS.md holds no such rule, and its routing table sends self-improvement work to docs/SELF_IMPROVEMENT.md.
• Does --fail-on-missing stay in the command?: The option stays in devops pr hardening-report. The workflow stops passing it. (best practice) [sources: src/devops_cli/commands/pr.py]
  • Chosen: Keep the option. A person or another job can still ask the command to fail, and the issue asks only that the workflow stop passing the option.
  • Rejected: Remove the option. The issue does not ask for its removal, and the option is a documented command flag.
• Does the workflow keep the issues: write permission?: The workflow drops issues: write. It keeps contents: read and pull-requests: read. (best practice) [sources: .github/workflows/defect-pr-hardening-audit.yml]
  • Chosen: Drop issues: write. No step creates or edits an issue after the change, so the permission is unused.
  • Rejected: Keep issues: write. An unused write permission widens the token that the workflow holds.
• How does a test check pull request 3467 offline?: A test audits a file list recorded for pull request 3467. It makes no GitHub request. (best practice) [sources: src/devops_cli/github/defect_classifier.py:155, src/devops_cli/commands/pr.py:2088]
  • Chosen: Audit a recorded file list. The audit takes a file list as input, so a fixture proves the real case offline.
  • Rejected: Read pull request 3467 from GitHub in the test. The test must run offline and must not need a token.

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

None.