Important
Archived 2026-10-15. Superseded by Assay, a reference architecture for deterministic agent control. Why: a sentence in a system prompt is not an enforced control. CrewScore measures written coverage only, so the lasting value is the 356-prompt corpus study and the control taxonomy, kept here as the record. The last release, 0.6.14, stays on PyPI. The rest of this README is unchanged.
An offline linter that checks an AI agent's system prompt against 23 published controls (injection defense, human approval, cost limits, stop conditions) and fails CI when a control you require is missing. No model, no API key, open rules.
Try it live in your browser, no install →
Quickstart · Watch the film · How it works · CI · Guardrails · Status and limits
Created and maintained by Sarosh Hussain, an independent open-source maintainer. Technical claims are grounded in the code, tests, and cited validation material.
A support assistant ships on Friday. Does its prompt say a human approves a refund? Does it say what to do with instructions hidden in a customer's message, or when to stop? Usually nobody checks, because checking means reading every prompt by hand, and runtime red-teaming is slow, paid and hard to reproduce.
We scanned 356 publicly collected agent prompts: 83 production-labeled prompts and 273 general-purpose prompts. Among the production-labeled subset, median coverage was 10 of 100. GPT-Store median: 0 of 100. Numbers → · Shareable card →
CrewScore makes the written part cheap to check. It reads prompt files, and
SYSTEM_PROMPT / system_prompt string literals in .py, .ts and .js
source, and reports which of 23 published controls are written down. It
plans fixes without touching your file unless you pass --apply. In CI you name the
controls your workflow needs, and the build fails only when one of those is
missing.
Public-good thesis: written safeguards should be inspectable with an offline, open ruleset before teams depend on runtime claims that are harder to reproduce or independently review.
pip install crewscore
mkdir crewscore-try && cd crewscore-try # a fresh folder, so scan . sees only this prompt
mkdir -p prompts && echo "You are a friendly support assistant for an online store." > prompts/support.md
crewscore test --prompt-file prompts/support.md # coverage N/23 and the first gap to review
crewscore fix --prompt-file prompts/support.md --plan # which dimensions need wording; writes nothing
crewscore scan . --require human_gate.approval_required # exit 2: the required control is missing
echo "A human must approve any refund or account change before it happens." >> prompts/support.md
crewscore scan . --require human_gate.approval_required # exit 0: the control is written downPython 3.11 or later. The CLI scores locally: no account, no network call, no
LLM. Inside an existing project, scan . --require checks every prompt it
finds, so use test --prompt-file <file> --require <control> to gate one file.
Watch the 90-second film → A one-line support prompt covers 0 of 23 controls. The release gate requires a human-approval rule and fails with exit 2. One sentence, written by a person, turns it to exit 0, and the film ends in the rule source that matched it.
flowchart LR
SRC["Prompt files, AGENTS.md,<br/>SYSTEM_PROMPT literals in .py / .ts / .js"] --> KIND{"Artifact type<br/>by filename and path"}
KIND -- "coding-agent config<br/>AGENTS.md, CLAUDE.md, .cursorrules" --> SMELL["Configuration smells<br/>never change the score"]
KIND -- "agent system prompt" --> RULES["Open ruleset<br/>deterministic patterns + negation guards"]
RULES --> CTRL["23 controls<br/>across 8 dimensions"]
CTRL --> OUT1["coverage N/23,<br/>first gap, fix plan"]
CTRL --> GATE{"--require control"}
GATE -- missing --> FAIL(["exit 2: CI fails and names the control"])
GATE -- written --> PASS(["exit 0"])
CTRL --> CI["Prompt-free JSON, SARIF,<br/>sticky PR comment"]
Each rule is a deterministic pattern with guards, so "do not ask permission" never counts as a human-approval rule. Several rules roll up to one control, and the gate checks controls by name rather than an average. How scoring works → · Architecture →
- Offline, deterministic scoring. The CLI and Action score with no model, no API key and no network call, and the same prompt text always gets the same governance findings. Configuration-smell checks also read the file's Git history and neighboring linter config. The browser checker scores locally too; the site itself sends anonymous, opt-out usage telemetry, described in its privacy page.
- Writes nothing unless asked.
fix --planlists what is missing; onlyfix --applywrites template text, and a person decides what stays in the prompt. - Gate on named controls, never the average.
--requireandrequired-controlsfail only on the controls you name. - CI output is prompt-free by default. Logs, JSON, Markdown and SARIF carry rule IDs and control labels, not the matched prompt text. The deprecated
include-snippets: "true"input re-admits snippets to JSON and Markdown; SARIF stays prompt-free. - Config files are not scored as prompts.
AGENTS.mdand similar files get a smell verdict, not a governance grade. - Every scoring change is measured. The CHANGELOG records each change and its corpus delta.
CrewScore is a checklist, not a benchmark. The number is the share of 23 published controls your prompt states — nothing about whether they are well specified, mutually consistent, or obeyed at runtime.
| What the prompt does | What it scores |
|---|---|
| Nothing written down | 0 |
| One control in each of the 8 dimensions | 36 |
| All 23 controls | 100 |
| One control restated five different ways | same as stating it once |
So a low score is actionable — you probably have not written down an injection policy, a human gate, or a safe-stop rule, and those are worth writing. A high score means the text is present, not that the agent obeys it. Don't rank prompts, teams, or vendors by this number, and don't treat a threshold as a safety bar. Prefer the findings to the total.
Three dimensions — Cost, Compliance, Audit — ship with known-thin
construct validity and say so. Ruleset 0.6.0 tightened their patterns against
measured corpus false positives; Compliance is still not lawful handling.
📄 The validation study → — including the arithmetic
showing our own scale was broken through 0.1.0, which we published before
fixing.
📊 Measured against 356 publicly collected prompts → — Cliff's δ = 0.614 separating production-labeled agent prompts from general-purpose ones, generated by a committed harness rather than typed by hand.
crewscore scan . # prompts + AGENTS.md + inline SYSTEM_PROMPT=...
crewscore scan . --no-inline # file discovery only
crewscore init . # prompt-free regression baseline + PR workflow
crewscore scan . --fail-on-regression --baseline .crewscore-baseline.json
crewscore scan . --require human_gate.approval_required # one-control CI habit
crewscore test --prompt-file ./prompt.md # coverage N/23 + first gap to review
crewscore fix --prompt-file ./prompt.md --plan # what's missing, no writes
crewscore rules --concepts # the 23 controls, and the rules behind themFull CLI reference → · How scoring works →
Start report-only — it never fails a build:
- uses: shmindmaster/crewscore@v2
with:
scan-path: "."When you know which controls your workflow actually needs, name them — the build then fails only when a named control is missing, never on the coverage average:
- uses: shmindmaster/crewscore@v2
with:
scan-path: "."
required-controls: "human_gate.approval_required,safe_stop.stop_condition"
sarif: "crewscore.sarif"Posts a sticky PR comment with the open rule findings (give the job
permissions: pull-requests: write; without it the action logs a warning and
moves on). Failed gates surface as ::error annotations naming the control.
CI output is prompt-free by default: rule IDs, dimensions, status, and
control labels — never the matched text of the prompt that was scanned, because
that build log is more public than the prompt is. include-snippets: "true"
re-admits it to JSON and Markdown output and is deprecated; it will be removed
after one release. SARIF remains prompt-free regardless of that input.
Guard downstream steps on the scored output, not on score — an empty
score casts to 0.
Action inputs, outputs, and the CLI variant →
CrewScore judges two kinds of file, and tells you which it thinks it is looking at. Detection is by filename and path — never by sniffing content.
| Artifact | Examples | Judged on |
|---|---|---|
| Coding-agent config | AGENTS.md, CLAUDE.md, .cursorrules |
Configuration smells |
| Agent system prompt | system-prompt.md, anything under prompts/ or agents/ |
The 8 governance dimensions |
A file saying "always use pnpm" is telling a coding agent how to work in your repo. It has no reason to contain HIPAA language, and scoring it against that is a category error.
We know the size of that error because we measured it: against the 100
most-starred repos with an AGENTS.md
(arXiv:2606.15828), the governance ruleset
put all 100 in the worst tier. A scale the entire population fails carries
no information. So config files get a smell verdict instead — and in --json,
no governance grade at all.
crewscore test --prompt-file AGENTS.md
# -> CONFIG: NO SMELLS DETECTED (not "0/100 CRITICAL GAPS")Problems in the shape of an instruction file rather than its content, from a published catalog — Configuration Smells in AGENTS.md Files (dos Santos et al., 2026), which found 91 of 100 popular projects carried at least one.
| Smell | Heuristic | Found in |
|---|---|---|
| Context Bloat | ≥ 200 lines | 42% of studied projects |
| Lint Leakage | Style rules a configured linter already enforces | 62% |
| Init Fossilization | Tracked by git with exactly one commit | 24% |
The paper's other three smells need an LLM to detect. We would rather ship three honest detectors than six approximate ones. Lint Leakage is an approximation of the paper's detector and says so in its output; Init Fossilization cannot tell "never needed revising" from "never got revised."
Smells never change the score. Folding them in would silently change what
every existing --threshold means.
| Validation | What the number does and does not measure |
| Corpus validation | Generated result over 356 publicly collected prompts |
| Scoring and controls | Formula, 23 controls, charter, governance |
| CLI | Every command and flag |
| GitHub Action | Action inputs/outputs and CLI-in-CI |
| Policies and SARIF | Regression and required-control CI without score gating |
| Architecture | Modules, data flow, lean target |
| Development | Local setup, rules, packaging, media |
| Live eval handoff | Promptfoo / garak after structural gate |
| Roadmap | Available work and deliberately deferred capabilities |
| Security | Private vulnerability reporting |
| Community discussions | Questions, adoption feedback, and open-ended ideas |
| Comparison | Other tools, and what to use after this one |
| CHANGELOG | Including every scoring change and its measured delta |
Current release: see PyPI and the
CHANGELOG.
Ruleset 0.6.0. The GitHub Action is shmindmaster/crewscore@v2, and the
checker at crewscore.ai runs the same engine locally
in your browser. The project is Alpha: the JSON payload and exit codes can
still change between releases, and every change is recorded.
What this is not: live adversarial red-teaming, runtime tool-gate enforcement, a security or compliance certification, or proof that the model will obey the text. Three dimensions (Cost, Compliance, Audit) ship with disclosed thin construct validity. For runtime testing after the structural gate, see the live eval handoff to Promptfoo or garak.
Roadmap: framework adapters that extract prompts from LangGraph, CrewAI and AutoGen graphs, and optional live adversarial testing (not the default path). See the roadmap.
-
Contributing: read CONTRIBUTING.md and the development guide. A new rule needs a corpus measurement and a negative test. Good first issues are labelled
good first issue; questions go to Discussions.git clone https://github.com/shmindmaster/crewscore.git cd crewscore pip install -e ".[dev]" pytest
-
Security: report vulnerabilities privately through a GitHub security advisory, as described in SECURITY.md.
-
License: MIT · Governance · Roadmap
Related open source by the same maintainer: Assay keeps an agent's decision out of its side effects, and GitPin makes agent-authored pull requests carry exact, re-checkable evidence. More at saroshhussain.com.