This repository was archived by the owner on Oct 15, 2026. It is now read-only.
Repository navigation
docs: verification pack (evidence ladder, tests, readiness) #114
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
94e4627
docs(verification): add verification pack (overview, functional, tech…
shmindmaster 5324792
docs(verification): plan the technical deep dive and vertical short
shmindmaster 8b21aed
docs(verification): tests, readiness verdict and status block at 90598a1
shmindmaster File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # GitPin Coverage Matrix | ||
|
|
||
| Scope: GitPin 0.6.4. The code was read at `e4c550e`. The suites ran at main `90598a1` (2026-10-15); the commits since `e4c550e` change HTTP loopback binding (#112), the CI trigger (#113) and the README (#111). Rungs use the evidence ladder: Planned, Implemented, Integrated, Tested, Demonstrated, Production-Verified. | ||
|
|
||
| How to read the rung column: it is the highest rung this pack can support from what was read or checked. Source and file paths give Implemented. A live trace through the pieces gives Integrated; the published `gitpin@0.6.4` gate was run against this repository's own history (see [TESTS.md](TESTS.md#production-smoke-checks-2026-10-15)). "Tested" needs an executed command with a result; the test run column points to [TESTS.md](TESTS.md). "Demonstrated" comes from the delivered films, listed in [DEMOS.md](DEMOS.md). "Production-Verified" needs a deployed SHA plus a health check and a smoke test of the path; the deploy column lists what was checked. | ||
|
|
||
| ## Evidence matrix | ||
|
|
||
| | Capability | Journey | Surface | Logic module | Integration | Test files | Test run | Demo | Deploy status | Rung | | ||
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ||
| | PR evidence gate (`gitpin gate --base --head`) | Run gate locally; fix failed gate | CLI | `src/gate.ts`, `src/gate-policy.ts`, `src/gate-types.ts`, `src/cli-gate.ts` | Local `git` only | `src/gate.test.ts`, `src/gate-hero.test.ts` | see TESTS.md | see DEMOS.md: gate fails on `uncovered-change` at exit 1, a claim with line + sha256, gate passes at exit 0, plain-git sha256 recheck, `gate.ts` coverage code | npm `gitpin` 0.6.4 matches main's version (checked `npm view`) | Production-Verified (npm 0.6.4 smoke); Tested; Demonstrated | | ||
| | Coverage by merge-base diff and `uncovered-change` | Fix a failed gate | CLI | `src/gate.ts:48-60,169-173` | Local `git` | `src/gate.test.ts`, `src/gate-hero.test.ts` | see TESTS.md | see DEMOS.md | as above | Production-Verified (npm smoke: `uncovered-change` on 2 paths); Tested; Demonstrated | | ||
| | Base policy, `policyChanges: block`, limits | Install policy; run gate | CLI, Action | `src/gate-policy.ts`, `src/gate.ts:61-67` | Local `git` | `src/gate.test.ts` (base policy, blocked policy change) | see TESTS.md | see DEMOS.md (policy trust boundary is not named in the facts given; check there) | as above | Tested | | ||
| | Per-claim line-slice sha256 verify | Run gate; plain-git recheck | CLI, Action | `src/gate.ts:108-115` | Local `git` | `src/gate.test.ts` (hash mismatch, deleted file, replace refs) | see TESTS.md | see DEMOS.md | as above | Production-Verified (npm smoke: `evidence-verified`); Tested; Demonstrated | | ||
| | Hero fixture `pnpm demo:gate` | Hero demo | CLI | `scripts/demo-gate-hero.mjs` | Synthetic scratch git repo | `src/gate-hero.test.ts` | see TESTS.md | see DEMOS.md | Local only; labelled synthetic | Tested; Demonstrated | | ||
| | GitHub Action | Run gate in CI | Action | `action.yml`, `scripts/render-gate-action-report.mjs` | npm registry (installs `gitpin@0.6.4`), GitHub runner | `src/gate-action.test.ts`; self-run via `.github/workflows/evidence-gate.yml` | see TESTS.md | Not in the films' list above | The repo's own evidence gate passed on PRs #111 and #113 | Integrated; Tested | | ||
| | 12 `pin.*` MCP tools, stdio | Connect client; prove; verify | MCP | `src/server.ts`, `src/pin-tools.ts`, `src/wiki*.ts`, `src/git*.ts` | MCP SDK, local Git | `src/wiki.test.ts`, `src/git.test.ts`, `src/evidence.test.ts`, `src/structure.test.ts` | see TESTS.md | Not shown in the two films | npm 0.6.4; no real MCP client session checked | Tested | | ||
| | MCP over HTTP (bearer, host allowlist, health) | Serve over HTTP | MCP | `src/http.ts` | MCP SDK Streamable HTTP | `src/http.test.ts` | see TESTS.md | Not shown | Container and remote deploy scripts exist (`Dockerfile.remote`, `scripts/verify-remote.mjs`, `deploy/`); no deployed instance checked | Tested; works locally only (no deployed instance) | | ||
| | Evidence prove / prove-set | Prove a claim | MCP, CLI | `src/evidence-prove.ts`, `src/evidence-set.ts`, `src/evidence-citation.ts` | Local Git | `src/evidence.test.ts` | see TESTS.md | Not shown | npm 0.6.4 | Tested | | ||
| | Evidence verify / verify-set / verify-cites | Verify a claim; verify citations | MCP, CLI | `src/evidence-verify.ts`, `src/cli-verify.ts` | Local Git | `src/evidence.test.ts`, `src/cli.test.ts` | see TESTS.md | Plain-git recheck appears in the films (see DEMOS.md) | npm 0.6.4 | Tested | | ||
| | Setup and health (`init`, `doctor`, `brief`) | Connect an MCP client | CLI | `src/onboarding.ts`, `src/doctor.ts`, `src/context-brief.ts` | Client config files on disk | `src/onboarding.test.ts`, `src/doctor.test.ts` | see TESTS.md | Not shown | npm 0.6.4 | Tested | | ||
| | Pages site | View product page | Site | `site/`, `scripts/build-site.mjs` | GitHub Pages | `tests/browser/site.spec.mjs`, `tests/browser/analytics-privacy.spec.mjs`, `src/site-build.test.ts` | see TESTS.md | Not shown | Redeployed from `90598a1` on 2026-10-15 (run 38737568718; the live footer had still shown the attribution removed in #98). HTTP 200, and the new footer was confirmed live. | Production-Verified | | ||
| | MCP Registry listing | Discover server | Registry | `server.json` | MCP Registry | `scripts/verify-mcp-registry.mjs` (run by `pnpm validate`) | see TESTS.md | Not shown | Registry lists `io.github.shmindmaster/gitpin` 0.6.4 as `isLatest: true`, status active (checked 2026-10-15) | Production-Verified for the listing; install from the registry not checked | | ||
| | npm package | Install `npx gitpin` | CLI | `package.json`, `scripts/verify-package.mjs`, `.github/workflows/release.yml` | npm | `src/launch-readiness-truth.test.ts` (version drift), `src/launch-readiness-task3.test.ts` | see TESTS.md | Not shown | `npm view gitpin version` returned 0.6.4, equal to main. `npx -y gitpin@0.6.4 gate` smoke passed (TESTS.md) | Production-Verified | | ||
|
|
||
| ## Integrations | ||
|
|
||
| | Integration | Direction | Verification level | Evidence | | ||
| | --- | --- | --- | --- | | ||
| | npm (`gitpin`) | Publish; Action installs from it | Production-Verified: 0.6.4 installed with `npx` and run against this repository's history | `npm view gitpin version` | | ||
| | MCP Registry | Publish; clients discover | Listing checked: 0.6.4 latest, active. Client install via registry not checked | Registry v0.1 search for `io.github.shmindmaster/gitpin` | | ||
| | GitHub Pages | Deploy site | Production-Verified: deployed from `90598a1` (run 38737568718), HTTP 200, live content checked | `pages.yml` run plus a `curl` of the footer | | ||
| | GitHub Action | Consumer CI | Integrated: the repo's own evidence gate passed on PRs #111 and #113 | `action.yml`, `.github/workflows/evidence-gate.yml` | | ||
| | MCP clients (Claude Code, Codex, Cursor and others) | stdio and HTTP | Config templates validated by `pnpm verify:clients` (not run here). No live client session checked | `docs/clients.md`, `scripts/validate-client-configs.mjs` | | ||
| | GitHub rulesets and auto-merge | Required check, merge | Workflow exists; ruleset settings not checked | `.github/workflows/auto-merge-owner-prs.yml` | | ||
| | PostHog (site analytics) | Site to PostHog | Opt-out logic exists with a privacy spec; live capture not checked | `site/analytics.js`, `tests/browser/analytics-privacy.spec.mjs` | | ||
|
|
||
| ## Lists | ||
|
|
||
| **UI without backend:** none found. The site is static and the product has no interactive UI. | ||
|
|
||
| **Backend without UI:** all of it by design (CLI, Action, MCP). Two items have no demo: the HTTP transport and the MCP tools (the films cover the gate only). | ||
|
|
||
| **Untested integrations (no executed test or live check in this pack):** | ||
| - GitHub Action as run on GitHub's hosted runners | ||
| - A real MCP client session over stdio or HTTP | ||
| - MCP Registry install path from a client | ||
| - A deployed HTTP instance | ||
| - Fresh `npx gitpin` install from npm | ||
|
|
||
| **Works locally only (as far as checked):** | ||
| - `pnpm demo:gate` (synthetic repo in a scratch directory) | ||
| - HTTP MCP server (tests use a local server) | ||
| - Local snapshot index (`pnpm index:build`) | ||
| - The gate against a real repository other than GitPin's own: not checked | ||
|
|
||
| **Pending, not evaluated:** PR #101 (faster git fixture tests, gate text output) and PR #102 (HTTP loopback by default, validate Origin). Both could change rows above once merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Demos | ||
|
|
||
| Every film below is a **real recording**. The real `gitpin` CLI runs in a real shell against fixture GP1, a synthetic git repository built by `scripts/build-pr-gate-fail-to-pass-artifact.mjs --persist --verify` (base `43439f8`, failing head `f0137d9`, passing head `57bce1a`, author `demo@gitpin.invalid`). Capture is headless Playwright (WebM) finished with Recast. There is no mock UI and no substitute footage. "Rosa" (release lead) is a fictional persona and is labelled as one on screen. | ||
|
|
||
| Videos are kept outside the repository, in the owner's media library. SHA-256 hashes identify each exact file. | ||
|
|
||
| ## Delivered films | ||
|
|
||
| | Film | Length | Recorded at | What it shows | QA on the exact file | SHA-256 | | ||
| |---|---|---|---|---|---| | ||
| | `Videos\Portfolio\GitPin\01 - The Gate That Wants Evidence.mp4` (+ `.srt`) | 1:35 | captured on the `03c1e69` build; every filmed command re-run byte-identical at main `e4c550e` | An agent PR changes `docs/protocol.md` and its manifest has `"claims": []`. `gitpin gate` returns `uncovered-change` and exit 1. One claim is added (line 5 plus `contentSha256`), and the same gate returns `evidence-verified` and exit 0. `git show … \| sha256sum` reproduces the hash. Shows the gate's coverage code in `src/gate.ts`, the CI signal (exit 1 fails, exit 0 clears), and the stated limit ("committed content, not semantic correctness"). | -16.0 LUFS; WER 0.0 (ElevenLabs scribe_v2 vs script); validate-encoded-picture PASS; story-experience review 90/100 | `d046bd6217fe7800c8d3b65ccc59caa8d0ab6c3ef417ce9870f3ffaa7f980012` | | ||
| | `Videos\Portfolio\GitPin\02 - The Gate in 30 Seconds.mp4` (+ `.srt`) | 0:32 | 1x frame-accurate cut of film 01; main still `e4c550e` on 2026-10-14 | Social cut with burned-in captions: zero claims → exit 1; one pinned claim → exit 0; plain-git recheck; the stated limit. | -16.0 LUFS; WER 0.0; no frozen picture ≥ 4 s; story-experience review 86/100 | `aff4b0f14fe89f226890f62e35e247aed53fea788d3fbb572a7209ee7fac8e7b` | | ||
| | `Videos\Evergreen\Proof Clips\02 - GitPin Evidence Gate.mp4` (+ `.srt`) | 0:13 | cut from film 01 | Fail, add claim, pass, plain-git recheck. Modular clip. | WER 0.0 | `2fd69c991c8897114fddfff604138965d3fd119f1dca883316889629acf82b75` | | ||
|
|
||
| Narration: | ||
| - Film 01 uses ElevenLabs Eric. | ||
| - Film 02 and the proof clip use ElevenLabs Bella. | ||
| - Both run on `eleven_v4`, third person. | ||
|
|
||
| Formats: | ||
| - Film 01 is format **B**, the functional walkthrough. | ||
| - Film 02 is a social teaser cut from it. | ||
| - Format **C**, the architecture and integration walkthrough, has not been made yet. | ||
|
|
||
| ## Traces behind the films | ||
| These are in the production workspace `D:\Local-AI\data\artifacts\media-studio\gitpin-hero-reel-20261014\`: | ||
| - `evidence/runtime-terminal-log.jsonl`: every command typed on camera, with output and exit code. | ||
| - `receipts/rerun-at-e4c550e.json`: the nine filmed commands re-run at `e4c550e` with byte-identical output. | ||
| - `receipts/gp1-verify-e4c550e.json`: the fixture verified at `e4c550e`. | ||
| - `receipts/claims-trace.json`: each spoken claim mapped to file:line. | ||
| - `qa/QA-RECORD.md`: the full gate table. | ||
|
|
||
| The social cut's EDL, which records what each segment proves, is in `D:\Local-AI\data\artifacts\media-studio\gitpin-social30-20261014\story\edl.json`. | ||
|
|
||
| ## In-repo demo assets | ||
| - [`docs/demos/pr-gate-fail-to-pass.md`](../demos/pr-gate-fail-to-pass.md), with its `.artifact.json` and `.svg`. | ||
| - `pnpm demo:gate`, a live fail-to-pass run on a scratch synthetic repo (`scripts/demo-gate-hero.mjs`, covered by `src/gate-hero.test.ts`). | ||
|
|
||
| ## Not demonstrated | ||
| - An MCP client (Claude Code or another) calling `pin.search → pin.prove → pin.verify`. | ||
| - The GitHub Action failing a real agent PR in the Checks UI. | ||
| - The HTTP transport with a bearer token. | ||
|
|
||
| ## Production plan: the next two assets | ||
|
|
||
| Both reuse the GP1 footage. Heavy work (capture, TTS, render) waits for an assigned slot. | ||
|
|
||
| **C. Technical deep dive, "Inside the gate's trust boundary" (about 6 min, developer education).** Planned, not made. | ||
| - **Hook (0:00–0:15).** A PR tries to weaken its own gate by editing `.gitpin/gate.yml` in head. The gate reads policy from base, reports `policy-change-blocked`, and exits 1. | ||
| - **Axes (0:15–0:45).** Base versus head trust. Coverage from the merge-base diff, not from the PR description. A hash-pinned line versus a semantic check. | ||
| - **Behaviour, then diagram (0:45–3:15).** Reuse the GP1 fail and pass takes. Walk the 17-hop trace in [TECHNICAL.md](TECHNICAL.md) on the code stage: `gate.ts` policy block (55–67), line-slice hash (108–115), coverage (169–173). Show the Mermaid diagram after the behaviour. | ||
| - **MCP (3:15–4:15).** A real MCP client over stdio calling `pin.search` → `pin.prove` → `pin.verify`, ending with the evidence pack's full SHA and hash. | ||
| - **Trade-off (4:15–4:45).** "Committed content, not semantic correctness." The claim text is not judged, except through optional `mustContain`. | ||
| - **Failure and recovery (4:45–5:30).** Edit the cited line after proving: the gate reports `content-hash-mismatch`. Re-prove, and it passes. | ||
| - **Payoff (5:30–6:00).** The Action YAML and `npx -y gitpin@0.6.4`. One call to action. | ||
| - **New captures needed:** the policy-change block, the hash mismatch and the MCP session. | ||
| - **Voice:** the local Qwen3-TTS narrator (no ElevenLabs spend), with WER checked by local ASR. | ||
|
|
||
| **Vertical short (≤ 45 s, 1080x1920).** Planned, not made. | ||
| - Reframed from film 02's beats: zero claims (exit 1), one pinned claim (exit 0), the plain-git recheck and the limit. | ||
| - Cropped from the GP1 Recast takes, not by letterboxing the 16:9 master. | ||
| - Burned-in captions. It reuses film 02's Bella lines, so there is no new TTS. | ||
|
|
||
| **ElevenLabs usage to date:** | ||
| - Film 02 used 7 `eleven_v4` lines, 411 characters. The API's `character-cost` headers total 180. | ||
| - Its music bed was reused from film 01. | ||
| - Film 01's usage is recorded in its own job workspace. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use either
pin.search_docsorpin.search_codein the planned MCP recording. The server's 12-tool contract has nopin.searchmethod, as the inventory inFUNCTIONAL.md:33also confirms, so the scriptedpin.search → pin.prove → pin.verifysession here—and the same sequence at line 40—cannot be executed by a real MCP client.Useful? React with 👍 / 👎.