diff --git a/.claude/skills/shadcn-audit/SKILL.md b/.claude/skills/shadcn-audit/SKILL.md index 49fd20833..214a6a948 100644 --- a/.claude/skills/shadcn-audit/SKILL.md +++ b/.claude/skills/shadcn-audit/SKILL.md @@ -42,7 +42,12 @@ an error as an empty result. Talk to the user in their language; - `gh` authenticated (`gh auth status`) with read access to `ruby-ui/ruby_ui`; posting the comment needs write access. - Node and pnpm (the shadcn sandbox and the docs assets). -- The chrome-devtools MCP server, for every browser step. +- Playwright for `collect`: `cd audit && pnpm install` in every checkout or + worktree (it installs `audit/node_modules`), then `pnpm exec playwright + install chromium` (the browser download is shared by every checkout on the + machine). +- The chrome-devtools MCP server, for the browser steps `collect` doesn't + cover (the kind-specific behavior sequences, animation, exploration). - Optional: Herdr. When `HERDR_ENV=1` the servers run in Herdr panes; otherwise as background shell processes (step 2.6). @@ -168,24 +173,62 @@ For each folder in `components`: list is ⏸ "map gap". - `divergences ` → registered differences (`dimension`, `rubyui`, `shadcn`, `reason`). A matching difference is ✅ basis `intentional`. + Entries with a structured `match` are applied by `collect` on its own; + prose-only entries you apply by hand. +- `collect --run --preview ` (add + `--shadcn-url`/`--rubyui-url` with the `url_template`s when step 2.6 picked + other ports). It drives both fixtures in headless Chromium at 1280×800, + light and dark, through the states the `kind` has (closed, open, focus, + after-esc; `form_control`: closed, focus; `static`: closed), and grades the + objective lines: computed styles per part (Look ❌ / ✅ grouped), the + accessibility snapshot plus `aria-expanded`/`aria-controls`/`aria-haspopup`/ + `aria-describedby` per part (ARIA ⚠️ / ✅ grouped), structured divergences + (✅ intentional), differences explained by a theme token or an environment + diff (⏸), unreachable states and selector count mismatches (⏸). It writes + `/collect.json` and `/screens/`, with a pixel diff per state as + evidence only, and only for states reached on both sides. Append its `lines` and `screenshots` to the component and + copy its `environment_diffs` into the report. Running it twice on the same + SHAs gives the same `collect.json`. + - `{"error": "… has no selectors …"}` → the component has no `selectors` + in `component_map.yml`: tell the user to add them (RubyUI part → CSS + selector in the rendered fixture; `null` = shadcn renders no element for + that part). Until they exist, collect that component by hand with the + chrome-devtools MCP: for each mapped part, the shadcn classes from + `[data-slot=]` against `effective`, the RubyUI element found from + its Ruby source at `/gem/lib/ruby_ui//` (tag, + Stimulus target, class string), look-alikes settled with + `getComputedStyle`; then the whole set for its `kind` in + `references/interactions.md`, including the Setup, the states and the + screenshots. + - `{"error": "playwright not found …"}` / `"chromium failed to launch"` → + install as in Prerequisites and re-run. ### 4. Look -Light and dark. For each mapped part: shadcn classes from the DOM -(`[data-slot=]`) checked against `effective`; RubyUI classes from the -preview DOM via `evaluate_script`. RubyUI parts have no `data-slot`: find the -element from the part's Ruby source at `/gem/lib/ruby_ui//` -(tag, Stimulus target, class string). List the tokens missing and extra on each -side; settle look-alikes with computed styles. Variant props (size, side) are +`collect` already graded Look from computed styles. Review its lines instead +of re-collecting: explain each ❌ with the class tokens behind it (shadcn +`effective` from `reference` vs the RubyUI class string) in the `note`, and +settle the ⏸ it left (count mismatches, theme/environment explanations, +reference `warnings`) when the DOM lets you. Variant props (size, side) are compared against `variants`. +**Overrides.** Change a `collect` grade only with a verified reason that the +rules in `references/grading.md` allow, written in that line's `note` (e.g. +`text-align: left` vs `start` in LTR renders the same → ✅ with the note). A ✅ +with a basis still needs its usual route: a registered divergence, a native +replacement or the spec. Never drop a `collect` line. + Fixture-originated wrappers aren't findings (`references/grading.md`): Sheet body (shadcn `div.px-4` vs `SheetMiddle`), Dialog body (bare `

` vs `DialogMiddle`). ### 5. HTML, ARIA, Behavior, Animation -Run the set for the component's `kind` from `references/interactions.md` on +`collect` already covered the setup, the environment probe, the states +closed/open/focus/after-esc, the ARIA snapshot and attributes per part, and +the screenshots. Run the rest of the `kind`'s set from +`references/interactions.md` (focus trap order, Esc and focus return, outside +click, scroll lock, arrow keys, typeahead, Enter/Space, animation) on both URLs (`url_template`s; `{folder}` = folder, `{light|dark}` = theme), in the same viewport on both sides (aim for 1280×800; record the actual size from the environment probe), with chrome-devtools MCP. Open by clicking `[data-audit-trigger]`. diff --git a/.claude/skills/shadcn-audit/references/grading.md b/.claude/skills/shadcn-audit/references/grading.md index 9002c1300..1b69047ec 100644 --- a/.claude/skills/shadcn-audit/references/grading.md +++ b/.claude/skills/shadcn-audit/references/grading.md @@ -21,7 +21,7 @@ Behavior or Animation. When unsure, use ⏸ and say what's missing. | `basis` | Shows as | Rule | | --- | --- | --- | -| `intentional` | ✅ … — intentional | The difference matches an entry from `divergences ` (same `dimension`). `source` = `audit/divergences.yml`, `note` = its `reason`. | +| `intentional` | ✅ … — intentional | The difference matches an entry from `divergences ` (same `dimension`). `source` = `audit/divergences.yml`, `note` = its `reason`. Entries with a structured `match` are applied by `collect` automatically; prose-only entries by hand. | | `native` | ✅ … — native | RubyUI swaps shadcn's JS for a native feature **and keeps** what shadcn delivers: focus in and back, Esc, dismiss, inertness of the page behind. Labelling is markup, graded on the ARIA line, not here. Tag differences go in `note`. No registration needed. | | `spec` | ✅ … — follows spec | shadcn breaks a spec (stage 2) and RubyUI follows it. `source` = the spec URL#anchor. No registration needed. | @@ -52,6 +52,16 @@ still graded on its own and is still a finding. `native-html.md` does → ❌ "candidate for ", citing MDN. Native already used and nothing lost → ✅ basis `native`. +## Lines from `collect` + +`collect` grades Look (computed styles: ❌ / ✅) and ARIA (snapshot + attributes: +⚠️ / ✅) per part, by the same rules as below. Its ⏸ lines say why (state not +reached, selector count mismatch, theme token, environment diff). Keep every +`collect` line. Change a grade only with a verified reason that the rules +below allow (e.g. values that render the same), written in its `note`; a ✅ +with a basis still needs its usual route (registered divergence, native, spec). +Otherwise keep the grade. + ## Per-dimension notes - **Look**: compare per part, light and dark. Class tokens that differ but diff --git a/.claude/skills/shadcn-audit/references/interactions.md b/.claude/skills/shadcn-audit/references/interactions.md index a7758a1e0..9dd05f37c 100644 --- a/.claude/skills/shadcn-audit/references/interactions.md +++ b/.claude/skills/shadcn-audit/references/interactions.md @@ -6,6 +6,13 @@ from the `.tsx` or the Ruby source. Run the same set on both URLs: - shadcn: `http://localhost:5179/?c=&theme=` (sandbox `url_template`) - RubyUI: `http://localhost:3179/__audit/?theme=` (preview `url_template`) +`collect` (SKILL.md step 3) already runs the setup below, the environment +probe, the states closed/open/focus/after-esc, the computed-style and ARIA +probes per part, and the screenshots, deterministically. Use this file for +what it doesn't cover: the per-`kind` sequences (focus trap order, Esc and +focus return, outside click, scroll lock, arrows, typeahead, Enter/Space), +animations, and any state `collect` reported as not reached. + ## Setup (once per page) 1. `new_page` (one per side), then `resize_page` both to the **same viewport**, diff --git a/audit/.gitignore b/audit/.gitignore index 32fe75168..e73b382b8 100644 --- a/audit/.gitignore +++ b/audit/.gitignore @@ -1 +1,2 @@ /.bundle/ +/node_modules/ diff --git a/audit/Gemfile b/audit/Gemfile index 08384a002..0f3ccf275 100644 --- a/audit/Gemfile +++ b/audit/Gemfile @@ -3,3 +3,5 @@ source "https://rubygems.org" gem "minitest" gem "rake" gem "standard" +gem "playwright-ruby-client", "1.64.0" +gem "chunky_png", "1.4.0" diff --git a/audit/Gemfile.lock b/audit/Gemfile.lock index ded1d6763..02a480e49 100644 --- a/audit/Gemfile.lock +++ b/audit/Gemfile.lock @@ -2,10 +2,18 @@ GEM remote: https://rubygems.org/ specs: ast (2.4.3) + base64 (0.3.0) + chunky_png (1.4.0) + concurrent-ruby (1.3.8) drb (2.2.3) json (3.1.0) language_server-protocol (3.17.0.6) lint_roller (1.1.0) + logger (1.7.0) + mime-types (3.7.0) + logger + mime-types-data (~> 3.2025, >= 3.2025.0507) + mime-types-data (3.2026.1027) minitest (6.0.6) drb (~> 2.0) prism (~> 1.5) @@ -13,6 +21,10 @@ GEM parser (3.3.13.0) ast (~> 2.4.1) racc + playwright-ruby-client (1.64.0) + base64 + concurrent-ruby (>= 1.1.6) + mime-types (>= 3.0) prism (1.9.0) racc (1.8.1) rainbow (3.1.1) @@ -58,7 +70,9 @@ PLATFORMS ruby DEPENDENCIES + chunky_png (= 1.4.0) minitest + playwright-ruby-client (= 1.64.0) rake standard diff --git a/audit/README.md b/audit/README.md index 24429ddc2..ca5d399ac 100644 --- a/audit/README.md +++ b/audit/README.md @@ -10,11 +10,13 @@ mise exec -- bundle exec exe/shadcn-audit workspace [--sha SHA] `workspace` keeps a sparse clone of `shadcn-ui/ui` in `.shadcn-audit/ui` (git-ignored) and checks out `main`, or the given SHA. +`collect --run

[--preview ] [--shadcn-url T] [--rubyui-url T]` drives both fixtures in headless Chromium (playwright-ruby-client) with the sandbox and preview servers running, and writes `/collect.json`: computed styles, accessibility snapshots, screenshots and pixel diffs per state, plus the objectively graded Look/ARIA lines. It needs `pnpm install && pnpm exec playwright install chromium` here once. Part selectors come from `selectors` in `component_map.yml`; a `match` key in `divergences.yml` lets `collect` apply that divergence on its own. + Tests use `ShadcnAudit::FakeShell` (`test/test_helper.rb`): stub canned stdout by argv or matcher, then assert on recorded calls. ## Prerequisites -Ruby 3.4 (via mise, as below, or an equivalent version manager; without mise, drop `mise exec --`), `gh` authenticated (`gh auth status`), Node and pnpm, and, for the browser steps, the chrome-devtools MCP. The audit runs its two dev servers in Herdr panes when `HERDR_ENV=1`, otherwise as background shell processes; either way it stops them at the end. `preview` boot commands use `mise exec --` only when `mise` is on PATH. +Ruby 3.4 (via mise, as below, or an equivalent version manager; without mise, drop `mise exec --`), `gh` authenticated (`gh auth status`), Node and pnpm, Playwright's Chromium for `collect` (above), and, for the remaining browser steps, the chrome-devtools MCP. The audit runs its two dev servers in Herdr panes when `HERDR_ENV=1`, otherwise as background shell processes; either way it stops them at the end. `preview` boot commands use `mise exec --` only when `mise` is on PATH. A component added to `component_map.yml` also needs fixtures in `fixtures//` (`content.yml`, `shadcn.tsx`, `rubyui.rb`). diff --git a/audit/component_map.yml b/audit/component_map.yml index 643b3d811..c96a195ce 100644 --- a/audit/component_map.yml +++ b/audit/component_map.yml @@ -2,6 +2,8 @@ # parts: RubyUI class => shadcn data-slot (null = no shadcn part). # props: RubyUI class => {shadcn prop => RubyUI kwarg (null = none)}; shadcn props only. # rubyui_only: RubyUI class => [kwargs with no shadcn counterpart]. className is never listed (all parts take class via attrs). +# selectors: RubyUI class => CSS selector of that part in the rendered fixture (fixed, so collect is deterministic); +# null = not compared (shadcn renders no element for it, e.g. a Base UI root). # fixture_shadcn: other shadcn components the fixture renders (added to the sandbox, not audited here). dialog: shadcn: dialog @@ -16,6 +18,14 @@ dialog: DialogTitle: dialog-title DialogDescription: dialog-description DialogMiddle: null + selectors: + Dialog: null + DialogTrigger: '[data-ruby-ui--dialog-target="trigger"] > button' + DialogContent: 'dialog[data-ruby-ui--dialog-target="dialog"]' + DialogHeader: 'dialog[data-ruby-ui--dialog-target="dialog"] > div:first-child' + DialogFooter: 'dialog[data-ruby-ui--dialog-target="dialog"] > div:last-of-type' + DialogTitle: '[data-ruby-ui--dialog-target="title"]' + DialogDescription: '[data-ruby-ui--dialog-target="description"]' props: Dialog: {open: open} DialogContent: {showCloseButton: show_close_button} @@ -37,6 +47,16 @@ alert_dialog: AlertDialogDescription: alert-dialog-description AlertDialogAction: alert-dialog-action AlertDialogCancel: alert-dialog-cancel + selectors: + AlertDialog: null + AlertDialogTrigger: '[data-controller="ruby-ui--alert-dialog"] > [data-action] > button' + AlertDialogContent: '[role="alertdialog"]' + AlertDialogHeader: '[role="alertdialog"] > div:first-child' + AlertDialogFooter: '[role="alertdialog"] > div:last-child' + AlertDialogTitle: '[role="alertdialog"] > div:first-child > h2' + AlertDialogDescription: '[role="alertdialog"] > div:first-child > p' + AlertDialogAction: '[role="alertdialog"] > div:last-child > button:last-child' + AlertDialogCancel: '[role="alertdialog"] > div:last-child > button:first-child' props: AlertDialogContent: {size: null} AlertDialogCancel: {variant: variant, size: size} @@ -54,6 +74,14 @@ sheet: SheetTitle: sheet-title SheetDescription: sheet-description SheetMiddle: null + selectors: + Sheet: null + SheetTrigger: '[data-controller="ruby-ui--sheet"] > [data-action] > button' + SheetContent: '[data-ruby-ui--sheet-content-target="panel"]' + SheetHeader: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child' + SheetFooter: '[data-ruby-ui--sheet-content-target="panel"] > div:last-of-type' + SheetTitle: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child > h3' + SheetDescription: '[data-ruby-ui--sheet-content-target="panel"] > div:first-child > p' props: SheetContent: {side: side, showCloseButton: null} @@ -71,6 +99,15 @@ command: CommandDialog: dialog CommandDialogTrigger: null CommandDialogContent: dialog-content + selectors: + Command: '[data-ruby-ui--command-target="panel"] > div' + CommandInput: 'input[data-ruby-ui--command-target="input"]' + CommandList: '[data-ruby-ui--command-target="panel"] > div > div:has([data-ruby-ui--command-target="group"])' + CommandEmpty: '[data-ruby-ui--command-target="empty"]' + CommandGroup: '[data-ruby-ui--command-target="group"]' + CommandItem: '[data-ruby-ui--command-target="item"]' + CommandDialog: null + CommandDialogContent: '[data-ruby-ui--command-target="panel"]' props: CommandInput: {placeholder: placeholder} CommandGroup: {heading: title} @@ -97,6 +134,17 @@ form: FormFieldContent: field-content FormFieldTitle: field-label FormFieldSeparator: field-separator + selectors: + FormField: '[data-slot="field"]' + FormFieldLabel: '[data-slot="field-label"]' + FormFieldHint: '[data-slot="field-description"]' + FormFieldError: '[data-slot="field-error"]' + FormFieldSet: '[data-slot="field-set"]' + FormFieldLegend: '[data-slot="field-legend"]' + FormFieldGroup: '[data-slot="field-group"]' + FormFieldContent: '[data-slot="field-content"]' + FormFieldTitle: null + FormFieldSeparator: '[data-slot="field-separator"]' props: FormField: {orientation: orientation} FormFieldLegend: {variant: variant} diff --git a/audit/divergences.yml b/audit/divergences.yml index f51763a73..f9790bb46 100644 --- a/audit/divergences.yml +++ b/audit/divergences.yml @@ -1,5 +1,7 @@ # Intentional RubyUI-vs-shadcn differences. A divergence listed here makes the # matching finding a `pass` with basis `intentional`. +# Optional `match: {part, property (Look) | attribute (ARIA, or `snapshot`), rubyui, shadcn}` lets `collect` apply it +# automatically; entries without it are applied by the agent. - component: dialog dimension: HTML rubyui: " + showModal()" diff --git a/audit/docs/2026-11-01-collect-design.md b/audit/docs/2026-11-01-collect-design.md new file mode 100644 index 000000000..d9435af40 --- /dev/null +++ b/audit/docs/2026-11-01-collect-design.md @@ -0,0 +1,201 @@ +# shadcn-audit `collect`: deterministic browser collection + +Status: design, approved for planning once reviewed. +Date: 2026-11-01. + +## Intent + +The shadcn-audit skill compares a RubyUI component with its shadcn +(Base UI + vega + neutral) counterpart. Today the agent does every browser +step by hand through the chrome-devtools MCP: opens both fixtures, clicks the +trigger, reads classes and ARIA, takes screenshots, and grades each line. + +Nothing is broken. The goal is **determinism**: running the audit twice on the +same commit should produce the same collected facts and the same starting +grades, line by line. Today that depends on the agent collecting the same +things the same way every run (which selector it derives for a RubyUI part, +which properties it compares, which states it screenshots). + +## Goals + +1. A new CLI subcommand, `collect`, in `audit/` (same one-JSON-object + contract as every other subcommand, run from `audit/`), that drives a + browser through **playwright-ruby-client** and collects, for both sides: + - **Setup**: fixed 1280×800 viewport, light and dark, and the environment + probe from `interactions.md` (viewport, DPR, fonts, body colors, form + control defaults, dark class). Differences go into `environment_diffs`. + - **States** common to every `kind`: closed, open (click on + `[data-audit-trigger]`), keyboard focus, after Escape. A state that does + not apply to the kind, or can't be reached, is recorded as unreachable + with the reason (→ ⏸). + - **Per state and per mapped part**: computed styles, the accessibility + tree (Playwright `ariaSnapshot`), plus a screenshot per state and side. + - **Pixel diff** per state (same viewport, both sides). +2. `collect` grades the **objective** lines itself: + - computed style equal on both sides → `pass` (even when class tokens + differ: this settles look-alikes such as `ring-1` vs `border`); + - computed style differs on a visual property → `fail`, both values named; + - ARIA difference (role, accessible name, state, or one of the checked + attributes present on one side only) → `warn`, as `grading.md` rule 4 + says (ARIA is ⚠️, not ❌; see "Builder-check decisions"); + - difference registered in `divergences.yml` → `pass`, basis + `intentional`; + - state unreachable / trigger missing / page error → `inconclusive` with + the reason. + Every graded line carries a `source` the way `report.json` requires. +3. The skill (`.claude/skills/shadcn-audit/SKILL.md` and + `references/interactions.md`) is updated so the agent runs `collect` + first and builds `report.json` on top of its output. +4. The agent keeps: Spec, Native, Animation, the kind-specific behavior + sequences (focus-trap order, scroll lock, outside click, arrow keys, + typeahead, Enter/Space toggling), fixture-originated wrappers, and + exploration of anything `collect` didn't cover. The agent **may override** + a `collect` grade, but only with a written reason that ends up in the + report line's `note`. + +## Non-goals + +- Running `collect` in CI. Local only for now (the user's call). The `audit` + CI job keeps running unit tests only. +- Kind-specific behavior sequences in v1. They move into `collect` later, one + kind at a time. +- Pixel diff as a grade. It is **evidence** for the agent (number of + differing pixels / ratio, plus a diff image), never a status on its own. + Font and antialiasing noise make it unreliable as an automatic verdict. +- Booting the servers. Sandbox (5179) and preview (3179) keep being started + as today (Herdr panes or background processes); `collect` receives the URLs. +- Changing `report.json`'s schema in a breaking way, or the renderers. +- Cross-browser runs (Firefox/WebKit). Chromium only, even though Playwright + would allow more later. + +## Anti-goals + +- `collect` producing grades that need judgment (spec conformance, "is this + JS replaceable by native HTML", "is this a fixture wrapper"). A rule-based + guess there is a false positive generator. +- Output that differs between two runs on the same SHAs for reasons other + than timestamps, absolute paths, or pixel noise under the tolerance. +- Running anything from `.shadcn-audit/ui` (still untrusted), or booting fork + code without the existing cross-repository confirmation step. + +## Constraints and decisions already made + +- **Library: playwright-ruby-client** (user's call). It needs the Node + `playwright` package and a Chromium installed locally; the README and the + skill's prerequisites must say how (the audit already requires Node and + pnpm). +- **Local only** (user's call). +- **Grading split** as in Goals 2 and 4 (user agreed). +- **Pixel diff = evidence only** (proposed, user agreed). +- **v1 scope** = setup + common states + styles/ARIA/screenshots/pixel diff + for every kind; behavior sequences stay with the agent (user agreed). +- **RubyUI part selectors become fixed data.** RubyUI parts have no + `data-slot`; today the agent derives a selector from the Ruby source on + every run, which is a source of non-determinism. `component_map.yml` gains + a selector per RubyUI part, next to the existing class → `data-slot` + mapping. `ComponentMap` validation and `map` output include it. + `collect` refuses (⏸ for that part) when a mapped part has no selector. +- Existing conventions hold: Ruby 3.4 via mise, StandardRB, Minitest, one JSON + object per subcommand, `{"error": …}` + exit 1 on failure, errors listed + in `CLI::ERRORS`. + +## Divergences become machine-matchable (user's call) + +`divergences.yml` entries today are free text (`rubyui`, `shadcn`, +`reason`) keyed by component + dimension, and `grading.md` says an entry +covers only the difference its text describes. Reading that prose is +judgment, so: + +- Entries gain **optional structured match keys**: the part, the computed + property or ARIA attribute/role, and the value on each side (exact shape + left to the builder). `Divergences` validates them. +- `collect` marks a difference `pass` / basis `intentional` **only** when a + structured key matches it exactly. It never matches on component + + dimension alone. +- Entries without match keys stay valid; the agent applies them as overrides, + as today. +- The four existing entries are migrated to include match keys where the + difference is expressible that way. + +## Builder-check decisions (designer's calls, check these) + +- **ARIA grade and basis.** ARIA differences are `warn`, per `grading.md` + rule 4, so the old flow's verdicts are preserved. The comparison is on the + computed accessibility tree (`ariaSnapshot`: role, name, states), so a native + `` vs `div[role=dialog]` is not a difference. On top of that, an + attribute probe checks `aria-expanded`, `aria-controls` (resolves or + dangling), `aria-haspopup`, `aria-describedby` (resolved text) and a + computed `modal` fact (`:modal` or `aria-modal="true"`, so a native + `` counts). `aria-labelledby` isn't read raw: the accessible name it + produces is already in the snapshot. ID-valued attributes are compared by + what they resolve to, never by the raw ID. Snapshots compare exactly + (hierarchy, case, numbers); only Look values are normalized. +- **Line granularity follows `grading.md`.** One Look line per differing + part, with all matching parts grouped into a single `pass` line; same for + ARIA. State and theme where the difference shows up go into `note` + (e.g. "open, light+dark"). No per-property or per-state lines, so a re-run + can be compared line by line with the old report. +- **Theme and environment explain before they fail.** A differing computed + value that equals a `theme` diff pair (RubyUI value vs shadcn value of the + same token) is not graded as a component `fail`; it becomes `inconclusive` + with a note naming the token (theme diffs are already reported on their + own). Same for typography properties when `environment_diffs` has a font + difference. Only what's left is graded. +- **Selectors come from the audited head.** `collect` loads + `component_map.yml` from the `preview` checkout (the audited SHA), falling + back to the working tree only when the head has no selectors yet, and says + which one it used. The expected count is the shadcn side's: the RubyUI + selector must match as many visible elements as `[data-slot=]` does + in the same state (one, or N for repeated parts such as accordion items); + otherwise that part is `inconclusive` ("selector matched 0/3 visible + elements"). A part never visible on either side is `inconclusive` too. A PR that + changes a part's markup updates its selector in the same PR. + +## Shape of the output (guidance, not a contract) + +`collect` writes its artifacts into the run folder and prints one JSON +object, roughly: + +- `viewport`, `environment` (per side) and `environment_diffs`; +- `states`: for each state × theme, per side: reached or not (+ reason), + screenshot path (relative to the run folder), per-part computed styles, + `ariaSnapshot` text; plus the pixel diff (ratio + diff image path); +- `lines`: the objectively graded lines, already in `report.json` line shape + (`dimension`, `status`, `basis`, `rubyui`, `shadcn`, `source`, `note`), so + the agent can append them to the component's `lines`; +- `screenshots`: entries already in `report.json`'s screenshot shape. + +The agent then adds its own lines and overrides, and writes `report.json` as +today; `render` and `comment` are unchanged. + +## Success criteria + +- Two `collect` runs on the same `head_sha`/`shadcn_sha`, with servers + booted the same way, produce identical JSON after removing timestamps and + absolute paths, and identical `lines`. +- Re-running the audit of an already-mapped component (e.g. Dialog) with the + new flow yields every Look/ARIA line the old flow had, with the same + verdicts or a documented reason for the change. +- Unit tests cover grading rules and map validation without a browser; the + browser-driving code is exercised by a local smoke test that is skipped + when Playwright/Chromium is not installed. + +## Left to the builder + +- Subcommand arguments and flags (how URLs, run folder, folder/component and + states are passed), and whether light/dark runs in one call. +- Which computed-style properties count as "visual" and how they are + normalized (color formats, sub-pixel rounding). +- How "keyboard focus" is reached generically (e.g. Tab to the trigger) and + how a state is declared not applicable per kind. +- Pixel diff implementation and the noise tolerance used to report it. +- The exact YAML shape of the RubyUI selectors in `component_map.yml`, and + filling them in for the five mapped components. +- How Playwright is installed/located and how its absence is reported. +- How `source` is filled for `collect` lines (e.g. `style-vega.css:` + from `reference` effective classes, or the fixture URL + state). + +## Not in this project + +- The accordion audit itself (no map entry or fixture yet). It runs with the + current flow or after this lands, separately. diff --git a/audit/docs/2026-11-01-collect-plan.md b/audit/docs/2026-11-01-collect-plan.md new file mode 100644 index 000000000..c15f85a89 --- /dev/null +++ b/audit/docs/2026-11-01-collect-plan.md @@ -0,0 +1,251 @@ +# shadcn-audit `collect` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A `collect` subcommand that drives both fixtures in Chromium through playwright-ruby-client, collects computed styles, accessibility facts, screenshots and pixel diffs per state, and grades the objective Look/ARIA lines deterministically. + +**Architecture:** A browser-free core (`Collect::Normalize`, `Collect::Grader`, `Collect::PixelDiff`) fed by a thin Playwright layer (`Collect::Driver`) and glued by `Collect::Runner`. The core is unit-tested with plain hashes; the driver gets a smoke test against local HTML files that skips when Chromium isn't installed. `ComponentMap` gains `selectors`, `Divergences` gains `match`. + +**Tech Stack:** Ruby 3.4 (mise), Minitest, StandardRB, playwright-ruby-client 1.64.0 + Node `playwright` 1.64.1 (pnpm, in `audit/`), chunky_png 1.4.0. + +**Spec:** `audit/docs/2026-11-01-collect-design.md` + +## Global Constraints + +- Every subcommand prints exactly one JSON object; failures are `{"error": …}` + exit 1 via `CLI::ERRORS`. +- Run from `audit/`: `mise exec -- bundle exec …`. `bundle exec rake` (tests + standardrb) must pass after every task. +- Viewport fixed at **1280×800**, DPR 1, Chromium headless only. +- Pixel diff is evidence only: it never sets a line's `status`. +- ARIA differences grade `warn`; Look differences `fail`; registered (structured `match`) → `pass` basis `intentional`; theme/environment-explained or unreachable → `inconclusive`. Nothing unverified is `pass`. +- Lines use `report.json` line shape: `dimension, status, basis, rubyui, shadcn, source, note`. Every `pass`/`warn`/`fail` has a non-empty `source`. +- Line granularity: one line per part per outcome; all fully matching parts grouped into one `pass` line per dimension. State/theme go in `note`. +- Screenshot names: `screens/---.png`; diff images `screens/--diff-.png`. Paths in output are relative to the run folder. +- Local only: CI keeps running unit tests; nothing browser-bound runs there. +- Never run anything from `.shadcn-audit/ui`. + +## Review Focus + +1. **Repeated parts with mismatched counts** (accordion items: 3 on shadcn, 2 visible on RubyUI): the part is `inconclusive` "selector matched 2/3 visible elements in open/light", never compared index-by-index. → Task 3 test. +2. **Hidden-but-present elements** (RubyUI keeps closed content in the DOM, Base UI unmounts it): only visible elements (`checkVisibility()`) are counted and collected. → Task 6 smoke test. +3. **Missing trigger, page error, navigation timeout**: the state is `reached: false` with the reason and the run continues; lines become `inconclusive`. → Task 6 smoke test + Task 7 runner test. +4. **Playwright/Chromium not installed**: `collect` returns `{"error": "…run `pnpm install && pnpm exec playwright install chromium` in audit/"}`, not a backtrace. → Task 7 CLI test. +5. **Order stability**: the same observations in a different hash/array order produce byte-identical `lines`. → Task 3 test. + +--- + +### Task 1: Dependencies and `selectors` in the component map + +**Files:** +- Modify: `audit/Gemfile` (+ `Gemfile.lock`), `audit/.gitignore`, `audit/lib/shadcn_audit/component_map.rb` +- Create: `audit/package.json`, `audit/pnpm-lock.yaml` +- Test: `audit/test/component_map_test.rb` + +**Interfaces:** +- Produces: `entry(folder)["selectors"]` → `Hash{String part => String css}` (may be absent). `ComponentMap#selectors(folder) -> Hash` (empty hash when absent). + +- [ ] **Step 1: Write failing tests** in `component_map_test.rb`: + - `test_selectors_are_returned`: entry with `parts: {DialogContent: dialog-content}`, `selectors: {DialogContent: "dialog > div"}` → `map.selectors("dialog") == {"DialogContent" => "dialog > div"}`. + - `test_selectors_default_to_empty`: no `selectors` key → `{}`. + - `test_selector_for_unknown_part_is_rejected`: `selectors: {Nope: "div"}` → `ConfigError` matching `/selectors.Nope is not a mapped part/`. + - `test_blank_selector_is_rejected`: `selectors: {DialogContent: " "}` → `ConfigError` matching `/selectors.DialogContent must be a non-empty CSS selector/`. +- [ ] **Step 2: Run** `mise exec -- bundle exec rake test TEST=test/component_map_test.rb` → the four fail. +- [ ] **Step 3: Implement** validation in `ComponentMap#validate!` and `#selectors(folder)`. Update the header comment of `component_map.yml` with one line: `# selectors: RubyUI class => CSS selector of that part in the rendered fixture (fixed, so collect is deterministic).` +- [ ] **Step 4: Add dependencies**: Gemfile `gem "playwright-ruby-client", "1.64.0"` and `gem "chunky_png", "1.4.0"`; `bundle install`. `audit/package.json` `{"private": true, "devDependencies": {"playwright": "1.64.1"}}`; `pnpm install`. `.gitignore` += `/node_modules/`. +- [ ] **Step 5: Run** `mise exec -- bundle exec rake` → all pass, standardrb clean. +- [ ] **Step 6: Commit** `[Feature] shadcn-audit: selectors in component map + playwright deps`. + +### Task 2: Structured `match` keys in divergences + +**Files:** +- Modify: `audit/lib/shadcn_audit/divergences.rb`, `audit/divergences.yml` (header comment only) +- Test: `audit/test/divergences_test.rb` + +**Interfaces:** +- Produces: `Divergences#match(component:, dimension:, part:, key:, rubyui:, shadcn:) -> Hash | nil` — returns the entry whose `match` equals `{part, property|attribute: key, rubyui, shadcn}` after `Collect::Normalize.value` on both values (Task 3 defines it; until then compare `strip` — Task 3 swaps it in). Never matches entries without `match`. + +Entry shape: +```yaml +- component: dialog + dimension: Look # Look → property; ARIA → attribute + rubyui: "border 1px" + shadcn: "ring 1px" + reason: "…" + since: "2026-11-01" + match: {part: DialogContent, property: border-top-width, rubyui: "1px", shadcn: "0px"} +``` +`attribute` may be `snapshot` to match the whole accessibility snapshot of the part. + +- [ ] **Step 1: Write failing tests**: + - `test_match_finds_structured_entry`: entry above → `match(component: "dialog", dimension: "Look", part: "DialogContent", key: "border-top-width", rubyui: "1px", shadcn: "0px")` returns it. + - `test_match_ignores_prose_only_entries`: same component/dimension without `match` → `nil`. + - `test_match_requires_exact_values`: `rubyui: "2px"` → `nil`. + - `test_match_validation`: `match` on dimension `HTML` → `ConfigError` `/match is only for Look or ARIA/`; Look with `attribute` → `/Look match needs property/`; missing `rubyui` → `/match needs part, property\|attribute, rubyui and shadcn/`. + - `test_existing_registry_still_loads`: `Divergences.load(ShadcnAudit.root.join("audit/divergences.yml"))` doesn't raise. +- [ ] **Step 2: Run** `rake test TEST=test/divergences_test.rb` → new tests fail. +- [ ] **Step 3: Implement** validation + `#match`. Header comment in `divergences.yml`: one line explaining `match`. The four existing entries are HTML/Behavior differences, not computed Look/ARIA values, so none gets a `match` (say so in the commit message). +- [ ] **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: machine-matchable divergences`. + +### Task 3: Normalize + Look grading + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/normalize.rb`, `audit/lib/shadcn_audit/collect/grader.rb` +- Modify: `audit/lib/shadcn_audit.rb` (requires), `divergences.rb` (use `Normalize.value`) +- Test: `audit/test/collect/normalize_test.rb`, `audit/test/collect/grader_test.rb` + +**Interfaces:** +- Consumes: `Divergences#match` (Task 2). +- Produces: + - `Collect::Normalize.value(String) -> String`: downcase, collapse whitespace, every decimal number rounded to 2 places with trailing zeros dropped (`"0.6459"→"0.65"`, `"1.50px"→"1.5px"`). + - `Collect::Grader.new(folder:, parts:, divergences:, theme_diffs:, environment_diff_keys:)` where `parts` is the ordered RubyUI part list from the map (`Array`), `theme_diffs` is `theme["diffs"]`, `environment_diff_keys` is e.g. `["font"]`. + - `Grader#lines(observations) -> Array` in report line shape. `observations`: `Array<{"state"=>, "theme"=>, "rubyui"=>SideFacts, "shadcn"=>SideFacts}>`. + - `SideFacts = {"reached"=>Bool, "reason"=>String|nil, "screenshot"=>String, "parts"=>{part=>[{"styles"=>{prop=>val}, "aria"=>String, "attrs"=>{name=>val|nil}}]}}`. + - Constants: `Grader::FONT_PROPS = %w[font-family font-size font-weight line-height letter-spacing width height]`; `Grader::LOOK_SOURCE = "computed styles (shadcn-audit collect)"`; `Grader::ARIA_SOURCE = "accessibility tree (shadcn-audit collect)"`. + +Look rules, per part, over every observation where both sides were reached: +1. Element counts differ → one `inconclusive` Look line for the part: note `"selector matched / visible elements in /"` (first mismatch, then `; …` for others). Skip property comparison for that observation. +2. Per element index and property, values equal after `Normalize.value` → no diff. +3. Diff explained by theme: some `theme_diffs` entry with same `mode` as the observation's theme and normalized `rubyui`/`shadcn` equal to the observed values → bucket `theme` (note names the tokens). +4. Diff explained by environment: `environment_diff_keys` includes `"font"` and property ∈ `FONT_PROPS` → bucket `environment`. +5. `divergences.match(dimension: "Look", part:, key: property, …)` → bucket `intentional`. +6. Otherwise → bucket `diff`. + +Output per part (in `parts` order), at most one line per non-empty bucket, in this order: `diff` → `fail` (`rubyui`/`shadcn` = `"prop: value"` joined by `"; "`, unique, sorted by property; note = the `state/theme` pairs where seen, joined `", "`); `intentional` → `pass` basis `intentional`, source `audit/divergences.yml`, note = reason; `theme`/`environment` → `inconclusive`, note `"explained by theme token(s) "` / `"explained by environment diff: font"`. Parts with no lines at all → collected into one `pass` Look line: `rubyui` = `", : computed styles match"`, `shadcn` = `"same"`, source `LOOK_SOURCE`. + +- [ ] **Step 1: Write failing tests** (`normalize_test.rb`): the two examples above, `" RGB(0, 0, 0) "→"rgb(0, 0, 0)"`, `"10px"` unchanged. +- [ ] **Step 2: Write failing tests** (`grader_test.rb`), each building observations by hand: + - `test_matching_parts_group_into_one_pass_line`: two parts identical → exactly one line `{dimension: "Look", status: "pass", rubyui: "DialogContent, DialogTitle: computed styles match"}`. + - `test_style_difference_is_fail_with_both_values`: `padding-top` `24px` vs `16px` in `open/light` and `open/dark` → one `fail` line, `rubyui: "padding-top: 24px"`, `shadcn: "padding-top: 16px"`, `note: "open/light, open/dark"`, source `LOOK_SOURCE`. + - `test_count_mismatch_is_inconclusive`: 2 vs 3 elements → `inconclusive`, note `"selector matched 2/3 visible elements in open/light"`, and no `fail` for that part. + - `test_theme_token_explains_difference`: `color` `oklch(0.646 0.222 41.116)` vs `oklch(0.809 0.105 251.813)` with theme diff `chart-1` light → `inconclusive`, note includes `chart-1`. + - `test_font_environment_explains_typography_only`: env keys `["font"]`; `font-size` differs → `inconclusive`; `padding-top` differs → `fail`. + - `test_structured_divergence_passes_as_intentional`: matching `match` entry → `pass` basis `intentional`, source `audit/divergences.yml`. + - `test_output_is_order_independent` (Review Focus 5): shuffle observations and the styles hash key order → `lines` equal. + - `test_unreached_observations_are_skipped_for_comparison`: rubyui `reached: false` → no Look lines from that observation (Task 4 adds the inconclusive line). +- [ ] **Step 3: Run** `rake test TEST=test/collect/grader_test.rb` → fail. +- [ ] **Step 4: Implement** `Normalize.value` and `Grader#lines` (Look only); switch `Divergences#match` to `Normalize.value`. +- [ ] **Step 5: Run** `rake` → pass. +- [ ] **Step 6: Commit** `[Feature] shadcn-audit: deterministic Look grading for collect`. + +### Task 4: ARIA grading and unreachable states + +**Files:** +- Modify: `audit/lib/shadcn_audit/collect/grader.rb` +- Test: `audit/test/collect/grader_test.rb` + +**Interfaces:** +- Produces: `Grader#lines` also emits ARIA lines and unreachable lines. `Grader::ARIA_ATTRS = %w[aria-expanded aria-controls aria-haspopup aria-describedby modal]` (`modal` added in review: computed `:modal` or `aria-modal="true"`; `aria-labelledby` is covered by the name in the snapshot; the driver fills `attrs` with exactly these keys; `aria-controls` is reported as `"resolves"`/`"dangling"`/nil, `aria-describedby` as the referenced text, `"dangling"` for a missing target, or nil). + +ARIA rules, per part, same count/skip logic as Look: +- `aria` (Playwright aria snapshot of the element) differs, compared exactly (hierarchy, case and numbers kept; `Normalize.value` is for Look only) → diff with key `snapshot`. +- Each `ARIA_ATTRS` value differs (incl. present vs nil) → diff with key = attribute. +- `divergences.match(dimension: "ARIA", part:, key:, …)` → `intentional`; theme/environment never apply to ARIA. +- Output like Look but `diff` bucket → `warn`; `rubyui`/`shadcn` = `": "`, snapshots trimmed to 200 chars; grouped `pass` line `": accessibility tree matches"`, source `ARIA_SOURCE`. + +Unreachable: for each observation where a side has `reached: false` → one `inconclusive` line per dimension (Look, ARIA), deduplicated by reason, note `"/ not reached on : "`, all observations sharing the same `state`+`side`+`reason` merged into one note. + +- [ ] **Step 1: Write failing tests**: + - `test_aria_snapshot_difference_is_warn`: snapshots `- button "Open" [expanded]` vs `- button "Open"` → `warn`, `rubyui` starts with `"snapshot: - button"`, source `ARIA_SOURCE`. + - `test_missing_attribute_is_warn`: `aria-controls` `"resolves"` vs nil → `warn` with `shadcn: "aria-controls: (none)"`. + - `test_matching_aria_groups_into_pass`. + - `test_aria_divergence_by_attribute_is_intentional`. + - `test_unreached_state_yields_inconclusive_look_and_aria`: rubyui `open/light` and `open/dark` unreached with reason `"no [data-audit-trigger]"` → exactly two lines (Look, ARIA), note `"open/light, open/dark not reached on rubyui: no [data-audit-trigger]"`. +- [ ] **Step 2: Run** → fail. **Step 3: Implement.** **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: ARIA and unreachable-state grading for collect`. + +### Task 5: Pixel diff + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/pixel_diff.rb` +- Test: `audit/test/collect/pixel_diff_test.rb` + +**Interfaces:** +- Produces: `Collect::PixelDiff.compare(rubyui_png:, shadcn_png:, diff_png:) -> {"pixels"=>Integer, "ratio"=>Float (6 decimals), "diff"=>String path}` or `{"error"=>"size 1280x800 vs 1280x720"}` when dimensions differ (no diff image written). `PixelDiff::CHANNEL_TOLERANCE = 16`: a pixel differs when any RGBA channel differs by more than 16. Diff image: differing pixels red (`#ff0000`), others the shadcn pixel at 25% alpha. + +- [ ] **Step 1: Write failing tests** generating PNGs with `ChunkyPNG::Image.new(4, 4, …)` in `Dir.mktmpdir`: identical → `pixels: 0, ratio: 0.0`; one pixel off by 10 → `0`; one pixel off by 40 → `pixels: 1, ratio: 0.0625`, diff file exists; 4×4 vs 4×3 → `{"error" => "size 4x4 vs 4x3"}`. +- [ ] **Step 2: Run** → fail. **Step 3: Implement.** **Step 4: Run** `rake` → pass. +- [ ] **Step 5: Commit** `[Feature] shadcn-audit: pixel diff evidence for collect`. + +### Task 6: Playwright driver + +**Files:** +- Create: `audit/lib/shadcn_audit/collect/driver.rb`, `audit/lib/shadcn_audit/collect/probes.js` +- Test: `audit/test/collect/driver_smoke_test.rb`, `audit/test/fixtures/collect/{a,b}.html` + +**Interfaces:** +- Produces: + - `Collect::BrowserError < StandardError` (message used verbatim by the CLI). + - `Collect::Driver.open(audit_dir:) { |driver| … }`: launches headless Chromium via `Playwright.create(playwright_cli_executable_path: /node_modules/.bin/playwright)`; raises `BrowserError` with the install hint (Review Focus 4) when the CLI or browser is missing. + - `Driver#environment(url) -> Hash` keys `w h dpr font fonts body forms dark` (the probe in `interactions.md`, `fonts` sorted). + - `Driver#observe(url:, state:, selectors:, screenshot_path:) -> SideFacts` (shape in Task 3). `selectors` is `{part => css}`; the runner passes `[data-slot=""]` for shadcn. + - `Driver::STATES = %w[closed open focus after-esc]`. + +Behavior (fresh `page` per call, viewport 1280×800, `reduced_motion: "reduce"`): +- Navigate, wait for `load` (timeout 15 s); page error / timeout → `reached: false`, reason = message's first line. +- `closed`: nothing more. `open`: real `click` on `[data-audit-trigger]` (missing → `reached: false`, reason `"no [data-audit-trigger]"`). `focus`: press `Tab` up to 20 times until `document.activeElement` is or contains the trigger (else unreached `"trigger never received keyboard focus"`). `after-esc`: `open` then press `Escape`. +- Settle: await all `document.getAnimations()` `finished` (cap 2 s), then two `requestAnimationFrame`s. +- Per part: elements matching the selector **and** `checkVisibility()` (Review Focus 2); for each: computed styles for the property list in `Driver::STYLE_PROPS` (color, background-color, border-{top,right,bottom,left}-{width,style,color}, border-{top-left,top-right,bottom-right,bottom-left}-radius, box-shadow, outline-width, outline-style, outline-color, outline-offset, padding-{top,right,bottom,left}, gap, row-gap, column-gap, width, height, font-family, font-size, font-weight, line-height, letter-spacing, text-align, text-decoration-line, opacity, display, transform), `locator.aria_snapshot`, `attrs` per `Grader::ARIA_ATTRS` rules. +- Screenshot `page.screenshot(path:, animations: "disabled", caret: "hide")`; `screenshot` in facts is the path relative to the run folder (runner passes both). + +- [ ] **Step 1: Write the smoke test** (skips with `skip "chromium not installed"` when `node_modules/.bin/playwright` is absent or `Driver.open` raises `BrowserError`), loading `file://` URLs of two fixtures: `a.html` has ` + + + + diff --git a/audit/test/fixtures/collect/b.html b/audit/test/fixtures/collect/b.html new file mode 100644 index 000000000..92f8e85c8 --- /dev/null +++ b/audit/test/fixtures/collect/b.html @@ -0,0 +1,2 @@ + +

No trigger here

diff --git a/audit/test/fixtures/collect/c.html b/audit/test/fixtures/collect/c.html new file mode 100644 index 000000000..c4b4e438b --- /dev/null +++ b/audit/test/fixtures/collect/c.html @@ -0,0 +1,4 @@ + + + + diff --git a/audit/test/fixtures/collect/d.html b/audit/test/fixtures/collect/d.html new file mode 100644 index 000000000..972759520 --- /dev/null +++ b/audit/test/fixtures/collect/d.html @@ -0,0 +1,19 @@ + + + + + + + +

Modal

 + +
hidden by opacity
+
pulsing
+
slow finite fade-in
+ + diff --git a/audit/test/fixtures/collect/e.html b/audit/test/fixtures/collect/e.html new file mode 100644 index 000000000..086c7a872 --- /dev/null +++ b/audit/test/fixtures/collect/e.html @@ -0,0 +1,11 @@ + + +
+ + diff --git a/audit/test/fixtures/collect/f.html b/audit/test/fixtures/collect/f.html new file mode 100644 index 000000000..cc3275c2f --- /dev/null +++ b/audit/test/fixtures/collect/f.html @@ -0,0 +1,8 @@ + + +
+ +