Skip to content

ci: run every e2e suite on pull requests and in test:e2e:all - #144

Merged
vycdev merged 2 commits into
developfrom
ci/issue-138-e2e-suites
Nov 10, 2026
Merged

vycdev merged 2 commits into
developfrom
ci/issue-138-e2e-suites

Conversation

@vycdev

@vycdev vycdev commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Refs #138. It doesn't close the issue; that's left for review.

What changes

npm run test:e2e:all (scripts/run-e2e-suites.mjs) runs @smoke and then each area config one after another: docs, landing, privacy, terms, not-found, i18n, and i18n build. It runs every suite even after one fails, prints a pass/fail summary, and exits non-zero if any failed. Each suite keeps its own test-results/<suite> and playwright-report/<suite>, so a later suite doesn't delete an earlier one's traces.

Before the suites run, it checks coverage, using Playwright's --list (about 4s, no browsers). The check fails if:

  • a playwright*.config.ts isn't in the suite list, or
  • a test listed by the default config isn't run by any suite, unless it's tagged @full.

npm run check:e2e runs only that check.

@matrix, @stress, and @full stay out of the default run. Every @full test is one of the 12 kromacut-flow.spec.ts export cases (2 images × 3 profiles × 2 meshers). The 3888px @stress cases allow 90 minutes each, and the @matrix cases 30 minutes each. Running them on every PR would turn a ~10 minute gate into hours. The one cheap flow case (1024 PNG, 2 colors, greedy) is already tagged @smoke, so the export flow is still covered. The scripts test:e2e:matrix, test:e2e:stress, and test:e2e:full are unchanged for running the rest on demand.

Five tests weren't run by any script. The coverage check found them, and all five pass on develop. Following the existing @smoke @matrix convention, they're now tagged @smoke:

  • palette-proof.spec.ts: Palette Proof round trip
  • settings-legal-links.spec.ts: all 3 tests
  • stack-matrix-settings.spec.ts: the adaptive thickness round trip, which was @matrix only

Fresh servers. playwright.config.ts and playwright.i18n.config.ts reuse an existing server outside CI. While testing this, my @smoke run reused a vite preview on port 5173 that belonged to another worktree. It tested that checkout's build and failed when that server restarted. test:e2e:all now sets KROMACUT_E2E_FRESH_SERVERS=1, which turns reuse off, so a port that's already in use fails clearly. Running npm run test:e2e on its own still reuses a server, as before.

Config guard. The docs, i18n, i18n-build, not-found, privacy, and terms specs call requirePlaywrightConfig() (tests/e2e/helpers/playwrightConfig.ts, next to the fake desktop helper from #139). Under any other config, a beforeAll fails with, for example:
This spec only runs with playwright.i18n.config.ts, not playwright.config.ts. Use \npm run test:e2e:i18n`.It fails instead of skipping so that a renamed config can't turn a CI suite into a silent all-skipped pass. The landing specs are tagged@smoke` and also run under the default config, so they aren't guarded.

.github/workflows/ci.yml

  • Runs on PRs into develop and main, on pushes to them (base-branch runs create the npm and Chromium caches that PRs restore), and on workflow_dispatch.
  • checks job: lint, build, npm test, check:e2e.
  • e2e job: a parallel matrix with one job per suite and fail-fast: false. The matrix comes from run-e2e-suites.mjs --list, so CI and test:e2e:all share one suite list.
  • Chromium is cached by Playwright version. A cache hit only installs the system deps.
  • On failure, each job uploads its Playwright report and traces as an artifact.
  • CI passed job: a single check that fails if any job failed, was cancelled, or was skipped. Branch protection can require just this check as suites are added.
  • Permissions: contents: read at the top level and none for CI passed. Checkout uses persist-credentials: false. The suite name reaches the shell through env, not ${{ }}. Node 22 matches the deploy and release workflows.
  • Fork PRs: it uses pull_request, never pull_request_target, so fork code gets a read-only token and no secrets. Maintainer approval for fork runs is a repository setting, not a workflow setting. I suggest Settings → Actions → General → "Require approval for all external contributors". The default only covers first-time contributors.

AGENTS.md testing guidance and CHANGELOG.md (Unreleased → Added) are updated. The desktop fakes from #139/#134 aren't touched. The only edits to specs are the guard call and the @smoke tags.

Verification (macOS)

🤖 Generated with Claude Code

Only the @smoke tests had a script that ran them all, and no workflow
ran tests on pull requests, so changes broke other areas' suites without
anyone noticing (#136, #135).

- npm run test:e2e:all runs @smoke and then every area config one
  after another on fresh servers, keeps each suite's report, and fails
  if any suite fails. It also fails if a playwright*.config.ts is not
  in the suite list or a test is not run by any suite. The @Full
  export matrix stays on demand.
- .github/workflows/ci.yml runs lint, the build, npm test, the coverage
  check, and each e2e suite as a parallel job on PRs into develop and
  main (and pushes to them, which seeds the caches). It caches npm and
  Chromium, uses a read-only token, and adds one "CI passed" check.
- Area specs fail up front under the wrong Playwright config and name
  the script to use.
- Palette Proof, the Settings legal links, and the adaptive Stack Matrix
  round trip were in no suite; they now run with @smoke.

Refs #138

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vycdev
vycdev force-pushed the ci/issue-138-e2e-suites branch from c0461a7 to 13b3ef8 Compare November 7, 2026 12:02
Playwright's JSON --list output puts load errors, such as a spec that
does not compile, in stdout, so the coverage check printed an empty
error and test:e2e:all crashed before running any suite. Report those
errors as coverage failures and keep running the suites.

Match the on-demand @Full tag against Playwright's parsed tags instead
of a substring of the title, so a title like "@fullscreen" is not
silently excluded.

Refs #138

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vycdev
vycdev merged commit 5370f46 into develop Nov 10, 2026
8 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant