Repository navigation
ci: run every e2e suite on pull requests and in test:e2e:all - #144
Merged
Merged
Conversation
Only the @smoke tests had a script that ran them all, and no workflow ran tests on pull requests, so changes broke other areas' suites without anyone noticing (#136, #135). - npm run test:e2e:all runs @smoke and then every area config one after another on fresh servers, keeps each suite's report, and fails if any suite fails. It also fails if a playwright*.config.ts is not in the suite list or a test is not run by any suite. The @Full export matrix stays on demand. - .github/workflows/ci.yml runs lint, the build, npm test, the coverage check, and each e2e suite as a parallel job on PRs into develop and main (and pushes to them, which seeds the caches). It caches npm and Chromium, uses a read-only token, and adds one "CI passed" check. - Area specs fail up front under the wrong Playwright config and name the script to use. - Palette Proof, the Settings legal links, and the adaptive Stack Matrix round trip were in no suite; they now run with @smoke. Refs #138 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vycdev
force-pushed
the
ci/issue-138-e2e-suites
branch
from
November 7, 2026 12:02
c0461a7 to
13b3ef8
Compare
Playwright's JSON --list output puts load errors, such as a spec that does not compile, in stdout, so the coverage check printed an empty error and test:e2e:all crashed before running any suite. Report those errors as coverage failures and keep running the suites. Match the on-demand @Full tag against Playwright's parsed tags instead of a substring of the title, so a title like "@fullscreen" is not silently excluded. Refs #138 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #138. It doesn't close the issue; that's left for review.
What changes
npm run test:e2e:all(scripts/run-e2e-suites.mjs) runs@smokeand then each area config one after another: docs, landing, privacy, terms, not-found, i18n, and i18n build. It runs every suite even after one fails, prints a pass/fail summary, and exits non-zero if any failed. Each suite keeps its owntest-results/<suite>andplaywright-report/<suite>, so a later suite doesn't delete an earlier one's traces.Before the suites run, it checks coverage, using Playwright's
--list(about 4s, no browsers). The check fails if:playwright*.config.tsisn't in the suite list, or@full.npm run check:e2eruns only that check.@matrix,@stress, and@fullstay out of the default run. Every@fulltest is one of the 12kromacut-flow.spec.tsexport cases (2 images × 3 profiles × 2 meshers). The 3888px@stresscases allow 90 minutes each, and the@matrixcases 30 minutes each. Running them on every PR would turn a ~10 minute gate into hours. The one cheap flow case (1024 PNG, 2 colors, greedy) is already tagged@smoke, so the export flow is still covered. The scriptstest:e2e:matrix,test:e2e:stress, andtest:e2e:fullare unchanged for running the rest on demand.Five tests weren't run by any script. The coverage check found them, and all five pass on
develop. Following the existing@smoke @matrixconvention, they're now tagged@smoke:palette-proof.spec.ts: Palette Proof round tripsettings-legal-links.spec.ts: all 3 testsstack-matrix-settings.spec.ts: the adaptive thickness round trip, which was@matrixonlyFresh servers.
playwright.config.tsandplaywright.i18n.config.tsreuse an existing server outside CI. While testing this, my@smokerun reused avite previewon port 5173 that belonged to another worktree. It tested that checkout's build and failed when that server restarted.test:e2e:allnow setsKROMACUT_E2E_FRESH_SERVERS=1, which turns reuse off, so a port that's already in use fails clearly. Runningnpm run test:e2eon its own still reuses a server, as before.Config guard. The docs, i18n, i18n-build, not-found, privacy, and terms specs call
requirePlaywrightConfig()(tests/e2e/helpers/playwrightConfig.ts, next to the fake desktop helper from #139). Under any other config, abeforeAllfails with, for example:This spec only runs with playwright.i18n.config.ts, not playwright.config.ts. Use \npm run test:e2e:i18n`.It fails instead of skipping so that a renamed config can't turn a CI suite into a silent all-skipped pass. The landing specs are tagged@smoke` and also run under the default config, so they aren't guarded..github/workflows/ci.ymldevelopandmain, on pushes to them (base-branch runs create the npm and Chromium caches that PRs restore), and onworkflow_dispatch.checksjob: lint, build,npm test,check:e2e.e2ejob: a parallel matrix with one job per suite andfail-fast: false. The matrix comes fromrun-e2e-suites.mjs --list, so CI andtest:e2e:allshare one suite list.CI passedjob: a single check that fails if any job failed, was cancelled, or was skipped. Branch protection can require just this check as suites are added.contents: readat the top level and none forCI passed. Checkout usespersist-credentials: false. The suite name reaches the shell throughenv, not${{ }}. Node 22 matches the deploy and release workflows.pull_request, neverpull_request_target, so fork code gets a read-only token and no secrets. Maintainer approval for fork runs is a repository setting, not a workflow setting. I suggest Settings → Actions → General → "Require approval for all external contributors". The default only covers first-time contributors.AGENTS.mdtesting guidance andCHANGELOG.md(Unreleased → Added) are updated. The desktop fakes from #139/#134 aren't touched. The only edits to specs are the guard call and the@smoketags.Verification (macOS)
npm run lint,npm run build, andnpm test(1105 passing after the rebase) all pass.npm run check:e2epasses: every e2e test except@fullis covered.npm run test:e2e:alltook 11 minutes, both before and after rebasing ontodevelop(which now includes test: share one fake desktop runtime across e2e specs #139 and refactor: share one decodeImage helper for image loading #141). 6 of 8 suites pass: smoke 81, docs 6, landing 19, not-found 10, i18n 17, i18n build 4.develop: 4 + 4 tests in the narrow-screen footer checks expect 7 footer links, but the Awards link (Add awards page with Snapmaker Innovation Fund winner badge #126) made it 8. It's the same cause as i18n landing footer e2e tests fail after the Awards link #136, but those suites weren't covered by test: align the i18n footer test with the Awards link #137. I reported it as Privacy and terms e2e footer tests still expect 7 links after the Awards link #143 and didn't fix it here, so the privacy and terms CI jobs (andCI passed) will fail until Privacy and terms e2e footer tests still expect 7 links after the Awards link #143 lands.actionlint1.7.12 reports no problems inci.yml. shellcheck isn't installed, so the shell snippets were reviewed by hand.@smoketakes about 11.5 minutes). privacy and terms fail only on the 8 Privacy and terms e2e footer tests still expect 7 links after the Awards link #143 footer tests, and their traces were uploaded as artifacts. The first run missed the Chromium cache, as expected; pushes todevelopwill fill it.🤖 Generated with Claude Code