build(deps-dev): bump the playwright group in /demo with 2 updates - #1120
Conversation
6cd3cb0 to
d7f10a1
Compare
|
@makhnatkin needs a human: Playwright 1.52 → 1.59 changes Chromium rendering, so every visual snapshot drifts; I rebased the branch and regenerated all snapshots in the v1.59.1-jammy container (208 pass), but 7 tests then fail on real assertions, not images — e.g. |
d7f10a1 to
bc20f40
Compare
|
@makhnatkin needs human: Playwright 1.59’s bundled Chromium changes arrow-key caret navigation in the ProseMirror editor, which breaks several visual tests in a way no lockfile/snapshot change can fix. What I did (pushed): rebased onto current What is left (needs a real decision, not a mechanical fix):
Fixing these means rewriting how those tests move the caret (or asserting on editor state instead of visual line position), which is test-design work rather than a dependency adaptation — so I stopped here rather than merging. |
|
@makhnatkin needs human: the Playwright 1.59.1 bump leaves three visual tests that I can't fix with a snapshot refresh. What I did: reproduced the CI failures in the What's left (run 35718068536):
All three sit on text-wrap boundaries and flip between runs (they were in the Not approved and not merged: checks are red. |
fc43ec8 to
367d651
Compare
|
@makhnatkin Rebased onto
Snapshots above were generated in the same container image CI uses, with |
8dabae6 to
c6329c2
Compare
|
@makhnatkin needs human: this bump needs real test-stability work, not just a snapshot refresh. What I did on this branch:
Why the revert: the earlier snapshot refresh on this branch was generated from a local run whose renders were structurally wrong for those three tests, e.g. the Clipboard one was committed at 668x836 (one copy of the content) while CI renders 668x1354 (original + pasted copy, which is what the test actually does). Same pattern for Colorify (450 vs 486) and Link (464 vs 536). That fixed the bulk of it — the remaining diffs dropped from ~20 000 px to 123–2512 px, i.e. ordinary antialiasing churn from the Chromium in playwright 1.59. Still red though (run, 4 failed / 3 flaky / 208 passed):
1, 2 and 4 look like plain snapshot refreshes now. 3 is genuinely non-deterministic under 1.59 and is the blocker: across CI attempts the markup preview came back as Note for whoever picks this up: |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 1.59 Playwright 1.59 ships the new Chromium headless shell, which changes where the caret lands after arrow-key navigation and how copied HTML is serialised. Updates the two snapshots and the one text assertion that encoded the old behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous snapshot refresh on this branch was produced by a local run whose renders were systematically truncated for three tests. CI renders them at main's original dimensions, so revert those snapshots and the Link caret assertion to main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Colorify, Link and Clipboard screenshots listed here still carried their playwright 1.52 baselines. Regenerated in the mcr.microsoft.com/playwright:v1.59.1-jammy image used by CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c6329c2 to
63e3295
Compare
|
@makhnatkin needs human: playwright 1.59.1's Chromium makes What I did push (63e3295): rebased onto What is left, measured on one machine, full suite,
And the Link test alone,
The cause is the caret setup at Picking a stable replacement means deciding what state this test is supposed to screenshot, and rebaselining accordingly — a call on test intent that I don't want to make on your behalf, especially as the earlier attempts on this branch (8dabae6, then the revert in 20218ce) went the same way. The broader 1→13 flaky jump is also worth a look before this lands; it is caret-blink noise that Not approved and not merged — Visual Tests is still red on this one test. |
|
@makhnatkin needs a human: Playwright 1.59’s Chromium changes Measured in the CI container image (
So this is a real behaviour change introduced by the bump, not a pre-existing flake and not a stale snapshot. It is also load-dependent: the test passes when run alone, which is why CI shows it as a hard failure only sometimes (it lost all 3 retries in the last run) and why the rest of the suite reports several other tests as flaky. What the test does: after Why I stopped rather than adapting the assertion:
Everything else on this PR is in good shape: Not merging; leaving this one for you. |
|
@makhnatkin Needs a human: the one remaining red test ( What I verified locally in the CI container image (amd64,
Root cause: the setup relies on Things I tried that did not make it deterministic:
A real fix means positioning the caret deterministically instead of depending on visual-line navigation, which also changes the resulting document shape and therefore the Everything else on the PR is green, and the branch is up to date with Unrelated note: |
Chromium bundled with Playwright 1.59 no longer treats ArrowUp on the first line as a no-op: the second press now collapses the caret to offset 0 instead of leaving it at offset 25, so Enter stopped splitting "amet" and the markup assertion added in #1194 never matched. Pressing ArrowUp once reproduces the pre-bump caret position exactly (verified against the 1.52 image: offset 25 -> "t," -> "Lorem ipsum dolor sit ame"), so the existing assertion is kept untouched as the oracle. The extra settle wait after fill() removes a pre-existing race where the first ArrowUp landed before layout settled, and the refreshed form-with-url snapshots carry only the new Chromium's anti-aliasing delta (123px). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed the Cause — not Playwright itself, but the Chromium bundled with 1.59. Verified rather than guessed — I probed
So one Also added a short settle wait after The two Full visual suite in the 1.59 Docker image: 211 passed, 0 failed (4 flaky "mark via toolbar" screenshots that recover on retry, pre-existing). |
The light snapshot for "copy and paste with preserve markup" refreshed in an earlier commit on this branch showed the pasted heading as "## **Emphasis**" (bold kept in the markup preview). Neither the baseline on main nor the 1.59.1 CI container ever renders that: both produce "## Emphasis". The bold variant came from a local run and was the only snapshot still failing CI, at a stable 2389px delta over two retries. Replaced with the image the 1.59.1 CI container actually produces, which differs from main's baseline by 567px of anti-aliasing only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same corruption as the light snapshot fixed in the previous commit, and hidden behind it: expectScreenshot compares light first, so while the light assertion failed the dark one was never reached and its committed image was never validated against CI. With light passing, dark now fails at a stable 2368px delta, and the two differing bands are exactly the ones from the light case — the markup preview reading "## **Emphasis**" instead of "## Emphasis". Replaced with the image the 1.59.1 CI container produces. It agrees with main's dark baseline everywhere except anti-aliasing (max per-channel delta of 1) and a 30x30px region at the top right. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
makhnatkin
left a comment
There was a problem hiding this comment.
Playwright 1.52.0 -> 1.59.1 for the demo package, with the container image in pr-visual-tests.yml and playwright-docker.sh kept in sync.
Reviewed the non-snapshot changes: the Link caret adaptation is justified by the new Chromium collapsing the caret to offset 0 on the second ArrowUp, and the two Clipboard snapshots were corrected against the images the 1.59.1 CI container actually produces (the previously committed pair showed ## **Emphasis**, which neither main nor CI renders). All checks green.
Bumps the playwright group in /demo with 2 updates: @playwright/experimental-ct-react and @playwright/test.
Updates
@playwright/experimental-ct-reactfrom 1.52.0 to 1.59.1Release notes
Sourced from @playwright/experimental-ct-react's releases.
... (truncated)
Commits
d466ac5chore: mark v1.59.1 (#40005)530e7e5cherry-pick(#4004): fix(cli): kill-all should kill dashboard9aa216ccherry-pick(#39994): Revert "fix(windows): hide console window when spawning ...01b2b15cherry-pick(#39980): chore: more release notes fixesa5cb6c9cherry-pick(#39972): chore: expose browser.bind and browser.unbind APIs99a17b5cherry-pick(#39975): chore: support opening .trace files via .link indirection43607c3cherry-pick(#39974): chore(webkit): update Safari user-agent version to 26.462cabe1cherry-pick(#39969): chore(npm): include all *.md from lib (#39970)0c65a75cherry-pick(#39968): chore: screencast.showActions apif04155bcherry-pick(#39958): chore: release notes for langs v1.59Maintainer changes
This version was pushed to npm by GitHub Actions, a new releaser for
@playwright/experimental-ct-reactsince your current version.Updates
@playwright/testfrom 1.52.0 to 1.59.1Release notes
Sourced from @playwright/test's releases.
... (truncated)
Commits
d466ac5chore: mark v1.59.1 (#40005)530e7e5cherry-pick(#4004): fix(cli): kill-all should kill dashboard9aa216ccherry-pick(#39994): Revert "fix(windows): hide console window when spawning ...01b2b15cherry-pick(#39980): chore: more release notes fixesa5cb6c9cherry-pick(#39972): chore: expose browser.bind and browser.unbind APIs99a17b5cherry-pick(#39975): chore: support opening .trace files via .link indirection43607c3cherry-pick(#39974): chore(webkit): update Safari user-agent version to 26.462cabe1cherry-pick(#39969): chore(npm): include all *.md from lib (#39970)0c65a75cherry-pick(#39968): chore: screencast.showActions apif04155bcherry-pick(#39958): chore: release notes for langs v1.59Maintainer changes
This version was pushed to npm by GitHub Actions, a new releaser for
@playwright/testsince your current version.You can trigger a rebase of this PR by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore <dependency name> major versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)@dependabot ignore <dependency name> minor versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)@dependabot ignore <dependency name>will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)@dependabot unignore <dependency name>will remove all of the ignore conditions of the specified dependency@dependabot unignore <dependency name> <ignore condition>will remove the ignore condition of the specified dependency and ignore conditions