Repository navigation
🧪 fix: Revert the Live Spec List After Every Journey, Keep Base Overrides Out of the Visual Baselines and Refresh the Two Stale 390 px Ones - #210
Conversation
…ides Out of the Visual Baselines and Refresh the Two Stale 390 px Ones e2e/specs.spec.ts reverted the Base spec list only as the last statement of its two saving journeys. When an assertion after Save failed, "E2E Spec" or "E2E Last" stayed in Base, the next run's new spec was named e2e-spec-2, and the serial describe skipped every journey after it. revertList now runs in a beforeAll, for a list an earlier run left behind, and in an afterEach, after every journey, passed or failed. It opens a page in a context of its own, so whatever a journey left on its page (a review dialog, unsaved edits) cannot get in the way, returns early when Base stores no list, and checks that the revert took. e2e/import.spec.ts left the Starting credits it imports into Base behind after every live run. An afterEach now reverts them. Every Base override shows in the configuration visual baselines: its tab's override count, the header's revert action at 1440 px, and its chip and section count on its own tab. Before its screenshot, each baseline now reloads until Base stores no admin overrides, since another spec's journey holds one for a few seconds, and fails naming the tabs that still have one. A leftover therefore never reaches a comparison or an --update-snapshots run. The 390 px system and features baselines failed on main on a clean database. #170 wrote them before its own last changes: at phone width a row's source chip moved under its label, the startup note gained a title and new wording, and "OAuth state TTL (ms)" became "OAuth state TTL". #193 then moved the collapsed rail's icons 1.5 px to its center. Both baselines are regenerated from main on a clean database. No threshold changed.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f95ba34af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…gh Its Screenshot and Around Every Live Base Write
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
# Conflicts: # e2e/a11y/config.spec.ts # e2e/config-shell.spec.ts
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…evert Actions Start With Their Visible Words
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Pull Request Template
Summary
This PR is e2e test hygiene only. It changes
e2e/specs.spec.ts,e2e/import.spec.ts,e2e/a11y/config.spec.tsand two of that file's baseline images, and adds a Base lock toe2e/live.tsthat the other specs saving to Base (features,providers,saves,mcp) now hold. No panel code changes.e2e/specs.spec.ts: one failed live run broke every run after it.The two journeys that save, "a new spec asks for Endpoint and Model up front…" and "reordering is saved…", called
revertList(page)only as their last statement. There was notry/finally,afterEachorafterAll. The cascade went like this:addSpecderived the namee2e-spec-2, sotoHaveValue('e2e-spec')failed and the serial describe skipped the rest.The file's doc comment already required a Base without a spec list of its own, but nothing restored one.
The fix:
revertListruns in abeforeAll, for a list an earlier run left behind. It also runs after every journey, passed or failed, as the cleanup of the Base lock (below), so the list is back before the lock is released. The two in-test calls are gone.baseURLandstorageState(the same pattern as the staging page inaccess.spec.ts). That way it doesn't depend on what a failed journey left on its own page: a review dialog, unsaved edits, or the 390 px viewport the last journey switches to. AbeforeAllhas nopagefixture anyway, so one helper serves both hooks.e2e/import.spec.tsleft a Base override behind after every live run."imports a snippet into Base, previewing exactly its keys" imports a random Starting credits value into Base and never reverts it. After one live run of the file, Base stores
balance.startBalance. That was the leftover behind part of the system baseline diff in the original report. The test now sits in a describe whose Base lock cleanup reverts Starting credits to its librechat.yaml value, passed or failed, before the lock is released.e2e/a11y/config.spec.ts: the visual baselines drew whatever Base stored.An admin override in Base changes the configuration page in several places:
On a stack with a Base override left behind (here, main's
import.spec.tsleftovers), the images changed by these amounts:Each change stays under its limit. So a leftover passed silently, used up part of the margin, and would have been written into the image by the next
--update-snapshots.The fix has two parts.
A check before each screenshot. Each visual baseline opens its tab and lists the tabs whose count shows an admin override. If the list isn't empty, the test fails before its screenshot is compared or written, with
Tabs where Base stores admin overrides, which no baseline shows, and the received value names the tab, e.g.["System: 1 admin override"]. The thresholds are unchanged.A Base lock held through the screenshot. A check alone only proves that Base was clean when the page loaded. Six spec files save to Base during a parallel live run, each holding the override for 1 to 5 s, and nothing kept one from saving between a baseline's check and its screenshot:
So the check and the screenshot now run under a lock that every journey saving to Base also holds:
e2e/live.ts: the cross-workermkdirlock behindlockUserProfilenow backs two named locks, the USER role profile and Base.lockBase(cleanup?)holds the Base lock through each test of the calling describe and its cleanup, which runs before the release, also after a failure.withBaseLockholds it around a hook.import.spec.ts, the base configurationbalance.startBalance, 2.1 sspecs.spec.ts, Model specsmodelSpecs.list, 3.0 s and 5.1 srevertList), and abeforeAllunder the lockfeatures.spec.ts, Features editorsmemory.agent.*2.3 s,actions.allowedDomains1.2 s,summarization.trigger.*1.0 sproviders.spec.ts, the whole fileendpoints.anthropic.models, 1.0 ssaves.spec.ts, custom endpoints by identityendpoints.custom, 2.5 s, 3.5 s and 2.8 smcp.spec.ts, MCP serversmcpServers.<new server>, 4.2 sThe times are from a MongoDB change stream on the
configscollection during a one-worker live run. The web search journey fails before it saves on this LibreChat (see Not in this PR), so it has no time.A few details:
lockUserProfileis used, so the read-only journeys in those describes hold it for a second or two as well.lockBase's doc comment asks for that order, so two workers can't each hold one lock while waiting for the other.Why a lock rather than a Playwright project that runs after every other live test (project
dependencies):-g 'visual baseline'or an--update-snapshotsrun of the baselines would first run every other live test.-chromium-linux.png), so all nine images would have to move or get a customsnapshotPathTemplate.The two 390 px baselines were stale, not regressed.
visual baseline: light-390 › systemand› featuresfail onmainon a clean database. To find out why, I measured every baseline at zero tolerance on a clean database, against a build ofmainand against a build of #170's merge commit, the last commit that wrote these images:mainvs baseline (px)So the 390 px images already failed against the code they were committed with. #170 wrote them before its own last changes landed, and every difference in them is a change #170 itself made:
librechat.yamlchip from beside the label to its own line under the help text, so the help text gets the row's width. This is most of the features diff (Model select, Parameters, Presets and Context cost rows) and the Allowed domains row in system.com_config_registration_startup_title,com_config_registration_startup_yaml).Between #170 and
main, the 390 px images changed by only ~1,025 px:#209 changes none of these images: the
mainbuild at #209 and at #206 render identically.Both 390 px baselines are regenerated with
--update-snapshots -g 'visual baseline: light-390 (system|features)$', against amainbuild on a database with no admin overrides. The new images match the first capture on a fresh instance pixel for pixel. Each shows Base with librechat.yaml values only: no override count, chip or revert action. They show:system-light-390: the Registration section with the Allowed domains chip under its help text, the titled startup note, "OAuth state TTL" and the centered rail.features-light-390: the Interface section's Chat controls with eachlibrechat.yamlchip on its own line under the row's help text, and the centered rail.After merging #207. #207 made every revert action's name start with its visible words ("Revert to librechat.yaml (1,000,000): Starting credits").
revertBaseStartBalancestill looked for the old "Revert Starting credits to …" name, found no button, and took its "nothing to revert" path, so the Base import's Starting credits stayed in Base. The live run on the merged tree showed it: everything passed, but Base kept the override afterwards, and the next run's baselines would have failed on it. It now matches every revert name form (/^(Revert to .+|Remove override): Starting credits$/), asfeatures.spec.tsdoes. On that leftover, the baselines failed first with "System: 1 admin override" and no screenshot taken. Then the fixed Base import test passed and reverted it (Base stored{}), and the baselines passed again.On the merged tree: the full mock suite gave 127 passed, exit 0. Live, all eight Base-writing specs together at
--workers=2gave 146 passed, with the one known web search failure (and the 2 serial journeys after it not run). Base stored no overrides afterwards, and no lock directory was left.Not in this PR
import.spec.tsstill leaves a USER role profile's Starting credits, an ADMIN profile's welcome message and a new role behind on each live run. The USER one draws an "Overridden for USER" link into the 1440 px system images (~1,480 px, within the limit). The new check covers Base overrides only, not profile ones.features.spec.ts,providers.spec.ts,saves.spec.tsandmcp.spec.tsrevert inside the journey. A journey that fails between its save and its revert leaves the override in Base, the wayspecs.spec.tsdid. The lock is still released, and the next baseline fails naming the tab. Moving those reverts into lock cleanups is a follow-up.features.spec.ts's "a web search key is masked…" fails before it saves. The server reads web search settings only from environment variable references, so the panel shows that note instead of "Will be stored encrypted when you save.". The serial describe then skips its last two journeys. It fails the same way withmain's spec.e2e/a11y/config.spec.ts(mcp at 1440 and 390, features at 1440, and the two change review dialogs, in both themes) fail live onmainwithlabel-content-name-mismatch. They fail the same way withmain's copy of the spec, and nothing here touches them.Change Type
Testing
All live runs used
E2E_LIVE=1against a local LibreChat backend with its own database. The investigation started on a fresh instance (new database, new backend).Live,
e2e/specs.spec.tsmainmaintoHaveValue('e2e-spec'): received"e2e-spec-2". 8 journeys did not run.mainbeforeAllreverted the list.The skip is "a profile shows the specs it inherits…", whose documented precondition is a USER role profile. A same-page revert in an
afterEachalso recovered in the open-dialog case, becausepage.gotoleaves the dirty page without its leave prompt. So the separate context is for thebeforeAll, and so that one helper can serve both hooks.Live,
e2e/import.spec.ts: 6 passed. Afterwards Base stored no overrides.Live, the Base override check in
e2e/a11y/config.spec.tsReceived: ["System: 1 admin override"]import.spec.tsleavesLive, the Base lock
These runs used two temporary helpers that aren't in the PR:
visibilitychangeevent)--update-snapshotswrites it.Live, every changed spec together (
a11y/config,import,specs,features,providers,saves,mcp,config-shell), with 2 workers and again with 4: 109 passed each time. 11 failed: the 10 axelabel-content-name-mismatchtests listed above, as they do withmain's spec, and the web search journey. 2 did not run (the features describe's last two). All 9 visual baselines passed. Afterwards Base stored no overrides, and no lock was left in the temp dir.Mock backend, full suite: exit 0, with 114 passed and 148 skipped (the live-only specs). The new hooks never run there, because every describe that calls them is live-only.
Test Configuration:
npx tsc --noEmit,npx eslint src/ --max-warnings 0, Prettier on the changed files andbun run test:lint-rules(19 passed) are clean. The ESLint config does not covere2e/.src/changes, so vitest and the build are unaffected. No baseline image changed beyond the two above.Checklist