Skip to content

🧪 fix: Revert the Live Spec List After Every Journey, Keep Base Overrides Out of the Visual Baselines and Refresh the Two Stale 390 px Ones - #210

Merged
danny-avila merged 4 commits into
mainfrom
pr/fu-e2e-specs-cleanup
Oct 29, 2026
Merged

danny-avila merged 4 commits into
mainfrom
pr/fu-e2e-specs-cleanup

Conversation

@danny-avila

@danny-avila danny-avila commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request Template

⚠️ Before Submitting a PR, Please Review:

  • Please ensure that you have thoroughly read and understood the Contributing Docs before submitting your Pull Request.

⚠️ Documentation Updates Notice:

  • Kindly note that documentation updates are managed in this repository: librechat.ai

Summary

This PR is e2e test hygiene only. It changes e2e/specs.spec.ts, e2e/import.spec.ts, e2e/a11y/config.spec.ts and two of that file's baseline images, and adds a Base lock to e2e/live.ts that the other specs saving to Base (features, providers, saves, mcp) now hold. No panel code changes.

e2e/specs.spec.ts: one failed live run broke every run after it.

The two journeys that save, "a new spec asks for Endpoint and Model up front…" and "reordering is saved…", called revertList(page) only as their last statement. There was no try/finally, afterEach or afterAll. The cascade went like this:

  • If an assertion after Save failed (a slow reload, for example), "E2E Spec" or "E2E Last" stayed in Base.
  • On the next run, addSpec derived the name e2e-spec-2, so toHaveValue('e2e-spec') failed and the serial describe skipped the rest.
  • A leftover "E2E Last" would also make the "Move E2E Last up" button ambiguous.

The file's doc comment already required a Base without a spec list of its own, but nothing restored one.

The fix:

  • revertList runs in a beforeAll, for a list an earlier run left behind. It also runs after every journey, passed or failed, as the cleanup of the Base lock (below), so the list is back before the lock is released. The two in-test calls are gone.
  • It opens its page in a browser context of its own, built from the project's baseURL and storageState (the same pattern as the staging page in access.spec.ts). That way it doesn't depend on what a failed journey left on its own page: a review dialog, unsaved edits, or the 390 px viewport the last journey switches to. A beforeAll has no page fixture anyway, so one helper serves both hooks.
  • It still returns early when Base stores no list. After saving, it now checks that "Revert list to librechat.yaml" is gone, so a revert that fails to take is reported.

e2e/import.spec.ts left a Base override behind after every live run.

"imports a snippet into Base, previewing exactly its keys" imports a random Starting credits value into Base and never reverts it. After one live run of the file, Base stores balance.startBalance. That was the leftover behind part of the system baseline diff in the original report. The test now sits in a describe whose Base lock cleanup reverts Starting credits to its librechat.yaml value, passed or failed, before the lock is released.

e2e/a11y/config.spec.ts: the visual baselines drew whatever Base stored.

An admin override in Base changes the configuration page in several places:

  • Every tab: the overridden tab gains an override count in the tab bar.
  • 1440 px: the header gains "Revert to librechat.yaml (N)".
  • The override's own tab: its row gains an Admin override chip and a revert link, and its section count gains "1 override".

On a stack with a Base override left behind (here, main's import.spec.ts leftovers), the images changed by these amounts:

Baselines Pixels changed by the leftovers Limit (2%)
system, 1440 (light, dark) ~9,100 31,680
features and custom, 1440 (light, dark) ~3,080 31,680
system, 390 1,777 8,580
features and custom, 390 0 8,580

Each change stays under its limit. So a leftover passed silently, used up part of the margin, and would have been written into the image by the next --update-snapshots.

The fix has two parts.

A check before each screenshot. Each visual baseline opens its tab and lists the tabs whose count shows an admin override. If the list isn't empty, the test fails before its screenshot is compared or written, with Tabs where Base stores admin overrides, which no baseline shows, and the received value names the tab, e.g. ["System: 1 admin override"]. The thresholds are unchanged.

A Base lock held through the screenshot. A check alone only proves that Base was clean when the page loaded. Six spec files save to Base during a parallel live run, each holding the override for 1 to 5 s, and nothing kept one from saving between a baseline's check and its screenshot:

  • The page reads Base once per load. A save after the check therefore reaches the image only if the page fetches Base again before the screenshot. React Query does that on window focus once the data is 30 s old.
  • Without a refetch, the check and the screenshot read the same page load, so today a save after the check doesn't reach the image. That only holds while nothing makes the page fetch Base again in between. The experiments in Testing show both cases.

So the check and the screenshot now run under a lock that every journey saving to Base also holds:

  • e2e/live.ts: the cross-worker mkdir lock behind lockUserProfile now backs two named locks, the USER role profile and Base. lockBase(cleanup?) holds the Base lock through each test of the calling describe and its cleanup, which runs before the release, also after a failure. withBaseLock holds it around a hook.
  • Both locks keep the existing behavior. A lock whose owner has exited, or that is held past 5 min, is taken over. Waiting doesn't spend the test's own timeout, and a waiter gives up after 6 min.
  • Each of the nine visual baselines holds the lock from loading its tab until its screenshot is taken.
  • Every describe whose journeys save to Base holds it too:
Spec, describe Journeys that save to Base Stored, then reverted Reverted by
import.spec.ts, the base configuration imports a snippet into Base balance.startBalance, 2.1 s the lock's cleanup
specs.spec.ts, Model specs a new spec…; reordering is saved… modelSpecs.list, 3.0 s and 5.1 s the lock's cleanup (revertList), and a beforeAll under the lock
features.spec.ts, Features editors the memory agent…; an Actions allowlist entry…; a web search key…; the summarization trigger… memory.agent.* 2.3 s, actions.allowedDomains 1.2 s, summarization.trigger.* 1.0 s the journey
providers.spec.ts, the whole file the Anthropic model list never ends up empty endpoints.anthropic.models, 1.0 s the journey
saves.spec.ts, custom endpoints by identity editing one field…; an endpoint added in the panel…; renaming an endpoint whose key is stored… endpoints.custom, 2.5 s, 3.5 s and 2.8 s the journey
mcp.spec.ts, MCP servers create, rename with an edit, and delete a server mcpServers.<new server>, 4.2 s the journey

The times are from a MongoDB change stream on the configs collection during a one-worker live run. The web search journey fails before it saves on this LibreChat (see Not in this PR), so it has no time.

A few details:

  • The lock covers whole describes, the way lockUserProfile is used, so the read-only journeys in those describes hold it for a second or two as well.
  • Model specs also locks the USER role profile, in a nested describe, so it takes Base first. lockBase's doc comment asks for that order, so two workers can't each hold one lock while waiting for the other.
  • The visual baselines hold the USER role profile lock too, taken after the Base lock. The configuration a11y spec's role-profile test (♿ fix: Keep Names, Focus and Status Messages Right Across the Configuration Page and App Shell #207) can save Starting credits to the USER profile, and the System tab draws an "Overridden for USER" link while it is there, so a baseline must not be captured during that test either.
  • Every journey saving to Base holds the lock until it has reverted. So an override that a baseline finds was left by a run that failed before reverting, or saved by a test that doesn't take the lock. The check therefore fails at once instead of polling. The first revision's 30 s poll and 90 s describe timeout are gone. A wait for the lock extends the test's timeout by the time waited.

Why a lock rather than a Playwright project that runs after every other live test (project dependencies):

  • Playwright skips a dependent project when any test it depends on fails. A full live run has known failures (Not in this PR), so the baselines would never run in one.
  • Dependency projects ignore CLI filters, so -g 'visual baseline' or an --update-snapshots run of the baselines would first run every other live test.
  • The snapshot names carry the project name (-chromium-linux.png), so all nine images would have to move or get a custom snapshotPathTemplate.
  • The lock lives in the OS temp dir, keyed by the panel URL, so it also orders two Playwright processes run against the same stack.

The two 390 px baselines were stale, not regressed.

visual baseline: light-390 › system and › features fail on main on a clean database. To find out why, I measured every baseline at zero tolerance on a clean database, against a build of main and against a build of #170's merge commit, the last commit that wrote these images:

Baseline main vs baseline (px) Limit #170 vs the same baseline (px)
system-light-390 12,294 8,580 11,865
features-light-390 10,320 8,580 9,890
system-light-1440 20,407 31,680 19,821
system-dark-1440 20,526 31,680 19,994
features-light-1440 4,997 31,680 4,410
features-dark-1440 5,042 31,680 4,508
custom-light-1440 587 31,680 0
custom-dark-1440 534 31,680 0
custom-light-390 431 8,580 0

So the 390 px images already failed against the code they were committed with. #170 wrote them before its own last changes landed, and every difference in them is a change #170 itself made:

Between #170 and main, the 390 px images changed by only ~1,025 px:

#209 changes none of these images: the main build at #209 and at #206 render identically.

Both 390 px baselines are regenerated with --update-snapshots -g 'visual baseline: light-390 (system|features)$', against a main build on a database with no admin overrides. The new images match the first capture on a fresh instance pixel for pixel. Each shows Base with librechat.yaml values only: no override count, chip or revert action. They show:

  • system-light-390: the Registration section with the Allowed domains chip under its help text, the titled startup note, "OAuth state TTL" and the centered rail.
  • features-light-390: the Interface section's Chat controls with each librechat.yaml chip on its own line under the row's help text, and the centered rail.

After merging #207. #207 made every revert action's name start with its visible words ("Revert to librechat.yaml (1,000,000): Starting credits"). revertBaseStartBalance still looked for the old "Revert Starting credits to …" name, found no button, and took its "nothing to revert" path, so the Base import's Starting credits stayed in Base. The live run on the merged tree showed it: everything passed, but Base kept the override afterwards, and the next run's baselines would have failed on it. It now matches every revert name form (/^(Revert to .+|Remove override): Starting credits$/), as features.spec.ts does. On that leftover, the baselines failed first with "System: 1 admin override" and no screenshot taken. Then the fixed Base import test passed and reverted it (Base stored {}), and the baselines passed again.

On the merged tree: the full mock suite gave 127 passed, exit 0. Live, all eight Base-writing specs together at --workers=2 gave 146 passed, with the one known web search failure (and the 2 serial journeys after it not run). Base stored no overrides afterwards, and no lock directory was left.

Not in this PR

  • The six 1440 px baselines carry drift of the same kind and still pass. Among other changes, they predate ✅ fix: Wave 2 Integration, Verification Fixes and Schema Coverage Guarantees #170's note and labels, 🔭 feat: Preview as a Role, Group or User, Scope Catalog Pickers and Env Status from the Config Contract #178's Preview button and 🎯 fix: Center Collapsed Sidebar Icons and Unclip Sidebar Focus Rings #193's rail, which add up to ~20,500 px against a 31,680 px limit. I left them alone, so system at 1440 keeps about a third of its margin. Regenerating them the same way would restore it.
  • import.spec.ts still leaves a USER role profile's Starting credits, an ADMIN profile's welcome message and a new role behind on each live run. The USER one draws an "Overridden for USER" link into the 1440 px system images (~1,480 px, within the limit). The new check covers Base overrides only, not profile ones.
  • features.spec.ts, providers.spec.ts, saves.spec.ts and mcp.spec.ts revert inside the journey. A journey that fails between its save and its revert leaves the override in Base, the way specs.spec.ts did. The lock is still released, and the next baseline fails naming the tab. Moving those reverts into lock cleanups is a follow-up.
  • On this stack's LibreChat, features.spec.ts's "a web search key is masked…" fails before it saves. The server reads web search settings only from environment variable references, so the panel shows that note instead of "Will be stored encrypted when you save.". The serial describe then skips its last two journeys. It fails the same way with main's spec.
  • The lock isn't first come, first served. A baseline can wait behind several journeys: up to ~52 s in the two-process run below, well under the 6 min limit.
  • Ten axe tests in e2e/a11y/config.spec.ts (mcp at 1440 and 390, features at 1440, and the two change review dialogs, in both themes) fail live on main with label-content-name-mismatch. They fail the same way with main's copy of the spec, and nothing here touches them.

Change Type

  • Bug fix (non-breaking change which fixes an issue)

Testing

All live runs used E2E_LIVE=1 against a local LibreChat backend with its own database. The investigation started on a fresh instance (new database, new backend).

Live, e2e/specs.spec.ts

Code Run Result
main Failure injected right after Save in "a new spec…" That journey failed and 8 did not run. Base kept the spec list, with "E2E Spec" in it.
main Re-run without the injected failure Failed at toHaveValue('e2e-spec'): received "e2e-spec-2". 8 journeys did not run.
This branch Against that leftover from main 10 passed, 1 skipped. The beforeAll reverted the list.
This branch Failure injected right after Save in "a new spec…" That journey failed. Afterwards Base stored no spec list.
This branch Re-run without the injected failure, then once more back to back 10 passed, 1 skipped, both times. No spec list in Base after either.
This branch Failure injected in "reordering is saved…" after its first Save, with the review dialog open over the unsaved move That journey failed. Afterwards Base stored no spec list ("E2E Last" reverted).
This branch, with a USER role profile Twice back to back 11 passed, both times. No spec list in Base after either.

The skip is "a profile shows the specs it inherits…", whose documented precondition is a USER role profile. A same-page revert in an afterEach also recovered in the open-dialog case, because page.goto leaves the dirty page without its leave prompt. So the separate context is for the beforeAll, and so that one helper can serve both hooks.

Live, e2e/import.spec.ts: 6 passed. Afterwards Base stored no overrides.

Live, the Base override check in e2e/a11y/config.spec.ts

Base Run Result
No overrides All 9 visual baselines 7 passed and the two stale 390 px ones failed on pixels before regeneration. All 9 passed after.
Starting credits saved to Base light-1440 system and custom, light-390 custom All three failed in 1 to 2.3 s with Received: ["System: 1 admin override"]
No Base overrides, with the USER and ADMIN profiles import.spec.ts leaves All 9 visual baselines 9 passed

Live, the Base lock

These runs used two temporary helpers that aren't in the PR:

  • A copy of the visual test that compares system-light-390 at zero tolerance (it matches a clean Base pixel for pixel) and logs the override counts the page shows after its screenshot.
  • A "hammer" journey in another worker. It saves Base's Starting credits through the panel, holds them 3 s, reverts them and pauses 0.5 s, over and over.
Code Run Result
First revision (check, then screenshot) A save from a second context right after the clean check, 1.5 s before the screenshot. 3 runs. 3 exact matches. The page still showed no override after its screenshot: the check and the screenshot read the same page load.
First revision The same save, then a React Query refetch once the data was 30 s old (a visibilitychange event) 830 px changed: the System tab's override count. That's under the 8,580 px limit, so a normal run passes and --update-snapshots writes it.
First revision The hammer, and the refetch 34 s after the check. 3 runs. 3 of 3 drew the override, 830 px each
This branch The hammer holding the Base lock, and the same refetch. 3 runs. 3 exact matches. The hammer saved only between baselines.
First revision The hammer, no refetch. 8 runs. 8 exact matches. The poll waited up to 23 s of its 30 s for a clean Base.
This branch The hammer holding the lock, no refetch. 8 runs. 8 exact matches. Each baseline waited ~7 s for the hammer's save in progress to be reverted, then checked in one load.
This branch Two Playwright processes against one panel, with each lock acquisition and release logged: the six spec files that save to Base (2 workers), and the nine baselines 4 times (2 workers) 65 Base lock holds (36 baselines, 29 journeys and hooks), no two overlapping. 25 saves to Base, 18 of them while the baselines ran, none inside a baseline's hold and none outside a journey's. 36 of 36 baselines passed.
This branch A failure injected 5 s after the import in import's Base journey, next to the nine baselines (2 workers) The journey failed and its cleanup reverted Starting credits before the release. The baseline waiting for the lock passed after 16 s.
This branch A Base lock left in the temp dir by a process that no longer runs Taken over at once

Live, every changed spec together (a11y/config, import, specs, features, providers, saves, mcp, config-shell), with 2 workers and again with 4: 109 passed each time. 11 failed: the 10 axe label-content-name-mismatch tests listed above, as they do with main's spec, and the web search journey. 2 did not run (the features describe's last two). All 9 visual baselines passed. Afterwards Base stored no overrides, and no lock was left in the temp dir.

Mock backend, full suite: exit 0, with 114 passed and 148 skipped (the live-only specs). The new hooks never run there, because every describe that calls them is live-only.

Test Configuration:

  • npx tsc --noEmit, npx eslint src/ --max-warnings 0, Prettier on the changed files and bun run test:lint-rules (19 passed) are clean. The ESLint config does not cover e2e/.
  • No src/ changes, so vitest and the build are unaffected. No baseline image changed beyond the two above.

Checklist

  • My code adheres to this project's style guidelines
  • I have performed a self-review of my own code
  • I have commented in any complex areas of my code
  • My changes do not introduce new warnings
  • I have written tests demonstrating that my changes are effective or that my feature works

…ides Out of the Visual Baselines and Refresh the Two Stale 390 px Ones

e2e/specs.spec.ts reverted the Base spec list only as the last statement of
its two saving journeys. When an assertion after Save failed, "E2E Spec" or
"E2E Last" stayed in Base, the next run's new spec was named e2e-spec-2, and
the serial describe skipped every journey after it. revertList now runs in
a beforeAll, for a list an earlier run left behind, and in an afterEach,
after every journey, passed or failed. It opens a page in a context of its
own, so whatever a journey left on its page (a review dialog, unsaved edits)
cannot get in the way, returns early when Base stores no list, and checks
that the revert took.

e2e/import.spec.ts left the Starting credits it imports into Base behind
after every live run. An afterEach now reverts them.

Every Base override shows in the configuration visual baselines: its tab's
override count, the header's revert action at 1440 px, and its chip and
section count on its own tab. Before its screenshot, each baseline now
reloads until Base stores no admin overrides, since another spec's journey
holds one for a few seconds, and fails naming the tabs that still have one.
A leftover therefore never reaches a comparison or an --update-snapshots
run.

The 390 px system and features baselines failed on main on a clean
database. #170 wrote them before its own last changes: at phone width a
row's source chip moved under its label, the startup note gained a title
and new wording, and "OAuth state TTL (ms)" became "OAuth state TTL". #193
then moved the collapsed rail's icons 1.5 px to its center. Both baselines
are regenerated from main on a clean database. No threshold changed.
@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 1. Head: 8f95ba3

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T11:22:26.746194Z ff7004f Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f95ba34af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread e2e/a11y/config.spec.ts Resolved
…gh Its Screenshot and Around Every Live Base Write
@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 2. Head: 76c47ac

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 76c47ac4ed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# Conflicts:
#	e2e/a11y/config.spec.ts
#	e2e/config-shell.spec.ts
@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 3. Head: 1cb8bcc

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 1cb8bcc4f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 4. Head: ff7004f

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: ff7004f5bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit 6f02cf7 into main Oct 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant