Skip to content

⏱️ fix: Stop the Event-Loop Delay Monitor From Waking the Idle Server 100 Times a Second - #199

Merged
danny-avila merged 2 commits into
mainfrom
pr/redo-idle-eventloop-monitor-wakeups
Oct 28, 2026
Merged

danny-avila merged 2 commits into
mainfrom
pr/redo-idle-eventloop-monitor-wakeups

Conversation

@danny-avila

Copy link
Copy Markdown
Contributor

Pull Request Template

⚠️ Before Submitting a PR, Please Review:

  • Please ensure that you have thoroughly read and understood the Contributing Docs before submitting your Pull Request.

⚠️ Documentation Updates Notice:

  • Kindly note that documentation updates are managed in this repository: librechat.ai

Summary

With no traffic, bun server.ts woke its main thread about 100 times a second. The cause was client.collectDefaultMetrics() in src/server/metrics.ts. prom-client 15 defaults eventLoopMonitoringPrecision to 10 ms and passes it as the resolution of a perf_hooks.monitorEventLoopDelay histogram. prom-client skips that histogram only when the runtime reports it as not implemented, and Bun 1.4.0 and 1.4.3 implement it. So the server ran a 10 ms timer on its main thread for as long as it was up.

The panel now passes eventLoopMonitoringPrecision: 1000.

  • The histogram still feeds nodejs_eventloop_lag_{min,max,mean,stddev,p50,p90,p99}_seconds, now from one sample a second instead of 100. nodejs_eventloop_lag_seconds is measured when Prometheus scrapes and doesn't change.
  • ADMIN_PANEL_METRICS_EVENT_LOOP_PRECISION_MS overrides the interval for deployments that want the finer percentiles back. It is documented in .env.example.
  • A value that isn't a whole number from 1 to 2147483647 falls back to 1000. monitorEventLoopDelay throws for any other value (2147483647 is Bun's ceiling, lower than Node's), and prom-client rethrows everything except "not implemented", so a bad value would otherwise stop the server at startup.
  • The call is now a small exported collectDefaultMetrics in metrics.ts, still run when the module loads, so the tests can run it against a fresh registry.

About #97. This removes the server's only steady source of idle wakeups, but this PR refers to #97 instead of closing it. The issue reports about 8% of a core and an active timerfd re-arming every second. Neither reproduced here:

  • Before this change the production build idled at 0.25 to 0.6% of a core on this host.
  • It had no timerfd open at all on Bun 1.4.0 or 1.4.3, only one epoll fd and one eventfd.
  • The /proc/<pid>/statm descriptor the issue points to is Bun's own. Bun 1.4.0 keeps it open for a bare Bun.serve with no app code.
  • The server code has no other timers. Its only setTimeout is a short wait before a save is read back, inside the save request. A 10-second setInterval like the memory tick proposed in 🪦 feat: Request Tombstones, Memory Watermarks & Server Error Logging #102 would add about 0.1 wakeups a second, which doesn't show up in these counts.

If the reporter's container still idles near 8% with this change, the one-second timer comes from something this host doesn't have.

Refs #97

Change Type

  • Bug fix (non-breaking change which fixes an issue)

Testing

Unit tests in src/server/metrics.test.ts run prom-client's real collectDefaultMetrics against a fresh registry. A spy on perf_hooks.monitorEventLoopDelay calls through, records the resolution and then disables the histogram. The tests check that:

  • by default the histogram resolution is 1000 ms;
  • ADMIN_PANEL_METRICS_EVENT_LOOP_PRECISION_MS=250 gives 250;
  • "", 0, -5, 1.5, abc and 2147483648 fall back to 1000;
  • after the module loads, the shared registry that /metrics serves has nodejs_eventloop_lag_p99_seconds.

With prom-client's default restored, every precision test fails because the monitor gets 10. Without the upper bound, the 2147483648 case fails. The registry test fails if the module-level call is dropped.

Idle measurement: a production build (bun run build) served with bun server.ts and NODE_ENV=production, with no traffic after one readiness request. Measured 60 seconds after start over three 20-second windows, counting the main thread's voluntary context switches from /proc/<pid>/status:

Bun main this PR
1.4.0 99 to 110 wakeups/s, 0.4 to 0.55% of a core 1 wakeup/s, 0 to 0.1%
1.4.3 (the Docker image's version) 99 to 100 wakeups/s, 0.25 to 0.3% 1 wakeup/s, 0 to 0.05%

On this PR's build, the first window after start or after a few requests can show 6 to 13 wakeups/s while Bun's worker threads wind down. Counts across all threads aren't in the table because those threads start and exit during a window.

Live checks on this PR's build:

  • With ADMIN_PANEL_METRICS_EVENT_LOOP_PRECISION_MS=10 it goes back to 99 to 111 wakeups/s, so the variable reaches the monitor under Bun.
  • With 2147483648 the server starts and serves the login page. prom-client on its own, given that value, throws at startup under Bun.
  • /metrics with the bearer token still serves all seven histogram gauges, nodejs_eventloop_lag_seconds, the process metrics and the admin_http_* counters. Without a token it answers 401.
  • Against a LibreChat backend: signed in and opened Home, Configuration, Access and Grants, with no console errors or failed requests.

Playwright e2e wasn't run because it serves the panel with vite dev, which never loads server.ts or src/server/metrics.ts.

Test Configuration:

  • Bun 1.4.0 and 1.4.3, Linux (WSL2).
  • LibreChat dev, MongoDB replica set, for the click-through.
  • npx tsc --noEmit, npx eslint src/ e2e/ --max-warnings 0, npx prettier --check on the changed files, bun run test:lint-rules, npx vitest run (3347 tests), bun run build.

Checklist

  • My code adheres to this project's style guidelines
  • I have performed a self-review of my own code
  • I have commented in any complex areas of my code
  • I have made pertinent documentation changes
  • My changes do not introduce new warnings
  • I have written tests demonstrating that my changes are effective or that my feature works
  • Local unit tests pass with my changes

… 100 Times a Second

collectDefaultMetrics() ran with prom-client's default
eventLoopMonitoringPrecision of 10 ms. That is a monitorEventLoopDelay
histogram sampling every 10 ms, so Bun's main thread woke about 100 times a
second with no traffic. The panel now samples once a second, which keeps the
nodejs_eventloop_lag_{min,max,mean,stddev,p50,p90,p99}_seconds metrics.

ADMIN_PANEL_METRICS_EVENT_LOOP_PRECISION_MS overrides the interval and is
documented in .env.example. A value that isn't a whole number from 1 to
2147483647 falls back to 1000: monitorEventLoopDelay throws for anything else
(Bun's ceiling is 2147483647), and prom-client would rethrow it at startup.

Idle main-thread wakeups of `bun server.ts` on a production build went from
about 100 a second to 1 a second on Bun 1.4.0 and 1.4.3.

Refs #97
@danny-avila

Copy link
Copy Markdown
Contributor Author

@codex review

Pass 1. Head: 0ea3703

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T10:25:56.177776Z 0ea3703 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 0ea37036d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# Conflicts:
#	src/server/metrics.test.ts
@danny-avila
danny-avila merged commit 7d56516 into main Oct 28, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant