Skip to content

perf: Stop holding the global mutex during scope access - #2091

Merged
jpnurmi merged 4 commits into
jpnurmi/perf/rwscopefrom
jpnurmi/perf/global-scope-mutex
Sep 20, 2026
Merged

jpnurmi merged 4 commits into
jpnurmi/perf/rwscopefrom
jpnurmi/perf/global-scope-mutex

Conversation

@jpnurmi

@jpnurmi jpnurmi commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Make sentry-native multi-threading friendly by allowing concurrent scope data readers.

The global scope data mutex became a serious bottleneck for multi-threaded logs and metrics. Most notably, when wired up with UE_LOG that may capture Unreal Engine logs from 100+ threads.

The previous step (#2042) enabled RW locks on the scope data. This final step removes the remaining, now superfluous global mutex around the data. With this, multiple logger threads can enrich incoming log items with scope data/attributes in parallel.

In theory, other concurrent scope operations may benefit too, but logs and metrics are the most important high-volume scenario.

Before

tests/benchmark.py::test_benchmark_logs[1] PASSED
Min 0.007ms, Max 0.007ms, Mean 0.007ms, Median 0.007ms, CPU 0.007ms
tests/benchmark.py::test_benchmark_logs[8] PASSED
Min 0.037ms, Max 0.037ms, Mean 0.037ms, Median 0.037ms, CPU 0.012ms
tests/benchmark.py::test_benchmark_logs[16] PASSED
Min 0.067ms, Max 0.067ms, Mean 0.067ms, Median 0.067ms, CPU 0.008ms
tests/benchmark.py::test_benchmark_logs[32] PASSED
Min 0.137ms, Max 0.137ms, Mean 0.137ms, Median 0.137ms, CPU 0.008ms
tests/benchmark.py::test_benchmark_metrics[1] PASSED
Min 0.004ms, Max 0.004ms, Mean 0.004ms, Median 0.004ms, CPU 0.003ms
tests/benchmark.py::test_benchmark_metrics[8] PASSED
Min 0.033ms, Max 0.033ms, Mean 0.033ms, Median 0.033ms, CPU 0.007ms
tests/benchmark.py::test_benchmark_metrics[16] PASSED
Min 0.059ms, Max 0.059ms, Mean 0.059ms, Median 0.059ms, CPU 0.007ms
tests/benchmark.py::test_benchmark_metrics[32] PASSED
Min 0.126ms, Max 0.126ms, Mean 0.126ms, Median 0.126ms, CPU 0.007ms

After

tests/benchmark.py::test_benchmark_logs[1] PASSED
Min 0.005ms, Max 0.005ms, Mean 0.005ms, Median 0.005ms, CPU 0.004ms
tests/benchmark.py::test_benchmark_logs[8] PASSED
Min 0.013ms, Max 0.013ms, Mean 0.013ms, Median 0.013ms, CPU 0.012ms
tests/benchmark.py::test_benchmark_logs[16] PASSED
Min 0.014ms, Max 0.014ms, Mean 0.014ms, Median 0.014ms, CPU 0.012ms
tests/benchmark.py::test_benchmark_logs[32] PASSED
Min 0.016ms, Max 0.016ms, Mean 0.016ms, Median 0.016ms, CPU 0.008ms
tests/benchmark.py::test_benchmark_metrics[1] PASSED
Min 0.005ms, Max 0.005ms, Mean 0.005ms, Median 0.005ms, CPU 0.005ms
tests/benchmark.py::test_benchmark_metrics[8] PASSED
Min 0.011ms, Max 0.011ms, Mean 0.011ms, Median 0.011ms, CPU 0.010ms
tests/benchmark.py::test_benchmark_metrics[16] PASSED
Min 0.017ms, Max 0.017ms, Mean 0.017ms, Median 0.017ms, CPU 0.015ms
tests/benchmark.py::test_benchmark_metrics[32] PASSED
Min 0.034ms, Max 0.034ms, Mean 0.034ms, Median 0.034ms, CPU 0.020ms

Close: #1862

@jpnurmi jpnurmi changed the title perf: Stop holding the global mutex during scope access [WIP] perf: Stop holding the global mutex during scope access Sep 14, 2026
@jpnurmi
jpnurmi added this pull request to stack #2092 September 14, 2026 07:07
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from e3facb1 to 5a1d374 Compare September 14, 2026 07:08

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/sentry_scope.c Outdated
Comment thread src/sentry_scope.c
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.44444% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.21%. Comparing base (233ea8b) to head (34af03d).

Additional details and impacted files
@@                   Coverage Diff                    @@
##           jpnurmi/perf/rwscope    #2091      +/-   ##
========================================================
+ Coverage                 74.47%   75.21%   +0.73%     
========================================================
  Files                       100      103       +3     
  Lines                     27403    27930     +527     
  Branches                   4968     5094     +126     
========================================================
+ Hits                      20408    21007     +599     
+ Misses                     5724     5574     -150     
- Partials                   1271     1349      +78     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from 5a1d374 to 58d161c Compare September 14, 2026 08:57
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from 58d161c to 47cf101 Compare September 14, 2026 10:14
@jpnurmi jpnurmi changed the title [WIP] perf: Stop holding the global mutex during scope access perf: Stop holding the global mutex during scope access Sep 14, 2026
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from 47cf101 to 816f367 Compare September 15, 2026 06:45

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/sentry_scope.c Outdated

@limbonaut limbonaut left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caught a race: two threads can update the same tag, for example, but their notifications can arrive in the opposite order. An observer can therefore end up remembering an older value instead of the value actually stored in the scope. See comments below for more details.

Comment thread src/sentry_scope.c
Comment thread src/sentry_scope.c
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from 3475564 to 3b8961a Compare September 16, 2026 13:52

@limbonaut limbonaut left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet! Benchmarks with the recent changes:

Median benchmark-reported real time, in milliseconds.

Benchmark Threads Master PR Speedup
Logs 1 0.00332 0.00348 0.96×
Logs 8 0.01571 0.00619 2.54×
Logs 16 0.04656 0.00917 5.08×
Logs 32 0.14999 0.02016 7.44×
Metrics 1 0.00266 0.00279 0.95×
Metrics 8 0.02256 0.00552 4.09×
Metrics 16 0.05678 0.00922 6.16×
Metrics 32 0.13757 0.01945 7.07×

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread tests/benchmark.py
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from 031883a to ea0c4fb Compare September 17, 2026 10:37

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ea0c4fb. Configure here.

Comment thread src/sentry_scope.h
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch 2 times, most recently from bd0d0a7 to c5493f6 Compare September 18, 2026 14:47
Comment thread src/sentry_scope.c
Comment thread src/sentry_scope.c
The global scope mutex still serializes readers despite the per-scope
read-write locks, limiting concurrent log and metric capture.

Retain the scope during access instead of holding the global mutex.
Coordinate cleanup with outstanding references and serialize observer
callbacks with a separate mutex. Keep the global mutex for scope
initialization and lifetime coordination.
Block new scope callers while existing references drain, and skip stale
cleanup after concurrent initialization. Allow reentrant access to finish
without accessing thread-local state from signal handlers.
Hold the observer lock across each observed scope mutation and its
notification so callbacks cannot receive an older update after a newer one.
@jpnurmi
jpnurmi force-pushed the jpnurmi/perf/global-scope-mutex branch from c5493f6 to 34af03d Compare September 19, 2026 18:53
@jpnurmi
jpnurmi merged commit 6b0d5f0 into master Sep 20, 2026
65 checks passed
@jpnurmi
jpnurmi deleted the jpnurmi/perf/global-scope-mutex branch September 20, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Logs: optimize performance

3 participants