Skip to content

webrtc-sys: Resolve memory leak for server-initiated room closure - #1473

Merged
alan-george-lk merged 4 commits into
mainfrom
alan/bugfix-remote-room-closure
Oct 7, 2026
Merged

alan-george-lk merged 4 commits into
mainfrom
alan/bugfix-remote-room-closure

Conversation

@alan-george-lk

@alan-george-lk alan-george-lk commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Joint Summary

Stacked/works in junction with #1500. Relationship:

These in combination fix a fairly hefty memory leak reported by a customer. Their use case is remotely killing a room via server API, which kills the client room locally, and joining another room in the same client process.

This PR Summary

Previously, unpublish_track called remove_track(sender)? first. When a remote room deletion had already closed the connection, that call errored and returned early, leaving the publication → track → transceiver graph attached, which could retain the room session.

The patch:

  • Saves the remove_track result instead of returning immediately.
  • Always detaches the transceiver and clears the publication’s track reference.
  • Requests renegotiation only when sender removal succeeded.
  • Returns the original removal error only after cleanup.

The added reconnection test publishes a local video track, deletes the room remotely, then verifies both that the publication loses its track and that the room session can actually drop. That directly demonstrates the retained-reference lifecycle issue is resolved.

Testing/Validation

A standalone C++ app was created to reproduce the behavior that runs relevant code in cycles (described below). The SDK is initialized once for the entire run with MALLOC_ARENA_MAX=1 set to tighten memory measurements. The environment was a Linux desktop with a 5090.

For each of the 100 cycles:

  1. Create a unique room using lk --dev room create.
  2. Mint a fresh participant token for that room.
  3. Connect one participant.
  4. Publish three 1280×720 H.264 tracks, explicitly requiring NVENC.
  5. Feed synthetic I420 frames to each track at 30 FPS for one second.
  6. Confirm every track encoded and sent data using NVIDIA H264 Encoder.
  7. Remotely delete the occupied room using lk --dev room delete <room name>.
  8. Confirm both the RoomDeleted disconnect reason and room EOS event.
  9. Destroy/release the room, participant, tracks, sources, and encoders.
  10. Sample idle RSS and thread count before starting the next cycle.

Note: The first 20 cycles are treated as warmup, growth is measured from cycle 20 through cycle 100.

Results Pre-PRs

Fresh clean build and 100-cycle run completed on Rust main commit 535ead0d.

Cycle Idle RSS Delta from cycle 20 Threads Thread delta from cycle 20
1 223.207 MiB — 45 —
10 251.770 MiB — 153 —
20 288.160 MiB 0 273 0
50 388.801 MiB +100.641 MiB 633 +360
80 501.766 MiB +213.606 MiB 993 +720
90 526.793 MiB +238.633 MiB 1,113 +840
100 563.527 MiB +275.367 MiB 1,233 +960
Result Value
Completed cycles 100/100
NVENC tracks 300/300
RoomDeleted + EOS 100/100
RSS growth, cycles 20–100 +275.367 MiB
Average RSS growth +3.442 MiB/cycle
Thread growth, cycles 20–100 +960
Average thread growth +12 threads/cycle
Test failures 0
Remaining rooms 0

Results Post-PRs

Note: this combines this branch + #1500.

Cycle Idle RSS Delta from cycle 20 Threads Thread delta from cycle 20
1 157.223 MiB — 29 —
10 163.199 MiB — 29 —
20 163.215 MiB 0 29 0
50 163.402 MiB +0.187 MiB 29 0
80 163.402 MiB +0.187 MiB 29 0
90 163.402 MiB +0.187 MiB 29 0
100 163.402 MiB +0.187 MiB 29 0
Result Value
Completed cycles 100/100
NVENC tracks 300/300
RoomDeleted + EOS 100/100
RSS growth, cycles 20–100 +0.188 MiB
Average RSS growth ~0.002 MiB/cycle
Thread growth, cycles 20–100 0
Average thread growth 0 threads/cycle
Test failures 0
Remaining rooms 0

Customer Validation

I cut a specific C++ SDK branch build against this Rust fix and the customer confirmed this resolved the problem.

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Changeset ✓

This PR includes a changeset covering all affected packages:

Package Bump
livekit patch
livekit-capture patch
livekit-ffi patch

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Failed unpublish sends contradictory binding events

When sender removal fails, unpublish_track now emits LocalTrackUnpublished but returns an error. The FFI unpublish request forwards that event, then reports failure to its caller.

(Refers to this code)

Learn more

The room emits LocalTrackUnpublished from this callback. The FFI event handler forwards it and removes the publication lookup entry. Meanwhile, the FFI request treats unpublish_track's returned error as a failed operation and sends an error callback. A closed peer connection triggers this mismatch even though the local track has been detached. Clients can receive a publication-removed event alongside a failed unpublish request.

Example: A participant unpublishes camera TR_1 while a server deletion closes its connection. Sender removal fails, so the SDK detaches TR_1 and sends LocalTrackUnpublished; the FFI request then reports an error for the same unpublish.

Recommended fix: Decide how the API represents completed local cleanup after sender-removal failure. Coordinate the unpublish_track return value with FFI's callback and event forwarding so the same operation has one consistent outcome; preserve the underlying removal error through logging or a distinct diagnostic if it cannot be returned as the operation result.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@lukasIO lukasIO left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@xianshijing-lk xianshijing-lk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with one nit.

Comment thread livekit/src/room/participant/local_participant.rs Outdated
@alan-george-lk
alan-george-lk force-pushed the alan/bugfix-remote-room-closure branch from efcb517 to 10f4813 Compare October 6, 2026 21:44
@alan-george-lk alan-george-lk changed the title Resolve memory leak for server-initiated room closure webrtc-sys: Resolve memory leak for server-initiated room closure Oct 6, 2026
@alan-george-lk
alan-george-lk force-pushed the alan/bugfix-remote-room-closure branch from 10f4813 to d4b5c40 Compare October 7, 2026 00:25
@alan-george-lk
alan-george-lk merged commit 7dd7903 into main Oct 7, 2026
33 checks passed
@alan-george-lk
alan-george-lk deleted the alan/bugfix-remote-room-closure branch October 7, 2026 01:36
alan-george-lk added a commit that referenced this pull request Oct 7, 2026
## Joint Summary

Stacked/works in junction with #1473. Relationship:

- #1473: releases local publications and tracks after remote room
deletion, allowing their encoders to be destroyed
- #1500: terminates VAAPI displays at the correct lifecycle boundaries,
releasing the associated driver resources

See #1473 for full write up and test results.

## This PR Summary

- Make the platform VAAPI display wrappers own the complete display
lifecycle: acquire and initialize once, then call `vaTerminate` during
`Close`/destruction.
- Keep capability-probe displays valid while they are queried and
terminate them when the probe's display owner goes out of scope.
- Keep encoder displays initialized for the full encoder lifetime, then
release VA buffers, context, surfaces, and configuration before
terminating the display.
- Make encoder teardown idempotent and safe after partial initialization
or reinitialization.
- Propagate VAAPI encoder initialization failures instead of continuing
with an unusable encoder.
- Replace manual entrypoint arrays with scoped storage and improve VAAPI
initialization diagnostics.

This resolves the premature-display-termination issue identified in
review while retaining cleanup of the VA driver resources and threads
created by capability detection.

## Validation

Tested on Linux with an AMD VAAPI device and an RTX 5090:

- Release C++ SDK/test build succeeds with LLVM 21.
- Forced-VAAPI `local_video` test-pattern publishing continuously
encoded and sent 329/329 frames in 12 seconds using `VAAPI H264
Encoder`. The previous implementation stalled after its first frame.
- See #1473 for full 100 cycle test results.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants