[Quinntyx]

Problem

Server browser tabs streamed to a remote viewer are unusable over a slower link. On the link from araveia to syssec-canis (a Tailscale DERP relay, about 21 Mbit/s with a 17–40 ms RTT median and frequent spikes past 500 ms), a keypress took 0.6–2 s to appear, a scroll trailed the wheel by 2–3 s, and animation reached about 3 fps.

Three things caused it:

• Every frame was a 150–350 KB JPEG.
• Only one frame could be in flight per round trip.
• Each queued wheel waited for the software-rendered page in turn, so a scroll kept replaying long after it stopped.

Change

I took the parts of the upstream discussion that help this link and left out the parts that don't (decisions below):

• Video for viewers that can decode it. The client checks VideoDecoder.isConfigSupported and sends codecs=avc1.42E034,vp8 on the stream URL.
  • The server re-encodes its JPEG screencast as H.264 (or VP8) with WebCodecs, in a scratch page of the Chromium it already runs. No ffmpeg and no new host dependency.
  • A frame is encoded only when it is about to be sent. Frames superseded on a slow link are never encoded, so the P-frame chain never needs dropping.
  • Once the page settles (400 ms quiet), the existing full-quality JPEG still follows, so text ends sharp.
  • A delta frame is about 10–20 KB, where the JPEG it replaces was about 260 KB.
• A bounded in-flight window instead of one frame per round trip. Up to 6 frames, or 128 KiB, may be unacknowledged.
  • A frame larger than the budget still travels alone, so fix(server): browser control replies no longer wait behind old frames on slow links pingdotgg/t3code#20290's guarantee holds: control replies never queue behind images.
  • Chromium is released once a video frame is encoded; the window, not the viewer's ack, bounds the link.
• Merging viewer wheels and pointer moves that wait behind a slow page. While one is dispatching, later wheels sum into the waiting one, and later moves replace it. Any other input (keys, clicks) keeps its place in order.
• A GPU and render scale per host, in Settings. See the section below.
• Skipping unchanged frames. A screencast frame byte-identical to the previous one is acked and dropped. On a GPU, the still's screenshot repaints the page, and Chromium sent the same picture again. That counted as motion, so another still followed 400 ms later, forever (about 7 Mbit/s on an idle page).
• Keyframe recovery. A painter whose decoder fails asks for a keyframe and ignores deltas until it arrives. Read-only viewers can ask too.

Compatibility goes both ways:

• A JPEG starts with 0xFF, and video messages start with 1 (keyframe, carrying codec and size) or 2 (delta). The format lives in packages/contracts/src/preview.ts.
• Old clients send no codecs and get the same JPEG stream as before. New clients on old servers just receive JPEG.
• Clients without WebCodecs keep JPEG: insecure contexts, older WebViews, possibly some iOS WebViews.
• If the encoder fails, for example above the codec's size limit, that viewer falls back to JPEG.

The web panel, web picture-in-picture and the mobile WebView all share the client-runtime stream client and painter, so they all get this. Desktop-rendered tabs and agent recordings don't use this path and are unchanged.

Host GPU and render scale

Settings → Browser → Rendering has two new per-environment rows. Mobile has them on the Threads settings page, when no project is selected.

• Browser GPU. Software is the default and today's --disable-gpu. Automatic enables the GPU (on Linux, ANGLE over EGL, without which headless Chromium stays on SwiftShader). A specific GPU is offered on Linux hosts.
  • The host lists GPUs through a new read-scoped preview.listGpus RPC. It reads /dev/dri/by-path/*-render, sysfs vendor/device/driver and pci.ids names.
  • A GPU is stored by its PCI path, which survives render nodes renumbering across boots. It is passed as --render-node-override.
  • A saved GPU that is gone, or an id that doesn't match the PCI-path format, falls back to software.
• Browser render scale. 1×, 1.5×, 2× (default, today's behavior) or 3×. It drives --force-device-scale-factor, the context device scale, zoom emulation and snapshot capture scale.

Both are launch flags. A change relaunches that host's Chromium:

• Server-run tabs tell their viewers to reconnect (close 1012) instead of going away.
• Each tab reopens at its page on first use. Tabs reopen only after the old browsers have closed, so a persistent profile is never opened twice.
• Desktop-rendered tabs are untouched.

docs/user/remote-access.md says where to find the settings.

Approaches considered and dropped

• DOM streaming, or tunneling the page on Take control. Either gives the viewer a different page from the one the agent drives (layout, storage, logins, recordings). Desktop already has a manual version with Open on this computer (fix(web): desktop opens remote environments' browser tabs locally pingdotgg/t3code#17316).
• Turning on the GPU or 1× rendering for every host (upstream [Bug]: Server preview browser is laggy: always launched with --disable-gpu and device scale 2 pingdotgg/t3code#26523). Software at 2× stays the default, and a host opts in from Settings.
• WebRTC tab capture ([Bug]: Server preview browser is laggy: always launched with --disable-gpu and device scale 2 pingdotgg/t3code#26523, fix(preview): server recordings capture the tab instead of re-encoding the JPEG screencast pingdotgg/t3code#26684). It needs full Chrome instead of the headless shell, plus UDP reachability, which T3 Connect's tunnel doesn't give. WebCodecs over the existing WebSocket works through every connection mode.
• Disabling smooth scrolling. Measured with no effect (283 → 275 ms over loopback).

Scope and approval

This is a performance fix for our fork, at the request of the fork's owner. Upstream would need prior approval, since it changes the default stream format and the frame-rate/bandwidth tradeoff (as pingdotgg#19462 was told).

Verification

Benchmark, araveia → syssec-canis over the real relay.

Setup:

• The bench server runs this branch's (or main's) real ServerBrowser and /api/preview-stream route, with real Chromium; only auth and the desktop channel are stubbed. Tailscale serves it over HTTPS on the same relay path the desktop uses.
• The viewer runs on syssec-canis in headless Chromium, with the real client-runtime stream client and painter bundled from each checkout.
• The test page draws three counters as black/white squares: key/click count, scroll position and animation frame. The viewer decodes them from the painted canvas, so latency is measured from send until the result is on screen, all on one clock.
• Panel: 1100×1000 CSS px at 1.5× scaling (cap 1650×1500), matching syssec-canis's 4K display at 150%.
• Rounds alternated baseline/branch, with relay RTT sampled before each (p50 17–40 ms). Each figure is the median of the per-round values over 3 rounds.

Phase	Latency p50 (ms)	Latency p90 (ms)	Frame rate shown	Avg message	Link use
Keypress every 300 ms	591 → 116	2033 → 173		278 → 36 KB	6.2 → 1.0 Mbit/s
Typing, key every 60 ms	435 → 96	578 → 174		271 → 27 KB	8.2 → 2.5 Mbit/s
Scroll, 60 px wheel every 16 ms	1896 → 356	2769 → 692	7–12 → 17–21 fps	81 → 68 KB	4.4 → 8.1 Mbit/s
Animation			3.3 → 30.4 fps	167 → 14 KB	7.9 → 2.7 Mbit/s

• Congested relay. In an earlier set of 3 rounds during relay congestion (ping 45–893 ms), the branch still led on every median: keypress 415 → 136 ms, scroll 1504 → 394 ms, animation 3.1 → 30.4 fps. Tail spikes hit both sides then; one branch round had a typing-burst p90 of 2 s.
• Loopback on araveia. This shows the server floor. Keypress p50 is 55 ms on main and 66 ms here, since encoding costs about 11 ms of server time per frame. Scroll is 1482 → 262 ms and animation 14 → 28 fps.
• Server cost. Encoding raises server CPU for this one viewer from about 1.7 to 3 cores during animation (24-core host).

GPU and render scale, araveia → syssec-canis. The relay p50 was 15–28 ms. Each cell gives the range over 2 alternating rounds; everything is on this branch:

Phase	Software 2× (default)	RTX 3060 2×	RTX 3060 1×
Keypress p50	104–153 ms	70–90 ms	65–84 ms
Scroll p50 / p90	262–365 / 338–662 ms	180–292 / 269–674 ms	134–194 / 167–369 ms
Scroll frame rate	17–27 fps	19–31 fps	34–41 fps
Animation	23–31 fps	37–38 fps	35 fps
Server CPU, animation	2.9–3.1 cores	2.2–2.3 cores	1.2–1.3 cores

• Loopback. With the RTX 3060 at 2×, scroll p50 is 159 ms against 262 ms on software and keypress 50 ms against 66 ms.
• Live relaunch. Changing the setting on a running bench server closed the open viewer with 1012 within 26 ms. The viewer reconnected and got the page back. Chromium was running with --render-node-override=/dev/dri/by-path/pci-0000:2b:00.0-render --force-device-scale-factor=1, and WebGL reported the NVIDIA renderer.

Tests. These fail on main and pass here:

• ServerBrowserStream.test.ts: small frames share the window, frames that wait are encoded only when sent, and the existing large-frame case still holds. Key frame requests and the codecs parameter are covered too.
• ServerBrowser.test.ts: wheels and moves waiting behind a slow page merge into one dispatch, and a key keeps its place.
• preview.test.ts: the frame header round-trips, and JPEG is told apart from video.
• ServerBrowser.test.ts: a rendering change relaunches the browser and the tab reopens at its last URL (fails without the re-navigation). An identical screencast frame is skipped (fails without the skip).
• ServerBrowserContexts.test.ts: relaunch closes every browser, and the next launch uses the current flags and scale.
• ServerBrowserGpus.test.ts: discovery against a fake sysfs/devfs/pci.ids tree, plus the flag mapping, including a missing GPU and an injected flag.

vp run check:changed passes.

Not checked:

• The packaged desktop app and the mobile app were not run against this build. The client path was tested in Arch's system Chromium through the real client-runtime code. Electron and Android WebView are Chromium-based too.
• Whether iOS WKWebView exposes VideoDecoder here is unverified. Without it, iOS stays on JPEG.
• The new Settings rows (web and mobile) were not rendered in a running app; they typecheck and follow the existing row patterns.
• GPU discovery was tested only on Linux with NVIDIA. Automatic on macOS and Windows is unverified.
• No before/after screen recording.

Model: Claude Opus 5.5. Harness: Claude Code through T3 Code.

🤖 Generated with Claude Code