SiteBench

A/B performance tests of a web page across git revisions, with a TUI and a scriptable CLI.

SiteBench checks out two revisions of a repository, installs and serves each one itself, and loads the page in a pinned Chrome build. A and B runs are interleaved and CPU throttling is on by default (4x). Results are saved as JSON (with a Markdown export) in a results directory.

How results are reported

• Time and CPU metrics are relative only. They depend on the machine and on background activity, so the report shows only B/A median ratios with a 95% bootstrap interval. The verdict is "better" or "worse" only when the interval is entirely on one side of 1.00×, and only with 5 or more runs per side.
• Memory and element counts are absolute. These are JS heap (measured after a forced garbage collection), renderer memory (RSS, Linux only), DOM elements and event listeners. Optional budgets in bench.toml mark them pass/fail.
• Host details are a reference note only. Every report records the Chrome version and the test machine (CPU, memory, OS, power, load average), but nothing is adjusted or compared with them.

Tests

Test	What it does
load	Opens the page and measures it up to "ready": the ready selector appearing, or the page settling (no DOM change for 500 ms) when element targeting isn't available.
scroll-steady	Sends 180 mouse-wheel events of 100 px into the scroll element, then scrolls back to the top and checks heap growth.
scroll-fast	The same with 400 px per event.

Each measurement uses a fresh Chrome process and profile. Each test starts with one warm-up run per side, then A and B alternate (AB, BA, AB…).

Element targeting with data-* attributes

Scroll tests need a ready selector and a scroll selector, and both must use data-* attributes (e.g. [data-product-list]). Before a run, SiteBench checks both revisions' source for those attributes. If either revision lacks them, it warns that element targeting isn't supported and runs only the whole-page load comparison.

Install

Requires Go 1.27+, git, and whatever your target needs to install and serve itself (e.g. Node/npm).

go build -o sitebench .

On first use SiteBench pins the current stable Chrome for Testing in bench.toml and downloads it into its cache. The host's Chrome is never used.

Usage

sitebench                     # TUI for the current directory
sitebench --results ~/bench   # TUI for another results directory

From the command line:

sitebench target add --name <site> --repo ~/src/<site> \
  --dir web --path / \
  --ready '[data-product-card]' --scroll '[data-product-list]' \
  --viewports mobile=390x844,desktop=1440x1000

sitebench target check <site> --rev @worktree   # one load + screenshot per viewport
sitebench run --a main --b @worktree           # compare; prints Markdown, saves JSON
sitebench run --a v1.0 --b v1.1 --tests load --runs 9 --throttle 6
sitebench sessions
sitebench export sessions/<file>.json          # writes <file>.md

sitebench chrome available                     # channels and recent versions
sitebench chrome use 158.0.8127.18             # pin (and download) another version
sitebench chrome status
sitebench chrome remove 157.0.8103.78
sitebench cache prune                          # delete cached source snapshots

A revision is anything git understands (main, HEAD~3, a1b2c3d, v1.2), or @worktree for the repository's current files including uncommitted changes. Comparing a revision with itself is an A/A run, which shows how much results vary on your machine.

Files

Results directory (--results, default ., or $SITEBENCH_RESULTS):

bench.toml          targets, pinned Chrome version + zip SHA-256, defaults, budgets, host notes
sessions/*.json     saved sessions (raw samples + comparisons)
sessions/*.md       Markdown exports
checks/*.png        screenshots from `target check`

Cache (~/.cache/sitebench, or $SITEBENCH_CACHE):

chrome/<version>/<platform>/   downloaded Chrome builds
snapshots/<repo>/<commit>/     extracted revisions with dependencies installed
logs/                          install and server output for each session

Revisions are extracted with git archive, so SiteBench never creates branches or worktrees in your repository. @worktree is served directly from the repository, and nothing is installed into it.

bench.toml

[chrome]
  version = "157.0.8103.78"
  [chrome.sha256]
    linux64 = "9223…"        # recorded on first download, checked on other machines

[defaults]
  throttle = 4.0
  runs = 5
  tests = ["load", "scroll-steady", "scroll-fast"]

[host]
  memory_speed = "DDR5-5600" # memory speed can't be read without root on Linux
  include_hostname = false
  notes = "laptop, on battery saver off"

[budgets]                    # optional maximums for absolute metrics
  heap_mb = 25
  dom_elements = 2000

[[target]]
  name = "<site>"
  repo = "/home/me/src/<site>"
  install_dir = "."
  install = "npm ci"
  dir = "web"
  serve = "npx vite --port {port} --strictPort"
  path = "/"
  ready = "[data-product-card]"
  scroll = "[data-product-list]"
  [target.viewports]
    mobile = [390, 844]
    desktop = [1440, 1000]

Notes and limits

• Linux and macOS. Process-group handling uses Unix signals. Renderer RSS is read from /proc, so it's Linux only.
• No Chrome sandbox on Linux. Chrome runs with --no-sandbox there, because Chrome for Testing has no AppArmor profile and recent distributions block its sandbox. Pages are your own local builds.
• Missing system libraries (e.g. libnss3) are reported on first start with the library names.
• Headless frame pacing is coarse, so frame-time percentiles aren't reported. Scroll smoothness shows up as long tasks and frames over 50 ms.

License

BSD 2-Clause. See LICENSE.