Skip to content

Add a C microbenchmark harness with memcmp example - #911

Open
msuozzo wants to merge 1 commit into
WebAssembly:mainfrom
msuozzo:push-nvpqytzrkymy
Open

msuozzo wants to merge 1 commit into
WebAssembly:mainfrom
msuozzo:push-nvpqytzrkymy

Conversation

@msuozzo

@msuozzo msuozzo commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Add a small header-only C benchmark runner under test/bench that
calibrates iteration batches between CLOCK_MONOTONIC reads so host clock
overhead does not dominate nanosecond-scale libc calls.

Wire benchmark binaries into test/CMakeLists.txt via a bench target
using the configured Wasm engine, with a 1ms CTest smoke pass on CLI
runners to keep benchmarks building cleanly.

@alexcrichton

Copy link
Copy Markdown
Collaborator

I'm a bit hesitant to bake our own here as it's somewhat nontrivial to flesh out custom benchmarking, for example you kind of want things like statistics, saving records to files to compare against previous iterations, etc, and that can get pretty gnarly pretty quickly. Since wasi-sdk is typically used to compile wasi-libc and it's already got libc++, would it be possible to rely on that when compiling benchmarks? They don't have to compile with every toolchain under the sun for example.

@msuozzo

msuozzo commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

i should have mentioned this in the pr description: bench.h prints go's benchmark format so benchstat could potentially handle the complexity you're describing (stats and cross-run comparisons, etc):

$ cmake -B build -DBUILD_TESTS=ON -DBENCH_ARGS="-count 6"
$ git checkout 4ec32c50~ -- libc-top-half/musl/src/string/memcmp.c
$ cmake --build build --target bench > before.txt
$ git checkout 4ec32c50 -- libc-top-half/musl/src/string/memcmp.c
$ cmake --build build --target bench > after.txt
$ go run golang.org/x/perf/cmd/benchstat@latest before.txt after.txt
                       │  before.txt  │             after.txt              │
                       │    sec/op    │   sec/op     vs base               │
_memcmp/equal/4           4.410n ± 2%   4.680n ± 1%   +6.12% (p=0.002 n=6)
_memcmp/equal/8           6.860n ± 1%   3.110n ± 1%  -54.66% (p=0.002 n=6)
_memcmp/equal/16         11.870n ± 1%   4.350n ± 1%  -63.35% (p=0.002 n=6)
_memcmp/equal/64          42.80n ± 3%   11.79n ± 0%  -72.44% (p=0.002 n=6)
_memcmp/equal/1024        644.4n ± 3%   166.4n ± 7%  -74.18% (p=0.002 n=6)
_memcmp/diff_first/8      2.175n ± 1%   1.890n ± 2%  -13.10% (p=0.002 n=6)
_memcmp/diff_first/64     2.170n ± 0%   1.900n ± 2%  -12.44% (p=0.002 n=6)
_memcmp/diff_last/8       6.820n ± 5%   2.185n ± 1%  -67.96% (p=0.002 n=6)
_memcmp/diff_last/64      48.22n ± 1%   10.90n ± 0%  -77.41% (p=0.002 n=6)
_memcmp/diff_last/1024    645.6n ± 1%   166.3n ± 0%  -74.24% (p=0.002 n=6)
geomean                   19.94n        8.408n       -57.83%

i did prototype google benchmark on top of wasi-sdk, which works, though it comes with a few restrictions:

  • introduces the first c++ into the repo
  • requires a full wasi-sdk install rather than plain clang
  • needs a separate ExternalProject subbuild (~100 lines of cmake) to handle non-c-only flags
  • mixes wasi-sdk's prebuilt libc++ (built against the last release's headers) with in-tree headers and libc, which feels like an unnecessary risk of header/abi skew

would documenting this benchstat workflow address your concern, or would you still prefer something like the google benchmark setup? or is there another framework that could work?

@alexcrichton

Copy link
Copy Markdown
Collaborator

Ok that makes sense to me, so yeah seems like the Google C++ benchmark framework isn't worth using here. Documenting something like benchstat seems ok as well, but I'm still kind of hesitant to pull in too much here without fully understanding it, for example the bench.h here has some oddities to me and things that I don't fully understand. While this could likely be resolved through discussion and/or documentation, I think it would still be ideal if something could be used off-the-shelf mostly from elsewhere. Searching for C/CMake benchmarking turns up https://github.com/sheredom/ubench.h for example. I've never used that before but it seems like it might fit this use case to get integrated here and it's got documentation as well. What would you think of finding something off the shelf to use from elsewhere?

Add a small header-only C benchmark runner under test/bench that
calibrates iteration batches between CLOCK_MONOTONIC reads so host clock
overhead does not dominate nanosecond-scale libc calls.

Wire benchmark binaries into test/CMakeLists.txt via a bench target
using the configured Wasm engine, with a 1ms CTest smoke pass on CLI
runners to keep benchmarks building cleanly.
@msuozzo

msuozzo commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

So ubench.h did come up when i was investigating but:

  1. It fails with "unsupported platform!" for wasi. fixable but not usable as-is.
  2. It probes the clock on each iteration which, in our case where clock_time_get() >> iteration, is untenable.
  3. It truncates avg iteration time at ns precision where we'd really want that additional precision for these sorts of benchmarks.

It also weighs ~1k LoC, much of it repeating the logic benchstat provides us. Admittedly isn't not ideal to roll our own here but it's <100 LoC of pretty straightforward code. i also updated the logic to fix some edge-cases i found. ptal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants