vcache

A compilation cache for C, C++ and Rust that solves the cross-directory problem head-on.

ccache and sccache work well when you build the same tree, from the same directory, twice. Move the checkout — a second worktree, a CI runner, a colleague's machine — and the hit rate collapses, because absolute paths are baked into preprocessed output, __FILE__, and debug info. Their documented workaround requires building from the top of the source tree with only relative include paths, which is not what CMake, Meson or Cargo generate.

vcache takes ownership of path rewriting instead. You declare your source roots; vcache canonicalises them on the compiler command line and normalises the two artifacts the compiler leaves un-remapped. The result:

$ cd /work/checkout-one && vcache g++ -g -O2 -c -I include src/lib.cc -o lib.o
$ cd /tmp/checkout-two  && vcache g++ -g -O2 -c -I include src/lib.cc -o lib.o
   ^ cache hit, and the two lib.o files are byte-identical

Building vcache itself from two unrelated directories: 25 of 25 objects served from cache, and the linked binaries are byte-identical (4.5s → 0.77s).

What the v stands for

Either:

• Vlad's cache.
• Virtuous Compile Avoidance, Conscientiously Hashing Everything.

The backronym describes the actual design. Avoiding a compile is the entire point, and vcache is conscientious to a fault about earning it: it never trusts an mtime, it re-hashes every file a cached dependency scan claims to have read, and it checksums every entry so a corrupt one reads as a miss instead of a bad object. That costs a preprocessor run on every lookup, which is why a vcache hit is slower than a ccache hit — and why it is a hit at all when a ccache hit would not have been.

Documentation

docs/quickstart.md	Zero to cross-directory hits in five minutes
docs/configuration.md	Every option, precedence, worked examples
docs/preprocessor-problem.md	Why this is needed, with measurements
docs/design.md	How it is put together
docs/daemon.md	The optional cache daemon: async S3 uploads, kept connections
docs/linux-kernel.md	Recipe: caching Linux kernel builds
docs/prompts/vcache-friendly-audit.md	A prompt for a coding agent that audits your build for vcache friendliness

Quick start

Prerequisites: a C++20 compiler, make and curl. vcache loads libcurl at runtime with dlopen, so the shared library is only needed if you use the S3 layer. libcurl's development headers (libcurl4-openssl-dev on Debian/Ubuntu, libcurl-devel on Fedora and RHEL, already present in the macOS SDK) are used when installed; without them the build falls back to the few declarations it needs, vendored in third-party/curl.

$ ./third-party/fetch.sh     # builds tcmalloc (one time)
$ make -j
$ make test

Then either call it explicitly:

$ vcache g++ -c foo.cc -o foo.o

or drop symlinks named after your compilers early on $PATH:

$ ln -s /path/to/vcache ~/.local/libexec/vcache/g++
$ ln -s /path/to/vcache ~/.local/libexec/vcache/gcc
$ ln -s /path/to/vcache ~/.local/libexec/vcache/rustc
$ export PATH=~/.local/libexec/vcache:$PATH

Keep other compiler wrappers' directories (ccache's /usr/lib64/ccache, for example) off $PATH. Each wrapper runs the next same-named compiler on $PATH, so vcache and such a wrapper would run each other; vcache stops with an error after four such hops.

Configuring roots

A root is a directory whose location should not affect the cache key. Almost always: your source tree.

$ export VCACHE_ROOTS=/home/you/myproject
$ vcache g++ -c src/a.cc -o a.o

/home/you/myproject is rewritten to /vcache/myproject everywhere it appears. On another machine, export VCACHE_ROOTS=/build/myproject maps to the same canonical prefix, and the two share cache entries.

Roots may be given as PATH, PATH=name (→ /vcache/name), or PATH=/absolute/target to choose the canonical prefix outright. They can be repeated, come from --vcache-root=, VCACHE_ROOTS (colon-separated), or the config file.

vcache also maps the current directory by default, because DW_AT_comp_dir records the build directory rather than the source directory. Disable with VCACHE_MAP_CWD=0 if you have a reason to.

Check what a mapping resolves to:

$ VCACHE_ROOTS=/home/you/myproject vcache --show-roots
/home/you/myproject -> /vcache/myproject
/home/you/myproject/build -> /vcache/cwd

Prefix-map flags from your build system

If the command line already contains -ffile-prefix-map, -fdebug-prefix-map, -fmacro-prefix-map or --remap-path-prefix, vcache refuses to run:

$ vcache g++ -ffile-prefix-map=/build=/usr/src -c a.cc -o a.o
vcache: refusing to run: the command line contains -ffile-prefix-map=/build=/usr/src
vcache manages path prefix mapping itself, and combining the two would silently change
which paths end up in your output. Choose one:
  --vcache-allow-prefix-maps        drop them; vcache's mapping wins
  --vcache-incoming-prefix-maps=keep pass them through (disables caching)

Silently overriding a mapping the build deliberately asked for would change its output without saying so, and gcc's last-match-wins resolution makes the combination genuinely ambiguous. Override deliberately, by any of:

$ vcache --vcache-allow-prefix-maps g++ ...          # command line
$ export VCACHE_INCOMING_PREFIX_MAPS=strip           # environment

[vcache]
incoming_prefix_maps = "strip"   # error (default) | strip | keep

Precedence is command line > environment > config file. keep passes the flags through untouched and marks the compilation uncacheable.

Configuration file

$VCACHE_CONFIG, else ~/.config/vcache/config.toml, else /etc/vcache/config.toml. The layout follows sccache's.

[vcache]
roots = ["/home/you/myproject", "/opt/toolchain=toolchain"]
map_cwd = true
incoming_prefix_maps = "error"   # error (default) | strip | keep
hash_env_vars = ["SOURCE_DATE_EPOCH"]

[cache.disk]
dir = "~/.cache/vcache"
size = "20G"

[cache.s3]
bucket = "my-build-cache"
region = "us-west-2"
prefix = "vcache/"
# endpoint = "http://minio.internal:9000"   # for MinIO/Ceph
# path_style = true

Environment variables override the file, and command-line flags override those. docs/configuration.md lists every key, its environment and command-line equivalents, and its default.

Cache layers

Reads walk local disk first, then S3, and stop at the first hit. A remote hit is promoted into the local layer so the rest of the build serves it locally. Writes go to every writable layer. If S3 is unreachable, lookups degrade to misses and the build proceeds — a shared cache outage never breaks a build.

Optionally, a daemon (VCACHE_DAEMON=auto) owns the layers for every compile on the machine. Compiles then stop waiting for S3 uploads and stop paying for a new S3 connection each; see docs/daemon.md.

Commands

Command	Effect
vcache --show-stats	hit/miss counters, why runs were not cached, and cache size
vcache --zero-stats	reset counters
vcache --clear	delete all entries
vcache --trim	evict until under the size limit, and remove scratch directories killed compiles left
vcache --show-config	effective configuration
vcache --show-roots	resolved root mapping for this directory
vcache --start-daemon	start the cache daemon in the background
vcache --stop-daemon	drain pending uploads, then stop the daemon
vcache --daemon-status	what the daemon is doing

Debugging cached builds

Paths in debug info are canonical, so point your debugger at the real sources:

(gdb) set substitute-path /vcache/myproject /home/you/myproject
(lldb) settings set target.source-map /vcache/myproject /home/you/myproject

Performance

Measured on one translation unit from this repository (gcc 13.3, -O2 -g, warm page cache, median of three):

	per file
plain g++	1.50 s
vcache hit	0.073 s
ccache hit (direct mode)	0.009 s

A vcache hit is ~19× faster than compiling, and ~8× slower than a ccache hit, because vcache always runs the preprocessor while ccache's direct mode skips it by hashing the source plus a stored manifest of includes. A precompiled header does not shorten that step: its header is expanded on every lookup, though that is small next to the rest of a large translation unit.

The trade is deliberate: vcache competes on hit rate, not per-hit latency. A ccache hit is faster, but only when ccache hits at all — move the checkout and it does not. 73 ms against a 1.5 s compile is comfortably on the right side of the line.

Runtime dependencies

$ ldd bin/vcache
	linux-vdso.so.1
	libm.so.6        (tcmalloc uses log2)
	libc.so.6
	/lib64/ld-linux-x86-64.so.2

That is the whole list. vcache is spawned once per compilation, so every DT_NEEDED entry is mapped and relocated on every invocation — linking libcurl directly pulled in about thirty shared objects (TLS, HTTP/2, Kerberos, LDAP, SSH, IDN, compression) and cost ~2.4 ms of startup on builds that never make a request. Two changes removed them:

• libcurl is dlopend, only when an S3 layer is actually constructed. A disk-only build never maps it; if it is missing when S3 is configured, vcache says so and continues with the local cache.
• SHA-256 and HMAC are vendored (src/hash/sha256.cc). SigV4 was the only use of OpenSSL — three functions — which is a thin reason to link it.

Process startup went from 6.1 ms to 3.0 ms as a result.

A toolchain without the static libstdc++ archive (libstdc++-static on Fedora and RHEL) gets a dynamically linked libstdc++ instead, which adds libstdc++.so.6 and libgcc_s.so.1 to the list above.

What is not cached

Linking (unless the link cache is on, VCACHE_LINK_CACHE=1, Linux only), -E-only runs, -MG, multiple inputs in one invocation, -save-temps, PGO flags, .incbin (the assembler reads a file the preprocessed text never mentions), a clang precompiled header built without -Xclang -fno-pch-timestamp, and rustc without --out-dir/--emit. All of these fall through to the compiler unchanged, so a build always makes progress. vcache --show-stats counts them as uncacheable, broken down by the rule that declined each.

Three things that look like they belong on that list but are cached:

• -march=native. The flag means something different on every machine, so vcache asks the compiler which target it resolved to and puts that in the key — see native_target.
• -M/-MM dependency scans. Keying those on preprocessed text would cost more than the run being cached, so they are keyed on the command line and verified against a manifest of the files they read — see dep_scan.
• Precompiled header builds. A PCH records the absolute path of every header it read and loads nowhere else, so its key holds the directory and the command line as typed: a rebuild of the same tree hits, another checkout does not. vcache writes <pch>.vcache-key beside it, a path-independent digest of its inputs that consumers' manifests share across checkouts — see the design notes.

How it works

1. Parse the command line; decide whether it is cacheable at all.
2. Refuse if the caller supplied -ffile-prefix-map / --remap-path-prefix, unless overridden; vcache owns path rewriting.
3. Derive -ffile-prefix-map flags from the configured roots, ordered so the most specific root wins under gcc's last-match-wins rule.
4. Preprocess with those flags plus -fno-working-directory, then hash the output with linemarker paths rewritten — gcc does not remap those itself.
5. Hash alongside it: compiler identity, codegen flags (never -I/-D, whose effect is already in the preprocessed text, and never -o), and the canonical root targets.
6. Look up; on a hit, write the object, replay diagnostics, and reverse-map the dependency file into local paths.
7. On a miss, compile into a temporary directory, place the artifacts, and store.

Steps 4 and 5 cost a preprocessor run on every lookup, so with cxx_manifest on (it is off by default) a C or C++ lookup first checks a manifest, as ccache's depend mode does. It is keyed on the source, the whole command line (-I and -D included), the compiler, the roots and any precompiled header the command names, and each state in it lists the files an earlier compile read, with their digests, the paths it probed and did not find, and the key that compile stored under. When every file of a state still hashes the same and every one of those paths is still absent, that key is looked up and the preprocessor never runs. The key is the ordinary preprocessor key, so trees with the manifest on and off share entries. A miss records its state from the compile's own -MD output, which vcache requests when the build did not, and on Linux from a trace of the compile through vcache-fstrace.so, which sees what the preprocessor looked for and did not find. The trade is exactness where the trace is not available (another platform, no tracer, a compiler process that does not load it): a depfile names the headers a compile read, not the paths it probed, so a header created later earlier on the include path is not seen and a stale object can be served; there, a gcc or clang built with an absent-deps patch can be asked to list those paths itself (-fabsent-deps=, opt-in) instead. Even traced, a header created inside the compiler's own trees outside its include directories is not seen. See cxx_manifest.

Rust follows the same shape, with --emit=dep-info standing in for preprocessing and --extern dependencies hashed by content rather than path. The variables that dep-info says the crate read through env!/option_env! go into the key with their raw values, so an unset or path-free variable still hits across directories. Dep-info expands every macro, so a lookup first checks a manifest of earlier dep-info runs, re-hashing the files each one recorded, and runs rustc only when none still matches — see rust_dep_info.

docs/preprocessor-problem.md records the measurements this design rests on, including the two compiler behaviours that make the naive approach fail.

Build configuration

Release builds use -O3 -ggdb3, LTO, and zstd-compressed debug info. Both LTO and -gz=zstd are probed for at configure time, so a toolchain without them still builds — just larger. make BUILD=debug gives -O0 -ggdb3 with no LTO. -static-libstdc++ -static-libgcc is probed the same way, and make prints which of static or dynamic libstdc++ it chose. So is curl/curl.h: without it, make says so and builds against third-party/curl, and S3 works as before wherever libcurl is installed at runtime.

Effect on the shipped binary, which keeps full -ggdb3 debug info throughout:

configuration	size
-O2 -g, no LTO, no compression	12.9 MB
-O3 -ggdb3 + LTO + --gc-sections	9.3 MB
the above + -gz=zstd	4.0 MB

zstd compression of DWARF is the larger win (−5.3 MB); LTO with -ffunction-sections -fdata-sections -Wl,--gc-sections accounts for −1.9 MB. LTO on its own tends to grow a binary through inlining — the dead-code elimination is what pays for it. Debug info survives intact: 32 compilation units, full line tables, and 27,909 macro definitions from -ggdb3.

Implementation notes

• C++20, built with plain make.
• Boost.Spirit X3 parses Makefile-syntax dependency files, which have real structure (line continuations, escaped spaces, $$). The compiler command line is a flag list rather than a grammar, so it uses a table-driven scanner; preprocessed output is normalised by a streaming scanner because it is a hot path over tens of megabytes per compile.
• BLAKE3 for cache keys, tcmalloc for allocation, toml++ for config. S3 uses hand-rolled SigV4 over a lazily loaded libcurl, with SHA-256/HMAC vendored — no AWS SDK and no OpenSSL.
• Boost is vendored as a 932-header subset (4.2 MB of Boost 1.86.0's 104 MB) — exactly what Spirit X3 opens. make boost-subset regenerates it against a full Boost tree if an include ever reaches further. Only gperftools is downloaded at setup time; everything else is committed.
• Statically linked apart from libc and libcurl, and libstdc++ where the toolchain has no static archive of it.
• Cache entries carry a BLAKE3 checksum; a corrupt entry reads as a miss rather than yielding a bad object.

Tests

$ make test

451 unit assertions and 309 integration assertions, covering cross-directory hits, out-of-tree builds, dependency-file replay, diagnostics replay, uncacheable fallback, masquerade mode, Rust, cache management, -march=native resolution, dependency-scan and Rust dep-info manifests, kbuild-shaped -Wp, command lines, .incbin, and the S3 layer against a mock object store. SigV4 is checked against AWS's documented signing-key vector and an independent reference implementation.

$ make kernel-test

A real Linux kernel build, kept out of make test because it downloads two kernel tarballs, wants ~10 GB of disk and takes a few minutes. It is the only test that exercises kbuild end to end — see docs/linux-kernel.md.

Licence

Copyright © 2026 Unto Labs.

vcache is licensed under the Apache License, Version 2.0. The full text is in LICENSE; you may also obtain it at https://www.apache.org/licenses/LICENSE-2.0. Unless required by applicable law or agreed to in writing, the software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Every first-party source file carries both SPDX-FileCopyrightText: 2026 Unto Labs and SPDX-License-Identifier: Apache-2.0, because this tree deliberately mixes licences and the top-level LICENSE alone would not say which file is under which, or who holds it.

Third-party components keep their own terms

Nothing under third-party/ is covered by the Apache licence above. Each component remains under the licence it was published with, and each directory carries that licence text verbatim:

Component	Licence	Text
BLAKE3 1.5.4	Apache 2.0 with LLVM exception	third-party/blake3/LICENSE_A2
toml++ 3.4.0	MIT	third-party/tomlplusplus/LICENSE
Boost 1.86.0 subset	Boost Software License 1.0	third-party/boost/LICENSE
curl 8.14.1 curl.h subset	curl licence	third-party/curl/COPYING
gperftools 2.16	BSD 3-clause	fetched at build time, not committed; licence ships in the tarball

All five are permissive and impose no term Apache 2.0 does not already accommodate, which is what the combination requires: their notice-retention obligations sit comfortably inside Apache 2.0's own attribution rules, and the combined binary ships under Apache 2.0 with the vendored notices intact. Vendoring a file here does not place it under Apache 2.0 — it keeps its original terms, and the notices in third-party/ must survive any redistribution.