Computing crate_hash from metadata encoding instead of HIR (implements #94878) by Daniel-B-Smith · Pull Request #154724 · rust-lang/rust

Daniel-B-Smith · 2026-04-02T18:15:01Z

This PR converts the crate_hash/SVH to depend on metadata instead of HIR whenever metadata is needed. A trimmed down crate_hash is kept for dylib/binary cases where metadata is not present. It is believed that the metadata will be a more sound way to track dependency changes.

Related to #94878

The metadata hash is calculated on the raw bytes before flush because hashing the elements incrementally turned out to be too expensive.

The change to the HIR hash is potentially safe even without the metadata crate_hash change. Without that change, this PR is a performance regression. I am bundling them because I believe the HIR hash change is more likely to be safe in light of the HIR hash being less load bearing on the SVH.

The dylib/binary crate_hash removes components that I believe are not relevant to those cases. Analysis:

resolutions.visibilities_for_hashing:

The comment says: "Hash visibility information since it does not appear in HIR." Visibilities only matter to external users of the crate (i.e. consumers of metadata). Without metadata, visibility differences are not observable, so this contributes nothing for incremental correctness.

debugger_visualizers:

The comment says: "that content is exported into crate metadata, so any changes to it need to be reflected in the crate hash." Without metadata, there is nothing to export, so this is purely metadata-motivated. The visualizer file path is already covered by HIR (the attribute) and the content is loaded later only if metadata is being written.

source_file_names:

These exist solely so that the crate-hash reflects remapped source paths that get embedded into metadata (the comment explicitly says "If we included the full mapping in the SVH, we could only have reproducible builds…"). The remapping itself is already captured via dep_tracking_hash, so for incremental this is redundant.

The crate hash calculation can be reverted via -Z metadata-crate-hash=no. I manually confirmed that setting that flag generates the same hash as the commit prior to my change. The test for that flag only checks that it changes the value of the hash and does not check the specific value of the hash. I could add a check for the latter, but I'm nervous that a hash golden test would be a maintenance headache.

nnethercote · 2026-04-09T23:26:03Z

@bors try @rust-timer queue

oli-obk · 2026-09-17T08:30:58Z

@bors r+

rust-bors · 2026-09-17T08:31:02Z

📌 Commit 56647f0 has been approved by oli-obk

It is now in the queue for this repository.

rust-bors · 2026-09-17T13:04:56Z

💔 Test for 5ca7eca failed: CI. Failed job:

auto - test-armhf-gnu (web logs, enhanced plaintext logs)

Daniel-B-Smith · 2026-09-24T22:44:08Z

The last failure was due to the proc macro rmake test not working on cross compile. A proc macro is dlopened by the host rustc, so it has to be built for the host, and the the --target added in #139244 doesn't work with this configuration. Since this is solely testing that the SVH is updated correctly, I see no reason to be concerned that SVH generation would be different in cross compilation since all the work is done during the metadata phase.

The ignore-musl is mostly being defensive since it also may not work work for this case, and there isn't any real benefit to test coverage.

Since the diff got rebased into an existing commit, the diff is solely in tests/run-make/proc-macro-dep-source-changes-crate-hash/rmake.rs.

//@ needs-crate-type: proc-macro

was changed into

//@ ignore-cross-compile
//@ needs-crate-type: proc-macro
//@ ignore-musl (FIXME: can't find `-lunwind`)

jieyouxu · 2026-09-25T05:05:57Z

@bors try jobs=* nolimit

rust-bors · 2026-09-25T08:10:03Z

💔 Test for d7ecec7 failed: CI. Failed jobs:

try - test-arm-android (web logs, enhanced plaintext logs)
try - test-armhf-gnu (web logs, enhanced plaintext logs)
try - test-i686-msvc (web logs, enhanced plaintext logs)
try - test-various (web logs, enhanced plaintext logs)

rustbot · 2026-09-25T18:49:07Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

The SVH is computed by hashing the entire encoded metadata blob, and SipHash-128 (~2.5 Ir/byte) dominated the cost — it accounted for the whole of the measured regression on metadata-heavy crates. Switch the byte pass to XXH3-128, which is ~10x cheaper per byte on its runtime-detected AVX2 path. XXH3 is non-cryptographic, which is fine here: the SVH is only a change detector, must be reproducible (so it is never keyed secretly, meaning even the previous SipHash use was not collision-resistant), and compiling a crate already executes its build scripts and proc-macros, so the crate author is trusted regardless. The untracked supplements (dep-tracking hash and the HIR hash for e.g. DefKind::GlobalAsm) are still folded in via a StableHasher fingerprint, so coverage is unchanged. twox-hash 2.x is already in the dependency graph via ruzstd; this only adds a direct edge (xxhash3_128 + std features for the runtime SIMD dispatch).

Daniel-B-Smith · 2026-09-25T18:57:11Z

I fixed tests/run-make/proc-macro-dep-source-changes-crate-hash/rmake.rs but forgot to fix the other two proc-macro tests I added. They have since been fixed.

oli-obk · 2026-09-25T20:38:51Z

@bors r+

rust-bors · 2026-09-25T20:38:55Z

📌 Commit 79ee8dc has been approved by oli-obk

It is now in the queue for this repository.

rust-bors · 2026-09-26T07:03:54Z

☀️ Test successful - CI
Approved by: oli-obk
Duration: 4h 42m 17s
Pushing 90c195e to main...

github-actions · 2026-09-26T07:06:59Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 5ceaf66 (parent) -> 90c195e (this PR)

Test differences

Show 50 test diffs

Stage 1

[run-make] tests/run-make/crate-hash-dep-extra-filename: [missing] -> pass (J1)
[run-make] tests/run-make/crate-hash-metadata-flag: [missing] -> pass (J1)
[run-make] tests/run-make/metadata-stub-incremental-reuse: [missing] -> pass (J1)
[run-make] tests/run-make/proc-macro-dep-source-changes-crate-hash: [missing] -> pass (J1)
[run-make] tests/run-make/proc-macro-env-changes-crate-hash: [missing] -> pass (J1)
[run-make] tests/run-make/proc-macro-global-asm-changes-crate-hash: [missing] -> pass (J1)

Stage 2

[run-make] tests/run-make/crate-hash-metadata-flag: [missing] -> pass (J0)
[run-make] tests/run-make/proc-macro-dep-source-changes-crate-hash: [missing] -> pass (J0)
[run-make] tests/run-make/proc-macro-env-changes-crate-hash: [missing] -> pass (J0)
[run-make] tests/run-make/proc-macro-global-asm-changes-crate-hash: [missing] -> pass (J0)
[run-make] tests/run-make/crate-hash-metadata-flag: [missing] -> ignore (ignored when cross-compiling) (J2)
[run-make] tests/run-make/crate-hash-dep-extra-filename: [missing] -> pass (J3)
[run-make] tests/run-make/metadata-stub-incremental-reuse: [missing] -> pass (J3)
[run-make] tests/run-make/proc-macro-dep-source-changes-crate-hash: [missing] -> ignore (ignored when cross-compiling) (J4)
[run-make] tests/run-make/proc-macro-env-changes-crate-hash: [missing] -> ignore (ignored when cross-compiling) (J4)
[run-make] tests/run-make/proc-macro-global-asm-changes-crate-hash: [missing] -> ignore (ignored when cross-compiling) (J4)
[run-make] tests/run-make/proc-macro-dep-source-changes-crate-hash: [missing] -> ignore (ignored when the target environment is musl ((FIXME: can't find -lunwind))) (J5)
[run-make] tests/run-make/proc-macro-env-changes-crate-hash: [missing] -> ignore (ignored when the target environment is musl ((FIXME: can't find -lunwind))) (J5)
[run-make] tests/run-make/proc-macro-global-asm-changes-crate-hash: [missing] -> ignore (ignored when the target environment is musl ((FIXME: can't find -lunwind))) (J5)

Additionally, 31 doctest diffs were found. These are ignored, as they are noisy.

Job group index

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 90c195e09786b4923a1e4e04216091de639f9fad --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

test-aarch64-apple-macos-26-1: 2h 22m -> 4h 40m (+97.2%)
test-x86_64-gnu-llvm-21-2: 1h 13m -> 1h 47m (+47.3%)
test-x86_64-gnu-llvm-22-1: 55m 59s -> 1h 21m (+45.1%)
test-x86_64-gnu-miri: 1h 31m -> 52m 44s (-42.6%)
test-x86_64-gnu-gcc: 1h 5m -> 1h 31m (+40.6%)
test-x86_64-gnu-aux: 2h 42m -> 1h 36m (-40.4%)
test-x86_64-gnu-parallel-frontend: 1h 28m -> 2h 4m (+39.7%)
test-x86_64-gnu-debug: 2h 10m -> 1h 19m (-38.9%)
dist-i686-linux: 1h 50m -> 1h 10m (-36.5%)
dist-x86_64-msvc: 3h 3m -> 1h 56m (-36.4%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2026-09-26T07:49:43Z

Finished benchmarking commit (90c195e): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.1%	[0.2%, 5.8%]	15
Improvements ✅ (primary)	-0.3%	[-0.7%, -0.1%]	81
Improvements ✅ (secondary)	-1.1%	[-4.6%, -0.1%]	52
All ❌✅ (primary)	-0.3%	[-0.7%, -0.1%]	81

Max RSS (memory usage)

Results (primary -3.5%, secondary 2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.2%	[2.0%, 4.7%]	6
Improvements ✅ (primary)	-3.5%	[-3.5%, -3.5%]	1
Improvements ✅ (secondary)	-1.6%	[-2.0%, -1.2%]	2
All ❌✅ (primary)	-3.5%	[-3.5%, -3.5%]	1

Cycles

Results (primary 13.8%, secondary 2.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	13.8%	[3.1%, 25.3%]	7
Regressions ❌ (secondary)	4.6%	[2.1%, 9.0%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.7%	[-4.8%, -2.1%]	3
All ❌✅ (primary)	13.8%	[3.1%, 25.3%]	7

Binary size

Results (primary 0.2%, secondary 0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.0%, 0.3%]	33
Regressions ❌ (secondary)	0.2%	[0.0%, 0.3%]	49
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.0%, 0.3%]	33

Bootstrap: 488.629s -> 487.474s (-0.24%)
Artifact size: 406.21 MiB -> 406.22 MiB (0.00%)

panstromek · 2026-09-29T07:36:44Z

perf triage:

Improvements outweigh regressions. Regressions are secondary, mostly in large-workspace, which is artifical stress tests for dependency overhead in large workspaces.

@rustbot label: +perf-regression-triaged

Daniel-B-Smith · 2026-10-01T16:58:22Z

I did some digging, and almost all of the improvements are a reduction in SipHash cost. needs_hir_hash() uses needs_metadata() as a proxy, so the rustdoc code is no longer calculating the HIR hash in some cases. Since the HIR hash getter will fail with an ICE, there's no risk of a subtle regression.

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 2, 2026