Outline and deduplicate cold panic and drop paths in MIR and LLVM by stephenduong1004 · Pull Request #164821 · rust-lang/rust

stephenduong1004 · 2026-10-23T20:31:04Z

Summary

Experimental prototype to outline and deduplicate cold panic! formatting, cold drop glue, and ? (FromResidual::from_residual) paths in MIR and LLVM, as an alternative to the macro-based approach in #115562 / #115670 (reverted in #145304).

Reproducer

use std::any::type_name;

#[inline]
pub fn checked_scale<T>(x: usize, factor: usize, limit: usize) -> usize {
    let scaled = x.wrapping_mul(factor);
    if scaled >= limit {
        panic!("overflow in {}: {} >= {}", type_name::<T>(), scaled, limit);
    }
    scaled + 1
}

pub fn entry(x: usize, limit: usize) -> usize {
    checked_scale::<u8>(x, 2, limit)
        + checked_scale::<u16>(x, 3, limit)
        + checked_scale::<u32>(x, 4, limit)
        + checked_scale::<u64>(x, 5, limit)
}

Today: Borrowing &scaled and &limit in panic! forces both out of SSA registers into stack allocas on the hot path, and entry gets 4 inlined copies of ~50 IR instructions of fmt::Arguments setup.
With this PR: scaled and limit stay in SSA registers on the hot path, the panic block is outlined, and all <T> copies merge into 1 shared .cold.1 helper.
Impact: Collapses 16 -> 1 generic panic helpers (-13.6% compile time on microbenchmarks, -5% to -12.8% on large crates) and 64 -> 1 inlined catch_unwind payload drops (-5.8% compile time).

Why MIR + LLVM Instead of Macro Outlining (#115562 / #115670 / #145304)?

Outlining inside panic_2021! (#115670, reverted in #145304) could not outline multi-argument formatted panics (panic!("... {}", x)), still borrowed &$arg in the hot caller, and created a new const fn DefId per panic!() site (requiring #[rustc_const_panic_str] hacks).

Doing this in MIR and LLVM (after borrowck and const-eval) avoids macro/const-eval hacks, preserves return/? in panic args, keeps hot locals in registers, outlines full fmt::Arguments setups, and also covers ? and cold drops.

What This PR Does

MIR (rustc_mir_transform):
- ColdBlockOutliner: Copies hot Copy locals into a cold temporary before borrowing &x in cold blocks, keeping x in an SSA register and unblocking LLVM outlining.
- KnownVariantDrops: Specializes shared Drop blocks when predecessors set a known enum variant (e.g., Result::Ok vs Err in catch_unwind), separating hot no-op drops from cold Err destructors.
- Inline: Skips MIR inlining into cold blocks (unless #[inline(always)]).
Codegen (rustc_codegen_ssa):
- Fixes find_cold_blocks to ignore is_cleanup unwind edges and marks catch_unwind error branches cold.
- Marks non-trivial calls in cold blocks and FromResidual::from_residual (?) calls as cold + noinline.
LLVM (PassWrapper.cpp):
- Runs HotColdSplittingPass (hot-cold-split-threshold=1) after inlining and marks outlined .cold. functions noinline.
- Lifts up to 4 constant pointers/integers (e.g. type_name::<T>(), Location) into parameters on .cold. functions so MergeFunctionsPass can merge generic duplicates.

Next Steps

This draft PR is LLM-generated; this is to gather end-to-end rustc-perf numbers and design feedback. If the direction makes sense, I'm happy to split this into small, independent PRs for review.

rust-log-analyzer · 2026-10-23T21:14:24Z

The job test-aarch64-gnu-llvm-22-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[RUSTC-TIMING] rustc_passes test:false 19.581
   Compiling rustc_mir_transform v0.0.0 (/checkout/compiler/rustc_mir_transform)
[RUSTC-TIMING] rustc_mir_build test:false 27.485
   Compiling rustc_hir_typeck v0.0.0 (/checkout/compiler/rustc_hir_typeck)
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp:74:35: error: expected template-name before '<' token
warning: rustc_llvm@0.0.0:    74 |     : public RequiredPassInfoMixin<ColdInvokeOutlinerPass> {
warning: rustc_llvm@0.0.0:       |                                   ^
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp:74:35: error: expected '{' before '<' token
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp:74:35: error: expected unqualified-id before '<' token
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp: In lambda function:
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp:1364:55: error: invalid use of incomplete type 'struct {anonymous}::ColdInvokeOutlinerPass'
warning: rustc_llvm@0.0.0:  1364 |               EarlyMPM.addPass(ColdInvokeOutlinerPass());
warning: rustc_llvm@0.0.0:       |                                                       ^
warning: rustc_llvm@0.0.0: llvm-wrapper/PassWrapper.cpp:73:8: note: forward declaration of 'struct {anonymous}::ColdInvokeOutlinerPass'
warning: rustc_llvm@0.0.0:    73 | struct ColdInvokeOutlinerPass
warning: rustc_llvm@0.0.0:       |        ^~~~~~~~~~~~~~~~~~~~~~
error: failed to run custom build command for `rustc_llvm v0.0.0 (/checkout/compiler/rustc_llvm)`

Caused by:
  process didn't exit successfully: `/checkout/obj/build/aarch64-unknown-linux-gnu/stage1-rustc/release/build/rustc_llvm/8b318b13520b47ee/out/build_script_build` (exit status: 1)
  --- stdout
  cargo:rustc-check-cfg=cfg(llvm_component,values("ipo"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bitreader"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bitwriter"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("linker"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("asmparser"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("lto"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("coverage"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("instrumentation"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("x86"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("arm"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("aarch64"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("amdgpu"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("avr"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("loongarch"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("m68k"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("csky"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("mips"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("powerpc"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("systemz"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("webassembly"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("msp430"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("sparc"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("nvptx"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("hexagon"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("riscv"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("xtensa"))
  cargo:rustc-check-cfg=cfg(llvm_component,values("bpf"))
  cargo:rerun-if-env-changed=RUST_CHECK
  cargo:rerun-if-env-changed=REAL_LIBRARY_PATH_VAR
  cargo:rerun-if-env-changed=REAL_LIBRARY_PATH
  cargo:rerun-if-env-changed=LLVM_CONFIG
  cargo:rerun-if-changed=/usr/lib/llvm-22/bin/llvm-config
  cargo:rustc-cfg=llvm_component="aarch64"
  cargo:rustc-cfg=llvm_component="amdgpu"
  cargo:rustc-cfg=llvm_component="arm"
  cargo:rustc-cfg=llvm_component="asmparser"
  cargo:rustc-cfg=llvm_component="avr"
  cargo:rustc-cfg=llvm_component="bitreader"
  cargo:rustc-cfg=llvm_component="bitwriter"
  cargo:rustc-cfg=llvm_component="bpf"
  cargo:rustc-cfg=llvm_component="coverage"
  cargo:rustc-cfg=llvm_component="hexagon"
  cargo:rustc-cfg=llvm_component="instrumentation"
  cargo:rustc-cfg=llvm_component="ipo"
  cargo:rustc-cfg=llvm_component="linker"
  cargo:rustc-cfg=llvm_component="loongarch"
  cargo:rustc-cfg=llvm_component="lto"
  cargo:rustc-cfg=llvm_component="m68k"
  cargo:rustc-cfg=llvm_component="mips"
  cargo:rustc-cfg=llvm_component="msp430"
  cargo:rustc-cfg=llvm_component="nvptx"
  cargo:rustc-cfg=llvm_component="powerpc"
  cargo:rustc-cfg=llvm_component="riscv"
  cargo:rustc-cfg=llvm_component="sparc"
  cargo:rustc-cfg=llvm_component="systemz"
  cargo:rustc-cfg=llvm_component="webassembly"
  cargo:rustc-cfg=llvm_component="x86"
  cargo:rustc-cfg=llvm_component="xtensa"
  cargo:rerun-if-env-changed=LLVM_COMPILER_IS_GNU_LIKE
  cargo:rerun-if-env-changed=LLVM_ENZYME
  cargo:rerun-if-env-changed=LLVM_OFFLOAD
  cargo:rerun-if-env-changed=LLVM_RUSTLLVM
  cargo:rerun-if-env-changed=LLVM_ASSERTIONS
  cargo:rerun-if-changed=llvm-wrapper/PassWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/Linker.cpp
  cargo:rerun-if-changed=llvm-wrapper/README
  cargo:rerun-if-changed=llvm-wrapper/SuppressLLVMWarnings.h
  cargo:rerun-if-changed=llvm-wrapper/RustWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/SymbolWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/LLVMWrapper.h
  cargo:rerun-if-changed=llvm-wrapper/.editorconfig
  cargo:rerun-if-changed=llvm-wrapper/offload/CMakeLists.txt
  cargo:rerun-if-changed=llvm-wrapper/offload/OffloadWrapper.cpp
  cargo:rerun-if-changed=llvm-wrapper/CoverageMappingWrapper.cpp
  cargo:rerun-if-env-changed=CC_FORCE_DISABLE
  CC_FORCE_DISABLE = None
  cargo:rerun-if-env-changed=CXX_aarch64-unknown-linux-gnu
  CXX_aarch64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CXX_aarch64_unknown_linux_gnu
  CXX_aarch64_unknown_linux_gnu = Some(sccache c++)
  cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
  CC_KNOWN_WRAPPER_CUSTOM = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  cargo:rerun-if-env-changed=CXXFLAGS
  CXXFLAGS = None
  cargo:rerun-if-env-changed=HOST_CXXFLAGS
  HOST_CXXFLAGS = None
  cargo:rerun-if-env-changed=CXXFLAGS_aarch64_unknown_linux_gnu
  CXXFLAGS_aarch64_unknown_linux_gnu = Some()
  cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
  CC_SHELL_ESCAPED_FLAGS = None
  cargo:rerun-if-env-changed=CXXFLAGS_aarch64-unknown-linux-gnu
  CXXFLAGS_aarch64-unknown-linux-gnu = None
  cargo:warning=llvm-wrapper/PassWrapper.cpp:74:35: error: expected template-name before '<' token
  cargo:warning=   74 |     : public RequiredPassInfoMixin<ColdInvokeOutlinerPass> {
  cargo:warning=      |                                   ^
  cargo:warning=llvm-wrapper/PassWrapper.cpp:74:35: error: expected '{' before '<' token
  cargo:warning=llvm-wrapper/PassWrapper.cpp:74:35: error: expected unqualified-id before '<' token
  cargo:warning=llvm-wrapper/PassWrapper.cpp: In lambda function:
  cargo:warning=llvm-wrapper/PassWrapper.cpp:1364:55: error: invalid use of incomplete type 'struct {anonymous}::ColdInvokeOutlinerPass'
  cargo:warning= 1364 |               EarlyMPM.addPass(ColdInvokeOutlinerPass());
  cargo:warning=      |                                                       ^
  cargo:warning=llvm-wrapper/PassWrapper.cpp:73:8: note: forward declaration of 'struct {anonymous}::ColdInvokeOutlinerPass'
  cargo:warning=   73 | struct ColdInvokeOutlinerPass
  cargo:warning=      |        ^~~~~~~~~~~~~~~~~~~~~~

  --- stderr


  error occurred in cc-rs: command did not execute successfully (status code exit status: 1): LC_ALL="C" "sccache" "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-w" "-I/usr/lib/llvm-22/include" "-std=c++17" "-D_GNU_SOURCE" "-D_GLIBCXX_USE_CXX11_ABI=1" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-fno-exceptions" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_AVR" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_BPF" "-DLLVM_COMPONENT_COVERAGE" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LOONGARCH" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_M68K" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-DLLVM_COMPONENT_XTENSA" "-o" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage1-rustc/aarch64-unknown-linux-gnu/release/build/rustc_llvm/3bf93504a4a21980/out/ef10e86dc40538c1-PassWrapper.o" "-c" "llvm-wrapper/PassWrapper.cpp"


warning: build failed, waiting for other jobs to finish...
[RUSTC-TIMING] rustc_hir_typeck test:false 44.141
[RUSTC-TIMING] rustc_codegen_ssa test:false 16.154

Testing outlining

6040092

This comment has been minimized.

Sign in to view

fix test

b7b037b

This comment has been minimized.

Sign in to view

fix tests

9f53dd0

Navigation Menu

Repository navigation

Outline and deduplicate cold panic and drop paths in MIR and LLVM - #164821

Outline and deduplicate cold panic and drop paths in MIR and LLVM#164821
stephenduong1004 wants to merge 3 commits into
rust-lang:mainfrom
stephenduong1004:outlinepanic

stephenduong1004 commented Oct 23, 2026 •

edited by rustbot

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

stephenduong1004 commented Oct 23, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reproducer

Why MIR + LLVM Instead of Macro Outlining (#115562 / #115670 / #145304)?

What This PR Does

Next Steps

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stephenduong1004 commented Oct 23, 2026 •

edited by rustbot

Loading