Make `Rc<T>::deref` zero-cost by EFanZh · Pull Request #141348 · rust-lang/rust

EFanZh · 2025-05-21T14:14:40Z

This PR makes Rc::deref zero-cost by changing the internal pointer to point directly to the value instead of to the allocation.

This PR is split from #132553, which will also make Arc::deref zero-cost.

Review status:

rustbot · 2025-05-24T10:22:23Z

The Miri subtree was changed

cc @rust-lang/miri

Make `Rc<T>::deref` zero-cost This PR makes `Rc::deref` zero-cost by changing the internal pointer so that it points to the value directly instead of the allocation. This is split out from #132553, which will also make `Arc::deref` zero-cost.

tgross35 · 2026-10-11T04:47:07Z

Hopefully rust-lang/rustc-perf#2615 will give some better hints about exactly where things are slowing down.

tgross35

So you can run perf yourself and experiment,

@bors delegate=try

I know the patterns in the current code have been optimized about as much as possible, any kind of perturbation may easily have negative effects on performance. The best thing to do is probably to slowly walk back changes to make the code's shape look more like what's currently there. If that means moving more out of common.rs and back into rc.rs then that's fine, more refactoring can happen after the two big PRs merge.

View changes since this review

tgross35 · 2026-10-11T09:41:44Z

    pub fn new_uninit() -> Rc<mem::MaybeUninit<T>> {
-        // ignore-tidy-undocumented-unsafe
-        unsafe {
-            Rc::from_ptr(Rc::allocate_for_layout(
-                Layout::new::<T>(),
-                |layout| Global.allocate(layout),
-                <*mut u8>::cast,
-            ))
-        }
+        Self::new_uninit_in(Global)
    }


Could you try adding this pattern back with allocate_for_layout and try_allocate_for_layout? With API in common:

fn rc_inner_layout_for_value_layout(layout: Layout) -> Layout; impl RcHeader { unsafe fn allocate_for_layout( value_layout: Layout, allocate: impl FnOnce(Layout) -> Result<NonNull<[u8]>, AllocError>, mem_to_rc_inner: impl FnOnce(ErasedRcValuePointer) -> RcValuePointer<T>, ) -> RcValuePointer<T>; }

I think we may be spending more time/complexity doing layout calculation and allocation while the existing pattern works quite well.

I inspected the generated assembly locally, I think the majority difference is caused by different inlining strategy and extra pointer offset operations. Most of the test cases use sized types, I ensured the layouts are computed at compile time, so the layout calculation are not likely the main cause.

I am now trying different optimization strategies to see if they improve the perf result.

I think we should keep this pattern anyway, actually. It's still easier to follow than what is currently in this PR, and we still want RcLayout to go away which this will help with.

At least we should have some better perf numbers before that with the added benchmarks.

rust-bors · 2026-10-11T10:10:10Z

✌️ @EFanZh, you can now perform try builds on this pull request!

Warning

You used the legacy format of the delegate command.
The new format is @bors delegate[=<username>] [<try|review>].

You can now post @bors try to start a try build.

tgross35 · 2026-10-14T19:20:06Z

Ah can't work with the old build, @EFanZh could you rebase?

tgross35 · 2026-10-15T18:42:28Z

@bors try @rust-timer queue

rust-bors · 2026-10-15T19:41:04Z

☀️ Try build successful (CI)
Build commit: 9e78396 (9e78396fdec429ba7badc774e4d66429ee890fd3)
Base parent: 56df459 (56df4595f93c41b7d956612e6211f9a274cc6a5d)

rust-timer · 2026-10-15T20:33:51Z

Finished benchmarking commit (9e78396): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.1%, 0.3%]	11
Regressions ❌ (secondary)	6.5%	[0.3%, 13.3%]	7
Improvements ✅ (primary)	-0.8%	[-1.0%, -0.6%]	3
Improvements ✅ (secondary)	-0.6%	[-0.6%, -0.6%]	1
All ❌✅ (primary)	0.0%	[-1.0%, 0.3%]	14

Max RSS (memory usage)

Results (primary -0.7%, secondary 2.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.3%, 2.3%]	1
Regressions ❌ (secondary)	3.7%	[1.8%, 9.4%]	8
Improvements ✅ (primary)	-3.7%	[-3.7%, -3.7%]	1
Improvements ✅ (secondary)	-3.5%	[-3.5%, -3.5%]	1
All ❌✅ (primary)	-0.7%	[-3.7%, 2.3%]	2

Cycles

Results (primary 2.6%, secondary 7.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	7.0%	[1.8%, 15.6%]	7
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.6%	[2.6%, 2.6%]	1

Binary size

Results (primary 0.1%, secondary 1.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.0%, 1.0%]	6
Regressions ❌ (secondary)	7.0%	[0.0%, 10.1%]	5
Improvements ✅ (primary)	-1.2%	[-1.6%, -0.9%]	2
Improvements ✅ (secondary)	-5.8%	[-8.9%, -4.8%]	4
All ❌✅ (primary)	0.1%	[-1.6%, 1.0%]	8

Bootstrap: 489.662s -> 490.032s (0.08%)
Artifact size: 406.87 MiB -> 406.95 MiB (0.02%)

tgross35 · 2026-10-16T08:33:48Z

Perf results at #141348 (comment) actually look pretty good, all things considered. There are some regressions for more niche issues, but the runtime benchmarks at https://perf.rust-lang.org/compare.html?start=56df4595f93c41b7d956612e6211f9a274cc6a5d&end=9e78396fdec429ba7badc774e4d66429ee890fd3&stat=instructions%3Au&tab=runtime&nonRelevant=true&runtimeName=rc_ are pretty positive.

It would be nice if we could shave off the 13% regression (maybe by getting some alloc computations back?) but don't think that's worth blocking on once we get the rest of this reviewed.

View changes since the review

@tgross35: the 13% regression is a debug build. I inspected it locally, and think it was mainly caused by debug assertions in various unchecked calls, I replaced them with the ones from the intrinsics mod, which skip debug assertions, and my local result shows improvement.

Good idea for an experiment, we'll just have to remember to put it back once verifying that's where it comes from. (We shouldn't use intrinsics directly since the assertions in the unsafe wrappers are still useful in library API.)

tgross35 · 2026-10-16T08:39:17Z

This test is a good idea, but could you put this into a separate PR? So we can see what this changes as a result of this one.

It would be good to also include an x86-only assembly test, covering most of the functions at https://rust.godbolt.org/z/TrbvGr4a8 for both Rc and Arc (excluding rc_new_slice because that's too big).

View changes since the review

@tgross35: Do you have a preference for the file layout for that PR? The current file have optimization in its name, I think the baseline probably need something else.

The same naming is fine, as even without this PR we're checking how Rc and Arc optimize. Or just rc-arc

tgross35 · 2026-10-16T08:52:58Z

+    #[inline(always)]
+    pub(super) fn try_allocate_uninit_for_type<A, T>(
+        alloc: &A,
+    ) -> Result<RcValuePointer<T>, AllocError>
+    where
+        A: Allocator,
+    {
+        Ok(try_allocate_uninit(alloc, RcLayout::new::<Self, T>())?.unerase())
+    }
+
+    #[inline(always)]
+    pub(super) fn try_allocate_zeroed_for_type<A, T>(
+        alloc: &A,
+    ) -> Result<RcValuePointer<T>, AllocError>
+    where
+        A: Allocator,
+    {
+        Ok(try_allocate_zeroed(alloc, RcLayout::new::<Self, T>())?.unerase())
+    }


Can you restore the previous allocate_for_layout pattern found at

rust/library/alloc/src/rcs/rc.rs

Lines 2375 to 2424 in 3e014ee

impl<T: ?Sized> Rc<T> {

/// Allocates an `RcInner<T>` with sufficient space for

/// a possibly-unsized inner value where the value has the layout provided.

///

/// The function `mem_to_rc_inner` is called with the data pointer

/// and must return back a (potentially fat)-pointer for the `RcInner<T>`.

#[cfg(not(no_global_oom_handling))]

unsafe fn allocate_for_layout(

value_layout: Layout,

allocate: impl FnOnce(Layout) -> Result<NonNull<[u8]>, AllocError>,

mem_to_rc_inner: impl FnOnce(*mut u8) -> *mut RcInner<T>,

) -> *mut RcInner<T> {

let layout = rc_inner_layout_for_value_layout(value_layout);

// ignore-tidy-undocumented-unsafe

unsafe {

Rc::try_allocate_for_layout(value_layout, allocate, mem_to_rc_inner)

.unwrap_or_else(|_| handle_alloc_error(layout))

}

}

/// Allocates an `RcInner<T>` with sufficient space for

/// a possibly-unsized inner value where the value has the layout provided,

/// returning an error if allocation fails.

///

/// The function `mem_to_rc_inner` is called with the data pointer

/// and must return back a (potentially fat)-pointer for the `RcInner<T>`.

#[inline]

unsafe fn try_allocate_for_layout(

value_layout: Layout,

allocate: impl FnOnce(Layout) -> Result<NonNull<[u8]>, AllocError>,

mem_to_rc_inner: impl FnOnce(*mut u8) -> *mut RcInner<T>,

) -> Result<*mut RcInner<T>, AllocError> {

let layout = rc_inner_layout_for_value_layout(value_layout);

// Allocate for the layout.

let ptr = allocate(layout)?;

// Initialize the RcInner

let inner = mem_to_rc_inner(ptr.as_non_null_ptr().as_ptr());

// ignore-tidy-undocumented-unsafe

unsafe {

debug_assert_eq!(Layout::for_value_raw(inner), layout);

(&raw mut (*inner).strong).write(Cell::new(1));

(&raw mut (*inner).weak).write(Cell::new(1));

}

Ok(inner)

}

}

? That removes the need of a lot of these single-line functions as well as for RcLayout, and goes a bit further to reduce the diff.

Those new functions can still live in common as methods on RcHeader so they can be reused by Arc.

View changes since the review

@tgross35: I can try restore allocate_for_layout to check the perf, but may I ask the reason behind removing RcLayout but keeping RcValuePointer? In my opinion, they play the same role of differentiating value properties from allocation properties.

It's difficult to say, but the gist is that we felt RcLayout doesn't really do enough to be needed. It's only used in common and pretty much always constructed in a module-public function, then immediately passed to a module-private function that deconstructs it. The struct wrapper doesn't really prevent much more than calling a layout function, but it adds more indirection and type invariants to think about.

RcValuePointer OTOH is a smaller abstraction that winds up used more, e.g. passed through API surface across common, rc, and arc. It also gets stored in some structs where it would be easier to forget whether a NonNull<T> can access the header.

rustbot · 2026-10-17T10:03:31Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

EFanZh · 2026-10-17T10:08:23Z

So you can run perf yourself and experiment.

@tgross35: My previous attempt of a perf run got rejected: #141348 (comment). Did I use the wrong command or do I also need the timer permission?

tgross35 · 2026-10-19T09:03:45Z

@bors try @rust-timer queue

tgross35 · 2026-10-19T09:06:41Z

So you can run perf yourself and experiment.

@tgross35: My previous attempt of a perf run got rejected: #141348 (comment). Did I use the wrong command or do I also need the timer permission?

I guess those can't be delegated. You can ask at #t-compiler/help > perf run whenever you're ready for a perf run.

rust-bors · 2026-10-19T10:03:24Z

☀️ Try build successful (CI)
Build commit: 7384739 (7384739eba9948d19c26afd3fdfd521b9e42ec3b)
Base parent: 588c7e9 (588c7e92f931498b1552ee64cad77fb0344cabfa)

rust-timer · 2026-10-19T10:48:43Z

Finished benchmarking commit (7384739): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=iffy rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.1%, 0.5%]	4
Regressions ❌ (secondary)	3.4%	[0.4%, 6.9%]	8
Improvements ✅ (primary)	-0.5%	[-0.5%, -0.5%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[-0.5%, 0.5%]	5

Max RSS (memory usage)

Results (primary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.4%	[-2.4%, -2.4%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.1%	[-2.4%, 2.6%]	2

Cycles

Results (secondary 3.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.8%	[1.6%, 6.3%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (primary 0.2%, secondary -0.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.7%	[0.5%, 0.9%]	2
Regressions ❌ (secondary)	2.0%	[1.9%, 2.0%]	4
Improvements ✅ (primary)	-0.7%	[-0.7%, -0.7%]	1
Improvements ✅ (secondary)	-3.8%	[-6.1%, -3.1%]	4
All ❌✅ (primary)	0.2%	[-0.7%, 0.9%]	3

Bootstrap: 493.578s -> 488.646s (-1.00%)
Artifact size: 409.12 MiB -> 409.13 MiB (0.00%)

rustbot assigned joboet May 21, 2025

rustbot added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels May 21, 2025

This comment has been minimized.

Sign in to view

EFanZh force-pushed the zero-cost-rc-deref branch from df34f84 to d3a7429 Compare May 24, 2025 05:01

This comment has been minimized.

Sign in to view

EFanZh force-pushed the zero-cost-rc-deref branch 2 times, most recently from bc84ec6 to 19fb34b Compare May 24, 2025 09:00

EFanZh marked this pull request as ready for review May 24, 2025 10:22

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 24, 2025

EFanZh mentioned this pull request May 24, 2025

Make Rc<T>::deref and Arc<T>::deref zero-cost #132553

Open

teor2345 reviewed May 26, 2025

View reviewed changes

Comment thread library/alloc/src/raw_rc/mod.rs Outdated

EFanZh force-pushed the zero-cost-rc-deref branch from 19fb34b to f5245ba Compare May 26, 2025 15:02