Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths. - #998
Draft
copybara-service[bot] wants to merge 1 commit into
Draft
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
…aths.
FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).
Dispatch directly from FreeSmallSlow instead:
- HaveHooks() -> FreeSmallHooked (hook + DeallocateSlow, or the
per-thread callee if per-CPU is inactive),
- !UsePerCpuCache() -> FreeSmallPerThread (ThreadCache::Deallocate /
transfer cache only, no hook plumbing),
- otherwise -> inlined DeallocateSlowNoHooks (unchanged).
Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Sampled-object
handling is untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.
x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
after FreeSmallPerThread: 32 insns, 9 loads, 5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).
PiperOrigin-RevId: 983583456
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths.
FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).
Dispatch directly from FreeSmallSlow instead:
per-thread callee if per-CPU is inactive),
transfer cache only, no hook plumbing),
Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Sampled-object
handling is untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.
x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
after FreeSmallPerThread: 32 insns, 9 loads, 5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).