Skip to content

Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths. - #998

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983583456
Draft

copybara-service[bot] wants to merge 1 commit into
masterfrom
test_983583456

Conversation

@copybara-service

Copy link
Copy Markdown

Split FreeWithHooksOrPerThread into per-thread and hooked free slow paths.

FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).

Dispatch directly from FreeSmallSlow instead:

  • HaveHooks() -> FreeSmallHooked (hook + DeallocateSlow, or the
    per-thread callee if per-CPU is inactive),
  • !UsePerCpuCache() -> FreeSmallPerThread (ThreadCache::Deallocate /
    transfer cache only, no hook plumbing),
  • otherwise -> inlined DeallocateSlowNoHooks (unchanged).

Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Sampled-object
handling is untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.

x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
after FreeSmallPerThread: 32 insns, 9 loads, 5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).

…aths.

FreeSmallSlow routed both "delete hooks installed" and "per-CPU caches
inactive" into one NOINLINE callee. That callee unconditionally built a
DeleteInfo (class_to_size load + 4 stack stores), tested delete_hooks_, then
re-tested CpuCacheActive before falling into the inlined ThreadCache::
Deallocate. Per-thread-mode binaries therefore paid the hook plumbing on
every free, and the hooked per-CPU path carried the ThreadCache body in its
frame (3 callee-saved pushes, 0x38 stack).

Dispatch directly from FreeSmallSlow instead:
 - HaveHooks()          -> FreeSmallHooked (hook + DeallocateSlow, or the
                           per-thread callee if per-CPU is inactive),
 - !UsePerCpuCache()    -> FreeSmallPerThread (ThreadCache::Deallocate /
                           transfer cache only, no hook plumbing),
 - otherwise            -> inlined DeallocateSlowNoHooks (unchanged).

Both are tail calls out of FreeSmallSlow; its hot per-CPU body is unchanged
(register renaming only). Routing still keys on HaveHooks() rather than
delete_hooks_.empty() because new-hook-only installs still need
DeallocateSlow -> MaybeForceSlowPath to uncache the slab. Sampled-object
handling is untouched. The FreeSmall inline body and fast_path goldens are
byte-identical.

x86-64 -c opt, per-thread free (thread cache present, no overflow), callee
after FreeSmallSlow's dispatch:
  before FreeWithHooksOrPerThread: 51 insns, 15 loads, 10 stores, 4 pushes
  after  FreeSmallPerThread:       32 insns,  9 loads,  5 stores, 1 push
Hooked per-CPU path: same instruction sequence, one fewer ptr spill/reload;
function shrinks 402 -> 249 bytes. FreeSmallSlow +16 bytes (second tail
call).

PiperOrigin-RevId: 983583456
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant