You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(racer): hold warmed pages on each page's primary owner - #954
Cache warming now stores and hard-holds each warmed page on its primary owner in the peer mesh, instead of forcing every page onto the requesting node's disk. Pages whose primary owner is the requester stay local as before. A HEAD warm reserves holds on every involved owner up front through new peer Warm hold, fill, and release operations, and is refused if any owner declines. Reads still wait on the warm frontier, then use the normal fill path to reach the owner's held copy. Holds are leased per requester and job, are not re-homed when membership changes, and require all nodes to run a version that understands the new peer operations.
Add a Warm peer operation with hold, fill, and release actions plus
held, warmed, and released acknowledgements. A hold carries the object
page count and a wall-clock expiry instead of a page list; the owner
derives its own primary pages. Acknowledgements are destination-only
and must match the requested action, and a held expiry may never exceed
the request. Receivers reject warm operations until they are served.
A warm HEAD now plans each page's primary owner under the admission
membership. Pages this node owns are held and filled locally as before;
every remote primary owner must accept a lease-scoped hold before the
HEAD succeeds, and the warm job asks that owner to fill each page and
acknowledge publication without sending the payload. Any refused hold
or persistently failed fill releases the job's leases. Reads still wait
on the warm frontier and then read remote pages through normal Fill.
Owners serve warm requests only for pages they are the primary owner
of, track leases per requester and job, and release unused capacity
only after the last live lease is gone.
Add requester tests for remote-only placement, all-or-nothing refusal
with releases to every owner, and a failing owner fill that fails the
job and releases its hold. Add an owner test for non-primary refusal,
lease-gated fills, and capacity shared across two requester leases.
Rewrite the warm section for primary-owner holds, all-or-nothing
admission, acknowledgement-only remote fills, leases, and the limits on
membership drift, clock skew, and mixed versions. List the new remote
and owner warm counters and note owner placement in the SDK Warm doc.
Preserve the completing job when determining return placement
cmd/racer-dataplane/src/read/dispatch.rs:537
Return placement from the same job that satisfied wait. The wait retains one Job (warm.rs:591-606), but this locality check looks up the current job again (warm.rs:319-322). Another worker can replace a completed remote job with a new job whose plan is still None, or remove an expired job, between these calls. The lookup then defaults to local, so a read can fail with Unavailable when only the remote owner has the page (fill.rs:115-125). Carry the retained job's local/remote outcome through the wait result and add a replacement-race regression test.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Reservation cleanup can race with pending holds, and hold recovery introduces quadratic work.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Fence hold reservations against concurrent release cleanup
cmd/racer-dataplane/src/read/warm.rs:1493
Fence reservation against a concurrent Release of the same lease. After Hold grants the lease and awaits worker reservations, Release can remove the last lease and queue cleanup. Workers process cleanup before queued Hold work (read/dispatch.rs:853-860), so cleanup can finish before Hold recreates the reservation. The owner can then acknowledge Hold with no live lease, leaving unused capacity reserved until the hold expires. Coordinate pending reservations with release, or recheck lease state after reservation and queue generation-fenced cleanup when needed. Cover this ordering with a test.
Avoid quadratic held-page reconstruction during recovery
cmd/racer-dataplane/src/store/catalog.rs:494
Recovery visits pages in hash order (alloc/src/index.rs:60-61), but each out-of-order HoldPages::insert shifts a vector suffix (store/holds.rs:43-47). Rebuilding a large held version therefore takes quadratic work on the synchronous recovery path, slowing restarts. Collect the locations first, then construct each page set once with FromIterator, which sorts once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cache warming now stores and hard-holds each warmed page on its primary owner in the peer mesh, instead of forcing every page onto the requesting node's disk. Pages whose primary owner is the requester stay local as before. A HEAD warm reserves holds on every involved owner up front through new peer Warm hold, fill, and release operations, and is refused if any owner declines. Reads still wait on the warm frontier, then use the normal fill path to reach the owner's held copy. Holds are leased per requester and job, are not re-homed when membership changes, and require all nodes to run a version that understands the new peer operations.