Repository navigation
feat(preload): hold a schema's images on every worker ahead of its launches - #55
Merged
Merged
Conversation
instanceImageName built the repository inline. Preloading needs it on its own, to scope worker cleanup to one challenge's tags, so it is now imageRepository, and instanceImageName is that plus the content tag. The names it produces are unchanged.
hi-liang
force-pushed
the
feat/schema-preload
branch
from
October 15, 2026 04:32
663d52e to
28687cf
Compare
hi-liang
force-pushed
the
feat/schema-preload
branch
4 times, most recently
from
October 17, 2026 02:20
a1b566f to
219c46a
Compare
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
…unches A worker pulls an image the first time it launches it. That is the right mode for a catalogue too large to fit on one box, and costs a pull on each worker's first launch of each image: a warm launch never contacts the registry at all (the tag is checked on the worker's own daemon), so the cost is exactly those cold pulls, and it lands hardest when many players launch at once. A set that does fit on the workers need not pay it at all. A preloaded schema (the preloads table; AddPreload, RemovePreload, ListPreloads, put in the API in a later commit) has every image it launches pulled onto every worker ahead of time. It is state, not a one-shot, so it covers workers added later (worker-add, recovery, the autoscaler) and follows redeploys. Each worker runs a pass when it joins placement, when a schema is preloaded, as each build lands, and every five minutes as a backstop. A pass that fails -- a pull, the worker's image list, the database -- goes again after 30 seconds, doubling while it keeps failing, up to the sweep. A pass pulls what the worker lacks, one image at a time, outside the launch slots and under the restart pull ceiling, and never holds the worker out of placement: a box added under load serves at once, and its launches pull as they always have until the pass catches up. Each worker pulls in its own random order, and at most CORK_PRELOAD_PULLS (default 8) preload pulls run at once across the fleet. Before each pull the pass reads afresh whether the image is still wanted, so a preload removed, or an image superseded, while it runs is not pulled; a read that fails pulls nothing and fails the pass. That is also the difference in cleanup. An on-demand schema leaves it out of band: corkd removes no images, and docker-reaper evicts what goes unused. While a challenge is preloaded its cleanup is corkd's, and opportunistic: after its pulls, each worker deletes the tags in the challenge's repository that no current build launches, in any schema. An old version waits for a successor that landed too late for the pass to try, which the next pass pulls first: deleting it before would take the layers the two share, for that pull to fetch again. A failed pull holds nothing back, since on a full disk the delete is what makes room for the retry. The delete is never forced, so dockerd itself refuses it while a container uses the image -- an instance the redeploy has yet to stop or restart on the new version, a launch it overtook -- and corkd tags that image <repo>:stale-<tag> instead, a reference-store write, for a later pass, an operator or a reaper to delete once nothing uses it. Nothing is held or paced around these deletes. Measured on Docker 29.9 with dockerd's IO held to gp3's baseline, deleting a whole challenge image's own layers takes about 1s for a typical one and up to 6s for the heaviest, while the daemon's creates, image checks and lists and container removals wait; but a redeploy's old version shares most of its layers with the new one, which the worker by then holds, so the delete that follows it is small. Removals read the database afresh and take no lock: a rollback that lands meanwhile can at worst have an unused image deleted that has just become current again, which the next launch or pass pulls back. A launch caught in such a transition may pull cold or fail as retryable, which is preferred to any machinery that would protect it. A delete or stale tag that times out ends the worker's removals until its next pass, and none of the preloader's timeouts count as the worker's stalls. A failed delete waits for the five-minute sweep rather than the 30-second retry: deletes come last, and the usual failure is a busy daemon; the removal phase's own database read, failing, fails the pass like any other. What an operation takes out of preloading -- the preload removed, the schema deleted, a challenge dropped -- may stay on the workers rather than being tracked down: a stale image costs disk that a preloaded worker has room for and an operator can vet, where a wrong delete costs launches. corkd logs the images that may remain, read from its database without asking any worker (releasedImages, against what preloading covered when the operation began), and RemovePreload returns them. A preload ends with RemovePreload or DeleteSchema only: an update that leaves the schema empty, such as a converge that failed, keeps it for the schema's return, and the renumbering refusal of a hand-over says to preload-add again after the remove-schema it asks for. Every operation still takes the one plain updateMu, now through lockUpdates/unlockUpdates, which note what preloading covered before it and keep the preloading flag current. initWorkers now registers its workers under workersMu, since each conn's goroutine starts as it is made, and Build draws its schema name under randMu, which the preloader's shuffle now shares. cork can see counts, not bytes: sizing the workers' image store for the set, plus a second version of what one update redeploys, stays the operator's, as does turning docker-reaper's image eviction off on those workers and back on after the preload is removed. Unit tests cover the planning, including a tag an on-demand schema launches in a preloaded challenge's repository, which stays, and an old version that waits for a successor landing mid-pass but not for a failed pull; the sets read from the database; add, remove and delete, and what each releases; and passes against a fake daemon: a worker admitted to placement starting its preloader, the pull cap, the worker re-checked after the wait for a fleet slot, a pass that stops pulling what is no longer wanted, one that cannot read the database before a pull or before its removals and so fails, the retry, its backoff and its reset, unforced deletes and the stale tag on an image in use, timeouts on either that neither eject the worker nor go on, an unreachable worker left alone, kicks from preload-add and landing builds only, and a preload that outlives an emptied schema. The fake answers as dockerd 29.9.1 was found to where the code depends on it: 404 for a missing tag, 409 for an image's last tag while a container uses it, the tag gone before a slow delete answers, and a tag that moves. Each guard's test fails with the guard removed, and the cork package passes under the race detector.
POST /preloads {"schema": "<name>"}, DELETE /preloads/<name> and GET
/preloads put the preloading of the previous commit in corkd's API,
and cork preload-add, preload-remove and preload-list in front of it. An
unknown schema is a 404, a daemon without a challenge registry a 409.
preload-add answers whether any worker is registered, and the CLI says
when none is: the preload then waits for workers to be added, which is
otherwise indistinguishable from it working. preload-list shows, per
schema and worker, how many of its images the worker was last seen to
hold, the pull in progress for that schema, failed pulls, when the
worker was last checked, and "ready" once it holds them all and has
been checked ("no images" for a schema left with no builds).
preload-remove answers with the images corkd no longer cleans up
({"released": [...]}), and the CLI lists them as images that may
remain on the workers, since they come from the database, not from
asking every worker.
The README and both usage texts describe the two modes by what selects
them -- whether a schema's images fit on a worker -- rather than by any
one use, and what each leaves to whom: on demand, corkd removes no
images and docker-reaper evicts; preloaded, cleanup is corkd's while the
preload lasts, an old version deleted after any successor that landed
meanwhile is pulled, never forced, with an image still in use kept
under a stale tag, and what leaves preloading may stay for an
operator. They also say what preloaded workers need (room for the set
plus a second version of what one update redeploys, docker-reaper's
image eviction off, and back on after preload-remove), that a worker
serves without pulling only once it shows ready, how to list stale
tags (a reference filter's * stops at a slash, so it is a grep), and
CORK_PRELOAD_PULLS. Tests pin the handler's status codes and when
preload-list says ready.
A full-mode step after generation 5, against real dockerd and zot, which is where the image names a worker lists and the unforced removals have to match what cork computes, and where dockerd's refusal to delete an image in use is real rather than the unit tests' fake. It takes binex101's image off both workers first, so the wait that follows cannot pass on a fleet that already held everything; then preload-add the e2e schema and waits for GET /preloads to report both workers holding all three images the schema launches (the flag-only challenge launches none), checking each daemon holds them too and that preload-list shows both ready. A container corkd does not manage then holds generation 5 of the on-demand challenge on one worker, and the step redeploys it: both workers must come to hold the new tag and drop generation 5's, the one with the container keeping the image as <repo>:stale-<tag> (the other may too, when its pass beats the rebuild's stop of its own instance), while zot keeps generation 5 as the rollback target and the other challenges' images stay. Once the container is removed, preload-add again (idempotent; it asks for a pass) must see the stale tag deleted from both. preload-remove hands the schema back and must list every image it leaves, and the step leaves tracked instances on both workers as generation 5 did. The removal of the second schema, also full mode, now preloads it first and waits for both workers to hold its two runtime images. After remove-schema they must still be on both, since what leaves preloading is an operator's to vet, and the step removes them by hand. The redeploy uses a generation number of its own (50): the base-pins step builds generation 6 next and fails if its source change is already in place, which is how the first run of this step found out.
hi-liang
force-pushed
the
feat/schema-preload
branch
from
October 17, 2026 02:53
219c46a to
e54e8f3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second way for workers to hold challenge images, chosen per schema by whether its images fit on a worker:
cork preload-add <schema>): every worker, including ones added later, pulls the schema's whole set ahead of time, so a worker that showsreadyinpreload-listlaunches without pulling. A redeploy's new images are pulled as each build lands.While a challenge is preloaded, cleanup is corkd's and opportunistic:
<repo>:stale-<tag>instead; a later pass deletes it once nothing uses it.preload-remove, a deleted schema, a dropped challenge) may stay on the workers; corkd logs the images that may remain, andpreload-removelists them. A preload ends only withpreload-removeor the schema's deletion, so an update that fails and leaves the schema empty keeps it.Load on live workers:
CORK_PRELOAD_PULLS(default 8). Each pull first checks its image is still wanted, so preload-remove stops a pass that is already pulling; a check that cannot read the database pulls nothing.Operator notes: size preloaded workers for the set plus one extra version of whatever an update redeploys, and turn docker-reaper's image eviction off on them (companion ansible-roles PR #46). List stale tags with
docker images --format '{{.Repository}}:{{.Tag}}' | grep ':stale-'; a rare race between a delete and a launch can leave an untagged image, whichdocker image pruneremoves.History was rewritten since the last review: forced removal, its pacing and the 503 refusal of overtaken launches are gone. Commits, each of which builds and passes unit tests:
imageRepositoryTested with: