Repository navigation
Share halide-cache entries through a server - #10
Conversation
|
|
||
| pub fn retrieve(&self, address: &Address, destination: &Path) -> Result<()> { | ||
| let mut source = self.root.join(shard_path(address, 2)); | ||
| pub fn store_raw<R: Read>(&self, address: &Address, mut blob: R) -> Result<bool> { |
There was a problem hiding this comment.
I would not add a second method that stores data into the lager. In this case we can easily generalize lager to make it fit both. You already have most of it: the write_entry method. IMO this is the shape that lager's API should have.
The concept of a lager in my mind is a space into which you store things. The important things the lager does for you is to keep track of where you stored the things, and in this case also keep track of which inventory has been sitting in the lager for the longest.
It does not have to include the concept of what form the inventory is stored into, and how to extract the inventory from the lager, which it currently does, and this PR adds a second way via the raw methods.
There was a problem hiding this comment.
I would not add a second method that stores data into the lager. In this case we can easily generalize lager to make it fit both. You already have most of it: the
write_entrymethod. IMO this is the shape that lager's API should have.
imho that is only valid if lager is meant to be an external crate because this just pushes the work to the client. This is just something I extended from what already existed. I did the change you requested (thanks LLMs) but it should have been a nitpick.
| pub fn store_at( | ||
| &self, | ||
| address: &Address, | ||
| write: impl FnOnce(&mut File) -> std::io::Result<()>, |
There was a problem hiding this comment.
nit: can be more generic: &mut dyn std::io::Write instead of File.
There was a problem hiding this comment.
johningve
left a comment
johningve
left a comment
There was a problem hiding this comment.
Haven't looked too closely at the server, but LGTM
The lager keeps track of where entries are and which have gone unused the longest. What an entry holds, and how it is packed, is now the caller's business: - store_at(address, write) writes an entry through a closure, atomically, and reports whether it is new - retrieve(address) opens an entry for reading and refreshes its LRU time Entries no longer have an extension. The scan removes entries left in an older format, which the new key scheme never hits anyway. halide-cache packs the files of an entry into a deterministic zstd-compressed tar archive, in a fixed order, and unpacks them by position. Prework for the server, which stores and serves entries as they are.
The object and header are always produced together, so this commit changes how halide-cache stores, restores and later: fetches, uploads and evicts them together with the same address in lager.
Entries are about to be shared, so the key must drop what differs between machines and include what makes an object valid elsewhere: - strip the base directory wherever it occurs in an argument and normalise path separators - prefix a key scheme version, and include the host OS and architecture - hash the builder executable, the Python interpreter, resolved through PATH. Ask it where it imports halide from and hash that package's native libraries. A missing or unusable halide is an error rather than a weaker key. - reject non-UTF-8 output paths instead of hashing a lossy conversion
Make local cache cleanup best effort, since Windows refuses to delete an entry another parallel build has open and the outputs are already in place. Run the client's tests on Windows in CI.
An HTTP server sharing cache entries between machines, storing them
through lager in the same layout as the local cache.
GET/PUT /v1/blobs/{address}
Ships a systemd unit, and static musl binaries for x86_64 and aarch64 in
CI and releases to be self-contained and independent of the Glibc
version of the host.
Look locally, then on the server, then build and upload. A remote hit is stored locally and triggers the local LRU cleanup. The remote is fail-open: connection errors, failed uploads and entries that do not decompress are warnings, and the build proceeds locally. The end-to-end tests start the server and cover hits across machines, a server that is down, corrupt entries, oversized uploads, and concurrent clients racing on one entry under constant eviction. They run on Unix only, since the server is deployed on Linux.
2 participants
This is the meat, best reviewed commit by commit. We introduce a new halide-cache-server binary and the corresponding
--remoteflag on the client.