Skip to content

Skip free space check when overlays are used. - #3203

Open
3405691582 wants to merge 2 commits into
google:mainfrom
3405691582:space_space
Open

3405691582 wants to merge 2 commits into
google:mainfrom
3405691582:space_space

Conversation

@3405691582

Copy link
Copy Markdown
Collaborator

Currently, after fetch, disk images are filesystem-sparse, so we check free space available to guard against the case where regular instance usage depletes free space that was not actually reserved, potentially leaving the disk images in an incomplete state.

This check is problematic however when overlay images are used. As written, the check is for free space on the filesystem containing the base images, not the overlay images. Worse, since the overlay images need to potentially capture all disk mutations, the free space required needs to potentially balloon to the entire disk size, which is a more difficult limit to mandate.

While the same errant state when disk space is depleted may manifest, potential corruption is limited to the overlay only, so instead, just skip the check entirely when overlays are used.


Review note: one commit is for code motion, the other has the conditional that skips the check; it will be easier to read these separately.

No functional change.
Currently, after fetch, disk images are filesystem-sparse, so we check
free space available to guard against the case where regular instance
usage depletes free space that was not actually reserved, potentially
leaving the disk images in an incomplete state.

This check is problematic however when overlay images are used. As
written, the check is for free space on the filesystem containing the
base images, not the overlay images. Worse, since the overlay images
need to potentially capture all disk mutations, the free space required
needs to potentially balloon to the entire disk size, which is a more
difficult limit to mandate.

While the same errant state when disk space is depleted may manifest,
potential corruption is limited to the overlay only, so instead, just
skip the check entirely when overlays are used.
@3405691582
3405691582 requested a review from Databean September 18, 2026 18:52
@Databean

Copy link
Copy Markdown
Member

While the same errant state when disk space is depleted may manifest, potential corruption is limited to the overlay only, so instead, just skip the check entirely when overlays are used.

I don't see how this conclusion follows from the premises. With both filling in linux-sparse files or appending to qcow2 overlays, running out of disk space on the host results in crosvm getting IO errors in response to writing to a file on the host, and reporting those errors to the guest through errors in virtio-block. The outcome is the same, that after the guest spends some time doing writes, writes suddenly start failing with no way to recover.

Another motivation for the pessimistic check is that IIRC GCE gives IO read/write throughput budgets based on total disk size, so being pessimistic also results in having more IO budget to work with.

@3405691582

Copy link
Copy Markdown
Collaborator Author

I am assuming the purpose of the check is to guard against corruption of the base images; I could be wrong though. Without overlays, a bad write in the image that triggers corruption causes a problem in the base image. At least if that happens with the overlay, we can throw the overlay out and the base image is usable again, albeit reinitialized to base state. We could potentially checkpoint the overlays, if we aren't doing something like that already.

I do wonder about diminishing returns however: at some point if you run out of disk space, there's not as much we can do to be that defensive about it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants