Repository navigation
Treevox: never let a voxelization batch span two chunk stripes - #731
Conversation
Chunks are ordered in 2x2 blocks and cut into batches of num_workers. With an odd number of chunk columns the last block of each stripe has two chunks, so the next batch wrapped into the following stripe and its union read covered the full grid width (7.7x a normal batch at 11 columns). That batch OOM-killed treevox at 16 GiB on ~700M-voxel grids. End each batch at a stripe boundary; chunk order is unchanged.
|
Verified on Cloud Run, plus one correction to the description. Setup: the 2026-10-15 crashed job, re-run on a temporary private service at prod size (4 vCPU / 16 GiB, 1 instance). Same 251,198-tree inventory, 6×11 chunks, 50×2674×5224 at 1 m, with the user's bands (
The pre-fix run survived this time (2 of 3 identical jobs survived on 10-10). Cloud Run's per-minute memory metric (p99 42% vs 32%) is too coarse to show the seconds-long spike. The oversized batch is gone. Correction: the description says output is unchanged; that's true only up to float32 rounding. With |
Closes #729.
Treevox orders chunks in 2x2 blocks and cuts the list into batches of
num_workers. Each batch is read and written as one bounding box. With an odd number of chunk columns, the last block of each two-row stripe has only two chunks, so the next batch wrapped into the following stripe (e.g.(0,10), (1,10), (2,0), (2,1)), and its bounding box covered the full grid width:All four treevox OOM kills in the last 30 days (2026-10-10 ×3, 2026-10-15 ×1) were 6×11-chunk grids that died in batch 6/17, the wrapping batch.
Changes
voxelize.chunk_batchescuts the block-ordered chunk list into batches of up tobatch_size, ending a batch at every stripe (row // 2) boundary. With 4 workers, every batch is now one 2x2 block or, at the end of a stripe, one 2x1 block. The 2× batches that followed the wrap are gone too._run_voxelization_batchesiterates those batches. The number of batches can rise slightly (17 → 18 for 6×11).Chunk order is unchanged, so output is unchanged: additive bands sum the same contributions, and overwrite bands still resolve last-writer-wins in chunk order whether two chunks share a batch or not. (Within a batch, a worker whose write equals the pre-batch value doesn't overwrite. That edge case is already documented on
masked_merge.)Tests
tests/test_voxelize.py::TestChunkBatchesuses the prod shape (6×11 chunks, 2674×5224 cells, chunk_xy 500). It checks that the batches keep chunk order and coverage, that no batch spans two stripes, and that no bounding box exceeds one 2x2 block plus halo. It also checks that even column counts match the old slicing, plus a single-row grid and an empty list.tests/test_voxelize.pyandtests/handlers/test_voxelize.pypass: 256 tests.Not verified on Cloud Run: the memory reduction follows from the bounding-box sizes and was not measured in a deployed run.