Repository navigation
ngmix: each chunk holds only its own rows' tile stamps - #940
Merged
Merged
Conversation
Tile_cat memory-maps the tile catalogue (and the segmentation vignet catalogue) and copies the stamps of the chunk's rows only, into a ChunkStamps indexed by tile-catalogue row; rows outside the chunk raise IndexError. The per-object columns (obj_id, ra, dec, flux) stay full length. Ngmix.process passes its ID_OBJ_MIN/ID_OBJ_MAX bounds to Tile_cat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs
cailmdaley
added a commit
that referenced
this pull request
Oct 5, 2026
Tile_cat takes no seg_cat_path: it reads row_min/row_max and holds both stamp columns of the tile catalogue, VIGNET and SEG_VIGNET, for the chunk's rows as ChunkStamps. test_tile_cat_reads_seg_vignet compares row by row; test_tile_cat_holds_the_stamp_columns_once gives way to test_ngmix_tile_cat_chunk.py, which now writes SEG_VIGNET into the tile catalogue (its separate-seg-catalogue alignment test goes with the separate catalogue). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd
martinkilbinger
approved these changes
Oct 6, 2026
cailmdaley
added a commit
that referenced
this pull request
Oct 6, 2026
#886 landed as a squash whose tree equals feat/wire-external-masks@baa6430c, already merged here; the three conflicts (params_pin.json, universes/committed.yaml, workflow/README.md) take this branch's side, which extends #886's. The merge adds exactly #940's ngmix change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Nfmxobnf8yQJoqdk6kUAx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each ngmix chunk now loads only its own rows' postage stamps, not the whole tile's.
Snakemake splits each tile into 16 ngmix chunks, but every chunk read all ~36K stamps (~360 MiB) into memory and kept them for the whole fit. Now
Tile_catmemory-maps the catalogue and copies just the chunk's rows (VIGNET, and seg vignets when present). Stamps stay indexed by tile row, so the rest of the code is unchanged; reaching outside the chunk raisesIndexError. WithoutID_OBJ_MIN/ID_OBJ_MAXthe whole catalogue loads as before.Result (tile 202.301, Nibi):
develop, on three small chunks and on a full 2,132-object chunk;New unit tests in
tests/module/test_ngmix_tile_cat_chunk.py; full suite passes.Conflicts with #925 in
Tile_cat; the fix is mechanical (keep the row range, readSEG_VIGNETthroughChunkStamps).🤖 Generated with Claude Code
https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd