Repository navigation
tile_detect: build the catalogue in memory, run SExtractor on node-local disk - #947
Open
cailmdaley wants to merge 3 commits into
Open
cailmdaley wants to merge 3 commits into
cailmdaley wants to merge 3 commits into
Conversation
make_post_process appended each EPOCH HDU through FITSCatalogue.save_as_fits, which flushed and reopened the ~670 MB uberseg catalogue per exposure, then rebuilt LDAC_OBJECTS for N_EPOCH with add_col, another whole-file write. It now reads the catalogue with astropy, as add_seg_vignet and match_catalogue do, builds the EPOCH HDUs and N_EPOCH in memory, and writes the file once. The columns are laid out by FITSCatalogue.fits_column (add_cols' column builder, now a public static method), so the file is byte-identical to the per-step route: a new test builds the reference through save_as_fits and add_col and compares bytes, and on DR6 191.307 (31,736 rows, 8 epochs) the output is byte-identical to the previous code's. Measured on that tile: 16.4 s -> 5.0 s node-local, 43.7 s -> 28.5 s on idle /scratch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW
On nibi tile_detect took 9-44 min per tile in single-allocation campaigns, of which SExtractor was 1-2 min: the SEG_VIGNET cut, the DR6 join and the post-processing each rewrite the whole catalogue (~670 MB under uberseg) on /scratch, which a loaded node slows ~30x (191.307 in smk-g13-us: 18, 15 and 9 min). Every tile's ngmix waits on this step. sextractor_runner takes an optional WORK_DIR: SExtractor writes its catalogue and check images in a fresh directory under it (sp-detect<num>.*, which tile_local's stale-store sweep also matches), the rewrites happen there, and the files are moved to the output directory once complete. The directory is removed on the way out, also on failure, so a failed run publishes nothing. Without WORK_DIR the runner writes in the output directory as before. config_tile_Sx.ini sets WORK_DIR = /local/scratch, which both profiles bind and the node-local tile store already uses. The ini is not part of params.pre, so the params pins do not move and running campaigns are not invalidated; the published files and the completeness counts are unchanged. Measured on DR6 191.307 under uberseg in one 2-core step: 36 s end to end through WORK_DIR against 112 s in place on idle /scratch (44 min in the loaded g13-us allocation). Catalogue, BACKGROUND and SEGMENTATION check images are identical to the in-place run's except SExtractor's own extraction time cards (SEXTIME, SEXELAPS) in LDAC_IMHEAD. tests/module/test_sextractor_work_dir.py checks the staged run publishes the in-place bytes, leaves the work directory empty, and publishes nothing when the join refuses the catalogue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW
…node-local # Conflicts: # workflow/rules/tile.smk
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
tile_detectfast on a loaded shared filesystem. Every tile's ngmix waits on this step, and on Nibi it took 9–44 min per tile, about 13% of a campaign's bill. No single step was slow. The problem was that each step rewrote the whole 672 MB catalogue on NFS.make_post_processbuilds the catalogue in memory and writes it once. Before, it saved and reopened the file for every epoch, then rebuilt it again for N_EPOCH.sextractor_runnertakes an optionalWORK_DIR, set to/local/scratchinconfig_tile_Sx.ini. SExtractor, the check images and the rewrites all run on node-local disk. Each file moves to the output directory once, and a failed run publishes nothing.Result: tile 191.307 took 36 s node-local, against 112 s on idle
/scratchand 44 min in the g13 campaign. Post-processing output is byte-identical to before. The other outputs match too, except SExtractor's timestamp header cards.Based on #925, which adds the
SEG_VIGNETfile handling this rewires; retarget to develop once #925 lands.Claude Opus 5.5 on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd