UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) by cailmdaley · Pull Request #925 · CosmoStat/shapepipe

cailmdaley · 2026-09-30T00:24:04Z

Makes UberSeg blend handling run end to end. Each object's segmentation stamp (SEG_VIGNET) is cut from the tile SExtractor run, and blend_handling: uberseg is the default.

Tile 202.301, a blended galaxy. Left: VIGNET, with the brighter neighbour masked (orange). Middle: SEG_VIGNET. Right: the UberSeg weight. Pixels nearer the neighbour's footprint than the galaxy's own get zero weight.

Motivation and how the work is organised

Blending is handled in two steps. #922 (masked pixels, which this PR builds on and contains) gives ngmix a per-pixel mask and handles neighbours and detector defects separately. UberSeg is the neighbour option that zero-weights the pixels belonging to other objects. To do that it needs a segmentation stamp for each object, on the same grid as VIGNET and labelled with that object's NUMBER. Since #933 every tile, data or sim, goes through ShapePipe's own SExtractor run, so that run's segmentation map is where the stamps come from. UberSeg is the default from now on. That is a choice, not yet a validated result: the noisefill-vs-UberSeg comparison is #813, and blend_handling: noisefill stays available for it. On 25 real tiles UberSeg halved the blend flux excess at negligible cost.

What changes

Cutting the stamps. sextractor_runner cuts SEG_VIGNET out of SExtractor's segmentation check image, centred exactly the way SExtractor centres VIGNET (vignet_centre reproduces its rounding rule).
Relabelling. On data, match_catalogue replaces SExtractor's numbering with DR6 NUMBERs. The two numberings overlap almost entirely, so a partial relabel could leave a neighbour carrying the object's own number (second figure below). For that reason every label goes through one lookup: a matched detection takes its DR6 NUMBER, an unmatched one becomes −1 (still treated as a neighbour), and sky stays 0. The join refuses NUMBERs that are repeated or that don't fit exactly in int32.
Neighbours come from the segmentation map, under both blend handlings. With MASK_TYPE CORRECT, SExtractor also writes −1e30 into an object's VIGNET over sub-MINAREA sky blobs and over fragments CLEAN merged into the object itself, so noisefill was noise-filling the object's own light (15% of NEIGHBOUR_FLAG = 0 objects differed between noisefill and UberSeg on 25 tiles). A marker now counts as a neighbour only if it falls on another object's footprint in SEG_VIGNET. The remaining markers keep their weight and light, because the epoch stamps hold real data there. UberSeg ignores the markers, so its output is bit-identical (tested).
Workflow. tile_detect writes SEG_VIGNET (+3 GB) under both blend handlings. blend_handling: uberseg additionally sets ngmix's BLEND_HANDLING. The uberseg params pin is unchanged. The noisefill pin moves, because noisefill campaigns now need the seg stamps.
ngmix. Reads SEG_VIGNET from the tile catalogue. It includes ngmix: each chunk holds only its own rows' tile stamps #940, so each chunk only holds its own rows' stamps.

Why every label is relabelled. SExtractor label 23409 is dropped by the join, but 23409 is also the DR6 NUMBER of an unrelated object. A partial relabel (middle) gives that object a phantom footprint; the full relabel (right) marks it −1.

Verification

Unit tests cover relabelling a crowd where the numbers collide on purpose, the centring rule at exact halves, and the NUMBER guard's boundaries. Each injected bug fails at least one test: the wrong order, a partial relabel, np.rint centring, or no guard.
Real tile, DR6 202.301 (36,014 matched objects):
- 36,013 stamps carry their own NUMBER at the centre. The one exception's centre falls on sky.
- No stamp has the object's label outside its own footprint.
- SEG_VIGNET sits on VIGNET's pixel grid for every object where that grid can be measured.
- 70% of stamps contain a neighbour, and UberSeg zero-weights 27% of stamp pixels, which matches the earlier DR6-converter numbers.
- Peak memory is 2.9 GiB under uberseg vs 1.5 GiB under noisefill.
The full suite and tests/workflow pass.

Not covered here: validating UberSeg against noisefill in sims (#813).

Claude Opus 5.5 on behalf of Cail

🤖 Generated with Claude Code

https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

…defects Defects (nonzero flag, zero weight, invalid RMS) must be zero-weighted and noise-filled under every BLEND_HANDLING, with neighbour pixels left raw under uberseg. The masked-fraction cut must count that same raw set, and an epoch with a defect closer than EPOCH_CENTRAL_DEFECT_RADIUS to the stamp centre must be dropped, at a strict boundary and at the configured radius. The runner must pass both cut options on, the HSM centroid must ignore raw defect values, each epoch must keep its own OFFSET, and every tile must log its epoch-cut tally. The science test recovers the full 2x2 response for columns, 3-px bleeds, single pixels and edge bands at the veto radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs, and requires |c1|, |c2| < 5e-4 and |m11|, |m22| < 1%. Flagged pixels in the simulation hold a hot value, so an unfilled defect shows up. A column two pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

…ral ones Metacal shears the whole stamp without reading the weights, so a defect left raw under uberseg reached the fit. Every defect (nonzero flag, zero weight, invalid RMS) is now zero-weighted and noise-filled whatever BLEND_HANDLING is; uberseg only zeroes the weight of neighbour-side pixels and keeps their light. The option that meant "no neighbour treatment" is renamed from noisefill to none, and the retired name is rejected. The defect set is not symmetrized: at the veto radius the one-sided fill leaves |c| <= 3e-4, while a four-fold mask would quadruple m (-2.7% against -0.64% for a 3-px bleed at 10 px on a 0.5" galaxy) and, through an elliptical PSF, still leave c1 = 7.3e-4. An epoch is dropped when a defect lies closer than EPOCH_CENTRAL_DEFECT_RADIUS (default 10 px) to the stamp centre, the smallest radius at which columns, 3-px bleeds, single pixels and edge bands give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3" and 0.5" galaxies through a 0.7" PSF. The masked-fraction cut counts the same raw defect set; EPOCH_MASKED_FRACTION_CUT (default 1/3) makes it configurable. Each tile logs "epoch cuts: considered=N masked_fraction=A central_veto=B objects_emptied=C". The HSM centroid is measured on the filled image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

prepare_postage_stamps already holds the object's galaxy vignet dict, so the per-epoch OFFSET is read from it instead of a second sqlitedict lookup. The pixel scale sets only the centroid-prior width; the comments that also credited it with a noise window are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The interpolation fills only short defect runs (at most 3 px along a row or column, with clean pixels at both ends); anything wider, and runs reaching the stamp border, stays noise-filled. The interpolant must reproduce linear planes without reading defect values, commute with quarter turns of the stamp, and apply one operator to the science and metacal noise images. The weight is zero on the defects and on the quarter-turn orbit of the interpolated pixels, whose light stays. Each defect is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS for interpolated pixels, EPOCH_CENTRAL_DEFECT_RADIUS for noise-filled ones. The runner and Ngmix pass DEFECT_FILL to the epoch cuts and to metacal; noise stays the default. The science test recovers the full response matrix for interpolated columns, full and finite 3-px bleeds and single pixels at the interpolated radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs and on a 0.7" galaxy through a 0.9" PSF, and for noise-filled wide defects at the noise radius. A bleed three pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

… veto DEFECT_FILL = interpolate fills short defect runs (at most 3 px along a row or column, with clean pixels at both ends: columns, 3-px bleeds, single pixels) with a Clough-Tocher interpolant of the clean pixels within 4 px. The interpolant is averaged over the four quarter turns of the stamp, and each turn's triangulation serves the science image and the metacal noise image alike. Wider holes and runs reaching the stamp border keep the noise fill. noise stays the default and is unchanged. Interpolation restores the object's light, so the remaining bias is the hole in the likelihood: a one-sided zero-weight column 8 px from a 0.5" galaxy still gives c1 = -1.3e-3. The weight is therefore also zero on the quarter-turn orbit of the interpolated pixels (their light stays), which brings that column to c1 = +2e-6. Symmetrizing the fill instead would trade true light for interpolated light over four times the area. Each defect pixel is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS (default 7 px) for interpolated pixels and EPOCH_CENTRAL_DEFECT_RADIUS (10 px) for noise-filled ones. 7 px is the smallest radius at which interpolated columns, 3-px bleeds and single pixels give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3", 0.5" and 0.7" galaxies, round and elliptical PSFs; it does not scale with galaxy size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA

Brings the decision record to what this branch does: - central_defect_veto: committed option fixed_radii (was disabled); the 10 px EPOCH_CENTRAL_DEFECT_RADIUS is pinned by a Values entry on its module constant, now tagged. - epoch_masked_fraction_cut: the cut counts the defect set against EPOCH_MASKED_FRACTION_CUT (no longer [HARDCODED] on flags). - defect_fill: defects are noise-filled under every BLEND_HANDLING; the docstring [LINT] this branch fixes is dropped. - blend_handling: the no-treatment option is BLEND_HANDLING = none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings develop (#875's decision record) in through the updated base, and amends the record for this branch: - defect_fill: the interpolate option is implemented (DEFECT_FILL = interpolate); noise stays the committed fill. - central_defect_veto: EPOCH_INTERPOLATED_DEFECT_RADIUS = 7 joins the Values, on its now-tagged module constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conflicts: the ngmix runner imports both the epoch-cut defaults and write_empty_tile_output; Ngmix.process logs the epoch-cut tally and then calls log_run_health. Develop's run-health test stubs prepare_postage_stamps with the epoch-cut kwargs and tally and sets the epoch-cut attributes; the defect-fill process tests give their vignettes an OFFSET and their logger warning/error methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The neighbour option keeps develop's name, noisefill, in the code, the runner config key, tests, astra.yaml and the committed universe; the name none is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

SExtractor's -1e30 markers in the tile VIGNET form their own per-epoch mask (stamp.neighbours): a footprint 3 px from the centre, or covering 45% of the stamp, drops no epoch; noisefill zero-weights and noise-fills exactly the marked pixels, bit for bit as develop does; uberseg leaves them raw and weighted; a flagged column near the centre is still vetoed; and do_ngmix_metacal hands each epoch's mask to make_ngmix_observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sc

prepare_postage_stamps wrote the tile VIGNET's -1e30 markers (other detections' footprints) into each epoch's flag stamp as 2**10, so defect_mask counted them: they fed the masked-fraction cut and the central veto. Every epoch shares the tile VIGNET, so a neighbour within EPOCH_CENTRAL_DEFECT_RADIUS dropped every epoch of the object. The markers are now their own per-epoch mask, stamp.neighbours (MegaCam-flipped with the epoch), threaded through do_ngmix_metacal and make_ngmix_observation to prepare_ngmix_weights. The epoch cuts count genuine defects only. Under noisefill the marked pixels get weight 0 and the defects' noise realisation, the output noisefill gave when they arrived as flags; under uberseg they are ignored and the seg-based weight handles neighbours. Defects are treated identically under both. Docstrings, @sc contracts, runner comments and the astra.yaml decisions say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sc

SExtractor writes -1e30 into the tile VIGNET beyond the tile's edge as well as on other detections' footprints. Beyond the edge the epoch holds the object's own light, cut off, so treating those pixels as neighbours exempted edge objects from both epoch cuts and measured them with a noise-filled band near their centre. split_tile_markers classifies the markers geometrically on the flipped tile VIGNET: the union of stamp rows and columns that are entirely -1e30 (the off-image part of a rectangle clip) is off-tile, the rest is the neighbour mask. Off-tile pixels are flagged OFF_TILE_FLAG (2**10) in the epoch's flag stamp, so under both blend handlings they are zero-weighted, filled, and counted by the masked-fraction cut and the central veto. Docstrings, @sc contracts, runner comments and astra.yaml say so. Tests: an object 3 px from the tile edge loses every epoch; a band 12 px away passes the default cuts and is vetoed at radius 13; a corner's L-shaped off-tile region is flagged exactly and a border-touching neighbour stays a neighbour; off-tile pixels are filled under both blend handlings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Under DEFECT_FILL = interpolate and BLEND_HANDLING = noisefill, the Clough-Tocher support included marked neighbour pixels that noisefill replaces with noise, so a defect beside a bright neighbour was filled with light the image no longer contains. The support now excludes ~clean (the defects and the removed neighbour pixels); the NaN fallback to the noise fill covers degenerate supports. Under uberseg the neighbour light stays in the image and the support is unchanged. interpolate_defects names its second argument `excluded`. Test: a flagged pixel next to a 1e4 marked neighbour interpolates to the kept-pixel interpolant (1.4, against 3102 with the neighbour in the support) under noisefill, and from the raw neighbour light under uberseg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fill_defects used np.where, which promotes a float32 stamp to float64; develop's noisefill kept the galaxy stamp's dtype. The filled image is cast back to the input dtype, so noisefill stays bit-identical to develop on float32 stamps. The develop-equivalence test gains a float32 case and checks dtypes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…amp border Every entirely -1e30 stamp row or column counted as off-tile. In an edge stamp, a neighbour footprint that covers the on-image part of a row (a wide DR6 footprint running from the tile edge) completes that row, so the neighbour's pixels became OFF_TILE_FLAG defects: counted by the epoch cuts, interpolated under DEFECT_FILL = interpolate, filled under uberseg. Only the runs of marked rows and columns that start at a stamp border are off-tile now. Tests build tile VIGNETs with the catalogue-mode converter: on the real 202.301 DR6 patch every stamp splits exactly into off-image pixels and seg-map neighbour pixels, and the edge case above keeps its footprint in the neighbour mask through prepare_postage_stamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

… converter The neighbour-marking rationale and the converter's BIG comment describe the markers as ngmix neighbours under flag 2**10; ngmix splits them into off-tile defects and a neighbour mask. The masking text names the tile VIGNET, not SExtractor, as the markers' carrier, since catalogue mode paints them from the DR6 segmentation map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

A committed ini can then name a workflow knob with its default, and the workflow exports the variable only when the knob leaves its default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Under SEG_VIGNET = True the converter adds one int32 stamp per object of the relabelled segmentation map, cut with the window and centres VIGNET is cut with, so the two register pixel for pixel. The stamp cutter moves to sextractor_script.cut_stamps for the SExtractor path to share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Under SEG_VIGNET = True, sextractor_runner cuts the SEGMENTATION check image on each object's VIGNET grid and adds it to the sexcat. VIGNET is centred on the pixel nearest SExtractor's double barycentre; the float32 X_IMAGE/Y_IMAGE round to it except on exact half pixels, which are resolved by matching VIGNET against image minus BACKGROUND. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tile_cat takes the stamps from the catalogue it already reads, and BLEND_HANDLING = uberseg fails before the object loop when the column is missing. SEG_VIGNET_PATH, its separate catalogue and its checks go. The stamp columns stay views into the loaded table, so it is held once, and BLEND_HANDLING is environment-expanded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

blend_handling: uberseg (config.yaml, default noisefill) exports SP_SEG_VIGNET=True to tile_detect and SP_BLEND_HANDLING=uberseg to tile_ngmix, and adds the SEG_VIGNET column's memory to each ngmix chunk. The committed inis read them as ${SP_SEG_VIGNET:-False} and ${SP_BLEND_HANDLING:-noisefill}, so noisefill plans the campaign it planned without the knob and the params pin does not move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A noisefill campaign launched from a shell that exports SP_SEG_VIGNET or SP_BLEND_HANDLING (bare, or in the APPTAINERENV_/SINGULARITYENV_ forms that survive --cleanenv) would hand them to every job through the slurm executor's --export=ALL. The Snakefile now pops them at parse time, so only blend_env's exports under uberseg set them. Prologues and shells are unchanged; the DAG harness gains a launch_env to test it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Under SEG_VIGNET, sextractor_runner runs SExtractor with a copy of the param file that adds X_IMAGE_DBL and Y_IMAGE_DBL; add_seg_vignet centres each stamp at rint(DBL - 1), as SExtractor centres VIGNET, and drops the two columns, so the catalogue is noisefill's plus SEG_VIGNET. The half-pixel tie matching against image minus background goes, and with it the BACKGROUND check image dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

blend_handling defaults to uberseg in config.yaml and the Snakefile, so a campaign without the key writes SEG_VIGNET in both detection modes and runs ngmix with BLEND_HANDLING = uberseg. The committed inis keep the modules' own noise-fill defaults and the workflow exports the uberseg values, so blend_handling: noisefill plans exactly the previous noise-fill campaign: params_pin_noisefill.json is the previous pin, and a noise-fill run dir resumes with the key set explicitly. The default pin moves in tile_detect and tile_ngmix only. astra's committed blend_handling becomes uberseg, with noisefill the comparison arm. test_committed_blend_handling_fills_defects checks the committed blend_handling x defect_fill pair against prepare_ngmix_weights: the workflow default is the committed blend handling, and under it every defect is zero-weighted and filled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BbT81hykcvtuefzB3FhYvY

The sextractor, match_catalogue, tile.smk, workflow README and params pin conflicts are develop's squashed #933 against the same content here, so they keep this branch's side: add_seg_vignet before match_catalogue, the full-table relabel, the NUMBER guard and SExtractor's VIGNET centring. ngmix: #927 replaces the PIXEL_SCALE option with each object's own pixel scale from its epochs' Jacobians; develop's side is taken, and the masked-pixels tests stop passing a pixel scale to Ngmix. #927's centroid prior test sets the defect-fill attributes and epoch_cuts that process reads on this branch. The regenerated params pins are unchanged; the noisefill pin is develop's params_pin.json byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JQtu8oaZQEAqzbxZbPpWs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Follow #933: the tile VIGNET's -1e30 markers come from SExtractor alone; the catalogue converter and its neighbour-marking decision are gone, and the off-tile split tests cut SExtractor-style stamps from a segmentation map directly. Follow #927: no PIXEL_SCALE option; the centroid prior uses each object's own pixel scale. Tests that build Ngmix or drive process() carry the epoch-cut settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JkM3cmXPuuSrNAf7vJde9e

…l DR6 blend, and a defect-type gallery Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…for every defect type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…lasses it covers prepare_ngmix_weights sorts each stamp pixel into clean, light-replaced (noise-filled defects, noisefill neighbours) and light-kept-at-zero-weight (interpolated defects, the uberseg neighbour side). One step zeroes the weight on the quarter-turn copies, about the stamp centre, of the light-kept holes that SYMMETRIZE_WEIGHTS names: - interpolated (default): the interpolated defects, as before; - interpolated_and_neighbours: also the uberseg neighbour side (#815); - none. Light-replaced pixels are never symmetrized. Under noisefill + noise every value is a no-op, so the defaults are unchanged. uberseg_weight becomes uberseg_mask, returning the neighbour-side pixels, so the weight map is built once from the pixel classes. Measured with defect_response (6 seeds, 0.5" galaxy, 0.7" PSF, neighbour at 10 px): a lightless neighbour footprint gives c1 = -7.8e-3 under uberseg and 2e-6 symmetrized, but with neighbour light (0.1x flux) c1 goes from +1.1e-3 to +1.2e-2, because the one-sided hole partly offsets the neighbour light in the target's cell. Symmetrizing also keeps 117 of 1580 weighted pixels (97% of S/N) at 10 px and 25 (71%) at 7 px. Hence the neighbour side stays opt-in. tests/science/test_weight_symmetrization.py guards both results; defect_response gains a neighbour and the fit S/N. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…fill's interpolated set Short defect runs are always interpolated and the rest noise-filled; DEFECT_FILL, SYMMETRIZE_WEIGHTS and the three epoch-cut keys are gone, and the radii and the masked fraction are module constants. central_defect_vetoes takes the interpolated set prepare_ngmix_weights uses, removed noisefill neighbours included, so a run that ends on a neighbour is vetoed at the noise-fill radius it gets. A non-finite interpolant raises instead of falling back to noise. Off-tile pixels get zero weight instead of a flag sentinel. Only interpolated holes are weight-symmetrized; the #815 neighbour-side option is dropped. The defect science tests merge into tests/science/test_defect_recovery.py, and the module tests lose their option plumbing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

… the set interpolate_defects fits one Clough-Tocher interpolant instead of averaging four quarter-turned ones. For the defects the veto keeps the average moves c by at most 5e-5 and m by at most 4e-4 (test_defect_recovery cases), and the quarter-turn weight copies already make the likelihood symmetric. interpolated_defects(defect, neighbour, blend_handling) is the one rule for which defect pixels are interpolated; prepare_postage_stamps (the central veto) and prepare_ngmix_weights (the fill) both call it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

pixel_classes compares noisefill and uberseg; defect_gallery shows raw, image and weight under the default. Both take the interpolated set from interpolated_defects, and violet marks only pixels zeroed by the quarter-turn symmetrization. dr6_epochs gives off-tile pixels zero weight and stores the off_tile mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…ct type, noise fill as counterfactual Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

EPOCH_MASKED_FRACTION_CUT = 0.10 (was 1/3). The central veto is the bias control; far-field defects bias nothing measured up to 58% of the stamp, so the fraction cut is set for DES Y3/Y6 comparability and as a margin against real-data effects the sims do not model. On tile 202.301 it drops 5.08% of epochs (3.38% at 1/3) and empties 0.55% of objects (0.35%); the extra drops are all CCD-edge or off-tile bands 10-20 px from the object. astra.yaml: ten_percent is the default, one_third excluded with the reason; universes/committed.yaml follows. Tests: the defect-set cut test uses 9.8%/11.8% epochs; corner and edge-neighbour classification tests use geometries under 10%; the off-tile central-veto test lifts the fraction cut to isolate the veto; the far-field recovery cases no longer assert against the cut and are documented as the evidence it is not a bias control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…cut) into integration/smk-g13 Brings #922 from 0226edf to b0ce5af: one-orientation defect interpolant and interpolated_defects() as the one rule for the veto and the fill (6c75fd4), EPOCH_MASKED_FRACTION_CUT = 0.10 with astra's ten_percent default (3bd49d3), and validation figures. One conflict, test_a_neighbour_completing_rows_beside_the_tile_edge_ stays_a_neighbour: #922's geometry (under 10%) with #925's cut_stamps-based _marked_stamps helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz

…ough a 0.9" PSF A noise-filled 4-column cluster (8% of the stamp, kept by the 10% cut) on that galaxy gives m11 = -6.6% at 10 px and first passes at 13 px. The science test checks the cluster and the edge band there; the bias-vs-distance script gains the cluster. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

…s frame One function, defect_weighting_masks, decides how interpolated defects enter the image and the weight map: fourfold_zero (the PR's design, still the default), des_y6 (pizza-cutter: mask OR its k=1 quarter turn, interpolated at full weight), hole and full. The option reaches ngmix from the run config: `defect_weighting` -> SP_DEFECT_WEIGHTING (tile_ngmix) -> DEFECT_WEIGHTING in config_tile_Ng_template.ini, validated at parse time against the module's DEFECT_WEIGHTINGS; tile_vignets carries it too so a change reruns the whole shape group. Params pin updated (tile_ngmix only): a campaign boundary. Under every option the metacal noise image is now interpolated in the frame ngmix's fixnoise shears it in (rot90 k=1, back with k=3), by the science image's operator. Interpolated in the detector frame, its interpolated pixels sat a quarter turn from the science image's and left an S/N-dependent additive bias (c1 = 2.7e-3 for a 3-px bleed at 7 px on the 0.7"/0.9" galaxy at S/N 30). A test pins ngmix's k=1/k=3 behaviourally. astra: weight_symmetrization becomes shape_measurement.defect_weighting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

Over the epochs handed to the fit, each object now carries n_epoch_interp (epochs with an interpolated defect pixel), min_dist_interp and min_dist_noisefill (smallest distance in pixels from the stamp centre to an interpolated / noise-filled defect pixel; -1 when there is none). They read only masks, so distance-split null tests are catalogue queries. make_cat writes them as NGMIX_N_EPOCH_INTERP, NGMIX_MIN_DIST_INTERP and NGMIX_MIN_DIST_NOISEFILL (never-fit rows: 0, -1, -1), and both final_cat.param files list them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

params_pin.json regenerated with --update-params-pin on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

The defect_fill rationale in astra.yaml now says what a defect is, why its image value matters under metacal, what the fill does, and why. The ngmix template and config.yaml name the default defect weighting, fourfold_zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

tests/workflow/README.md and test_dag.py keep both sides (develop's test_tile_store_is_unique_per_campaign and LOCAL_TAG/run-dir-hash probes, this branch's blend-handling and noisefill rows); both params pins regenerated with --update-params-pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

#922 was rebuilt after this branch took it, so the two carry different histories of the defect fill. Resolved to #922's current design (interpolation is the only fill, DEFECT_WEIGHTING, 10% masked-fraction cut, 13 px noise-fill veto) with this branch's SEG_VIGNET UberSeg on top: - ngmix.py: no seg_cat_path; Raises names blend_handling/defect_weighting. - committed.yaml: defect_fill interpolate, blend_handling uberseg, defect_weighting fourfold_zero, epoch_masked_fraction_cut ten_percent. - tile.smk: tile_ngmix exports SP_DEFECT_WEIGHTING and the blend variables. - astra.yaml: #922's defect_fill rationale; this branch's uberseg-default blend_handling text. - tests: drop the DEFECT_FILL tests #922 removed; the committed-universe test checks the fill is the noisefill one under the committed blend handling; _marked_stamps keeps cut_stamps with #922's 20-px-edge geometry; test_sextractor_seg_vignet uses uberseg_mask; test_dag normalizes the run-dir hash in the node-local store name (develop's LOCAL_TAG), which differs between the campaigns these tests compare. - params pins regenerated with --update-params-pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

Tile_cat takes no seg_cat_path: it reads row_min/row_max and holds both stamp columns of the tile catalogue, VIGNET and SEG_VIGNET, for the chunk's rows as ChunkStamps. test_tile_cat_reads_seg_vignet compares row by row; test_tile_cat_holds_the_stamp_columns_once gives way to test_ngmix_tile_cat_chunk.py, which now writes SEG_VIGNET into the tile catalogue (its separate-seg-catalogue alignment test goes with the separate catalogue). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

…aches it A copy wedged in a corner of the clean region (an edge band beside the stamp edge) lies outside the support's hull, so the fill came back NaN and the finiteness guard dropped every epoch of the object. Such copies now keep their light, like copies on already-replaced pixels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An exception while building one epoch's ngmix observation (for example the non-finite interpolant guard in prepare_ngmix_weights) used to reach the per-object except, losing every epoch of the object. do_ngmix_metacal now catches it per epoch, drops that epoch and fits the object on the rest; the existing epoch logic (N_EPOCH, masked-fraction cut) decides the rest. An object whose every epoch fails still raises. The drop is kept: Postage_stamp.epoch_failures holds (epoch name, exception), process() logs each at warning level with tile, object and epoch, adds them to the "epoch cuts" tally as failed=, and writes the count as a new per-object column n_epoch_failed -> NGMIX_N_EPOCH_FAILED (final_cat.param for cfis and cfis_image_sims; never-fit rows 0). astra.yaml epoch_masked_fraction_cut records the rule; contract failed-epoch-dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

…seg-vignet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

With MASK_TYPE CORRECT, SExtractor writes -1e30 into an object's tile VIGNET over every above-threshold pixel outside its pre-CLEAN footprint: other detections, but also sub-MINAREA sky blobs and fragments CLEAN merged into the object itself. split_tile_markers took every interior marker as a neighbour, so noisefill noise-filled the object's own light (15% of NEIGHBOUR_FLAG=0 objects differed noisefill vs uberseg on 25 tiles). split_tile_markers(tile_vign, shape, seg, object_number) now keeps as neighbours only markers on a seg label other than 0 and the object's own (the join's -1, a detection that left the catalogue, counts). Off-tile detection is unchanged. The other interior markers become nothing: -1e30 is a flag in the tile VIGNET only, and the epoch stamps hold real light there (the object's own or sky), so they keep weight and image and are neither interpolated nor noise-filled. The narrower neighbour mask feeds the noisefill fill and interpolated_defects consistently; uberseg reads no marker, so its output is bit-identical (tested). Without a seg stamp every interior marker stays a neighbour, and ngmix warns once. tile_detect therefore writes SEG_VIGNET under both blend handlings (SP_SEG_VIGNET exported always; DETECT_SEG_MEM_MB / NGMIX_SEG_MEM_MB unconditional). blend_handling now only moves tile_ngmix's prologue: the uberseg params pin is unchanged and the noisefill pin moves (campaign boundary for noisefill only). astra.yaml blend_handling, defect_fill and pixel-mask text updated; contract neighbours-are-other-footprints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

…node-local # Conflicts: # workflow/rules/tile.smk

cailmdaley and others added 29 commits September 26, 2026 07:07

tests: name the none-mode uberseg test for none, not noisefill

9f0ea44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ngmix: restore the BLEND_HANDLING name noisefill

b9d92a9

The neighbour option keeps develop's name, noisefill, in the code, the runner config key, tests, astra.yaml and the committed universe; the name none is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/develop' into rebuild/masked-pixels

dcec8c4

Expand ${VAR:-default} in config values

d35af2e

A committed ini can then name a workflow knob with its default, and the workflow exports the variable only when the knob leaves its default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

make_cat: leave SEG_VIGNET out of final_cat, with VIGNET

0eb2cfb

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

astra.yaml: UberSeg reads the tile catalogue's SEG_VIGNET

f4e9515

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Point the stale SEG_VIGNET_PATH message and astra note at SEG_VIGNET

60ae234

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cailmdaley force-pushed the feat/uberseg-seg-vignet branch from 7af2d80 to c704e63 Compare September 30, 2026 01:10

cailmdaley added a commit to cailmdaley/shapepipe that referenced this pull request Oct 2, 2026

PR CosmoStat#925 figures

fbdf617

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cailmdaley and others added 11 commits October 2, 2026 17:26

Masked-pixel figures for #922: pixel classes on a synthetic and a rea…

74f5be3

…l DR6 blend, and a defect-type gallery Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

Defect gallery from real DR6 epochs; dr6_epochs.py scans an exposure …

b482d75

…for every defect type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

CI: run tests/science once, in its own step

0226edf

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

Bias-vs-distance figure for one defect: production treatment per defe…

b82c77e

…ct type, noise fill as counterfactual Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

bias_vs_distance: legend names the masked-fraction cut from its constant

b0ce5af

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

cailmdaley and others added 8 commits October 3, 2026 02:42

Merge origin/develop into feat/masked-pixels

93c5472

params_pin.json regenerated with --update-params-pin on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

cailmdaley requested a review from martinkilbinger October 5, 2026 13:28

cailmdaley and others added 2 commits October 5, 2026 15:34

Merge feat/masked-pixels (des_y6 copy fix) into feat/uberseg-seg-vignet

7c0b563

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cailmdaley mentioned this pull request Oct 5, 2026

tile_detect: build the catalogue in memory, run SExtractor on node-local disk #947

Open

cailmdaley and others added 3 commits October 5, 2026 15:59

Merge feat/masked-pixels (#922, failed-epoch drop) into feat/uberseg-…

f344fa6

…seg-vignet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yfkBvkzPv33iJxXZ6jAxd

cailmdaley added a commit that referenced this pull request Oct 5, 2026

Merge feat/uberseg-seg-vignet (#925) @ 6b9fa35 into perf/tile-detect-…

18cd66a

…node-local # Conflicts: # workflow/rules/tile.smk

Navigation Menu

UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run) - #925

UberSeg end to end on #933's tile detection (SEG_VIGNET from the tile SExtractor run)#925
cailmdaley wants to merge 77 commits into
developfrom
feat/uberseg-seg-vignet

cailmdaley commented Sep 30, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cailmdaley commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and how the work is organised

What changes

Verification

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cailmdaley commented Sep 30, 2026 •

edited

Loading