ngmix masked pixels: neighbours and defects as separate questions

[cailmdaley]

Every stamp we fit with ngmix has pixels we can't use as they are: light from a neighbouring object, or a defect (bad column, cosmic ray, bleed trail, chip or tile edge). Metacal shears the image in Fourier space, where a weight map can't enter, so whatever we put in those pixels is sheared and spread into the fit. This PR fixes one treatment for defects, keeps a single setting for neighbours, and adds the cuts that drop an epoch when a defect sits too close to the object.

What ngmix sees

One epoch run through prepare_ngmix_weights. Top: the image ngmix fits. Bottom: its weights (dark = zero; violet = zero weight with the light kept). (a–c) A synthetic blend under each BLEND_HANDLING. (d–e) A real DR6 blend with a real bad column: tile 202.301, object 4783, exposure 2086786p, CCD 24.

Pixel	BLEND_HANDLING = noisefill (default)	uberseg
Neighbour footprint	zero weight, filled with noise	pixels nearer a neighbour than the target get zero weight; their light stays
Defect: zero exposure weight, nonzero flag, bad background RMS, or beyond the tile edge	zero weight; narrow runs interpolated, the rest noise-filled	same

Neighbours. SExtractor marks other objects' footprints with −1e30 in each tile stamp, and we apply that one mask to every epoch. noisefill replaces those pixels with noise. uberseg (as in DES) gives each pixel to the nearest segmentation footprint and keeps the neighbour's light, which is real sky that metacal shears along with the target. Which default to use is decided by the blended-simulation comparison in #813.

Off-tile pixels are also −1e30 in the tile stamp, but the exposure there holds the object's own light, cut off by the tile edge. So they are defects. We find them as the unbroken runs of fully −1e30 rows and columns that start at the stamp border. This matches the true off-tile region for 210 of 212 edge objects on a simulated tile (to ±1 px), and for 32 of 32 on DR6.

Defects

Real DR6 defects (tile 202.301, exposure 2086786p) under the default. Green outline: interpolated. Orange: noise fallback. Blue: neighbour footprint.

A defect run at most 3 px across, with clean pixels on both sides, is filled by Clough–Tocher interpolation from the surrounding pixels. Wider holes, edge bands, and runs that end on a noise-filled neighbour get noise at the background RMS instead.

Interpolated pixels keep zero weight. A one-sided zero-weight hole pulls the model fit (c₁ = −1.3e-3 for a column 8 px from a 0.5″ galaxy). So we also zero the weight of each interpolated pixel's three quarter-turn copies, keeping their light. The holes then cancel (c₁ = 2e-6). These copies are the violet lines.

Interpolation is the only fill because noise fill near the object is biased in a size-dependent way: a 0.7″ galaxy through a 0.9″ PSF needs a 13–14 px veto under noise fill, but only 7 px with interpolation. The interpolated fill is unbiased, but it is about 1.6× noisier than the background across a 3 px gap; that is covered by the bias scan below.

Symmetrising neighbour holes (#815) was tried and rejected. With no neighbour light it cancels the pull (c₁ from −7.8e-3 to 2e-6). With real neighbour light, though, the one-sided hole was partly offsetting that light's pull. Symmetrising removes the offset and makes c₁ worse: from 1.1e-3 to 1.2e-2 for a neighbour at 0.1× the target's flux, 10 px away. It also leaves only 25 weighted pixels for a neighbour at 7 px. It is recorded as excluded in astra.yaml.

Which epochs we keep

An epoch is dropped when:

• a noise-filled defect pixel lies closer than 13 px to the object centre, or an interpolated one closer than 7 px;
• or more than 10% of the stamp is defect.
• or the epoch fails to prepare (e.g. the interpolant can't reach a defect). It is logged, counted in NGMIX_N_EPOCH_FAILED, and the object is fitted on its remaining epochs.

Neighbour pixels never count towards these cuts. The neighbour mask is the same in every epoch, so counting it would remove whole blended objects (31% of a simulated tile).

Each radius is the smallest at which every tested defect keeps |m| < 1% and |c| < 5e-4. The defects are columns, 3 px bleeds, single pixels, edge bands and a 4-column cluster too wide to interpolate. The cases cover 0.3″, 0.5″ and 0.7″ galaxies, and round and elliptical PSFs. The 0.7″ galaxy through a 0.9″ PSF sets both radii. Interpolated 3 px bleeds there need 7 px (c = 6.9e-4 at 6 px, 2.5e-4 at 7 px). Noise-filled defects need 13 px: a 4-column cluster gives m₁ = −6.6% at 10 px and −0.67% at 13 px. CFHT observes at a fixed orientation, so detector defects line up the same way in every epoch, and their biases add rather than average away.

Worst-axis |m| and |c| from a single defect at distance d (tests/helpers/defect_response, 6 seeds). Solid: the production treatment. Dashed: noise fill, shown as the counterfactual for narrow defects. Hollow: epochs the 10% cut drops anyway. Dotted: the 7 px and 13 px veto radii.

Why the masked fraction is 10%, as in DES. The cut matters for bias only through edge bands, which reach close to the object while covering a large part of the stamp: at 10% every edge band closer than about 20 px is dropped. Far-field masks are otherwise harmless (|m| ≤ 0.05% up to 58% of the stamp). On real tile 202.301 the cut is cheap. Compared with 1/3, for interior galaxy-like objects, epochs dropped go from 3.4% to 5.1% and objects left with no epoch from 0.35% to 0.55%. What 10% adds is CCD-edge and off-tile bands. Column clusters sit at 2–8% of a stamp, and only 4 epochs cross 10%.

Technical notes

• Settings. BLEND_HANDLING (noisefill | uberseg). The veto radii (13 px, 7 px) and the masked fraction (10%) are constants in ngmix.py, and each is recorded with its excluded alternatives in astra.yaml.

• What changes for existing runs.

  • Narrow defects are now interpolated rather than noise-filled.
  • The masked-fraction cut used to count every −1e30 marker. It now counts flagged, zero-weight and bad-RMS pixels, and ignores neighbours.
  • The central veto is new.
  • The masked-fraction cut is 10% rather than 1/3.
  • Under uberseg, defects used to keep their raw values; they are now filled.

  Both arms of A/B validation of off-by-default shape-measurement options #813 must run on this code.

• Which pixels are interpolated is decided once, by interpolated_defects, and shared by the veto and the fill. A non-finite interpolant raises an error and the object is lost. This never happened in a fuzz test of 4,000 random stamps, or in 104,069 real epochs on tile 202.301 (294,675 interpolated pixels).

• Object loss (measured with the earlier 1/3 cut and 10 px veto). On a simulated tile of 21,851 objects, the cuts lose 89 (0.41%). 88 of those lie outside the tile's deduplication box and inside a neighbouring tile's, so the merged catalogue drops them from this tile anyway. The 89th is a neighbour footprint that fills whole rows out to the stamp border. Its pixels are read as off-tile, which is the known approximation.

• Saturation. The survey flag map marks only the first pixel of each saturated run, so most saturated pixels reach ngmix unflagged. This is separate from this PR; see Saturated pixels reach ngmix unflagged: survey flag bit 8 marks only the first pixel of each saturated run #943.

• CI. tests/science now runs once, in its own step.

• Not included. Multi-object fitting (MOF), which remains a later option.

• Evidence.

  • Figure scripts: scripts/validation/masked_pixels/.
  • Bias tests: tests/science/test_defect_recovery.py.
  • Review measurements, on candide: /n17data/cdaley/scratch/922_review/ and /n17data/cdaley/scratch/review922/.

Supersedes #915 and #916. Closes #815. Part of #602.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4HLdzDyc3CiPLrgJ48GVo

[cailmdaley]

Defect: a neighbour that covers a full border row or column of the stamp is read as off-tile.

split_tile_markers (ngmix.py:1754) treats any unbroken run of all −1e30 border rows or columns as off-tile. A large neighbour can cover a full border row of a stamp that sits entirely on the tile. Those pixels then get noise-filled, or count against the masked fraction, when they should get UberSeg weighting.

Measured on tile 202.301, with our own SExtractor run: 45 of 35,621 fully on-tile objects (0.13%) are affected. The median is 301 misread pixels per stamp, and a third of the affected objects have at least 20% of their stamp misread. The affected objects are all in blends, so the selection is small but blend-correlated.

Fix: decide off-tile from the stamp's geometry (tile dimensions plus the vignet origin), or under UberSeg from SEG_VIGNET, instead of from −1e30 alone. This doesn't block the current campaign.

Claude Opus 5.5 on behalf of Cail, from the nibi Surveyor's check