Merge develop into the tomography branch - #394
Draft
cailmdaley wants to merge 73 commits into
Draft
cailmdaley wants to merge 73 commits into
cailmdaley wants to merge 73 commits into
Conversation
* chore: add Renovate config (ported from shapepipe) * chore: gate python bumps behind dashboard approval
The metacal estimator and calibrate_comprehensive_cat.py carried a workaround that overwrote the metacal no-shear reconvolution-PSF size with the 1P value — a real fix for an older ShapePipe catalogue whose no-shear PSF was wrong, now a no-op under the current stack: ngmix builds one magnitude-dilated reconv PSF and reuses it across all metacal types, so NOSHEAR and 1P are bit-identical per object. - calibration.py: drop the overwrite; the no-shear branch simply uses the no-shear reconv-PSF column it already reads. - calibrate_comprehensive_cat.py: drop the override and the now-redundant NGMIX_T_PSF_RECONV_NOSHEAR_orig column. - test_calibration.py: update the estimator-stdout note (no hack line now). No output changes under the current stack. test_calibration.py: 8 passed. Claude-Session: https://claude.ai/code/session_01Xthu9uZC17TsaeXh899fhc Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The provisional SHA pin predated the fork's merged protocol+packaging PRs (and its commit is now orphaned). The library is changing quickly, so pyproject tracks the fork's main and uv.lock carries the exact commit; updates land via uv lock --upgrade-package smokescreen. The DRAW_SCHEME custody assertion fails closed if an install's draw semantics ever drift from the committed blind. Claude-Session: https://claude.ai/code/session_01G9MahwJEQ1t9EuvUXijmy3 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
check.txt/format.txt/report.md are written by lint.yml during each run; the autofix step's 'git add -A' kept committing them back. Gitignore them (plus the residual-pass variants) and drop the tracked copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012b34pAS3bXRxN5hVdyq5Sw
* additive bias calculation for paper updated to v1.4.6.3 * added config * added fill_photoz script * fill photoz bands fixes * adding mag errors to fill_photoz * added Z_ML to fill_photoz * added more flags to fill_photoz * added 0p7 and 1p0 aperture magnitudes to PhotoPipe + SP output * Fixed fill photoz script with new MP_NAME type * ruff autofix (format + safe lint fixes) Pushed by the lint gate. * removed leftover git marker * fill_photoz_bands: spot-check FITS/HDF5 row order before filling a tile The fill pairs FITS row k with the k-th HDF5 row of the tile (sorted-index order) and only ever verified the row *count*, so a PhotoPipe tile ordered differently from the comprehensive catalogue would be filled with silently mismatched photo-z. After the size check, compare RA/Dec for a small sample of rows -- up to five at each end plus evenly spaced interior rows, --n_check_rows (default 10), --check_tol_arcsec (default 0.5). Only the sampled HDF5 rows are read (one fancy-index into the already-sorted index array), so the cost is a handful of point reads per tile rather than a full per-row match. A failing tile is warned about, counted as a row-order mismatch in the end-of-run summary, counted towards the consecutive-failure abort, and skipped without being added to done_tiles -- exactly as a size mismatch is, so a resume retries it. --n_check_rows 0 disables the check. HDF5 columns are RA/Dec (cat_config.yaml ra_col/dec_col for SP_v1.4.x). The FITS names are resolved at runtime from a candidate list; ALPHA_J2000 / DELTA_J2000 is first, verified against a real DR6 tile (/n17data/UNIONS/WL/photometry/UNIONS_DR6/UNIONS.001.227_SP_ugriz_photoz_ext.cat). If no candidate pair matches, the check disables itself with a warning rather than skipping tiles. Test: synthetic 3-tile HDF5 + FITS pair, tiles interleaved so the non-contiguous write path runs too; the two aligned tiles fill, the tile with reversed FITS rows is skipped and left empty, and --n_check_rows 0 reproduces the old unchecked behaviour. * fill_photoz_bands: row-order check ignores invalid positions, fails if none A sampled row with a non-finite or sentinel (|Dec| > 90) coordinate on either side carries no information about the FITS/HDF5 pairing. Before, such a row made `sep <= tol` False (a false row-order failure), and an all-NaN sample raised a RuntimeWarning from np.nanmax. Now those rows are excluded from the comparison, n_checked reports only the rows actually compared, and a tile whose sample has no comparable row is treated as unverifiable and skipped (distinct warning), so it is retried on resume rather than written blind. Adds a direct unit test covering partial/all-invalid samples, a reversed remainder, and a single-row tile (one-element h5py fancy index). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: martinkilbinger <martinkilbinger@cea.fr> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Cail Daley <cail.daley@cea.fr> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…vivor, guard against recurrence (#320) * cat_config: drop duplicate SP_v1.4.6 / SP_v1.3.6 blocks, repair the survivor `cosmo_val/cat_config.yaml` carried two top-level `SP_v1.4.6:` keys and two `SP_v1.3.6:` keys. PyYAML keeps the last, so the second block of each pair was what every consumer saw — and both were stale. They arrived in merge c22f075, which resolved a conflict by keeping both sides (and renamed the dead `SSP_v1.4.6_msel` key to `SP_v1.4.6` in the process). Delete the shadowing blocks and repair the surviving `SP_v1.4.6` to the post-aa774b65d convention: * `cov_th` stays at A = 2894.03 deg^2 / n_e = 5.0935 (the nside-4096 footprint mask every other live entry uses), not the 2405 / 6.128 pair the shadowing block held. * `shear.redshift_distr` -> `shear.redshift_path`: the code only ever reads `redshift_path`, so the v1.4.6 n(z) was under a key nothing reads while the winning block pointed at the v1.0-era `dndz_SP_A.txt`. * `shear.mask` (read nowhere) -> entry-level `mask:` pointing at `mask_map_footprint_nside_4096.fits`, matching SP_v1.4.5 / SP_v1.4.6.3 and the mask `A` was measured from. * `shear.R: 1.0` — the delivered e1/e2 columns already have the response applied; the shadowing block's `R: 0.92` double-counted it and inflated xi_pm by 1/R^2 = 18% against a covariance that never sees R. `SP_v1.3.6`'s surviving block already carries the nside-4096 footprint mask; its cov_th still holds the old 2405 / 6.128 pair and needs regenerating separately. * tests: reject duplicate keys in cat_config.yaml `yaml.safe_load` silently keeps the last of a repeated key, so a merge that resolves a conflict by keeping both sides leaves the shadowed block invisible to every consumer *and* to every test. That is how two `SP_v1.4.6` and two `SP_v1.3.6` entries survived on develop. Add a `yaml.SafeLoader` subclass whose `construct_mapping` raises on a repeated key, and assert `cosmo_val/cat_config.yaml` parses clean with it. Fails on the pre-fix config with `duplicate key 'SP_v1.3.6' at line 522 (first seen at line 148)`. * cat_config: recompute SP_v1.3.6 cov_th (A, n_e, sigma_e) for the nside-4096 footprint aa774b6 repointed SP_v1.3.6's mask to the 2894 deg² footprint but left its cov_th holding v1.4.6's old numbers. Recomputed from v1.3.6's own catalogue with sp_validation.survey; the same code path reproduces SP_v1.4.6.3's committed A / n_e / sigma_e bit-for-bit. n_psf left as it was.
* Fixes from the snakemake inference run report (#300) Rules: drop the /automnt prefix from the hardcoded paths in xi_highres (twopoint.smk) and covariance_glass_mock (covariance.smk). /automnt/nXXdataN does not exist on the node that owns that disk, so a job landing there fails immediately, before any log is written. Every canonical path in common.py already uses the plain /nXXdataN form. Docs: workflow/README.md gains a note on the /automnt trap and one on host ~/.local shadowing the container's pinned Snakemake; cosmo_inference/README.md recommends CosmoSIS --mpi over the fragile upstream --smp process pool (cosmosis#170) and cosmosis >= 3.16.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * workflow: profile-driven apptainer containerization (set A) candide profile now owns the container: software-deployment-method: apptainer + apptainer-args carries the bind mounts (matching the app bash-function binds in the top-level UNIONS CLAUDE.md), replacing the old rationale for leaving containerization to each rule. Rewrites the profile's doc comment to describe the new model and its two documented exceptions (xi_highres MPI, covariance_cosmocov host toolchain). Adds a container_smoke rule (workflow/rules/container_smoke.smk + scripts/container_smoke.py) as a cheap end-to-end check of the profile-driven container path (editable sp_validation import, numpy + OMP_NUM_THREADS, git provenance) via `script:`, wired unconditionally into workflow/Snakefile. Reconciles image_sims/Snakefile's container: None comment: it now documents that the two-image (SIF/SIF_PIPELINE) chain is a per-rule container: choice in image_sims.smk, still wrapped by the profile's apptainer deployment -- not a rule-owned apptainer exec call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * workflow: profile-driven apptainer containerization (set B: image_sims) Strip every rule's explicit apptainer exec wrapper (_EXEC_PREFIX/EXEC/ EXEC_PIPELINE) from image_sims.smk. Each compute rule now carries a plain per-rule container: SIF / container: SIF_PIPELINE directive; Snakemake wraps the shell: command via the profile's software-deployment-method: apptainer + apptainer-args (set A). PYTHONPATH/PSF_DICT/OMP_NUM_THREADS injection and the SLURM_* env strip move from apptainer --env/-u flags to plain shell VAR=value / env -u syntax at the front of each shell: string (_ENV_PREFIX) -- identical effect, no apptainer-specific mechanism, works the same whether or not the command is container-wrapped. im_mbias split into im_mbias_config (run:, host-side git/provenance introspection + yaml write -- Snakemake never containerizes run: regardless of container:, so this must stay a driver-side step) and im_mbias (shell:, container: SIF, runs the actual m-bias compute). This was the one rule whose apptainer call lived inside a run: block's trailing shell() -- splitting it out is what makes container: apply to it at all. binds: dropped from image_sims config/schema -- it's now the profile's apptainer-args (one bind list for the whole workflow), not a per-run-config value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * workflow: profile-driven apptainer containerization (set C: remaining rules + docs) Completes the pivot for the rules outside image_sims: xi_highres and covariance_cosmocov keep their container: None + inline apptainer exec / host-toolchain call (multi-node MPI and a host-compiled binary respectively, both genuinely incompatible with Snakemake's own container wrapping), now documented in their rule docstrings as deliberate exceptions rather than leftovers. No other rule outside image_sims called apptainer directly. While touching these files, retired the stale /pure_eb/ absolute paths left from the old repo layout: added workflow.common.WORKFLOW_SCRIPTS (Path(__file__)-based, correct under both standalone and module-composed runs) for the handful of shell: rules that call a workflow script directly, and reused the existing COSMO_INFERENCE constant elsewhere. Removed the run_cosmo_val rule in twopoint.smk, dead since cosmo_val.smk decomposed it into per-diagnostic rules (its own docstring says so) and still pointing at a stale path plus a nonsensical host .local PYTHONPATH injection. Flagged (not fixed) covariance_process: it calls cosmo_inference/scripts/ cosmocov_process.py, deleted in the #236 cleanup and never restored, so the rule fails on the default covariance target -- pre-existing, unrelated to this pivot. Rewrote workflow/README.md and cosmo_inference/README.md to the new model: snakemake is a thin host-side tool pinned via `uv tool install snakemake snakemake-executor-plugin-slurm`, run directly on the host, never from inside an apptainer shell; the candide profile's software-deployment-method puts each job in the container instead. Added a short pointer from the top-level README's dev-shell instructions to workflow/README.md so the two don't get conflated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * workflow: fix script: directive under the profile-driven container container_smoke failed for real (SLURM jobs 839818/839819) with ModuleNotFoundError: snakemake.iocontainers -- the container image had its own snakemake==9.16.3 pip-installed directly (leftover from the old apptainer-shell-then-snakemake-inside pattern this pivot retires), shadowing the host-mounted 9.23.1 orchestrator that script: bind-mounts in and sys.path.extends (appended, not prepended). Removed it and its snakemake-executor-plugin-slurm/-slurm-jobstep/-interface-* family from the image (verified Required-by: none outside the family itself). Separately, apptainer-args never actually isolated host tooling: the image's own /.singularity.d/env/50-bashrc.sh unconditionally sourced the host ~/.bashrc for every apptainer action, not just an interactive `apptainer shell` -- so a host dotfile (asdf init) ran on every exec too, pushing host PATH entries (~/.local/bin) ahead of the image's own /usr/local/bin. A bare `python` in any shell:/script: rule was silently running the host's interpreter, invisibly, surviving --cleanenv. Gated the bashrc sourcing on APPTAINER_COMMAND=shell (set by apptainer itself before these scripts run). Fixing both surfaced a third, previously-masked bug: every script: rule (19 files) imports `from snakemake.script import snakemake`, which is IDE-hint-only in this snakemake version -- snakemake.script exposes no such runtime attribute (only the Snakemake class), and the preamble that actually gets pickled in already provides `snakemake` as a plain global before the rest of the file executes. Removed the broken import repo-wide; the object resolves via normal global lookup exactly as before, including inside the functions/branches a few scripts defer it into. Verified end to end: container_smoke now completes for real through SLURM (jobid 839822, python 3.12.12, sp_validation editable install resolved, correct HEAD commit read from inside the job) with no apptainer exec left in any rule. Dry-run coverage for every rule that owns an edited script (masks_only, cv_weights, and friends) shows clean DAGs. The container-image invariants (no in-image snakemake, exec/run must not source host dotfiles) aren't reproducible from this repo -- the sandbox at /n17data/cdaley/containers/containers has no tracked build recipe -- so they're now documented in workflow/README.md for the next rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * workflow: remove last live apptainer-exec shell call + stale sweep-script docs presentation_pte_cosebis was the one rule left with an inline apptainer exec in its shell: block (container: None override); every sibling presentation_* rule already relies on the module-level container default. Drop the override and the raw call so it's wrapped like the rest. Also update the three sweep-script docstrings (run_xi_sweep, run_cosebis_ptes_sweep, run_cl_sweep) whose example invocations still showed the retired apptainer-exec-then-python pattern, to match the plain `python script.py ...` convention already used by their sibling CLI scripts. * profile: add --cleanenv to apptainer-args Per-node smoke tests showed default env passthrough makes container python resolution nondeterministic (host ~/.local shadowing — the #302 mechanism). --cleanenv makes every job's environment container-defined. Verified: container_smoke green through the profile with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MS3u2QuVSK1Q2CVaxcPNxU * tests: adapt pure-E/B and glass-mock tests to the tomographic API calculate_pure_eb now returns one results dict per tomographic bin pair (#297); the pure-E/B integration test still indexed the flat mode keys. Unwrap the non-tomographic "tomo_bin_all_tomo_bin_all" entry and correct the docstring that still advertised the flat return. glass_mock's map path imports cosmology.compat.camb, which is absent in the image, so the xfail's raises=AttributeError no longer matched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QndBZicN3QvyZG4XDDPmGs * docs: sweep comment bloat across the pivot diff Remove the 'snakemake is injected' comment repeated in 19 script: files (one note in workflow/README.md instead), trim the candide profile header to the operational lessons, and cut re-narrations of the container model in Snakefile/common.py/README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * image_sims: one image, drop dead SLURM env strip, im_mbias_config as script Collapse sif/sif_pipeline to the single sp_validation image (it ships shapepipe; must be rebuilt from uv.lock — the current 2026-07-04 image predates the lock and its numpy 2.5 breaks numba/ngmix). Remove the env -u SLURM_* prefix: shapepipe#744 gates mpi4py on OMPI/PMI vars, and --cleanenv strips the host env anyway (verified in-container). Convert im_mbias_config from a run: block to script:, drop the redundant os.makedirs, and trim pivot re-narration from comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * covariance: restore cosmocov_process.py as a containerized script: rule Deleted in the #236 cleanup with no replacement; restored from history to workflow/scripts/ (the rule is its only caller) and converted the rule from shell: to script:. Fixes on the way: bare exit() on a non-PD matrix returned 0 (Snakemake saw success) — now sys.exit(1); eigvalsh for the symmetric matrix; Agg backend; plot dpi 2000 -> 300. Verified round-trip on synthetic input in the container. Drop the NOTE and stale comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * tests: container_smoke becomes a real pytest, out of the main workflow The rule asserted nothing and put a non-scientific artifact in every paper's results/. Now: tests/data/container_smoke/{Snakefile,script} driven by test_container_smoke.py (@slow, skipped off-cluster), which submits one tiny SLURM job through the committed candide profile and asserts APPTAINER_CONTAINER is set (the job really ran in the image), the editable install resolved, seed-42 eigh values match, and git works inside the container. OMP_NUM_THREADS is recorded, not asserted — unset is the profile's designed state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * workflow: locate shell-invoked scripts via workflow.source_path Replace the hand-rolled WORKFLOW_SCRIPTS constant with Snakemake's first-class mechanism, which the docs specifically prescribe because manual path construction breaks under module composition. The script goes in input: (not params:, which would cause spurious reruns), so it also becomes an honest dependency. Fixes a live bug on the way: papers/bmodes unblinding_ceremony called 'python workflow/scripts/unblinding_ceremony.py', which from the paper workdir resolves into the paper's own scripts/ dir -- stale since 091bba8 and only detectable at run time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * workflow: use script: for single-process rules, source_path only for MPI script: resolves relative to the .smk defining it, module composition included, so it defeats the paper-workdir trap without making scripts into input files -- and matches what the other 20+ rules already do. Only xi_highres keeps source_path, where script: is structurally impossible (snakemake would wrap the whole mpiexec line in one container). unblinding_ceremony was already written for script: -- its _config_from_snakemake was dead code because the rule invoked it via shell:, so it silently ran _config_from_cli, which re-derives paths from constants that no longer exist (a cosmo_val dir deleted from the referenced checkout, and a _PROJECT_ROOT off by one since the script moved to papers/bmodes/scripts/). The rule declared 6 inputs and 9 params while passing 2 on the command line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * papers/bmodes: delete the unblinding ceremony Never used, and silently broken: the rule invoked the script via shell:, so the script's snakemake branch was dead code and its CLI branch re-derived paths from constants that no longer exist. Nothing imported it and no rule consumed its output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * containers: run the CI-published image from one shared path CI builds ghcr.io/cosmostat/sp_validation on every push from uv.lock; the hand-built SIFs it replaces were stale in ways that only failed at run time (no shapepipe.modules in one, numpy 2.5 breaking numba in the other). Every call site -- the Snakefiles, image_sims, the MPI rule's own apptainer exec, the paper shell drivers, interactive use -- now names one file, refreshed deliberately (see workflow/README.md). Overridable with --config container=<path or docker:// URI>. im_mbias_config read the OCI revision by opening the configured sif path; it now reads APPTAINER_CONTAINER, since the image a job actually ran in may be overridden and current.sif is a moving target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * deps: ship CosmoSIS in the image, drop the vestigial cosmology pin CosmoSIS was an undeclared, user-supplied dependency of the inference step -- which is why #303 was hand-patched in someone's ~/.local. It pip-installs into the image against the base gfortran/GSL/cfitsio in ~2 min, so declare it. MPIFC must be set at build time or the sampler Makefiles silently skip the MPI targets and --mpi fails at load; chains must run under MPI because the upstream --smp pool is still broken at 3.25.2. cosmology 2022.10.9 was vestigial: the cosmology.compat.camb adapter comes from cosmology-compat-camb via glass[examples]. Relocking drops it and nothing else. UV_PYTHON pins uv to the image's own interpreter, since $HOME is bind-mounted and uv would otherwise pick a host CPython carrying none of the stack. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * containers: pull the CI image by tag, park the MPI rule Snakemake pulls docker://ghcr.io/cosmostat/sp_validation:develop into a shared apptainer prefix on first use and never again, so no digest or path is written down. One constant, CONTAINER_URI in workflow/common.py, is the single source of truth; host-side callers that need a concrete file (the paper shell drivers, interactive use) derive it via workflow/scripts/container_path.py. xi_highres is parked as a comment block: it has never been runnable -- its shell is a bare 'python run_2pcf_highres.py' while the script requires --cat-config and --out -- and parking it leaves covariance_cosmocov as the workflow's only container exception. The MPI reasoning is preserved in the block for whoever revives it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * tests: drop the glass map-path xfail, its condition is met The xfail asked for a compatible glass+cosmology pair verified in a fresh image. glass 2026.2 with cosmology-compat-camb is that pair: the map path runs end to end (11 shells, 66 spectra, monotonic kappa accumulation), so the marker now only hides regressions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * docker: install liblapack-dev, pin MPIFC absolutely cosmosis's bundled MultiNest links -llapack and the base image ships only the runtime liblapack.so.3 with no dev symlink, so the build died at 'cannot find -llapack'. It passed on candide only because that sandbox had liblapack-dev installed at some point. MPIFC takes the absolute path: /opt/ompi/bin is not always on PATH, and a miss silently drops the MPI sampler libraries while the install still reports success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * docker: point UV_PYTHON at the venv, not the base interpreter uv pip honours UV_PYTHON over VIRTUAL_ENV, so naming the system interpreter sent the editable install of sp_validation there instead of /app/.venv -- the image built fine and then failed its own import smoke test. The venv's own python satisfies the original intent (uv can't wander onto a host CPython from the bind-mounted $HOME) while keeping uv pip pointed at the venv. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017hCncxwiiBZm9ixGgdzCNA * workflow: run the launched checkout's sp_validation by default Snakemake's `script:` directive already executes the checkout's script files, while `import sp_validation` resolved to the image's baked copy -- the two halves of one commit, split. `common.inject_checkout_pythonpath()` prepends the checkout's src/ to APPTAINERENV_PYTHONPATH (preserving any user-set value), so the image supplies the frozen dependency stack and the launched tree supplies sp_validation. This is what the image-sims chain has always done for both repos (`_ENV_PREFIX` in rules/image_sims.smk); the main workflow now matches it. Opt out with `--config checkout_pythonpath=false` to reproduce from the image alone. The flag is parsed tolerantly because `--config k=false` can arrive as the string "false". Also drops the hardcoded candide OpenMPI path from workflow/Snakefile: a machine path does not belong in generic workflow code, and it moves to the candide profile in a following commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * profiles: add a machine-independent default, make candide the machine layer workflow/profiles/default carries the container model and nothing else, so the workflow runs off candide with `--profile workflow/profiles/default -j N`. Snakemake cannot compose profiles (one --profile, no `inherits:`), so the small machine-independent set -- software-deployment-method, rerun-triggers, latency-wait -- is duplicated verbatim in both files, marked GENERIC and cross-referenced. That is the least-magic arrangement available. apptainer-args and apptainer-prefix stay out of the shared block: every machine has its own disks and image cache. candide's apptainer-args now carries `--env LD_LIBRARY_PATH=/softs/openmpi/...`, previously an os.environ line in workflow/Snakefile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * presentation: containerize the two rules that ran bare python The Moriond talk-figure rules called `python` with no container, so they ran against whatever interpreter the driver happened to have. Let them inherit the module-level `container:` like every other rule. The two ImageMagick `convert` rules keep `container: None`: `convert` is a host tool, absent from the image. Same for covariance_cosmocov, whose docstring says so directly rather than pointing at a list elsewhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * image_sims: default sif to null and fall back to the workflow's image `image_sims: {sif: ...}` was a required structural key, so every run config repeated the image path — a second place for it to drift from what the rest of the workflow runs. Default it to null and resolve it through the same code path as every other entry point; a run config still overrides it to name its own image or a branch tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * containers: give every user their own image, driven by spv-container The workflow ran out of one shared image directory on candide (/n17data/cdaley/containers/snakemake-sif) with a hand-maintained current.sif symlink pointing at whatever Snakemake last autopulled. That only worked for one person: refreshing the image or repointing the symlink needed write access to another user's directory, and a refresh moved the ground under everyone at once. Everyone now runs their own image file at one canonical per-user path, ~/.cache/sp_validation/sp_validation.sif (SPV_CONTAINER overrides it), owned by a small CLI: spv-container pull # fetch the tag there, atomically spv-container status # revision label vs. this checkout's HEAD spv-container exec <cmd...> # one-off run inside it, candide binds applied sp_validation.container is stdlib-only on purpose: it runs on the host, outside the container, so it must import without the scientific stack — and it works straight from a checkout (`python3 src/sp_validation/container.py status`) with nothing installed. It also holds CONTAINER_URI, which workflow/common.py loads from this checkout by file path, so the CLI and the workflow can never name different images. `container:` now resolves to that local .sif when it exists and to the registry tag otherwise (Snakemake accepts either, and autopulls the tag into .snakemake/singularity). `--config container=...` still overrides both. The candide profile drops apptainer-prefix accordingly, and common.configure() warns — once, never fatally — when the local image predates the checkout. Also drops workflow/scripts/container_path.py, which existed to locate the shared cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: tell the container story once, around the per-user image Rewrites the container sections of workflow/README.md, CLAUDE.md, CONTRIBUTING.md and README.md for the per-user model: one image per person at ~/.cache/sp_validation/sp_validation.sif, `spv-container` to fill and inspect it, and how `container:` resolves to it. The shared-prefix machinery is gone -- current.sif bootstrap, the atomic-mv refresh recipe, the group-writable TODO. Trims the commentary while there. The profile pair says "change one, change the other" once instead of shouting it in three places; off-candide gets a paragraph rather than parallel billing, since candide is where everyone runs; and the enumeration of container exceptions is dropped in favour of the docstring on each rule that opts out. Also repoints the two paper Snakefiles, which resolve `container:` themselves, at common.resolve_container -- and replaces the obsolete `apptainer build --sandbox` recipe in README.md and installation.rst with `apptainer pull`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * containers: add an opt-in writable sandbox, and one resolution order The pristine SIF is read-only, which is what you want almost always -- but it lost the one real advantage of the old hand-built sandbox workflow: `pip install` mid-analysis, when you need a package the image does not carry yet and a CI rebuild is too slow a loop to think in. `spv-container sandbox` unpacks the image into a writable directory at ~/.cache/sp_validation/sandbox/, and `spv-container exec --writable` runs against it so installs persist. Opt-in: nothing builds one for you. Resolution order is now one thing, shared by the CLI, the run_*.sh drivers and the workflow's `container:` -- sandbox if it exists, else SIF if it exists, else the registry tag. Snakemake execs a sandbox directory as happily as a .sif, so a package installed into the sandbox is there for workflow jobs too, with no further wiring. `resolve_image()` in sp_validation.container is the single implementation; common.resolve_container defers to it. The build stages into a sibling directory and swaps it in, as `pull` does, for a sharper reason than pull has: a half-written .sif fails loudly, but a half-unpacked sandbox is still a *directory*, so resolution would elect it and every job would silently run a broken tree. Building before removing also means a `--force` rebuild that fails -- a typo in --source, a network blip -- leaves the sandbox you already had intact, instead of deleting a working environment on the way to not replacing it. The cost of a sandbox is that what runs is no longer fully described by a revision label, so the divergence is made visible rather than left silent: `status` names which layer is live and says the revision only describes what the sandbox was built from (falling back to the SIF's label, marked as inferred, when the sandbox carries none), and the workflow prints one line at launch when a sandbox is in play. `spv-container pull && spv-container sandbox --force` resets. Verified on candide (apptainer 1.5.3): unprivileged `build --sandbox` works through user namespaces with no fakeroot and no subuid mapping; `exec --writable` persists writes while plain `exec` gets a read-only filesystem; `inspect --labels` still reports the source image's OCI labels from a sandbox directory; `--fix-perms` at build time is what keeps the tree removable afterwards (without it apptainer leaves directories that defeat `rm -rf`, which would strand `--force`); and a failed `--force` rebuild leaves the existing sandbox and its contents untouched, with no staging directory left behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * image: build the cosmosis-standard-library fork into the container The cosmo_inference .ini templates pointed COSMOSIS_DIR at two different people's home directories (/home/guerrini/... and a scratch path of Lisa's), so running the inference pipeline meant either being one of them or editing the templates by hand. CosmoSIS itself already ships in the image via the `workflow` extra; only the Standard Library — the tree of module files the pipelines name — was missing. Clone and build it at /opt/cosmosis-standard-library, pinned to Sacha Guerrini's fork at b26fa7ff. That fork is 4 commits ahead of upstream and 373 behind; the four are what the UNIONS pipelines need (tau statistics, sample_S8, two z-dependent linear-alignment modules). Carrying them onto current upstream is future work, noted in cosmo_inference/README.md. The templates now read COSMOSIS_DIR from %(CSL_DIR)s, which the image sets — CosmoSIS reads environment variables into an ini's [DEFAULT] section, which is how the existing %(SCRATCH)s references already work. Off-image, export CSL_DIR and the same templates work unchanged. The build follows CSL's documented procedure for a pip-installed cosmosis (`source cosmosis-configure && make`), but targets `shear/` rather than the top-level `make`: the top level also descends into likelihood/, building the Planck, WMAP and ACT likelihoods, which no UNIONS pipeline uses. Of the modules our templates do name, all are pure Python except two under shear/ — `limber`, which project_2d.py links, and cl_to_xi_nicaea's nicaea_interface.so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: prune duplicated and historical comments The container model, the checkout-PYTHONPATH default, the profile GENERIC mirroring and the CSL_DIR resolution were each explained in three to six places. Give every concept one home -- workflow/README.md for the user-facing story, the docstring of the thing itself for mechanism -- and leave pointers elsewhere. Drop comments narrating what the code used to do; git holds that. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * simplify: collapse duplication added by this branch The container model landed the same few lines in several places; fold each into one home. * the four run_*.sh sweep drivers resolved this user's image (and repeated the bind list) inline -- now one sourced papers/bmodes/scripts/container_env.sh, resolving exactly as sp_validation/container.py does (sandbox first, then SPV_CONTAINER/XDG_CACHE_HOME, binds from SPV_APPTAINER_BINDS) * container.py: one _require_apptainer() instead of three copies of the PATH guard, and compare_revision's merge-base calls go through _git * common.py: resolve_container takes the override value, so image_sims.smk no longer wraps IMSIM["sif"] in a synthetic config dict; drop the CONTAINER_URI / local_sif / local_sandbox re-exports, which have no callers * cosmocov_process.py: only Snakemake runs it, so drop the argv entry point and its main() indirection, matching im_mbias_config.py * xip_xim.py: one catalog() builder for the tomographic and non-tomographic paths instead of two near-identical treecorr.Catalog blocks Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * simplify: give the sweep drivers one shared preamble; fold a duplicated README section The four papers/bmodes sweep drivers each repeated the worktree path, the derived script/source dirs, and a full `apptainer exec ... /usr/local/bin/python` invocation (six sites). container_env.sh now owns all of it and exposes `spv_python` / `sweep_versions`; argv is byte-identical. workflow/README.md explained `--config container=` twice, once for a local .sif and once for a branch tag. One subsection now covers both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * image: actually build CSL — cosmosis-configure exits 0 under set -u without running make Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc * image: drop set -u in the CSL layer — the configure exports append to unset paths The test -f artifact guard is what keeps a no-op build loud; -u had become the thing breaking the build instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc * image: point limber's Makefile at Debian's GSL (GSL_INC/GSL_LIB) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C856B4eJ3LwXrj9SCiEuMc * rebase onto develop: shed tomography-branch remnants The container/workflow work is orthogonal to the tomography branch it was accidentally based on. Restore develop's pure_eb docstring, test_cosmo_val call shape, and glass_mock xfail; keep develop's glass==2025.1 pinned set (cosmology 2022.10.9 is load-bearing there, not vestigial) and relock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC * docs: make spv-container the install story README leads with the four-line install (clone, symlink onto PATH, pull, exec-check); container.py gets a shebang + exec bit so the symlink is a real CLI with no packaging. installation.rst carries the depth (subcommands, per-user model, sandbox, raw apptainer/docker); CONTRIBUTING and workflow/README point at the same symlink step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0143zcEsfSWr13AfSroMteEC * CSL runtime coverage: import-test template modules; patch fork for scipy>=1.15; add fast-pt The image build only compiled CSL; pure-Python modules were never loaded until a pipeline ran. Two runtime breaks shipped in a green image: scipy>=1.15 removed scipy.special.lpn (legendre.py, reached by every real-space likelihood via spec_tools -- #316), and project_2d.py imports fastpt, which the image never carried. - test_csl_modules.py imports every .py module the ini templates reference, in-image (skips without CSL_DIR/cosmosis) - Dockerfile applies upstream a8a941d5 (lpn fix) as a patch until the fork absorbs it (#316) - fast-pt>=3.2,<4 joins the workflow extra (4.0 restructured; CSL expects 3.x) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmGA86b7YyQn54JFj78ssM * ruff autofix (format + safe lint fixes) Pushed by the lint gate. * Repoint CSL at the UNIONS-WL org fork; trim fast-pt annotation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmGA86b7YyQn54JFj78ssM * ruff autofix (format + safe lint fixes) Pushed by the lint gate. * lint: noqa E402 on the in-section container-smoke import * Drop CSL scipy workaround after fork merge Pin the container to the current UNIONS-WL fork main commit, remove the now-obsolete lpn patch, and update the inference documentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XTNCvVQNLXZVTiDR1PZaVG --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Concurrent doc-deploy runs on back-to-back pushes to develop both hit peaceiris/actions-gh-pages at once, and the loser fails with "cannot lock ref 'refs/heads/gh-pages'" (seen in run 34250154443). Scope the job to a gh-pages-<ref> concurrency group; cancel-in-progress is fine since develop only ever has one active deploy target and the newest push should win over a stale one still building. Claude-Session: https://claude.ai/code/session_01Wk8SZkCRuKQ5xu5g38zHpx Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Resolve reporting bins from their edges rather than bin means. This keeps the 12--83 arcmin fiducial window at bins 9--15 and preserves the intended degrees of freedom. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The texlive-latex-base/-recommended/-fonts-recommended + cm-super set added in #338 costs ~1.3 GB and still lacks type1cm.sty, so matplotlib's usetex path stays broken. Replace it with TinyTeX (~270 MB measured), pinned to TeX Live 2025 on both halves: bundle v2026.02, and tlmgr pointed at that year's frozen tlnet-final historic mirror. Packages are an explicit tlmgr list covering matplotlib's usetex preamble. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Container: TinyTeX pinned to TeX Live 2025, replacing apt texlive
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Pin peaceiris/actions-gh-pages action to 84c30a8
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Deletes code with no live caller:
- workflow/scripts/run_2pcf_highres.py and the parked, unrunnable
xi_highres block in workflow/rules/twopoint.smk that was its only
reference; the mpi4py comment in pyproject.toml no longer names it.
- papers/bmodes/scripts/run_cov_sweep.sh, which calls
workflow/scripts/run_cosmocov_chain.sh, absent from develop.
Fixes docs that point at paths or tools that do not exist:
- CLAUDE.md, CONTRIBUTING.md: single-test example uses test_cosmo_val.py.
- CLAUDE.md: cosmo_inference runs through Snakemake (inference_fiducial),
not a pipeline.sh driver.
- papers/{bmodes,cosmo_val}/Snakefile: read cat_config through code/
rather than the deprecated pure_eb symlink.
- ecut_spec.md, update_survey_stats.py: paths under papers/bmodes/.
- README badge and installation docs: Python 3.12 floor, matching
requires-python.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeMhD6yCyrgBz5oz87bCtR
calculate_pure_eb_correlation returns a self-describing results dict (theta, bin edges, xi+/- on both grids, n_eff) in place of the gg/gg_int objects, so calculate_eb_statistics, the plot helpers and save_pure_eb_results work from values alone. New seams: bins_from_edges, log_bin_edges, pure_eb_from_xi, pure_eb_covariance_mc and cosebis_scan_from_xi. The pure-E/B .npz stores n_eff (the realisation count behind the covariance) instead of npatch. Callers in cosmo_val, the bmodes calculate_pure_eb_ptes script, its claims rule and PTE sweep driver are adapted. No numbers change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HeMhD6yCyrgBz5oz87bCtR
cs_util's get_theo_xi returns {pair: (xi+, xi-)}, so concatenating its
return value raised TypeError and the semi-analytic pure-E/B covariance
(pure_eb_covariance_mc, and the paper's precompute_pure_eb_chunk.py) could
not run. One n(z) is one tracer pair: unpack that single entry, which also
fails loudly if a tomographic n(z) is passed. The new test stubs the
kernel and checks the draws centre on the binned theory mean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeMhD6yCyrgBz5oz87bCtR
Remove dead scripts and fix stale doc paths
* tests: pin pure-E/B on exact-binning and committed ξ±; drop dead catalogue paths The synthetic pure-E/B pins passed in CI and failed on candide because TreeCorr's default bin_slop/angle_slop make ξ± follow the tree's top-level split, which varies with the jackknife patches and, through min_top, with the thread count TreeCorr takes from cpu_count(). Fixed patch centres alone leave a 4-vs-48-thread spread (reporting ξ− up to 16%); exact binning removes it (1e-12). The test measures with bin_slop = angle_slop = 0, and test_b_modes pins pure_eb_from_xi on the same ξ±, committed as tests/data/pure_eb_xi_fixture.npz. The configured-path guard skips cat_config's paths.output and directory-less calibration params.input_path values. The two LFmask entries (data gone) and the six unread covmat_file keys leave cat_config.yaml, and the slow duplicate test_catalog_paths_exist goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * cosmo_val: pin TreeCorr's min_top so ξ± does not depend on the machine TreeCorr derives min_top, the depth of its root cells, from its thread count (max(3, ceil(log2 n)), with n from cpu_count() when unset). The root cells set which pairs bin_slop approximates, so at default slop calculate_2pcf's ξ± depended on the node: on the synthetic catalogue, 48 threads move ξ± by 0.038σ against 4. The shared treecorr_config pins min_top = 6, which is what TreeCorr derives on candide's 48- and 64-CPU nodes; the same config reaches the aperture-mass, leakage and ρ/τ correlations. test_calculate_2pcf_does_not_depend_on_thread_count measures at production binning on 4 and 48 threads (fresh Catalog each, shared patch centres) and requires agreement below 1e-6σ; it is red without the pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * b_modes: one pure-E/B kernel call; tests pin each pure-E/B fact once calculate_pure_eb_correlation (modes and jackknife) and pure_eb_covariance_mc call cosmo_numba through pure_eb_from_xi, so the transform test_b_modes pins is the one every pure-E/B product runs through. The committed ξ± fixture is now a conftest fixture (pure_eb_xi) with two users. The synthetic pure-E/B test asserts that the ξ± and edges it measures equal the fixture (agreement 2e-13), and that its modes are pure_eb_from_xi of them; this replaces its copy of the mode pins, and np.savez of what it measures regenerates the fixture. test_b_modes pins pure_eb_from_xi on the fixture, which fails loudly if cosmo_numba does not import, with no skip. A failure now names what moved: the measured ξ±, the wiring into the kernel, or the transform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * b_modes: one pure-E/B transform per jackknife realisation The jackknife covariance function indexed pure_EB(x) once per key, so every realisation ran cosmo_numba's transform six times (55 transforms at npatch=8 where 10 suffice; values unchanged). _eb_vector concatenates the modes in _EB_KEYS order for both the jackknife and the MC covariance. The synthetic pure-E/B test counts transforms around calculate_pure_eb and is red on the per-key closure (55 > npatch + 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * tests: glue-section header stops enumerating which tests compare values Two tests in the section compare values (pure-E/B against committed ξ±, ξ± across TreeCorr thread counts); the header defers to each test's docstring instead of listing them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: host/image parity check, catalogue config from the checkout, one output root, one integration grid - Launch-time parity: container.image_runtime reads the image's Python minor and Snakemake version (SIF via apptainer, sandbox off disk, tags skipped); common.check_host_parity stops a launch whose host Snakemake differs, with the reinstall command. configure() and image_sims.smk call it; the README install line pins --python 3.12. - The catalogue config is the launched checkout's cosmo_val/cat_config.yaml, loaded by configure() into CATALOG_CONFIG; the paper Snakefiles no longer merge it into `config`, and covariance.smk / ecut.smk read CATALOG_CONFIG. - One output root: cv_init_params passes output_dir=COSMO_VAL, cv_runner no longer chdirs into the live checkout, and CosmologyValidation drops its COSMO_VAL environment fallback. - One integration grid (R12): the cosebis grid is gone; cv_cosebis reads the integration part and the CosmoCov g covariance on that grid (the one pure-E/B uses). An npatch=1 grid defaults to the diagonal covariance, and a binning outside the named grids takes its covariance from its own patches. - The candide profile sets jobs: 100. - workflow/tests: host-launcher DAG tests on a toy checkout (P1-P3) and the real papers on candide (P4, P5 = the container smoke test, moved here); CI runs them in a workflow-dag job. test_bmodes_workflow_dry_run.py is replaced by P4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: the image lives under ~/.cache; launch guards read the real image - container.CACHE_DIR is ~/.cache/sp_validation whatever XDG_CACHE_HOME says. A job runs the image from the path the launching host resolved, and on candide XDG_CACHE_HOME is node-local /scratch: from such a shell the launch fell back to the registry tag and skipped the parity check. SPV_CONTAINER / SPV_SANDBOX remain the overrides. - test_launch_reads_the_image_under_home: a mismatched image under a fake ~/.cache stops the launch while XDG_CACHE_HOME points elsewhere. - test_papers_resolve_on_candide asserts unconditionally that the launch read a local image (no "parity unchecked"). - test_image_sims_checks_parity_at_launch: the standalone image-sims Snakefile stops on a mismatched image. - test_assemble_resolves pins each terminal file's inputs: the reporting ξ± part with its CosmoCov ng covariance, the fiducial-binning pseudo-Cl part with its NaMaster covariance, COSEBIs, pure-E/B and ρ/τ. - The container smoke test launches as the README does (no --jobs), so the candide profile's job bound is under test. - Test docstrings drop the design-table row IDs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: a ξ± part's covariance follows its patches, in one place A grid is a binning; rule xi takes cov=patch_cov(npatch) directly, so grid_cov, _named_grid and the grids' cov key go. cv_init_params loses its unused version_list, and the cosmo_val rules pass one CV_INIT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow/README: scope the output-root sentence to papers/cosmo_val Name where the other rules write: masks under the run directory's output/masks/, papers/bmodes' figures and macros under its docs/, image sims under grids_base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: SLURM jobs start whatever the launch's XDG_CACHE_HOME; P4/P5 test what a launch does A job's Snakemake inherits the launching shell's environment (--export=ALL), and a login shell may point XDG_CACHE_HOME at node-local storage: every job then fails creating its source cache. Two halves close it, each needed (shown on n33 by replaying the captured sbatch job command with srun stubbed): - the candide profile leaves source-cache out of shared-fs-usage, so a job neither reuses the launch's cache path nor creates its own under XDG; - common.py drops XDG_CACHE_HOME from what jobs inherit, because the slurm-jobstep executor forces a shared source cache on the Snakemake it starts for each job step. test_a_job_needs_no_launch_cache covers both (each mutation turns it red). P4 (test_papers_resolve_on_candide) now passes from a shell with node-local XDG_CACHE_HOME under srun, and its bmodes case also resolves an e-cut catalogue, so both CATALOG_CONFIG readers in ecut.smk are exercised. P5 hands the smoke Snakefile the image a launch resolves (no registry pull). The host launcher and CI carry snakemake-executor-plugin-slurm, as the README install line does, so the candide profile is parsed in CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * papers/bmodes: the sweep drivers read the image from ~/.cache, as container.py does container_env.sh still followed XDG_CACHE_HOME, so from a login shell that points it at node-local storage the sweep drivers ran a missing image while spv-container and the workflow found the real one. test_sweep_drivers_run_the_resolved_image pins the shell copy of the resolution (cache and sandbox precedence) to container.resolve_image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * run_2pcf: a ξ± part carries the covariance its measurement estimated Which covariance a part carries had two homes that disagreed: rule xi passed cov=patch_cov(npatch) ("diagonal" at npatch=1), while run_2pcf's own default, reached by the CLI and papers/bmodes' run_xi_sweep, was "none". The same file name then held variances or not depending on who wrote it. run_2pcf now carries what TreeCorr estimated (gg.var_method, which calculate_2pcf sets from npatch): the jackknife covariance with patches, the shot-noise diagonal without. The cov argument, --cov, the jackknife guard, rule xi's cov param and common.patch_cov (with its test) go. Rule xi hands its wildcards to grid_of directly (xi_binning_of goes). test_xi_part_carries_the_covariance_the_measurement_estimated runs run_2pcf on the synthetic catalogue at npatch 1 and 4; the previous run_2pcf fails both cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow/tests: the launch-cache test answers the profile's apptainer check with a stub The candide profile deploys with apptainer, and Snakemake asks for the apptainer binary and its version even in a dry-run, so test_a_job_needs_no_launch_cache failed on GitHub's runners, which have no apptainer. A stub apptainer on the test's PATH answers that check; the dry-run reads nothing else from it. In a CI emulation (a clean checkout, a fresh HOME, no apptainer or SLURM on PATH, the workflow-dag command) the suite goes from 1 failed, 11 passed to 12 passed. Both mutations still turn the test red there: source-cache added back to the profile's shared-fs-usage (NotADirectoryError under the blocked XDG_CACHE_HOME) and the XDG_CACHE_HOME pop removed from common.py (the job sees the launch's cache). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow/tests: the smoke Snakefile composes workflow/ as the entry Snakefiles do P5's Snakefile never imported common, so its launch did not drop XDG_CACHE_HOME: from a login shell pointing it at node-local /scratch, the job's inner Snakemake died creating its source cache (PermissionError), which says nothing about what a real launch does. The Snakefile now imports common, resolves its image with common.resolve_container and checks host/image parity, as the entry Snakefiles do. P5 launches exactly as the README does, with no --config container=, and the literal default image and the test that kept it in step with CONTAINER_URI go. Checked on n33 with sbatch stubbed to record what it would submit, then the recorded job replayed with srun stubbed, from a shell with XDG_CACHE_HOME set to /scratch/cdaley/tmp/xdg: at the parent commit the job environment carries XDG_CACHE_HOME and the replay fails with PermissionError; with this commit it carries none, the job runs in the SIF, and P5's assertions pass on its report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: resolve_container checks the image it returns The host/image parity check had a home beside the resolver, so configure() resolved the image once only to check it, and every Snakefile that picks its own image (image_sims.smk, the smoke Snakefile) had to remember a second line. resolve_container now runs check_host_parity on the image it returns: the image checked is the image that runs, and the extra call sites go. check_host_parity stays cached, since composed Snakefiles evaluate container: more than once. Removing the check from resolve_container turns 5 host tests red (both parity cases, the unreadable-image line, the image under ~/.cache, image sims). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow/tests: P5 checks that the job imports the launched checkout's src The smoke Snakefile imported common but never put the checkout's src/ on the job's PYTHONPATH, so its job imported the image's baked sp_validation, and P5's assertion (any editable src/ layout) passed on it. The smoke Snakefile now calls inject_checkout_pythonpath as configure() does for the entry Snakefiles, and P5 asserts the job's sp_validation is this checkout's src/sp_validation/__init__.py. Run on n08 through the default profile (apptainer, no SLURM): the job reports <checkout>/src/sp_validation/__init__.py; with the injection removed it reports /sp_validation/src/sp_validation/__init__.py, which the new assertion rejects and the old one accepted. P5 itself needs a SLURM submit host and has not been run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow: COSMO_VAL defaults to the launched checkout's own output tree A launch from any checkout that named no COSMO_VAL read that checkout's catalogue config and code but wrote every cosmo_val product into the production tree, silently. COSMO_VAL now defaults to the launched checkout's (gitignored) cosmo_val/output, so production is written only from the production checkout or when a launch names it; COSMO_INFERENCE keeps its shared default. README says so. P2 gains an unnamed case: with COSMO_VAL unset, every declared output lies under the toy checkout's cosmo_val/output, the inference root or results/. Restoring the production default turns it red (outputs under /n17data/cdaley/unions/code/sp_validation/cosmo_val/output). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * calculate_2pcf: the ξ± text dump carries columns only, so it reads back With patches, calculate_2pcf wrote its .txt with write_cov=True and no per-patch results. TreeCorr 5.1.4 writes num_rows only alongside patch results, so its reader ran on into the cov block and raised ("got 12 columns instead of 11"). The two ξ± figure rules re-enter calculate_2pcf on the reporting grid (npatch=100) and hit exactly that read. The dump now carries the columns only; the covariance matrix lives in the SACC part, and the figure readers use the columns. test_a_patched_xi_dump_reads_back measures at npatch=4, then re-enters calculate_2pcf from a fresh CosmologyValidation and compares the columns; under write_cov=True it fails with the ValueError above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * workflow/tests: the candide profile's job bound is checked on any host A real launch through the committed candide profile, of an up-to-date target, submits nothing, so it runs in CI and on an allocation; the same launch through the profile without `jobs` is refused. test_container_smoke submits a real job and needs sbatch, which only a login node has; it is documented as the login-node check it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EGDJZeA8v26eKGAHQqh5j * profiles: wait 60 s for a job's outputs to appear P5 (one real SLURM job through the candide profile, from a login node) passed, but only on its retry: the job finished, its output took more than 5 s to show on the login node's /home, and Snakemake re-ran it. On a multi-hour job that retry costs hours, and a second miss fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * rho_tau_stats: 6 h wall clock; the jackknife ρ/τ outruns the 60 min default Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * Scrub the n(z) A/B/C blind: an n(z) is a catalogue entry's Choosing an n(z) is choosing a catalogue entry: each entry's shear.redshift_path is its n(z), read by CosmologyValidation.get_redshift and by the workflow's redshift_path(version). CosmologyValidation loses its blind override, and the pseudo-Cl, covariance, inference and papers/bmodes filenames lose their blind token and wildcard. - common: catalogue_entry()/redshift_path() replace build_redshift_path; covariance_{base,dir,path} and pseudo_cl_tag drop blind; the version constraint admits the _A/_B/_C entries. - covariance.smk: get_cat_params and the star-halo mask resolve through the catalogue entry. - papers/bmodes: bb_covariance_blind_independence and the talk n(z) plot compare fiducial.nz_realisations, the SP_v1.4.6.3_{A,B,C}_leak_corr entries; pure-E/B, COSEBIs PTE and pseudo-Cl paths are per version. - cat_config: SP_v1.4.6.3_{A,B,C} match SP_v1.4.6.3 but for their n(z); SP_v1.4.8 and SP_v1.4.11.3(_ecut07) name the n(z) their covariances used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * cat_config: the leakage n(z) is a path template, not an A/B/C blind key nz.dndz.path names the file with {pipeline}; the same file as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * cat_config: SP_v1.4.11.2 reads the n(z) its covariance used Like SP_v1.4.11.3, its covariance was built from nz_SP_v1.4.6_A.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * workflow: no snakemake in the image, so any host Snakemake works A script: job unpickles the host's snakemake object with whichever snakemake it imports first, and the image's site-packages precede the host's appended sys.path. The image carried its own (the workflow extra, and the base image's jupyter extra), so the host had to match its version exactly. The Dockerfile now uninstalls every snakemake* package after the sync and the workflow extra drops snakemake; the job then reads the pickle with the package that wrote it. The launch check keeps only the Python minor (check_host_python): the host's snakemake and its compiled dependencies load into the image's interpreter. The README install line, CI's DAG job and the test harness no longer pin a Snakemake version. The container smoke job now reports which snakemake unpickled its object and asserts it is the host's version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * profiles: every key needed and non-default, each with its reason Drops the candide LD_LIBRARY_PATH to /softs/openmpi: /softs is not bound, so the path does not exist inside the container, and no containerized rule uses MPI. Drops the default profile's --bind /home (apptainer mounts $HOME already). States the real reasons for the rest: rerun-triggers leaves out software-env because it hashes the per-person image path; shared-fs-usage leaves out source-cache because jobs would be handed the launch's node-local cache path; slurm_account because the executor's guess fails on candide; retries and kept logs for fan-outs and their printed output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * workflow: jobs read the repo's matplotlibrc, never the launching user's Apptainer binds $HOME, so a job read the launcher's own matplotlibrc, and a LaTeX preamble there the image cannot typeset stopped every figure rule. common.py points each job's MATPLOTLIBRC (through APPTAINERENV_, past --cleanenv) at an empty workflow/matplotlibrc. The container smoke job reports the matplotlibrc it would read and the test asserts it is the repo's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * CONTRACTS: the host-side workflow imports only stdlib and snakemake workflow.common runs in the host Snakemake with no sp_validation installed, and loads container.py by path; container.py imports only the standard library, since it also runs as the spv-container CLI before any image exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * profiles/default: bind /home; Snakemake's --home <workdir> keeps apptainer from mounting it A job's home is its working directory, so a checkout or output tree under the launching user's home was invisible to the job: the toy run's jobs imported the image's sp_validation instead of the checkout's src/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TJSfyQjoQXjPKsEfkGhZLj * workflow: jobs read the user's matplotlibrc; no snakemake-uninstall step The image's TeX now carries sfmath, so a usetex preamble typesets inside jobs and the repo matplotlibrc override has nothing left to guard. The shapepipe:develop base ships no snakemake, so the uninstall step is gone; pyproject says why the workflow extra must not re-add it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * workflow: resolve the image without a host/image Python check; trim profile and README The launch resolves its image and runs it; the README states that the host Snakemake runs on the image's Python (3.12). The CONTRACTS notes go (the workflow-dag CI job is what enforces host-importability). The candide profile excludes n17 and n36 only, and the README's output-roots paragraph and the profile's shared-fs-usage comment say what they need to in fewer lines. The DAG tests drop the Python-check tests and the profile job-bound test, and test_one_integration_grid checks only the shared COSEBIs/pure-E/B inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: drop the smoke test's numeric check and perf-pinning asserts The container smoke job reports and checks placement, imports, Snakemake version and provenance; the host-side suite no longer needs numpy. test_xi_grids loses its grid-count test, and the pure-E/B cosmo_val test keeps its value checks without counting kernel calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * generate_paper_macros: read the single pure E/B joint PTE, drop per-blind macros pure_eb_data_vector writes one fiducial `pte_joint`, so \ebfiducialPte and \ebfullPte read that key. The covariance_blind_consistency and per-blind PTE-spread macros go: nothing produces their inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * paper macros: drop the unused COSEBIS macros and the inputs no script reads generate_paper_macros no longer emits \cosebisfiducialPte, \cosebisfullPte or \cosebisthetaMin/Max: they read keys cosebis_version_comparison evidence does not carry and no paper uses them (the COSEBIS PTEs come from \configPte<Ver>Cosebis[Full]). The macro rules drop the cosebis and BB realisation evidence inputs accordingly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * bb_covariance_nz_independence: realisations from fiducial.nz_realisations The BB-covariance n(z)-independence check (renamed from bb_covariance_blind_independence: rule, script, tapestry outputs) takes its realisation labels from config fiducial.nz_realisations, the first as the reference and every other compared against it; figure markers, evidence keys (<label>_to_<reference>, plus reference_realisation) and summary maxima follow the configured set rather than a hardcoded A/B/C. The summary flag is cosebis_bb_nz_independent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * workflow: redshift_path reads shear.redshift_path as written Matches CosmologyValidation.get_redshift, which does not join the entry's subdir; base_version's docstring names what uses it (footprint, plotting style). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Lift the scipy cap; require camb>=2.0 camb 2.0 makes its BBN Y_He predictor return a scalar, so set_cosmology works under scipy 1.18.0 (whose RectBivariateSpline returned shape (1,) for scalar input) and the scipy cap has no reason to exist. camb 2.0 moves the glass-mock CAMB fingerprint beyond the test's 1e-6 tolerance (As +1.4e-4, P(k) up to 2.9e-4 relative), so regenerate that reference under camb 2.0.4. The test now pins camb.config.AccuracyTarget to CAMB's default: pyccl's CAMB path sets that process-global to 0, which made the reference test depend on whether a pyccl test ran first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ * Require pyccl>=3.3.5 (boltzmann_camb on camb 2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * uv.lock: camb 2.0.4, scipy 1.18.1, pyccl 3.3.6, cs_util/shear_psf_leakage develop heads cs_util develop@e14cd686 (CosmoStat/cs_util#93) and shear_psf_leakage develop@649edf4a (CosmoStat/shear_psf_leakage#45) lift their scipy caps and admit camb 2; shear_psf_leakage no longer pulls gsl or stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ * test_glass_mock: widen the tolerance instead of pinning AccuracyTarget Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Relock on the merged pyproject: camb 2.0.4, shear_psf_leakage develop tip Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W4pb4vZJimNoYkWqQCqEwR --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…XdataN paths (#368) * rho/tau rule script without its interactive scaffolding; docs for a private COSMO_INFERENCE - workflow/scripts/run_rho_tau.py is a plain rule script: the IPython autoreload branch, hard-coded interactive paths and progress prints go; stream buffering and the output check come from cv_runner. - workflow/README: launching with COSMO_INFERENCE=<tree> of one's own, with the mask link the CosmoCov chain needs. - Docstrings: calculate_rho_tau_stats; sacc_writers notes that the integration-grid ξ± part feeds COSEBIs and pure-E/B and does not join {version}.sacc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cosmo_val: TreeCorr runs on the CPUs the process holds treecorr_config defaults num_threads to the process's CPU affinity. TreeCorr's own default is multiprocessing.cpu_count(), the node's count: a job holding 16 of a node's 24 CPUs ran 24 threads. A caller's num_threads still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * workflow: the checkout and output roots take the plain /nXXdataN spelling common._plain resolves a path and spells /automnt/<disk>/... back as /<disk>/..., on every host; REPO_ROOT, COSMO_VAL and COSMO_INFERENCE go through it. Snakemake keys its persistence records and matches targets by path string, so a symlinked or /automnt-spelled root no longer declares paths a plainly named target cannot match, and a job step re-deriving the launch's paths on the node that owns the disk (which has no /automnt/<disk>) declares paths it can reach. The README names the disks a job sees under their plain spelling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sacc_writers: module docstring as on develop Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…GB on SP_v1.4.6.3 (#369) Claude-Session: https://claude.ai/code/session_01UzeJqdtGgoeWizre32L5fD Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…he rule (#371) glass_mock_xi_fine measured mocks on a hardcoded 0.5'–500' / 1000-bin grid while the data's E/B transforms run on XI_GRIDS["integration"]; the two must match node for node, or mock and data B-mode vectors integrate different θ samplings. The mock rule, its output name (grid tag, as the data ξ± carry), the COSEBIS scatter/bias-test inputs and the bias test's covariance grid now all come from XI_GRIDS["integration"]. The COSEBIS scripts read the binning from the file (GGCorrelation.from_file) instead of restating it. run_glass_mock_2pcf passes the rule's thread count to gg.process; TreeCorr's own default is every CPU on the node. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
calibrate_min_pte takes a (n_mocks, n_stats) PTE matrix from noise-only realisations and returns the alpha-quantile of the per-mock minimum PTE (with a distribution-free order-statistic interval), the implied effective number of independent tests k_eff from 1 - (1 - t)^k = alpha, and a global_pte method giving the data's global p-value with a Wilson interval. Statistic-agnostic: columns can be any statistics, bin pairs or scale cuts. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Pure E/B as a fixed-quadrature operator with exact covariance
The pure-E/B estimator is one data-independent matrix K = (I_6 ⊗ P)·M on
the fine ξ± grid: the Schneider (2022) transform with fixed Gauss-Legendre
weights (cosmo-numba get_pure_EB_operator), evaluated at the fine nodes
inside the reporting range with [tmin, tmax] at the fine-grid extent, then
averaged into the reporting bins with pair-count weights. Its covariance is
K C_ξ Kᵀ, exact for an analytic or a jackknife ξ± covariance; `npatch`
travels with the results (None for analytic) and sets the Hartlap factor of
every χ² built on them, with the combined ξ+/ξ− χ² debiased over its full
length.
calculate_pure_eb_correlation works from fine-grid arrays; the reported
ξ±, θ and variances are the same pair-count average, so ξ± = E ± B + amb
holds bin by bin. The adaptive pointwise path, the Monte-Carlo covariance
and the separate pure-E/B jackknife are removed. sacc_io gains
get_xi_npairs; cosmo-numba is pinned to the fork commit carrying the
operator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cv_pure_eb: modes and exact covariance from the integration part
The rule reads only the integration-grid ξ± part (with its pair counts)
and the CosmoCov ξ± covariance on that grid, and calls
calculate_pure_eb_correlation. The Monte-Carlo parameters and the
reporting-part input go; the covariance takes seconds, so the rule's
threads and runtime shrink. The summary labels the covariance from the
npz's npatch record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* papers/bmodes: pure E/B through the library operator, no Monte-Carlo chunks
pure_eb_modes.py replaces the chunk scatter/gather (precompute_pure_eb_chunk,
gather_pure_eb_chunks): one call to calculate_pure_eb_correlation on the
fine ξ± and its Gaussian covariance, written to {version}_pure_eb.npz.
The PTE and data-vector scripts drop the MC Hartlap factor, the n(z)
covariance comparison drops its MC-noise band, and the n_samples /
n_chunks config keys go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cv_pure_eb: describe the covariance input without naming its source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pure E/B: average fine bins with TreeCorr's pair weight, not npairs
TreeCorr's ξ± and meanr in a bin are averages over pairs weighted by
w_i w_j, so pooling fine bins with their `weight` reproduces the
reporting-bin measurement exactly when the fine edges nest the reporting
edges; npairs does so only for an unweighted catalogue. The operator,
calculate_pure_eb_correlation, the mixin, cv_pure_eb and pure_eb_modes.py
take `weight_int`; sacc_io.get_xi_npairs becomes get_xi_weight (parts
already carry the tag). A new test checks the nesting identity on a
weighted catalogue with exact binning; the fixture and operator pins are
regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pure E/B: reporting bins as unions of fine bins
Assigning fine bins to reporting bins by meanr sends a fine bin that
straddles a reporting edge wholly to one side, so P cannot reproduce
TreeCorr's pair average there. The library now takes the fine-grid edges,
snaps each requested reporting edge to the nearest fine edge, pools whole
fine bins, and returns the edges it used; the results' left/right edges,
the pure-E/B npz and the B-mode summary carry those snapped edges.
Edges that snap together or reach outside the fine grid raise.
Callers pass the fine edges: the mixin from gg_int's edges, cv_pure_eb and
pure_eb_modes.py from the integration grid spec. The reproduction test now
covers requested edges that do not nest, measuring each snapped bin
directly with exact binning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* papers/bmodes: resolve pure-E/B scale-cut windows on the saved edges
The pure-E/B PTE intermediate now carries the reporting edges its
matrices are indexed on, and config_space_pte_matrices resolves the
fiducial windows on them rather than on a nominal geomspace grid. A PTE
file that records no edges was binned on the nominal grid, which stays
its window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pin cosmo-numba a64cb2e; evaluate the pure-E/B transform at every fine node
cosmo-numba's get_pure_EB_operator (schneider2022, local_from_int=True)
now builds the fixed-quadrature transform. It is evaluated at every fine
node: the zero-padded ξ− window extends from the evaluation grid, so this
keeps the transform a function of the fine grid alone. P then selects its
rows; NaN rows at the grid edges are dropped first and only a NaN row that
P uses raises.
The new operator clamps a θ-dependent integration limit to the
interpolator's extrapolation bound, as the reference does, where the
previous pin extrapolated the stencil polynomial; the operator pins move
accordingly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pure E/B: resolve scale cuts by snapping to the nearest reporting edge
Exact containment on snapped reporting edges makes a cut placed on a
nominal edge select different bins on different fine grids: [12, 83]′ gave
bins 9–15 on a 0.5–300′ grid but 9–14 on cosmo_val's 0.08–300′ one.
bins_from_scale_cut snaps each end of a cut to the nearest reporting edge
in log θ, the rule the edges themselves were snapped by, so the same cut
selects the same bins on every grid. _get_pte_from_scale_cut (hence the
B-mode summaries), the pure-E/B plots and papers/bmodes
config_space_pte_matrices use it; COSEBIs keep bins_from_edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pure E/B: run the transform on the regular log grid (TreeCorr rnom)
cosmo-numba's interpolator places its samples on a regular grid in log θ,
so handing it TreeCorr meanr put every sample at its nominal position
while the evaluation limit sat at the true meanr; where a spacing exceeded
the mean step the limit fell past the extrapolation bound, and the old and
new cosmo-numba operators resolved that knife-edge differently. The
transform now runs on the geometric centres of the log-uniform fine edges
(TreeCorr rnom) and is evaluated there; meanr enters only the reported θ.
The edges are checked to be log-uniform, and pure_eb_operator no longer
takes theta_int. On rnom the d78a189 and a64cb2e operators agree to
round-off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* catalog: metacal bitmasks use FITS format K, not I or the dead FLAGS_ key
NGMIX_MCAL_FLAGS carries bit 30 (shapepipe#854's absent-measurement flag)
alongside the native fitter bits. Format I truncates it to int16 and
silently zeroes bit 30 on write, in both the FITS and the HDF5 branch of
write_shape_catalog (the HDF5 path reuses the FITS column's array).
The per-type NGMIX_FLAGS_{1P,1M,2P,2M,NOSHEAR} columns had the same
problem from the other direction: their format entry was keyed as
"FLAGS_{suffix}", which never matches the real column name
"{prefix}_FLAGS_{suffix}", so the writer's float64 default masked the
dead key rather than narrowing anything. Both parameter files now key
and format all six metacal bitmasks the same way, at K, so they round-
trip exactly and survive JointCat's optional memory-reduction pass
(which only downcasts int32/float64, not int64).
NGMIX_MCAL_TYPES_FAIL stays at I: it is a count in [0, 5], not a bitmask.
Adds a round-trip test parametrized over both parameter files, FITS and
HDF5, and reduce_mem on/off, checking 0, bit 30 alone, and bit 30 with a
native bit together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* catalog: metacal bitmasks as int32; reduce_mem never narrows integers
ShapePipe sets bit 2**30 in the metacal flags for a missing measurement,
which needs 32 bits. The parameter files now write NGMIX_MCAL_FLAGS and
the five NGMIX_FLAGS_* columns as FITS J (int32).
JointCat.dtype_out with reduce_mem narrowed every int32 column outside a
keep-list to int8, silently wrapping any value above 127. It now only
narrows float64 to float32 (RA/Dec excepted) and leaves integer columns
alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* catalog: layout-agnostic campaign and star catalogue readers
Replace read_hdf5_file's hardcoded patches/<name>/<tile-ID> lookup with
find_dataset_group(), which descends from the file root through single
container groups until it reaches the per-unit datasets. This reads the
legacy patches/<campaign>/ layout that ShapePipe still writes as a
compatibility shim, a future flat tiles/ layout, and the exposures/<exp>
layout of full_starcat_<campaign>.hdf5 with the same code.
read_star_catalogue() keeps the FITS path for files ending in .fits.
Requested columns missing from the data now raise a clear KeyError.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* galaxy: replace IMAFLAGS_ISO cut with config-driven mask columns
ShapePipe v2 drops IMAFLAGS_ISO for eleven boolean MASK_n* columns.
galaxy.mask_cut() ORs a configurable list of them (default MASK_n4,
MASK_n1, MASK_n2, MASK_n8, MASK_n1024 — stars, star halos, manual galaxy
mask, MaxiMask) and returns the keep mask; a catalogue missing any of the
requested columns raises a KeyError naming them.
classification_galaxy_base takes mask_columns; extract_info.py passes the
params.py mask_columns list and uses it for the star-sample cut too, and
now reads both catalogues through the new readers. Column lists in
params.py and masking.py's SPATIAL_CUTS updated to the MASK_n* names.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* retire patch logic (#340)
ShapePipe v2 processes a campaign (a tile list); P1-P7 no longer exist.
- catalog_builders.JointCat: get_patches()/get_n_obj() and the per-patch
FITS merge are replaced by a merge over a list of campaign hdf5 files
(-i final_cat_A.hdf5+final_cat_B.hdf5), read through
read_campaign_catalogue. The 'patch' int8 column becomes a 'campaign'
string column; the hdf5 root attr becomes 'campaigns'.
- survey.get_footprint() deleted: it was a lookup table of P1-P7 (plus W3)
RA/Dec boundaries, meaningless for a campaign. Its only caller,
catalog.check_matching, used it as an optional pre-filter via a 'name'
argument that every caller passed as None; the argument goes too, along
with the test_survey test that exercised P5.
- merge_psf_cat.py, combine_results.py, stats_tile_id_gal_counts.py,
compute_area.py: patch vocabulary and v1/v1.5/v1.6 P-name shortcuts
generalised to an explicit list of campaigns.
- params.py: 'name = "P7"' becomes 'campaign = None'.
Deleted (only ever meaningful for the P1-P7 era):
- scripts/prepare_patch_for_spval.sh: symlinks a v1 per-patch run tree
(~/psfex/${patch}/output/run_sp_Ms/.../full_starcat-0000000.fits,
tiles_${patch}.txt) into a working dir; neither the layout nor the file
names exist in v2.
- scripts/plot_rho_stats_patches.py: globs P* directories and reads
P*/output/run_sp_Pl/mccd_plots_runner/output/rho_stats_id.fits, one
curve per patch. No campaign analogue.
- scripts/survey_stats_all.sh: hardcoded 'for patch in P1 ... P7' over v1
run-directory bookkeeping.
- scripts/star_match_stats.py: sums stats_file.txt over the seven patches.
- scripts/check_tile_IDs_SP_LF.py: compares per-patch ShapePipe tile IDs
against CFIS3500_THELI_P<n>.list Lensfit files; both sides P-named.
The word 'patch' survives only in catalog.py's comment naming the legacy
hdf5 group, and in the treecorr jackknife sense (npatch/patch_number).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* tests: campaign readers, mask cut, and campaign merge
Seventeen unit tests on tiny synthetic hdf5 fixtures built in a temp dir:
both campaign layouts (legacy patches/<campaign>/<tile-ID> and flat
tiles/<tile-ID>) read identically, param-list restriction, missing-column
and ambiguous-layout errors; the star reader on exposures/<exp> hdf5 and
on FITS; galaxy.mask_cut defaults, explicit list, empty list, missing
column, and a v1 IMAFLAGS_ISO-only catalogue; JointCat.merge_catalogues
across two campaigns of different layouts, plus its error paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* catalog: safe, low-memory campaign reading and merging
- concatenate_datasets preallocates the output and fills it column by
column, so peak memory is the packed output plus one tile instead of the
full-width uncut catalogue; the verbose estimate uses the real itemsize.
- validate the requested columns against every tile dataset, not only the
first, and name the offending dataset in the error.
- read_campaign_catalogue checks the root n_tiles attribute and refuses a
truncated file; campaign_shape reports row count and dtype from metadata.
- JointCat.merge_catalogues preallocates the merged array from that first
pass (no more accumulate-then-concatenate, which doubled peak memory),
promotes each column's dtype across all campaigns so a wider string or
integer column in a later file is no longer silently truncated, and
refuses multi-dimensional columns explicitly.
- reduce_mem reduces int32 to int16 (int8 wrapped N_EPOCH/CCD_NB values)
and every assignment is range-checked, raising instead of wrapping.
- check_matching drops the identity index over d1 that the retired
footprint prefilter left behind, and extract_info applies mask_cut to
the matched subset instead of the whole catalogue.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* config: v2 mask configuration with the MASK_n* columns
Add config/calibration/mask_v2.0.yaml: the v1.X.11 cut set with
IMAFLAGS_ISO and the v1 post-processing masks replaced by the boolean
MASK_n<bit> columns (True = masked, hence kind: equal, value: False), the
coverage bits and MASK_n2048 listed but commented out. The v1.X configs are
left untouched: each describes a legacy catalogue that really has
IMAFLAGS_ISO, and rewriting them would break reproducing published versions.
plots.sky_plots no longer hardcodes the v1 label set; it combines whichever
of IMAFLAGS_ISO / MASK_n* / npoint3 / 1024_Maximask the config declared.
The comprehensive-to-minimal demo asks for the v2 mask labels.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* scripts: finish the campaign rename in configs and entry points
- params.py interpolated the removed 'name' into two paths (NameError on
import); use 'campaign', give it a real default, and point star_cat_path
at full_starcat_<campaign>.hdf5 (hdu_star_cat now documented as legacy
FITS only). params_im_sim.py renames 'name' to 'campaign' likewise.
- combine_results.get_area matches both the campaign and the legacy patch
wording, and raises on a missing file or unmatched pattern instead of
returning None / a 1 deg^2 placeholder that silently rescales densities.
- merge_psf_cat writes the campaign *name* as a string column (FITS 'A<n>'),
matching JointCat; the old 1-based ordinal depended on -p argument order.
- compute_m_bias_image_sims counts tiles via the n_tiles attribute or
find_dataset_group, not a hardcoded legacy group inside a bare except.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* tests, docs: pin row order, cover the new failure modes
Tests: assert the exact concatenation instead of sorted values, add a
fixture whose keys are inserted out of order, and cover the truncated
n_tiles file, a column missing from a later tile, dtype promotion across
campaigns in both argument orders, reduce_mem overflow, and a
multi-dimensional column.
Docs: CLAUDE.md, scripts/calibration/README.md, homogenize_cat_extended.py
and the catalog_builders docstrings now say campaign.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* catalog: promote dtypes across tiles, stream the campaign merge
Three defects in the campaign reading path, found in review:
- concatenate_datasets and campaign_shape both took the output dtype from
the first dataset alone, so a campaign whose tiles differ (S7 next to
S12 tile IDs, i2 next to i4, f4 next to f8 after a partial
reprocessing) had the later tiles silently truncated, downcast or
wrapped. np.concatenate, which this code replaced, promoted. Both now
build the dtype with group_dtype(), promoting every column across every
dataset; catalog_builders._promote becomes an alias of the shared
catalog.promote_dtypes rather than a second copy of it.
- merge_catalogues still held a whole campaign in memory next to the
preallocated output, the very thing its comment claimed the rewrite
avoided -- and with one campaign per merge in v2, that is the normal
case, ~2x the merged catalogue at DR6 scale. It now fills the output
tile by tile through the new catalog.iter_campaign_tiles(), so peak
memory is the output plus a single tile.
- write_hdf5_file wrote the merged array twice, create_dataset(data=...)
followed by an immediate dset[:] = dat_all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* config: restore the r-band coverage cut in the v2 mask config
mask_v2.0.yaml dropped v1.X's '64_r' r-band imaging cut without a
replacement, so a v2 calibration run admitted objects outside the r-band
footprint that v1 rejected -- a silent change of effective area, n(z) and
galaxy-density normalisation. Enable MASK_n64, v1's '64_r' equivalent.
v1's other coverage cut, npoint3 >= 3, came from an external
post-processing catalogue and has no v2 counterpart; say so in the config
rather than leaving its absence unexplained.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* scripts: cheaper star mask cut, campaign-derived leakage labels
extract_info fancy-indexed the full-width catalogue, dd[ind_star], only
to read ~5 boolean mask columns from it -- a copy of every column for
every matched star (~GB at DR6 scale). Mask first, index the resulting
bool array.
plot_leakage still labelled its curves "all", "P1" ... "P7": a P-named
survivor of the #340 retirement, and a fixed length that silently
mismatched the number of input files. Labels and colours now follow the
input files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* masks: treat n64 as a reason bit, not r-band coverage
The v2 branch inferred from v1's '64_r' naming that MASK_n64 flags r-band
imaging coverage. It does not: bit 64 is an undocumented *reason* bit of
the r-band default bitmask, and OR{n1,n2,n4,n8,n64,n1024} reproduces
mask_r, the v1 r-band mask, exactly on the P3 region.
Add MASK_n64 to DEFAULT_MASK_COLUMNS so the default galaxy cut is exactly
that set, documented as reproducing mask_r, and mirror it in the
calibration params, the v2 mask config and the minimal-catalogue demo.
Document n16/n32/n128/n256 as the u/g/i/z coverage flags (no r flag: the
catalogue is r-selected) and n2048 as absent Pan-STARRS z2. The faint vs
bright assignment of n1/n2 is unconfirmed for the Aug-2026 products, so
the labels no longer claim one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* ruff autofix (format + safe lint fixes)
Pushed by the lint gate.
* galaxy: cut never-fit objects on NGMIX_N_EPOCH explicitly
ShapePipe's make_cat pre-fills the NGMIX_* columns with sentinels
(G1/G2 = -10, T/FLUX = 0) and overwrites them only for objects present
in the ngmix output, so an object ngmix never fit keeps
NGMIX_MCAL_FLAGS == 0 and passes a flag-only cut. In
final_cat_smk-g7.hdf5 that is 18,983 of 1,851,100 objects (1.03%);
admitting them drags mean e1 to -0.096 (std 0.98) from +0.0001.
classification_galaxy_ngmix already rejected all 18,983 via the
NGMIX_G1_PSF_ORIG_NOSHEAR != -10 guard, so the production selection was
never affected -- verified on the real file, which gives 1,105,851 rows
out with and without the new cut. But that protection was incidental:
it is an exact float equality against a sentinel ShapePipe may change,
and the coadd N_EPOCH >= 2 cut in classification_galaxy_base does not
substitute for it (18,750 of the 18,983 have N_EPOCH >= 1). Cut on
NGMIX_N_EPOCH > 0 explicitly so the guarantee is stated, not inferred.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* readers: NaN-safe mask cut, star-catalogue provenance and count check
mask_cut decided on truthiness via astype(bool). ShapePipe writes the
MASK_n* columns as float64 {0, 1} rather than bool (being fixed
upstream), and astype(bool) reads NaN as True, so an incomplete mask
column would have silently deleted sky. Decide on the value instead
(masked iff > 0.5), accept bool, int and float alike, and treat NaN as
"no verdict recorded" -- keep the object, but count and warn, since a
nonzero count means the product is defective. final_cat_smk-g7.hdf5
carries no NaNs and only exact 0.0/1.0, so this is defensive: the real
file gives 1,105,851 rows out before and after.
read_star_catalogue silently accepted a truncated file and threw away
exposure provenance. It now validates the n_exposures root attribute
against the datasets found, as the galaxy reader validates n_tiles
(check_n_tiles generalised to check_n_units), and adds an EXPID column
carrying the exposure number each star came from. The datasets are
named by that number and concatenating them discarded it, leaving no
way to group stars by exposure downstream. Names may be bare
("2086324", as smk-g7 writes them) or carry the CFIS suffix
("2110000p"), so EXPID takes the leading digits. On the real star
catalogue: 53,264 stars over 127 exposures, matching n_exposures.
Also note in group_dtype that ShapePipe writes TILE_ID as f8, so the
string-promotion branch is for a future string-valued TILE_ID, with a
TODO recording that as an open schema decision. No behaviour change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* ruff autofix (format + safe lint fixes)
Pushed by the lint gate.
* config: cut never-fit objects in the v2 mask config too
mask_v2.0.yaml, applied downstream to the comprehensive catalogue, cut
on neither NGMIX_MCAL_FLAGS nor any epoch column. It rejected the
never-fit objects only through its NGMIX_G1/G2_PSF_ORIG_NOSHEAR != -10
cuts, and only because make_cat happens to fill the PSF columns from
the same -10 literal it uses for the galaxy ellipticities. That is the
same accidental immunity just removed from
classification_galaxy_ngmix, one stage further downstream.
Add NGMIX_N_EPOCH >= 1. The column is already carried into the
comprehensive catalogue via add_cols_pre_cal in params.py. On
final_cat_smk-g7.hdf5 the cut keeps 1,832,117 of 1,851,100 objects,
removing exactly the 18,983 (1.03%) never-fit rows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
* mask_v2.0: FLAGS <= 2, matching the image-sims selection
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* combine_results: drop unused n_campaign
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VXzqmMYw7Kp8QMtoiVHjVq
* grammar: ShapePipe v1->v2 column-grammar adapter, wired into rho/tau
sp_validation reads only the v2 column grammar, while every catalogue
on disk (v1.3.x-v1.6.x) is v1, so e.g. rho/tau raised KeyError on the
v1.4.a PSF file. New module sp_validation.grammar is the one place that
knows the difference:
- V1_RULES: the v1->v2 map of the shape-measurement columns as data (HSM
renames, SIGMA -> T = 2 sigma^2 via cs_util.size.sigma_to_T, 2-vector or
flattened NGMIX_ELL* split into G1/G2, PSFo/Tpsf/MOM_FAIL renames),
applied to tables detect_generation calls v1. Mapping is by name, so v1
values reach the names the code reads.
- MASK_RULES: the healsparse mask bits {b}_{label} (the names
ApplyHspMasks gave them in the comprehensive HDF5's data_ext) ->
MASK_n{b}, applied whatever the generation: they are the same bits of
the same UNIONS bitmask ShapePipe v2 writes as MASK_n{b}
(MASK_LABELS; 512 = outside the tile's unique region). IMAFLAGS_ISO is
not mapped: its v1 bits mean different things.
- adapt(table, *tables): tables with nothing to rename come back
unchanged; otherwise a V2View, a lazy column view over numpy, FITS_rec
or h5py that joins row-aligned tables (data + data_ext), computes derived
columns on access, and composes row selections as ranges or selected
indices, reading only the window of rows they span.
- read_catalogue(path, hdu), materialise, v2_names, read_column_names.
rho_tau.get_rho_tau / get_jackknife_cov / get_theory_cov read each
catalogue once through read_catalogue and hand the table to
shear_psf_leakage (needs its loaded-catalogue support).
Tests: v1 twins adapt to their v2 twins for numpy (vector and flattened
ELL), FITS_rec and h5py; row selection commutes; dtype matches every
column read; data_ext mask names rename with or without a generation and
conflict with their new names; h5py selections read only their window;
the psf_size_error field and full get_rho_tau outputs agree between a v1
PSF catalogue and its v2 twin (and a rename-only sigma is caught).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cat_config: psf/shear columns match their files; guard it on candide
test_cat_config_columns_exist_on_candide reads each cat_config
catalogue's FITS header through grammar.read_column_names and checks
the psf block's declared columns and the shear block's *_col columns
are among the names the file presents. Known gaps are listed with
reasons and fail the test once healed.
Config fixes it surfaced:
- psf dec_col Dec -> DEC: every PSF file (and ShapePipe v2) names it
DEC; only FITS_rec's case-insensitive lookup hid the mismatch.
- SP_v1.4.12.3 / SP_v1.4.13.3 psf blocks named v1 columns and the
retired square_size flag; now the v2 names like every other entry.
- SP_axel_v0.0 / SP_v1.4.5.A shear: declare their lowercase ra/dec;
SP_v1.4.5.A's PSF shape columns are psf_g1/psf_g2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* rho_tau: build rho/tau catalogues without a scalar-bool mask
build_catalog indexes each column with ``mask``; a scalar True only
adds an axis treecorr reshapes away (no stars are cut), and a scalar
False selects nothing, so treecorr raises "Input arrays have zero
length". That broke get_rho_tau for DES and get_jackknife_cov for
every catalogue. No flag cut was ever applied, so dropping the
argument leaves the non-DES results unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* calibration: read v1 and v2 products through the grammar adapter
Every reader on the calibration path now presents its tables in the v2
column grammar, so a ShapePipe v1 product runs through the same code and
the same configs as a v2 one:
- catalog.read_campaign_catalogue, campaign_shape, iter_campaign_tiles and
the JointCat merge adapt each tile (param_list names v2 columns);
read_star_catalogue adapts FITS and HDF5 star catalogues.
- CalibrateCat.read_cat returns one table: the comprehensive HDF5's data
and data_ext joined by grammar.adapt, so mask columns read the same
whether they sit in data (v2) or data_ext (post-processed v1), and
galaxy.mask_cut works on either.
- get_masks_from_config takes that one table, and each mask config is one
`dat` cut list: the v1 configs' dat_ext cuts move into it under their
MASK_n{b} names (the selections are unchanged; IMAFLAGS_ISO stays in the
v1 configs). The image-sim overlay and scripts/masking.py follow.
- ApplyHspMasks writes mask bit b as MASK_n{b}, from grammar.MASK_LABELS.
- cosmo_val/compute_theory_cov.py hands CovTauTh loaded, adapted tables.
Tests: a v1 comprehensive HDF5 (v1 data + data_ext with the old mask
names) and its v2 twin give identical mask_cut and per-config mask
selections for every config in config/calibration, and identical metacal
inputs and response; the campaign and star readers present v1 files in v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* uv.lock: shear_psf_leakage develop@00e38c4 (loaded-catalogue rho/tau builders, seeded patches)
Pinned below develop's tip 372980f, whose scipy>=1.18 requirement cannot
resolve against cs_util<0.3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* grammar: v1 no-shear reconvolved-PSF size from NGMIX_Tpsf_1P
ShapePipe v1 wrote a wrong NGMIX_Tpsf_NOSHEAR: it differs by ~2% for
nearly every object (v1.4, v1.5, v1.6 comprehensive) from the
reconvolution kernel metacal applied, which NGMIX_Tpsf_{1P,1M,2P,2M}
record (they agree to ~1e-5). The v1.4.6.3 release used the 1P value
in its metacal size cut and wrote it as the no-shear column; #267
dropped that substitution, which is a no-op only on the v2 stack, so
the branch's v1 calibration selected ~3% more objects than the release.
The adapter now presents NGMIX_T_PSF_RECONV_NOSHEAR from NGMIX_Tpsf_1P
when the table has it, falling back to NGMIX_Tpsf_NOSHEAR (a cut
catalogue such as v1.4.6.3's already holds the 1P value there), and
hides the raw column. Rules gain a fallback source; a derived column is
presented where the first of its sources sits. The module docstring
and the migration doc state the rule: name mapping, except this one
documented v1 defect.
Every consumer now sees the 1P value, including the w_des and
PSF-leakage size-ratio binning, where the release used the raw value;
those two columns therefore do not reproduce the release bit for bit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* grammar: select rows of an HDF5-backed view in one pass; cache its dtype
metacal reads ~40 columns of data[mask]. Over h5py datasets the lazy
view read each column over the whole span of the selection, so a
full-sky v1 calibration re-read the 238 GB comprehensive file once per
column. Selecting rows of a view over h5py Datasets now reads the
selected rows of every column in one pass per dataset (whole rows, in
256 MB blocks over the selection's span, skipping empty blocks), as
indexing the Dataset did on develop, and wraps them in a view over the
in-memory arrays so renames and derived columns still apply. Views
over in-memory tables stay lazy. to_structured reads each dataset once
for all requested fields.
On a cold 10M-row window of v1.4.c (4.8M rows selected):
h5py Dataset[mask] 19.2 s; adapt(data, data_ext)[mask] plus the 40
metacal columns 23.5 s; the previous per-column path took 16.5 s for
its first column alone, i.e. one full re-read per column.
dtype is computed once per view and passed to row selections
(group_dtype asked for it per column, quadratically). view[()] and
view[...] select every row, as for an h5py Dataset; other tuples raise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* catalog.group_dtype: name the tables that lack a column
A column missing from some tiles (several v1 tiles carry no
SPREAD_MODEL) surfaced as numpy's bare "no field of name" KeyError.
group_dtype now checks every table first and raises a KeyError naming
how many tables lack which columns, and which ones; callers pass the
tables keyed by dataset name so the message names tiles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* masks: refuse a mask config's dat_ext list; masking keeps IMAFLAGS_ISO
Mask configs are one dat: list over the joined data + data_ext table,
but get_masks_from_config and scripts/masking.py read only config["dat"],
so an older config with a dat_ext: list (several sit under v1.4.x and
v1.5.x) silently lost those cuts. masks.catalogue_cuts returns the dat
list and raises on dat_ext, saying to merge it into dat with {b}_{label}
renamed MASK_n{b}; the calibration, footprint and demo scripts all read
cuts through it.
scripts/masking.py's footprint cuts had dropped IMAFLAGS_ISO, which the
v1 configs still cut on and which is spatial (halo, border, Messier,
NGC, spike bits); it is back in SPATIAL_CUTS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* galaxy.mask_cut: a NaN mask value masks the object, as the config cuts do
mask_cut kept objects whose mask column is NaN (with a warning), while
the config-driven cut that calibration runs (kind: equal, value: False)
drops them, so the two selections disagreed on a defective product.
mask_cut now drops them too, still counting and warning. Keeping them
was chosen to stop astype(bool) from dropping NaN rows silently; the
warning covers that, and dropping an object with no mask verdict is
the conservative choice for a shear catalogue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* Comments describe the present; calibration configs' relative input_path
Drop wording that goes stale ("being fixed upstream", "unconfirmed for
the Aug-2026 products", "legacy" layouts and files) in favour of what
is true now, and ApplyHspMasks.write_hdf5_header's documented but
nonexistent campaigns parameter.
test_configured_paths_exist_on_candide resolved a calibration config's
relative params.input_path against config/calibration; such configs
run as config_mask.yaml in their run directory, where the relative
path names a file, so the guard skips those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* tests: calibration path reproduces a v1.4.6.3 release window (candide, slow)
Runs scripts/calibration/calibrate_comprehensive_cat.py with
mask_v1.X.6.yaml on rows 200M..201M of the v1.4.c comprehensive HDF5
and checks the released cut catalogue's rows from that window, matched
by (RA, Dec) and bracketed by rows from outside it: same objects, same
order, and identical per-object columns, including the no-shear
reconvolved-PSF size against the release's NGMIX_Tpsf_NOSHEAR. Globally
calibrated columns (e1, e2, w_des, leakage-corrected) are not compared.
Skipped where the release products are absent.
Passes in ~30 s on n09 (131,526 objects); before the no-shear
reconvolved-PSF correction it fails, selecting 134,936.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsAQRxNg6yDsUX4y2SWxTJ
* Drop the SExtractor-only columns DR6 catalogue mode lacks
ShapePipe v2 catalogues in DR6 catalogue mode no longer carry MAG_WIN,
MAGERR_WIN, SNR_WIN, FLUX_AUTO, FLUXERR_AUTO, FLUX_APER, FLUXERR_APER,
FWHM_IMAGE, FWHM_WORLD (shapepipe #924). Stop passing them through in
extract_info (params.add_cols) and calibrate_comprehensive_cat, and drop
the SExtractor SNR_WIN curve from the galaxy SNR histogram. The v1.4.6.3
release regression compares the remaining per-object columns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ
* Describe halo bit identities and guard independent mask cuts
* Name mask columns MASK_{bit}_{label}
The external-mask column for bit b is MASK_{b}_{label}, labels from
grammar.MASK_LABELS (grammar.mask_column), so the column carries both the
bit identity and its meaning. ApplyHspMasks writes these names; every mask
config, script and the galaxy defaults cut on them (galaxy derives its
tuples from mask_column).
The adapter presents the v1 data_ext spelling {b}_{label} and the
pre-release ShapePipe v2 spelling MASK_n{b} under the canonical name;
neither marks a generation. Tests cover both spellings mapping to the same
column and the same cuts, including the fixed-name faint/bright halo check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Retire the remaining patch-era scripts and docs
- Delete scripts/combine_results.py and scripts/compute_area.py (no caller;
their R.txt/c.txt outputs feed only a create_joint_shape_cat.py that no
longer exists) and the post_processing.md sections describing them.
- Delete cosmo_val/compute_theory_cov.py: its version keys, home-directory
paths and base_dir + absolute-subdir join no longer resolve, and
rho_tau's CovTauTh path, which the workflow drives, does the same job.
- using_the_catalogues.md: the v1.0 example selects on the file's `patch`
column; v1.4.1+ comprehensive HDF5 (flat data/data_ext) is read with
grammar.adapt, cut FITS catalogues with grammar.read_catalogue.
- index.rst: per-campaign merging. mask_r agreement stated as verified on
the P3 sky area. Drop the uncalled ApplyHspMasks.get_label_struct.
Campaign-reader fixture named CAMPAIGN.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* One catalogue reader; column names configurable through column_map
Container and column naming are now two axes, each known in one place.
sp_validation.io.Catalogue / read_catalogue detect the container from file
contents (h5py.is_hdf5, else FITS) and read FITS table HDUs, a single HDF5
dataset, a comprehensive data + data_ext HDF5, and ShapePipe's HDF5 groups
of per-tile/per-exposure chunks (n_tiles / n_exposures checked; key_column
keeps chunk names, e.g. EXPID). Every catalogue read routes through it:
extract_info (fixing the np.load-on-.fits path), the builders' read_cat
(BaseCat and CalibrateCat merged) and campaign merge, cosmo_val pseudo-Cl,
catalogue characterisation and the shear_psf_leakage-backed results, the
rho/tau loader (temp FITS only when the input is not already plain HDU 1),
glass_mock, calibration and scripts/masking.py.
grammar.adapt takes a column_map {canonical name: name in the file},
explicit or one-'*' patterns, applied lazily ahead of the v1 rules; it
overrides any rule for the same name and the v1 detection runs on the rest.
Catalogue configs carry it (cat_config blocks, calibration params,
params.py galaxy_/star_column_map). The *_col keys still pick roles,
naming columns as presented after the map.
config/columns/shapepipe_v2.yaml lists every column the code reads by
fixed name; test_column_schema keeps code and file in step both ways, and
a Sphinx extension renders it as docs/source/catalogue_columns.md.
Removed: catalog.read_campaign_catalogue, read_star_catalogue,
campaign_shape, iter_campaign_tiles, concatenate_datasets,
STAR_CAT_COLUMNS, read_shape_catalog; grammar.read_catalogue,
read_column_names, requires_adaptation; metacal's prefix argument.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01816Up3mpuhwYq7cnmyRHqv
* Read lazily where columns are not named; column_map fixes
io.open_catalogue / open_entry return a catalogue as one table without
reading it: FITS memory-mapped, unchunked HDF5 read per column on access.
The cosmo_val leakage subclasses, pseudo-Cl and catalogue
characterisation use them instead of reading every column into memory.
Through the leakage path the v1.4.6.3 shear catalogue (61.4M rows, v1
names, so adapted) peaked at 29.6 GiB RSS, 15.6 GiB of it anonymous; it
now peaks at 15.4 GiB with 0.2 GiB anonymous, as upstream's memmap read
does (the rest is file-backed page cache).
grammar: a column_map entry overrides only the rule for its own canonical
name; the columns it renames still feed the other rules (a map onto
NGMIX_Tpsf_1P no longer sends NGMIX_T_PSF_RECONV_NOSHEAR to the wrong
NGMIX_Tpsf_NOSHEAR). adapt(V2View, column_map=...) applies the map over
the view instead of ignoring it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01816Up3mpuhwYq7cnmyRHqv
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cail Daley <cdaley@l1.nibi.sharcnet>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Resolved toward the tomographic API with develop's I/O:
- Bin-pair keyed results, basename() with _tomo_bin_all, the renamed
functions and the tomographic iNKA covariance come from the
tomography branch; pol_factor is the ±1 multiplier.
- Every catalogue read goes through io.open_entry / _CatalogueLoader,
and masks are built from the same rows handed on.
- patch_number is mandatory; blind is gone; develop's treecorr_config
(min_top, threads) is kept.
- The ("all","all") pair writes develop's SACC parts (pseudo-Cℓ with the
out_path override, rho/tau); tomographic pairs keep their own outputs.
The ("all","all") ξ± dump is xi_{basename}.txt.
- Pure E/B runs develop's arrays/operator API inside the bin-pair loop.
- glass 2026.2 with glass.ext.camb and cosmology.compat.camb, which the
tomography branch's glass_mock imports.
Breakages outside the conflicted files (workflow scripts, tests,
summarize_bmodes), the pixel-window and jackknife-seeding ports, and the
lock follow as separate commits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
The workflow scripts call the ("all", "all") pair explicitly
(calculate_2pcf_version(...)["tomo_bin_all_tomo_bin_all"],
calculate_pseudo_cl(compute_tomography=False, out_path=...),
calculate_pseudo_cl_inka_cov(compute_tomography=False),
calculate_rho_tau_stats(tomography=False)), and the rules declare the names
the methods write: cv_basename and the rho/tau outputs carry _tomo_bin_all,
the ξ± dump is xi_{basename}.txt (also in papers/bmodes, which imports these
rules). The pseudo-Cl covariance job computes in a private directory, since
the iNKA blocks are cached under binning-only names.
plot_2pcf and plot_ratio_xi_sys_xi come back as wrappers over
plot_2pcf_tomography, so the cv_plot_2pcf / cv_ratio_xi_sys_xi rules and
run_cosmo_val.py run; xi_psf_sys computes the non-tomographic fit first, as
rho_tau_fits does. The ρ/τ plot rules pass the figure names the methods no
longer default to, and plot_rho_tau_fits saves its contours into the output
directory.
pol_factor: true in the paper config passed the ±1 assert and meant no e2
flip; it is -1. GLASS_mock_validation's PSF columns take the v2 names its
file presents (as SP_v1.6.6, same file) and its bin column the file's
TOM_BIN_ID. compute_tomography and force_run are exempt from the
cv_init_params forwarding: rule scripts choose the pair per call, and
Snakemake decides what reruns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
summarize_bmodes looked up the pre-tomography shapes inside try/except KeyError, so pure E/B, COSEBIs and C_ℓ^BB all dropped out of the summary silently. It now reads the pair entry of each result (plot_cosebis stores its result under the pair key, as plot_pure_eb does) and catches only the RuntimeError of a scale cut that selects no bin. get_redshift, hence sacc_nz and every SACC writer, unpacked a multi-column tomographic n(z) file into (z, nz1, ..., nzN); it now returns the summed whole-survey n(z) through read_redshift_distribution, the n(z) the pseudo-Cl iNKA fiducial uses for the same pair. pol_factor=True passed the ±1 assert and meant no e2 flip; bools are rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 2PCF tests call calculate_2pcf_version(...)["tomo_bin_all_tomo_bin_all"]
and look for the xi_{basename}.txt dump; the pure-E/B test reads its pair
and re-measures the integration-grid ξ± on the same patches for the
jackknife check, since cat_ggs is only filled by calculate_2pcf.
get_rho_tau takes base_tau. The OneCovariance fixture carries the ℓ and
tomographic-bin columns the tomographic reshape reads (one triangle, as the
reshape mirrors the other). The column-schema scan learns the non-catalogue
keys the tomographic code reads (W{}, H0) and the GLASS mock's TOM_BIN_ID.
New: plot_2pcf and plot_ratio_xi_sys_xi write their figures;
summarize_bmodes reads every statistic's pair and raises on a shape it
cannot read; sacc_nz is the summed n(z) of a multi-column file; pol_factor
rejects bools; cv_basename spells CosmologyValidation.basename.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
glass_mock.compute_two_point_cl_map unpacked four values from get_n_gal_map, which returns the count map only; it takes the pixel bookkeeping from get_pixels and passes it on. Both mock pseudo-Cℓ helpers label the coupled spectrum ℓ = 0..lmax-1, the range compute_coupled_cell returns (checked at nside 32: 64 entries, labels 0..63). The test-reference generator and the namaster covariance paper script call get_params_rho_tau without survey=, the generator reads its catalogue through io.open_entry and carries the mandatory patch_number, and the paper script takes the pixel bookkeeping from get_pixels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gration branch psf_systematics: the galaxy masks keep reading through open_entry, the reader the ρ/τ loader and the object-wise leakage share. uv.lock is re-resolved against the merged pyproject, with shear-psf-leakage at develop f1a2c071 (object-wise row selection, #48). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
Catalogue spectra are not pixelized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OneCovariance fixture is the full row-major n×n table of one bin, as the file is written, now with the ℓ (cols 1-2) and bin (cols 5-8) columns the tomographic reshape reads; the perturbation teeth keep the table symmetric. The tomographic reshape stays: PseudoClMixin's OneCovariance covariance needs it, and on a single-bin table it returns the matrix the row-major reshape did. test_matter_maps_are_seed_deterministic passes (XPASS) with the merged pins (glass 2026.2, glass.ext.camb 2023.6, cosmology.compat.camb 0.2.0) overlaid on the current image, so its xfail goes. It fails on an image built from the old lock (no cosmology.compat) until uv.lock is re-locked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e writer test_matter_maps_are_seed_deterministic passes with the locked glass 2026.2 / cosmology-compat-camb 0.2.0 on the path, but the image is still built from the old lock, so the xfail stays (raises=ModuleNotFoundError) until it XPASSes in a rebuilt image. The OneCovariance fixture's full row-major table is what cov_output.__write_cov_list writes: every (ℓ1, ℓ2) pair for every ordered pair of spectra. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shear_psf_leakage saves each jackknife draw as cov_{rho,tau}_{catalog_id}.npy.
#295 renamed the files get_jackknife_cov reads to cov_{rho,tau}_{base}{i}.npy
but kept catalog_id = version + i, so a fresh cov_estimate_method="jk" run
computed every draw and then raised FileNotFoundError. The draws now take
catalog_id = base_tau + i, which keeps the bins of a tomographic run apart,
and the reader looks for exactly those files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pseudo-C_ell covariance's fiducial spectrum is built from self.cosmo, a CCL Cosmology. The tomographic pseudo-C_ell work (#221, reapplied as #240; a941f54, 753abc1) evaluated it with cs_util's CAMB backend, which rebuilds a CAMB cosmology from H0, ombh2, omch2, ns, As/sigma8 and w/wa only: the neutrino mass is dropped (CAMB's 0.06 eV default stands in) and CAMB's mead2020 non-linear model replaces the object's halofit. Against CCL that is -4% at ell=1000 for Planck 2018 and -8.5% with mnu=0.3. develop passed backend="ccl"; the default is CCL again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PseudoClMixin.get_pseudo_cls_catalog forwarded tomo_bin_a=tomo_bin_b=None over the primitive's "all" defaults, which the primitive reads as a tomographic selection: a call without bins selected an empty catalogue (NaMaster then fails inside ducc0) or tripped the tomography assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shear_psf_leakage f1a2c071 (#48) has temporarily_read_data(selection=...) call self.read_data(selection=selection). _LeakageObject overrode read_data without that argument, so every tomographic bin of the object-wise leakage raised TypeError. The override now keeps the rows a boolean selection marks, over the rows io.open_entry returns, which are the rows _get_galaxy_mask builds its masks on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The inverse of a covariance estimated from N realisations is biased high; the PSF-leakage fits correct it with the realisation count. develop passed 300 for cov_estimate_method="sim"; #295 (ee75f53) kept only the jackknife patch count, so simulation covariances went undebiased and the fitted leakage parameters came out too tight (about 12% for 60 tau points). Nothing records the simulation count: no code in this repository's history writes cov_tau_*_sim.npy, and the file holds only the matrix. The count is therefore the constructor parameter n_sim_cov, default 300 as on develop, forwarded by the workflow from config["cosmo_val"]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TreeCorr's unseeded k-means and thread-dependent tree depth changed patch layouts, ξ±, and jackknife covariances between runs. Compute seeded, fixed-depth centres once from each full version catalogue and share them across bin pairs; use the same helper for rho/tau covariance patches. Add reproducibility and cross-bin regressions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…AG test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
The tomography branch inserts _tomo_bin_all into CosmologyValidation.basename, but inference.smk and covariance.smk still hard-coded the pre-tomography rho_stats/tau_stats/cov_tau names, so they could never match what rho_tau_stats writes. cv_rho_stats, cv_tau_stats and a new cv_cov_tau now live in common.py (cosmo_val.smk is only included with a cosmo_val config, so its helpers were invisible to the compute rules) and take the binning explicitly: CV_FIDUCIAL in cosmo_val, the wildcards in inference_prep, FIDUCIAL for the glass-mock rules. cv_objectwise_leakage reads every version's rho/tau FITS through _load_alpha_leakage but declared no inputs; it now declares them so it is scheduled after rho_tau_stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es it Reverts d603c5c. The CAMB/CCL difference is mostly the non-linear model (mead2020 vs halofit), not an error, and the fiducial only sets the signal term of a Gaussian covariance. The one real defect, cs_util's CCL-to-CAMB conversion dropping m_nu, belongs in cs_util. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
…rectory The iNKA covariance was computed and moved into place, then the job failed removing its temporary directory: files still open leave .nfs placeholders. Also report the BB block's real shape (it printed 5x5 for a 32-bin block). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
…type is given A cov_type of None was formatted into the covariance file name (cov_tau_…_None.npy), so the workflow's tau plot failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
The rho/tau rule writes the covariance of cov_estimate_method next to the tau statistics; the plot now draws its error bars from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138Efdjkma53ysbYhhXDgxW
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This merges
developinto the tomography branch, so that the later merge of the tomography branch intodevelop(step 4 of #375) becomes a review rather than a rebuild.The result keeps the tomography branch's API and takes develop's I/O.
Every per-version result stays keyed by bin pair, with non-tomographic results as the
("all","all")pair.Catalogues are read through develop's readers, and the
("all","all")pair is written through develop's SACC writers.Closes #374. Part of #375.
Structure
The first commit, 0d5f2ad, is a true merge of
develop(8db2aa5) and only resolves conflicts.Each later change is its own commit:
pol_factor: -1, the B-mode summary and SACC n(z) on the bin-pair shapes, the tests, and this branch's own callers of reshaped helpersuv.lockre-resolved: shear-psf-leakage develop f1a2c071 (includes CosmoStat/shear_psf_leakage#48), glass 2026.2cell_method == "map", in both the SACC bandpower window and the iNKA fiducial("all","all"); the object-wise leakage reader accepts #48's row selection; Hartlap debiasing of simulation ρ/τ covariances restoredChoices worth a look
NmtFieldCatalog), which have no pixels. This branch already dropped pw² from the iNKA fiducial, and develop applied it. pw² now applies only on the map path.m_nu(_ccl_to_camb drops the neutrino mass cs_util#96)._tomo_bin_all, asbasename()builds them. Existing products under the old names are recomputed on the next run.Verification
pytest -m "not slow") and the workflow DAG tests pass on the branch head. CI builds the image and runs the unit tests in it.TOM_BIN_IDrows, and the τ theory covariance is computed per bin from the masked catalogue.papers/cosmo_valsuite on SP_v1.4.6.3, run from develop and from this branch. Results and figures for each deliberate change will go here.Develop's own suite fails in three places, independent of this merge: the pseudo-Cℓ covariance rule (the dict bug of closed #372), COSEBIs and pure-E/B refusing the unblinded ξ± part the same workflow writes, and the τ plot (CosmoStat/shear_psf_leakage#46).
— Claude Opus 5.5 on behalf of Cail
🤖 Generated with Claude Code