Skip to content

CLAUDE.md

Latest commit

 

History

History
159 lines (134 loc) · 8.7 KB

CLAUDE.md

File metadata and controls

159 lines (134 loc) · 8.7 KB
Raw

ShapePipe

ShapePipe is a galaxy shape-measurement pipeline for weak-lensing cosmology. It runs the full chain from raw survey images to calibrated shear catalogues — object detection, PSF modelling, and shape measurement — and was used to produce the first UNIONS cosmic-shear release.

The project is now entering a substantial rework of the shape-measurement pipeline, with the near-term goal of a tight loop between the pipeline and image simulations for validation and calibration. Development is by a small team; reproducibility and clarity matter more than breadth.

Environment — the container is the source of truth

ShapePipe is not a stand-alone library: it needs system tools (Source Extractor, PSFEx, WeightWatcher), MPI, and a specific scientific-Python stack. The supported way to get all of that is the container.

  • Dependencies are declared in pyproject.toml as floor ranges; exact versions live in uv.lock, the reproducible manifest generated by uv (never hand-edited). Renovate keeps the lock current (see renovate.json): uv's resolver owns version selection, so cross-package constraints resolve jointly instead of failing update PRs. Bump a floor only when the code actually requires the newer API. System-level tools live in the Dockerfile.
  • Images publish to ghcr.io/cosmostat/shapepipe on every push to an integration branch, one image per branch (:<tag>): full stack, interactive tooling, and the test / lint / doc extras all baked in. Batch jobs, downstream FROM clauses, and CI's own test run all use this same image.
  • On a cluster, run with Apptainer: 
    apptainer build --sandbox shapepipe docker://ghcr.io/cosmostat/shapepipe:develop
apptainer shell --writable shapepipe
cd /app && shapepipe_run -c /app/example/config.ini
  • For development, work inside the image — it carries vim, ripgrep, pytest, and the rest. A common pattern is a long-lived --writable Apptainer sandbox with a host clone of the repo bind-mounted in and pip install -e pointed at it, so edits on the host are live inside the container.

Shadow a library build with PYTHONPATH (no image rebuild). When you need the container to run a different build of a pure-Python library than the one baked into its venv — a feature worktree, an unreleased branch, this repo's own src/ against a stale image — prepend the host checkout to PYTHONPATH. Python resolves the prepended path first, so the on-disk version shadows /app/.venv/... without touching the image. This is the local-testing counterpart of a git-ref dependency (e.g. cs_util @ develop in pyproject.toml): the dep change makes CI build the right version; the shadow lets you test that version now, before any rebuild. The recipe:

apptainer exec --bind /n17data,/automnt <image.sif> bash -c \
  "cd <repo-worktree> && \
   PYTHONPATH=/path/to/libfoo-checkout:<repo-worktree>/src \
   python -m pytest <targets> -o addopts='' -q"

Notes: the checkout path is the parent of the importable package dir (the dir containing foo/, not foo/ itself); list several :-separated to stack shadows; -o addopts='' clears pyproject.toml's pytest defaults when a plugin they reference (e.g. pytest-cov) isn't in the image. Use this for a quick verify; land the real fix as the pyproject.toml / uv.lock dep change so CI and the next image agree.

The test suite runs inside the deploy-image.yml build — there is no separate test workflow.

Testing container changes: build remotely, pull locally. Don't apptainer build images on a cluster — quotas are tight and the build is slow. The loop for any change to Dockerfile / pyproject.toml / uv.lock is: edit → push → let GitHub Actions build and publish to GHCR → apptainer pull docker://ghcr.io/cosmostat/shapepipe:<branch> on the cluster → test. Watch the remote build with gh run watch (or gh run list --branch <branch>). The only things that run locally are the pull and the test. On a quota-limited cluster, keep SIFs and Apptainer's scratch off $HOME: point APPTAINER_TMPDIR / APPTAINER_CACHEDIR at a roomy data partition and pull SIFs there.

Full detail: docs/source/installation.md and docs/source/container.md.

Layout

  • src/shapepipe/ — the package (src-layout). modules/ holds the pipeline modules and their *_runner.py wrappers; pipeline/ is execution and file I/O; utilities/; canfar/ is CANFAR/cluster job orchestration. Console entry points (shapepipe_run, summary_run, canfar_*) are defined under [project.scripts].
  • tests/ — the whole test suite, one discovery root: module/ (per-module unit/property/integration tests), unit/ (structural), science/ (fast guardrails), cluster/ (candide-only), helpers/ (shared library code). See tests/README.md.
  • example/ — a runnable example pipeline (example/config.ini) on a single CFIS tile; doubles as the CI smoke test.
  • scripts/ — shell / Python / notebook helpers (sh/, python/, jupyter/), symlinked onto $PATH inside the image.
  • docs/ — Sphinx sources; the API docs are generated from docstrings.

Development workflow

We are in rapid iteration. Changes that require re-running the pipeline are fine; a PR that changes science defaults (detection parameters, cuts, module options) does not need a completed rerun to be accepted — state what we expect the rerun to show, and do the rerun when it comes up.

  • develop is the integration branch — open PRs against it. main / master are release branches.
  • Tests run with pytest. CI runs them inside the image, so the suite exercises exactly what ships — run them the same way, in the container.
  • CI (.github/workflows/deploy-image.yml): every PR and push builds the image and runs the test suite + the example pipeline + binary smokes; pushes to develop / main / master additionally publish the images. API docs deploy from master (cd.yml).
  • Style: PEP 8; numpydoc docstrings on public modules, classes, and methods. Match the surrounding code.

Project knowledge

Durable project knowledge — decisions, architecture, plans — lives in this repo's pull requests, issues, docstrings, and docs/, alongside notes some maintainers keep in their own stores outside it. A .felt/ directory (a markdown "fiber" note store used with the felt CLI) is not tracked here: it's gitignored, and where it exists it's a machine-local symlink into a private, separately git-synced store, so a fresh clone won't have one. Record durable decisions in the PR, issue, or docs where the change lives — and scientific decisions in astra.yaml, below.

Scientific decisions live in astra.yaml

astra.yaml at the repo root is the pipeline's decision record: every consequential scientific choice embedded in the code and committed configs, with its rationale, alternatives and selected default. universes/committed.yaml pins the option the committed configuration selects for every decision. @sc [decision:<id>] tags live at implementing code/config sites; optional Values: sentences assert committed values within those tagged sites. The rationale stays in ASTRA, while local @sc contracts hold site-specific constraints. tests/unit/test_decisions.py checks tag syntax, bidirectional decision/site coverage, values, and test decision markers. To inspect tags at an implementation site, run python -m tests.helpers.decisions <path>[:<line>]; --decision <id> lists its sites, and --diff origin/develop the decisions your branch touches and whether their record changed (CI posts this list on every PR). The format is ASTRA.

uvx astra-tools@0.2.17 guide is the briefing and uvx astra-tools@0.2.17 spec the field reference. The file's header states its conventions, including [HARDCODED] and [LINT].

Membership test: a different defensible choice would change which objects enter the shear catalogue, or the numbers attached to them. Detection thresholds, masking, star-selection cuts, PSF model degree, ngmix priors and seeding, flag semantics, completeness gates: in. Workflow policy (manifests, chunking, allocation, failure reporting, provenance) is out; it lives in the PR and in the PRD, #848.

A scientific change is not finished until the record is. When a change moves what the pipeline measures, amend astra.yaml in the same PR (add the decision, or edit its rationale, options, Values and site tags), pin the selected option in universes/committed.yaml, and say so in the PR description. tests/unit/test_decisions.py runs in CI; a scientific change that breaks its site/value checks or leaves the record stale is unfinished. Before committing:

uvx astra-tools@0.2.17 validate