Data/Algorithm Documentation

.devcontainer/Dockerfile

@@ -5,6 +5,9 @@ FROM mcr.microsoft.com/devcontainers/python:3.11
 RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && apt-get install -y libgtk-3-dev
 
 # Installs poetry and ipython
-RUN pip install "poetry>=1.0.0,<2.0" ipython
+RUN pip install "poetry>=2.0,<3.0" ipython
+
+# Create the poetry virtualenv at /workspaces/imap_processing/.venv so VS Code can find it
+ENV POETRY_VIRTUALENVS_IN_PROJECT=true
 
 WORKDIR /workspaces/imap_processing

.devcontainer/devcontainer.json

@@ -6,25 +6,23 @@
 	"features": {
 		"ghcr.io/devcontainers/features/desktop-lite:1": {
 			"version": "latest"
-		}, "ghcr.io/devcontainers/features/docker-in-docker:2": {
-			"version": "latest"
+		}, "ghcr.io/devcontainers/features/docker-in-docker:4.0.0": {
+			"version": "latest",
+			"moby": false
 		}
 	},
 	"forwardPorts": [6080],
-	"postCreateCommand": "poetry install",
-	"postStartCommand": ". $(poetry env info --path)/bin/activate",
+	"postCreateCommand": "poetry install --all-extras",
 	"customizations": {
 		"vscode": {
 			"extensions": [
-				"ms-python.python"
+				"ms-python.python",
+				"Anthropic.claude-code"
 			],
 			// Set *default* container specific settings.json values on container create.
 			"settings": {
-				// not using venvs? uncomment this
-				"python.defaultInterpreterPath": "/usr/local/bin/python"
-
-				// use the active venv
-				// "python.defaultInterpreterPath": ".venv/bin/python3"
+				// poetry installs into the in-project .venv (see Dockerfile)
+				"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"
 			}
 		}
 	}

AGENTS.md

@@ -0,0 +1,144 @@
+# AGENTS.md
+
+`imap-processing` is the science data processing pipeline for NASA's IMAP mission,
+run by the Science Operations Center (SOC) at LASP. It turns raw CCSDS telemetry
+packets into ISTP-compliant CDF science products, one instrument and one data level
+at a time.
+
+## Big picture
+
+- A high level overview of the mission lives in [docs/source/mission-overview.rst](docs/source/mission-overview.rst). This document is not needed for
+  software development, but provides a helpful reference for connecting together instrument-spanning questions or ideas.
+- Every product is produced by a single invocation of `imap_cli` for one
+  `(instrument, data-level, descriptor, start-date)` combination. See [imap_processing/cli.py](imap_processing/cli.py).
+- [imap_processing/cli.py](imap_processing/cli.py) is the only entry point. It defines an abstract
+  `ProcessInstrument` base class and one subclass per instrument (`Codice`, `Glows`,
+  `Hi`, `Hit`, `Idex`, `Lo`, `Mag`, `Spacecraft`, `Swapi`, `Swe`, `Ultra`). Each
+  subclass implements `do_processing(dependencies) -> list[xr.Dataset]`; the base
+  class handles downloading dependencies and writing the returned datasets to CDF.
+  **Adding a new data level means updating both `PROCESSING_LEVELS` in
+  [imap_processing/__init__.py](imap_processing/__init__.py) and the relevant `do_processing` branch.**
+- Instrument subpackages live at `imap_processing/<instrument>/`. Two layouts are in
+  use — flat modules (`hi/hi_l1a.py`, `codice/codice_l1a.py`) and level subpackages
+  (`mag/l1a/mag_l1a.py`, `swe/l1b/swe_l1b.py`). Follow whichever the instrument
+  already uses; do not restructure.
+- Data flows as `xarray.Dataset` objects throughout. Processing functions take
+  dependencies (an `imap_data_access.ProcessingInputCollection` or file paths) and
+  return `xr.Dataset` / `list[xr.Dataset]`. They should not write files themselves —
+  the CLI does that.
+
+## Algorithms
+- Algorithm behavior is documented per-instrument in
+  [docs/source/algorithm-code-documentation/](docs/source/algorithm-code-documentation/). Check there before changing science logic.
+
+  Some instruments have a full working reference distilled from their algorithm document —
+  a product inventory, the algorithms with equations, and an implementation-status
+  page listing deviations and gaps. **Read the instrument's page set before proposing or
+  estimating work on it.**
+  - CoDICE: [docs/source/algorithm-code-documentation/codice/index.rst](docs/source/algorithm-code-documentation/codice/index.rst)
+  - GLOWS: [docs/source/algorithm-code-documentation/glows/index.rst](docs/source/algorithm-code-documentation/glows/index.rst)
+  - HIT: [docs/source/algorithm-code-documentation/hit/index.rst](docs/source/algorithm-code-documentation/hit/index.rst)
+  - IDEX: [docs/source/algorithm-code-documentation/idex/index.rst](docs/source/algorithm-code-documentation/idex/index.rst)
+  - MAG: [docs/source/algorithm-code-documentation/mag/index.rst](docs/source/algorithm-code-documentation/mag/index.rst)
+  - SWAPI: [docs/source/algorithm-code-documentation/swapi/index.rst](docs/source/algorithm-code-documentation/swapi/index.rst)
+  - SWE: [docs/source/algorithm-code-documentation/swe/index.rst](docs/source/algorithm-code-documentation/swe/index.rst)
+
+  **Many of these pages are unreviewed AI-generated drafts.** They contain the line
+  `.. include:: /algorithm-code-documentation/_ai_generated_notice.inc` just below the
+  page title (this renders as a warning banner). Treat a page with that line as a
+  strong starting point, not as an authority:
+  - When a marked page disagrees with the code, do not assume the code is wrong. The
+    algorithm document or CMAD may be out of date, or the deviation may be intentional.
+    Point out the discrepancy and let a human decide; don't "fix" code to match the page.
+  - When you cite a marked page in a PR, review, or answer, say that it is unverified.
+  - Only a person who has verified the page against the code or with the instrument
+    team removes the include line. Never remove it yourself unless that person asks you
+    to. When you edit a marked page, keep the marker.
+  - As the pages are reviewed by humans to correct discrepancies, they will be added into
+    sphinx docs. Currently, the sphinx documentation only points to autogenerated docs from the code.
+  - The ENAs (Lo, Hi, and Ultra) will be included in more detail at a later date to this documentation, when their algorithm documents are finalized.
+
+
+## Use the shared infrastructure, don't reinvent it
+
+| Need | Use |
+|---|---|
+| Decommutate CCSDS packets | `packet_file_to_datasets()` in [imap_processing/utils.py](imap_processing/utils.py) |
+| Raw DN → engineering units | `convert_raw_to_eu()` in [imap_processing/utils.py](imap_processing/utils.py) |
+| CDF read/write | `load_cdf()` / `write_cdf()` in [imap_processing/cdf/utils.py](imap_processing/cdf/utils.py) |
+| CDF global/variable attributes | `ImapCdfAttributes` in [imap_processing/cdf/imap_cdf_manager.py](imap_processing/cdf/imap_cdf_manager.py) |
+| Time conversion (MET / ET / TT-J2000 ns) | [imap_processing/spice/time.py](imap_processing/spice/time.py) — never hand-roll epoch math |
+| Spin, repointing, pointing frames, geometry | [imap_processing/spice/](imap_processing/spice/) |
+| Data quality bitflags | [imap_processing/quality_flags.py](imap_processing/quality_flags.py) |
+| ENA sky maps / pointing sets | [imap_processing/ena_maps/ena_maps.py](imap_processing/ena_maps/ena_maps.py) |
+| Combining time-varying ancillary files | [imap_processing/ancillary/ancillary_dataset_combiner.py](imap_processing/ancillary/ancillary_dataset_combiner.py) |
+
+## Packet definitions (XTCE)
+
+Packet structures are XTCE XML files in `imap_processing/<instrument>/packet_definitions/`.
+They are generated as a first pass from the instrument team's telemetry spreadsheet via
+`imap_xtce <spreadsheet.xlsx> --output <file.xml>` ([imap_processing/ccsds/excel_to_xtce.py](imap_processing/ccsds/excel_to_xtce.py))
+and then hand-refined. Decommutation is always done through `packet_file_to_datasets()`,
+which returns `dict[apid, xr.Dataset]`; instrument code dispatches on APID enums
+defined in the instrument's `constants.py`.
+
+## CDF products (required for every new data product)
+
+1. Add/extend the YAML attribute configs in [imap_processing/cdf/config/](imap_processing/cdf/config/):
+   `imap_<instrument>_global_cdf_attrs.yaml` and
+   `imap_<instrument>_<level>_variable_attrs.yaml`.
+2. Load them with `ImapCdfAttributes.add_instrument_global_attrs()` and
+   `.add_instrument_variable_attrs()` and apply them to `dataset.attrs` and each
+   variable's `.attrs`.
+3. Set `Logical_source = "imap_<instrument>_<level>_<descriptor>"` and `Data_version`.
+   `write_cdf()` derives the output filename from these, so a typo here produces a
+   wrongly-named file rather than an error.
+4. Missing ISTP attributes surface as `cdflib` `ISTPError` at write time — test by
+   actually calling `write_cdf()` in a test, as [imap_processing/tests/hi/test_hi_l1a.py](imap_processing/tests/hi/test_hi_l1a.py) does.
+
+## Environment and commands
+
+Poetry 2.x with the dynamic-versioning plugin. Full setup instructions, including the
+required system libraries (`libnetcdf`, `libhdf5`, and openblas/gfortran on 3.14+), are
+in [docs/source/development/getting-started.rst](docs/source/development/getting-started.rst).
+
+```bash
+poetry install --extras "test,dev,doc"        # or --all-extras
+pre-commit install                            # do this once
+
+poetry run pytest -vvv -n auto -m "not external_kernel and not external_test_data"
+poetry run pytest imap_processing/tests/hi/test_hi_l1a.py::test_sci_de_decom -vvv
+
+pre-commit run --all-files                    # ruff check+format, mypy, numpydoc, codespell
+make -C docs html SPHINXOPTS="-W --keep-going"
+```
+
+- Default to the `-m "not external_kernel and not external_test_data"` selection.
+  Those markers trigger multi-hundred-MB downloads of SPICE kernels from NAIF and
+  test data from the SDC; assume the sandbox has no network access unless told otherwise.
+- Tests live in [imap_processing/tests/](imap_processing/tests/), mirroring the instrument packages.
+  Global fixtures (SPICE kernels, metakernels, fake spin/repoint data, temp `DATA_DIR`)
+  are in [imap_processing/tests/conftest.py](imap_processing/tests/conftest.py); instrument-specific fixtures in each
+  subdirectory's `conftest.py`. Prefer existing fixtures over new ad-hoc setup.
+- Codecov requires ~90% patch coverage, so new code needs tests.
+
+## Conventions
+
+- Ruff (`ruff check` / `ruff format`) and mypy (`strict = true`, tests excluded) are
+  enforced in pre-commit and CI. Configuration lives in [pyproject.toml](pyproject.toml).
+- **numpydoc docstrings are validated** (`numpydoc-validation` hook) on all non-test
+  code: summary, `Parameters`, `Returns` with types are effectively mandatory.
+  Tests are exempt from docstring rules (`D` is ignored under `*/tests/*`).
+- Modern type hints everywhere: `str | Path`, `list[xr.Dataset]`, `npt.NDArray`.
+  Target version is `py310`.
+- `logger = logging.getLogger(__name__)` — never `print()`.
+- Style details (naming, imports, PR checklist, review standards) are documented in
+  [docs/source/development/git-workflow-and-style-guide/index.rst](docs/source/development/git-workflow-and-style-guide/index.rst) — follow it rather than
+  inventing conventions.
+
+## Gotchas
+
+- Pre-commit blocks direct commits to `main` and `dev`, and files over 1000 KB. Work on
+  a feature branch; test data is downloaded at runtime, never committed (no git-lfs).
+- CI runs the suite on Linux/macOS/Windows × Python 3.10–3.14, so avoid
+  platform-specific paths and version-specific syntax.

docs/source/algorithm-code-documentation/_ai_generated_notice.inc

@@ -0,0 +1,20 @@
+..
+   Shared "unreviewed AI-generated draft" banner, pulled into each algorithm page
+   with ``.. include:: /algorithm-code-documentation/_ai_generated_notice.inc``.
+   Once a person familiar with the instrument has verified a page, delete that
+   include line from the page. To list the pages still awaiting review, run:
+   grep -rl "_ai_generated_notice.inc" docs/source/algorithm-code-documentation
+
+.. admonition:: AI-generated draft — not yet reviewed
+   :class: warning
+
+   This page was generated by an AI assistant from the instrument's algorithm
+   document, the IMAP Calibration and Measurement Algorithms Document (CMAD), and
+   the source code in this repository. No one familiar with the instrument has
+   checked it yet, so it may contain errors or leave out context that only the
+   instrument team knows.
+
+   If this page and the code disagree, the code is not necessarily wrong. The
+   source document may be out of date, or the difference may be intentional.
+   Check against the code, and with the instrument team if you need to, before
+   relying on anything here.

docs/source/algorithm-code-documentation/codice.rst

@@ -28,4 +28,4 @@ L2 processing:
     :recursive:
 
     utils
-    decompress
+    decompress