diff --git a/.zenodo.json b/.zenodo.json index 8e25c69..4208f78 100644 --- a/.zenodo.json +++ b/.zenodo.json @@ -1,5 +1,5 @@ { - "title": "CoReNet: Combinatorial Residual Networks for multijet topologies", + "title": "CoReNet: Combinatorial Residual Network", "upload_type": "software", "description": "Reference implementation of CoReNet, a configuration-driven combinatorial residual network for jet assignment and event classification in multijet final states, with the HH to bbbb, HH to bbWW and ttH examples.", "creators": [ diff --git a/CITATION.cff b/CITATION.cff index f059209..30c12bb 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -20,5 +20,5 @@ authors: given-names: "Alejandro" orcid: "https://orcid.org/0000-0002-9443-7769" affiliation: "Carnegie Mellon University" -# doi: "10.xxxx/zenodo.xxxxxxx" # to be confirmed +doi: "10.5281/zenodo.23577720" # preferred-citation: # to be confirmed (article describing the architecture) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd648c9..1de6d70 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,21 +14,11 @@ cd CoReNet ### 2. Environment Setup -#### Option A: Using Pixi (Recommended on Linux with CUDA GPU) ```bash pixi install pixi shell ``` -#### Option B: Using Pip / Virtual Environment (CPU / Local Dev) -```bash -python -m venv .venv -source .venv/bin/activate -pip install --upgrade pip -pip install torch --index-url https://download.pytorch.org/whl/cpu -pip install -e ".[test,export,lint]" -``` - --- ## Code Quality & Standards @@ -68,6 +58,36 @@ To run with test coverage: pytest --cov=corenet tests/ ``` +### Parity with the research code + +```bash +CORENET_REFERENCE_REPO=/path/to/research-checkout pytest tests/reference -s +``` + +- `CORENET_REFERENCE_REPO`: checkout of the research repository; its `src/hcr` is imported, `data/h5/{hh4b,bbww}_all.h5` and the checkpoints under `models/` are read. Without it these tests are skipped. + +The tests require identical configurations, parameter counts, initial weights, loader tensors, eval-mode outputs and checkpoint evaluation. Training-mode outputs are not compared with the research code, because CoReNet embeds the slots of one leaf role in one node together (they share the ghost-batch statistics); the golden-file test below covers them. + +### Golden files of the training-mode numerics + +`pytest tests` runs `tests/test_training_regression.py`: training-mode outputs, first-step gradients, five SGD steps and the eval-mode outputs after them, on 1,024 events of `examples/data/{hh4b,bbww}_small.h5`, on the CPU with one thread, compared with `tests/golden/{hh4b,bbww}.npz`. + +After an intentional change of the training-mode numerics, or of `examples/data/{hh4b,bbww}_small.h5`: + +1. Regenerate the golden files: + ```bash + python scripts/make_golden.py + ``` + - `hh4b` or `bbww`: regenerate this case only (default: both) +2. Read the printed change per stage against the old files and the two noise probes. The script writes nothing if a probe exceeds a tenth of the test tolerance or the files exceed 2 MB. +3. Run `pytest tests/test_training_regression.py`. +4. Commit `tests/golden/*.npz` in the same commit as the change and give the reason in the commit message. + +| file | content | +|---|---| +| `tests/golden/hh4b.npz` | HH → 4b: initial weights, frozen input statistics, training-mode outputs, first-step gradients, losses, weights after five SGD steps, eval-mode outputs | +| `tests/golden/bbww.npz` | the same for HH → bbWW | + --- ## Pull Request Guidelines diff --git a/README.md b/README.md index 55ea878..6b4e1a0 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ # CoReNet: Combinatorial Residual Network +[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23577720.svg)](https://doi.org/10.5281/zenodo.23577720) [![CI](https://github.com/cms-cmu/CoReNet/actions/workflows/ci.yml/badge.svg)](https://github.com/cms-cmu/CoReNet/actions/workflows/ci.yml) @@ -19,7 +20,7 @@ pixi shell # activate; or prefix every command with `pixi ``` pixi itself is a single static binary (`curl -fsSL https://pixi.sh/install.sh | sh`) that needs no root -access, so the same commands work on a laptop and on a cluster front end. +access. The environment is built for Linux (x86-64) with an NVIDIA GPU (CUDA 12). This installs the command-line tools `corenet-validate`, `corenet-train`, `corenet-evaluate`, `corenet-export` and `corenet-report` (each has `--help`), and the Python package `corenet`. The unit tests run with @@ -94,6 +95,11 @@ outputs: combine: [HH] ``` +The optional `labels:` section names the labels to train on, in label order (e.g. +`labels: [multijet, HH, ttbar]`: label 0 = multijet, 1 = HH, 2 = ttbar). Without it the classes are +"rest" (label 0) plus the outputs in order; with it every output name must be one of the labels and the +main score is the class of the first output ([EventInfo.md](docs/EventInfo.md)). + `jets` is the jet collection: `max: 12` is the number of jet slots read per event, and the input file must have at least that many. The listed `features` are the inputs of the network: these columns of the input file, and only these, are fed to it, for the `b` jets and for the `other` jets alike. \ @@ -143,7 +149,10 @@ corenet-train --config examples/bbww/bbww.yaml --input examples/data/bbww_all.h5 * `--freeze-epoch 15 --finetune-layers head_main. head_reduce. aux_heads. --finetune-lr 1.0e-3 --finetune-batch-size 4096`: the fine-tuning stage; from epoch 15 on only the parameters whose names start with the given prefixes (default: the output layers; `none` freezes nothing) are trained, at this learning rate and batch size - (default: the schedule continues). + (default: the schedule continues); `--finetune-reset-optimizer` starts a new optimizer and learning-rate + schedule at that epoch instead of continuing the main stage's. +* `--masked-classes SR=d4`: for the events with `EVENT/SR` true, remove the class `d4` from the + softmax in training (a blinded signal region); the evaluation uses all classes ([docs/Options.md](docs/Options.md)). * `--epochs`, `--lr`, `--seed`, ...: override the training settings of the configuration file. See [docs/Options.md](docs/Options.md) or `corenet-train --help`. @@ -269,4 +278,7 @@ class_probabilities, views_HH = sess.run(None, feeds) If you use this software for a publication, please cite it as given in [CITATION.cff](CITATION.cff) (to be completed) and the paper describing the method: [paper reference / arXiv to be added]. +The software is archived on Zenodo with the concept DOI +[10.5281/zenodo.23577720](https://doi.org/10.5281/zenodo.23577720) (version 1.0.0: +10.5281/zenodo.23577721); [CITATION.cff](CITATION.cff) holds the citation. Released under the MIT licence ([LICENSE](LICENSE)). diff --git a/docs/BBWW.md b/docs/BBWW.md index ef71361..542e54a 100644 --- a/docs/BBWW.md +++ b/docs/BBWW.md @@ -65,7 +65,6 @@ training: attention_gate: true kv_projection: true phase_symmetric: true - reflect_eta: false ``` ### `objects` @@ -101,7 +100,7 @@ Two outputs, hence three classes: 0 = rest, 1 = signal (HH), 2 = tt̄; the examp ### `training` -All network settings at their defaults except `reflect_eta: false`. +All network settings at their defaults. `examples/bbww/bbww_hyperparameters.yaml` writes every setting out ([Options.md](./Options.md)). ## Training, evaluation and export diff --git a/docs/Dataset.md b/docs/Dataset.md index ffdae51..fd653a3 100644 --- a/docs/Dataset.md +++ b/docs/Dataset.md @@ -43,6 +43,8 @@ of shape `[NUM_EVENTS, MAX_JETS]`. Every event is stored with the same number of `MAX_JETS`, so events with fewer jets are padded; `MASK` marks a real jet with `True` and a padding slot with `False`. Store all jets of the event, not only the b-jet candidates: which jets are the b jets is decided by the event description when the file is loaded, so one file serves any choice. +The loader turns `MASK` into the last channel of every object tensor (+1 real, -1 padding); for the +`context` objects this channel is fed to the network as given (+1/-1 from the loader). The column names are the ones used in the event description. The columns listed as `features` there are the inputs of the network: `pt`, `eta`, `phi` and `mass` are the kinematics, any other feature @@ -64,7 +66,9 @@ A collection named `x` is read from `INPUTS/x`, matched case-insensitively, or f * **Labels.** `EVENT/label` (or, when absent, `CLASSIFICATIONS/EVENT/signal`) holds the class of the event: `0` for background not described by a hypothesis ("rest"), and `1, 2, ...` for the outputs - of the event description in order (`1` = signal in both examples). + of the event description in order (`1` = signal in both examples). A `labels:` section in the event + description replaces this convention by its own list of class names in label order + ([EventInfo.md](./EventInfo.md)). * **Weights.** `EVENT/weight` is the event weight (1 when absent). Each class is normalised during training, so only the relative weights within a class matter. Negative weights count as 0. * **Folds.** The cross-validation fold of an event is `EVENT/fold` when the file has that column; the @@ -80,6 +84,10 @@ A collection named `x` is read from `INPUTS/x`, matched case-insensitively, or f `(k + 1) % n_folds`). Only selecting a fold needs the folds: `corenet-evaluate --fold -1` and `corenet-export` also work on a file with neither `EVENT/fold` nor `--n-folds` (the events then carry fold -1 in the prediction file). +* **Flags.** Boolean columns `EVENT/` named by `masked_classes` ([Options.md](./Options.md)) mark + the events whose listed classes are removed from the softmax in training. They are boolean, or + integers 0/1. No flagged event may carry one of the classes masked for its flag: `corenet-train` + and `corenet-validate` reject such a file and give the number of offending events per flag and class. * **Event identifier.** `EVENT/event` is copied to the prediction files and is the input of the `squares(event) % n_folds` rule. diff --git a/docs/EventInfo.md b/docs/EventInfo.md index 2a12d77..488831f 100644 --- a/docs/EventInfo.md +++ b/docs/EventInfo.md @@ -12,6 +12,7 @@ hypotheses: # the decay chains (required) attention: # which candidates are compared with which (optional) relations: # pairs of objects the classifier looks at (optional) outputs: # the classes (required) +labels: # labels to train on, in label order (optional) combine: # the hypotheses the classifier is built from (required) training: # network settings (optional, see Options.md) trainer: # training settings (optional, see Options.md) @@ -162,6 +163,16 @@ first output is the main one: the classifier gives the probability of every clas hypotheses in `combine`. Every further output also gets its own score from its hypothesis alone (`heads/` in the prediction files). Every hypothesis must appear in `combine` or in `outputs`. +```yaml +labels: [multijet, HH, ttbar] # labels to train on, in label order (0, 1, 2, ...); optional +``` + +`labels` (optional) names the labels to train on, in label order: label `0` is the first name, `1` the +second, and so on (at least two, all different). Without it the classes are "rest" plus the outputs, as +above. With it the classes are exactly these names, every output name must be one of them, and the main +score is the probability of the class of the first output; auxiliary heads are still built for the +further outputs only, so a network can have any number of classes without auxiliary heads. + ## `training` and `trainer` ```yaml diff --git a/docs/Options.md b/docs/Options.md index e9fc4c2..9a5afe0 100644 --- a/docs/Options.md +++ b/docs/Options.md @@ -35,11 +35,26 @@ Changing one of these gives a different network: validate the file with `corenet | `finetune_layers` | `--finetune-layers PREFIX [PREFIX ...]` | `head_main.`, `head_reduce.`, `aux_heads.` (the output layers) | parameter-name prefixes that stay trainable from `freeze_epoch` on; `none` (the word, alone; `[]` in the file) freezes nothing | | `finetune_lr` | `--finetune-lr` | unset | learning rate set at `freeze_epoch`; the `lr_milestones` after it multiply this value by `lr_gamma`; unset keeps the schedule | | `finetune_batch_size` | `--finetune-batch-size` | unset | batch size from `freeze_epoch` on; the ghost batches of the stage are kept when they divide it, else the largest smaller number that does, so that every ghost batch has at least 8 events; unset keeps the schedule | +| `finetune_reset_optimizer` | `--finetune-reset-optimizer` / `--no-finetune-reset-optimizer` | off | at freeze_epoch, start a new optimizer and learning-rate schedule over the trainable parameters (a fresh fine-tuning stage) instead of continuing the main stage's optimizer | +| `masked_classes` | `--masked-classes FLAG=CLASS[,CLASS]` | none | classes removed from the softmax for the events flagged by `EVENT/FLAG`; the flagged events still train the other classes; the file must hold no flagged event of an excluded class | | `class_norm` | `--class-norm` / `--no-class-norm` | true | give every class the same total weight in training | | `seed` | `--seed` | 42 | random seed; fold `k` uses `seed + k` | -| `early_stop_patience` | `--early-stop-patience` | 0 | stop when the validation AUC (on the `--exclude-folds` fold) has not improved for this many epochs, and keep the weights of the best epoch; 0 trains every epoch. The validation fold must differ from the held-out fold and hold events of class 1 and of another class, otherwise the run stops with an error | +| `early_stop_patience` | `--early-stop-patience` | 0 | stop when the validation AUC (on the `--exclude-folds` fold) has not improved for this many epochs, and keep the weights of the best epoch; 0 trains every epoch. The validation fold must differ from the held-out fold and hold events of the main class (the class of the first output; class 1 without `labels:`) and of another class, otherwise the run stops with an error | | `rest_order` | `--rest-order` | `pt` | order of the jets taken by a `rest` line: `pt`, or `selector` to keep the order of the previous candidate line on the same collection (e.g. `b: jets btag 4`) | +`masked_classes` serves a training with a blinded region. Example: a classifier of the classes +`d3`, `d4` and `ttbar` is trained with the 4-tag data of the signal region left out of the file. The +3-tag data and the tt̄ events of the signal region are kept and flagged by `EVENT/SR`; with + +```yaml +trainer: + masked_classes: + SR: [d4] # for events with EVENT/SR true, the d4 class is removed from the softmax +``` + +these events train the network to tell `d3` from `ttbar` only, and no event of the signal region +pulls the `d4` score. `corenet-evaluate` and the prediction files always use all classes. + Write floating-point numbers with a decimal point (`0.005` or `5.0e-3`): YAML reads `5e-3` as a string, which the validation rejects. diff --git a/docs/TTH.md b/docs/TTH.md index a7b6cfa..e1743b5 100644 --- a/docs/TTH.md +++ b/docs/TTH.md @@ -75,7 +75,7 @@ training: attention_gate: true kv_projection: true phase_symmetric: true - reflect_eta: false + reflect_eta: true # use the forward-backward symmetry of the detector ``` ### `objects` @@ -116,7 +116,7 @@ One output, `signal: ttH`, hence two classes: 0 = rest (semileptonic tt̄), 1 = ### `training` -All network settings at their defaults except `reflect_eta: false`. [Options.md](./Options.md) +All network settings at their defaults. [Options.md](./Options.md) describes every setting. ## Training, evaluation and export diff --git a/examples/bbww/bbww.yaml b/examples/bbww/bbww.yaml index afee0e8..06b5b89 100644 --- a/examples/bbww/bbww.yaml +++ b/examples/bbww/bbww.yaml @@ -38,4 +38,4 @@ training: attention_gate: true kv_projection: true phase_symmetric: true - reflect_eta: false + reflect_eta: true # use the forward-backward symmetry of the detector diff --git a/examples/bbww/bbww_hyperparameters.yaml b/examples/bbww/bbww_hyperparameters.yaml index ca8ebd1..4e1329d 100644 --- a/examples/bbww/bbww_hyperparameters.yaml +++ b/examples/bbww/bbww_hyperparameters.yaml @@ -1,6 +1,6 @@ -# The HH -> bbWW network of bbww.yaml with EVERY tunable option written out at its default value -# (except reflect_eta, which this example switches off). Training with this file gives the same -# result as with bbww.yaml. Copy it and edit the values. Reference: docs/Options.md. +# The HH -> bbWW network of bbww.yaml with EVERY tunable option written out at its default value. +# Training with this file gives the same result as with bbww.yaml. Copy it and edit the values. +# Reference: docs/Options.md. objects: jets: {features: [pt, eta, phi, mass], max: 12} b: jets btag 2 @@ -41,7 +41,7 @@ training: attention_gate: true # gated comparison of candidates kv_projection: true # reduced-size projections when candidates are compared phase_symmetric: true # phase-symmetric layers - reflect_eta: false # (default true) use the forward-backward symmetry of the detector + reflect_eta: true # use the forward-backward symmetry of the detector head_weights: {ttbar: 1.0} # loss weight of the extra ttbar score (default 1.0) # ---- training settings (read by corenet-train; a command-line flag overrides the value here) @@ -52,6 +52,8 @@ trainer: lr_gamma: 0.25 batch_size_init: 1024 # epoch-0 batch size; doubled at epochs 1, 3, 6 and 10 (fixed schedule) freeze_epoch: 15 # from this epoch on only the output layers are trained + finetune_reset_optimizer: false # true: new optimizer and lr schedule at freeze_epoch class_norm: true # normalise the event weights of every class to 1 over the training set seed: 42 # random seed; the seed of fold k is seed + k rest_order: pt # order of the jets taken by a `rest` line: pt | selector + masked_classes: {} # flag -> classes removed from the softmax for the flagged events, e.g. {SR: [d4]} diff --git a/examples/hh4b/hh4b_hyperparameters.yaml b/examples/hh4b/hh4b_hyperparameters.yaml index 318e958..2e6d6e7 100644 --- a/examples/hh4b/hh4b_hyperparameters.yaml +++ b/examples/hh4b/hh4b_hyperparameters.yaml @@ -36,6 +36,8 @@ trainer: lr_gamma: 0.25 batch_size_init: 1024 # epoch-0 batch size; doubled at epochs 1, 3, 6 and 10 (fixed schedule) freeze_epoch: 15 # from this epoch on only the output layers are trained + finetune_reset_optimizer: false # true: new optimizer and lr schedule at freeze_epoch class_norm: true # normalise the event weights of every class to 1 over the training set seed: 42 # random seed; the seed of fold k is seed + k rest_order: pt # order of the jets taken by a `rest` line: pt | selector + masked_classes: {} # flag -> classes removed from the softmax for the flagged events, e.g. {SR: [d4]} diff --git a/examples/tth/tth.yaml b/examples/tth/tth.yaml index cad3da1..240cf0b 100644 --- a/examples/tth/tth.yaml +++ b/examples/tth/tth.yaml @@ -37,4 +37,4 @@ training: attention_gate: true kv_projection: true phase_symmetric: true - reflect_eta: false + reflect_eta: true # use the forward-backward symmetry of the detector diff --git a/pyproject.toml b/pyproject.toml index 51d19af..1d6c2bd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "CoReNet" -version = "1.0.0" +version = "1.1.0" description = "CoReNet (Combinatorial Residual Network): a decay-chain-configurable neural network for event classification in collider physics" readme = "README.md" license = "MIT" diff --git a/scripts/make_golden.py b/scripts/make_golden.py new file mode 100644 index 0000000..1c54cb8 --- /dev/null +++ b/scripts/make_golden.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Regenerate the golden files of the training-mode regression test (tests/golden/.npz). + + python scripts/make_golden.py # both cases: tests/golden/hh4b.npz, tests/golden/bbww.npz + python scripts/make_golden.py hh4b # one case + +Run it on the CPU, only after an intentional change of the training-mode numerics (or of +examples/data/_small.h5), and commit the files with that change. Per case it prints the change +against the file it replaces (per stage, the largest relative error as defined in +tests/training_regression.py) and two noise probes: another thread count, and the initial weights moved by +one float32 rounding. It writes nothing when a probe exceeds a tenth of the test tolerance, when a value is +not finite, or when the golden files would exceed 2 MB in total. +""" + +from __future__ import annotations + +import argparse +import datetime +import io +import subprocess +import sys +from pathlib import Path + +import numpy as np +import torch + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "tests")) +import training_regression as tr # noqa: E402 + +PROBE_THREADS = 4 +MAX_BYTES = 2_000_000 + + +def _git(*args) -> str: + try: + return subprocess.run( + ["git", "-C", str(REPO), *args], capture_output=True, text=True, check=True + ).stdout.strip() + except (OSError, subprocess.CalledProcessError): + return "unknown" + + +def _by_stage(errors: dict) -> dict: + out = {} + for k, e in errors.items(): + s = k.split(":", 1)[0] + out[s] = max(out.get(s, 0.0), e) + return out + + +def _fmt(by_stage: dict) -> str: + return " ".join(f"{s} {e:.1e}" for s, e in by_stage.items()) + + +def main(argv=None) -> int: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + p.add_argument( + "cases", nargs="*", help=f"cases to regenerate {list(tr.CASES)} [all]" + ) + args = p.parse_args(argv) + cases = args.cases or list(tr.CASES) + unknown = [c for c in cases if c not in tr.CASES] + if unknown: + p.error(f"unknown case(s) {unknown}; choose from {list(tr.CASES)}") + commit = _git("rev-parse", "--short", "HEAD") + src_dirty = _git("status", "--porcelain", "--", "src") != "" + blobs, ok = {}, True + for case in cases: + arrays = tr.compute(case) # initial state from torch.manual_seed(tr.SEED) + init = tr.stage(arrays, "init") + nonfinite = [ + k + for k, v in arrays.items() + if np.issubdtype(v.dtype, np.floating) and not np.all(np.isfinite(v)) + ] + if nonfinite: + print(f"{case}: non-finite values in {nonfinite[:10]}") + ok = False + probes = { + f"threads={PROBE_THREADS}": tr.compute( + case, init=init, threads=PROBE_THREADS + ), + "init+1ulp": tr.compute(case, init=tr.perturb(case, init)), + } + for name, other in probes.items(): + by_stage = _by_stage(tr.compare(other, arrays)[0]) + print(f"{case}: probe {name}: {_fmt(by_stage)}") + if max(by_stage.values()) > tr.TOL / 10: + print(f"{case}: probe {name} exceeds TOL/10 = {tr.TOL / 10:g}") + ok = False + old = tr.GOLDEN_DIR / f"{case}.npz" + if old.exists(): + g_old, m_old = tr.load(old) + errors, one_sided = tr.compare(arrays, g_old) + print( + f"{case}: change against {old.relative_to(REPO)} (made at {m_old.get('commit')}): " + f"{_fmt(_by_stage(errors))}; arrays added/removed: {len(one_sided)} {one_sided[:10]}" + ) + else: + print(f"{case}: no previous golden file") + info = tr.meta(case) | { + "commit": commit, + "src_dirty": src_dirty, + "date": datetime.date.today().isoformat(), + "torch": torch.__version__, + "numpy": np.__version__, + } + buf = io.BytesIO() + tr.save(buf, arrays, info) + blobs[case] = buf.getvalue() + print( + f"{case}: {len(arrays)} arrays, {len(blobs[case]):,} bytes; " + f"losses {np.round(arrays['loss:sgd'], 6).tolist()}" + ) + others = ( + [f for f in tr.GOLDEN_DIR.glob("*.npz") if f.stem not in blobs] + if tr.GOLDEN_DIR.exists() + else [] + ) + total = sum(len(b) for b in blobs.values()) + sum(f.stat().st_size for f in others) + print(f"golden files in total: {total:,} bytes (limit {MAX_BYTES:,})") + if total > MAX_BYTES: + ok = False + if not ok: + print("NOT WRITTEN") + return 1 + tr.GOLDEN_DIR.mkdir(parents=True, exist_ok=True) + for case, blob in blobs.items(): + path = tr.GOLDEN_DIR / f"{case}.npz" + path.write_bytes(blob) + print(f"wrote {path.relative_to(REPO)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/corenet/checkpoint.py b/src/corenet/checkpoint.py index 1cbe27b..fb2a681 100644 --- a/src/corenet/checkpoint.py +++ b/src/corenet/checkpoint.py @@ -3,7 +3,7 @@ {"model": state dict of CoReNet, "config": the configuration mapping the network was built from (so a checkpoint is self-contained), "config_path": path of the configuration file at training time, - "classes": ["rest", ...], + "classes": spec.classes (["rest", ...], or the ``labels:`` list), "metadata": {"fold", "epoch", "exclude_folds", "input", "n_folds", "fold_rule" ("EVENT/fold" or "squares(event) % n_folds"), "fold_seed", , "finetune_trainable_parameters"}, diff --git a/src/corenet/data/folds.py b/src/corenet/data/folds.py index a975ede..9eec8e1 100644 --- a/src/corenet/data/folds.py +++ b/src/corenet/data/folds.py @@ -8,11 +8,10 @@ The fold of an event therefore depends only on its event number, the number of folds and the seed: any subset of a sample, in any order, gets the same folds, and no random split is ever made. -This is a copy of the rule of the research converter that wrote the example files -(``hcr.data.delphes_h5.assign_folds`` with ``hcr.math_tools.random.Squares``), kept bit for bit so -that the package reproduces their ``EVENT/fold`` column: the shipped files -``examples/data/{hh4b,bbww}_{small,all}.h5`` were written with ``n_folds=5`` and ``seed=0``, so -``--n-folds 5`` (``--fold-seed 0``) gives exactly the folds stored in them. +This is the rule that wrote the ``EVENT/fold`` column of the example files, kept bit for bit so that +the package reproduces it: the shipped files ``examples/data/{hh4b,bbww}_{small,all}.h5`` were +written with ``n_folds=5`` and ``seed=0``, so ``--n-folds 5`` (``--fold-seed 0``) gives exactly the +folds stored in them. """ from __future__ import annotations diff --git a/src/corenet/data/h5.py b/src/corenet/data/h5.py index 9c38dfe..837c6c2 100644 --- a/src/corenet/data/h5.py +++ b/src/corenet/data/h5.py @@ -75,6 +75,7 @@ "none", ) # H5Dataset.fold_rule; "none": the file has no EVENT/fold and no n_folds was given NO_FOLD = -1 # the fold of every event of a file loaded without folds +FLAG_GROUP = "EVENT" # flag columns of masked_classes: EVENT/ # ------------------------------------------------------------------ what the loader reads (shared with corenet-validate) @@ -118,6 +119,76 @@ def resolve_group(f: h5py.File, name: str) -> str: ) +def read_flag(f: h5py.File, name: str, n: int | None = None) -> np.ndarray: + """The boolean event column ``EVENT/`` (a flag of ``masked_classes``): boolean, or integer + 0/1 (cast). KeyError when missing, ValueError for another dtype, shape or length.""" + key = f"{FLAG_GROUP}/{name}" + where = Path(f.filename).name + if key not in f: + raise KeyError( + f"{where}: no {key} dataset (flag column named by masked_classes)" + ) + a = f[key][...] + if a.ndim != 1 or (n is not None and len(a) != n): + raise ValueError( + f"{where}: {key} has shape {a.shape}; a flag column must hold one value per event" + + (f" ({n})" if n is not None else "") + ) + if a.dtype.kind == "b": + return a.astype(bool) + if a.dtype.kind in "iu" and np.isin(a, (0, 1)).all(): + return a.astype(bool) + raise ValueError( + f"{where}: {key} has dtype {a.dtype}" + + (" with values other than 0/1" if a.dtype.kind in "iu" else "") + + "; a flag column must be boolean or integer 0/1" + ) + + +def masked_label_problems( + flags: dict[str, np.ndarray], + labels: np.ndarray, + spec: NetworkSpec, + masked_classes: dict, +) -> list[str]: + """One message per (flag, excluded class) with flagged events of that class: such an event would + have its own class removed from the softmax (a blinding or configuration error).""" + problems = [] + for flag, classes in masked_classes.items(): + for c in classes: + i = spec.classes.index(c) + n_bad = int((flags[flag] & (labels == i)).sum()) + if n_bad: + problems.append( + f"masked_classes: {n_bad:,} events with {FLAG_GROUP}/{flag} true carry the excluded " + f"class `{c}` (label {i}); events of an excluded class must not be flagged" + ) + return problems + + +def check_masked_classes( + f: h5py.File, spec: NetworkSpec, masked_classes: dict +) -> list[str]: + """Problems of a file for ``masked_classes`` (empty list: none): every flag column exists and is + boolean or 0/1 (``read_flag``), and no flagged event carries one of its excluded classes.""" + key = next((k for k in LABEL_KEYS if k in f), None) + labels = f[key][...].astype(np.int64) if key else None + problems, flags = [], {} + for flag in masked_classes: + try: + flags[flag] = read_flag( + f, flag, len(labels) if labels is not None else None + ) + except (KeyError, ValueError) as e: + problems.append(str(e.args[0])) + if problems: + return problems + if labels is None: # the loader gives every event label 0 + n = len(next(iter(flags.values()))) if flags else 0 + labels = np.zeros(n, dtype=np.int64) + return masked_label_problems(flags, labels, spec, masked_classes) + + def _sortable(a: np.ndarray) -> np.ndarray: """A column as read from the file, in a dtype that sorts exactly and can be negated: floats and signed integers unchanged, bool -> int8, unsigned integers -> int64.""" @@ -184,9 +255,14 @@ def __init__( rest_order: str = "pt", n_folds: int | None = None, fold_seed: int = FOLD_SEED, + flags: list[str] | None = None, + masked_classes: dict | None = None, ): """``n_folds``: use fold = squares(event) % n_folds (``fold_of``, seed ``fold_seed``) instead of the - ``EVENT/fold`` column. Without both, the file loads without folds (``has_folds`` is False).""" + ``EVENT/fold`` column. Without both, the file loads without folds (``has_folds`` is False). + ``flags``: boolean event columns ``EVENT/`` read into ``self.flags`` (default: the flags of + ``masked_classes``); ``masked_classes`` (flag -> class names): a flagged event of an excluded class + raises ValueError.""" if rest_order not in ("pt", "selector"): raise ValueError( f"rest_order must be 'pt' or 'selector', got {rest_order!r}" @@ -235,6 +311,19 @@ def __init__( ) self.fold_seed = int(fold_seed) self.folds, self.n_folds, self.fold_rule = self._folds(f, n_folds, n) + masked_classes = masked_classes or {} + names = list(dict.fromkeys([*(flags or []), *masked_classes])) + self.flags: dict[str, torch.Tensor] = { + k: torch.from_numpy(read_flag(f, k, n)) for k in names + } + problems = masked_label_problems( + {k: v.numpy() for k, v in self.flags.items()}, + self.labels.numpy(), + spec, + masked_classes, + ) + if problems: + raise ValueError(f"{self.path.name}: " + "; ".join(problems)) self.inputs = inputs @property @@ -349,11 +438,12 @@ def _labels(self, f: h5py.File, n: int) -> np.ndarray: if key not in f: return np.zeros(n, dtype=np.int64) lab = f[key][...].astype(np.int64) - bad = np.unique(lab[(lab < 0) | (lab > len(self.spec.outputs))]) + top = len(self.spec.classes) - 1 + bad = np.unique(lab[(lab < 0) | (lab > top)]) if len(bad): raise ValueError( - f"{self.path.name}: labels {bad.tolist()} outside 0..{len(self.spec.outputs)} " - f"({len(self.spec.outputs)} outputs: {[o.name for o in self.spec.outputs]})" + f"{self.path.name}: labels {bad.tolist()} outside 0..{top} " + f"(classes {self.spec.classes})" ) return lab @@ -374,6 +464,7 @@ def __getitem__(self, idx): "label": self.labels[idx], "weight": self.weights[idx], "fold": self.folds[idx], + "flags": {k: v[idx] for k, v in self.flags.items()}, } def fold_indices(self, fold: int) -> tuple[np.ndarray, np.ndarray]: @@ -395,10 +486,13 @@ def fold_indices(self, fold: int) -> tuple[np.ndarray, np.ndarray]: "NO_FOLD", "WEIGHT_KEY", "H5Dataset", + "check_masked_classes", "columns_to_read", "event_source", "fold_of", "group_candidates", + "masked_label_problems", + "read_flag", "resolve_group", "used_collections", ] diff --git a/src/corenet/evaluate.py b/src/corenet/evaluate.py index 92a5427..411af8c 100644 --- a/src/corenet/evaluate.py +++ b/src/corenet/evaluate.py @@ -14,7 +14,8 @@ main output, the ROC curve; the AUCs are printed and written as JSON next to it (``.json``). Metrics use the event weights of the file, clipped at >= 0 (``--unweighted`` ignores them): - * main AUC: class 1 (the first output) against every other class, scored by P(class 1); + * main AUC: the main class (the class of the first output: class 1 without ``labels:``, + ``spec.main_class``) against every other class, scored by its probability; * one-vs-rest AUC of every class c present, scored by P(c); * the AUC of every auxiliary head against ``label == its class``. """ @@ -47,8 +48,24 @@ class Batches: def __init__(self, ds: H5Dataset, device): self.inputs = {k: v.to(device) for k, v in ds.inputs.items()} self.labels = ds.labels.to(device) + self.flags = {k: v.to(device) for k, v in ds.flags.items()} + self.classes = list(ds.spec.classes) self.device = device + def class_mask( + self, sel: torch.Tensor, masked_classes: dict, n_classes: int + ) -> torch.Tensor | None: + """(len(sel), n_classes) bool, True where the class is removed from the event's softmax + (``masked_classes``: flag -> class names, for the events with that flag true); None when empty.""" + if not masked_classes: + return None + mask = torch.zeros((len(sel), n_classes), dtype=torch.bool, device=self.device) + for flag, classes in masked_classes.items(): + rows = self.flags[flag][sel] + for c in classes: + mask[rows, self.classes.index(c)] = True + return mask + def slice(self, idx: np.ndarray) -> dict[str, torch.Tensor]: sel = torch.as_tensor(idx, device=self.device) return {k: v[sel] for k, v in self.inputs.items()} @@ -146,8 +163,9 @@ def auc(y, s): return None return float(roc_auc_score(y, s, sample_weight=w)) - y_main = label == 1 - out["auc"]["main"] = auc(y_main, probs[:, 1]) + main = spec.main_class + y_main = label == main + out["auc"]["main"] = auc(y_main, probs[:, main]) out["auc"]["one_vs_rest"] = { c: auc(label == i, probs[:, i]) for i, c in enumerate(spec.classes) @@ -159,7 +177,7 @@ def auc(y, s): } roc = None if y_main.any() and not y_main.all(): - fpr, tpr, thr = roc_curve(y_main, probs[:, 1], sample_weight=w) + fpr, tpr, thr = roc_curve(y_main, probs[:, main], sample_weight=w) roc = { "fpr": fpr.astype(np.float32), "tpr": tpr.astype(np.float32), diff --git a/src/corenet/export.py b/src/corenet/export.py index a7fd052..02dd4e9 100644 --- a/src/corenet/export.py +++ b/src/corenet/export.py @@ -10,7 +10,7 @@ positional input per leaf in the order listed in the JSON, then ``event``), with a dynamic batch dimension. Its outputs are probabilities: - class_probabilities (N, nClasses) softmax of the main head, classes ["rest", ...] + class_probabilities (N, nClasses) softmax of the main head, classes ``spec.classes`` views_ (N, V) softmax over the candidates of every multi-candidate hypothesis head_ (N,) sigmoid of every auxiliary head diff --git a/src/corenet/models/config.py b/src/corenet/models/config.py index f3a0473..08fca1d 100644 --- a/src/corenet/models/config.py +++ b/src/corenet/models/config.py @@ -15,6 +15,7 @@ attention: [{W_lep: W_had}, {t_had: t_lep}] # query: keys (exclusivity mask inferred) relations: [[lepton, met], [H_bb, met]] # named pairs -> (log M, dR, dPhi) pixels into the main head outputs: {signal: HH, ttbar: tt} # first = main score; names are the classes (labels 1, 2, ...) + labels: [multijet, HH, ttbar] # labels to train on, in label order (0, 1, 2, ...); optional combine: [HH, tt] # must contain the main hypothesis training: {features: 8, attention_heads: 2, ...} # network-wide options trainer: {epochs: 20, lr: 0.01, ...} # optional training-recipe defaults, read by corenet-train only @@ -42,6 +43,7 @@ "attention", "relations", "outputs", + "labels", "combine", "training", "trainer", @@ -131,13 +133,25 @@ class NetworkSpec: default_factory=dict ) # node -> hypothesis in which it is declared min_jets: int = 4 # minimum jet multiplicity of the selection: nSelJets enters as log(nSelJets - (min_jets - 1)) + labels: list[str] | None = ( + None # ``labels:``: the class names in label order (None: rest + outputs) + ) # ---- derived @property def classes(self) -> list[str]: - """Label convention: 0 = rest, then the outputs in order.""" + """Label convention: the ``labels:`` list when given, else 0 = rest, then the outputs in order.""" + if self.labels is not None: + return list(self.labels) return ["rest"] + [o.name for o in self.outputs] + @property + def main_class(self) -> int: + """Class index of the main score (the first output): 1 without ``labels:``.""" + if self.labels is not None: + return self.labels.index(self.outputs[0].name) + return 1 + @property def main(self) -> OutputSpec: return self.outputs[0] @@ -447,6 +461,19 @@ def parse_config(cfg: dict) -> NetworkSpec: for h in combine: if h not in hypotheses: raise ConfigError(f"combine: {h!r} is not a declared hypothesis") + # labels ----------------------------------------------------------------------------------- + labels = cfg.get("labels") + if labels is not None: + labels = _str_list(labels, "labels") + if len(labels) < 2: + raise ConfigError( + f"labels: at least two classes are needed, got {labels!r}" + ) + for o in outputs: + if o.name not in labels: + raise ConfigError( + f"outputs.{o.name}: the output name must be one of the labels {labels}" + ) used_h = set(combine) | {o.hypothesis for o in outputs if o.hypothesis} for h in hypotheses: if h not in used_h: @@ -496,6 +523,7 @@ def parse_config(cfg: dict) -> NetworkSpec: relations, owner, min_jets, + labels, ) diff --git a/src/corenet/models/network.py b/src/corenet/models/network.py index 85cb4d7..04edc35 100644 --- a/src/corenet/models/network.py +++ b/src/corenet/models/network.py @@ -10,9 +10,10 @@ 2. features log(1+pt), eta, dphi-to-parent, log(1+m) for every child slot; the top node drops phi; event scalars whose name contains ``nSelJets`` enter as log(nSelJets - (min_jets - 1)) - 3. embedding GBN embed (+ event scalars for leaves) -> conv, per role / node; the ``context`` - objects are embedded from their kinematics, a flag and the context role's extra - (non-kinematic) features + 3. embedding GBN embed (+ event scalars for leaves) -> conv, per role / node (all slots of one + role in one node in a single call, so they share the ghost-batch statistics); the ``context`` + objects are embedded from their kinematics, a flag (the last input channel as + given) and the context role's extra (non-kinematic) features 4. reinforce bottom-up ResNetBlock per composite node (a node reused by several hypotheses is computed once); ``context`` attention after the first level; ``attention`` entries as soon as both nodes exist, with the pair @@ -443,7 +444,7 @@ def _prepare(self, inputs): n = next(iter(inputs.values())).shape[0] # per leaf (role or single object): four-vectors (n, 4, slots), its non-kinematic features (isE / isM, ...; # every feature of the collection is an input), and which slots hold a real object - leaf_kin, leaf_extra, leaf_present, etaSign = {}, {}, {}, None + leaf_kin, leaf_extra, leaf_present, leaf_flag, etaSign = {}, {}, {}, {}, None for r in self.leaf_order: if r not in inputs: continue @@ -458,6 +459,10 @@ def _prepare(self, inputs): extras = [feats.index(f) for f in self._extras(r)] leaf_extra[r] = x[:, extras, :] if extras else None leaf_present[r] = x[:, -1, :] != -1 # presence flag is the last channel + # the raw flag channel, fed to the context objects as given: the loader writes +1/-1 there (so + # this equals where(present, 1, -1)); an external caller may put a 0/1 quality flag in it (-1 = + # padded), e.g. a jet-quality flag + leaf_flag[r] = x[:, -1, :] if self.spec.training.reflect_eta: if etaSign is None: # leading member of the first role etaSign = 1 - 2 * (kin[:, 1, 0:1] < 0).float() @@ -486,6 +491,7 @@ def _prepare(self, inputs): "leaf_kin": leaf_kin, "leaf_extra": leaf_extra, "leaf_present": leaf_present, + "leaf_flag": leaf_flag, "event": a, "hyp": {}, } @@ -493,7 +499,7 @@ def _prepare(self, inputs): kin, kinP, present = {}, {}, {} for h, T in self.tables.items(): out["hyp"][h] = self._prepare_hypothesis( - h, T, leaf_kin, leaf_extra, leaf_present, kin, kinP, present + h, T, leaf_kin, leaf_extra, leaf_present, leaf_flag, kin, kinP, present ) out["kin"], out["kinP"], out["present"] = kin, kinP, present # a node reused by several hypotheses is embedded once; its dphi reference is the parent in the @@ -533,7 +539,7 @@ def vec(name): return out def _prepare_hypothesis( - self, h, T, leaf_kin, leaf_extra, leaf_present, kin, kinP, present + self, h, T, leaf_kin, leaf_extra, leaf_present, leaf_flag, kin, kinP, present ): # kin / kinP / present are shared across hypotheses: a reused node is computed once for node in self.levels[h]: @@ -620,12 +626,20 @@ def _prepare_hypothesis( hs = self.spec.hypotheses[h] if hs.context: prep["context"] = self._prepare_context( - h, T, hs.context, leaf_kin, leaf_extra, leaf_present, kin, kinP + h, + T, + hs.context, + leaf_kin, + leaf_extra, + leaf_present, + leaf_flag, + kin, + kinP, ) return prep def _prepare_context( - self, h, T, crole, leaf_kin, leaf_extra, leaf_present, kin, kinP + self, h, T, crole, leaf_kin, leaf_extra, leaf_present, leaf_flag, kin, kinP ): """Objects = this hypothesis' leaves (jet roles, lepton, MET; flag 2) + the context role (its own flag); pair matrices object-object (flag 0) and first-level-candidate-object (flag 1), with masks. @@ -634,7 +648,10 @@ def _prepare_context( ``feats`` (n, 4 + E, S) is the embedding input of the objects: log pt, eta, log m, flag, then the E extra (non-kinematic) features of the context role in its feature order. An object of another leaf carries the same E channels: an extra its own collection also has is taken from its tensor, one it lacks is -1 (the - padding value); the pair features are kinematic only and unchanged.""" + padding value); the pair features are kinematic only and unchanged. + The flag of a context-role object is the last channel of its input tensor as given (``leaf_flag``): the + loader writes +1 (present) / -1 (padded) there, so for H5 input it is where(present, 1, -1); an external + caller may put a 0/1 quality flag there instead. The mask (flag == -1) is unchanged.""" first = self.levels[h][0] # every leaf of the hypothesis (jet roles and single-object collections such as lepton / MET), flag 2 leaf_roles = [ @@ -647,11 +664,7 @@ def _prepare_context( parts.append(leaf_kin[r]) flags.append(2 * torch.ones_like(leaf_kin[r][:, 0:1, :])) parts.append(leaf_kin[crole]) - flags.append( - torch.where(leaf_present[crole], 1.0, -1.0) - .unsqueeze(1) - .to(leaf_kin[crole].dtype) - ) + flags.append(leaf_flag[crole].unsqueeze(1).to(leaf_kin[crole].dtype)) o = torch.cat([torch.cat(parts, 2), torch.cat(flags, 2)], 1) # (n,5,S) mask = o[:, 4, :] == -1 S = o.shape[2] @@ -715,6 +728,24 @@ def _prepare_context( "P": P, } + # ------------------------------------------------------------------ grouped leaf slots + def _leaf_groups(self, nt) -> dict[str, list[int]]: + """Leaf role -> the child slots of node table ``nt`` that come from it (in slot order).""" + groups: dict[str, list[int]] = {} + for j, c in enumerate(nt.children): + if c in self.leaves: + groups.setdefault(c, []).append(j) + return groups + + @staticmethod + def _grouped_slots(hp, node, js): + """Embedding input (n, F, P * len(js)) and presence (n, P * len(js)) of the slots ``js`` of ``node``, + candidate-major, slot-minor (slot js[0] of candidate 0, slot js[1] of candidate 0, ..., candidate 1, ...).""" + feats = torch.stack([hp["slot_feats"][node][j] for j in js], dim=3) # (n,F,P,m) + pres = torch.stack([hp["slot_present"][node][j] for j in js], dim=2) # (n,P,m) + n, F_, P, m = feats.shape + return feats.reshape(n, F_, P * m), pres.reshape(n, P * m) + # ------------------------------------------------------------------ forward def forward(self, inputs): prep = self._prepare(inputs) @@ -731,20 +762,27 @@ def forward(self, inputs): continue # reused node: computed once nt = T.nodes[node] k = len(nt.children) - # child slot pixels: leaves are embedded per slot (padded slots zeroed); composites are gathered - cols, cols0 = [], [] + # child slot pixels: leaves are embedded (padded slots zeroed); composites are gathered. + # All slots of one leaf role in this node are embedded in ONE call (pixels candidate-major, + # slot-minor), because the ghost-batch statistics of a GBN are taken over all pixels of a call: + # the slots of one role in one node must share them (one embedding of + # all candidate jets of a dijet layer). Different roles keep separate calls. + cols, cols0 = [None] * k, [None] * k + for role, js in self._leaf_groups(nt).items(): + feats, pres = self._grouped_slots(hp, node, js) + x = self.leaf_embed[role](feats, add=a) + x0 = x.clone() + x = NonLU(x) + keep = pres.unsqueeze(1).to(x.dtype) + x, x0 = x * keep, x0 * keep + x = x.view(n, self.d, nt.n, len(js)) + x0 = x0.view(n, self.d, nt.n, len(js)) + for i, j in enumerate(js): + cols[j], cols0[j] = x[:, :, :, i], x0[:, :, :, i] for j, c in enumerate(nt.children): - idx = nt.child_index[:, j] - if c in self.leaves: - x = self.leaf_embed[c](hp["slot_feats"][node][j], add=a) - x0 = x.clone() - x = NonLU(x) - keep = hp["slot_present"][node][j].unsqueeze(1).to(x.dtype) - x, x0 = x * keep, x0 * keep - else: - x, x0 = pixels[c][:, :, idx], pixels0[c][:, :, idx] - cols.append(x) - cols0.append(x0) + if c not in self.leaves: + idx = nt.child_index[:, j] + cols[j], cols0[j] = pixels[c][:, :, idx], pixels0[c][:, :, idx] x = torch.stack(cols, dim=3).reshape(n, self.d, nt.n * k) x0 = torch.stack(cols0, dim=3).reshape(n, self.d, nt.n * k) xx = self.node_embed[node](prep["embed_in"][node]) @@ -883,11 +921,10 @@ def updateMeanStd(self, inputs): continue done.add(node) nt = T.nodes[node] - for j, c in enumerate(nt.children): - if c in self.leaves: - self.leaf_embed[c].embed.updateMeanStd( - hp["slot_feats"][node][j], ~hp["slot_present"][node][j] - ) + # grouped per (node, role) exactly as in forward + for role, js in self._leaf_groups(nt).items(): + feats, pres = self._grouped_slots(hp, node, js) + self.leaf_embed[role].embed.updateMeanStd(feats, ~pres) self.node_embed[node].embed.updateMeanStd(prep["embed_in"][node]) if "context" in hp: ctx = hp["context"] diff --git a/src/corenet/report.py b/src/corenet/report.py index 45d1e53..861e385 100644 --- a/src/corenet/report.py +++ b/src/corenet/report.py @@ -725,11 +725,12 @@ def section_training(self) -> str: # ---- classification def _main_curves(self): - """{split: [(fold, fpr, tpr, auc)]} of the main score (class 1 against the rest).""" + """{split: [(fold, fpr, tpr, auc)]} of the main score (the main class against the rest).""" + m = self.spec.main_class out = {} for name in SPLITS: for fd, s in self.splits_of(name): - y, sc, w = s.label == 1, s.probs[:, 1], np.clip(s.weight, 0, None) + y, sc, w = s.label == m, s.probs[:, m], np.clip(s.weight, 0, None) if y.all() or not y.any(): continue f, t = roc_points(y, sc, w) @@ -793,7 +794,7 @@ def section_classification(self) -> str: parts.append( figure( fig, - f"ROC of the main score P({self.classes[1]}) (class 1 against every other class, weighted): thin lines are the test folds, thick lines the mean over folds with the ± 1 sd band; dashed = training folds, dash-dotted = validation fold, where predictions exist.", + f"ROC of the main score P({self.classes[self.spec.main_class]}) (class {self.spec.main_class} against every other class, weighted): thin lines are the test folds, thick lines the mean over folds with the ± 1 sd band; dashed = training folds, dash-dotted = validation fold, where predictions exist.", ) ) # AUC table @@ -837,18 +838,19 @@ def _score_figure(self) -> str: if not test: return "" bins = np.linspace(0, 1, 41) + m = self.spec.main_class fig, (a, b) = plt.subplots(1, 2, figsize=(7.8, 3.2)) for name in ("test", "train"): sp = self.splits_of(name) if not sp: continue lab = np.concatenate([s.label for _, s in sp]) - sc = np.concatenate([s.probs[:, 1] for _, s in sp]) + sc = np.concatenate([s.probs[:, m] for _, s in sp]) w = np.concatenate([s.weight for _, s in sp]) series = self.class_series(sc, lab, w, {"ls": SPLIT_STYLE[name]}) series = [(f"{c} ({name})", v, ww, st) for c, v, ww, st in series] - hist_panel(a, series, bins, f"P({self.classes[1]})") - hist_panel(b, series, bins, f"P({self.classes[1]})", logy=True) + hist_panel(a, series, bins, f"P({self.classes[m]})") + hist_panel(b, series, bins, f"P({self.classes[m]})", logy=True) a.legend(frameon=False) a.set_title("(a) linear", loc="left") b.set_title("(b) log", loc="left") diff --git a/src/corenet/train.py b/src/corenet/train.py index 93717b6..278037e 100644 --- a/src/corenet/train.py +++ b/src/corenet/train.py @@ -18,16 +18,16 @@ * per-class weight normalisation (``class_norm``): the event weights of every class sum to 1 over the training set; weights are clamped at >= 0 inside the loss. -Loss = weighted cross entropy over ``spec.classes`` (0 = rest, 1 = first output, 2 = second, ...) on -the main logits, plus, for every further output ``o``, ``head_weights.get(o, 1.0)`` times a weighted +Loss = weighted cross entropy over ``spec.classes`` (0 = rest, 1 = first output, 2 = second, ...; or the +``labels:`` list of the configuration in label order) on the main logits, plus, for every further output ``o``, ``head_weights.get(o, 1.0)`` times a weighted binary cross entropy of its head against ``label == class index of o``. The view (candidate selection) scores are not supervised. Folds: ``EVENT/fold`` of the file (ids exactly 0..K-1), or ``--n-folds K`` for fold = squares(event) % K from the event numbers (``corenet.data.folds.fold_of``, seed ``--fold-seed``, 0 = the rule of the example files); ``--exclude-folds next`` is the fold (k + 1) % K of held-out fold k. Training needs one -of the two. ``--early-stop-patience`` needs a validation set with events of class 1 and of another -class; an empty validation set or a missing validation AUC stops the run with an error. +of the two. ``--early-stop-patience`` needs a validation set with events of the main class (``spec.main_class``, +class 1 without ``labels:``) and of another class; an empty validation set or a missing validation AUC stops the run with an error. For each held-out fold k this writes, in ``--output-dir``: model_fold{k}.pkl checkpoint: state dict, configuration, classes and metadata @@ -78,6 +78,8 @@ "finetune_layers": list(HEAD_PREFIXES), "finetune_lr": None, "finetune_batch_size": None, + "finetune_reset_optimizer": False, + "masked_classes": {}, } @@ -155,6 +157,18 @@ def freeze_heads(model: CoReNet, prefixes=HEAD_PREFIXES) -> int: return n +def _reset_optimizer(model: torch.nn.Module, lr: float, milestones, gamma: float): + """(optimizer, scheduler) of a fresh fine-tuning stage: a new Adam over the trainable parameters at + learning rate ``lr`` and a new MultiStepLR whose milestones count from the reset.""" + optimizer = torch.optim.Adam( + [p for p in model.parameters() if p.requires_grad], lr=lr + ) + scheduler = torch.optim.lr_scheduler.MultiStepLR( + optimizer, milestones=list(milestones), gamma=gamma + ) # milestones counted from the reset, as a fresh stage would + return optimizer, scheduler + + def count_trainable(model: torch.nn.Module) -> int: return int(sum(p.numel() for p in model.parameters() if p.requires_grad)) @@ -176,6 +190,47 @@ def parse_finetune_layers(value) -> list[str]: return list(value) +def parse_masked_classes(entries: list[str]) -> dict[str, list[str]]: + """``--masked-classes FLAG=CLASS[,CLASS...] ...`` as the ``masked_classes`` mapping.""" + out: dict[str, list[str]] = {} + for entry in entries: + flag, sep, classes = entry.partition("=") + names = classes.split(",") + if not sep or not flag or not all(names): + raise ConfigError( + f"--masked-classes: expected FLAG=CLASS[,CLASS...], got {entry!r}" + ) + if flag in out: + raise ConfigError(f"--masked-classes: flag {flag!r} given twice") + out[flag] = names + return out + + +def check_masked_classes(masked_classes: dict, spec: NetworkSpec) -> None: + """Every masked class is one of ``spec.classes`` and at least one class stays unmasked per flag.""" + for flag, classes in masked_classes.items(): + unknown = [c for c in classes if c not in spec.classes] + if unknown: + raise ConfigError( + f"trainer.masked_classes.{flag}: unknown classes {unknown} (classes {spec.classes})" + ) + if len(classes) >= len(spec.classes): + raise ConfigError( + f"trainer.masked_classes.{flag}: masks every class {spec.classes}; at least one must stay" + ) + + +def masked_cross_entropy( + logits: torch.Tensor, label: torch.Tensor, class_mask: torch.Tensor | None = None +) -> torch.Tensor: + """Per-event cross entropy; ``class_mask`` (events, classes) True removes that class from the + event's softmax (its logit is set to -inf: the softmax renormalises over the other classes and the + removed logit gets zero gradient).""" + if class_mask is not None: + logits = logits.masked_fill(class_mask, float("-inf")) + return F.cross_entropy(logits, label, reduction="none") + + def normalize_class_weights( labels: np.ndarray, weights: torch.Tensor, idx: np.ndarray, n_classes: int ) -> torch.Tensor: @@ -189,10 +244,12 @@ def normalize_class_weights( return w -def weighted_auc(label: np.ndarray, score: np.ndarray, weight: np.ndarray) -> float: - """ROC AUC (trapezoidal, weighted) of class 1 against everything else, without scikit-learn.""" +def weighted_auc( + label: np.ndarray, score: np.ndarray, weight: np.ndarray, positive: int +) -> float: + """ROC AUC (trapezoidal, weighted) of class ``positive`` against everything else, without scikit-learn.""" order = np.argsort(-score, kind="stable") - y, w = label[order] == 1, np.clip(weight[order], 0, None) + y, w = label[order] == positive, np.clip(weight[order], 0, None) tps, fps = np.cumsum(w * y), np.cumsum(w * ~y) if tps[-1] <= 0 or fps[-1] <= 0: return float("nan") @@ -203,8 +260,8 @@ def weighted_auc(label: np.ndarray, score: np.ndarray, weight: np.ndarray) -> fl def aux_targets(spec: NetworkSpec) -> dict[str, int]: - """Auxiliary head name -> the class label it discriminates (classes = ['rest'] + outputs).""" - return {o.name: i + 1 for i, o in enumerate(spec.outputs) if i > 0} + """Auxiliary head name -> the class label it discriminates (its index in ``spec.classes``).""" + return {o.name: spec.classes.index(o.name) for o in spec.outputs[1:]} def validation_metrics( @@ -213,8 +270,13 @@ def validation_metrics( val_idx: np.ndarray, labels: np.ndarray, weights: torch.Tensor, + class_mask: np.ndarray | None = None, ) -> dict: - """Weighted cross entropy and signal-vs-rest AUC on the validation fold(s). + """Weighted cross entropy and main-class-vs-rest AUC on the validation fold(s). + + ``class_mask`` (len(val_idx), classes), True = class removed from the event's softmax + (``masked_classes``): the loss uses the probabilities with those classes set to 0 and the row + renormalised, as the training loss; the AUC uses the full probabilities. Side-effect free: the torch RNG state is saved and restored so the training stream is identical to a run without validation; the caller restores ghost batches. @@ -227,13 +289,20 @@ def validation_metrics( model.train() label = labels[val_idx] w = np.clip(weights.numpy()[val_idx], 0, None) - p = np.clip(probs[np.arange(len(label)), label], 1e-12, None) + p_loss = probs + if class_mask is not None: + rows = class_mask.any(1) + p_loss = probs.copy() + kept = np.where(class_mask[rows], 0.0, probs[rows]) + p_loss[rows] = kept / kept.sum(1, keepdims=True) + p = np.clip(p_loss[np.arange(len(label)), label], 1e-12, None) out = { "val_events": len(label), "val_loss": float(-(w * np.log(p)).sum() / max(w.sum(), 1e-12)), } - if 0 < int((label == 1).sum()) < len(label): - out["val_auc"] = weighted_auc(label, probs[:, 1], w) + main = model.spec.main_class + if 0 < int((label == main).sum()) < len(label): + out["val_auc"] = weighted_auc(label, probs[:, main], w, main) return out @@ -244,11 +313,14 @@ def loss_of_batch( w: torch.Tensor, aux: dict[str, int], head_weights: dict[str, float], + class_mask: torch.Tensor | None = None, ) -> torch.Tensor: - """Weighted CE on the class logits + sum_o head_weight[o] * weighted BCE of the auxiliary heads.""" + """Weighted CE on the class logits + sum_o head_weight[o] * weighted BCE of the auxiliary heads. + ``class_mask``: classes removed from the softmax per event (``masked_cross_entropy``).""" out = model(batch) w_sum = w.sum() - loss = (F.cross_entropy(out["logits"], label, reduction="none") * w).sum() / w_sum + logits = out["logits"] + loss = (masked_cross_entropy(logits, label, class_mask) * w).sum() / w_sum for name, cls in aux.items(): target = (label == cls).to(out["heads"][name].dtype) bce = F.binary_cross_entropy_with_logits( @@ -302,6 +374,11 @@ def train_fold( data = Batches(ds, device) w_dev = weights.to(device) aux, head_weights = aux_targets(spec), dict(spec.training.head_weights or {}) + val_class_mask = data.class_mask( + torch.as_tensor(val_idx, device=device), args.masked_classes, n_classes + ) + if val_class_mask is not None: + val_class_mask = val_class_mask.cpu().numpy() model.train() with torch.no_grad(): @@ -353,6 +430,15 @@ def metadata(epoch: int) -> dict: n_trainable = freeze_heads(model, args.finetune_layers) else: n_trainable = count_trainable(model) + if args.finetune_reset_optimizer: + lr_now = ( + args.finetune_lr + if args.finetune_lr is not None + else optimizer.param_groups[0]["lr"] + ) + optimizer, scheduler = _reset_optimizer( + model, lr_now, args.lr_milestones, args.lr_gamma + ) if args.finetune_lr is not None: for group in optimizer.param_groups: group["lr"] = args.finetune_lr @@ -365,6 +451,7 @@ def metadata(epoch: int) -> dict: ) + f", {n_trainable:,} trainable parameters, lr {optimizer.param_groups[0]['lr']:.2e}, " f"batch size {bs}, {n_gb} ghost batches" + + (", optimizer reset" if args.finetune_reset_optimizer else "") ) model.train() loss_sum, w_sum = 0.0, 0.0 @@ -373,7 +460,10 @@ def metadata(epoch: int) -> dict: w = w_dev[sel].clamp(min=0) if w.sum() <= 0: continue - loss = loss_of_batch(model, batch, data.labels[sel], w, aux, head_weights) + class_mask = data.class_mask(sel, args.masked_classes, n_classes) + loss = loss_of_batch( + model, batch, data.labels[sel], w, aux, head_weights, class_mask + ) optimizer.zero_grad() loss.backward() optimizer.step() @@ -396,7 +486,11 @@ def metadata(epoch: int) -> dict: "train_loss": loss_sum / max(w_sum, 1e-12), } if len(val_idx): - entry.update(validation_metrics(model, data, val_idx, labels, val_weights)) + entry.update( + validation_metrics( + model, data, val_idx, labels, val_weights, val_class_mask + ) + ) model.setGhostBatches(n_gb) # predict() set them to 0 history.append(entry) history_path.write_text( @@ -436,7 +530,7 @@ def metadata(epoch: int) -> dict: raise SystemExit( f"fold {fold}: epoch {epoch}: no validation AUC, so --early-stop-patience cannot be applied: " f"the validation folds {exclude_folds} hold {counts} events (positive-weight events of " - f"class 1 `{spec.classes[1]}` and of another class are needed)" + f"class {spec.main_class} `{spec.classes[spec.main_class]}` and of another class are needed)" ) if best_auc is None or auc > best_auc: best_auc, best_epoch, bad = auc, epoch, 0 @@ -566,6 +660,20 @@ def build_parser() -> argparse.ArgumentParser: type=int, help="batch size from --freeze-epoch on [keep the schedule]", ) + g.add_argument( + "--finetune-reset-optimizer", + action=argparse.BooleanOptionalAction, + default=None, + help="at --freeze-epoch, start a new optimizer and learning-rate schedule over the trainable " + "parameters [off]", + ) + g.add_argument( + "--masked-classes", + nargs="*", + metavar="FLAG=CLASS[,CLASS...]", + help="classes removed from the softmax for events with EVENT/FLAG true, e.g. SR=d4; " + "overrides the trainer section", + ) g.add_argument( "--lr", type=float, help=f"Adam learning rate [{RECIPE_DEFAULTS['lr']}]" ) @@ -646,6 +754,19 @@ def is_int(x): def is_num(x): return isinstance(x, (int, float)) and not isinstance(x, bool) + def is_masked_classes(x): + if not isinstance(x, dict): + return False + return all( + isinstance(k, str) + and k + and isinstance(v, list) + and v + and all(isinstance(c, str) for c in v) + and len(set(v)) == len(v) + for k, v in x.items() + ) + rules = { "epochs": (lambda x: is_int(x) and x >= 1, "an integer >= 1"), "freeze_epoch": (lambda x: is_int(x) and x >= 0, "an integer >= 0"), @@ -669,14 +790,20 @@ def is_num(x): lambda x: x is None or (is_int(x) and x >= 1), "an integer >= 1", ), + "finetune_reset_optimizer": (lambda x: isinstance(x, bool), "true or false"), + "masked_classes": ( + is_masked_classes, + "a mapping flag name -> non-empty list of distinct class names", + ), } for key, (ok, what) in rules.items(): if key in r and not ok(r[key]): raise ConfigError(f"trainer.{key} must be {what}, got {r[key]!r}") -def resolve_recipe(args, config: dict): - """Fill the recipe options of ``args``: command line > ``trainer:`` section > built-in default.""" +def resolve_recipe(args, config: dict, spec: NetworkSpec | None = None): + """Fill the recipe options of ``args``: command line > ``trainer:`` section > built-in default. + With ``spec``, the class names of ``masked_classes`` are checked against ``spec.classes``.""" section = config.get("trainer", {}) or {} if not isinstance(section, dict): raise ConfigError("trainer must be a mapping") @@ -685,11 +812,15 @@ def resolve_recipe(args, config: dict): raise ConfigError( f"trainer: unknown keys {sorted(unknown)} (known: {', '.join(RECIPE_DEFAULTS)})" ) + if args.masked_classes is not None: # command line: FLAG=CLASS[,CLASS...] + args.masked_classes = parse_masked_classes(args.masked_classes) for key, default in RECIPE_DEFAULTS.items(): if getattr(args, key) is None: setattr(args, key, section.get(key, default)) args.finetune_layers = parse_finetune_layers(args.finetune_layers) _check_recipe({k: getattr(args, k) for k in RECIPE_DEFAULTS}) + if spec is not None: + check_masked_classes(args.masked_classes, spec) return args @@ -700,7 +831,7 @@ def main(argv=None): ) config = read_config_dict(args.config) spec = parse_config(config) - resolve_recipe(args, config) + resolve_recipe(args, config, spec) if args.early_stop_patience and not args.exclude_folds: build_parser().error( "--early-stop-patience needs a validation fold: pass --exclude-folds" @@ -718,6 +849,7 @@ def main(argv=None): rest_order=args.rest_order, n_folds=args.n_folds, fold_seed=args.fold_seed, + masked_classes=args.masked_classes, ) if not ds.has_folds: raise SystemExit( diff --git a/src/corenet/validate.py b/src/corenet/validate.py index fb1d01a..ce335bc 100644 --- a/src/corenet/validate.py +++ b/src/corenet/validate.py @@ -9,7 +9,7 @@ built (candidate enumeration, relation restrictions) and summarised. 2. Data (with ``--input``): every dataset the loader reads is looked up with the loader's own rules (``corenet.data.h5``: group aliases, used collections, computed event scalars, label keys) and its - dtype and shape are checked; the labels must lie in 0..len(outputs), the weights must be finite, + dtype and shape are checked; the labels must lie in 0..len(classes) - 1, the weights must be finite, the folds must exist for training (``EVENT/fold`` with ids exactly 0..K-1, or ``--n-folds K`` for fold = squares(event) % K from the event numbers, seed ``--fold-seed``; the report gives the fold source and the events per fold); the present members @@ -108,7 +108,9 @@ def check_config(path, report: Report) -> NetworkSpec | None: from .train import build_parser, resolve_recipe args = resolve_recipe( - build_parser().parse_args(["--config", str(path), "--input", "-"]), cfg + build_parser().parse_args(["--config", str(path), "--input", "-"]), + cfg, + spec, ) if cfg.get("trainer"): report.ok( @@ -181,6 +183,7 @@ def check_data( n_forward: int = 512, n_folds: int | None = None, fold_seed: int = FOLD_SEED, + masked_classes: dict | None = None, ) -> None: sec = f"data {path}" try: @@ -358,11 +361,12 @@ def length(name, ds): ) elif length(key, f[key]): label = f[key][...].astype(np.int64) - bad = np.unique(label[(label < 0) | (label > len(spec.outputs))]) + top = len(spec.classes) - 1 + bad = np.unique(label[(label < 0) | (label > top)]) if len(bad): report.fail( sec, - f"{key}: labels {bad.tolist()} outside 0..{len(spec.outputs)} (classes {spec.classes})", + f"{key}: labels {bad.tolist()} outside 0..{top} (classes {spec.classes})", ) else: counts = { @@ -374,6 +378,18 @@ def length(name, ds): report.warn( sec, f"class `{c}` (label {i}) has no events in this file" ) + if masked_classes: + problems = loader.check_masked_classes(f, spec, masked_classes) + for p in problems: + report.fail(sec, p) + if not problems: + for flag, classes in masked_classes.items(): + n_flag = int(loader.read_flag(f, flag).sum()) + report.ok( + sec, + f"masked_classes: {loader.FLAG_GROUP}/{flag} boolean, {n_flag:,} flagged events, " + f"none of the excluded classes {classes}", + ) if loader.WEIGHT_KEY not in f: report.warn( sec, f"{loader.WEIGHT_KEY} not found: every event gets weight 1" @@ -480,7 +496,12 @@ def length(name, ds): # ---- the actual loader and a forward pass try: ds = loader.H5Dataset( - path, spec, rest_order=rest_order, n_folds=n_folds, fold_seed=fold_seed + path, + spec, + rest_order=rest_order, + n_folds=n_folds, + fold_seed=fold_seed, + masked_classes=masked_classes, ) except Exception as e: report.fail(sec, f"the loader (H5Dataset) fails: {type(e).__name__}: {e}") @@ -582,6 +603,14 @@ def validate( ) except Exception: rest_order = "pt" + try: # an invalid masked_classes is already reported by check_config + from .train import check_masked_classes + + trainer = read_config_dict(config).get("trainer") or {} + masked_classes = trainer.get("masked_classes") or {} + check_masked_classes(masked_classes, spec) + except Exception: + masked_classes = {} check_data( data, spec, @@ -589,6 +618,7 @@ def validate( rest_order=rest_order, n_folds=n_folds, fold_seed=fold_seed, + masked_classes=masked_classes, ) elif data is not None: report.fail(f"data {data}", "not checked: the configuration is invalid") diff --git a/tests/golden/bbww.npz b/tests/golden/bbww.npz new file mode 100644 index 0000000..e5464f7 Binary files /dev/null and b/tests/golden/bbww.npz differ diff --git a/tests/golden/hh4b.npz b/tests/golden/hh4b.npz new file mode 100644 index 0000000..6d396d8 Binary files /dev/null and b/tests/golden/hh4b.npz differ diff --git a/tests/reference/test_parity_reference.py b/tests/reference/test_parity_reference.py index c9a665a..f332296 100644 --- a/tests/reference/test_parity_reference.py +++ b/tests/reference/test_parity_reference.py @@ -9,23 +9,31 @@ CORENET_REFERENCE_REPO=/path/to/reference pytest tests/reference -s Checks, for the HH -> 4b and HH -> bbWW example configurations: - * the example YAML files describe the same network as the reference configurations; + * the example YAML files describe the same network as the reference configurations (sections other + than ``training`` as written, ``training`` after the defaults are filled in); for HH -> bbWW the + reference is the eta-reflection configuration ``configs/hcr/dscan/bbww_reflect_d8.yaml`` and its + trained checkpoint ``models/bbww_d8_e20bestreflect_r3_best`` (the paper's model); * with the same seed both code bases initialise identical state dicts (same module creation order); parameter counts 1,508 and 3,832; * the two loaders produce identical input tensors from the same HDF5 file; - * identical batches give bit-identical (torch.equal) logits, view logits, auxiliary heads and event - pixels in training mode (ghost batches, twice, so that the running statistics update) and in eval - mode, identical buffers, identical gradients, and identical parameters after 5 Adam steps; + * after updateMeanStd / initMeanStd on identical batches the frozen statistics are identical, except + those of the leaf embedders, which agree to float32 rounding (CoReNet accumulates all slots of one + leaf role in one node in one call); with the reference statistics copied, eval-mode logits, view + logits, auxiliary heads and event pixels are bit-identical (torch.equal). Training-mode outputs, + gradients and optimiser steps differ by design (the slots of one leaf role share the ghost-batch + statistics) and are pinned by the golden-file test tests/test_training_regression.py; * the trained reference checkpoints load strictly into CoReNet and give torch.equal outputs; * the primitives (GBN, conv1d, MinimalAttention, ReinforceLayer, ResNetBlock) are bit-identical; - * the two trainers, run for 2 epochs with the same arguments, produce bit-identical weights and - predictions; + * the two trainers, run for 2 epochs with the same arguments, write checkpoints with the same + state-dict keys, shapes and integer buffers, predict the same events in the same order and follow + the same learning-rate and batch-size schedule (the trained weights differ by design, see above); * CoReNet with the 4b configuration reproduces the reference repository's original hard-coded 4b network (``hcr.models.HCR.HCR``) loaded from its trained checkpoint, torch.equal. """ from __future__ import annotations +import dataclasses import os import sys from pathlib import Path @@ -63,15 +71,15 @@ ), "bbww": dict( config=repo / "examples/bbww/bbww.yaml", - ref_config=REF / "configs/hcr/bbww.yaml", - # models/bbww_d8_r3: trained after W_lep became a plain [lepton, met] sum (models/bbww_d8_r2 predates it) + # the eta-reflection configuration of the paper (configs/hcr/bbww.yaml has reflect_eta: false) + ref_config=REF / "configs/hcr/dscan/bbww_reflect_d8.yaml", data=REF / "data/h5/bbww_all.h5", params=3832, - ckpt=REF / "models/bbww_d8_r3/model_fold0.pkl", + # the paper's bbWW model (best-validation epoch, fold 0), trained with bbww_reflect_d8.yaml + ckpt=REF / "models/bbww_d8_e20bestreflect_r3_best/model_fold0.pkl", ), } N_EVENTS = 4096 -GHOST_BATCHES = 16 def _cfg(topology): @@ -136,9 +144,17 @@ def _same_state(ref, new, what): @pytest.mark.parametrize("topology", TOPOLOGIES) def test_example_config_matches_reference(topology): cfg, ref_cfg = _cfg(topology) - assert cfg == ref_cfg, ( + + def rest(c): + return {k: v for k, v in c.items() if k != "training"} + + assert rest(cfg) == rest(ref_cfg), ( f"{topology}: example configuration differs from the reference" ) + # training options after the defaults are filled in: an omitted key equal to its default passes + new_t = dataclasses.asdict(parse_config(cfg).training) + ref_t = dataclasses.asdict(ref_parse(ref_cfg).training) + assert new_t == ref_t, f"{topology}: training options {new_t} vs {ref_t}" @pytest.mark.parametrize("topology", TOPOLOGIES) @@ -174,69 +190,57 @@ def test_loader_parity(topology): ) -# ----------------------------------------------------------------------------------- forward / backward +# ----------------------------------------------------------------------------------- forward (eval mode) @pytest.mark.parametrize("topology", TOPOLOGIES) -def test_forward_backward_parity(topology): +def test_eval_forward_parity(topology): + """Frozen statistics and eval-mode outputs. Training-mode outputs, gradients and optimiser steps are + not compared: CoReNet embeds all slots of one leaf role in one node in one call (they share the + ghost-batch statistics), the research code embedded every slot on its own; + tests/test_training_regression.py pins them instead.""" ref_ds, new_ds, idx = _data(topology) ref, new = _models(topology, seed=3) new.load_state_dict( ref.state_dict(), strict=True ) # explicit copy (no key renaming needed) - x_ref, x_new = _batch(ref_ds, idx), _batch(new_ds, idx) ref.train() new.train() - ref.updateMeanStd(x_ref) - new.updateMeanStd(x_new) + ref.updateMeanStd(_batch(ref_ds, idx)) + new.updateMeanStd(_batch(new_ds, idx)) ref.initMeanStd() new.initMeanStd() - _same_state(ref, new, f"{topology} after initMeanStd") - ref.setGhostBatches(GHOST_BATCHES) - new.setGhostBatches(GHOST_BATCHES) - for mode in ("train", "train", "eval"): - getattr(ref, mode)() - getattr(new, mode)() + # frozen statistics: identical, except those of the leaf embedders, which CoReNet accumulates over all + # slots of a role in one call (the same pixels summed in another order: float32 rounding only) + rs, ns = ref.state_dict(), new.state_dict() + assert list(rs) == list(ns), f"{topology}: state-dict keys differ" + rounded = [] + for k in rs: + if torch.equal(rs[k], ns[k]): + continue + assert k.startswith("leaf_embed.") and k.endswith((".m", ".s")), ( + f"{topology} after initMeanStd: {k} differs" + ) + torch.testing.assert_close( + ns[k], rs[k], rtol=1e-5, atol=1e-6, msg=f"{topology} after initMeanStd: {k}" + ) + rounded.append(k) + new.load_state_dict( + rs, strict=True + ) # the reference statistics in both from here on + ref.setGhostBatches(0) + new.setGhostBatches(0) + ref.eval() + new.eval() + for _ in range(2): with torch.no_grad(): _same_tree( ref(_batch(ref_ds, idx)), new(_batch(new_ds, idx)), - f"{topology} [{mode}] outputs", + f"{topology} [eval] outputs", ) - _same_state(ref, new, f"{topology} [{mode}] buffers") - # gradients and 5 optimiser steps, training mode - ref.train() - new.train() - opt_r = torch.optim.Adam(ref.parameters(), lr=1e-2) - opt_n = torch.optim.Adam(new.parameters(), lr=1e-2) - label = ref_ds.labels[idx] - for step in range(5): - losses = [] - for m, opt, ds in ((ref, opt_r, ref_ds), (new, opt_n, new_ds)): - out = m(_batch(ds, idx)) - loss = torch.nn.functional.cross_entropy(out["logits"], label) - for h in out["heads"].values(): - loss = loss + h.pow(2).mean() - opt.zero_grad() - loss.backward() - opt.step() - losses.append(loss.detach()) - assert torch.equal(losses[0], losses[1]), ( - f"{topology} step {step}: loss {losses[0]} vs {losses[1]}" - ) - for (k, pr), (_, pn) in zip(ref.named_parameters(), new.named_parameters()): - assert (pr.grad is None) == (pn.grad is None), k - if pr.grad is not None: - assert torch.equal(pr.grad, pn.grad), ( - f"{topology} step {step}: gradient of {k} differs" - ) - _same_state(ref, new, f"{topology} after 5 Adam steps") - ref.eval() - new.eval() - with torch.no_grad(): - o_ref, o_new = ref(_batch(ref_ds, idx)), new(_batch(new_ds, idx)) - _same_tree(o_ref, o_new, f"{topology} outputs after training steps") + _same_state(ref, new, f"{topology} [eval] buffers") print( - f" {topology}: {N_EVENTS} events, train x2 / eval / 5 Adam steps: logits, views, heads, event pixel, " - f"buffers, gradients and weights torch.equal" + f" {topology}: {N_EVENTS} events: frozen statistics identical except {rounded} (float32 " + f"rounding); eval-mode logits, views, heads, event pixel and buffers torch.equal" ) @@ -458,27 +462,36 @@ def test_trainer_parity(topology, tmp_path): n = torch.load( tmp_path / "new/model_fold0.pkl", map_location="cpu", weights_only=False )["model"] + # trained weights, predictions and losses differ by design (grouped leaf embedding, see + # test_eval_forward_parity); tests/test_training_regression.py pins the training numerics assert list(r) == list(n) for k in r: - assert torch.equal(r[k], n[k]), f"{topology}: trained {k} differs" + assert r[k].shape == n[k].shape and r[k].dtype == n[k].dtype, ( + f"{topology}: trained {k}: shape/dtype {tuple(r[k].shape)} {r[k].dtype} vs " + f"{tuple(n[k].shape)} {n[k].dtype}" + ) + if not torch.is_floating_point(r[k]): + assert torch.equal(r[k], n[k]), f"{topology}: integer buffer {k} differs" with ( h5py.File(tmp_path / "ref/predictions_fold0.h5") as fr, h5py.File(tmp_path / "new/predictions_fold0.h5") as fn, ): - assert np.array_equal( - fr["CLASSIFICATIONS/EVENT/signal"][...], fn["class_probabilities"][...] + assert ( + fr["CLASSIFICATIONS/EVENT/signal"].shape == fn["class_probabilities"].shape ) assert np.array_equal(fr["EVENT/event"][...], fn["EVENT/event"][...]) import json hr = json.loads((tmp_path / "ref/history_fold0.json").read_text())["history"] hn = json.loads((tmp_path / "new/history_fold0.json").read_text())["history"] + assert len(hr) == len(hn) == 2, (len(hr), len(hn)) for a, b in zip(hr, hn): - for key in ("train_loss", "val_loss", "val_auc", "lr", "batch_size"): + for key in ("lr", "batch_size"): assert a[key] == b[key], (key, a[key], b[key]) print( - f" {topology}: 2-epoch training on {small.name}: weights, predictions and history identical " - f"(train loss {[round(h['train_loss'], 6) for h in hn]})" + f" {topology}: 2-epoch training on {small.name}: same state-dict keys, shapes and integer " + f"buffers, same predicted events, same lr / batch-size schedule; train loss (differs by design) " + f"reference {[round(h['train_loss'], 6) for h in hr]}, CoReNet {[round(h['train_loss'], 6) for h in hn]}" ) diff --git a/tests/synthetic.py b/tests/synthetic.py index 8175a0b..91d6748 100644 --- a/tests/synthetic.py +++ b/tests/synthetic.py @@ -30,9 +30,18 @@ def _jets(rng, n, n_slots, n_present, boost): def write_synthetic( - path: Path, topology: str, seed: int = 0, n: int = N_EVENTS + path: Path, + topology: str, + seed: int = 0, + n: int = N_EVENTS, + flag: str | None = None, + masked_label: int | None = None, + n_violations: int = 0, ) -> Path: - """A small file in the layout of examples/data/{hh4b,bbww}_small.h5.""" + """A small file in the layout of examples/data/{hh4b,bbww}_small.h5. ``flag``: also write a boolean + ``EVENT/`` (20% of the events true, from a separate random stream so the other datasets are + unchanged); ``masked_label``: no flagged event carries this label, apart from ``n_violations`` + events of that label forced to true (a blinding error).""" rng = np.random.default_rng(seed) if topology == "hh4b": label = (rng.uniform(size=n) < 0.3).astype(np.int64) # 0 rest/QCD, 1 signal @@ -82,4 +91,11 @@ def write_synthetic( for c in np.unique(label): # per-class sum = 1, as the real files w[label == c] /= float((label == c).sum()) f.create_dataset("EVENT/weight", data=w) + if flag is not None: + frng = np.random.default_rng(seed + 1000) + flagged = frng.uniform(size=n) < 0.2 + if masked_label is not None: + flagged &= label != masked_label + flagged[np.flatnonzero(label == masked_label)[:n_violations]] = True + f.create_dataset(f"EVENT/{flag}", data=flagged) return path diff --git a/tests/test_config.py b/tests/test_config.py index 7c507d4..8272c07 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -275,3 +275,38 @@ def test_tth_example_parses(): assert model.tables["ttH"].nodes["ttH"].n == 12 n_par = sum(p.numel() for p in model.parameters()) assert n_par == 3543, n_par + + +def test_labels(): + """``labels:`` (optional): the classes in label order; every output must be a label; main_class is + the class of the first output; without it the classes are rest + outputs and main_class is 1.""" + s = parse_config(copy.deepcopy(BBWW)) + assert s.classes == ["rest", "signal", "ttbar"] and s.main_class == 1 + c = copy.deepcopy(BBWW) + c["labels"] = ["ttbar", "multijet", "signal"] + s = parse_config(c) + assert s.classes == ["ttbar", "multijet", "signal"] and s.main_class == 2 + model = CoReNet(s, device="cpu") + assert model.nC == 3 and set(model.aux_heads) == {"ttbar"} + c = copy.deepcopy(HH4B) + c["labels"] = ["d4", "d3", "t4", "t3"] + c["outputs"] = {"d4": "HH"} + s = parse_config(c) + assert s.classes == ["d4", "d3", "t4", "t3"] and s.main_class == 0 + model = CoReNet(s, device="cpu") + assert model.nC == 4 and len(model.aux_heads) == 0 + + +def test_labels_rejected(): + c = copy.deepcopy(BBWW) + c["labels"] = ["multijet", "signal"] # ttbar is an output but not a label + _fails(c, "outputs.ttbar: the output name must be one of the labels") + c = copy.deepcopy(HH4B) + c["labels"] = ["signal"] + _fails(c, "labels: at least two classes are needed") + c = copy.deepcopy(HH4B) + c["labels"] = ["signal", "rest", "signal"] + _fails(c, "labels: duplicate entries ['signal']") + c = copy.deepcopy(HH4B) + c["labels"] = ["signal", 3] + _fails(c, "labels: expected a list of names") diff --git a/tests/test_labels.py b/tests/test_labels.py new file mode 100644 index 0000000..7ac4619 --- /dev/null +++ b/tests/test_labels.py @@ -0,0 +1,93 @@ +"""``labels:``: a network with N classes named in label order, trained and evaluated on the CPU on a small +synthetic file with labels 0, 1, 2 (``labels: [a, b, c]``, ``outputs: {b: HH}``): the checkpoint and the +prediction file carry the labels as classes, and the main AUC is that of class 1 = ``b``.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import h5py +import numpy as np +import torch +import yaml +from synthetic import write_synthetic + +from corenet import evaluate, train + +repo = Path(__file__).resolve().parent.parent +HH4B = repo / "examples/hh4b/hh4b.yaml" + + +def test_labels_train_and_evaluate(tmp_path: Path) -> None: + data = write_synthetic(tmp_path / "abc.h5", "hh4b", seed=3, n=600) + with h5py.File(data, "r+") as f: + label = f["EVENT/label"][...] + rng = np.random.default_rng(5) + label = np.where( + label == 0, np.where(rng.uniform(size=len(label)) < 0.5, 0, 2), 1 + ) # a = 0, b = 1 (the boosted signal), c = 2 + f["EVENT/label"][...] = label + assert set(np.unique(label)) == {0, 1, 2} + + cfg = yaml.safe_load(HH4B.read_text()) + cfg["labels"] = ["a", "b", "c"] + cfg["outputs"] = {"b": "HH"} + cfg.pop("trainer", None) + config = tmp_path / "abc.yaml" + config.write_text(yaml.safe_dump(cfg)) + + out_dir = tmp_path / "models" + train.main( + [ + "--config", + str(config), + "--input", + str(data), + "--fold", + "0", + "--exclude-folds", + "1", + "--output-dir", + str(out_dir), + "--epochs", + "1", + "--batch-size-init", + "128", + "--freeze-epoch", + "1", + "--device", + "cpu", + ] + ) + ckpt = torch.load( + out_dir / "model_fold0.pkl", map_location="cpu", weights_only=False + ) + assert ckpt["classes"] == ["a", "b", "c"] + with h5py.File(out_dir / "predictions_fold0.h5", "r") as f: + assert f["class_probabilities"].shape[1] == 3 + assert json.loads(f.attrs["classes"]) == ["a", "b", "c"] + assert "heads" not in f or len(f["heads"]) == 0 + + out = tmp_path / "eval.h5" + metrics = evaluate.main( + [ + "--checkpoint", + str(out_dir / "model_fold0.pkl"), + "--input", + str(data), + "--output", + str(out), + ] + ) + from sklearn.metrics import roc_auc_score + + with h5py.File(out, "r") as f: + probs = f["class_probabilities"][...] + lab = f["EVENT/label"][...] + w = np.clip(f["EVENT/weight"][...], 0, None) + assert probs.shape[1] == 3 + expected = roc_auc_score(lab == 1, probs[:, 1], sample_weight=w) + assert metrics["auc"]["main"] == expected == metrics["auc"]["one_vs_rest"]["b"] + saved = json.loads(Path(str(out) + ".json").read_text()) + assert saved["auc"]["main"] == expected diff --git a/tests/test_network_grouped.py b/tests/test_network_grouped.py new file mode 100644 index 0000000..9f6d78b --- /dev/null +++ b/tests/test_network_grouped.py @@ -0,0 +1,84 @@ +"""Grouped leaf embedding: all child slots of one leaf role in one node are embedded in a single call +(candidate-major, slot-minor), so in training mode they share the ghost-batch statistics. In eval mode +(running statistics, 0 ghost batches) the embedded pixel of a slot must not depend on which other slots +are in the call: the grouped call, split back per slot, equals the per-slot call exactly. This is why +stored checkpoints evaluate as before the grouping.""" + +from __future__ import annotations + +from pathlib import Path + +import torch +from synthetic import write_synthetic + +from corenet.data.h5 import H5Dataset +from corenet.models.config import load_config +from corenet.models.network import CoReNet + +repo = Path(__file__).resolve().parent.parent + + +def _model_and_inputs(tmp_path: Path, topology: str): + data = write_synthetic(tmp_path / f"{topology}.h5", topology, n=400) + torch.manual_seed(0) + model = CoReNet(load_config(repo / f"examples/{topology}/{topology}.yaml"), "cpu") + ds = H5Dataset(data, model.spec) + model.train() + model.updateMeanStd(ds.inputs) + model.initMeanStd() + model.eval() + model.setGhostBatches(0) + return model, ds.inputs + + +def _check_grouping_invariance(model: CoReNet, inputs) -> int: + checked = 0 + with torch.no_grad(): + prep = model._prepare(inputs) + n = prep["n"] + a = model.event_embed(prep["event"]) if model.event_embed is not None else None + for h, T in model.tables.items(): + hp = prep["hyp"][h] + for node in model.levels[h]: + nt = T.nodes[node] + for role, js in model._leaf_groups(nt).items(): + feats, pres = model._grouped_slots(hp, node, js) + assert feats.shape[2] == nt.n * len(js) + grouped = model.leaf_embed[role](feats, add=a) + grouped = grouped.view(n, model.d, nt.n, len(js)) + for i, j in enumerate(js): + alone = model.leaf_embed[role](hp["slot_feats"][node][j], add=a) + assert torch.equal(grouped[:, :, :, i], alone), ( + h, + node, + role, + j, + ) + # the presence is laid out the same way + assert torch.equal( + pres.view(n, nt.n, len(js))[:, :, i], + hp["slot_present"][node][j], + ) + checked += 1 + return checked + + +def test_grouped_embedding_eval_invariance_hh4b(tmp_path): + model, inputs = _model_and_inputs(tmp_path, "hh4b") + nt = model.tables["HH"].nodes[model.levels["HH"][0]] + assert model._leaf_groups(nt) == {"b": [0, 1]} # a dijet: both slots in one call + assert _check_grouping_invariance(model, inputs) >= 2 + + +def test_grouped_embedding_eval_invariance_bbww(tmp_path): + model, inputs = _model_and_inputs(tmp_path, "bbww") + groups = { + node: model._leaf_groups(T.nodes[node]) + for h, T in model.tables.items() + for node in model.levels[h] + } + assert groups["W_lep"] == { + "lepton": [0], + "met": [1], + } # different roles: separate calls + assert _check_grouping_invariance(model, inputs) > 0 diff --git a/tests/test_recipe_options.py b/tests/test_recipe_options.py new file mode 100644 index 0000000..0a232c9 --- /dev/null +++ b/tests/test_recipe_options.py @@ -0,0 +1,334 @@ +"""Training-recipe options that default to off, on small synthetic files, on the CPU: + +1. finetune_reset_optimizer: a new optimizer and learning-rate schedule at the freeze epoch; +2. masked_classes: classes removed from the softmax for flagged events (loss, validation loss, + loader and validator checks); the evaluation keeps every class. +""" + +from __future__ import annotations + +import io +import json +import tempfile +from pathlib import Path + +import h5py +import numpy as np +import pytest +import torch +import torch.nn.functional as F +import yaml +from synthetic import write_synthetic + +from corenet import evaluate, train, validate +from corenet.data.h5 import H5Dataset +from corenet.evaluate import Batches +from corenet.models.config import ConfigError, load_config, parse_config +from corenet.models.network import CoReNet + +repo = Path(__file__).resolve().parent.parent +HH4B = repo / "examples/hh4b/hh4b.yaml" + + +def _train(data: Path, out_dir: Path, *extra: str, epochs: int = 3) -> dict: + """corenet-train on the synthetic hh4b file: fold 0 held out, fold 1 validation, freeze at epoch 2.""" + train.main( + [ + "--config", + str(HH4B), + "--input", + str(data), + "--output-dir", + str(out_dir), + "--fold", + "0", + "--exclude-folds", + "1", + "--epochs", + str(epochs), + "--batch-size-init", + "256", + "--freeze-epoch", + "2", + "--seed", + "7", + "--device", + "cpu", + *extra, + ] + ) + return json.loads((out_dir / "history_fold0.json").read_text()) + + +def _state(out_dir: Path) -> dict: + return torch.load( + out_dir / "model_fold0.pkl", map_location="cpu", weights_only=False + )["model"] + + +def test_finetune_reset_optimizer_training() -> None: + """Off and on with the same seed: both run, the history records the option, the weights differ.""" + with tempfile.TemporaryDirectory(prefix="corenet_reset_") as tmp: + tmp = Path(tmp) + data = write_synthetic(tmp / "hh4b_all.h5", "hh4b") + off = _train(data, tmp / "off") + on = _train(data, tmp / "on", "--finetune-reset-optimizer") + assert off["recipe"]["finetune_reset_optimizer"] is False + assert on["recipe"]["finetune_reset_optimizer"] is True + assert len(off["history"]) == len(on["history"]) == 3 + assert off["history"][:2] == on["history"][:2] # identical before the freeze + key = "head_main.conv.module.weight" + assert not torch.equal(_state(tmp / "off")[key], _state(tmp / "on")[key]) + + +def test_reset_optimizer_helper() -> None: + """The fresh optimizer has no state, the requested lr and exactly the trainable parameters.""" + model = CoReNet(load_config(HH4B), device="cpu") + train.freeze_heads(model) + opt, sched = train._reset_optimizer(model, 1e-3, [1, 2], 0.5) + assert len(opt.state) == 0 + assert opt.param_groups[0]["lr"] == 1e-3 + trainable = [p for p in model.parameters() if p.requires_grad] + params = opt.param_groups[0]["params"] + assert len(params) == len(trainable) + assert all(a is b for a, b in zip(params, trainable)) + assert sorted(sched.milestones) == [1, 2] and sched.gamma == 0.5 + + +def test_finetune_reset_optimizer_config() -> None: + cfg = yaml.safe_load(HH4B.read_text()) + cfg["trainer"] = {"finetune_reset_optimizer": True} + args = train.build_parser().parse_args(["--config", "x", "--input", "y"]) + assert train.resolve_recipe(args, cfg).finetune_reset_optimizer is True + args = train.build_parser().parse_args( + ["--config", "x", "--input", "y", "--no-finetune-reset-optimizer"] + ) + assert train.resolve_recipe(args, cfg).finetune_reset_optimizer is False + cfg["trainer"] = {"finetune_reset_optimizer": "yes"} + with pytest.raises( + ConfigError, match="finetune_reset_optimizer must be true or false" + ): + train.resolve_recipe( + train.build_parser().parse_args(["--config", "x", "--input", "y"]), cfg + ) + + +# ----------------------------------------------------------------------------------- masked_classes +def _abc_file(tmp: Path, n_violations: int = 0) -> Path: + """Synthetic file with labels a = 0, b = 1 (the boosted signal), c = 2 and a flag EVENT/SR that + avoids class b, apart from ``n_violations`` flagged events of class b.""" + data = write_synthetic( + tmp / "abc.h5", + "hh4b", + seed=3, + n=600, + flag="SR", + masked_label=1, + n_violations=n_violations, + ) + with h5py.File(data, "r+") as f: + label = f["EVENT/label"][...] + rng = np.random.default_rng(5) + f["EVENT/label"][...] = np.where( + label == 0, np.where(rng.uniform(size=len(label)) < 0.5, 0, 2), 1 + ) + return data + + +def _abc_config(tmp: Path, masked_classes=None) -> Path: + cfg = yaml.safe_load(HH4B.read_text()) + cfg["labels"] = ["a", "b", "c"] + cfg["outputs"] = {"b": "HH"} + cfg.pop("trainer", None) + if masked_classes is not None: + cfg["trainer"] = {"masked_classes": masked_classes} + path = tmp / "abc.yaml" + path.write_text(yaml.safe_dump(cfg)) + return path + + +def test_masked_cross_entropy_equals_column_deletion() -> None: + """-inf logits = deleting the column for the flagged rows; zero gradient on the removed logits.""" + gen = torch.Generator().manual_seed(0) + n, n_classes, excluded = 64, 4, 2 + logits = torch.randn(n, n_classes, generator=gen, dtype=torch.float64) + logits.requires_grad_(True) + flagged = torch.rand(n, generator=gen) < 0.4 + label = torch.randint(0, n_classes, (n,), generator=gen) + kept = torch.tensor([0, 1, 3]) + label[flagged] = kept[torch.randint(0, 3, (int(flagged.sum()),), generator=gen)] + w = torch.rand(n, generator=gen, dtype=torch.float64) + mask = torch.zeros(n, n_classes, dtype=torch.bool) + mask[flagged, excluded] = True + + masked = train.masked_cross_entropy(logits, label, mask) + explicit = torch.empty(n, dtype=torch.float64) + explicit[~flagged] = F.cross_entropy( + logits[~flagged], label[~flagged], reduction="none" + ) + shifted = label[flagged] - (label[flagged] > excluded).long() + explicit[flagged] = F.cross_entropy( + logits[flagged][:, kept], shifted, reduction="none" + ) + assert torch.allclose(masked, explicit, atol=1e-6, rtol=0) + loss = (masked * w).sum() / w.sum() + assert torch.allclose(loss, (explicit * w).sum() / w.sum(), atol=1e-6, rtol=0) + loss.backward() + zero = torch.zeros(int(flagged.sum()), dtype=torch.float64) + assert torch.equal(logits.grad[flagged, excluded], zero) + assert (logits.grad[~flagged, excluded] != 0).all() + plain = train.masked_cross_entropy(logits, label, None) + assert torch.equal(plain, F.cross_entropy(logits, label, reduction="none")) + + +def test_masked_classes_training(tmp_path: Path) -> None: + """1 epoch with masked_classes {SR: [b]}: the history records it; the evaluation is untouched.""" + data = _abc_file(tmp_path) + config = _abc_config(tmp_path, {"SR": ["b"]}) + out_dir = tmp_path / "models" + train.main( + [ + "--config", + str(config), + "--input", + str(data), + "--fold", + "0", + "--exclude-folds", + "1", + "--output-dir", + str(out_dir), + "--epochs", + "1", + "--batch-size-init", + "128", + "--freeze-epoch", + "1", + "--device", + "cpu", + ] + ) + hist = json.loads((out_dir / "history_fold0.json").read_text()) + assert hist["recipe"]["masked_classes"] == {"SR": ["b"]} + assert np.isfinite(hist["history"][0]["train_loss"]) + assert np.isfinite(hist["history"][0]["val_loss"]) + with h5py.File(out_dir / "predictions_fold0.h5", "r") as f: + probs = f["class_probabilities"][...] + assert probs.shape[1] == 3 + assert np.allclose(probs.sum(1), 1.0, atol=1e-5) + + +def test_masked_classes_blinding_error(tmp_path: Path) -> None: + """A flagged event of an excluded class: the loader raises, the validator reports NOT VALID.""" + data = _abc_file(tmp_path, n_violations=3) + config = _abc_config(tmp_path, {"SR": ["b"]}) + spec = load_config(config) + message = "3 events with EVENT/SR true carry the excluded class `b`" + with pytest.raises(ValueError, match=message): + H5Dataset(data, spec, masked_classes={"SR": ["b"]}) + report = validate.validate(config, data) + assert any(message in m for m in report.errors), report.errors + buf = io.StringIO() + report.print(buf) + assert "NOT VALID" in buf.getvalue() + assert validate.main(["--config", str(config), "--input", str(data)]) == 1 + # without the violations the same configuration is valid and the flags are loaded + (tmp_path / "clean").mkdir() + clean = _abc_file(tmp_path / "clean") + report = validate.validate(config, clean) + assert not report.errors, report.errors + ds = H5Dataset(clean, spec, masked_classes={"SR": ["b"]}) + assert ds.flags["SR"].dtype == torch.bool and len(ds.flags["SR"]) == len(ds) + assert set(ds[:5]["flags"]) == {"SR"} + assert H5Dataset(clean, spec).flags == {} + + +def test_flag_column_rules(tmp_path: Path) -> None: + """Missing flag column: KeyError; integer 0/1 is accepted; other values: ValueError.""" + data = _abc_file(tmp_path) + spec = load_config(_abc_config(tmp_path)) + with pytest.raises(KeyError, match="EVENT/CR"): + H5Dataset(data, spec, flags=["CR"]) + with h5py.File(data, "r+") as f: + sr = f["EVENT/SR"][...] + f.create_dataset("EVENT/SR01", data=sr.astype(np.int64)) + f.create_dataset("EVENT/SR2", data=sr.astype(np.int64) * 2) + ds = H5Dataset(data, spec, flags=["SR01"]) + assert torch.equal(ds.flags["SR01"], torch.from_numpy(sr.astype(bool))) + with pytest.raises(ValueError, match="boolean or integer 0/1"): + H5Dataset(data, spec, flags=["SR2"]) + + +def test_masked_validation_loss(tmp_path: Path) -> None: + """validation_metrics with a class mask equals the masked formula on the full probabilities.""" + data = _abc_file(tmp_path) + spec = load_config(_abc_config(tmp_path)) + ds = H5Dataset(data, spec, masked_classes={"SR": ["b"]}) + model = CoReNet(spec, device="cpu") + batches = Batches(ds, "cpu") + with torch.no_grad(): + model.train() + model.updateMeanStd(batches.slice(np.arange(len(ds)))) + model.initMeanStd() + val_idx = np.arange(200) + mask = batches.class_mask(torch.as_tensor(val_idx), {"SR": ["b"]}, 3).numpy() + assert mask[:, 1].sum() == int(ds.flags["SR"][:200].sum()) > 0 + labels, weights = ds.labels.numpy(), ds.weights + out = train.validation_metrics(model, batches, val_idx, labels, weights, mask) + probs = evaluate.predict(model, batches, val_idx)[0].astype(np.float64) + label, w = labels[val_idx], weights.numpy()[val_idx].astype(np.float64) + p = np.empty(len(val_idx)) + for i, row in enumerate(probs): + keep = ~mask[i] + p[i] = row[label[i]] / row[keep].sum() + expected = -(w * np.log(p)).sum() / w.sum() + assert out["val_loss"] == pytest.approx(expected, rel=1e-5) + unmasked = train.validation_metrics(model, batches, val_idx, labels, weights) + assert unmasked["val_loss"] != pytest.approx(expected, rel=1e-5) + assert out["val_auc"] == unmasked["val_auc"] + + +def test_masked_classes_config() -> None: + """The trainer key and the --masked-classes flag; malformed values are rejected.""" + cfg = yaml.safe_load(HH4B.read_text()) + cfg["labels"] = ["a", "b", "c"] + cfg["outputs"] = {"b": "HH"} + spec = parse_config(cfg) + cfg["trainer"] = {"masked_classes": {"SR": ["b"]}} + args = train.build_parser().parse_args(["--config", "x", "--input", "y"]) + assert train.resolve_recipe(args, cfg, spec).masked_classes == {"SR": ["b"]} + args = train.build_parser().parse_args( + ["--config", "x", "--input", "y", "--masked-classes", "SR=b,c", "CR=a"] + ) + assert train.resolve_recipe(args, cfg, spec).masked_classes == { + "SR": ["b", "c"], + "CR": ["a"], + } + args = train.build_parser().parse_args( + ["--config", "x", "--input", "y", "--masked-classes"] + ) + assert train.resolve_recipe(args, cfg, spec).masked_classes == {} + with pytest.raises(ConfigError, match="expected FLAG=CLASS"): + train.resolve_recipe( + train.build_parser().parse_args( + ["--config", "x", "--input", "y", "--masked-classes", "SR"] + ), + cfg, + spec, + ) + assert "--masked-classes" in train.build_parser().format_usage() + for bad, message in [ + ({"SR": ["d"]}, "unknown classes"), + ({"SR": []}, "masked_classes must be a mapping"), + ({"SR": ["a", "b", "c"]}, "masks every class"), + (["SR"], "masked_classes must be a mapping"), + ({"SR": "b"}, "masked_classes must be a mapping"), + ({"SR": ["b", "b"]}, "masked_classes must be a mapping"), + ]: + cfg["trainer"] = {"masked_classes": bad} + with pytest.raises(ConfigError, match=message): + train.resolve_recipe( + train.build_parser().parse_args(["--config", "x", "--input", "y"]), + cfg, + spec, + ) diff --git a/tests/test_training_regression.py b/tests/test_training_regression.py new file mode 100644 index 0000000..50942f9 --- /dev/null +++ b/tests/test_training_regression.py @@ -0,0 +1,66 @@ +"""Regression test of CoReNet's training-mode numerics against golden files (tests/golden/.npz). + +All slots of one leaf role in one node are embedded in one call and share their ghost-batch statistics, so +training-mode outputs, gradients and optimiser steps no longer match the research code bit for bit +(tests/reference compares eval mode only). This test pins them to the stored values of this code: the +HH -> 4b and HH -> bbWW example configurations on 1,024 events of the shipped small example files, CPU, +one thread. What is stored and the tolerance: tests/training_regression.py. After an intentional change of +the numerics: python scripts/make_golden.py (CONTRIBUTING.md). +""" + +from __future__ import annotations + +import numpy as np +import pytest +import training_regression as tr + +CASES = list(tr.CASES) +REGEN = "regenerate with `python scripts/make_golden.py` (CONTRIBUTING.md)" + + +def _golden(case): + path = tr.GOLDEN_DIR / f"{case}.npz" + data = tr.CASES[case][1] + if not data.exists(): + pytest.skip(f"{data} not found") + assert path.exists(), f"{path} missing: {REGEN}" + golden, info = tr.load(path) + current = tr.meta(case) + for key in ("input_md5", *tr.RECIPE): + assert info.get(key) == current[key], ( + f"{case}: {key} is {current[key]!r}, the golden file was made with " + f"{info.get(key)!r}: {REGEN}" + ) + return golden, info + + +@pytest.mark.parametrize("case", CASES) +def test_seeded_initialisation(case): + golden, _ = _golden(case) + init, actual = tr.stage(golden, "init"), tr.initial_state(case) + assert actual.keys() == init.keys(), ( + f"{case}: state-dict keys differ from the golden file" + ) + for k in init: + assert np.array_equal(actual[k], init[k]), ( + f"{case}: {k} after torch.manual_seed({tr.SEED}) differs from the golden file" + ) + + +@pytest.mark.parametrize("case", CASES) +def test_training_numerics(case): + golden, info = _golden(case) + actual = tr.compute(case, init=tr.stage(golden, "init")) + errors, one_sided = tr.compare(actual, golden) + assert not one_sided, f"{case}: arrays present on one side only: {one_sided[:10]}" + worst = sorted(((e, k) for k, e in errors.items()), reverse=True) + bad = [(e, k) for e, k in worst if not e <= tr.TOL] + assert not bad, ( + f"{case}: {len(bad)} of {len(errors)} arrays differ from the golden file (made at " + f"{info.get('commit')}) by more than {tr.TOL:g} (relative); largest: " + + "; ".join(f"{k} {e:.2e}" for e, k in bad[:10]) + ) + print( + f" {case}: {len(errors)} arrays within {tr.TOL:g}; largest relative error " + f"{worst[0][0]:.1e} ({worst[0][1]})" + ) diff --git a/tests/training_regression.py b/tests/training_regression.py new file mode 100644 index 0000000..1152447 --- /dev/null +++ b/tests/training_regression.py @@ -0,0 +1,234 @@ +"""Training-mode numerics of CoReNet on a fixed batch of a shipped example file: the computation behind +the golden-file regression test (tests/test_training_regression.py) and its generator +(scripts/make_golden.py), kept in one place so that both compute exactly the same thing. + +One case = one example configuration on its small example file, on the CPU with THREADS intra-op threads. +Stages (array names ":"): + + init: state dict after torch.manual_seed(SEED); the test loads the stored one, so the stages below + do not depend on the random-number stream (test_seeded_initialisation checks the stream) + stats: every GBN mean / std buffer (*.m, *.s) after updateMeanStd + initMeanStd on the batch + train1: training-mode outputs with GHOST_BATCHES ghost batches (logits, views, heads, event pixel) + train2: the same outputs from a second call + grad: gradients of the first of STEPS SGD steps (cross entropy + squared auxiliary heads) + loss: the loss of every step + trained: every floating-point state-dict entry after the STEPS steps + eval: eval-mode outputs after the steps + +Stored as tests/golden/.npz: one array per name plus "__meta__" (a JSON string). +""" + +from __future__ import annotations + +import hashlib +import json +from contextlib import contextmanager +from pathlib import Path + +import numpy as np +import torch + +from corenet.data import H5Dataset +from corenet.models.config import load_config +from corenet.models.network import CoReNet + +REPO = Path(__file__).resolve().parents[1] +GOLDEN_DIR = REPO / "tests" / "golden" +CASES = { + "hh4b": (REPO / "examples/hh4b/hh4b.yaml", REPO / "examples/data/hh4b_small.h5"), + "bbww": (REPO / "examples/bbww/bbww.yaml", REPO / "examples/data/bbww_small.h5"), +} +N_EVENTS = 1024 # every (len // N_EVENTS)-th event of the file +GHOST_BATCHES = 4 # 256 events per ghost batch +SEED = 3 +STEPS = 5 +# plain SGD: linear in the gradient. Adam's first step is lr * sign(gradient), which turns the rounding +# noise of a gradient that cancels into a full step that differs between machines. +LR = 0.1 +THREADS = 1 +TOL = 1e-4 # largest relative error allowed per array (see compare) +FLOOR = 1e-3 # scale floor of an array: FLOOR x the largest |value| of its stage +RECIPE = ("n_events", "ghost_batches", "seed", "steps", "optimizer", "threads") + + +@contextmanager +def fixed_threads(n): + old = torch.get_num_threads() + torch.set_num_threads(n) + try: + yield + finally: + torch.set_num_threads(old) + + +def file_md5(path) -> str: + h = hashlib.md5() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def meta(case) -> dict: + cfg, data = CASES[case] + return { + "case": case, + "config": cfg.relative_to(REPO).as_posix(), + "input": data.relative_to(REPO).as_posix(), + "input_md5": file_md5(data), + "n_events": N_EVENTS, + "ghost_batches": GHOST_BATCHES, + "seed": SEED, + "steps": STEPS, + "optimizer": f"SGD lr={LR}", + "threads": THREADS, + "tol": TOL, + } + + +def save(file, arrays: dict, info: dict) -> None: + np.savez_compressed( + file, __meta__=np.array(json.dumps(info, sort_keys=True)), **arrays + ) + + +def load(path): + with np.load(path, allow_pickle=False) as z: + arrays = {k: z[k] for k in z.files if k != "__meta__"} + info = json.loads(str(z["__meta__"])) + return arrays, info + + +def stage(arrays: dict, name: str) -> dict: + p = name + ":" + return {k[len(p) :]: v for k, v in arrays.items() if k.startswith(p)} + + +def _model(case) -> CoReNet: + torch.manual_seed(SEED) + return CoReNet(load_config(CASES[case][0]), "cpu") + + +def _numpy(t) -> np.ndarray: + return t.detach().cpu().numpy().copy() + + +def initial_state(case) -> dict: + return {k: _numpy(v) for k, v in _model(case).state_dict().items()} + + +def perturb(case, init: dict) -> dict: + """``init`` with every parameter (not buffer) multiplied by 1 +- 2^-23 (random signs, seed 0): one + float32 rounding of the starting point, the noise probe of scripts/make_golden.py.""" + names = {k for k, _ in _model(case).named_parameters()} + rng = np.random.default_rng(0) + out = {} + for k, v in init.items(): + if k in names: + sign = rng.choice(np.array([-1.0, 1.0], dtype=np.float32), size=v.shape) + v = (v * (np.float32(1) + sign * np.float32(2.0**-23))).astype(v.dtype) + out[k] = v + return out + + +def _batch(case): + cfg, data = CASES[case] + ds = H5Dataset(data, load_config(cfg)) + idx = torch.arange(N_EVENTS) * (len(ds) // N_EVENTS) + return {k: v[idx].clone() for k, v in ds.inputs.items()}, ds.labels[idx].clone() + + +def _outputs(name, out, arrays): + def walk(path, x): + if isinstance(x, dict): + for k, v in x.items(): + walk(f"{path}.{k}" if path else str(k), v) + else: + arrays[f"{name}:{path}"] = _numpy(x) + + walk("", out) + + +def compute(case, init: dict | None = None, threads: int = THREADS) -> dict: + """The arrays of one case (module docstring). ``init``: initial state dict to load (numpy arrays as + stored under ``init:``); None = the state after torch.manual_seed(SEED).""" + x, label = _batch(case) + + def inputs(): + return {k: v.clone() for k, v in x.items()} + + arrays = {} + with fixed_threads(threads): + model = _model(case) + if init is not None: + model.load_state_dict( + {k: torch.from_numpy(np.array(v)) for k, v in init.items()}, strict=True + ) + for k, v in model.state_dict().items(): + arrays[f"init:{k}"] = _numpy(v) + model.train() + model.updateMeanStd(inputs()) + model.initMeanStd() + for k, v in model.state_dict().items(): + if k.endswith((".m", ".s")) and v.is_floating_point(): + arrays[f"stats:{k}"] = _numpy(v) + model.setGhostBatches(GHOST_BATCHES) + for name in ("train1", "train2"): + with torch.no_grad(): + _outputs(name, model(inputs()), arrays) + opt = torch.optim.SGD(model.parameters(), lr=LR) + losses = [] + for step in range(STEPS): + out = model(inputs()) + loss = torch.nn.functional.cross_entropy(out["logits"], label) + for h in out["heads"].values(): + loss = loss + h.pow(2).mean() + opt.zero_grad() + loss.backward() + if step == 0: + for k, p in model.named_parameters(): + if p.grad is not None: + arrays[f"grad:{k}"] = _numpy(p.grad) + opt.step() + losses.append(loss.item()) + arrays["loss:sgd"] = np.array(losses) + for k, v in model.state_dict().items(): + if v.is_floating_point(): + arrays[f"trained:{k}"] = _numpy(v) + model.eval() + model.setGhostBatches(0) + with torch.no_grad(): + _outputs("eval", model(inputs()), arrays) + return arrays + + +def compare(actual: dict, golden: dict): + """Relative error of every golden array: max |actual - golden| / scale, scale = max |golden| of that + array, floored at FLOOR x the largest |golden| of its stage (an array that is zero up to rounding, such + as a gradient that cancels, is not measured against its own noise). Integer arrays must be equal + (error 0 or inf); a shape mismatch or a non-finite value is inf. + Returns ({name: error}, names present on one side only).""" + stage_max = {} + for k, g in golden.items(): + if np.issubdtype(g.dtype, np.floating) and g.size: + s = k.split(":", 1)[0] + stage_max[s] = max(stage_max.get(s, 0.0), float(np.abs(g).max())) + errors = {} + for k, g in golden.items(): + if k not in actual: + continue + a = actual[k] + if a.shape != g.shape: + errors[k] = np.inf + elif not np.issubdtype(g.dtype, np.floating): + errors[k] = 0.0 if np.array_equal(a, g) else np.inf + elif g.size == 0: + errors[k] = 0.0 + elif not np.all(np.isfinite(a)): + errors[k] = np.inf + else: + floor = FLOOR * stage_max[k.split(":", 1)[0]] + scale = max(float(np.abs(g).max()), floor, 1e-30) + diff = np.abs(a.astype(np.float64) - g.astype(np.float64)).max() + errors[k] = float(diff) / scale + return errors, sorted(set(golden) ^ set(actual))