CLI --help without torch; difference-map phases, DELFWT and extrapolations

docs/changelog.rst

@@ -13,6 +13,8 @@ Breaking changes
 - Anomalous scattering is off unless ``--wavelength`` is given (it defaulted to 1.0 Å)
 - Bulk-solvent phase offset removed; ``optimize_phase`` is deprecated
 - ``phased-difference-map`` is now ``difference-map``: fewer default columns (``--all-columns``), renamed (``dF``, ``W_InVa``)
+- ``import torchref`` no longer imports torch: the top-level names (``Model``, ``ReflectionData``, ``dtypes``, ...) load on first access, and ``TORCHREF_DTYPE_*``/``TORCHREF_DEVICE`` are read then
+- ``torchref.difference-map`` takes ``-o`` as a directory (``difference_map.mtz``, ``summary.json``) and gains ``--dmax``; the difference MTZ adds ``DELFWT`` (``dF`` times the ``--ded-weight`` weight), writes the phase-aware and scalar extrapolations by default, and renames ``FC``/``PHIC``/``PHIC_diff`` to ``Fc_light``/``PHFc_light``/``PHdFc``
 - Refinement checkpoints restore only through ``load_state``; ``Refinement.create_from_state_dict`` is removed
 - ``french_wilson_auto`` moved to ``torchref.io.datasets``; ``CollectionRiceTarget`` is now ``CollectionMLTarget``
 - Refinement writes its own PDB/mmCIF header instead of copying the input's
@@ -33,6 +35,7 @@ Performance
 ~~~~~~~~~~~
 - CPU density kernels ship prebuilt in the new ``torchref-kernels`` package (Rust): nothing is compiled at runtime, ``ninja`` is no longer a dependency, and the CPU splat is multithreaded on macOS. Wheels cover Linux x86_64/aarch64, macOS arm64 and Windows x86_64; other platforms run the portable CPU kernels unless ``torchref-kernels`` is built from source
 - Faster CPU VDW pair search, hydrogen-mode switching and ADP-locality neighbours; ``SfFFT`` stores no real-space grid
+- Every command-line tool answers ``--help`` and argument errors without importing torch
 - The command-line tools set ``OMP_WAIT_POLICY=PASSIVE``, so idle OpenMP workers no longer spin on the cores the CPU kernels need: multithreaded CPU refinement is faster. Library use is unchanged; see the CLI guide to opt in
 
 Fixes that change results
@@ -45,6 +48,7 @@ Fixes that change results
 - Refinement: frozen parameters stay frozen, rigid-body translations refine, the scaler is refit warm each cycle
 - Density kernels on CPU, CUDA and MPS; long matrix products on MPS; REMARK 3 angle RMSD reported in degrees
 - CUDA ``ls``/``nll`` X-ray kernels score a signed ``F_calc`` as ``|F_calc|``, as the eager path does
+- Difference maps take their phases from a fit of the dark model alone, so they no longer change with ``-lm``; without ``--dmin`` the models' FFT grid follows the data instead of a fixed 1.0 Å
 - ``WilsonNormaliser`` with ``s_lo``/``s_hi`` wider than the data: stable in float32 and no longer truncated by its log clamp
 
 Other fixes

docs/user_guide/cli.rst

@@ -236,29 +236,36 @@ Compute difference and extrapolated map coefficients without refinement.
 Uses the same pipeline as ``torchref.difference-refine`` but the input
 models are kept as-is.
 
-The default output is the difference map: the amplitude difference ``dF``/``SIGdF``
-on the **dark** model's phases ``PHDELWT``, with one mean-one weight column per
-registered scheme beside it -- ``W_Q``, the q-weight (the default), and ``W_InVa``,
-the inverse variance ``1/sigma^2`` -- and ``KSCALE``, the scaler's factor from model to observed
-scale. This is the construction ``torchref.validate-ded`` correlates against. Build the
-map with ``torchref.mtz2map -csf dF -cw W_Q -cphi PHDELWT``, adding
-``--units electrons`` for e/A^3. It needs no light-state model, so ``-lm`` is optional:
+The default output is the difference map ``DELFWT``/``PHDELWT``: the amplitude
+difference ``dF`` times the ``--ded-weight`` weight, on the **dark** model's phases, which
+Coot opens directly. The phases come from a fit of the dark model alone, so the
+difference map is the same with or without a light model. Beside it sit ``dF``/``SIGdF``,
+one mean-one weight column per registered scheme -- ``W_Q``, the q-weight (the default),
+and ``W_InVa``, the inverse variance ``1/sigma^2`` -- and ``KSCALE``, the scaler's factor
+from model to observed scale, so another weighting is
+``torchref.mtz2map -csf dF -cw W_InVa -cphi PHDELWT`` (``--units electrons`` for e/A^3).
+This is the construction ``torchref.validate-ded`` correlates against. ``-o`` is a
+directory; it receives ``difference_map.mtz`` and ``summary.json`` (inputs, dataset
+scaling, R-factors, the ``q``-weight fit and the extrapolation shrinkage).
 
 .. code-block:: bash
 
    torchref.difference-map \
        -dm dark.pdb \
-       -dsf dark.mtz -lsf light.mtz -o results.mtz
+       -dsf dark.mtz -lsf light.mtz -o results/
 
-Supplying ``-lm`` (with ``--fraction``) adds the light state's amplitude and
-phase and the extrapolated map ``FWT``/``PHWT``:
+The light-state model is optional. Supplying ``-lm`` (with ``--fraction``) adds the
+light data's model ``Fc_light``/``PHFc_light`` and three extrapolated light-state maps:
+``FWT``/``PHWT`` from the shrunk amplitudes ``FEXT``, ``2FEXT_PHASED-Fc``/``PHFEXT_PHASED``
+from the phase-aware ones and ``2FEXT_SCALAR-Fc``/``PHFEXT_SCALAR`` from the classic
+``(Fo_light - w_dark Fo_dark) / w_light``, each on its own scale fit:
 
 .. code-block:: bash
 
    torchref.difference-map \
        -dm dark.pdb -lm light.pdb \
        -dsf dark.mtz -lsf light.mtz \
-       --fraction 0.37 -o results.mtz
+       --fraction 0.37 -o results/
 
 ``FWT``/``PHWT`` keep the standard labels so Coot auto-opens the map, but here they
 are the extrapolated light-state map ``2*FEXT - Fc``, not a ``2mFo-DFc``. The file
@@ -267,18 +274,18 @@ records this: the columns sit in named MTZ datasets -- ``observed``, ``differenc
 column chooser shows ``/torchref/extrapolated_light/FWT``, and ``gemmi mtz`` prints a
 history line per dataset. ``torchref.difference-refine`` writes the same file.
 
-**Key options:** ``--ded-weight {q,inverse_variance,none}`` selects the scheme the
-model-phased and two-moment difference columns carry (default ``q``; its fit uses the
-intensity differences when the data carry ``I``/``SIGI``, and falls back to inverse
-variance with a warning when too few reflections exist to fit); ``--difference-gamma``
-fixes the dark-amplitude exponent of the difference power law instead of fitting it;
-``--difference-sigma-scale`` scales the reported sigmas the ``q`` weight and the
-extrapolation take as the noise (default 1; ``fit`` estimates the scale from the
-differences, which reads noise as signal when the sigmas' calibration changes with
-resolution, as French-Wilson and other posterior amplitude sigmas' does);
-``--all-columns`` writes every alternative map coefficient and diagnostic -- the
-model-phased difference, the two other extrapolations and the intensity block -- at
-the cost of two further scale fits.
+**Key options:** ``--dmin``/``--dmax`` resolution limits (without ``--dmin`` the
+models' grid follows the data); ``--ded-weight {q,inverse_variance,none}`` selects the
+scheme ``DELFWT`` and the model-phased difference columns carry (default ``q``; its fit
+uses the intensity differences when the data carry ``I``/``SIGI``, and falls back to
+inverse variance with a warning when too few reflections exist to fit);
+``--difference-gamma`` fixes the dark-amplitude exponent of the difference power law
+instead of fitting it; ``--difference-sigma-scale`` scales the reported sigmas the ``q``
+weight and the extrapolation take as the noise (default 1; ``fit`` estimates the scale
+from the differences, which reads noise as signal when the sigmas' calibration changes
+with resolution, as French-Wilson and other posterior amplitude sigmas' does);
+``--all-columns`` (needs ``-lm``) adds the model-phased difference and the intensity
+block.
 
 :API: :mod:`torchref.cli.difference_map`
 

tests/integration/test_cli_ded_weights.py

@@ -24,6 +24,7 @@
     "SIGFo_light": "Stddev",
     "dF": "SFAmplitude",
     "SIGdF": "Stddev",
+    "DELFWT": "SFAmplitude",
     "PHDELWT": "Phase",
     "W_InVa": "Weight",
     "W_Q": "Weight",
@@ -105,7 +106,7 @@ def _run(project_root, module, *argv, timeout=1800):
 
 @pytest.fixture(scope="module")
 def diff_mtz(project_root, pair):
-    out = pair["dir"] / "diff.mtz"
+    out = pair["dir"] / "diff"
     _run(
         project_root,
         "difference_map.py",
@@ -126,7 +127,7 @@ def diff_mtz(project_root, pair):
         "-o",
         out,
     )
-    return out
+    return out / "difference_map.mtz"
 
 
 def _read(path):

tests/integration/test_cli_two_moment_mtz.py

@@ -19,20 +19,29 @@
     "SIGFo_light": "Stddev",
     "dF": "SFAmplitude",
     "SIGdF": "Stddev",
-    # The difference map: dF on the dark phases, one weight column per registered
-    # scheme, and the observed-to-model scale.
+    # The difference map: weighted dF on the dark phases, one weight column per
+    # registered scheme, and the observed-to-model scale.
+    "DELFWT": "SFAmplitude",
     "PHDELWT": "Phase",
     "W_InVa": "Weight",
     "W_Q": "Weight",
     "KSCALE": "MTZReal",
     "Fc_dark": "SFAmplitude",
-    # The mixed model, and the extrapolated map to refine against.
-    "FC": "SFAmplitude",
-    "PHIC": "Phase",
+    # The light data's (mixed) model, and the three extrapolated maps.
+    "Fc_light": "SFAmplitude",
+    "PHFc_light": "Phase",
     "FEXT": "SFAmplitude",
     "SIGFEXT": "Stddev",
     "FWT": "SFAmplitude",
     "PHWT": "Phase",
+    "FEXT_PHASED": "SFAmplitude",
+    "SIGFEXT_PHASED": "Stddev",
+    "2FEXT_PHASED-Fc": "SFAmplitude",
+    "PHFEXT_PHASED": "Phase",
+    "FEXT_SCALAR": "SFAmplitude",
+    "SIGFEXT_SCALAR": "Stddev",
+    "2FEXT_SCALAR-Fc": "SFAmplitude",
+    "PHFEXT_SCALAR": "Phase",
 }
 FLAG_COLUMNS = {"FreeR_flag_dark", "FreeR_flag_light"}
 
@@ -49,19 +58,9 @@
 ALL_COLUMNS_EXTRA = {
     "2mDFop-DFc": "SFAmplitude",
     "mDFop-DFc": "SFAmplitude",
-    "PHIC_diff": "Phase",
+    "PHdFc": "Phase",
     "dFc": "SFAmplitude",
     "dFc_phased": "SFAmplitude",
-    "FEXT_PHASED": "SFAmplitude",
-    "SIGFEXT_PHASED": "Stddev",
-    "2FEXT_PHASED-Fc": "SFAmplitude",
-    "FEXT_PHASED-Fc": "SFAmplitude",
-    "PHFEXT_PHASED": "Phase",
-    "FEXT_SCALAR": "SFAmplitude",
-    "SIGFEXT_SCALAR": "Stddev",
-    "2FEXT_SCALAR-Fc": "SFAmplitude",
-    "FEXT_SCALAR-Fc": "SFAmplitude",
-    "PHFEXT_SCALAR": "Phase",
 }
 
 # And what it adds again once the two-moment model is on.
@@ -229,8 +228,8 @@ def test_map_coefficients_are_grouped_into_described_datasets(two_moment_all_mtz
 
     assert where["FWT"] == where["PHWT"] == where["FEXT"] == "extrapolated_light"
     assert where["FEXT_PHASED"] == where["FEXT_SCALAR"] == "extrapolated_light"
-    assert where["dF"] == where["PHDELWT"] == where["W_Q"] == "difference"
-    assert where["FC"] == where["PHIC"] == "light_model"
+    assert where["DELFWT"] == where["PHDELWT"] == where["W_Q"] == "difference"
+    assert where["Fc_light"] == where["PHFc_light"] == "light_model"
     assert where["dF_corr"] == "two_moment"
     assert where["H"] == where["Fo_dark"] == where["FreeR_flag_dark"] == "observed"
     assert all(ds.crystal_name == "torchref" for ds in mtz.datasets)
@@ -246,8 +245,8 @@ def test_the_difference_columns_carry_df_and_the_registered_weights(
         self, baseline_mtz
     ):
         """``dF`` must be ``Fo_light - Fo_dark``, ``W_InVa`` the mean-normalised inverse
-        variance and ``W_Q`` a mean-one weight -- the constructions
-        ``torchref.validate-ded`` correlates against. If these ever diverge, the map
+        variance, ``W_Q`` a mean-one weight and ``DELFWT`` ``dF * W_Q`` -- the
+        constructions ``torchref.validate-ded`` correlates against. If these ever diverge, the map
         built from the file stops being the map the validation reports on, which is how
         the output drifted from the science before."""
         import numpy as np
@@ -270,12 +269,14 @@ def test_the_difference_columns_carry_df_and_the_registered_weights(
         assert np.abs(got_w - w).max() / max(float(np.abs(w).max()), 1e-30) < 1e-4
         w_q = df["W_Q"].to_numpy().astype(float)
         assert np.isfinite(w_q).all() and abs(w_q.mean() - 1.0) < 1e-4
+        got_delfwt = df["DELFWT"].to_numpy().astype(float)
+        assert np.abs(got_delfwt - got_df * w_q).max() / scale < 1e-5
         assert (df["KSCALE"].to_numpy().astype(float) > 0).all()
 
         # And the phase is the dark model's, not the mixed model's.
         assert not np.allclose(
             df["PHDELWT"].to_numpy().astype(float),
-            df["PHIC"].to_numpy().astype(float),
+            df["PHFc_light"].to_numpy().astype(float),
         )
 
 

torchref/__init__.py

@@ -11,6 +11,9 @@
     )
     refinement.refine(macro_cycles=10)
 
+``import torchref`` does not import torch: the names re-exported here and the
+subpackages load on first access, and :mod:`torchref.config` reads its environment then.
+
 Modules
 -------
 io
@@ -42,13 +45,15 @@
 __version__ = "0.7.0"
 
 
+import importlib
 import os
-
-# Must be set before torch is imported below, or it has no effect.
-os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
+import sys
 import warnings
 from pathlib import Path
 
+# Must be set before torch is first imported, or it has no effect.
+os.environ.setdefault("PYTORCH_ENABLE_MPS_FALLBACK", "1")
+
 from torchref._bootstrap import configure_threading, detect_available_cpus
 
 # Threading must be configured before torch is imported.
@@ -68,12 +73,10 @@
 
 configure_threading(N_CPUS)
 
-import torch
-
-torch.set_num_threads(N_CPUS)
-
-# Must come after torch: reads its dtype/device state at import.
-from torchref.config import device, dtypes, sigma_cutoff_ed
+# A torch already imported has read its thread count; one imported later reads it from
+# the OMP_NUM_THREADS just set.
+if "torch" in sys.modules:
+    sys.modules["torch"].set_num_threads(N_CPUS)
 
 # Project root path for referencing package files
 ROOT_TORCHREF = Path(__file__).parent.parent.resolve()
@@ -83,80 +86,44 @@
 
 PATH_TORCHREF_DATA = PATH_TORCHREF / "data"
 
-# =============================================================================
-# Convenience imports for common classes
-# =============================================================================
-
-
-# Data I/O
-from torchref.io import (
-    DatasetCollection,
-    ReflectionData,
-    ScaledDataset,
-    read_cif,
-    read_mtz,
-    read_pdb,
-)
-
-# Maps
-from torchref.maps import DifferenceMap, Map
-
-# Model
-from torchref.model import Model, ModelFT
-from torchref.model.rigid_xyz import RigidXYZTensor
-
-# Refinement
-from torchref.refinement import LBFGSRefinement, Refinement
-from torchref.refinement.rigid_body_refinement import RigidBodyRefinementStep
-
-# Scaling
-from torchref.scaling import Scaler, ScalerBase, SolventModel
-from torchref.symmetry import Cell, SpaceGroup, Symmetry
-
-# Device movement mixin (public API for extension code)
-from torchref.utils.device_mixin import DeviceMixin
-
-# Restraints
-# torchref.topology.restraints.Restraints is not imported here: constructing it can
-# trigger a monomer-library download, so it stays lazy.
-
-
-__all__ = [
-    # Version and paths
-    "__version__",
-    "ROOT_TORCHREF",
-    "PATH_TORCHREF",
-    "N_CPUS",
-    # Dtype and device configuration
-    "dtypes",
-    "device",
-    "sigma_cutoff_ed",
-    # Data I/O
-    "ReflectionData",
-    "ScaledDataset",
-    "DatasetCollection",
-    "read_mtz",
-    "read_cif",
-    "read_pdb",
-    # Model
-    "Model",
-    "ModelFT",
-    "RigidXYZTensor",
-    # Refinement
-    "Refinement",
-    "LBFGSRefinement",
-    "RigidBodyRefinementStep",
-    # Scaling
-    "Scaler",
-    "ScalerBase",
-    "SolventModel",
-    # Symmetry
-    "Cell",
-    "SpaceGroup",
-    "Symmetry",
-    # Maps
-    "Map",
-    "DifferenceMap",
-    # Device mixin
-    "DeviceMixin",
-]
+# Defining module -> public names, imported on first access by ``__getattr__`` so that
+# ``import torchref`` (and every CLI ``--help``) doesn't import torch. ``Restraints`` is
+# deliberately absent: constructing it can trigger a monomer-library download.
+_LAZY_EXPORTS = {
+    name: module
+    for module, names in {
+        "torchref.config": "dtypes device sigma_cutoff_ed",
+        "torchref.io": "ReflectionData ScaledDataset DatasetCollection"
+        " read_mtz read_cif read_pdb",
+        "torchref.model": "Model ModelFT",
+        "torchref.model.rigid_xyz": "RigidXYZTensor",
+        "torchref.refinement": "Refinement LBFGSRefinement",
+        "torchref.refinement.rigid_body_refinement": "RigidBodyRefinementStep",
+        "torchref.scaling": "Scaler ScalerBase SolventModel",
+        "torchref.symmetry": "Cell SpaceGroup Symmetry",
+        "torchref.maps": "Map DifferenceMap",
+        "torchref.utils.device_mixin": "DeviceMixin",
+    }.items()
+    for name in names.split()
+}
+
+
+def __getattr__(name: str):
+    """Import a re-exported name, or a subpackage, on first access."""
+    if name in _LAZY_EXPORTS:
+        value = getattr(importlib.import_module(_LAZY_EXPORTS[name]), name)
+    elif not name.startswith("_") and any(
+        (PATH_TORCHREF / n).exists() for n in (name, f"{name}.py")
+    ):
+        value = importlib.import_module(f"{__name__}.{name}")
+    else:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+    globals()[name] = value
+    return value
+
+
+def __dir__():
+    return sorted(set(globals()) | set(_LAZY_EXPORTS))
+
+
+__all__ = ["__version__", "ROOT_TORCHREF", "PATH_TORCHREF", "N_CPUS", *_LAZY_EXPORTS]