Repository navigation
[BUG][pt_expt][DPA1] Automatic graph-lower export changes predictions for dense-trained se_atten_v2 models with nonzero davg #5973
Description
Bug summary
I trained a DPA1-L0 model with the standard PyTorch backend (dp --pt train) and converted the
frozen .pth model to .pt2 using dp convert-backend. The descriptor used the default
set_davg_zero: false and has nonzero davg.
The automatically selected graph-lower model loads and runs, but it is not numerically
equivalent to the source model. On the same six-frame system, the force-component difference
is:
- MAE:
1.1381067 eV/Angstrom - RMSE:
1.7619205 eV/Angstrom - Maximum absolute difference:
6.7789032 eV/Angstrom
In contrast, a manually exported nlist/dense-lower .pt2 model reproduces the source model
with a force-component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/ Angstrom.
The discrepancy is reproduced by an uncompiled CPU eager comparison between
forward_common_lower and forward_common_lower_graph, and is already present in the
descriptor output. As a diagnostic, setting the descriptor mean (davg) to zero in memory
reduces the descriptor discrepancy to numerical noise.
I also independently retrained four models with set_davg_zero: true. After automatic graph-
lower conversion, their force-component MAE differences are 1.76e-5–1.84e-5 eV/Angstrom,
with a maximum observed difference of 1.33e-4 eV/Angstrom.
These results indicate that automatic conversion accepts a dense-trained se_atten_v2 model
with nonzero davg, even though its graph-lower representation is not functionally equivalent.
DeePMD-kit Version
0.1.dev1+gbc902daf7.d20260809
Backend and its version
PyTorch 2.11.0+cu128
How did you download the software?
Built from source
Input Files, Running Commands, Error Log, etc.
A minimal reproducer archive is attached containing:
reproducer/
├── frozen_model.pth
├── input.json
├── compare_predictions.py
└── test_system/
├── type.raw
├── type_map.raw
├── nopbc
└── set.000/
├── box.npy
├── coord.npy
├── energy.npy
└── force.npy
The test system is nonperiodic and contains six frames with seven atoms per frame.
Relevant model settings:
{
"type_map": ["C", "Cl", "H", "O"],
"descriptor": {
"type": "se_atten_v2",
"sel": 250,
"rcut": 6.0,
"attn": 128,
"attn_layer": 0,
"tebd_input_mode": "strip",
"precision": "float32"
}
}
set_davg_zero was not specified during training, so its effective value is false. The trained
descriptor has nonzero davg; its largest absolute value is approximately 0.01418.
Environment:
DeePMD-kit: 0.1.dev1+gbc902daf7.d20260809
Git commit: bc902daf747d93c5d7314338d287fd2dd0a7e32c
PyTorch: 2.11.0+cu128
Python: 3.12.12
OS: Linux x86_64
The build is based on the official master branch. The only local source change preserves an
explicitly requested `CMAKE_CUDA_ARCHITECTURES=120`; no Python, model, descriptor,
serialization, or inference code was modified.
There is no exception or error traceback. Conversion and inference both exit successfully. This
is a silent numerical correctness failure.
Observed source .pth versus automatic graph-lower .pt2 difference:
Energy MAE: 1.3291797 eV/frame
Energy maximum difference: 2.0507687 eV/frame
Force-component MAE: 1.1381067 eV/Angstrom
Force-component RMSE: 1.7619205 eV/Angstrom
Force maximum difference: 6.7789032 eV/Angstrom
For comparison, manually exporting the same model through nlist/dense lower gives a force-
component MAE of 5.13e-6 eV/Angstrom and a maximum difference of 1.91e-5 eV/Angstrom.
### Steps to Reproduce
[deepmd-dpa1-graph-lower-reproducer.zip](https://github.com/user-attachments/files/31117915/deepmd-dpa1-graph-lower-reproducer.zip)
1. Extract the attached minimal reproducer and enter its directory:
```bash
unzip deepmd-dpa1-graph-lower-reproducer.zip
cd deepmd-dpa1-graph-lower-reproducer
2. Convert the supplied source model. dp convert-backend automatically selects graph lower:
dp convert-backend frozen_model.pth model_graph.pt2
3. Evaluate the source and converted models on the same six-frame system:
dp --pt test -m frozen_model.pth -s test_system -n 32 -d pth
dp --pt-expt test -m model_graph.pt2 -s test_system -n 32 -d graph
4. Compare their predictions:
python compare_predictions.py pth graph
Expected reproduction output:
Energy MAE difference: 1.3291797 eV/frame
Energy maximum absolute difference: 2.0507687 eV/frame
Force-component MAE: 1.1381067 eV/Angstrom
Force-component RMSE: 1.7619205 eV/Angstrom
Force-component maximum difference: 6.7789032 eV/Angstrom
No retraining, LAMMPS, or Kokkos is required. The same underlying dense-versus-graph
discrepancy was also reproduced independently in uncompiled CPU eager execution.
### Further Information, Files, and Links
_No response_
Activity
dosubot commentedon Aug 16, 2026
Hi @hcustc! I'm Dosu and I'm helping the DeepModeling team.
This is a variant of a previously reported bug. The root cause is the same phantom padding-neighbor residual documented in
deepmd/dpmodel/descriptor/dpa1.py: the dense lower path (forward_common_lower) retains a-davg/dstdcontribution for every vacant padding slot in the fixed-capacity neighbor list, while the graph lower path (forward_common_lower_graph) omits padding neighbors entirely [1]. Whendavg ≠ 0, the two paths are not numerically equivalent.Your case differs from issue #5862 (which covers
.ptcheckpoint loading): here the discrepancy appears duringdp convert-backendto.pt2. The export gatemodel_uses_graph_lowerindeepmd/pt_expt/utils/serialization.pydoes not check whether the model has nonzerodavg, so a dense-trained model withset_davg_zero: falsegets silently exported into a graph-lower.pt2whose descriptor semantics differ from the source.Your
set_davg_zero: trueretrained models round-trip cleanly (MAE ~1.8×10⁻⁵ eV/Å) because withdavg = 0the padding residual vanishes and the two paths produce identical outputs [2].Two potential fixes to consider:
davg ≠ 0— makedp convert-backendfall back to the nlist/dense-lower format and warn the user, rather than silently producing an inequivalent model.davgin the graph-lower export path — mathematically valid if the graph path has already absorbed the normalization elsewhere, but this needs verification against the DPA1 aggregation semantics.The companion issue #5824 proposes decoupling DPA1 graph normalization and statistics from the dense
sel-based representation [3], which is the structural prerequisite for option 2. Issue #5862 tracks the related.ptcheckpoint path mismatch [1].Immediate workaround: Manually export via the nlist/dense-lower path (as you've already done), which you confirmed gives a force-component MAE of ~5×10⁻⁶ eV/Å. Until the export gate logic is fixed, models trained with
set_davg_zero: falseshould not use the automatically selected graph-lower.pt2format.To reply, just mention @dosu.
Docs are dead. Just use Dosu.
njzjz-bot commentedon Aug 23, 2026
Dosu's root-cause diagnosis is correct, but one suggested fix needs correction.
The source
.pthmodel was trained with the legacy dense-lower semantics. With nonzerodavg, converting it to a graph lower changes the represented function because the dense path retains the historical padding contribution while the graph path has no padding edges.Do not zero
davgin an already trained model. The normalization statistics are part of the trained function; changing them after training changes the descriptor inputs and generally changes the model predictions.set_davg_zero: trueis a valid workaround only when used for training/retraining, as the reporter tested.The conversion-side fix is PR #5975: preserve the source artifact's concrete lower-input semantics instead of automatically reinterpreting every graph-capable reconstructed model as graph-native. For a legacy dense
.pthsource, conversion should therefore emit the nlist/dense lower. Until that fix is merged, the reporter's manual dense-lower export is the correct workaround.Coding agent: Codex
Codex version: codex-cli 0.149.0
Model: gpt-5.6-sol
Reasoning effort: xhigh