Repository navigation
Record exact label availability when building LMDB datasets #5954
Description
Summary
LMDB generation should record exact per-frame label availability in dataset metadata. Data readers should consume that metadata instead of reading or probing frame payloads at training startup to infer label-availability signatures.
Motivation
A training batch exposes one find_* value for each label. Frames with different optional-label availability therefore need either exact homogeneous grouping or conservative reduction at batching time.
Determining those groups online is problematic for large LMDB datasets:
- An exact result requires inspecting every frame before useful training work starts.
- A bounded probe avoids that startup scan for common uniform datasets, but it is inherently probabilistic.
- Rare availability signatures outside the sample may be missed. If they later enter a mixed batch, conservative reduction preserves correctness but discards valid supervision from the other frames in that batch.
- Genuinely heterogeneous datasets may still trigger an expensive full scan.
Label availability is an intrinsic property of the serialized dataset, so it should ideally be computed once when the LMDB is created rather than rediscovered by every consumer.
Proposal
When generating an LMDB dataset:
- Normalize label names according to the same key mapping used by readers.
- Compute each frame's effective availability from source presence and any explicit
find_*value. - Encode distinct availability bitsets as compact signature IDs.
- Store a schema-versioned signature table and one compact signature ID per frame in LMDB metadata.
- Store dataset-level uniformity directly when all frames share one signature.
At read time:
- Select only labels required by the active model and loss.
- Project the stored signatures onto those labels without decoding frame payloads.
- Partition tracked optional labels exactly.
- Keep mandatory-label validation during frame decoding as a corruption and contract check.
- Fall back to the existing reader-side detection path for legacy LMDBs without this metadata.
Implementation considerations
- Metadata should describe source facts, not model-specific policy. Decisions such as whether a missing label may use a default should remain with the consumer.
- LMDB merge and filtering utilities should preserve or rebuild the signature metadata, including remapping signature IDs where necessary.
- The metadata schema should be versioned so readers can reject malformed or incompatible indexes safely.
- A compact representation should have low overhead: typically one small integer per frame plus a small table of unique signatures.
Expected benefits
- Exact label grouping without probabilistic misses.
- No
O(number of frames)payload scan at training startup. - Reusable availability information for training, statistics, validation, sampling, and inference.
- A clearer separation between immutable dataset facts and consumer policy.
Relationship to reader-side mitigation
A reader-side change remains useful for existing datasets: defer availability checks until label requirements are known, probe only active optional labels, and build an exact compact index only when necessary. That approach should be treated as backward-compatible mitigation for legacy LMDBs; generation-time metadata should be the preferred long-term source of truth.
Activity
feat(lmdb): support mixed-size batches and lazy label availability (d…
LOGO127 commentedon Sep 8, 2026
I reproduced this with the real dpdata writer and DeePMD LMDB reader, and would be interested in a coordinated implementation if nobody is already working on it.
Tested dpdata
991a38dd132272656ad4397f463e58815b7d342eand DeePMD28b7d068801716765ab8119257f814596e49a10con Linux/Python 3.12. I built the latter with TF/PT/Paddle backends disabled for this reader-only check; no mocked reader or import stubs were used.The fixture uses
dump_systemswith three ordered one-H-atom systems containing 1, 1 and 1,022 frames. Every frame has coordinates, a periodic cell and energy; the middle system alone omits forces. Withwrite_batch_size=127, the real output has 1,024 frames and two label signatures, but no availability metadata. Registering optional energy/force requirements and callingcollect_lmdb_sampling_groupsyields one group of 1,024 frames. Actualdecode_batchresults:find_force[0][1][0, 1][0, 2]find_energyremains 1 in the mixed batch. This confirms the existing conservative behavior described in #5962: the rare missing-force signature falls outside the bounded probe, and force supervision is disabled for an explicitly decoded mixed batch. It does not fabricate labels. I have not run a stochastic training sampler, model loss/backpropagation, or an accuracy benchmark.Before choosing permanent metadata keys, could we agree on a producer/consumer contract? My proposed scope is the complete generation-time index plus reader projection, merge/filter preservation, and legacy fallback—not simply increasing the probe bound:
frame_nlocs/frame_system_ids, preserving staged publication and map-full retries.dpdata currently rejects custom disk names beginning with
find_, while the DeePMD reader also accepts externally produced explicit flags. The schema should preserve that distinction instead of repurposing reserved dpdata fields. I also noticed #5565's dpdata-to-LMDB conversion path; an additive shared metadata contract should avoid duplicating that conversion work.Is this cross-repository scope welcome, and is there already a preferred schema/versioning convention? No assignment or fix is claimed yet. Source investigation and reproduction were prepared with Codex assistance.
njzjz-bot commentedon Sep 29, 2026
Yes, the cross-repository producer/consumer scope you outlined is welcome, and the real dpdata→LMDB→DeePMD reproducer is a good basis for the work.
I suggest splitting implementation into coordinated PRs rather than coupling it to #5962:
The transactional point you called out is important: per-frame metadata must advance only with a successful frame transaction, consistent with
frame_nlocs/frame_system_idsand map-full retry behavior.There is no need to repurpose dpdata's reserved
find_*fields. The metadata should encode source availability facts, while DeepMD keeps model/loss policy on the consumer side.Please proceed with the schema + producer/consumer PRs and link them here. A minimal, additive schema with strict version/length/ID validation is preferable to trying to redesign the full LMDB format in the first change.
Agent: ChatGPT
Model: GPT-5.6 Sol