Skip to content

Fix scalar CUDA hashing and enable it for CUDA tensor dumps - #158

Open
Essoz wants to merge 1 commit into
mainfrom
codex/cuda-scalar-hashing
Open

Essoz wants to merge 1 commit into
mainfrom
codex/cuda-scalar-hashing

Conversation

@Essoz

@Essoz Essoz commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

CUDA tensor dumps always passed with_cuda=False, so they never used the existing CUDA hash implementation. Enabling that backend also exposed a scalar bug: unpacking a 0-D tensor's shape raised ValueError.

This change represents scalars as one-element rows before CUDA hashing and routes CUDA tensor dumps to that backend. CPU and MPS tensors retain the CPU path. It preserves the existing quantization and hash recurrence, propagates backend failures, and adds explicit contracts to the changed functions. dump_tensor returns object because its unchanged statistics path can return application-cached metadata without validating its type.

Validation

Baseline: 6dccb47; candidate: b10876dd9610284b3c6defd87d081c4d12184d02.

  • Actual SDSC Expanse GPU jobs: baseline 55324509, candidate 55324590; NVIDIA H100 80GB HBM3, Python 3.12.3, PyTorch 2.16.0.dev20261012+cu130, Numba 0.63.1. The scalar torch.tensor(1.0, device="cuda") changed from the shape-unpacking exception to the expected hash 100000001. All 9 regression tests passed on the H100.
  • Tests compare CUDA with CPU and an independent signed-64 recurrence for scalar dtypes, vectors, matrices, 3-D tensors, noncontiguous tensors, and an empty vector. They check actual dumper routing, values, RNG states, gradients, exception propagation, and unchanged dump modes.
  • The CPU-executable routing regression fails against the baseline (with_cuda=False) and passes against the candidate. Local relevant suites: 53 passed, 4 CUDA-only tests skipped.
  • Configured Ruff, mypy, Black, and isort hooks pass. The new test module passes stricter mypy with real Torch stubs. The same stricter check on the two legacy modules reports 22 baseline errors → 20 candidate errors, none added.

Reproduce GPU validation on an allocated CUDA node:

python -m unittest tests.test_cuda_hashing_contracts -v

Local validation also includes tests/test_metadata_classification.py, tests/test_constructor_snapshot.py, and tests/test_trace_integrity.py. GPU logs and preflight JSON are under /home/yjiang21/traincheck-experiments/runs/checking-20261027/, with prefixes preflight-55324509 and cuda-contracts-55324590.

Scope and limitations

The historical algorithm returns only the first row's hash after flattening leading dimensions; this PR preserves that behavior. Empty vectors still hash to zero. Zero-row matrices remain unsupported. This is a correctness and backend-enablement change; no speedup is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant