Repository navigation
MultiGPU / End2end test test_get_total_number_of_trainable_parameters in test_utils.py fails / succeeds non-deterministically #332
Description
The test_get_total_number_of_trainable_parameters in test_utils.py test fails from time to time with:
E torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5
E ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
E Last error:
E socketStartConnect: Connect to 10.183.75.12<50669> failed : Software caused connection abort
I'm running pytorch nightly with cuda 12.8 which might have caused this.
Activity
[-]Test test_get_total_number_of_trainable_parameters in test_utils.py fails / succeeds non-deterministically[/-][+]MultiGPU / End2end test test_get_total_number_of_trainable_parameters in test_utils.py fails / succeeds non-deterministically[/+]