The NCCL (NVIDIA Collective Communications Library) package is often indicated by the error message RuntimeError: Distributed package doesn't have NCCL built in if it is not installed or configured correctly in your system.
NVIDIA created the NCCL software library to provide effective multi-GPU and multi-node communication. Distributed deep learning frameworks like PyTorch and TensorFlow frequently use them.
Windows doesn't support NCCL as a backend. Therefore, if you are working on Windows and encounter this issue, you can resolve it by following these instructions.
One of the ways is that you add this to your main Python script.
import osos.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"
Another way to solve this is by making the following changes in your launch.json file.
"env": {"PL_TORCH_DISTRIBUTED_BACKEND":"gloo"},
You have to set the environmental variable PL_TORCH_DISTRIBUTED_BACKEND = gloo, such that now your script will use the gloo backend instead of NCCL.
Other methods to resolve this issue includes:
Check NCCL installation: Ensure that NCCl is installed on your system. You can do this by running the following command.
nccl --version
In case it is not installed you have to set it up.
Check environment variables: Make that the required environment variables are correctly configured. You must specifically provide the path where NCCL is installed in the LD_LIBRARY_PATH variable.
export LD_LIBRARY_PATH=/usr/local/nccl/lib/:$LD_LIBRARY_PATH
You have to update the path to your NCCl installation location.
Rebuild PyTorch or related packages: You might need to rebuild PyTorch or any other deep learning framework if you're using NCCL after installing NCCL. To rebuild or reinstall the package, you can follow the directions in the documentation of the relevant framework.
Verify GPU drivers: Ensure your computer has the necessary GPU drivers installed. For NCCL to work appropriately, suitable GPU drivers are needed. The recommended GPU driver version for your particular GPU model can be found in the NVIDIA documentation.
Compatibility check: Confirm that the installed NCCL and deep learning framework versions are compatible. Some framework versions can have special specifications or dependencies on the NCCL version. For compatibility details, consult the release notes or documentation for the framework.
You should be able to fix the RuntimeError: Distributed package doesn’t have NCCL built in error by carrying out the actions listed above.
Free Resources