appendix-c
appendix C Troubleshooting and development practices
C.1 Troubleshooting
This section collects the failures that bite most often, grouped by where they show up: installation, compilation, runtime, PyTorch, distributed execution, and performance.
C.1.1 CUDA installation problems
Problem: nvcc: command not found
Solution:
# Add CUDA to PATH export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH # Make permanent (add to ~/.bashrc) echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc source ~/.bashrc
Problem: CUDA error: no CUDA-capable device detected
Solution:
# Check GPU is detected lspci | grep -i nvidia # Check NVIDIA driver nvidia-smi # If the driver is missing, install the recommended Ubuntu package sudo ubuntu-drivers install sudo reboot
C.1.2 Compilation errors
Problem: fatal error: cuda_runtime.h: No such file or directory
Solution: Ensure that the CUDA Toolkit is installed and nvcc is in your PATH (see appendix A, section A.2.1).
Problem: fatal error: mpi.h: No such file or directory (chapter 10)
Solution:
sudo apt-get install -y openmpi-bin libopenmpi-dev
Problem: undefined reference to cublasCreate
Solution: Add -lcublas to your nvcc command:
nvcc your_file.cu -o output -lcublas
C.1.3 Runtime errors
Problem: CUDA error: out of memory
Solution: