appendix-c

appendix C  Troubleshooting and development practices

 

C.1 Troubleshooting

This section collects the failures that bite most often, grouped by where they show up: installation, compilation, runtime, PyTorch, distributed execution, and performance.

C.1.1 CUDA installation problems

Problem: nvcc: command not found

Solution:

# Add CUDA to PATH
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# Make permanent (add to ~/.bashrc)
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >>
~/.bashrc
source ~/.bashrc

Problem: CUDA error: no CUDA-capable device detected

Solution:

# Check GPU is detected
lspci | grep -i nvidia

# Check NVIDIA driver
nvidia-smi

# If the driver is missing, install the recommended Ubuntu package
sudo ubuntu-drivers install
sudo reboot

C.1.2 Compilation errors

Problem: fatal error: cuda_runtime.h: No such file or directory

Solution: Ensure that the CUDA Toolkit is installed and nvcc is in your PATH (see appendix A, section A.2.1).

Problem: fatal error: mpi.h: No such file or directory (chapter 10)

Solution:

sudo apt-get install -y openmpi-bin libopenmpi-dev

Problem: undefined reference to cublasCreate

Solution: Add -lcublas to your nvcc command:

nvcc your_file.cu -o output -lcublas

C.1.3 Runtime errors

Problem: CUDA error: out of memory

Solution:

C.1.4 PyTorch problems

C.1.5 MPI and distributed problems

C.1.6 Performance debugging

C.2 Development best practices

C.2.1 Kernel development workflow

C.2.2 Debugging CUDA kernels

C.2.3 Testing strategy