A GPU accelerates tensor computations for deep learning, complementing CPUs and enabling frameworks to train models on parallel hardware. Frameworks can also distribute work across multiple GPUs or computers. section 3.1
Practical takeaways:
- Use GPU support through a deep-learning framework rather than implementing low-level GPU computation yourself. section 3.1
- Treat a single GPU as an introductory training scale; multi-GPU training is a route to greater scale and speed. section 18.0
- Keep GPUs distinct from TPUs, mixed precision, and quantization: these appear as separate approaches to accelerating or scaling training. section 18.0
From CUDA to framework training
CUDA, released by NVIDIA in 2006, made consumer-GPU neural-network computation possible; researchers began building CUDA-based networks from 2009. Modern frameworks package GPU computation, automatic differentiation, and multi-device distribution together. section 3.1
