chapter four
4 Training and verifying neural nets in raw CUDA
This chapter covers
- The challenge of building and verifying a neural network from scratch for reproducibility
- Establishing a “gold standard” implementation in PyTorch
- Using NumPy and C to peel back layers of abstraction
- Building a naive CUDA MLP from our C blueprint
- Optimizing the CUDA implementation with the cuBLAS library
- Techniques for debugging and profiling low-level GPU code
4.1 Level 1: The PyTorch approach
In our journey from high-level frameworks to bare-metal CUDA, we need a reliable point of reference: a “gold standard” against which we can verify every subsequent implementation. For this, we turn to PyTorch. Its combination of a high-level API, automatic differentiation, and highly optimized, battle-tested backend libraries makes it the perfect candidate for our trusted implementation.