chapter four

4 Training and verifying neural nets in raw CUDA

 

This chapter covers

  • The challenge of building and verifying a neural network from scratch for reproducibility
  • Establishing a “gold standard” implementation in PyTorch
  • Using NumPy and C to peel back layers of abstraction
  • Building a naive CUDA MLP from our C blueprint
  • Optimizing the CUDA implementation with the cuBLAS library
  • Techniques for debugging and profiling low-level GPU code

4.1 Level 1: The PyTorch approach

In our journey from high-level frameworks to bare-metal CUDA, we need a reliable point of reference: a “gold standard” against which we can verify every subsequent implementation. For this, we turn to PyTorch. Its combination of a high-level API, automatic differentiation, and highly optimized, battle-tested backend libraries makes it the perfect candidate for our trusted implementation.

4.1.1 PyTorch implementation: Setup, training, and benchmarking

4.2 Level 2: Backpropagation theory

4.2.1 Building intuition: From slopes to gradients

4.2.2 Scaling to vectors and matrices

4.2.3 Layer-by-layer gradient derivations

4.2.4 Common pitfalls and debugging tips

4.2.5 Vectorized implementation for batches

4.3 Level 3: The NumPy approach

4.3.1 Manual implementation in NumPy

4.3.2 Verification technique: Isolate and conquer

4.3.3 Performance

4.4 Level 4: The C approach

4.4.1 Setup: Headers, constants, and data structures

Summary