chapter twelve

12 Parallel matrix multiplication

 

This chapter covers

  • Arithmetic intensity applied to matrix multiplication
  • Loading tiles from global memory into shared memory
  • Tiled matrix multiplication and why it is faster
  • Using cuBLAS instead of writing your own kernel

Every time you ask a large language model a question, a GPU somewhere multiplies two large matrices. Rendering a frame of a 3D game is matrix multiplication, and so is every training step of a neural network. It is among the most common computations on a GPU, and a naive implementation of it spends nearly all of its time waiting for data.

A modern GPU can perform tens of trillions of multiply-adds per second. But global memory can only deliver data at a fraction of that speed. It's like a grading room with a hundred students, but only one narrow doorway to the filing cabinet where all the data is kept. The students grade fast, but they're starved for data. The trick is to put a small whiteboard right next to each cluster of desks so the students can copy down what they need without walking back to the filing cabinet every time. That whiteboard is shared memory, and that trick is called tiling.

In this chapter, we’ll set a naive kernel that makes a billion trips to global memory against a tiled version that cuts that traffic by 32 times. We’ll also look at why production code hands the job to cuBLAS.

Understanding the problem

The solution: tiled matrix multiplication

Tiles: breaking a matrix into bite-sized chunks

Walkthrough of a tiled kernel

Step 1: Load tiles into shared memory

Step 2: Synchronize

Step 3: Compute using shared memory

Step 4: Synchronize again, then repeat

In production: use cuBLAS

Common mistakes

Mistake 1: Missing the second __syncthreads()

Mistake 2: Row-major vs column-major confusion

Mistake 3: Tile size over the limits

Summary