12 Parallel matrix multiplication
This chapter covers
- Arithmetic intensity applied to matrix multiplication
- Loading tiles from global memory into shared memory
- Tiled matrix multiplication and why it is faster
- Using cuBLAS instead of writing your own kernel
Every time you ask a large language model a question, a GPU somewhere multiplies two large matrices. Rendering a frame of a 3D game is matrix multiplication, and so is every training step of a neural network. It is among the most common computations on a GPU, and a naive implementation of it spends nearly all of its time waiting for data.
A modern GPU can perform tens of trillions of multiply-adds per second. But global memory can only deliver data at a fraction of that speed. It's like a grading room with a hundred students, but only one narrow doorway to the filing cabinet where all the data is kept. The students grade fast, but they're starved for data. The trick is to put a small whiteboard right next to each cluster of desks so the students can copy down what they need without walking back to the filing cabinet every time. That whiteboard is shared memory, and that trick is called tiling.
In this chapter, we’ll set a naive kernel that makes a billion trips to global memory against a tiled version that cuts that traffic by 32 times. We’ll also look at why production code hands the job to cuBLAS.