chapter eleven

11 Shared memory

 

This chapter covers

  • The GPU memory hierarchy
  • How to declare and use shared memory
  • The 3-Step Recipe, the pattern behind this book’s shared memory algorithms
  • Bank conflicts and the padding trick that fixes column access
  • Tiling and transposing a matrix with a padded shared memory tile

Imagine you’ve written a matrix multiplication kernel. It produces the correct result. But suppose it takes 850 milliseconds on a large matrix. Your colleague’s version takes 45 milliseconds with the same input and output, so it’s ≈19 times faster (850 ÷ 45 = 18.9).

The difference is where the data lives during computation. Your slow kernel reads every value from the GPU’s global memory. Your colleague’s kernel loads data into a small, fast memory right next to the processing cores, and reads from there instead.

When your kernel is slow

Introducing shared memory

The 3-Step Recipe

Producers and consumers

Applying the 3-Step Recipe

Bank conflicts

Predicting bank conflicts from the access pattern

Avoiding bank conflicts with padding

A first taste of tiling: transpose

Common mistakes

Mistake 1: Bank conflicts in column access

Mistake 2: Exceeding shared memory limits

Mistake 3: Reading uninitialized shared memory

Summary