liveBook · Manning

Tiling

As explained in Grokking GPU Programming

Tiling is a technique that partitions data or computation into smaller blocks to enable reuse and reduce memory traffic.

A matrix multiplication kernel can spend much of its time fetching the same values from global memory. Tiling divides the work into smaller blocks so threads can copy a chunk into shared memory, reuse it, and fetch the next chunk only when needed. section 12.0 section 12.2

Think of global memory as a distant filing cabinet and shared memory as a whiteboard beside a study group. Instead of every student repeatedly fetching the same values, the group copies a few values onto the whiteboard and shares them. section 12.2

For tiled matrix multiplication, each thread block computes one tile of the output matrix C:

  • The block loads a matching chunk of A and a matching chunk of B into shared memory. section 12.2
  • Threads synchronize, then use the shared values to accumulate their parts of the output. section 12.8
  • Threads synchronize again before replacing the shared tiles with the next pair of chunks. After all chunks are processed, valid output elements are written to global memory. section 12.8
Figure 12.6 The big picture of tiling. C is partitioned into T×T tiles, one per thread block; Block(m,n) owns the highlighted tile. To compute it, the block walks tile-row m of A and tile-column n of B, loading one T×T chunk of each into shared memory, computing on the pair, and moving on to the next chunk.

Follow one output tile as its thread block works through corresponding chunks of A and B. section 12.2

Both synchronization points matter. Without the first, a thread could read a tile before another thread finishes loading it. Without the second, a thread could overwrite a tile while another is still using it. Threads outside matrix boundaries load zeros, and the final write checks the output bounds. section 12.8

Counting memory requests

For a 1024×1024 matrix A multiplied by a 1024×512 matrix B, a 32×32 tile reduces requested global memory traffic from about 4 GB to 128 MB. Each element of A is requested 16 times rather than 512; each element of B is requested 32 times rather than 1,024. The 32-fold traffic reduction is for this comparison, not a promise of a 32-fold speedup. section 12.8

Tile size must fit the kernel’s resources. In the one-thread-per-output-element kernel, a 32×32 block uses 1,024 threads; a 64×64 block would exceed the stated 1,024-thread limit. Tiling also helps matrix transpose stage data so that both global reads and writes are coalesced. section 12.8 section 11.14

How Grokking GPU Programming covers it

How deep it goes
Works through tiled matrix multiplication, including block mapping, boundary checks, synchronization, traffic counts, and tile-size limits.
Examples
Uses a study-group whiteboard analogy and compares naive and 32×32-tiled multiplication of 1024×1024 and 1024×512 matrices.
What you can do after
Builds a tiled matrix multiplication kernel from scratch using cooperative loads, synchronization, accumulation, and boundary checks.
How it connects
Connects tiling to shared memory, thread synchronization, matrix multiplication, matrix transpose, and coalesced memory access.

Also explained in

Compare how 3 books explain tiling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.