Tiling
Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
CUDA for LLMs
Tiling is a computation technique that partitions data into smaller blocks to improve locality and reuse.
- How it is used
- Optimizes GPU matrix multiplication and attention by keeping reusable tiles and intermediates near computation.
- How deep it goes
- Traces tiled GEMM from shared-memory scheduling and synchronization through register micro-tiles and warp-level WMMA fragments.
- Examples
- Uses 128 × 128 GEMM block tiles, 8 × 8 thread micro-tiles, and tiled Flash Attention.
- What you can do after
- Build tiled CUDA kernels using shared-memory staging, synchronization, and register accumulation.
Grokking GPU Programming
Tiling is a technique that partitions data or computation into smaller blocks to enable reuse and reduce memory traffic.
- How it is used
- Reduces repeated global-memory requests by staging matrix chunks for cooperative reuse in shared memory.
- How deep it goes
- Works through tiled matrix multiplication, including block mapping, boundary checks, synchronization, traffic counts, and tile-size limits.
- Examples
- Uses a study-group whiteboard analogy and compares naive and 32×32-tiled multiplication of 1024×1024 and 1024×512 matrices.
- What you can do after
- Builds a tiled matrix multiplication kernel from scratch using cooperative loads, synchronization, accumulation, and boundary checks.
Transformers in Action
Tiling is a technique that partitions a large computation into smaller subproblems whose data can be reused in faster local memory.
- How it is used
- Uses tiling to reduce GPU memory traffic in matrix multiplication and attention.
- How deep it goes
- Explains cooperative tile loading, blockwise softmax, and hardware overlap without developing a kernel from scratch.
- Examples
- Illustrates tiled matrix multiplication and blockwise query-key-value attention.
- What you can do after
- Apply it via FlashAttention-3 by selecting its attention implementation when loading a causal language model.
Also covered in
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- arithmetic intensityArithmetic intensity is the ratio of computational operations performed to bytes of data transferred by a computation.
- attention matrixAn attention matrix is a matrix of scores or weights relating query positions to key positions and indicating their attention relationships.
Related concepts
- shared memoryuses · Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
- FlashAttentionused by · FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.
- hardware registeruses · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.
- CUDA kernelused by · A CUDA kernel is a function launched on a GPU for execution by multiple threads in parallel.
- data reuseproduces · Data reuse is the use of a value in multiple computations without repeatedly fetching it from its original storage.
- memory trafficaddresses · Memory traffic is the amount of data transferred between memory and processing units during computation.
Questions about tiling
What is tiling?
Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
Which Manning books explain tiling?
CUDA for LLMs, Grokking GPU Programming and Transformers in Action.
What should I know before learning tiling?
Manning books that teach tiling usually introduce attention, matrix-matrix multiplication, arithmetic intensity and attention matrix first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


