Tiling

As explained in Transformers in Action

Tiling is a technique that partitions a large computation into smaller subproblems whose data can be reused in faster local memory.

Tiling divides a large GPU computation into blocks whose data fits in fast on-chip shared memory. In matrix multiplication, a naïve kernel repeatedly fetches rows and columns from global memory. Loading smaller matrix tiles once and reusing them for multiple calculations reduces redundant transfers and helps keep compute units busy. section 9.4

Tiles work best when threads load nearby addresses together. A GPU warp contains 32 threads; consecutive accesses can be combined into a coalesced memory transaction. Matrix layout therefore matters alongside tile size: a layout that makes the needed elements scattered can waste memory bandwidth or require a transpose. CUDA blocks map naturally to tiles because their threads can cooperate through shared memory. section 9.4

Figure 9.6 Tiled matrix multiplication. Large matrices are divided into tiles that fit into shared memory. Each tile is computed cooperatively, enabling data reuse and coalesced memory access.

The tiled matrix-multiplication diagram shows how cooperating threads reuse matrix blocks held in shared memory. section 9.4

FlashAttention applies the same locality principle to attention. Standard attention materializes an n × n score matrix; FlashAttention instead streams tiles of queries and keys into on-chip SRAM. It computes partial scores, updates softmax normalization incrementally, and accumulates the result with values. This avoids storing the full intermediate matrix in high-bandwidth memory while preserving exact attention rather than approximating it. section 9.4.1

Figure 9.8 Blockwise computation in FlashAttention. Rather than materializing the full n × n attention matrix, the algorithm streams tiles of Q and K into SRAM, applies softmax locally, rescales for global normalization, and accumulates outputs. This avoids expensive HBM traffic.

The blockwise attention diagram shows how streamed tiles contribute to globally normalized outputs without materializing the full score matrix. section 9.4.1

Enable a tiled attention implementation

When loading a causal language model with AutoModelForCausalLM.from_pretrained, set attn_implementation="kernels-community/vllm-flash-attn3". This selects FlashAttention-3; it does not require changing the attention formula. FlashAttention-3 targets NVIDIA Hopper features, including hardware-assisted tile transfers and overlapping matrix multiplication with softmax. section 9.4.1

How Transformers in Action covers it

How deep it goes

Explains memory layouts, coalesced loads, blockwise attention, and incremental normalization, but does not provide a tiled kernel implementation.

Examples

Illustrates tiled matrix multiplication, blockwise attention, and a model-loading option that selects FlashAttention-3.

What it assumes

Introduces GEMM, warps, and memory layouts while relying on the surrounding context of GPU-based transformer attention.

What you can do after

Apply it via Hugging Face AutoModelForCausalLM by selecting the shown FlashAttention-3 implementation.

How it connects

Connects tiling to GEMM, CUDA blocks, shared memory, coalesced memory access, softmax, and FlashAttention.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

• matrix-matrix multiplicationused by · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• FlashAttentionused by · FlashAttention is an exact, input-output-aware attention algorithm that tiles computation to reduce memory traffic and avoid materializing the full attention matrix in high-bandwidth memory.
• shared memoryuses · Shared memory is a memory region or storage area accessible to multiple processes, threads, or other execution components for accessing or exchanging data.
• throughputproduces · The amount of work, data, or output a system processes or completes per unit of time.
• vectorizationused by · Vectorization is the representation of data or computations as vectors or arrays so they can be processed numerically or collectively.
• nested loopimplemented by · A nested loop is a loop placed inside another loop so that the inner loop executes within each iteration of the outer loop.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.