chapter eight

8 Flash Attention

 

This chapter covers

  • Why standard attention becomes memory-bound and how to spot that shift
  • Core ideas behind Flash Attention: tiling, online softmax, and SRAM residency
  • Restructuring loops and work distribution so tiles cooperate efficiently
  • Building a fused Flash Attention kernel with WMMA tensor cores
  • Evaluating Flash Attention across versions and hardware generations
  • Deciding when Flash Attention is the right optimization for your model

Tensor cores gave us raw compute throughput. But attention has a different bottleneck: the sheer volume of data shuffled between on-chip SRAM and off-chip high-bandwidth memory (HBM).

8.1 From compute-bound to memory-bound

In the last chapter, we explored tensor cores and took GEMM from 43 to 618 TFLOPS on an H100. Our focus was almost entirely on optimizing raw arithmetic, ensuring that every GPU cycle was spent doing useful math. But as we move into optimizing complex operations like the attention mechanism, we encounter a different kind of performance barrier.

Standard self-attention’s problem is memory traffic. It requires computing a large intermediate matrix, often called the attention score matrix, which must be written out to and then read back from HBM. As the context length of our models grows, this matrix becomes enormous, and the cost of moving that data completely overshadows the cost of the actual computation. Our optimization problem shifts from compute-bound to memory-bound.

8.1.1 The HBM bottleneck in standard attention

8.2 Designing Flash Attention

8.2.1 Building intuition: Why this problem is harder than it looks

8.2.2 The Flash Attention solution: Tiling and online softmax

8.2.3 Tiling: Breaking the problem down

8.2.4 Online softmax: Computing global statistics iteratively

8.3 Implementing Flash Attention

8.3.1 Dissecting the naive implementation

8.3.2 Understanding the work distribution strategy

8.3.3 The three-level loop hierarchy

8.3.4 The fused Flash Attention kernel

8.3.5 Benchmarking our kernel

8.4 The evolution of Flash Attention