8 Flash Attention
This chapter covers
- Why standard attention becomes memory-bound and how to spot that shift
- Core ideas behind Flash Attention: tiling, online softmax, and SRAM residency
- Restructuring loops and work distribution so tiles cooperate efficiently
- Building a fused Flash Attention kernel with WMMA tensor cores
- Evaluating Flash Attention across versions and hardware generations
- Deciding when Flash Attention is the right optimization for your model
Tensor cores gave us raw compute throughput. But attention has a different bottleneck: the sheer volume of data shuffled between on-chip SRAM and off-chip high-bandwidth memory (HBM).
8.1 From compute-bound to memory-bound
In the last chapter, we explored tensor cores and took GEMM from 43 to 618 TFLOPS on an H100. Our focus was almost entirely on optimizing raw arithmetic, ensuring that every GPU cycle was spent doing useful math. But as we move into optimizing complex operations like the attention mechanism, we encounter a different kind of performance barrier.
Standard self-attention’s problem is memory traffic. It requires computing a large intermediate matrix, often called the attention score matrix, which must be written out to and then read back from HBM. As the context length of our models grows, this matrix becomes enormous, and the cost of moving that data completely overshadows the cost of the actual computation. Our optimization problem shifts from compute-bound to memory-bound.