FlashAttention
FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.
CUDA for LLMs
FlashAttention-2 is a version of FlashAttention that improves GPU utilization through reduced non-matrix-multiplication work and improved parallelism and work partitioning.
- How it is used
- Uses fused, tiled attention to reduce HBM traffic from large intermediate matrices at long sequence lengths.
- How deep it goes
- Builds a fused kernel through work distribution and nested loops, then compares version-level optimizations and benchmark limitations.
- Examples
- Benchmarks a WMMA kernel against naive CUDA and PyTorch attention on an RTX 3090 across three sequence lengths.
- What you can do after
- Build a fused WMMA attention kernel from scratch and evaluate its correctness and performance against a naive baseline.
GPU Programming with Triton
FlashAttention is an IO-aware exact attention algorithm that tiles computation to avoid materializing the full attention matrix in global memory.
- How it is used
- Accelerates transformer attention in LLM inference by reducing global-memory traffic.
- How deep it goes
- Details tiled query/key/value processing and online softmax, while deferring a complete kernel implementation.
- Examples
- Uses an attention-fusion figure to depict reduced memory traffic and idle time.
- What you can do after
- Understand how tiled fusion cuts global-memory traffic and supports stable online softmax accumulation.
Read how GPU Programming with Triton explains FlashAttention →
Transformers in Action
FlashAttention is an exact attention-computation technique that tiles queries, keys, and values to reduce memory traffic and linearize memory usage.
- How it is used
- Deploys transformer attention more efficiently on resource-constrained hardware and longer input sequences.
- How deep it goes
- Explains tiled computation, memory movement, normalization, complexity, speedups, and FlashAttention-3 hardware features.
- Examples
- Demonstrates blockwise Q-K streaming, incremental output accumulation, and a Transformers configuration enabling FlashAttention-3.
- What you can do after
- Operate/configure FlashAttention-3 in a Transformers model and evaluate its memory and throughput benefits.
Also covered in
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- quantizationModel quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
- shared memoryShared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
Related concepts
- online softmaxuses · Online softmax is an incremental algorithm that computes softmax by updating normalization statistics as input values or blocks are processed.
- tilinguses · Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
- parallelismuses · Parallelism is the execution of multiple independent tasks or computational operations at the same time rather than sequentially.
- bottleneckaddresses · A bottleneck is a constraint, component, or stage that limits the throughput or performance of a larger system or workflow.
Questions about FlashAttention
What is FlashAttention?
FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.
Which Manning books explain FlashAttention?
CUDA for LLMs, GPU Programming with Triton and Transformers in Action.
What should I know before learning FlashAttention?
Manning books that teach FlashAttention usually introduce attention, matrix-matrix multiplication, quantization and shared memory first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


