liveBook · Manning

FlashAttention

FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.

How 3 Manning books explain it

CUDA for LLMs

FlashAttention-2 is a version of FlashAttention that improves GPU utilization through reduced non-matrix-multiplication work and improved parallelism and work partitioning.

How it is used
Uses fused, tiled attention to reduce HBM traffic from large intermediate matrices at long sequence lengths.
How deep it goes
Builds a fused kernel through work distribution and nested loops, then compares version-level optimizations and benchmark limitations.
Examples
Benchmarks a WMMA kernel against naive CUDA and PyTorch attention on an RTX 3090 across three sequence lengths.
What you can do after
Build a fused WMMA attention kernel from scratch and evaluate its correctness and performance against a naive baseline.

Read how CUDA for LLMs explains FlashAttention →

GPU Programming with Triton

FlashAttention is an IO-aware exact attention algorithm that tiles computation to avoid materializing the full attention matrix in global memory.

How it is used
Accelerates transformer attention in LLM inference by reducing global-memory traffic.
How deep it goes
Details tiled query/key/value processing and online softmax, while deferring a complete kernel implementation.
Examples
Uses an attention-fusion figure to depict reduced memory traffic and idle time.
What you can do after
Understand how tiled fusion cuts global-memory traffic and supports stable online softmax accumulation.

Read how GPU Programming with Triton explains FlashAttention →

Transformers in Action

FlashAttention is an exact attention-computation technique that tiles queries, keys, and values to reduce memory traffic and linearize memory usage.

How it is used
Deploys transformer attention more efficiently on resource-constrained hardware and longer input sequences.
How deep it goes
Explains tiled computation, memory movement, normalization, complexity, speedups, and FlashAttention-3 hardware features.
Examples
Demonstrates blockwise Q-K streaming, incremental output accumulation, and a Transformers configuration enabling FlashAttention-3.
What you can do after
Operate/configure FlashAttention-3 in a Transformers model and evaluate its memory and throughput benefits.

Read how Transformers in Action explains FlashAttention →

Also covered in

What to know first

Related concepts

Questions about FlashAttention

What is FlashAttention?

FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.

Which Manning books explain FlashAttention?

CUDA for LLMs, GPU Programming with Triton and Transformers in Action.

What should I know before learning FlashAttention?

Manning books that teach FlashAttention usually introduce attention, matrix-matrix multiplication, quantization and shared memory first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.