liveBook · Manning

FlashAttention

As explained in GPU Programming with Triton

FlashAttention is an IO-aware exact attention algorithm that tiles computation to avoid materializing the full attention matrix in global memory.

FlashAttention is an exact, IO-aware attention algorithm for transformer models. It fuses matrix multiplication, softmax normalization, and value weighting into one tiled kernel instead of materializing the full attention matrix in global memory. section 3.3 section 3.7

Each program instance processes a tile of query vectors, iterates over key and value tiles, and accumulates weighted outputs in shared memory and registers. This reduces global-memory accesses from quadratic to linear in sequence length; careful online accumulation of softmax statistics also improves numerical stability. Reported speedups over standard implementations are 2× to 4×. section 3.3

Figure 3.5 Kernel fusion in attention reduces memory traffic and idle time.
CH03 F05 Fartale

The figure summarizes how fusion in attention reduces memory traffic and idle time. section 3.3

How GPU Programming with Triton covers it

How deep it goes
Details tiled query/key/value processing and online softmax, while deferring a complete kernel implementation.
Examples
Uses an attention-fusion figure to depict reduced memory traffic and idle time.
What you can do after
Understand how tiled fusion cuts global-memory traffic and supports stable online softmax accumulation.
How it connects
Connects kernel fusion, tiling, matrix multiplication, softmax normalization, and value weighting.

Also explained in

Compare how 3 books explain FlashAttention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.