Quadratic growth means that full self-attention costs increase proportionally to the square of sequence length, making very long contexts a computational bottleneck. section 3.1.1
Common responses to quadratic attention costs include:
- Use sparse or local attention patterns to restrict which tokens interact. section 3.1.1
- Apply low-rank or kernel approximations, memory layers, or retrieval-augmented approaches. section 3.1.1
- Consider recurrent or state-space architectures such as RWKV, Mamba, or RetNet, while recognizing their reported performance trade-offs. section 3.1.1
- Reduce runtime or memory pressure with gradient checkpointing and mixed-precision training. section 3.1.1






