Kernel fusion
Kernel fusion is an optimization that combines multiple computational operations into fewer execution units or passes, reducing overhead and often intermediate data movement.
GPU Programming with Triton
Kernel fusion is an optimization technique that combines multiple operations into fewer GPU kernel launches to reduce intermediate data movement and launch overhead.
- How it is used
- Uses kernel fusion to reduce intermediate global-memory traffic and launch overhead in GPU workloads.
- How deep it goes
- Works through memory-traffic counts and a fused Triton softmax, including masking, reduction scope, launch configuration, and register-pressure tradeoffs.
- Examples
- Uses SiLU followed by multiplication and compares unfused and fused softmax implementations.
- What you can do after
- Apply it via Triton to fuse a single-tile softmax while handling masked lanes and choosing launch parallelism.
Read how GPU Programming with Triton explains kernel fusion →
Domain-Specific Small Language Models
An optimization that combines adjacent operators into one to reduce overhead and improve inference latency.
- How it is used
- Uses fusion to improve inference latency by combining adjacent operators into optimized operations.
- How deep it goes
- Works through custom ONNX-node construction, output checks, and timing, but does not detail GPU kernel implementation.
- Examples
- Demonstrates an EyeLike-plus-Add custom node and identifies Skip Layer Normalization Fusion and Fast GELU Fusion in transformer optimization.
- What you can do after
- Build it from scratch with ONNX OpRun, then compare outputs and benchmark execution with ReferenceEvaluator and timeit.
Read how Domain-Specific Small Language Models explains kernel fusion →
CUDA for LLMs
Operator fusion combines multiple computational operations into a single execution unit to reduce overhead or intermediate data movement.
- How it is used
- Reduces kernel launches and intermediate global-memory traffic in deep-learning workloads.
- How deep it goes
- Traces CUDA implementations through shared-memory loading, reductions, and writes, while noting tradeoffs against production kernels.
- Examples
- Uses softmax, LayerNorm followed by ReLU, bias-plus-ReLU GEMM epilogues, and tiled attention.
- What you can do after
- Build a design for a fused CUDA ReLU-after-LayerNorm kernel using row-wise reductions and minimizing global-memory traffic.
Deep Learning with PyTorch, Second Edition
An optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.
- How it is used
- Accelerates PyTorch model execution during production inference by reducing execution overhead.
- How deep it goes
- Explains operation combination, memory effects, compiled traces, and GPU kernels without detailing fusion compiler internals.
- Examples
- Uses cosine and ReLU, matrix multiplication with addition, compiled model traces, and optimized linear-layer kernels.
- What you can do after
- Apply it through torch.compile and diagnose its effect by comparing profiler traces before and after compilation.
Read how Deep Learning with PyTorch, Second Edition explains kernel fusion →
Also covered in
- The Joy of JavaScript
- Functional Programming in JavaScript
- Quantization and Fast InferenceIn early access (MEAP)
- Transformers in Action
- Grokking Simplicity
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- tilingTiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
Related concepts
- CUDA kernelproduces · A CUDA kernel is a function launched on a GPU for execution by multiple threads in parallel.
- GPU kernelproduces · A GPU kernel is a program or function executed on a graphics processing unit to perform a computation, often in parallel across many threads.
- online softmaxused by · Online softmax is an incremental algorithm that computes softmax by updating normalization statistics as input values or blocks are processed.
- compiler optimizationkind of · Compiler optimization is a transformation performed during compilation to improve a program’s execution efficiency or resource use while preserving its defined behavior.
- FlashAttentionused by · FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.
- epiloguehas part · An epilogue is a postprocessing stage that applies operations such as scaling or activation after a main matrix computation.
Questions about kernel fusion
What is kernel fusion?
Kernel fusion is an optimization that combines multiple computational operations into fewer execution units or passes, reducing overhead and often intermediate data movement.
Which Manning books explain kernel fusion?
GPU Programming with Triton, Domain-Specific Small Language Models, CUDA for LLMs and Deep Learning with PyTorch, Second Edition.
What should I know before learning kernel fusion?
Manning books that teach kernel fusion usually introduce attention, language model, matrix-matrix multiplication and tiling first.
What is kernel fusion a kind of?
Kernel fusion is a kind of compiler optimization. Compiler optimization is a transformation performed during compilation to improve a program’s execution efficiency or resource use while preserving its defined behavior.
What are the parts of kernel fusion?
Kernel fusion includes epilogue.
What is kernel fusion also called?
Kernel fusion is also called operation fusion or operator fusion.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



