liveBook · Manning

Kernel fusion

Also called operation fusion, operator fusion

Kernel fusion is an optimization that combines multiple computational operations into fewer execution units or passes, reducing overhead and often intermediate data movement.

How 4 Manning books explain it

GPU Programming with Triton

Kernel fusion is an optimization technique that combines multiple operations into fewer GPU kernel launches to reduce intermediate data movement and launch overhead.

How it is used
Uses kernel fusion to reduce intermediate global-memory traffic and launch overhead in GPU workloads.
How deep it goes
Works through memory-traffic counts and a fused Triton softmax, including masking, reduction scope, launch configuration, and register-pressure tradeoffs.
Examples
Uses SiLU followed by multiplication and compares unfused and fused softmax implementations.
What you can do after
Apply it via Triton to fuse a single-tile softmax while handling masked lanes and choosing launch parallelism.

Read how GPU Programming with Triton explains kernel fusion →

Domain-Specific Small Language Models

An optimization that combines adjacent operators into one to reduce overhead and improve inference latency.

How it is used
Uses fusion to improve inference latency by combining adjacent operators into optimized operations.
How deep it goes
Works through custom ONNX-node construction, output checks, and timing, but does not detail GPU kernel implementation.
Examples
Demonstrates an EyeLike-plus-Add custom node and identifies Skip Layer Normalization Fusion and Fast GELU Fusion in transformer optimization.
What you can do after
Build it from scratch with ONNX OpRun, then compare outputs and benchmark execution with ReferenceEvaluator and timeit.

Read how Domain-Specific Small Language Models explains kernel fusion →

CUDA for LLMs

Operator fusion combines multiple computational operations into a single execution unit to reduce overhead or intermediate data movement.

How it is used
Reduces kernel launches and intermediate global-memory traffic in deep-learning workloads.
How deep it goes
Traces CUDA implementations through shared-memory loading, reductions, and writes, while noting tradeoffs against production kernels.
Examples
Uses softmax, LayerNorm followed by ReLU, bias-plus-ReLU GEMM epilogues, and tiled attention.
What you can do after
Build a design for a fused CUDA ReLU-after-LayerNorm kernel using row-wise reductions and minimizing global-memory traffic.

Read how CUDA for LLMs explains kernel fusion →

Deep Learning with PyTorch, Second Edition

An optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.

How it is used
Accelerates PyTorch model execution during production inference by reducing execution overhead.
How deep it goes
Explains operation combination, memory effects, compiled traces, and GPU kernels without detailing fusion compiler internals.
Examples
Uses cosine and ReLU, matrix multiplication with addition, compiled model traces, and optimized linear-layer kernels.
What you can do after
Apply it through torch.compile and diagnose its effect by comparing profiler traces before and after compilation.

Read how Deep Learning with PyTorch, Second Edition explains kernel fusion →

Also covered in

What to know first

Related concepts

Questions about kernel fusion

What is kernel fusion?

Kernel fusion is an optimization that combines multiple computational operations into fewer execution units or passes, reducing overhead and often intermediate data movement.

Which Manning books explain kernel fusion?

GPU Programming with Triton, Domain-Specific Small Language Models, CUDA for LLMs and Deep Learning with PyTorch, Second Edition.

What should I know before learning kernel fusion?

Manning books that teach kernel fusion usually introduce attention, language model, matrix-matrix multiplication and tiling first.

What is kernel fusion a kind of?

Kernel fusion is a kind of compiler optimization. Compiler optimization is a transformation performed during compilation to improve a program’s execution efficiency or resource use while preserving its defined behavior.

What are the parts of kernel fusion?

Kernel fusion includes epilogue.

What is kernel fusion also called?

Kernel fusion is also called operation fusion or operator fusion.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.