Kernel fusion

As explained in Deep Learning with PyTorch, Second Edition

An optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.

Kernel fusion combines multiple operations into one executable kernel, reducing launch and memory overhead. It is especially useful for pointwise operations such as cosine and ReLU, where separate passes can be limited by memory bandwidth. section 17.1.5

Compilation can expose several performance improvements:

• torch.compile can fuse model forwards into compiled regions and generate optimized GPU kernels. section 17.4
• Fusing operations reduces separate kernel invocations, intermediate memory use, and potentially memory-bandwidth pressure. section 17.1.5
• The related optimization of operator reordering can combine an addition with matrix multiplication to improve cache efficiency. section 17.1.5

Compiling a model

Wrap an existing model with torch.compile before inference: model = AutoModelForCausalLM.from_pretrained(checkpoint); model = model.to(device); model = torch.compile(model). The resulting speedup depends on hardware and model architecture, and some cases may become slower. section 17.1.5

Figure 17.5 Trace of the model execution after compilation

A post-compilation execution trace makes fusion visible: forwards appear under a compiled region, alongside fused GPU operations such as optimized linear-layer kernels. section 17.4

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes

Explains operation combination, memory effects, compiled traces, and GPU kernels without detailing fusion compiler internals.

Examples

Uses cosine and ReLU, matrix multiplication with addition, compiled model traces, and optimized linear-layer kernels.

What it assumes

Assumes familiarity with PyTorch models and inference while explaining the motivation from Python interpretation and separate operations.

What you can do after

Apply it through torch.compile and diagnose its effect by comparing profiler traces before and after compilation.

How it connects

Connects kernel fusion with torch.compile, JIT compilation, operator reordering, GPU operations, and performance profiling.

liveBookmanning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsAn optimization that combines adjacent operators into one to reduce overhead and improve inference latency.

Compare how 2 books explain kernel fusion →

Related concepts

• FlashAttentionused by · FlashAttention is an exact, input-output-aware attention algorithm that tiles computation to reduce memory traffic and avoid materializing the full attention matrix in high-bandwidth memory.
• memory boundaddresses · A memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.
• hardware registeruses · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.