Kernel fusion

As explained in Domain-Specific Small Language Models

An optimization that combines adjacent operators into one to reduce overhead and improve inference latency.

Kernel fusion combines adjacent operators into one optimized operator. By reducing intermediate operations and execution overhead, it can improve inference latency while preserving model behavior. section 4.3.3 section 10.4

Fusion operates on a computation graph rather than changing the model’s intended calculation. A graph optimizer can replace consecutive nodes with a semantics-preserving fused node, either automatically or through a custom operator. section 10.3 section 10.4

Practical fusion work typically includes these checks:

• Identify expensive or frequently executed operator sequences through profiling. section 10.4
• Replace compatible nodes with a fused implementation, such as combining EyeLike and Add into AddEyeLike. section 10.3
• Compare original and optimized outputs before claiming a speedup. section 10.3
• Benchmark representative inputs and deployment hardware because the benefit depends on execution costs. section 10.3 section 4.3.3

Fusing EyeLike and Add

An ONNX graph containing EyeLike followed by Add can replace both nodes with AddEyeLike. In the demonstrated benchmark, 10,000 runs took 1.142791 seconds before fusion and 0.863603 seconds afterward, while the maximum output difference was zero. section 10.3

Figure 10.12 Comparing part of the GPT-2 ONNX-converted model’s graph, before and after optimization, showing Skip Layer Normalization Fusion applied

Inspect a GPT-2 ONNX graph before and after Skip Layer Normalization Fusion to see how multiple transformer operations become one optimized operation. section 10.4

How Domain-Specific Small Language Models covers it

How deep it goes

Defines graph-level fusion, demonstrates custom-node implementation and validation, and stops short of deriving kernel-level hardware behavior.

Examples

Uses EyeLike-plus-Add fusion, Skip Layer Normalization Fusion, GPT-2 ONNX graphs, output comparison, and timing benchmarks.

What it assumes

Assumes familiarity with computation graphs and inference, while demonstrating ONNX nodes, custom operators, evaluation, and benchmarking.

What you can do after

Diagnose and evaluate fusion with ONNX Runtime, custom OpRun implementations, output checks, profiling, and representative benchmarks.

How it connects

Connects operator fusion with graph optimization, profiling, custom operators, ONNX Runtime, quantization, and transformer execution providers.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.

Compare how 2 books explain kernel fusion →

Related concepts

• FlashAttentionused by · FlashAttention is an exact, input-output-aware attention algorithm that tiles computation to reduce memory traffic and avoid materializing the full attention matrix in high-bandwidth memory.
• memory boundaddresses · A memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.
• hardware registeruses · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.