Kernel fusion
Kernel fusion is an optimization that combines multiple computational operations into fewer executable units to reduce launch overhead and intermediate data movement.
Domain-Specific Small Language Models
An optimization that combines adjacent operators into one to reduce overhead and improve inference latency.
- How it is used
- Improves neural-network and LLM inference by reducing operator and execution overhead in computation graphs.
- How deep it goes
- Defines graph-level fusion, demonstrates custom-node implementation and validation, and stops short of deriving kernel-level hardware behavior.
- Examples
- Uses EyeLike-plus-Add fusion, Skip Layer Normalization Fusion, GPT-2 ONNX graphs, output comparison, and timing benchmarks.
- What you can do after
- Diagnose and evaluate fusion with ONNX Runtime, custom OpRun implementations, output checks, profiling, and representative benchmarks.
Read how Domain-Specific Small Language Models explains kernel fusion →
Deep Learning with PyTorch, Second Edition
An optimization that combines multiple operations into fewer executable units to reduce overhead and improve speed.
- How it is used
- Accelerates PyTorch model execution during production inference by reducing execution overhead.
- How deep it goes
- Explains operation combination, memory effects, compiled traces, and GPU kernels without detailing fusion compiler internals.
- Examples
- Uses cosine and ReLU, matrix multiplication with addition, compiled model traces, and optimized linear-layer kernels.
- What you can do after
- Apply it through torch.compile and diagnose its effect by comparing profiler traces before and after compilation.
Read how Deep Learning with PyTorch, Second Edition explains kernel fusion →
Related concepts
- FlashAttentionused by · FlashAttention is an exact, input-output-aware attention algorithm that tiles computation to reduce memory traffic and avoid materializing the full attention matrix in high-bandwidth memory.
- memory boundaddresses · A memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.
- hardware registeruses · A hardware register is a small, rapidly accessible storage location within a processor or processing device that holds data, addresses, or control information during execution.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.