Tensor fusion

As explained in LLMs in Production

Tensor fusion is the merging of multiple tensor-operation kernels into a combined operation to reduce launches and improve memory efficiency.

Tensor fusion combines multiple tensor-operation kernels into one operation. It targets workloads where fast GPU computation is undermined by repeated CUDA kernel launches, tensor transfers, and memory-bandwidth pressure. section 6.1.1

A fused neural-network operation

A fully connected layer can combine matrix multiplication, bias addition, and ReLU into a composite kernel. Convolution, bias, and ReLU provide another fused-layer pattern. section 6.1.1

Figure 6.2 An example of an unoptimized network compared to the same network optimized using graph optimization. CBR is a NVIDIA fused layer kernel that simply stands for Convolution, Bias, and ReLU. See the following NVIDIA blog post for reference: https://mng.bz/PNvw.

Compare an unoptimized network with a graph-optimized version using a fused Convolution, Bias, and ReLU kernel. section 6.1.1

How LLMs in Production covers it

How deep it goes

Explains the launch and memory bottleneck and gives fused-operation examples, but stops short of implementation details.

Examples

Uses fused matrix multiplication, bias addition, and ReLU, plus a Convolution-Bias-ReLU graph-optimization figure.

What you can do after

Evaluate how combining tensor operations can reduce launches and improve memory efficiency on GPU workloads.

How it connects

Connects computation graphs, kernel tuning, CUDA kernel launches, memory bandwidth, graph optimization, and tensor parallelization.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.