Kernel tuning

As explained in LLMs in Production

Kernel tuning is the selection and optimization of processor kernels to minimize execution time for a specific workload and hardware configuration.

A kernel is a small processor program. Kernel tuning selects and optimizes kernels, their order, and execution configuration to minimize runtime on particular hardware and workloads. section 6.1.1

For convolution, competing algorithms are tested against GPU type, input and filter sizes, tensor layout, batch size, and related deployment characteristics. section 6.1.1

Practical implications:

• Optimizing for a specific neural network and platform improves resource use and performance, but requires uncommon hardware and machine-learning expertise. section 6.1.1
• Treat tuning as platform-specific rather than universally optimal: changing shapes, layouts, batch sizes, or GPU targets can change the best kernel. section 6.1.1

Compiling an LLM for a target GPU

A TensorRT workflow traces a GPT-2 model, declares static or dynamic input shapes and tensor data types, and compiles it for GPU execution. section 6.1.1

Figure 6.2 An example of an unoptimized network compared to the same network optimized using graph optimization. CBR is a NVIDIA fused layer kernel that simply stands for Convolution, Bias, and ReLU. See the following NVIDIA blog post for reference: https://mng.bz/PNvw.

Compare an unoptimized network with graph optimization that fuses convolution, bias, and ReLU into a CBR kernel. section 6.1.1

How LLMs in Production covers it

How deep it goes

Explains kernel selection and benchmarking factors, but stops short of implementation details for thread, register, and shared-memory optimization.

Examples

Uses convolution algorithm selection and a TensorRT GPT-2 compilation workflow with configured input shapes and data types.

What it assumes

Assumes basic deep-learning and processor concepts while defining kernels and identifying hardware and model expertise requirements.

What you can do after

Apply it via PyTorch 2.x compilation or TensorRT to optimize models for selected hardware and input configurations.

How it connects

Connects model compilation with convolution, graph optimization, GPU execution, and vendor kernel libraries.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.