Code compilation

As explained in LLMs in Production

Compilation is the transformation of high-level program or model code into lower-level code optimized for execution on target hardware.

Compilation lowers high-level model code into machine-level code that better matches target hardware, improving inference speed and potentially reducing cost. A practical pipeline converts PyTorch or TensorFlow models into an intermediate representation such as TorchScript, MLIR, or ONNX, then generates hardware-specific code for GPUs, TPUs, or CPUs. section 6.1.1

The intermediate representation separates framework tooling from hardware-specific compilation:

• Framework developers provide conversion into shared intermediate representations. section 6.1.1
• Hardware vendors provide compilation from those representations to their processors. section 6.1.1
• Existing tools such as llama.cpp and pre-quantized model formats reduce repeated optimization work. section 6.1.1

TensorRT compilation

A TensorRT workflow loads GPT-2 with PyTorch, moves it to CUDA, traces it with TorchScript, and supplies input shapes and tensor types for compilation. Inputs can use static dimensions or bounded dynamic shapes. section 6.1.1

Figure 6.2 An example of an unoptimized network compared to the same network optimized using graph optimization. CBR is a NVIDIA fused layer kernel that simply stands for Convolution, Bias, and ReLU. See the following NVIDIA blog post for reference: https://mng.bz/PNvw.

Graph optimization can replace an unoptimized network with fused operations such as NVIDIA's convolution-bias-ReLU kernel. section 6.1.1

How LLMs in Production covers it

How deep it goes

Explains the framework-to-intermediate-representation-to-machine-code pipeline and demonstrates TensorRT configuration.

Examples

Uses GPT-2 TensorRT tracing, ONNX export commands, llama.cpp, and graph-optimized fused kernels.

What it assumes

Assumes familiarity with high-level model frameworks and hardware targets while explaining intermediate representations from first principles.

What you can do after

Apply it via TensorRT or ONNX export tooling to produce hardware-oriented model artifacts.

How it connects

Connects PyTorch, TensorFlow, TorchScript, MLIR, ONNX, TensorRT, quantization, and hardware-specific inference.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionPyTorch compilation is the transformation of PyTorch computations into an optimized executable representation.
• 

  liveBookmanning.com
  Deep Learning with JAXThe process of converting code into optimized executable form.
• 

  liveBookmanning.com
  Modern AngularCompilation is the process of translating source code or templates into executable code or an intermediate representation.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe process of converting code into a compiled form for execution, used here as a performance and debugging tradeoff in TensorFlow, JAX, and PyTorch.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIModel compilation is the transformation of a trained model into lower-level executable code for faster inference or deployment.
• 

  liveBookmanning.com
  Secrets of the JavaScript Ninja, Second EditionThe process of translating a program or expression into an internal form suitable for execution.

Compare how 7 books explain code compilation →

Related concepts

• Treeliteproduced by · Treelite is a tool that compiles decision-tree ensemble models into portable low-dependency C or Java code.
• compilation testmeasured by · A compilation test checks whether source code, including a macro invocation, compiles successfully or fails as expected.
• Dockerused by · Docker is a platform and ecosystem for building, distributing, and running applications in containers.
• just-in-time compilationincludes · Just-in-time compilation translates or optimizes program code during execution, often using runtime information to generate specialized executable instructions.
• mix environmentconfigured by · A Mix environment is a project configuration context, such as development, test, or production, that determines the compilation target and related build settings.
• generationused by · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.