liveBook · Manning

Inference

Inference is the process of applying a trained model to input data to produce predictions or other outputs.

How 4 Manning books explain it

CUDA for LLMs

Model inference is the execution of a trained machine-learning model on input data to produce predictions or other outputs.

How it is used
Uses autoregressive transformer inference to motivate latency-focused CUDA strategies for single-token decode.
How deep it goes
Traces inference from model phases and PyTorch bindings to a row-wise CUDA GEMV kernel and its memory bottleneck.
Examples
Uses a 64-token, 128-dimensional transformer and a single-token query projection to illustrate inference workloads.
What you can do after
Apply inference via PyTorch and custom CUDA extensions, selecting kernels suited to single-token workloads.

Read how CUDA for LLMs explains inference →

Quantization and Fast Inference

Inference is the execution of a trained machine-learning model to produce outputs from inputs.

How it is used
Frames inference as a production deployment problem constrained by data movement, latency, energy, and cost.
How deep it goes
Explains bandwidth costs and deployment comparisons, then checks runtime execution and measurement conditions.
Examples
Compares Llama-2-7B CPU and GPU token costs and describes GPU language-model and on-device vision deployments.
What you can do after
Operate and configure quantized inference while checking execution, throughput, and user-facing measurements.

Read how Quantization and Fast Inference explains inference →

Machines that Think

The derivation of conclusions or outputs from learned parameters and supplied input during model execution.

How it is used
Distinguishes model execution from training and relates inference to answer generation and deployment economics.
How deep it goes
Explains forward passes and language-model generation conceptually, then examines information limits and serving costs without implementation detail.
Examples
Uses post-training events, omitted documents, and prior exchanges to illustrate information unavailable during inference unless supplied as input.
What you can do after
Evaluate model answers and deployments by recognizing input limits, fixed parameters, latency, and ongoing operating costs.

Read how Machines that Think explains inference →

Build Applications with Local AI Models on a Mac

Model inference is the process of applying a trained model to input data to produce an output.

How it is used
Uses language model inference to generate responses in a local voice assistant pipeline.
How deep it goes
Describes inference as a pipeline stage and deployment choice without explaining model internals.
Examples
Uses a voice-enabled chatbot, an airplane-mode test, and a lightweight agent connected to an LLM backend.
What you can do after
Evaluate/understand where inference runs and how that affects offline operation and privacy.

Read how Build Applications with Local AI Models on a Mac explains inference →

Also covered in

What to know first

Related concepts

Questions about inference

What is inference?

Inference is the process of applying a trained model to input data to produce predictions or other outputs.

Which Manning books explain inference?

CUDA for LLMs, Quantization and Fast Inference, Machines that Think and Build Applications with Local AI Models on a Mac.

What should I know before learning inference?

Manning books that teach inference usually introduce training, neural network, attention and batching first.

What is the difference between inference and training?

Inference is the process of applying a trained model to input data to produce predictions or other outputs. Training is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.

What kinds of inference are there?

Kinds of inference include local-model inference.

What is inference part of?

Inference is part of pipeline. A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.