Inference
Inference is the process of applying a trained model to input data to produce predictions or other outputs.
CUDA for LLMs
Model inference is the execution of a trained machine-learning model on input data to produce predictions or other outputs.
- How it is used
- Uses autoregressive transformer inference to motivate latency-focused CUDA strategies for single-token decode.
- How deep it goes
- Traces inference from model phases and PyTorch bindings to a row-wise CUDA GEMV kernel and its memory bottleneck.
- Examples
- Uses a 64-token, 128-dimensional transformer and a single-token query projection to illustrate inference workloads.
- What you can do after
- Apply inference via PyTorch and custom CUDA extensions, selecting kernels suited to single-token workloads.
Quantization and Fast Inference
Inference is the execution of a trained machine-learning model to produce outputs from inputs.
- How it is used
- Frames inference as a production deployment problem constrained by data movement, latency, energy, and cost.
- How deep it goes
- Explains bandwidth costs and deployment comparisons, then checks runtime execution and measurement conditions.
- Examples
- Compares Llama-2-7B CPU and GPU token costs and describes GPU language-model and on-device vision deployments.
- What you can do after
- Operate and configure quantized inference while checking execution, throughput, and user-facing measurements.
Read how Quantization and Fast Inference explains inference →
Machines that Think
The derivation of conclusions or outputs from learned parameters and supplied input during model execution.
- How it is used
- Distinguishes model execution from training and relates inference to answer generation and deployment economics.
- How deep it goes
- Explains forward passes and language-model generation conceptually, then examines information limits and serving costs without implementation detail.
- Examples
- Uses post-training events, omitted documents, and prior exchanges to illustrate information unavailable during inference unless supplied as input.
- What you can do after
- Evaluate model answers and deployments by recognizing input limits, fixed parameters, latency, and ongoing operating costs.
Build Applications with Local AI Models on a Mac
Model inference is the process of applying a trained model to input data to produce an output.
- How it is used
- Uses language model inference to generate responses in a local voice assistant pipeline.
- How deep it goes
- Describes inference as a pipeline stage and deployment choice without explaining model internals.
- Examples
- Uses a voice-enabled chatbot, an airplane-mode test, and a lightweight agent connected to an LLM backend.
- What you can do after
- Evaluate/understand where inference runs and how that affects offline operation and privacy.
Read how Build Applications with Local AI Models on a Mac explains inference →
Also covered in
- LLM Customization and Fine-TuningIn early access (MEAP)
- AI GovernanceIn early access (MEAP)
- Building Reliable AI Systems
- Rearchitecting LLMsIn early access (MEAP)
What to know first
- trainingTraining is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.
- neural networkA neural network is a computational model composed of interconnected, parameterized units or transformations that learn to map inputs to outputs.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- batchingBatching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
Related concepts
- Ollamauses · Ollama is software for downloading, managing, running, and serving language models locally.
- trainingcontrasts with · Training is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.
- pipelinepart of · A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
- local-model inferenceincludes · Local-model inference is the execution of a trained model on hardware or infrastructure controlled by the user or deploying organization rather than through a remote service.
- artificial intelligence systemused by · An artificial intelligence system is a computational system that uses artificial intelligence techniques to perform tasks such as learning, reasoning, interpreting, generating, predicting, or making decisions.
Questions about inference
What is inference?
Inference is the process of applying a trained model to input data to produce predictions or other outputs.
Which Manning books explain inference?
CUDA for LLMs, Quantization and Fast Inference, Machines that Think and Build Applications with Local AI Models on a Mac.
What should I know before learning inference?
Manning books that teach inference usually introduce training, neural network, attention and batching first.
What is the difference between inference and training?
Inference is the process of applying a trained model to input data to produce predictions or other outputs. Training is the process of fitting a machine-learning model to data by adjusting its parameters or learned representations to improve its performance on a task.
What kinds of inference are there?
Kinds of inference include local-model inference.
What is inference part of?
Inference is part of pipeline. A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



