liveBook · Manning

Inference

As explained in Machines that Think

The derivation of conclusions or outputs from learned parameters and supplied input during model execution.

Inference is the use of a trained model to produce outputs from new inputs. It differs from training, when parameters are repeatedly updated to reduce error. During inference, those parameters remain fixed, and computation typically requires only a forward pass through the model. section 4.6.1

In a language model, the current input and learned weights jointly shape the response. As tokens pass through the model’s layers, their representations are transformed, making some continuations more likely than others. Knowledge acquired during training influences these predictions through the weights rather than through a database lookup. section 6.4.1

A question about a recent event

If a relevant fact is neither encoded in the model’s parameters nor supplied in the prompt, it cannot influence the answer. A model therefore cannot use an event that occurred after training unless the relevant information is provided as input. Likewise, apparent continuity in a conversation depends on the surrounding system supplying earlier turns again. section 6.4.1

Inference does not normally incorporate experience into the model’s parameters. Online learning updates parameters as new data arrives, but those updates are still training; execution and adjustment remain distinct. A fluent response should not by itself be treated as evidence of human-like understanding. section 5.4.4 section 1.7

Figure 5.4 Human cognition continuously updates through interaction with the world, so learning and action form a single loop. Artificial neural networks separate learning from use: parameters are updated during training, after which the trained model performs inference without incorporating new experience.

The diagram contrasts continuous human learning and action with the separation of neural-network training from inference. section 5.4.4

Although inference is usually less expensive than training, serving a model creates ongoing costs. Each query requires computation, and real-time expectations limit how much work can be delayed or batched. Peak demand, accessible model memory, concurrent users, and regional replicas all affect infrastructure needs. Response time, throughput, and operating cost consequently shape how models are integrated into products. section 4.6.1 section 4.6.2

How Machines that Think covers it

How deep it goes
Explains forward passes and language-model generation conceptually, then examines information limits and serving costs without implementation detail.
Examples
Uses post-training events, omitted documents, and prior exchanges to illustrate information unavailable during inference unless supplied as input.
What you can do after
Evaluate model answers and deployments by recognizing input limits, fixed parameters, latency, and ongoing operating costs.
How it connects
Connects inference with training, neural networks, attention, online learning, and continual learning.

Also explained in

Compare how 4 books explain inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.