Inference is the use of a trained model to produce outputs from new inputs. It differs from training, when parameters are repeatedly updated to reduce error. During inference, those parameters remain fixed, and computation typically requires only a forward pass through the model. section 4.6.1
In a language model, the current input and learned weights jointly shape the response. As tokens pass through the model’s layers, their representations are transformed, making some continuations more likely than others. Knowledge acquired during training influences these predictions through the weights rather than through a database lookup. section 6.4.1
A question about a recent event
If a relevant fact is neither encoded in the model’s parameters nor supplied in the prompt, it cannot influence the answer. A model therefore cannot use an event that occurred after training unless the relevant information is provided as input. Likewise, apparent continuity in a conversation depends on the surrounding system supplying earlier turns again. section 6.4.1
Inference does not normally incorporate experience into the model’s parameters. Online learning updates parameters as new data arrives, but those updates are still training; execution and adjustment remain distinct. A fluent response should not by itself be treated as evidence of human-like understanding. section 5.4.4 section 1.7
Figure 5.4 Human cognition continuously updates through interaction with the world, so learning and action form a single loop. Artificial neural networks separate learning from use: parameters are updated during training, after which the trained model performs inference without incorporating new experience.

The diagram contrasts continuous human learning and action with the separation of neural-network training from inference. section 5.4.4
Although inference is usually less expensive than training, serving a model creates ongoing costs. Each query requires computation, and real-time expectations limit how much work can be delayed or batched. Peak demand, accessible model memory, concurrent users, and regional replicas all affect infrastructure needs. Response time, throughput, and operating cost consequently shape how models are integrated into products. section 4.6.1 section 4.6.2



