liveBook · Manning

Parameter

As explained in Machines that Think

A value specified or learned within a model that helps determine its behavior.

A parameter is a learned numerical weight that helps determine how a model transforms inputs into outputs. In a language model, parameters encode statistical regularities as a web of constraints on how features interact and which patterns are treated as similar. They do not store sentences, rules, or facts literally. section 4.2.1 section 6.4.1

The number of parameters is a visible measure of model size and affects representational capacity. Wider layers can represent more features at each step, while deeper stacks can combine and transform features across layers. More capacity gives a model room to distinguish more situations, but a higher parameter count alone does not determine capability. section 4.2.1

Training data influences which structures a model encounters, while parameters determine how much structure it can represent. Performance depends on balancing model size with data and computation; increasing all three tends to improve results, though additional gains diminish. section 4.2.2

During ordinary inference, learned weights remain fixed. They transform representations of the current input, influencing which continuations are more likely. A prompt can change the response without changing the parameters: this in-context adaptation is temporary, not accumulated knowledge. Online learning can update parameters as new data arrives, but those updates are still training. section 5.4.4 section 6.4.1

Figure 5.4 Human cognition continuously updates through interaction with the world, so learning and action form a single loop. Artificial neural networks separate learning from use: parameters are updated during training, after which the trained model performs inference without incorporating new experience.

The figure contrasts parameter updates during training with fixed-parameter inference. section 5.4.4

This distinction matters for reliability. If a fact is neither encoded in the parameters nor supplied in the active input, it cannot influence the answer. Recent events, omitted documents, and earlier exchanges therefore need to be supplied as input when they are otherwise unavailable. section 6.4.1

How Machines that Think covers it

How deep it goes
Explains representational capacity, layer-level effects, and the distinction between parameter updates and inference-time conditioning.
Examples
Contrasts post-training events and omitted documents with information supplied in a prompt.
What you can do after
Enables evaluation of whether model size or available information supports a claim about capability or reliability.
How it connects
Links parameters to training data, scaling laws, inference, in-context learning, and online learning.

Also explained in

Compare how 41 books explain parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.