liveBook · Manning

Context

As explained in Machines that Think

Context is the surrounding information and circumstances that determine how an expression, event, or task should be interpreted.

Context is the surrounding information and circumstances that shape how language or a task is interpreted. Words rarely carry meaning on their own: intent and the surrounding conversation also matter. For language models, the information available during generation helps define the task and frame the response. section 2.0 section 6.4.1

An answer draws on two sources: patterns encoded in the model’s parameters during training and text supplied at inference time. The prompt provides the active context; the parameters shape how the model transforms that input and predicts continuations. Encoded knowledge is not retrieved as if from a database. section 6.4.1

This distinction sets a practical boundary. Attention can connect only tokens in the active sequence. If a fact is neither encoded in the parameters nor supplied in the prompt, it cannot influence the answer. Documents, post-training events and earlier exchanges are inaccessible unless their relevant information is provided; conversational continuity depends on the surrounding system reintroducing prior turns. section 6.4.1

Available context does not guarantee a reliable answer. Instructions, examples, evidence and prior turns can reinforce or displace one another. Next-token prediction favors plausible continuations without requiring factual verification, so a fluent response can extend weak patterns rather than acknowledge missing evidence. section 6.0 section 6.1.2

Context can also be built during a response. In chain-of-thought, intermediate steps become input for later tokens and may help guide a problem toward a valid answer. Such steps can serve as a scaffold or an explanation, but they can also be a misleading trace; their presence does not establish exactly how the model reached its answer. section 9.2.3

How Machines that Think covers it

How deep it goes
Explains active token sequences, learned parameters and attention, while leaving the mechanisms behind chain-of-thought improvements unsettled.
Examples
Illustrates inaccessible information with post-training events, omitted documents and prior turns that must be reintroduced.
What you can do after
Enables evaluation of whether a model has the information needed to answer reliably, without treating fluency as proof of understanding.
How it connects
Connects context to attention, next-token prediction, hallucination and chain-of-thought.

Also explained in

Compare how 30 books explain context →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.