6 All that glitters is not smart
This chapter covers
- Analyzing why language models produce hallucinated knowledge.
- Interpreting the probabilistic nature of model responses.
- Evaluating how safeguards and jailbreaks reshape model outputs.
- Assessing how context limits shape model responses.
Large language models often produce responses that feel precise, informed, and closely aligned with the questions they receive, creating the impression that the system understands the request and assembles a reasoned reply. In many interactions, this impression holds, with outputs that appear confident, coherent, and consistent with the task.
Yet this surface reliability can crack in ways that are difficult to anticipate. The model does not reason over a stable store of facts or apply rules in the way traditional software might. Instead, each response emerges from a process that predicts the next fragment of text based on patterns learned from vast collections of human language and on the information available at that moment. Within that process, facts, instructions, examples, retrieved evidence, and prior context all become signals that can reinforce, weaken, or displace one another.