GPT-2’s “Ovid’s Unicorn” sample posed a puzzle: its invented news story held together across paragraphs without an explicit memory mechanism, yet it also described four-horned unicorns and fires burning underwater. Fluent writing could coexist with a fragile grasp of how the world works. section 1.2
A world model is an internal representation of regularities in an environment. One way to understand it is as compression: rather than retain every observation, a model captures rules that can produce typical outcomes. A model learning an environment’s physics, for instance, would compress its history of observations into such rules. section 8.2.1
The trade-off resembles both generalization and scientific explanation. A model that memorizes noise carries something like a giant lookup table; one that is too simple misses important patterns. The aim is to capture regularities while leaving room for accidental variation. GPT-2’s unicorn story illustrates why convincing output alone cannot establish that this balance has been struck. section 8.2.1 section 1.2
