Emergent behavior is a capability that arises from interactions among a system’s components rather than from an explicitly programmed feature. In generative AI, it helps models interpret prompts, solve problems, and produce step-by-step approaches. section 1.3.5 section 2.8.9
Large language models can acquire unexpected abilities by learning from large amounts of natural-language data. These abilities may appear only after training reaches a scale threshold; smaller models can perform near chance, while larger models improve sharply. Smaller-model performance does not reliably predict the behavior that emerges at scale, creating both opportunity and uncertainty. section 2.8.9
Translating unfamiliar code
A prompt can ask GPT-4 to convert a C function that reverses a string using pointers, XOR operations, and macros into Python. Without being given the function’s purpose as a separate instruction, the model produces a Python implementation that accounts for immutable strings, list conversion, pointer-like indices, and reconstruction of the result. This illustrates emergent code understanding rather than a conversion rule explicitly programmed for that function. section 2.8.9
Figure 2.12 Model scaling as measured by training computation requirements in FLOP
Use the scaling figure to inspect how benchmark performance changes with training computation and where emergent abilities appear. section 2.8.9
