chapter thirteen
13 Expanding the boundaries of learning: OpenAI and the scaling laws of neural language models
This chapter covers
- Jared Kaplan et al.’s Scaling Laws for Neural Language Models (2020) and the discovery that prediction error declines according to power-law relationships
- How increases in training resources create measurable and predictable improvements in language-model performance
- Why model parameters, training data, and compute must grow in balance to maximize performance gains
- How increasing training resources not only reduces loss but also improves sample efficiency, generalization, and transfer across tasks
- How modern AI applies old algorithms with new methods and architectures at scale to produce increasingly capable learning systems
Many of the ideas that power modern artificial intelligence originated independently across different centuries and disciplines, often in response to seemingly unrelated problems. David MacKay later exposed deeper connections among probability, inference, information theory, and learning. His synthesis marked an intellectual inflection point: methods developed in isolation could be understood as complementary parts of a common framework for reasoning and learning from incomplete information under different constraints.