10 From foundational algorithms to modern AI: David J.C. MacKay and the unifying logic of learning
This chapter covers
- David J.C. MacKay’s Information Theory, Inference, and Learning Algorithms (2003) and how it synthesized foundational ideas into a common framework
- How MacKay exposed recurring patterns obscured by time, terminology, and purpose
- How earlier contributions supplied complementary building blocks rather than competing solutions
- Why seeing their connections makes knowledge cumulative rather than discrete
- How modern AI combined these foundational ideas and operationalized them at unprecedented scale
Only in retrospect—and occasionally through the insight of a single unconventional thinker—does a pattern become visible.
Across history, landmark methods emerged to answer very different questions. Bayes transformed prior probabilities into posterior probabilities as new evidence arrived. Fisher used maximum likelihood to estimate unknown parameters from observed samples. Shannon measured entropy in bits and established limits on reliable transmission through noisy channels. Bellman expressed sequential decisions through value functions and recursive equations. Vapnik separated classes with maximum-margin hyperplanes and kernels. Breiman constructed forests from bootstrap samples and randomized feature subsets, then aggregated their predictions. Each method possessed its own objects, notation, and computational machinery.