10 From isolated algorithms to coherent systems: David J.C. MacKay and the unifying logic of learning
This chapter covers
- David J.C. MacKay’s Information Theory, Inference, and Learning Algorithms (2003) and how it synthesized foundational ideas into a single framework
- Why compression, communication, estimation, and learning are all forms of probabilistic inference under uncertainty
- How MacKay revealed shared computational patterns across seemingly unrelated algorithms
- What it means to move from isolated algorithms to integrated learning systems
- Why this unifying perspective remains timeless in designing, interpreting, and evaluating modern learning systems
Only in retrospect—and occasionally through the insight of a single unconventional thinker—does a pattern become visible.
10.1 The unifying insight: probabilistic inference under constraints
10.2 Example 1 of 2: decoding a message and fitting a model are the same computation
10.2.1 A noisy channel: the classical decoding problem
10.2.2 From decoding to a probabilistic model
10.2.3 Posterior inference and optimal decoding
10.2.4 The same computation, seen as model fitting
10.2.5 Learning enters through uncertainty about the model
10.2.6 A simple numerical demonstration
10.2.7 Why this equivalence matters
10.3 Example 2 of 2: one algorithm, many disguises
10.3.1 The recurring pattern: passing information locally
10.3.2 Message passing in error-correcting codes
10.3.3 The same computation in graphical models
10.3.4 Neural computation and iterative signaling
10.3.5 Why the disguises persist
10.3.6 The teaching moment
10.4 From algorithms to systems: what MacKay changed in how we think
10.4.1 Why intelligence cannot live in a single algorithm
10.4.2 Why this framing dissolves traditional boundaries
10.4.3 What changes once the system is visible
10.4.4 The epistemic implications of probabilistic learning
10.5 A turning point: from foundational ideas to learning systems
10.6 Summary