liveBook · Manning

What you will learn in Timeless Algorithms, The Foundational Ideas

by Gary Sutton · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Seeing inside the black box

The introduction explains why producing model results is not the same as judging them and presents foundational ideas as the layers beneath modern learning systems.

1.1 Why foundational ideas matter · 1.2 The hidden stack of modern intelligence · 1.2.1 Raw Data + Feature Engineering · 1.2.2 Modeling & Decision Frameworks · 1.2.3 Algorithmic Assumptions · 1.2.4 Mathematical Foundations · 1.2.5 Epistemology & Ethics · 1.2.6 Seeing the stack in practice · 1.3 How foundational ideas map to the stack · 1.4 Where these ideas work—and where they don’t · 1.5 Summary

What you learn in chapter 1 →

Chapter 2: From effect to cause: Thomas Bayes and the first algorithm of learning

The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.

2.1 From observation to belief: how Bayesian reasoning begins · 2.2 From prior to posterior: how Bayes’ Theorem works · 2.2.1 The mathematical structure of Bayes’ Theorem · 2.2.2 Bayesian inference in real-world scenarios · 2.2.3 The structure that makes inference possible · 2.3 Building the machinery: the rules that power the theorem · 2.3.1 Partitioning belief: the addition rule · 2.3.2 Contrasting outcomes: the complement rule · 2.3.3 Expressing likelihoods: odds and expected value · 2.3.4 Linking outcomes: the logic of the multiplication rule · 2.3.5 Independent repetition: compound events · 2.3.6 Likelihood from repetition: binomial probability · 2.4 Applications in machine learning and AI · 2.4.1 Bayesian inference in modern algorithms · 2.4.2 The addition and multiplication probability rules at work · 2.4.3 The enduring legacy of Bayes · 2.5 Bayesian reasoning beyond ML and AI · 2.6 Where Bayesian inference goes wrong · 2.7 Summary

What you learn in chapter 2 →

Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood

The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.

3.1 Fisher’s approach to statistical inference · 3.1.1 Estimating unknown parameters · 3.1.2 What is likelihood? · 3.1.3 Maximum likelihood estimation · 3.1.4 Maximum likelihood estimation · 3.1.5 Why maximum likelihood endures · 3.2 The criteria for a good estimator · 3.2.1 Consistency: converging toward truth · 3.2.2 Efficiency: getting the most from the data · 3.2.3 Sufficiency: capturing all the information · 3.2.4 Fisher’s case for maximum likelihood · 3.3 Maximum likelihood and logistic regression · 3.3.1 Why logistic regression requires maximum likelihood · 3.3.2 The logistic model · 3.4 Linear regression as a special case · 3.4.1 Primer on linear regression · 3.4.2 A maximum likelihood interpretation · 3.4.3 Worked example · 3.4.4 Why linear regression mattered · 3.5 Maximum likelihood and neural networks · 3.6 MLE beyond regression analysis and neural networks · 3.6.1 Poisson and exponential models · 3.6.2 Naïve Bayes · 3.6.3 Hidden Markov Models · 3.7 Model Selection · 3.7.1 Likelihood Ratio Tests · 3.7.2 Akaike and Bayesian Information Criteria · 3.8 Fisher's enduring legacy · 3.9 Summary

What you learn in chapter 3 →

Chapter 4: From evidence to decisions: Neyman, Pearson, and the principles of hypothesis testing

The introduction explains why learning from data is not enough to make decisions and how the Neyman–Pearson framework adds explicit, advance rules for controlling errors.

4.1 The stepwise framework of hypothesis testing · 4.1.1 Step 1: state the hypotheses · 4.1.2 Step 2: choose a significance level · 4.1.3 Step 3: select the test · 4.1.4 Step 4: compute the test statistic · 4.1.5 Step 5: establish the decision rule · 4.1.6 Step 6: make a decision · 4.1.7 Step 7: draw a conclusion · 4.1.8 From framework to practice · 4.2 Hypothesis testing in action · 4.2.1 Example 1: beverage fill volumes · 4.2.2 Example 2: safety monitoring · 4.2.3 Example 3: a small clinical trial · 4.2.4 From examples to principles · 4.3 Why it matters · 4.3.1 From evidence to action · 4.3.2 Making decisions reproducible · 4.3.3 A framework beyond statistics · 4.4 An enduring framework for decision-making · 4.5 Summary

What you learn in chapter 4 →

Chapter 5: Entropy and the bit: Claude Shannon and the quantification of information

The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.

5.1 How Shannon redefined information · 5.1.1 Information as the reduction of uncertainty · 5.1.2 Entropy as the measure of uncertainty · 5.1.3 Measuring entropy in bits · 5.1.4 Entropy, redundancy, and compression · 5.2 Shannon’s framework of communication · 5.2.1 Source coding (compression) · 5.2.2 Channel capacity · 5.2.3 Noisy channel coding · 5.2.4 Signal design · 5.2.5 From communication to modern learning methods · 5.3 From information theory to data partitioning · 5.3.1 Decision trees · 5.3.2 Random forests · 5.3.3 Feature selection · 5.3.4 Clustering · 5.4 Information theory in deep learning · 5.4.1 Neural networks · 5.4.2 Representation learning · 5.5 From communication to learning · 5.6 Summary

What you learn in chapter 5 →

Chapter 6: Dynamic programming: Richard Bellman and recursive optimization

The introduction frames dynamic programming as Bellman’s way to optimize interconnected decisions by reusing solutions to smaller problems.

6.1 The dynamic programming paradigm · 6.1.1 Structural prerequisites for dynamic programming · 6.1.2 Expressing problems as recurrence relations · 6.1.3 States, transitions, and base cases: the anatomy of a dynamic programming solution · 6.1.4 Implementation methods · 6.1.5 From principles to practice · 6.2 Constrained optimization: the knapsack problem · 6.2.1 Verifying prerequisites · 6.2.2 Specifying the recurrence · 6.2.3 Defining states, transitions, and base cases · 6.2.4 Choosing an implementation strategy · 6.2.5 Solving the 0/1 knapsack problem · 6.2.6 Reconstructing the selected items · 6.3 Counting solutions: the coin change problem · 6.3.1 Verifying the prerequisites · 6.3.2 Specifying the recurrence · 6.3.3 Defining states, transitions, and base cases · 6.3.4 Choosing an implementation strategy · 6.3.5 Solving the coin change problem · 6.3.6 Interpreting the solution · 6.4 Real-world applications of dynamic programming · 6.4.1 Constrained optimization · 6.4.2 Inventory control · 6.4.3 Shortest routes and networks · 6.4.4 Bioinformatics and string matching · 6.5 Markov Decision Processes and the Bellman equation · 6.5.1 Markov Decision Processes explained · 6.5.2 The Bellman equation: recursive decomposition of value · 6.5.3 From dynamic programming to reinforcement learning · 6.6 Dynamic programming in machine learning and AI · 6.6.1 Probabilistic sequence decoding · 6.6.2 Dynamic programming’s place in modern AI · 6.7 Pitfalls and limitations · 6.7.1 State explosion and the curse of dimensionality · 6.7.2 Memory versus computation · 6.7.3 Defining the right state · 6.8 Summary

What you learn in chapter 6 →

Chapter 7: Beyond Bayesian inference: Raiffa, Schlaifer, and rational decision-making

The introduction explains how Raiffa and Schlaifer extended Bayesian inference into a framework for choosing actions and deciding whether more information is worth obtaining.

7.1 Revisiting Bayes: from diagnosis to decision · 7.1.1 From posterior probability to expected utility · 7.1.2 Decision trees: visualizing choice under uncertainty · 7.1.3 Understanding the value of information · 7.1.4 Applying the value of information · 7.2 The decision framework of Raiffa and Schlaifer · 7.2.1 Acts, states, and experiments · 7.2.2 The expected utility formulation · 7.2.3 Terminal versus preposterior analysis · 7.3 Opportunity loss, regret, and expected utility · 7.3.1 From utility to opportunity loss · 7.3.2 A simple illustration · 7.4 Conjugate priors and computable Bayesian learning · 7.4.1 From integration to parameter updating · 7.4.2 Beta-Binomial updating · 7.4.3 Why conjugacy matters · 7.5 The enduring value of a rational decision-making framework · 7.5.1 Decision analysis as a managerial discipline · 7.5.2 Beyond managerial decision-making · 7.5.3 From decision analysis to machine learning and AI · 7.6 Summary

What you learn in chapter 7 →

Chapter 8: Learning to generalize: Vladimir Vapnik and support vector machines

The introduction explains why SVMs were designed to generalize beyond training data, using wide margins and, when necessary, tolerated violations.

8.1 From separation to maximum margin · 8.1.1 Separation in one dimension · 8.1.2 From one dimension to two · 8.2 Hard-margin support vector machines · 8.2.1 Decision function, weights, bias, and classification · 8.2.2 Parallel margins and support vectors · 8.2.3 Constraints, margin width, and optimization · 8.3 Soft-margin support vector machines · 8.3.1 Why hard margins break down · 8.3.2 Slack variables and soft-margin constraints · 8.3.3 The penalty parameter and soft-margin optimization · 8.3.4 Soft margins in action · 8.3.5 Selecting the penalty parameter through cross-validation · 8.4 Predictions using the direct form · 8.5 Predictions using the support-vector form · 8.5.1 How support vectors reproduce the fitted weight vector · 8.5.2 Making predictions directly from support vectors · 8.6 Kernels and nonlinear separation · 8.6.1 From explicit transformations to the kernel trick · 8.6.2 Common kernels and the boundaries they produce · 8.6.3 Kernel parameters and model selection · 8.7 Why support vector machines generalize · 8.7.1 Controlling capacity rather than maximizing it · 8.7.2 VC dimension and the theoretical case for generalization · 8.8 The continuing influence of support vector machines · 8.8.1 Extensions of the support-vector framework · 8.8.2 Where SVMs remain useful · 8.8.3 Margins beyond SVMs · 8.9 Summary

What you learn in chapter 8 →

Chapter 9: From single trees to random forests: Leo Breiman and the logic of ensemble learning

This introduction explains why unstable decision trees can benefit from an ensemble and contrasts Breiman’s approach to generalization with Vapnik’s.

9.1 The strengths and limitations of single decision trees · 9.1.1 Instability and sensitivity to early splits · 9.1.2 Depth, overfitting, and variance · 9.1.3 Why pruning is not enough · 9.2 Bagging: randomness in the observations · 9.2.1 How bagging creates different training sets · 9.2.2 Aggregation: turning variation into stability · 9.2.3 What bagging fixes—and what it does not · 9.3 Random forests: randomness in the features · 9.3.1 Random feature selection at each split · 9.3.2 Creating diversity by limiting choice · 9.3.3 The complete random forest algorithm · 9.4 Why random forests generalize · 9.4.1 Classification margins: measuring collective confidence · 9.4.2 Balancing tree strength and correlation · 9.4.3 Why adding trees stabilizes the forest · 9.5 Out-of-bag evaluation · 9.6 Variable importance: recovering interpretation · 9.6.1 Measuring variable importance · 9.6.2 interpreting importance responsibly · 9.7 Random forests versus support vector machines · 9.8 The enduring legacy of random forests · 9.9 Summary

What you learn in chapter 9 →

Chapter 10: From foundational algorithms to modern AI: David J.C. MacKay and the unifying logic of learning

The introduction presents MacKay’s synthesis of seemingly separate algorithms as ways of learning from incomplete evidence under practical constraints, then connects that synthesis to modern AI.

10.1 Synthesis rather than invention · 10.2 Looking back: isolated contributions, shared foundations · 10.2.1 Bayes: revising belief with new evidence · 10.2.2 Fisher: estimating unknown parameters through maximum likelihood · 10.2.3 Neyman and Pearson: making decisions while controlling errors · 10.2.4 Shannon: quantifying information and the limits of communication · 10.2.5 Bellman: making global optimization tractable through decomposition · 10.2.6 Raiffa and Schalifer: From Bayesian inference to rational action · 10.2.7 Vapnik: generalizing through soft margins · 10.2.8 Breiman: stabilizing predictions through randomness and aggregation · 10.3 When ideas cross boundaries · 10.3.1 Decoding as Bayesian model comparison · 10.3.2 Message passing across codes and graphical models · 10.3.3 Cross-entropy: from coding efficiency to model training · 10.4 Why the connections matter · 10.5 Looking forward: AI operationalizes the synthesis · 10.5.1 Neural networks: the synthesis becomes trainable · 10.5.2 Transformers: the synthesis gains a scalable architecture · 10.5.3 Scaling laws: expanding the boundaries of what is feasible · 10.6 Summary

What you learn in chapter 10 →

Chapter 11: Learning representations: LeCun, Bengio, Hinton, and the rise of deep learning

The introduction explains why deep learning became influential, centering on learned representations and the convergence of earlier ideas and practical advances.

11.1 From early neural models to deep learning · 11.2 How a neural network operates, end to end · 11.2.1 Architecture and representation · 11.2.2 From input to prediction · 11.2.3 Measuring prediction error · 11.2.4 Tracing error backward · 11.2.5 Updating the network · 11.2.6 The complete learning cycle · 11.3 A worked example: pedestrian detection in autonomous driving · 11.3.1 Defining the problem and network · 11.3.2 Making the initial prediction · 11.3.3 Measuring the prediction error · 11.3.4 Tracking the error backward · 11.3.5 Updating the parameters · 11.3.6 Results from one training step · 11.4 The deeper foundations of neural network learning · 11.4.1 Activation functions shape what neural networks can learn · 11.4.2 Cross-entropy as likelihood and information · 11.4.3 Bayes, MacKay, and uncertainty · 11.5 What depth, width, and learned representations contribute · 11.5.1 Depth and width provide different forms of capacity · 11.5.2 Learned representations simplify prediction · 11.5.3 What makes larger networks possible · 11.6 Limits, constraints, and generalization · 11.6.1 Optimization and architectural decisions · 11.6.2 Overfitting and regularization · 11.6.3 Data, bias, and computational constraints · 11.6.4 Reliability, calibration, and opacity · 11.7 From the 2015 synthesis to modern AI · 11.8 Summary

What you learn in chapter 11 →

Chapter 12: Beyond recurrence: Google Brain and the transformer architecture

The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.

12.1 From sequential processing to relational representation · 12.1.1 The constraints of sequential modeling · 12.1.2 From recurrence to self-attention · 12.1.3 From representation to capability · 12.2 Worked examples of attention in action · 12.2.1 Example 1: resolving a pronoun through attention · 12.2.2 Semantic association with factual recall · 12.2.3 Example 3: autoregressive generation · 12.3 Intellectual positioning · 12.3.1 Foundations carried forward · 12.3.2 Foundations reconsidered · 12.3.3 What was genuinely new · 12.4 The transformer’s prospects for lasting influence · 12.4.1 Why the transformer is positioned to persist · 12.4.2 Why continued dominance is not assured · 12.5 Conclusion and transition · 12.6 Summary

What you learn in chapter 12 →

Chapter 13: Expanding the boundaries of learning: OpenAI and the scaling laws of neural language models

This introduction connects earlier learning ideas and transformer architecture to evidence that scaling model parameters, data, and compute yields predictable gains while leaving important constraints in place.

13.1 Discovering the scaling relationships · 13.1.1 The empirical puzzle · 13.1.2 Measuring performance across resources · 13.1.3 The empirical pattern · 13.2 Parameters, data, compute, and power laws · 13.2.1 Model parameters: representational capacity · 13.2.2 Training data: available information · 13.2.3 Compute: the work of learning · 13.2.4 Expressing improvement as a power law · 13.2.5 Balancing resources: the compute-efficient frontier · 13.3 What scaling revealed—and what it did not · 13.3.1 Larger models and sample efficiency · 13.3.2 Generalization at high capacity · 13.3.3 From next-token prediction to task transfer · 13.3.4 The reach of the evidence · 13.4 From isolated algorithms to scaled learning systems · 13.4.1 What scaling did not change · 13.4.2 From learned representations to relational architecture · 13.4.3 Scaling the reach of learning · 13.5 Constraints pushed outward (not eliminated) · 13.5.1 Computational and physical boundaries · 13.5.2 The limits of training data · 13.5.3 The economics of the frontier · 13.5.4 Moving the boundary—again · 13.6 Summary

What you learn in chapter 13 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.