liveBook · Manning

Chapter 11: Learning representations: LeCun, Bengio, Hinton, and the rise of deep learning

What you will learn in chapter 11 of Timeless Algorithms, The Foundational Ideas

The introduction explains why deep learning became influential, centering on learned representations and the convergence of earlier ideas and practical advances.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why deep learning became influential, centering on learned representations and the convergence of earlier ideas and practical advances.

Key terms: representation learning, handcrafted features, successive layers, multilayer networks, Deep learning (2015), synthesis, adaptive system

11.1 From early neural models to deep learning

This section traces how neural networks became practical deep-learning systems and explains why learning representations from data was central to that change.

Key terms: neural models, layers, internal connections, training methods, graphics processing units, deep learning, representation, representation learning, handcrafted features, Yann LeCun

11.2 How a neural network operates, end to end

This section traces how a neural network turns inputs into predictions and learns from errors by adjusting its parameters.

Key terms: forward pass, input layer, weighted sum, bias, activation function, representation, regression, binary classifier, decision threshold, multiclass network

11.2.1 Architecture and representation

This section explains how a neural network’s layers and nodes transform inputs into learned representations, and how width and depth affect those representations.

Key terms: architecture, input layer, hidden layers, output layer, nodes, neurons, units, weights, bias, activation function

11.2.2 From input to prediction

This section explains how a neural network turns an input into a prediction through a forward pass, before any learning occurs.

Key terms: forward pass, weights, biases, activation function, activation, weighted combinations, nonlinear transformations, activation functions, representation, output layer

11.2.3 Measuring prediction error

This section explains how a loss function measures a neural network’s prediction error and why that measurement provides a useful learning signal.

Key terms: loss function, squared-error loss, cross-entropy loss, decision threshold, learning signal, forward pass, backpropagation, chain rule

11.2.4 Tracing error backward

This section explains how backpropagation traces prediction error through a multilayer network to calculate gradients for its parameters.

Key terms: loss, backpropagation, sensitivity, weights, biases, chain rule, local effects, gradients, forward pass, optimization method

11.2.5 Updating the network

This section explains how a neural network uses gradients to update its parameters repeatedly across mini-batches and epochs.

Key terms: gradient descent, gradient, learning rate, weights, biases, mini-batches, stochastic gradient descent, forward pass, loss, epoch

11.2.6 The complete learning cycle

This section brings the steps of neural-network training together as a repeated cycle and explains why learning takes many small updates.

Key terms: loss function, backpropagation, gradients, gradient descent, weights and biases, mini-batch, epoch, representations, loss

11.3 A worked example: pedestrian detection in autonomous driving

Introduces pedestrian detection as a binary classification task and explains why a simplified neural network makes it a useful worked example.

Key terms: binary classification, positive class, negative class, decision threshold, single linear rule, representation, pedestrian-like patterns, numerical input, hidden layer, nodes

11.3.1 Defining the problem and network

The section defines the inputs, outputs, decision rule, and small feedforward network used in a pedestrian-detection example.

Key terms: 𝑥, 𝑦, 𝑦̂, binary outcome, predicted probability, decision threshold, 𝜏, positive class, false alarms, hidden nodes

11.3.2 Making the initial prediction

This section traces a pedestrian-detection network’s first forward pass from image-derived input through hidden-node calculations to a probability and braking decision, before any parameters change.

Key terms: hidden nodes, weights, biases, pre-activation value, sigmoid, activation function, hidden-layer representation, nonlinear transformations, output node, logit

11.3.3 Measuring the prediction error

This subsection shows why a correctly classified pedestrian detection still has cross-entropy loss and why that continuous loss can guide learning.

Key terms: decision threshold, cross-entropy loss, probability, continuous loss, backward pass, cross-entropy

11.3.4 Tracking the error backward

This section traces a pedestrian-detection network’s prediction error backward to calculate gradients for its seven adjustable parameters, without updating them yet.

Key terms: backward pass, chain rule, gradients, parameter update, output-layer error signal, pre-activation value, cross-entropy loss, sigmoid output, output weight, bias gradient

11.3.5 Updating the parameters

This section shows how gradients update a small network’s parameters and distinguishes a single-observation update from mini-batch updates and an epoch.

Key terms: gradient descent, gradient, learning rate, parameters, loss, stochastic gradient descent, batch size of one, mini-batch, epoch

11.3.6 Results from one training step

This section checks the effect of one training update on the pedestrian detector and explains why small improvements can add up through repeated training.

Key terms: forward pass, sigmoid function, cross-entropy loss, 0.30 threshold, backpropagation, gradient descent, mini-batches, epochs

11.4 The deeper foundations of neural network learning

This section identifies mathematical choices behind neural-network learning and connects them to earlier statistical and information-theoretic ideas.

Key terms: activation functions, nonlinear relationships, cross-entropy, gradients, likelihood framework, information theory, probabilistic outputs, strength of a prediction, uncertainty about network parameters

11.4.1 Activation functions shape what neural networks can learn

This section compares sigmoid, tanh, and ReLU to show how activation choice changes hidden representations, gradient flow, and practical network design.

Key terms: nonlinear activation functions, hidden layers, linear transformation, sigmoid, tanh, ReLU, pre-activation values, hidden-layer representations, backpropagation, gradients

11.4.2 Cross-entropy as likelihood and information

This section explains why binary cross-entropy is an appropriate classification loss by connecting it to Fisher’s likelihood framework and Shannon’s measure of information.

Key terms: Bernoulli variable, likelihood, log-likelihood, binary cross-entropy loss, Shannon’s information theory, surprise, information cost, cross-entropy

11.4.3 Bayes, MacKay, and uncertainty

This section distinguishes a standard network’s prediction from uncertainty about its parameters, then explains how Bayesian neural networks and MacKay’s perspective connect that uncertainty to inference, regularization, and information.

Key terms: conditional probability, weights and biases, fixed parameter values, posterior distribution, Bayesian neural network, prior, likelihood, posterior, predictive distribution, approximation methods

11.5 What depth, width, and learned representations contribute

The section explains how width and depth expand what neural networks can represent, and why learned representations should help them predict unseen observations.

Key terms: width, depth, patterns, representations, generalization, internal representations, predictive patterns, unseen observations

11.5.1 Depth and width provide different forms of capacity

The section explains how width and depth give a neural network different capacities for recognizing pedestrians, and why capacity alone does not ensure generalization.

Key terms: width, nodes, patterns, depth, intermediate patterns, representations, adjustable parameters, capacity, generalization

11.5.2 Learned representations simplify prediction

This section explains how hidden layers can make classification simpler by learning a new representation, and when that simplification can help with unseen data.

Key terms: learned representation, hidden layer, input space, decision space, nonlinear transformation, representation learning, generalization, unseen observations, predictive relationships

11.5.3 What makes larger networks possible

Larger neural networks became practical through suitable data, faster computing, and more reliable training methods, though greater capacity brings new risks.

Key terms: depth, width, parameters, training data, computing power, learned representations, loss, predictive relationships, noise, matrix calculations

11.6 Limits, constraints, and generalization

The section introduces limits of neural-network learning that remain despite larger models and advances since 2015.

Key terms: depth, width, tradeoffs, generalization, biased data, confidence

11.6.1 Optimization and architectural decisions

This section explains why neural-network training is sensitive to optimization and architectural choices, and how validation helps select a network that performs beyond its training data.

Key terms: loss surface, non-convex, saddle points, local minima, convex objective, gradient descent, global minimum, training loss, mini-batches, generalize

11.6.2 Overfitting and regularization

This section explains how neural networks overfit, how model choices and regularization limit that risk, and how to evaluate whether performance carries over to new conditions.

Key terms: overfitting, training loss, generalization, depth, width, connectivity, hyperparameters, capacity, validation performance, regularization

11.6.3 Data, bias, and computational constraints

This section explains how training-data limitations, distribution shift, bias, and computational demands constrain neural networks despite their capacity and gains from scaling.

Key terms: data quality, representativeness, limited coverage, learned representations, distribution shift, validation and test results, production performance, bias, overall loss, aggregate accuracy

11.6.4 Reliability, calibration, and opacity

This section explains why neural-network accuracy alone does not establish reliability, focusing on calibration, interpretability, changing inputs, and uncertainty.

Key terms: reliability, calibration, classification accuracy, predicted probabilities, opacity, hidden representations, influential inputs, internal responses, distribution shifts, adversarial example

11.7 From the 2015 synthesis to modern AI

This section explains how modern AI extended the 2015 neural-network learning cycle through new training approaches, broader representations, transformers, and scaling, while retaining earlier foundations and constraints.

Key terms: representation, prediction, loss, parameter adjustment, constraints, self-supervised learning, labels, target, foundation models, representations

11.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 10: From foundational algorithms to modern AI: David J.C. MacKay and the unifying logic of learning Chapter 12: Beyond recurrence: Google Brain and the transformer architecture →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.