11 Learning representations: LeCun, Bengio, Hinton, and the rise of deep learning
This chapter covers
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton’s Deep learning (2015), which marked the arrival of deep neural networks as a dominant learning method
- How layered networks transform inputs through weighted connections and nonlinear activations to make predictions
- How backpropagation assigns responsibility for errors and gradient descent converts those errors into learning
- Why depth enables hierarchical representations to be learned rather than handcrafted
- How deep learning has evolved since 2015 and where its greatest promise and most important limitations lie
Neural networks are not a recent invention. Their origins extend to the middle of the twentieth century, when researchers began investigating whether networks of simple mathematical units could reproduce aspects of biological intelligence. Early models generated considerable interest, but their limited capabilities and the difficulty of training multilayer networks led to decades of uneven progress and recurring skepticism.
11.1 From early neural models to deep learning
11.2 How a neural network operates, end to end
11.2.1 Architecture and representation
11.2.2 From input to prediction
11.2.3 Measuring prediction error
11.2.4 Tracing error backward
11.2.5 Updating the network
11.2.6 The complete learning cycle
11.3 A worked example: pedestrian detection in autonomous driving
11.3.1 Defining the problem and network
11.3.2 Making the initial prediction
11.3.3 Measuring the prediction error
11.3.4 Tracking the error backward
11.3.5 Updating the parameters
11.3.6 Results from one training step
11.4 The deeper foundations of neural network learning
11.4.1 Activation functions shape what neural networks can learn
11.4.2 Cross-entropy as likelihood and information
11.4.3 Bayes, MacKay, and uncertainty
11.5 What depth, width, and learned representations contribute
11.5.1 Depth and width provide different forms of capacity
11.5.2 Learned representations simplify prediction
11.5.3 What makes larger networks possible
11.6 Limits, constraints, and generalization
11.6.1 Optimization and architectural decisions
11.6.2 Overfitting and regularization
11.6.3 Data, bias, and computational constraints
11.6.4 Reliability, calibration, and opacity
11.7 From the 2015 synthesis to modern AI
11.8 Summary