liveBook · Manning

Deep learning

As explained in Timeless Algorithms, The Foundational Ideas

Deep learning is a machine-learning paradigm that learns representations through multiple parameterized computational layers.

Recognizing an image requires a way to turn raw input into useful features. Deep learning addresses this problem with neural networks that learn representations through multiple layers, rather than relying entirely on features specified in advance. Successive transformations can reveal patterns that would be difficult to handcraft. section 11.0 section 11.1

A layer can be written as hₗ = σ(Wₗhₗ₋₁ + bₗ), where hₗ₋₁ is the previous layer’s output, Wₗ contains weights, bₗ contains biases, and σ is an activation function. The weights and biases parameterize the transformation; the activation adds nonlinearity. Depth determines how many transformations are composed, while width determines how many patterns a layer can represent at one stage. section 11.8 section 11.0

Training repeatedly connects prediction to correction. A forward pass produces an output; a loss function measures its discrepancy from the observed outcome. Backpropagation applies the chain rule in reverse to calculate each parameter’s influence on the loss. Gradient descent then updates the parameters using those gradients and a chosen learning rate. As parameters change, the intermediate representations change too. section 10.5.1 section 11.8

Prediction and decision

A classification network can produce a probability, which a predetermined threshold converts into a class or operational response. These are distinct from the probability itself. A regression network instead produces a continuous numerical prediction. Training evaluates the continuous output: cross-entropy is used for classification, while squared error is widely used for regression. section 11.8

The modern rise of deep learning followed decades of uneven progress. Larger data sets, greater computing power—including graphics processing units—and improved network design and training made multilayer networks practical at scale. The 2015 synthesis by Yann LeCun, Yoshua Bengio, and Geoffrey Hinton captured this shift toward learned representations. section 11.0 section 11.1

Learning a low training loss is not the same as generalizing to unseen observations. What a network can learn is constrained by its architecture, data and computation. Its training also brings together likelihood, cross-entropy and probabilistic reasoning, linking representation learning to broader questions of prediction and uncertainty. section 10.5.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains layer transformations and the training cycle through loss, backpropagation and gradient descent, alongside generalization constraints.
Examples
Uses image recognition, speech processing and classification and regression predictions to illustrate applications and outputs.
What you can do after
Evaluate and understand how learned representations, training decisions and generalization constraints affect deep-learning predictions.
How it connects
Connects representation learning with neural networks, cross-entropy, likelihood, Bayesian inference and generalization.

Also explained in

Compare how 20 books explain deep learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.