How deeply does Math and Architectures of Deep Learning cover Universal Approximation Theorem?

Develops tower-based intuition and perceptron constructions, but defers a formal proof and practical width estimates.

Universal Approximation Theorem

As explained in Math and Architectures of Deep Learning

A universal-approximation result stating that a sufficiently wide single-hidden-layer neural network can approximate continuous functions on compact domains arbitrarily well.

The Universal Approximation Theorem states that a neural network with one hidden layer can approximate continuous functions to arbitrary accuracy. Its significance is expressive power: multilayer perceptrons can represent functions that a single perceptron cannot, such as XOR. section 7.3.4 section 7.6

A useful intuition approximates a curve with narrow rectangular towers. Increasing the number of towers improves accuracy; each tower can be assembled from shifted, flipped perceptron steps and combined in an MLP. section 7.5.3

The theorem establishes possibility, not practical efficiency:

A single hidden layer is theoretically sufficient for continuous-function approximation. section 7.5.3
The required number of perceptrons can become arbitrarily large, so deeper networks are often more practical. section 7.3.4 section 7.5.3
Approximation does not guarantee accuracy on unseen data; training depends on representative labeled data and suitable weights and biases. section 7.6

Constructing a one-dimensional tower

Combine a left-shifted step, ϕ(x+5), with a flipped and right-shifted step, ϕ(−x+5), using ϕ(ϕ(x+5)+ϕ(−x+5)−1.5). The result forms a bounded tower; summing towers approximates a target curve. section 7.5.3

Figure 7.15 Generating a 1D tower with perceptrons.

(a) ϕ(x): positive w yields a regular step

(b) Perceptron for a regular step

(d) Perceptron for a laterally flipped step

(e) ϕ(−x+5): changing the bias shifts the step left or right

(f) Perceptron for a laterally shifted step

(g) (ϕ(x+5)+ϕ(−x+5)−1.5): ANDing a left-shifted step with a flipped, right-shifted step yields a tower

(h) MLP for a 1D tower

Use the construction diagram to trace how weights flip steps, biases shift them, and their combination creates a tower. section 7.5.3

How Math and Architectures of Deep Learning covers it

Cover of Math and Architectures of Deep Learning

liveBook manning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
continuous functionaddresses · A function whose output changes arbitrarily little when its input changes arbitrarily little.
universal approximatorequivalent to · A universal approximator is a model or function class capable of approximating a specified class of functions or distributions arbitrarily closely under suitable conditions.
function approximationequivalent to · Function approximation is representing an unknown input-output mapping with a parameterized model fitted from finite observations.
activation functionequivalent to · An activation function maps a neural-network unit’s or layer’s input to an output, often introducing nonlinearity into the network’s computation.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.