Manifold hypothesis

As explained in Deep Learning with Python, Third Edition

The assumption that high-dimensional data lie near a lower-dimensional manifold, making smooth interpolation useful for learning.

Why can a classifier recognize a handwritten digit it has never seen? It could memorize its training images—even a model trained on randomly shuffled labels can fit those labels. Generalization depends instead on a useful property of real data: valid inputs are not scattered randomly among every possible input. section 5.1.2

Handwritten digits

An unprocessed MNIST image is a 28 × 28 array whose entries can each take 256 values. Almost none of the possible arrays look like digits. The recognizable ones occupy a small, structured part of that vast space. Change a digit slightly and it usually remains recognizable; follow a smooth sequence of small changes and one digit can gradually morph into another. section 5.1.2

The manifold hypothesis describes this structure as data lying on or near a lower-dimensional manifold within the full input space. Nearby valid examples can make interpolation useful for learning. But interpolation must respect the data’s shape: simply averaging two digit images usually does not produce a valid digit. section 5.1.2

Figure 5.8 Difference between linear interpolation and interpolation on the latent manifold. Every point on the latent manifold of digits is a valid digit, but the average of two digits usually isn’t.

Compare a straight average of two digits with a path along the latent manifold; the distinction shows why not every point between two image arrays is a useful example. section 5.1.2

As a rule of thumb, deep learning is a strong choice for perception tasks when there is plenty of data and this hypothesis applies. Do not assume the same approach will solve every abstraction problem: discrete reasoning and perception call for different strengths. section 19.4.3

How Deep Learning with Python, Third Edition covers it

How deep it goes

Defines a manifold through the MNIST input space and illustrates smooth paths, without deriving manifold geometry.

Examples

Uses MNIST digit morphs, digit-image averaging, and a model fitted to shuffled labels.

What it assumes

Builds intuition from pixel arrays and recognizable digits rather than requiring formal manifold mathematics.

What you can do after

Evaluate/understand it to judge when data-rich perception problems favor deep learning over hand-coded discrete rules.

How it connects

Links generalization and interpolation to memorization, latent manifolds, perception, and discrete reasoning.

liveBookmanning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionThe manifold hypothesis is the claim that natural data lies on a low-dimensional manifold within the high-dimensional space in which it is embedded.

Compare how 2 books explain manifold hypothesis →

Related concepts

• interpolationproduces · Interpolation is the estimation or construction of values or representations between known points according to an assumed relationship or specified rule.
• language modelconfigures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• generalizationaddresses · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
• manifolduses · A manifold is a mathematical space that locally resembles Euclidean space, although its global geometry or topology may be more complex.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.