Manifold

As explained in Deep Learning with R, Third Edition

A manifold is a continuous surface or space that locally resembles a simpler coordinate space and can represent the structure of data.

Imagine every possible 28 × 28 image. Almost none looks like a handwritten digit. Recognizable digits occupy a much smaller, structured part of that enormous space: a manifold. Change a digit slightly and it usually remains recognizable; follow a smooth path through valid digit images and one digit can gradually morph into another. section 5.1.2

More precisely, a manifold is a lower-dimensional subspace that looks locally like an ordinary linear space. It need not be flat overall. Picture two colored sheets of paper crumpled together: each sheet remains a continuous surface, but the colors are hard to separate in the crumpled ball. Successive neural-network layers can be pictured as small transformations that gradually uncrumple the surfaces and separate the classes. section 2.3.6 section 5.1.2

A model can fit without learning useful structure

A counterexample uses MNIST images but randomly shuffled labels: random_train_labels <- sample(train_labels) model <- keras_model_sequential() |> layer_dense(512, activation = "relu") |> layer_dense(10, activation = "softmax") model |> compile(optimizer = "rmsprop", loss = "sparse_categorical_crossentropy", metrics = "accuracy") model |> fit(train_images, random_train_labels, epochs = 100, batch_size = 128, validation_split = 0.2) With enough parameters, a model can memorize arbitrary input–label pairs. Fitting training samples alone therefore does not explain why predictions work on new digits; their structured input space matters. section 5.1.2

A model that approximates the data’s latent manifold can generalize by interpolating between training samples. That does not promise sensible predictions everywhere: inputs far from those seen during training remain a problem. Even between two valid digits, a straight pixel-by-pixel average may not itself look like a valid digit; a path along the manifold is different. section 5.5 section 5.1.2

Figure 5.8 Difference between linear interpolation and interpolation on the latent manifold. Every point on the latent manifold of digits is a valid digit, but the average of two digits usually isn’t.

Compare straight interpolation between digit images with interpolation along the latent manifold; the figure makes the distinction between an average image and a valid digit visible. section 5.1.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Defines a lower-dimensional, locally linear subspace and develops visual intuitions, without a formal geometric derivation.

Examples

Illustrates the idea with crumpled colored paper, morphing MNIST digits, latent interpolation, and a shuffled-label training example.

What it assumes

Introduces manifolds through familiar surfaces and images while assuming basic familiarity with classification, layers, and training samples.

What you can do after

Evaluate/understand why interpolation across learned data structure can support generalization while memorizing arbitrary labels cannot explain it.

How it connects

Connects manifolds to neural-network layers, geometric transformations, interpolation, classification, and generalization.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA lower-dimensional geometric subspace that is locally similar to Euclidean space, used here to describe the structure of valid data samples.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA one-dimensional manifold.

Compare how 3 books explain manifold →

Related concepts

• latent spaceused by · A latent space is a learned representational space in which numerical coordinates encode information or features inferred from observed data.
• interpolationused by · Interpolation is the estimation or construction of values or representations between known points according to an assumed relationship or specified rule.
• text encoderproduced by · A text encoder is a neural-network model that transforms textual input into numerical representations for conditioning or other downstream computation.
• surfaceincludes · A surface is a two-dimensional geometric set, often embedded in three-dimensional space or representing a boundary or relation.
• UMAPused by · UMAP, or Uniform Manifold Approximation and Projection, is a nonlinear manifold-learning technique that embeds high-dimensional data in a lower-dimensional space while preserving important neighborhood and structural relationships.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.