liveBook · Manning

Representation learning

As explained in Timeless Algorithms, The Foundational Ideas

Representation learning transforms raw data into representations that retain information useful for downstream tasks.

A high-resolution image may contain millions of pixels, many of them nearly identical to their neighbors. Representation learning transforms such raw data into a structured form that retains useful information while reducing redundancy and noise. The resulting representation can make data easier to process, search, visualize, or use in downstream tasks. section 5.4.2 section 5.4

Neural networks commonly learn representations during training, although representation learning is broader than any one model. Earlier methods often relied on features chosen by experts; deep networks instead learn successive transformations of the input. In image recognition, these transformations may move from pixels to edges, shapes, and eventually objects. section 5.4.2 section 11.1

The information-theoretic perspective connects representation learning to source coding: both seek a compact description that preserves informative content while discarding predictable variation. But compactness alone is not the goal. What can safely be discarded depends on what the representation must support; information useful for reconstruction, for example, need not be identical to information useful for classification. section 5.4.2 section 11.5.2

Figure 5.3 Representation learning as information-preserving compression. An encoder transforms a high-dimensional image into a compact latent representation by preserving the image's informative structure while reducing redundant detail. Although many original pixels are no longer represented explicitly, the latent representation retains the information needed for a decoder to reconstruct an image that closely resembles the original.

The encoder compresses an image into a latent representation, from which a decoder reconstructs an image resembling the original. section 5.4.2

A learned representation can also change the geometry of a prediction problem. Hidden layers may transform observations that require a curved boundary in the input space into coordinates where a straight boundary separates the classes. The nonlinearity lies in the transformation, not necessarily in the final boundary. section 11.5.2

Figure 11.3 A nonlinear transformation reorganizes the same six observations so that classes requiring a curved boundary in the input space can be separated by a straight boundary in the learned representation.

The same circles and squares become linearly separable after their coordinates are transformed. section 11.5.2

Easier separation of training observations is not enough on its own. The transformation improves generalization only when it captures relationships that also occur in unseen observations. section 11.5.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains information-preserving compression and geometric class separation, without specifying an implementation procedure.
Examples
Uses image compression and a circles-versus-squares classification problem to illustrate learned representations.
What you can do after
Evaluate and understand how representations can reduce redundancy or simplify classification, and why training-set separation alone does not ensure generalization.
How it connects
Connects representation learning to source coding, entropy, neural networks, classification, and generalization.

Also explained in

Compare how 5 books explain representation learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.