liveBook · Manning

Data representation

As explained in Timeless Algorithms, The Foundational Ideas

A representation encodes informative properties of data in a form suitable for processing or interpretation.

A data representation describes data in a form suited to processing or interpretation. An image, for example, can be represented by its pixel values or by a numerical description of its edges, textures, and shapes. The useful description depends on the task: a representation need not preserve every detail of the original data. section 5.4.2 section 11.2.1

Representation learning transforms raw data into descriptions that retain useful information while reducing redundancy and noise. It can support storage, search, visualization, and downstream prediction. This objective resembles source coding, but the emphasis shifts from efficient transmission to retaining information useful for a task. section 5.4.2

Figure 5.3 Representation learning as information-preserving compression. An encoder transforms a high-dimensional image into a compact latent representation by preserving the image's informative structure while reducing redundant detail. Although many original pixels are no longer represented explicitly, the latent representation retains the information needed for a decoder to reconstruct an image that closely resembles the original.

The encoder compresses an image into a latent representation; the decoder uses it to reconstruct an image resembling the original. section 5.4.2

In a neural network, the outputs of a layer form a new numerical representation. A layer can be written as h = f(Wx + b), where x is its input, W contains weights, b contains biases, and f is an activation function. The weights and biases shape the description, while the activation introduces nonlinearity. Training adjusts these parameters in response to a loss function. section 11.2.1 section 11.2

Width lets a layer respond to more patterns at one stage; depth lets successive layers combine patterns into more complex descriptions. In visual recognition, simple patterns may contribute to contours and shapes, then to evidence for a classification. These are illustrative levels, not fixed meanings assigned to particular layers. section 11.2.1 section 11.5.1

Making classes easier to separate

Suppose three circles occupy the center of an input space and three squares surround them. A straight line cannot separate the classes in those original coordinates. A nonlinear hidden-layer transformation can give the same observations new coordinates in which a straight boundary separates them. The original inputs do not change; their description does. section 11.5.2

Figure 11.3 A nonlinear transformation reorganizes the same six observations so that classes requiring a curved boundary in the input space can be separated by a straight boundary in the learned representation.

The figure follows the same observations from a space requiring a curved boundary to a learned representation allowing a straight boundary. section 11.5.2

A representation improves prediction on unseen data only if it captures patterns that persist beyond training observations. Greater capacity alone does not guarantee this: narrow training data can produce representations that fail when conditions change. In sequence modeling, architecture also constrains what can be represented: recurrent networks compress preceding information into a changing state, while transformer attention builds contextual representations through relationships among tokens. section 11.5.1 section 11.5.3 section 13.4.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains layer transformations, representational width and depth, and the geometry of classification, while treating visual layer meanings as illustrative.
Examples
Illustrates image compression, pedestrian detection, and a six-observation transformation that makes classes linearly separable.
What you can do after
Evaluates how a representation can simplify prediction and why training-set separation may not generalize.
How it connects
Connects representation learning with source coding, entropy, neural networks, generalization, and attention.

Also explained in

Compare how 11 books explain data representation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.