liveBook · Manning

Data representation

As explained in Machines that Think

A structured encoding of information that can support reasoning or computation.

A data representation is a way of encoding information so a system can process it. The form matters: it determines what the system can observe and helps shape the patterns it can learn. In language processing, for example, a representation must contend with context and ambiguity, not just identify words. section 2.0 section 2.5

Tokenization breaks language into discrete, machine-readable units. Those units identify pieces of text, but do not by themselves express how the pieces relate. Numerical representations called embeddings address a different problem: capturing patterns of similarity, contrast, and association among tokens. section 2.0 section 2.5

Neural networks can learn representations rather than rely entirely on a predefined description of the data. Successive layers transform and refine inputs while learning to make predictions. With sufficient data and computation, networks can extract increasingly abstract features, such as edges, shapes, and objects. section 3.1.6 section 1.4.3

Useful representations capture patterns across examples, supporting responses to new data rather than merely reproducing familiar cases. This does not guarantee human-like understanding: capable behavior can arise from learned patterns without awareness. section 3.3 section 1.4.3

The strawberry test

A request to count the letter “r” in “strawberry” can expose a mismatch between a task and a model’s representation. Language models usually process tokens, which may be whole words or fragments rather than individual characters. Some newer models address the task with character-level steps or code, but the example illustrates why an apparently simple task can be brittle. section 9.4.4

How Machines that Think covers it

How deep it goes
Explains tokenization, relational embeddings, and learned layer-by-layer representations without detailing their implementation.
Examples
Uses edges, shapes, and objects to illustrate abstraction and letter counting in “strawberry” to illustrate representational brittleness.
What you can do after
Evaluate how a system’s representation can support generalization while making some tasks unexpectedly difficult.
How it connects
Connects representations to tokenization, embeddings, neural networks, and generalization.

Also explained in

Compare how 11 books explain data representation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.