liveBook · Manning

Encoder

As explained in Machines that Think

A sequence-processing component that transforms an input sequence into internal representations using attention over the available input.

An encoder transforms input into an internal representation. Its role depends on the architecture: in an autoencoder, it compresses data into a latent space for reconstruction; in a Transformer, it processes an input sequence into contextual representations. section 3.4.5 section 4.1.6

These uses share a transformation role, not an identical objective. An autoencoder’s encoder works with a decoder that reconstructs the original input from a compact code. A Transformer encoder uses unrestricted self-attention, so every token can attend to every other token. Its decoder differs: it generates tokens step by step and masks future positions. section 3.4.5 section 4.1.6

Two forms of representation

An autoencoder might compress an image into a code that retains the patterns needed to reconstruct it. A Transformer encoder instead processes a whole sentence, allowing each token to use information from tokens elsewhere in that sequence. section 3.4.5 section 4.1.6

Figure 3.16 An autoencoder compresses input into a compact internal code and then reconstructs it. The encoder reduces the data to a low-dimensional representation in the bottleneck, where only the most informative patterns remain. The decoder expands this representation back into the original space, producing an output that resembles the input. By learning to rebuild the data from such a constrained description, the network discovers the underlying structure that defines the input instead of its incidental details.

Shows how an autoencoder’s encoder compresses input into a bottleneck representation before the decoder reconstructs it. section 3.4.5

Figure 4.9 Encoder–decoder structure for next-token prediction. The encoder processes the full input sequence to build contextualized representations. The decoder receives a shifted version of the sequence, so that at each position it can attend only to previous tokens, and predicts the next token accordingly. This setup allows the model to learn from complete sequences while enforcing the causal structure required for generation.

Shows the encoder processing a full input sequence alongside a decoder that predicts tokens using only earlier positions. section 4.1.6

How Machines that Think covers it

How deep it goes
Explains compression and unrestricted self-attention, contrasting encoder and decoder roles without detailing implementation.
Examples
Uses compressed image patterns and whole-sentence token relationships to illustrate distinct encoder roles.
What you can do after
Understand how encoder roles differ between autoencoders and Transformers, and how attention access distinguishes Transformer encoders from decoders.
How it connects
Connects encoders to autoencoders, latent spaces, decoders, Transformers, and self-attention.

Also explained in

Compare how 28 books explain encoder →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.