An encoder transforms input into an internal representation. Its role depends on the architecture: in an autoencoder, it compresses data into a latent space for reconstruction; in a Transformer, it processes an input sequence into contextual representations. section 3.4.5 section 4.1.6
These uses share a transformation role, not an identical objective. An autoencoder’s encoder works with a decoder that reconstructs the original input from a compact code. A Transformer encoder uses unrestricted self-attention, so every token can attend to every other token. Its decoder differs: it generates tokens step by step and masks future positions. section 3.4.5 section 4.1.6
Two forms of representation
An autoencoder might compress an image into a code that retains the patterns needed to reconstruct it. A Transformer encoder instead processes a whole sentence, allowing each token to use information from tokens elsewhere in that sequence. section 3.4.5 section 4.1.6
Figure 3.16 An autoencoder compresses input into a compact internal code and then reconstructs it. The encoder reduces the data to a low-dimensional representation in the bottleneck, where only the most informative patterns remain. The decoder expands this representation back into the original space, producing an output that resembles the input. By learning to rebuild the data from such a constrained description, the network discovers the underlying structure that defines the input instead of its incidental details.

Shows how an autoencoder’s encoder compresses input into a bottleneck representation before the decoder reconstructs it. section 3.4.5
Figure 4.9 Encoder–decoder structure for next-token prediction. The encoder processes the full input sequence to build contextualized representations. The decoder receives a shifted version of the sequence, so that at each position it can attend only to previous tokens, and predicts the next token accordingly. This setup allows the model to learn from complete sequences while enforcing the causal structure required for generation.

Shows the encoder processing a full input sequence alongside a decoder that predicts tokens using only earlier positions. section 4.1.6



























