liveBook · Manning

Encoder-decoder architecture

As explained in Building Specialized AI Systems

An encoder-decoder architecture uses an encoder to represent an input sequence and a decoder to generate an output sequence.

An encoder-decoder architecture maps one sequence to another: the encoder reads input tokens sequentially and compresses them into a fixed-size hidden state, while the decoder generates output tokens one at a time. section 3.1.1

The architecture creates a practical trade-off:

  • Sequential processing preserves temporal dependencies but prevents parallel token processing during training. section 3.1.1
  • The fixed-size representation works reasonably for short inputs but can lose important information from long sequences. section 3.1.1
  • The decoder depends on the final encoder state and cannot directly access earlier encoder states, making long-range translation difficult. section 3.1.1

Language translation

For translation, an encoder reads the input sentence word by word, and a decoder uses the compressed representation to produce the translated sentence token by token.

How Building Specialized AI Systems covers it

How deep it goes
Explains sequential state updates, fixed-size compression, and information loss, then contrasts these limits with Transformer attention.
Examples
Uses language translation and a visual RNN encoder-decoder diagram as concrete examples.
What you can do after
Evaluate encoder-decoder bottlenecks and understand why attention-based Transformers replace recurrence for longer sequences.
How it connects
Connects RNNs, LSTMs, GRUs, Transformers, attention, and long-range dependencies.

Also explained in

Compare how 14 books explain encoder-decoder architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.