An encoder-decoder architecture maps one sequence to another: the encoder reads input tokens sequentially and compresses them into a fixed-size hidden state, while the decoder generates output tokens one at a time. section 3.1.1
The architecture creates a practical trade-off:
- Sequential processing preserves temporal dependencies but prevents parallel token processing during training. section 3.1.1
- The fixed-size representation works reasonably for short inputs but can lose important information from long sequences. section 3.1.1
- The decoder depends on the final encoder state and cannot directly access earlier encoder states, making long-range translation difficult. section 3.1.1
Language translation
For translation, an encoder reads the input sentence word by word, and a decoder uses the compressed representation to produce the translated sentence token by token.













