Encoder-decoder architecture
An encoder-decoder architecture is a neural model design in which an encoder transforms input into a representation and a decoder uses that representation to produce output, often mapping one sequence to another.
Natural Language Processing in Action, Second Edition
A neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
- How it is used
- Uses encoder-decoder models to transform sequences, with Transformer-based translation as the main application.
- How deep it goes
- Explains the recurrent context-vector bottleneck and works through Transformer masks, tensor shapes, training, inference, and evaluation.
- Examples
- Demonstrates German-English translation, a random-tensor shape check, attention inspection, and BLEU evaluation.
- What you can do after
- Apply it via PyTorch to configure, train, and evaluate a translation Transformer and inspect decoder attention weights.
Retrieval Augmented Generation, The Foundational Ideas
An encoder-decoder architecture transforms input into an encoded representation and generates output conditioned on that representation.
- How it is used
- Turns retrieved passages into generated answers in RAG and supports multi-passage synthesis in FiD.
- How deep it goes
- Explains encoder and decoder roles, cross-attention, and FiD’s tensor reshaping through an implementation.
- Examples
- Uses a translator analogy and a three-passage question-answering example with encoder states and a generated answer.
- What you can do after
- Build it from scratch using Hugging Face model calls to encode passages separately and generate from fused states.
Inside Deep Learning: Math, Algorithms, Models
A neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
- How it is used
- Maps input sequences to potentially different-length outputs, with English-to-French machine translation as the working task.
- How deep it goes
- Works through encoder states, stepwise attention and PyTorch components, then addresses loss and training choices.
- Examples
- Uses English-to-French sentences, predicted translations and attention heat maps to illustrate variable lengths and model decisions.
- What you can do after
- Build it from scratch in PyTorch as an attention-based translation model and inspect its predictions and attention scores.
Read how Inside Deep Learning: Math, Algorithms, Models explains encoder-decoder architecture →
Deep Learning with Python, Third Edition
A model that maps an input sequence to an output sequence, often used for translation or generation tasks.
- How it is used
- Frames translation and question answering as generation conditioned on a separate source sequence.
- How deep it goes
- Explains training and token-by-token inference, with Keras RNN construction and a Transformer encoder-decoder template.
- Examples
- Uses English-to-Spanish translation, including generated sentences and a token-by-token decoding loop.
- What you can do after
- Apply it via Keras to construct an RNN translation model and generate target tokens.
Read how Deep Learning with Python, Third Edition explains encoder-decoder architecture →
Build AI Drug Discovery Pipelines
A sequence-to-sequence architecture maps an input sequence to an output sequence, typically through an encoder and decoder.
- How it is used
- Supports de novo molecular design by compressing SMILES sequences and decoding latent representations into candidate molecules.
- How deep it goes
- Explains bidirectional GRU encoding, latent projection, sequential decoding, teacher forcing, and temperature sampling with implementation details.
- Examples
- Uses SMILES reconstruction, bidirectional encoder diagrams, latent-vector decoding, teacher forcing, and temperature-based molecule generation.
- What you can do after
- Build it from scratch with PyTorch by implementing bidirectional encoding, latent sampling, sequential decoding, and molecule generation.
Read how Build AI Drug Discovery Pipelines explains encoder-decoder architecture →
Knowledge Graphs and LLMs in Action
An encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
- How it is used
- Frames graph representation learning and downstream tasks as separate encoding and task-specific decoding stages.
- How deep it goes
- Explains embeddings, message passing, decoder choices, and reusable PyTorch Geometric implementations without detailing training objectives.
- Examples
- Uses Node2Vec on the karate-club network, transaction classification, and MovieLens user–movie link prediction.
- What you can do after
- Build graph predictors with PyTorch and PyTorch Geometric by selecting an encoder and pairing it with a classification or link-scoring decoder.
Read how Knowledge Graphs and LLMs in Action explains encoder-decoder architecture →
Deep Learning with R, Third Edition
A sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
- How it is used
- Supports machine translation and question answering by predicting a target sequence from a separate source sequence.
- How deep it goes
- Explains encoder representations, decoder conditioning, training, inference, and an RNN implementation, while stopping short of detailed derivations.
- Examples
- Demonstrates English-to-Spanish translation with GRU layers, start and end tokens, generated outputs, and an R/Keras training setup.
- What you can do after
- Build it from scratch in R/Keras and generate target sequences autoregressively from source inputs.
Read how Deep Learning with R, Third Edition explains encoder-decoder architecture →
Sutskever's List
A sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.
- How it is used
- Maps variable-length input sequences to variable-length outputs, especially for machine translation.
- How deep it goes
- Explains recurrent encoding, fixed-vector conditioning, and decoding, emphasizing how compression strains longer inputs.
- Examples
- Uses English-to-French translation and illustrates English-to-Spanish decoding from paired sentences.
- What you can do after
- Understand how an encoder–decoder generates variable-length outputs and why fixed-vector compression limits longer inputs.
Read how Sutskever's List explains encoder-decoder architecture →
Optimization Algorithms: AI techniques for design, planning, and control problems
A neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
- How it is used
- Maps an ordered input sequence to an output sequence, including solutions to discrete combinatorial problems.
- How deep it goes
- Defines sequence-to-sequence modeling and pointer selection, but does not detail network internals.
- Examples
- Uses convex-hull point selection and mentions travelling-salesperson, sorting, and named-entity-recognition tasks.
- What you can do after
- Understand how pointer networks select input items when output categories depend on input length.
Building Specialized AI Systems
An encoder-decoder architecture uses an encoder to represent an input sequence and a decoder to generate an output sequence.
- How it is used
- Frames sequence-to-sequence tasks such as language translation through sequential encoding and decoding.
- How deep it goes
- Explains sequential state updates, fixed-size compression, and information loss, then contrasts these limits with Transformer attention.
- Examples
- Uses language translation and a visual RNN encoder-decoder diagram as concrete examples.
- What you can do after
- Evaluate encoder-decoder bottlenecks and understand why attention-based Transformers replace recurrence for longer sequences.
Read how Building Specialized AI Systems explains encoder-decoder architecture →
Causal AI
A sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
- How it is used
- Deploys conditional sequence generation as components in a causal model over related textual variables.
- How deep it goes
- Works through tokenization, dataset construction, fine-tuning, generation, and log probabilities, but does not detail encoder-decoder internals.
- Examples
- Generates Prince actions from King actions, Kingdom outcomes from paired actions, and King actions from Prince actions.
- What you can do after
- Apply it via Hugging Face Transformers and PyTorch to train, load, generate from, and score conditional text models.
Learn Generative AI with PyTorch
- How it is used
- Uses encoder-decoder models to explain image reconstruction, generative modeling, and machine translation.
- How deep it goes
- Defines encoding and decoding through latent representations and reconstruction, without detailing architecture internals.
- Examples
- Uses handwritten-digit reconstruction, English-to-French translation, and image encoding in text-to-image Transformers as examples.
- What you can do after
- Build an autoencoder from scratch to encode handwritten digits into 20-value vectors and reconstruct their images.
Read how Learn Generative AI with PyTorch explains encoder-decoder architecture →
Build an Advanced RAG Application (From Scratch)
An encoder-decoder architecture is a neural-network design in which an encoder transforms an input sequence into representations that a decoder uses to produce an output sequence.
- How it is used
- Deploys encoder-decoder models for sequence-to-sequence generation and semantic retrieval over large document collections.
- How deep it goes
- Explains encoder embeddings, decoder generation, and a retrieval trade-off, without detailing layer-level computation or implementation code.
- Examples
- Uses climate-change articles, an author-specific semantic query, document embeddings, and retrieval as concrete examples.
- What you can do after
- Evaluate when to combine encoders and decoders, use encoders for retrieval-scale data, or choose decoder-only summarization.
Read how Build an Advanced RAG Application (From Scratch) explains encoder-decoder architecture →
Effective Conversational AI
An encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.
- How it is used
- Supports tasks that transform substantial input sequences into separate, relatively small outputs.
- How deep it goes
- Defines the architecture and contrasts it with encoder-only and decoder-only designs without detailing implementation internals.
- Examples
- Uses translation and summarization as examples of large-input, small-output tasks.
- What you can do after
- Evaluate and select an architecture for generative or non-generative sequence-transformation tasks.
Read how Effective Conversational AI explains encoder-decoder architecture →
Also covered in
- Introduction to Generative AI, Second Edition
- Build a Text-to-Image Generator (from Scratch)
- Build a Large Language Model (From Scratch)
- How Large Language Models Work
- Transformers in Action
- Machines that ThinkIn early access (MEAP)
- Generative AI in Action
- Math and Architectures of Deep Learning
- Data Without Labels
- Deep Learning with PyTorch, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
Related concepts
- BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder, pretrained to reconstruct corrupted text.
- decoding strategyhas part · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- encoderhas part · An encoder is a component or process that transforms input into an encoded or internal representation for subsequent processing, transmission, or decoding.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
- retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
Questions about encoder-decoder architecture
What is encoder-decoder architecture?
An encoder-decoder architecture is a neural model design in which an encoder transforms input into a representation and a decoder uses that representation to produce output, often mapping one sequence to another.
Which Manning books explain encoder-decoder architecture?
Natural Language Processing in Action, Second Edition, Retrieval Augmented Generation, The Foundational Ideas, Inside Deep Learning: Math, Algorithms, Models, Deep Learning with Python, Third Edition, Build AI Drug Discovery Pipelines, Knowledge Graphs and LLMs in Action, Deep Learning with R, Third Edition, Sutskever's List and 6 more.
What should I know before learning encoder-decoder architecture?
Manning books that teach encoder-decoder architecture usually introduce language model, recurrent neural network, attention and linear layer first.
What kinds of encoder-decoder architecture are there?
Kinds of encoder-decoder architecture include BART.
What are the parts of encoder-decoder architecture?
Encoder-decoder architecture includes decoding strategy and encoder.
What is encoder-decoder architecture also called?
Encoder-decoder architecture is also called sequence-to-sequence model, encoder-decoder model, seq2seq, encoder–decoder framework or encoder–decoder structure.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













