Encoder-decoder architecture
An encoder-decoder architecture is a model design in which an encoder transforms input into an intermediate representation and a decoder uses that representation to produce output.
Natural Language Processing in Action, Second Edition
A neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
- How it is used
- Uses encoder–decoder models to turn input text into translated output text.
- How deep it goes
- Works through Transformer preparation, training, inference, and evaluation; extends PyTorch attention modules rather than implementing multi-head attention from scratch.
- Examples
- Uses translation datasets, tensor-shape checks, sentence translations, attention visualizations, and BLEU evaluation.
- What you can do after
- Apply it via PyTorch `nn.Transformer` to build, train, inspect, and evaluate a translation model.
Inside Deep Learning: Math, Algorithms, Models
A neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
- How it is used
- Builds neural systems for machine translation and other tasks where sequences map to related sequences of different lengths.
- How deep it goes
- Implements encoder-decoder attention, tokenwise loss, decoding, and evaluation while deferring broader translation metrics.
- Examples
- Demonstrates English-to-French translation, attention heat maps, padding-aware loss, and concrete predicted sentences.
- What you can do after
- Builds and trains an attention-based PyTorch Seq2Seq translator, then inspects predictions and attention scores.
Read how Inside Deep Learning: Math, Algorithms, Models explains encoder-decoder architecture →
Deep Learning with Python, Third Edition
A model that maps an input sequence to an output sequence, often used for translation or generation tasks.
- How it is used
- Maps a fixed source sequence to a generated target sequence for tasks such as translation and question answering.
- How deep it goes
- Works through RNN encoder-decoder construction and token-by-token generation, then sketches a Transformer implementation without deriving attention internals.
- Examples
- Uses English-to-Spanish translation code, a generation loop, and sample translations that include both plausible and faulty outputs.
- What you can do after
- Build it from scratch with Keras GRU layers, generate translations token by token, and interpret next-token accuracy cautiously.
Read how Deep Learning with Python, Third Edition explains encoder-decoder architecture →
Knowledge Graphs and LLMs in Action
An encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
- How it is used
- Frames graph representation learning and downstream tasks as separate encoding and task-specific decoding stages.
- How deep it goes
- Explains embeddings, message passing, decoder choices, and reusable PyTorch Geometric implementations without detailing training objectives.
- Examples
- Uses Node2Vec on the karate-club network, transaction classification, and MovieLens user–movie link prediction.
- What you can do after
- Build graph predictors with PyTorch and PyTorch Geometric by selecting an encoder and pairing it with a classification or link-scoring decoder.
Read how Knowledge Graphs and LLMs in Action explains encoder-decoder architecture →
Deep Learning with R, Third Edition
A sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
- How it is used
- Supports machine translation and question answering by predicting a target sequence from a separate source sequence.
- How deep it goes
- Explains encoder representations, decoder conditioning, training, inference, and an RNN implementation, while stopping short of detailed derivations.
- Examples
- Demonstrates English-to-Spanish translation with GRU layers, start and end tokens, generated outputs, and an R/Keras training setup.
- What you can do after
- Build it from scratch in R/Keras and generate target sequences autoregressively from source inputs.
Read how Deep Learning with R, Third Edition explains encoder-decoder architecture →
Sutskever's List
A sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.
- How it is used
- Maps variable-length input sequences to variable-length outputs, especially for machine translation.
- How deep it goes
- Explains recurrent encoding, fixed-vector conditioning, and decoding, emphasizing how compression strains longer inputs.
- Examples
- Uses English-to-French translation and illustrates English-to-Spanish decoding from paired sentences.
- What you can do after
- Understand how an encoder–decoder generates variable-length outputs and why fixed-vector compression limits longer inputs.
Read how Sutskever's List explains encoder-decoder architecture →
Optimization Algorithms: AI techniques for design, planning, and control problems
A neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
- How it is used
- Maps an ordered input sequence to an output sequence, including solutions to discrete combinatorial problems.
- How deep it goes
- Defines sequence-to-sequence modeling and pointer selection, but does not detail network internals.
- Examples
- Uses convex-hull point selection and mentions travelling-salesperson, sorting, and named-entity-recognition tasks.
- What you can do after
- Understand how pointer networks select input items when output categories depend on input length.
Causal AI
A sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
- How it is used
- Deploys conditional sequence generation as components in a causal model over related textual variables.
- How deep it goes
- Works through tokenization, dataset construction, fine-tuning, generation, and log probabilities, but does not detail encoder-decoder internals.
- Examples
- Generates Prince actions from King actions, Kingdom outcomes from paired actions, and King actions from Prince actions.
- What you can do after
- Apply it via Hugging Face Transformers and PyTorch to train, load, generate from, and score conditional text models.
Learn Generative AI with PyTorch
- How it is used
- Uses compression and reconstruction to build autoencoders, translate sequences, and support generative image models.
- How deep it goes
- Explains encoder, decoder, latent space, reconstruction error, and a 20-value digit example, but defers model-specific internals.
- Examples
- Demonstrates handwritten-digit reconstruction, English-to-French translation, text-to-image generation, and black-blond hair image round trips.
- What you can do after
- Build an autoencoder from scratch and apply encoder-decoder representations to reconstruction, translation, and image generation.
Read how Learn Generative AI with PyTorch explains encoder-decoder architecture →
Effective Conversational AI
An encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.
- How it is used
- Supports tasks that transform substantial input sequences into separate, relatively small outputs.
- How deep it goes
- Defines the architecture and contrasts it with encoder-only and decoder-only designs without detailing implementation internals.
- Examples
- Uses translation and summarization as examples of large-input, small-output tasks.
- What you can do after
- Evaluate and select an architecture for generative or non-generative sequence-transformation tasks.
Read how Effective Conversational AI explains encoder-decoder architecture →
Related concepts
- BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
- decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
- retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.