Encoder-decoder architecture

As explained in Inside Deep Learning: Math, Algorithms, Models

A neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.

A sequence-to-sequence (Seq2Seq) model maps an input sequence X = (x₁, …, xT) to an output sequence Y = (y₁, …, yT′). The sequences may contain different tokens, lengths, ordering, and multiple valid mappings, making Seq2Seq more suitable than an RNN that emits one output for every input item. section 11.0 section 11.1

The model separates processing into an encoder and decoder. The encoder converts the source sequence into hidden states; the decoder generates target tokens one at a time. Attention avoids compressing the entire input into one vector: at each decoding step, it weights encoder states and forms a context vector x̄t = Σᵢ αᵢhᵢ. section 11.1 section 11.1.1

An attention-based implementation proceeds through four recurring stages:

• Run the source sequence through an encoder, such as PyTorch nn.GRU, and retain hidden states h₁ … hT. section 11.4
• Use a decoder RNN cell to produce a context for the current output position, using the previous decoder state or token according to the decoding strategy. section 11.1.1 section 11.4
• Score the decoder context against encoder states, normalize the scores into α₁ … αT, and compute the weighted attention context. section 11.4
• Concatenate the attention context with the decoder context, project it through a fully connected layer, and predict the next token until EOS or a maximum length. section 11.4

English-to-French translation

For input “what a nice day,” the target may be “quelle belle journée.” The target can reorder words and differ in length. A trained model may translate “some animals are afraid of fire” as “les animaux ont peur du feu,” while attention scores show which source words influenced each output token. section 11.2

Training commonly combines autoregressive decoding with teacher forcing. For outputs shaped (B, T, V), apply cross-entropy at each time step and ignore padding with ignore_index so the model learns to emit EOS rather than padding. Gradient clipping helps stabilize RNN training; attention heat maps provide an interpretable view of token alignment. section 11.4.2 section 11.6

Figure 11.6 The process of applying attention to predict one item at a time of the output. Each highlighted region shows one of the four steps of encoding, decoding the context, attention, and prediction. This is repeated until an EOS token is found or a maximum limit is reached.

Use the decoding diagram to trace encoding, decoder-context generation, attention, and token prediction for one output position. section 11.4

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Implements encoder-decoder attention, tokenwise loss, decoding, and evaluation while deferring broader translation metrics.

Examples

Demonstrates English-to-French translation, attention heat maps, padding-aware loss, and concrete predicted sentences.

What it assumes

Assumes RNN, GRU, attention, PyTorch modules, datasets, and DataLoaders while assembling them into manageable subcomponents.

What you can do after

Builds and trains an attention-based PyTorch Seq2Seq translator, then inspects predictions and attention scores.

How it connects

Connects encoder-decoder networks with RNNs, GRUs, attention mechanisms, teacher forcing, autoregressive models, and denoising autoencoders.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model that maps an input sequence to an output sequence, often used for translation or generation tasks.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionAn encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
• 

  liveBookmanning.com
  Sutskever's ListA sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
• 

  liveBookmanning.com
  Causal AIA sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Effective Conversational AIAn encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.

Compare how 10 books explain encoder-decoder architecture →

Related concepts

• BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
• decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
• retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.