Encoder-decoder architecture

As explained in Deep Learning with R, Third Edition

A sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.

An encoder-decoder architecture maps one sequence to another. The encoder converts a fixed source sequence into a representation; the decoder uses that representation and previous target tokens to predict the next token. section 15.2 section 15.6

Figure 15.1 Sequence-to-sequence learning: the source sequence is processed by the encoder and is then sent to the decoder. The decoder looks at the target sequence so far and predicts the target sequence offset by one step in the future. During inference, we generate one target token at a time and feed it back into the decoder.

Trace the source sequence through the encoder and into the decoder, which predicts the target sequence one position ahead. section 15.2

Use two operating modes:

• During training, provide the target sequence so the decoder learns next-token prediction conditioned on the encoded source and previous target tokens. section 15.2
• During inference, seed the decoder with a start token, feed each prediction back as the next input, and stop at an end token. section 15.2
• For an RNN design, encode the entire source before decoding; use the encoder representation as the decoder’s initial state. section 15.2.2

R-based GRU translation model

A practical Keras construction embeds the English source, processes it with a bidirectional GRU, and uses the resulting representation to initialize a Spanish decoder GRU. A dense softmax layer predicts the Spanish vocabulary at every step. Compile with Adam and sparse categorical cross-entropy, then generate translations token by token. Example outputs include “You know that.” → “[start] tú lo sabes [end]” and “I will tell you tomorrow.” → “[start] te lo voy mañana a decir [end]”. section 15.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains encoder representations, decoder conditioning, training, inference, and an RNN implementation, while stopping short of detailed derivations.

Examples

Demonstrates English-to-Spanish translation with GRU layers, start and end tokens, generated outputs, and an R/Keras training setup.

What it assumes

Builds on language modeling and next-token prediction while introducing source-sequence encoding and decoder initialization.

What you can do after

Build it from scratch in R/Keras and generate target sequences autoregressively from source inputs.

How it connects

Connects sequence-to-sequence modeling with language modeling, RNNs, GRUs, bidirectional processing, attention, and Transformers.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model that maps an input sequence to an output sequence, often used for translation or generation tasks.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionAn encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
• 

  liveBookmanning.com
  Sutskever's ListA sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
• 

  liveBookmanning.com
  Causal AIA sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Effective Conversational AIAn encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.

Compare how 10 books explain encoder-decoder architecture →

Related concepts

• BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
• decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
• retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.