Encoder-decoder architecture

As explained in Effective Conversational AI

An encoder-decoder architecture is a language-model architecture that encodes an input sequence and decodes a separate output sequence.

An encoder-decoder architecture handles both generative and non-generative tasks by encoding an input sequence and decoding a separate output sequence. It fits problems where the input is large but the desired output is relatively small. section 4.1.4

Translation and summarization

A long source passage can be encoded and converted into a shorter translation or summary. This contrasts with encoder-only architectures, which focus on extracting context, and decoder-only architectures, which generate text by predicting subsequent tokens. section 4.1.4

Figure 4.4 Traditional classification models use supervised learning to predict one of several predefined classifications. They look for the intent, or meaning, of a user input. Generative models use decoding transformers to create a text completion. They predict the next sequence of tokens (loosely, words or characters) that are most likely to occur after the user input.

Compare classification models, which select among predefined intents, with generative models, which decode likely continuations. section 4.1.4

How Effective Conversational AI covers it

How deep it goes

Defines the architecture and contrasts it with encoder-only and decoder-only designs without detailing implementation internals.

Examples

Uses translation and summarization as examples of large-input, small-output tasks.

What it assumes

Assumes familiarity with input sequences, output sequences, classification, and generative AI use cases.

What you can do after

Evaluate and select an architecture for generative or non-generative sequence-transformation tasks.

How it connects

Connects encoder-only architectures, decoder-only architectures, classification models, generative models, translation, and summarization.

liveBookmanning.com

Read Effective Conversational AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA neural architecture that encodes an input sequence into a representation and decodes that representation into an output sequence.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA neural sequence transduction model that maps an input sequence to an output sequence, potentially of different length, using an encoder and decoder.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA model that maps an input sequence to an output sequence, often used for translation or generation tasks.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionAn encoder-decoder architecture is a model design in which an encoder transforms input data into a learned representation and a decoder transforms that representation into predictions or reconstructed outputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA sequence-to-sequence model predicts a target sequence from a separate source sequence, typically conditioning each target token on prior target tokens.
• 

  liveBookmanning.com
  Sutskever's ListA sequence-to-sequence model maps an input sequence to an output sequence using learned encoder and decoder components or an equivalent transformation.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA neural model that transforms an input sequence into an output sequence, typically allowing the two sequences to have different lengths.
• 

  liveBookmanning.com
  Causal AIA sequence-to-sequence model maps an input sequence to an output sequence, often for tasks such as translation, summarization, or conditional generation.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch

Compare how 10 books explain encoder-decoder architecture →

Related concepts

• BARTincludes · BART is a denoising sequence-to-sequence Transformer model with a bidirectional encoder and an autoregressive decoder.
• decodehas part · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• encoderhas part · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• link predictionuses · Link prediction is a graph-learning task that estimates whether a connection exists or may occur between nodes, often by scoring candidate links.
• retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.

This explainer was generated by Manning's knowledge graph from the text of Effective Conversational AI. The section links point to the parts of the book it draws on.