Decoder-only transformer

As explained in Sutskever's List

A transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.

A decoder-only transformer generates text from left to right, conditioning each token on those before it through masked self-attention. Introduced in 2018, it handles long input sequences better than the original Transformer and scales more easily than encoder-decoder counterparts. section 6.1.1

Scaling experiments compared decoder-only transformers with other architectures and found advantages in maintaining performance across long contexts and computational efficiency at scale. The results also showed that scaling curves were not unique to one architecture. section 6.1.1 section 6.1.5

Directional knowledge

GPT-4 can identify Tom Cruise’s mother when asked about him, yet fail to retrieve Tom Cruise when prompted with her name. This illustrates how left-to-right prediction can make reversing a relation brittle. section 7.5.2

Figure 7.7 BERT is deeply bidirectional, OpenAI’s GPT is unidirectional, and ELMo is shallowly bidirectional.

Compare the directional context available to GPT with the bidirectional representations of BERT. section 7.5.2

How Sutskever's List covers it

How deep it goes

Explains left-to-right conditioning through masked self-attention and compares long-context performance and computational efficiency.

Examples

Contrasts LSTMs' short-sequence performance with their plateau and uses GPT-4's Tom Cruise example to illustrate directional brittleness.

What you can do after

Evaluate architecture trade-offs in long-context performance and computational efficiency at scale.

How it connects

Connects decoder-only transformers to autoregressive prediction, masked self-attention, GPT, BERT, LSTMs, and the Universal Transformer.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA Transformer design that uses only decoder stacks for autoregressive generation.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsA decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
• 

  liveBookmanning.com
  Transformers in ActionA decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
• 

  liveBookmanning.com
  How Large Language Models WorkA decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.
• 

  liveBookmanning.com
  Effective Conversational AIA decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.

Compare how 8 books explain decoder-only transformer →

Related concepts

• GPT modelincludes · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.
• TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
• Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
• masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.