Decoder-only transformer

As explained in Build a Text-to-Image Generator (from Scratch)

A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.

A decoder-only transformer consists of N identical decoder layers and generates sequences autoregressively. Given a prompt, it extracts semantic meaning and predicts the most likely next token. Unlike an encoder-only transformer, which produces continuous representations, it is designed for generation rather than representation alone. section 2.1.2

Trace generation as a repeated next-token prediction loop:

• Start with a prompt or previously generated tokens. section 2.1.2
• Process the sequence through repeated decoder layers. section 2.1.2
• Predict the next token using all previously predicted tokens, then append it to the sequence. section 2.1.2
• Continue until the required sequence is complete; translation decoders use EOS to signal termination. section 2.1.2

Generating image tokens

In min-DALL-E, a pretrained BART encoder converts text into embeddings, a decoder-only transformer predicts image tokens autoregressively, and a pretrained VQGAN decoder reconstructs those tokens as visual content. Discrete image tokens make the prediction space finite and align image generation with language modeling. section 12.1.2

Figure 12.3 The image generation process in min-DALL-E to generate an image from a text prompt

Follow the complete min-DALL-E inference path from a text prompt through image-token generation to reconstructed visual content. section 12.1.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Explains layer structure, autoregressive flow, and EOS termination, but does not detail decoder-layer internals.

Examples

Uses ChatGPT for text generation, translation decoding with EOS, and min-DALL-E for image-token generation.

What you can do after

Evaluate/understand how decoder-only models turn prompts into sequences through repeated next-token prediction.

How it connects

Connects decoder-only transformers with transformers, encoder-only transformers, encoder–decoder transformers, tokenization, and autoregressive generation.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA Transformer design that uses only decoder stacks for autoregressive generation.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsA decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
• 

  liveBookmanning.com
  Transformers in ActionA decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
• 

  liveBookmanning.com
  Sutskever's ListA transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.
• 

  liveBookmanning.com
  How Large Language Models WorkA decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.
• 

  liveBookmanning.com
  Effective Conversational AIA decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.

Compare how 8 books explain decoder-only transformer →

Related concepts

• GPT modelincludes · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.
• TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
• Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
• masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.