How deeply does How Large Language Models Work cover decoder-only transformer?

Explains embeddings, position, attention, probabilities, and sampling, but does not derive transformer mathematics or training procedures.

Decoder-only transformer

As explained in How Large Language Models Work

A decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.

A decoder-only transformer generates text by taking a partially written document and predicting a likely next token. It is the architecture behind systems such as GPT and Gemini, though its fluent output should not be confused with human understanding or reasoning. section 3.1

A simple generation loop looks like this:

Represent tokens as numerical embedding vectors and add position information so the model can track word order. section 3.5
Use attention, a fuzzy-dictionary process involving queries, keys, and values, to relate parts of the context. section 3.5
Produce probabilities for every possible next token, then select one through unembedding or sampling. section 3.5
Feed the selected token back with all previous tokens and repeat, making the process autoregressive. section 3.5

Continuing a document

Given a partially written document, the model predicts a likely continuation one token at a time. Adjusting sampling randomness can make the result more consistent or more creative, but not necessarily more accurate. section 3.1 section 3.5

How How Large Language Models Work covers it

liveBook manning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorchA Transformer design that uses only decoder stacks for autoregressive generation.
liveBook manning.com

Time Series Forecasting Using Foundation ModelsA decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
liveBook manning.com

Transformers in ActionA decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
liveBook manning.com

Sutskever's ListA transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.
liveBook manning.com

Effective Conversational AIA decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.

Compare how 8 books explain decoder-only transformer →

Related concepts

GPT modelincludes · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.
TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.