GPT-2
GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
Natural Language Processing in Action, Second Edition
GPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
- How it is used
- Uses GPT-2 to demonstrate next-token generation, fine-tuning, and the role of causal models in question answering.
- How deep it goes
- Works through tokenization, logits, sampling, and a fine-tuning workflow using pretrained components.
- Examples
- Compares continuations for NLP and neural-network prompts before and after fine-tuning on manuscript sentences.
- What you can do after
- Apply GPT-2 via Hugging Face Transformers to generate text, inspect next-token probabilities, and fine-tune on a text corpus.
Read how Natural Language Processing in Action, Second Edition explains GPT-2 →
Build a Large Language Model (From Scratch)
A family of transformer-based decoder language models pretrained by OpenAI.
- How it is used
- Deploys decoder-transformer components to predict the next token and generate text from tokenized prompts.
- How deep it goes
- Descends from architecture flow to PyTorch modules, tensor shapes, parameter counts, and pretrained weight assignment.
- Examples
- Demonstrates two four-token batches, 50,257-way logits, model dimension inspection, pretrained loading, and sampled continuation.
- What you can do after
- Builds a GPT-2 architecture, loads OpenAI weights, and generates continuations with configurable sampling.
Read how Build a Large Language Model (From Scratch) explains GPT-2 →
Learn Generative AI with PyTorch
A decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.
- How it is used
- Uses GPT-2XL to demonstrate decoder-only text generation, model construction, and loading pretrained weights.
- How deep it goes
- Works through token embeddings, positional encoding, causal self-attention, decoder blocks, and the assembled model.
- Examples
- Uses a four-token prompt to illustrate embeddings and a smaller model trained on Hemingway novels to compare generated passages.
- What you can do after
- Build it from scratch as GPT-2XL, then load pretrained weights to generate text.
Sutskever's List
GPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
- How it is used
- Serves as a turning point for discussions of convincing generated text, model scaling, and release risks.
- How deep it goes
- Describes training scale, zero-shot behavior, and selected outputs, but does not explain transformer internals.
- Examples
- Uses headline-to-article generation, reading-comprehension questions, and the cherry-picked “Ovid’s Unicorn” passage.
- What you can do after
- Evaluate/understand its historical significance while distinguishing striking generated samples from reliable general performance.
Rearchitecting LLMs
GPT2Model is a GPT-2 transformer model implementation that organizes embeddings, transformer blocks, normalization, and output projection.
- How it is used
- Maps GPT2Model’s components and high-level data flow for inspecting a DistilGPT2 implementation.
- How deep it goes
- Traces constructor components and nested forward calls, while deferring detailed operations inside attention and the MLP.
- Examples
- Uses DistilGPT2 dimensions and a diagram showing embeddings, six blocks, and the block’s module sequence.
- What you can do after
- Evaluate the model’s component structure and trace its high-level data flow from embeddings through final normalization.
Domain-Specific Small Language Models
A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
- How it is used
- Uses GPT-2 for prompt-conditioned text completion, domain fine-tuning, and text-generation serving.
- How deep it goes
- Explains data preparation and model loading for fine-tuning, and outlines prompt-based API inference; it does not detail model internals.
- Examples
- Uses paired context-completion data, protein-sequence generation through ProtGPT2, and a GPT-2 text-generation API.
- What you can do after
- Apply it via Hugging Face Transformers to prepare completion data, load pretrained weights, and serve generated text through FastAPI.
Read how Domain-Specific Small Language Models explains GPT-2 →
Causal AI
GPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.
- How it is used
- Uses GPT-2 to represent a causal Markov kernel for generating King’s action vignettes.
- How deep it goes
- Descends from pretrained-model loading through tokenization, fine-tuning, generation, log probabilities, and Pyro integration.
- Examples
- Generates King’s action vignettes and supplies them to downstream Prince and Kingdom sequence-to-sequence models.
- What you can do after
- Build and sample a fine-tuned GPT-2 model with Hugging Face Transformers, then expose it as a Pyro distribution.
Also covered in
- Transformers in Action
- LLMs in Production
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- generationGeneration is the production of an output by a computational model from supplied input or context, often using patterns learned from data.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- transformer architecturekind of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- decoder-only transformerkind of · A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
- GELUuses · GELU is a smooth neural-network activation function that scales an input by the standard normal cumulative distribution function evaluated at that input.
- language modelkind of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- PyTorchuses · PyTorch is an open-source machine-learning framework that provides tensor computation, automatic differentiation, and tools for developing and training neural networks.
Questions about GPT-2
What is GPT-2?
GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
Which Manning books explain GPT-2?
Natural Language Processing in Action, Second Edition, Build a Large Language Model (From Scratch), Learn Generative AI with PyTorch, Sutskever's List, Rearchitecting LLMs, Domain-Specific Small Language Models and Causal AI.
What should I know before learning GPT-2?
Manning books that teach GPT-2 usually introduce language model, transformer architecture, generation and token first.
What is GPT-2 a kind of?
GPT-2 is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
What is GPT-2 also called?
GPT-2 is also called GPT2.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






