liveBook · Manning

GPT-2

Also called GPT2

GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.

How 7 Manning books explain it

Natural Language Processing in Action, Second Edition

GPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.

How it is used
Uses GPT-2 to demonstrate next-token generation, fine-tuning, and the role of causal models in question answering.
How deep it goes
Works through tokenization, logits, sampling, and a fine-tuning workflow using pretrained components.
Examples
Compares continuations for NLP and neural-network prompts before and after fine-tuning on manuscript sentences.
What you can do after
Apply GPT-2 via Hugging Face Transformers to generate text, inspect next-token probabilities, and fine-tune on a text corpus.

Read how Natural Language Processing in Action, Second Edition explains GPT-2 →

Build a Large Language Model (From Scratch)

A family of transformer-based decoder language models pretrained by OpenAI.

How it is used
Deploys decoder-transformer components to predict the next token and generate text from tokenized prompts.
How deep it goes
Descends from architecture flow to PyTorch modules, tensor shapes, parameter counts, and pretrained weight assignment.
Examples
Demonstrates two four-token batches, 50,257-way logits, model dimension inspection, pretrained loading, and sampled continuation.
What you can do after
Builds a GPT-2 architecture, loads OpenAI weights, and generates continuations with configurable sampling.

Read how Build a Large Language Model (From Scratch) explains GPT-2 →

Learn Generative AI with PyTorch

A decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.

How it is used
Uses GPT-2XL to demonstrate decoder-only text generation, model construction, and loading pretrained weights.
How deep it goes
Works through token embeddings, positional encoding, causal self-attention, decoder blocks, and the assembled model.
Examples
Uses a four-token prompt to illustrate embeddings and a smaller model trained on Hemingway novels to compare generated passages.
What you can do after
Build it from scratch as GPT-2XL, then load pretrained weights to generate text.

Read how Learn Generative AI with PyTorch explains GPT-2 →

Sutskever's List

GPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.

How it is used
Serves as a turning point for discussions of convincing generated text, model scaling, and release risks.
How deep it goes
Describes training scale, zero-shot behavior, and selected outputs, but does not explain transformer internals.
Examples
Uses headline-to-article generation, reading-comprehension questions, and the cherry-picked “Ovid’s Unicorn” passage.
What you can do after
Evaluate/understand its historical significance while distinguishing striking generated samples from reliable general performance.

Read how Sutskever's List explains GPT-2 →

Rearchitecting LLMs

GPT2Model is a GPT-2 transformer model implementation that organizes embeddings, transformer blocks, normalization, and output projection.

How it is used
Maps GPT2Model’s components and high-level data flow for inspecting a DistilGPT2 implementation.
How deep it goes
Traces constructor components and nested forward calls, while deferring detailed operations inside attention and the MLP.
Examples
Uses DistilGPT2 dimensions and a diagram showing embeddings, six blocks, and the block’s module sequence.
What you can do after
Evaluate the model’s component structure and trace its high-level data flow from embeddings through final normalization.

Read how Rearchitecting LLMs explains GPT-2 →

Domain-Specific Small Language Models

A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.

How it is used
Uses GPT-2 for prompt-conditioned text completion, domain fine-tuning, and text-generation serving.
How deep it goes
Explains data preparation and model loading for fine-tuning, and outlines prompt-based API inference; it does not detail model internals.
Examples
Uses paired context-completion data, protein-sequence generation through ProtGPT2, and a GPT-2 text-generation API.
What you can do after
Apply it via Hugging Face Transformers to prepare completion data, load pretrained weights, and serve generated text through FastAPI.

Read how Domain-Specific Small Language Models explains GPT-2 →

Causal AI

GPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.

How it is used
Uses GPT-2 to represent a causal Markov kernel for generating King’s action vignettes.
How deep it goes
Descends from pretrained-model loading through tokenization, fine-tuning, generation, log probabilities, and Pyro integration.
Examples
Generates King’s action vignettes and supplies them to downstream Prince and Kingdom sequence-to-sequence models.
What you can do after
Build and sample a fine-tuned GPT-2 model with Hugging Face Transformers, then expose it as a Pyro distribution.

Read how Causal AI explains GPT-2 →

Also covered in

What to know first

Related concepts

Questions about GPT-2

What is GPT-2?

GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.

Which Manning books explain GPT-2?

Natural Language Processing in Action, Second Edition, Build a Large Language Model (From Scratch), Learn Generative AI with PyTorch, Sutskever's List, Rearchitecting LLMs, Domain-Specific Small Language Models and Causal AI.

What should I know before learning GPT-2?

Manning books that teach GPT-2 usually introduce language model, transformer architecture, generation and token first.

What is GPT-2 a kind of?

GPT-2 is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

What is GPT-2 also called?

GPT-2 is also called GPT2.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.