GPT-2
GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
Natural Language Processing in Action, Second Edition
GPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
- How it is used
- Serves as a hands-on causal text generator for inspecting next-token predictions, trying decoding strategies, and fine-tuning generation patterns.
- How deep it goes
- Works through model loading, tokenization, logits, sampling, and a Trainer-based fine-tuning workflow; does not build the transformer from scratch.
- Examples
- Uses “NLP is” and “NLP is a” prompts, ranked next-token candidates, and before-and-after continuations from corpus fine-tuning.
- What you can do after
- Apply it via Hugging Face Transformers to generate text, inspect token predictions, adjust sampling, and fine-tune on a sentence corpus.
Read how Natural Language Processing in Action, Second Edition explains GPT-2 →
Build a Large Language Model (From Scratch)
A family of transformer-based decoder language models pretrained by OpenAI.
- How it is used
- Deploys decoder-transformer components to predict the next token and generate text from tokenized prompts.
- How deep it goes
- Descends from architecture flow to PyTorch modules, tensor shapes, parameter counts, and pretrained weight assignment.
- Examples
- Demonstrates two four-token batches, 50,257-way logits, model dimension inspection, pretrained loading, and sampled continuation.
- What you can do after
- Builds a GPT-2 architecture, loads OpenAI weights, and generates continuations with configurable sampling.
Read how Build a Large Language Model (From Scratch) explains GPT-2 →
Learn Generative AI with PyTorch
A decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.
- How it is used
- Uses GPT-2 as a decoder-only model for building a next-token generator and producing text from prompts.
- How deep it goes
- Works through embeddings, decoder blocks, and model assembly, but loads pretrained weights rather than training the full GPT-2XL model.
- Examples
- Uses a four-token prompt to illustrate embedding shapes and a shark-related prompt to compare generated passages.
- What you can do after
- Build it from scratch as GPT-2XL, load released weights to generate text, or train a smaller GPT-style model.
Sutskever's List
GPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
- How it is used
- Serves as a turning point for discussions of convincing generated text, model scaling, and release risks.
- How deep it goes
- Describes training scale, zero-shot behavior, and selected outputs, but does not explain transformer internals.
- Examples
- Uses headline-to-article generation, reading-comprehension questions, and the cherry-picked “Ovid’s Unicorn” passage.
- What you can do after
- Evaluate/understand its historical significance while distinguishing striking generated samples from reliable general performance.
Domain-Specific Small Language Models
A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
- How it is used
- Supports text completion, domain fine-tuning, protein-sequence generation, and HTTP-served text generation.
- How deep it goes
- Explains decoder-only generation, data preparation, model loading, and serving, but does not detail Transformer internals.
- Examples
- Uses context-completion pairs, protein sequences, and a FastAPI prompt-generation endpoint.
- What you can do after
- Builds fine-tuning datasets, loads GPT-2, generates completions, and serves inference through FastAPI.
Read how Domain-Specific Small Language Models explains GPT-2 →
Causal AI
GPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.
- How it is used
- Uses GPT-2 to represent a causal Markov kernel for generating King’s action vignettes.
- How deep it goes
- Descends from pretrained-model loading through tokenization, fine-tuning, generation, log probabilities, and Pyro integration.
- Examples
- Generates King’s action vignettes and supplies them to downstream Prince and Kingdom sequence-to-sequence models.
- What you can do after
- Build and sample a fine-tuned GPT-2 model with Hugging Face Transformers, then expose it as a Pyro distribution.
Related concepts
- transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- decoder-only transformerkind of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
- PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
- GELUuses · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
- language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.