How deeply does Causal AI cover GPT-2?

Descends from pretrained-model loading through tokenization, fine-tuning, generation, log probabilities, and Pyro integration.

GPT-2

As explained in Causal AI

GPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.

GPT-2 is used as an autoregressive Transformer language model: it generates text sequentially from a prompt, or from an empty input when modeling a marginal distribution. section 13.3.2

A causal modeling workflow uses GPT-2 to:

Fine-tune pretrained language knowledge toward the distribution of a King’s action vignette. section 13.3.2

King’s action model

An empty input can produce a King’s action vignette such as a declaration of war; the generated sequence receives a model log probability before feeding downstream causal models.

How Causal AI covers it

liveBook manning.com

Read Causal AI on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionGPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
liveBook manning.com

Build a Large Language Model (From Scratch)A family of transformer-based decoder language models pretrained by OpenAI.
liveBook manning.com

Learn Generative AI with PyTorchA decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.
liveBook manning.com

Sutskever's ListGPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
liveBook manning.com

Domain-Specific Small Language ModelsA decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.

Compare how 6 books explain GPT-2 →

Related concepts

transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
decoder-only transformerkind of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
GELUuses · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Causal AI. The section links point to the parts of the book it draws on.