How deeply does Domain-Specific Small Language Models cover GPT-2?

Explains decoder-only generation, data preparation, model loading, and serving, but does not detail Transformer internals.

GPT-2

As explained in Domain-Specific Small Language Models

A decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.

GPT-2 is a decoder-only Transformer language model for autoregressive text generation. The smallest base model has 124 million parameters and can be fine-tuned for domain-specific completion tasks. section 2.1.2 section 8.5

Operationally, GPT-2 fine-tuning involves:

Pairing each context with a target completion. section 2.1.2
Tokenizing, adding sequence markers, and padding or truncating sequences to a consistent length. section 2.1.2
Loading the pretrained model through Hugging Face Transformers and training it with a custom data loader. section 2.1.2

from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained('gpt2')

A minimal Transformers loading pattern initializes GPT-2 for causal language modeling before fine-tuning. section 2.1.2

Serving GPT-2

A FastAPI endpoint can accept a prompt, generate text with the Hugging Face pipeline, and return the response; Docker supports deployment. section 11.2

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionGPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
liveBook manning.com

Build a Large Language Model (From Scratch)A family of transformer-based decoder language models pretrained by OpenAI.
liveBook manning.com

Learn Generative AI with PyTorchA decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.
liveBook manning.com

Sutskever's ListGPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
liveBook manning.com

Causal AIGPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.

Compare how 6 books explain GPT-2 →

Related concepts

transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
decoder-only transformerkind of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
GELUuses · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.