liveBook · Manning

Causal language model

As explained in Natural Language Processing in Action, Second Edition

A causal language model predicts each position using only information from earlier or currently available positions in the sequence.

A causal language model predicts each token from tokens earlier in the sequence, following the left-to-right order used when generating ordinary text. During training, a causal data collator prevents predicted tokens from using later prompt tokens. section 10.2.2 section 8.1.3

Distinguish causal prediction from masked or bidirectional modeling:

  • Set mlm=False when training on ordinary left-to-right text; set it to True for randomly masked-token datasets such as those used with BERT. section 10.2.2
  • Use causal models for normal text generation, while bidirectional or acausal models use information from both directions for tasks such as masking and OCR correction. section 10.2.2 section 8.1.3

Fine-tuning GPT-2

Load text lines, tokenize them with truncation and padding, split the dataset into training and evaluation sets, and configure Hugging Face Trainer with DataCollatorForLanguageModeling(mlm=False). Fine-tuning GPT-2 on the resulting dataset can change a completion such as “NLP is” into domain-shaped text.

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Defines directional token access, contrasts masked prediction, and demonstrates dataset preparation, collator configuration, and Trainer-based fine-tuning.
Examples
Uses GPT-2, NLPiA2 text lines, causal data collators, and generated completions comparing vanilla and fine-tuned models.
What it assumes
Assumes familiarity with tokenizers, PyTorch datasets, GPT-2, and Hugging Face Trainer configuration.
What you can do after
Build it from scratch by preparing tokenized text and fine-tuning GPT-2 with a causal collator and Trainer.
How it connects
Connects causal language models with masked language models, bidirectional language models, transformers, attention matrices, and recurrent neural networks.
Cover of Natural Language Processing in Action, Second Edition
liveBookmanning.com
Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

Compare how 3 books explain causal language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.