Causal language model
A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.
Natural Language Processing in Action, Second Edition
A causal language model predicts each position using only information from earlier or currently available positions in the sequence.
- How it is used
- Supports left-to-right text generation and fine-tuning generative models on domain-specific text.
- How deep it goes
- Defines directional token access, contrasts masked prediction, and demonstrates dataset preparation, collator configuration, and Trainer-based fine-tuning.
- Examples
- Uses GPT-2, NLPiA2 text lines, causal data collators, and generated completions comparing vanilla and fine-tuned models.
- What you can do after
- Build it from scratch by preparing tokenized text and fine-tuning GPT-2 with a causal collator and Trainer.
Read how Natural Language Processing in Action, Second Edition explains causal language model →
Retrieval Augmented Generation, The Foundational Ideas
A causal language model predicts each token from preceding tokens and is trained for autoregressive sequence modeling.
- How it is used
- Scores query-conditioned document tokens for contrastive prompt compression before retrieval-augmented generation.
- How deep it goes
- Explains next-token probability scoring and its compression role, but does not develop model architecture or training internals.
- Examples
- Uses Qwen/Qwen2.5-0.5B inside PromptCompressor to score and compress retrieved documents.
- What you can do after
- Apply it via PromptCompressor to perform query-conditioned document compression.
Read how Retrieval Augmented Generation, The Foundational Ideas explains causal language model →
Deep Learning with R, Third Edition
A causal language model is a language model that predicts tokens sequentially from preceding context.
- How it is used
- Uses sequential next-token prediction for text generation, then adapts it to instruction-response behavior through fine-tuning.
- How deep it goes
- Explains generation, tokenizer-model packaging, prompt templates, padding masks, labels, and response-weighted loss without detailing model internals.
- Examples
- Demonstrates Gemma generation with advice, brownie, and factual prompts, plus a 15,000-example instruction dataset and tokenized training batch.
- What you can do after
- Apply it via KerasHub to load Gemma, generate continuations, and prepare response-masked batches for instruction fine-tuning.
Read how Deep Learning with R, Third Edition explains causal language model →
Transfer Learning for Natural Language Processing
Causal language modeling trains a model to predict the next token from preceding tokens without access to future tokens.
- How it is used
- Frames next-token prediction as GPT’s training objective for text generation.
- How deep it goes
- Defines next-token prediction and future-token masking, but does not detail the objective’s implementation.
- Examples
- Uses pretrained GPT for text generation, including chatbot applications.
- What you can do after
- Apply pretrained GPT weights through Transformers pipelines to generate text.
Read how Transfer Learning for Natural Language Processing explains causal language model →
Deep Learning with Python, Third Edition
A language model trained to predict the next token from past tokens only, without access to future context.
- How it is used
- Frames text generation as sequential next-token prediction and supports autocomplete-style continuation.
- How deep it goes
- Defines the past-only prediction objective, contrasts masked modeling, and demonstrates generation without deriving model internals.
- Examples
- Uses Gemma prompts for advice, brownie recipes, and a fabricated historical answer to illustrate continuation quality and failure.
- What you can do after
- Apply it via KerasHub to instantiate a pretrained CausalLM, choose sampling behavior, and generate text from prompts.
Read how Deep Learning with Python, Third Edition explains causal language model →
Also covered in
- Build an Advanced RAG Application (From Scratch)In early access (MEAP)
- Causal AI
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- GPT-2GPT-2 is a family of autoregressive, decoder-only Transformer language models developed by OpenAI that generate text by predicting successive tokens from preceding context.
- GPT-4GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- GPT modelused by · A GPT model is a generative, autoregressive Transformer language model in the family developed by OpenAI.
- language modelkind of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- masked language modelingcontrasts with · Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
- ChatGPTincludes · ChatGPT is OpenAI’s conversational AI service, which lets users interact with generative AI models and related tools through natural-language prompts.
- tokenizeruses · A tokenizer is a text-processing component or procedure that divides text into discrete tokens and may map them to identifiers according to specified rules or a vocabulary.
Questions about causal language model
What is causal language model?
A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.
Which Manning books explain causal language model?
Natural Language Processing in Action, Second Edition, Retrieval Augmented Generation, The Foundational Ideas, Deep Learning with R, Third Edition, Transfer Learning for Natural Language Processing and Deep Learning with Python, Third Edition.
What should I know before learning causal language model?
Manning books that teach causal language model usually introduce language model, GPT-2, GPT-4 and token first.
What is the difference between causal language model and masked language modeling?
A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation. Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
What kinds of causal language model are there?
Kinds of causal language model include ChatGPT.
What is causal language model a kind of?
Causal language model is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




