liveBook · Manning

Causal language model

A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.

How 5 Manning books explain it

Natural Language Processing in Action, Second Edition

A causal language model predicts each position using only information from earlier or currently available positions in the sequence.

How it is used
Supports left-to-right text generation and fine-tuning generative models on domain-specific text.
How deep it goes
Defines directional token access, contrasts masked prediction, and demonstrates dataset preparation, collator configuration, and Trainer-based fine-tuning.
Examples
Uses GPT-2, NLPiA2 text lines, causal data collators, and generated completions comparing vanilla and fine-tuned models.
What you can do after
Build it from scratch by preparing tokenized text and fine-tuning GPT-2 with a causal collator and Trainer.

Read how Natural Language Processing in Action, Second Edition explains causal language model →

Retrieval Augmented Generation, The Foundational Ideas

A causal language model predicts each token from preceding tokens and is trained for autoregressive sequence modeling.

How it is used
Scores query-conditioned document tokens for contrastive prompt compression before retrieval-augmented generation.
How deep it goes
Explains next-token probability scoring and its compression role, but does not develop model architecture or training internals.
Examples
Uses Qwen/Qwen2.5-0.5B inside PromptCompressor to score and compress retrieved documents.
What you can do after
Apply it via PromptCompressor to perform query-conditioned document compression.

Read how Retrieval Augmented Generation, The Foundational Ideas explains causal language model →

Deep Learning with R, Third Edition

A causal language model is a language model that predicts tokens sequentially from preceding context.

How it is used
Uses sequential next-token prediction for text generation, then adapts it to instruction-response behavior through fine-tuning.
How deep it goes
Explains generation, tokenizer-model packaging, prompt templates, padding masks, labels, and response-weighted loss without detailing model internals.
Examples
Demonstrates Gemma generation with advice, brownie, and factual prompts, plus a 15,000-example instruction dataset and tokenized training batch.
What you can do after
Apply it via KerasHub to load Gemma, generate continuations, and prepare response-masked batches for instruction fine-tuning.

Read how Deep Learning with R, Third Edition explains causal language model →

Transfer Learning for Natural Language Processing

Causal language modeling trains a model to predict the next token from preceding tokens without access to future tokens.

How it is used
Frames next-token prediction as GPT’s training objective for text generation.
How deep it goes
Defines next-token prediction and future-token masking, but does not detail the objective’s implementation.
Examples
Uses pretrained GPT for text generation, including chatbot applications.
What you can do after
Apply pretrained GPT weights through Transformers pipelines to generate text.

Read how Transfer Learning for Natural Language Processing explains causal language model →

Deep Learning with Python, Third Edition

A language model trained to predict the next token from past tokens only, without access to future context.

How it is used
Frames text generation as sequential next-token prediction and supports autocomplete-style continuation.
How deep it goes
Defines the past-only prediction objective, contrasts masked modeling, and demonstrates generation without deriving model internals.
Examples
Uses Gemma prompts for advice, brownie recipes, and a fabricated historical answer to illustrate continuation quality and failure.
What you can do after
Apply it via KerasHub to instantiate a pretrained CausalLM, choose sampling behavior, and generate text from prompts.

Read how Deep Learning with Python, Third Edition explains causal language model →

Also covered in

What to know first

Related concepts

Questions about causal language model

What is causal language model?

A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.

Which Manning books explain causal language model?

Natural Language Processing in Action, Second Edition, Retrieval Augmented Generation, The Foundational Ideas, Deep Learning with R, Third Edition, Transfer Learning for Natural Language Processing and Deep Learning with Python, Third Edition.

What should I know before learning causal language model?

Manning books that teach causal language model usually introduce language model, GPT-2, GPT-4 and token first.

What is the difference between causal language model and masked language modeling?

A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation. Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.

What kinds of causal language model are there?

Kinds of causal language model include ChatGPT.

What is causal language model a kind of?

Causal language model is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.