Causal language model
A causal language model predicts each token from preceding tokens and generates sequences autoregressively.
Natural Language Processing in Action, Second Edition
A causal language model predicts each position using only information from earlier or currently available positions in the sequence.
- How it is used
- Supports left-to-right text generation and fine-tuning generative models on domain-specific text.
- How deep it goes
- Defines directional token access, contrasts masked prediction, and demonstrates dataset preparation, collator configuration, and Trainer-based fine-tuning.
- Examples
- Uses GPT-2, NLPiA2 text lines, causal data collators, and generated completions comparing vanilla and fine-tuned models.
- What you can do after
- Build it from scratch by preparing tokenized text and fine-tuning GPT-2 with a causal collator and Trainer.
Read how Natural Language Processing in Action, Second Edition explains causal language model →
Deep Learning with R, Third Edition
A causal language model is a language model that predicts tokens sequentially from preceding context.
- How it is used
- Uses sequential next-token prediction for text generation, then adapts it to instruction-response behavior through fine-tuning.
- How deep it goes
- Explains generation, tokenizer-model packaging, prompt templates, padding masks, labels, and response-weighted loss without detailing model internals.
- Examples
- Demonstrates Gemma generation with advice, brownie, and factual prompts, plus a 15,000-example instruction dataset and tokenized training batch.
- What you can do after
- Apply it via KerasHub to load Gemma, generate continuations, and prepare response-masked batches for instruction fine-tuning.
Read how Deep Learning with R, Third Edition explains causal language model →
Deep Learning with Python, Third Edition
A language model trained to predict the next token from past tokens only, without access to future context.
- How it is used
- Frames text generation as sequential next-token prediction and supports autocomplete-style continuation.
- How deep it goes
- Defines the past-only prediction objective, contrasts masked modeling, and demonstrates generation without deriving model internals.
- Examples
- Uses Gemma prompts for advice, brownie recipes, and a fabricated historical answer to illustrate continuation quality and failure.
- What you can do after
- Apply it via KerasHub to instantiate a pretrained CausalLM, choose sampling behavior, and generate text from prompts.
Read how Deep Learning with Python, Third Edition explains causal language model →
Related concepts
- masked language modelingcontrasts with · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
- tokenizeruses · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
- ChatGPTincludes · ChatGPT is OpenAI’s conversational AI service, which lets users interact with generative AI models and related tools through natural-language prompts.
- fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- generateimplements · A generate function is an API operation that produces model-generated outputs from specified inputs during inference.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.