Post-training
Post-training is the additional training and adaptation of a pretrained model to improve its capabilities, behavior, alignment, or usefulness for particular tasks.
Reinforcement Learning from Human Feedback
Post-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
- How it is used
- Transforms pretrained language models into instruction-following, preference-aligned, and task-capable systems through staged optimization.
- How deep it goes
- Develops multistage recipes, data choices, optimization tools, and evaluation trade-offs rather than treating post-training as a single method.
- Examples
- Uses raw versus assistant-like answers, supportive chat responses, InstructGPT, and the Tülu 3 multistage recipe.
- What you can do after
- Apply it via instruction tuning, preference optimization, and verifiable-reward training while assessing data and evaluation trade-offs.
Read how Reinforcement Learning from Human Feedback explains post-training →
Introduction to Generative AI, Second Edition
Post-training is the adaptation of a pretrained model using additional task-specific data and optimization rather than training the model from scratch.
- How it is used
- Adapts pretrained language models for specific abilities and desired response behaviors.
- How deep it goes
- Explains adaptation through additional data and adjusted model parameters, and names safety methods without implementation-level instructions.
- Examples
- Uses instruction following, code generation, safe responses to unsafe inputs, and a chatbot whose answers need selective shortening.
- What you can do after
- Evaluate/understand when targeted model adaptation may suit a task better than prompting, while recognizing its data, skill, and labor costs.
Read how Introduction to Generative AI, Second Edition explains post-training →
Build a Reasoning Model (From Scratch)
Methods applied after initial pretraining to improve model behavior or capabilities.
- How it is used
- Positions model refinement after pretraining to improve instruction following, preference alignment, and reasoning behavior.
- How deep it goes
- Distinguishes major stages and their ordering but defers detailed reinforcement-learning implementation to later treatment.
- Examples
- Compares pretrained, instruction-tuned, and preference-tuned answers to a sleep-and-health summary prompt.
- What you can do after
- Evaluate how training stages change instruction following, response style, and reasoning-like behavior.
Read how Build a Reasoning Model (From Scratch) explains post-training →
Related concepts
- fine-tuninghas part · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- foundational modeluses · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
- reinforcement learninghas part · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
- RLAIFincludes · Reinforcement learning from AI feedback is a training or alignment technique that uses AI-generated evaluations or preferences as feedback to optimize a model or agent.
- Direct Preference Optimizationincludes · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- reinforcement learning with human feedbackhas part · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.