Post-training

Q: How deeply does Introduction to Generative AI, Second Edition cover post-training?

Explains adaptation through additional data and adjusted model parameters, and names safety methods without implementation-level instructions.

As explained in Introduction to Generative AI, Second Edition

Post-training is the adaptation of a pretrained model using additional task-specific data and optimization rather than training the model from scratch.

A chatbot may be able to generate fluent sentences yet struggle to follow a simple instruction such as “Write me a poem.” Post-training helps bridge that gap. After a large language model’s initial training on vast amounts of data, developers train it further to improve particular abilities or behaviors. section 2.1 section 2.8

The initial phase, called pretraining, gives a model a broad foundation through a token-prediction task. Post-training starts with that existing model rather than building one from scratch. It typically uses additional examples showing the desired task, then adjusts the model’s parameters to fit them. Depending on the data and computing resources, this can take minutes to hours rather than the weeks needed to train a foundation model. section 2.1 section 2.8

Post-training can serve different goals:

Improve capabilities such as instruction following, summarization, question answering, or code generation. section 2.16 section 2.8
Steer responses toward safety policies. Methods include supervised fine-tuning on examples of safe responses and reinforcement learning from human or AI feedback. section 3.0 section 3.8

Making a chatbot concise

Suppose users want short answers to questions with concrete answers, but fuller replies to open-ended questions. A blanket prompt to “answer as concisely as possible” could shorten both. Post-training instead offers a way to show the model examples of when each response style is appropriate. That greater precision comes at a cost: collecting or generating examples requires more effort and technical skill than changing a prompt. section 7.11

Table 7.7 A comparison between prompting and post-training

	Prompting	Post-training
Post-accessibility	High—accessible to all generative AI users	Low—requires data collection or generation and technical skills
Level of precision	Low—small tweaks can have unexpected impacts	High—offers the ability to target narrow changes
Amount of data	A few examples or none	Typically, but not always, hundreds or thousands of examples
Amount of compute required	None for training, but it can require more at inference time	Some for training, but it can require less at inference time (if using a smaller model or fewer prompt tokens)

Compare prompting and post-training by accessibility, precision, example data, and computing needs. section 7.11

Better-targeted behavior does not settle what counts as a safe response: providers may disagree, and safety work can depend on people labeling disturbing content. Post-training is therefore both a technical choice and a decision about whose expectations and labor shape a model’s behavior. section 2.2 section 3.8

How Introduction to Generative AI, Second Edition covers it

Cover of Introduction to Generative AI, Second Edition

liveBook manning.com

Read Introduction to Generative AI, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Reinforcement Learning from Human FeedbackPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
liveBook manning.com

Build a Reasoning Model (From Scratch)Methods applied after initial pretraining to improve model behavior or capabilities.

Compare how 3 books explain post-training →

Related concepts

fine-tuninghas part · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
foundational modeluses · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
reinforcement learninghas part · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
RLAIFincludes · Reinforcement learning from AI feedback is a training or alignment technique that uses AI-generated evaluations or preferences as feedback to optimize a model or agent.
Direct Preference Optimizationincludes · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
reinforcement learning with human feedbackhas part · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.