Pretraining

Q: How deeply does Reinforcement Learning from Human Feedback cover pretraining?

Contrasts pretraining with post-training and next-token prediction, but does not detail datasets, optimization procedures, or model internals.

As explained in Reinforcement Learning from Human Feedback

Pretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.

Pretraining is the large-scale phase that produces a base model with broad knowledge and capabilities through next-token prediction. Its output establishes most of the potential later cultivated by post-training. section 1.4

Next-token prediction does not automatically yield useful conversation. Post-training reshapes that behavior toward question-answering and attempts to expose capabilities that pretraining makes available but does not directly express. section 1.4

Pretraining establishes the starting point for later optimization:

Produces a static base model whose knowledge and capabilities constrain the final model’s potential. section 1.4
Leaves substantial performance gains to mid-training, high-quality end-of-pretraining data, instruction tuning, RLVR, and preference tuning. section 1.4

OLMoE comparison

OLMoE Instruct’s second version changed little in pretraining but improved its popular-benchmark average from 35 to 48 through post-training, illustrating the distinction between creating capability and eliciting it. section 1.4

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with R, Third EditionUnsupervised pretraining is training a model on input data without manually supplied labels before adapting it to downstream tasks.
liveBook manning.com

Deep Learning with Python, Third EditionThe initial large-scale training phase used to learn broad language representations before task-specific fine-tuning.
liveBook manning.com

Build a Large Language Model (From Scratch)The initial training phase in which a language model learns general patterns from a large text corpus before task-specific adaptation.
liveBook manning.com

Time Series Forecasting Using Foundation ModelsPretraining is initial training of a model on a large dataset to learn general patterns and relationships before applying it to specific tasks.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA strategy of training a model on a source dataset before adapting it to a downstream task.
liveBook manning.com

Build a Reasoning Model (From Scratch)The initial training stage in which a language model learns general statistical structure from large amounts of data before task-specific adaptation.
liveBook manning.com

The Art of AI Product DevelopmentPretraining is the initial training of a model on broad data before adapting it to specific tasks or domains.
liveBook manning.com

How Large Language Models WorkPretraining is the initial training of a model on broad data before adapting it to particular tasks or applications.
liveBook manning.com

Introduction to Generative AI, Second EditionThe initial training phase in which a model learns general representations or predictive behavior from a large corpus before task-specific adaptation.
liveBook manning.com

LLMs in ProductionPretraining is the initial training of a model on a large, broad dataset before adapting it to a specific task or domain.
liveBook manning.com

Sutskever's ListPretraining is the initial training of a model on a broad dataset before task-specific adaptation or evaluation.
liveBook manning.com

Data Without LabelsPretraining is an initial model-training stage that learns general patterns or representations before adaptation to a specific task.
liveBook manning.com

Build a Website with ChatGPTPretraining is the initial training of a model on large amounts of data to learn general patterns before task-specific use.

Compare how 14 books explain pretraining →

Related concepts

foundational modelproduces · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
pretrained modelproduces · A pretrained model is a machine-learning model whose parameters were learned through prior training before it is used or adapted for a subsequent task.
next-token predictionuses · Next-token prediction is the task or training objective of estimating the token that follows a sequence, given its preceding tokens and any conditioning information.
datasetuses · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.
zero-shot learningused by · Zero-shot learning is a setting in which a model performs a task or predicts categories without task-specific examples during training, adaptation, or inference, relying on learned knowledge and available task information.
similarity searchused by · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.