Training from scratch
Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training.
LLMs in Production
Training from scratch is the process of creating a model by learning all of its parameters from randomly initialized weights rather than adapting a pretrained model.
- How it is used
- Introduces the foundation-model stage before specialization through fine-tuning and prompting.
- How deep it goes
- Outlines data preparation, architecture, initialization, training, and monitoring, with a code example and resource estimates.
- Examples
- Uses a training-from-scratch listing and VRAM estimates for 1-billion- and 70-billion-parameter models.
- What you can do after
- Evaluate whether to train a foundation model from scratch or fine-tune an existing one, considering data and resource demands.
Read how LLMs in Production explains training from scratch →
Starting Data Analytics with Generative AI and Python
Training from scratch is the process of learning a model's parameters from initialized parameters using task-relevant training data.
- How it is used
- Customizes sentiment-analysis models when domain-specific accuracy and relevance justify building and training a model independently.
- How deep it goes
- Explains benefits, costs, and expertise demands but stops short of implementation guidance and training procedures.
- Examples
- Uses sentiment analysis, labeled datasets, model design, and computational training as the concrete setting.
- What you can do after
- Evaluate whether domain-specific accuracy justifies the data, expertise, time, and computational investment required.
Read how Starting Data Analytics with Generative AI and Python explains training from scratch →
Sutskever's List
Training from scratch is optimization of a model starting from newly initialized parameters rather than pretrained weights.
- How it is used
- Explains why initializing and optimizing a neural network can make training difficult, especially as networks grow deeper.
- How deep it goes
- Explains symmetry breaking and how initialization scale affects gradients, convergence, and learning in earlier layers.
- Examples
- Compares identical zero or constant weights with random weights, and contrasts weights that are too large or too small.
- What you can do after
- Understand how initialization can prevent neurons from specializing or destabilize learning through vanishing and exploding gradients.
Also covered in
- Build a Text-to-Image Generator (from Scratch)
- Domain-Specific Small Language Models
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
Related concepts
- fine-tuningcontrasts with · Fine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- foundation modelalternative to · A foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- labeluses · A label is a target value, category, or reference annotation associated with an example for learning, prediction, or evaluation.
- pretrainingcontrasts with · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- open-source modelalternative to · An open-source model is a machine-learning model whose relevant artifacts, such as code or weights, are made available under terms permitting specified forms of access, use, modification, or redistribution.
Questions about training from scratch
What is training from scratch?
Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training.
Which Manning books explain training from scratch?
LLMs in Production, Starting Data Analytics with Generative AI and Python and Sutskever's List.
What should I know before learning training from scratch?
Manning books that teach training from scratch usually introduce language model, backpropagation, BERT and accuracy first.
What is the difference between training from scratch and fine-tuning?
Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training. Fine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


