liveBook · Manning

Training from scratch

Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training.

How 3 Manning books explain it

LLMs in Production

Training from scratch is the process of creating a model by learning all of its parameters from randomly initialized weights rather than adapting a pretrained model.

How it is used
Introduces the foundation-model stage before specialization through fine-tuning and prompting.
How deep it goes
Outlines data preparation, architecture, initialization, training, and monitoring, with a code example and resource estimates.
Examples
Uses a training-from-scratch listing and VRAM estimates for 1-billion- and 70-billion-parameter models.
What you can do after
Evaluate whether to train a foundation model from scratch or fine-tune an existing one, considering data and resource demands.

Read how LLMs in Production explains training from scratch →

Starting Data Analytics with Generative AI and Python

Training from scratch is the process of learning a model's parameters from initialized parameters using task-relevant training data.

How it is used
Customizes sentiment-analysis models when domain-specific accuracy and relevance justify building and training a model independently.
How deep it goes
Explains benefits, costs, and expertise demands but stops short of implementation guidance and training procedures.
Examples
Uses sentiment analysis, labeled datasets, model design, and computational training as the concrete setting.
What you can do after
Evaluate whether domain-specific accuracy justifies the data, expertise, time, and computational investment required.

Read how Starting Data Analytics with Generative AI and Python explains training from scratch →

Sutskever's List

Training from scratch is optimization of a model starting from newly initialized parameters rather than pretrained weights.

How it is used
Explains why initializing and optimizing a neural network can make training difficult, especially as networks grow deeper.
How deep it goes
Explains symmetry breaking and how initialization scale affects gradients, convergence, and learning in earlier layers.
Examples
Compares identical zero or constant weights with random weights, and contrasts weights that are too large or too small.
What you can do after
Understand how initialization can prevent neurons from specializing or destabilize learning through vanishing and exploding gradients.

Read how Sutskever's List explains training from scratch →

Also covered in

What to know first

Related concepts

Questions about training from scratch

What is training from scratch?

Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training.

Which Manning books explain training from scratch?

LLMs in Production, Starting Data Analytics with Generative AI and Python and Sutskever's List.

What should I know before learning training from scratch?

Manning books that teach training from scratch usually introduce language model, backpropagation, BERT and accuracy first.

What is the difference between training from scratch and fine-tuning?

Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training. Fine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.