Training from scratch
Training from scratch is the process of optimizing a model from newly initialized parameters without reusing parameters learned through prior training.
LLMs in Production
Training from scratch is the process of creating a model by learning all of its parameters from randomly initialized weights rather than adapting a pretrained model.
- How it is used
- Creates a foundation language model before later finetuning and prompting.
- How deep it goes
- Works through architecture, data preparation, tokenization, initialization, training, and overfitting, with a PyTorch example rather than a full production recipe.
- Examples
- Uses a PyTorch GPT listing and VRAM estimates for 1B- and 70B-parameter models.
- What you can do after
- Build it from scratch by preparing data and a model, running a training loop, and checking generalization against resource costs.
Read how LLMs in Production explains training from scratch →
Starting Data Analytics with Generative AI and Python
Training from scratch is the process of learning a model's parameters from initialized parameters using task-relevant training data.
- How it is used
- Customizes sentiment-analysis models when domain-specific accuracy and relevance justify building and training a model independently.
- How deep it goes
- Explains benefits, costs, and expertise demands but stops short of implementation guidance and training procedures.
- Examples
- Uses sentiment analysis, labeled datasets, model design, and computational training as the concrete setting.
- What you can do after
- Evaluate whether domain-specific accuracy justifies the data, expertise, time, and computational investment required.
Read how Starting Data Analytics with Generative AI and Python explains training from scratch →
Sutskever's List
Training from scratch is optimization of a model starting from newly initialized parameters rather than pretrained weights.
- How it is used
- Explains why initializing and optimizing a neural network can make training difficult, especially as networks grow deeper.
- How deep it goes
- Explains symmetry breaking and how initialization scale affects gradients, convergence, and learning in earlier layers.
- Examples
- Compares identical zero or constant weights with random weights, and contrasts weights that are too large or too small.
- What you can do after
- Understand how initialization can prevent neurons from specializing or destabilize learning through vanishing and exploding gradients.
Related concepts
- fine-tuningcontrasts with · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
- foundational modelalternative to · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
- labeluses · A label is a value or annotation associated with an item that identifies its target category, outcome, or reference answer for learning or evaluation.
- pretrainingcontrasts with · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- open-source modelalternative to · An open-source model is a machine-learning model whose relevant artifacts, such as code or weights, are made available under terms permitting public access and specified forms of use or modification.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.