Weight initialization
Weight initialization is the process of assigning initial values to a model’s trainable parameters before training or optimization.
Build AI Drug Discovery Pipelines
Weight initialization is the procedure for assigning initial numerical values to a model's learnable weights before training.
- How it is used
- Initializes a SMILES autoencoder’s parameters before training to support stable learning.
- How deep it goes
- Explains gradient-scale risks and Kaiming uniform bounds, then implements initialization for linear and embedding layers.
- Examples
- Calculates the sampling range for a 256-output layer and initializes a molecular SMILES autoencoder.
- What you can do after
- Apply it via PyTorch to initialize linear and embedding layers with different strategies.
Read how Build AI Drug Discovery Pipelines explains weight initialization →
Sutskever's List
Weight initialization is the procedure for assigning initial values to a neural network's trainable weights before training.
- How it is used
- Positions starting weights as a factor in whether deep neural networks train stably.
- How deep it goes
- Explains symmetry breaking and vanishing or exploding gradients, then gives AlexNet settings without deriving an initialization formula.
- Examples
- Contrasts identical weights with random weights and uses AlexNet’s Gaussian weights and selected bias values as a concrete configuration.
- What you can do after
- Operate/configure it by choosing distinct initial weights of suitable scale and considering layer-specific biases for stable early training.
Deep Learning with JAX
The process of assigning initial values to neural-network weights before training begins.
- How it is used
- Uses randomness to initialize neural-network parameters before training and to establish a JAX model’s functional structure.
- How deep it goes
- Explains the role of initialization and external PRNG keys, but stops short of specifying initialization distributions or algorithms.
- Examples
- Uses a fully connected network with 784 inputs, 512 hidden neurons, and 10 output neurons to illustrate initialization and application.
- What you can do after
- Build it from scratch by initializing layer parameters with externally supplied random state and passing them into a pure forward pass.
Read how Deep Learning with JAX explains weight initialization →
Also covered in
- Learn Generative AI with PyTorch
- Advanced Algorithms and Data Structures
- CUDA for LLMs
- Grokking AI Algorithms, Second Edition
What to know first
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- sigmoidThe logistic sigmoid is a smooth, S-shaped function that maps real-valued inputs to values between zero and one.
- batch normalizationBatch normalization is a neural-network technique that normalizes activations using statistics computed across a batch during training and stored running statistics during inference.
Related concepts
- parameterconfigures · A parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
- autoencoderused by · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
- exploding gradientaddresses · An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or hindering convergence.
- random restartused by · Random restart is a search or optimization technique that repeatedly begins from newly selected initial states to escape local optima or other poor regions.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink during backpropagation until parameter updates in earlier layers or steps become ineffective.
Questions about weight initialization
What is weight initialization?
Weight initialization is the process of assigning initial values to a model’s trainable parameters before training or optimization.
Which Manning books explain weight initialization?
Build AI Drug Discovery Pipelines, Sutskever's List and Deep Learning with JAX.
What should I know before learning weight initialization?
Manning books that teach weight initialization usually introduce backpropagation, objective function, sigmoid and batch normalization first.
What kinds of weight initialization are there?
Kinds of weight initialization include Kaiming initialization.
What is weight initialization also called?
Weight initialization is also called random initialization or parameter initialization.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


