liveBook · Manning

Weight initialization

Also called random initialization, parameter initialization

Weight initialization is the process of assigning initial values to a model’s trainable parameters before training or optimization.

How 3 Manning books explain it

Build AI Drug Discovery Pipelines

Weight initialization is the procedure for assigning initial numerical values to a model's learnable weights before training.

How it is used
Initializes a SMILES autoencoder’s parameters before training to support stable learning.
How deep it goes
Explains gradient-scale risks and Kaiming uniform bounds, then implements initialization for linear and embedding layers.
Examples
Calculates the sampling range for a 256-output layer and initializes a molecular SMILES autoencoder.
What you can do after
Apply it via PyTorch to initialize linear and embedding layers with different strategies.

Read how Build AI Drug Discovery Pipelines explains weight initialization →

Sutskever's List

Weight initialization is the procedure for assigning initial values to a neural network's trainable weights before training.

How it is used
Positions starting weights as a factor in whether deep neural networks train stably.
How deep it goes
Explains symmetry breaking and vanishing or exploding gradients, then gives AlexNet settings without deriving an initialization formula.
Examples
Contrasts identical weights with random weights and uses AlexNet’s Gaussian weights and selected bias values as a concrete configuration.
What you can do after
Operate/configure it by choosing distinct initial weights of suitable scale and considering layer-specific biases for stable early training.

Read how Sutskever's List explains weight initialization →

Deep Learning with JAX

The process of assigning initial values to neural-network weights before training begins.

How it is used
Uses randomness to initialize neural-network parameters before training and to establish a JAX model’s functional structure.
How deep it goes
Explains the role of initialization and external PRNG keys, but stops short of specifying initialization distributions or algorithms.
Examples
Uses a fully connected network with 784 inputs, 512 hidden neurons, and 10 output neurons to illustrate initialization and application.
What you can do after
Build it from scratch by initializing layer parameters with externally supplied random state and passing them into a pure forward pass.

Read how Deep Learning with JAX explains weight initialization →

Also covered in

What to know first

Related concepts

Questions about weight initialization

What is weight initialization?

Weight initialization is the process of assigning initial values to a model’s trainable parameters before training or optimization.

Which Manning books explain weight initialization?

Build AI Drug Discovery Pipelines, Sutskever's List and Deep Learning with JAX.

What should I know before learning weight initialization?

Manning books that teach weight initialization usually introduce backpropagation, objective function, sigmoid and batch normalization first.

What kinds of weight initialization are there?

Kinds of weight initialization include Kaiming initialization.

What is weight initialization also called?

Weight initialization is also called random initialization or parameter initialization.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.