Weight initialization
Weight initialization is the process of assigning starting values to a model’s trainable parameters before training or optimization.
Sutskever's List
Weight initialization is the procedure for assigning initial values to a neural network's trainable weights before training.
- How it is used
- Positions starting weights as a factor in whether deep neural networks train stably.
- How deep it goes
- Explains symmetry breaking and vanishing or exploding gradients, then gives AlexNet settings without deriving an initialization formula.
- Examples
- Contrasts identical weights with random weights and uses AlexNet’s Gaussian weights and selected bias values as a concrete configuration.
- What you can do after
- Operate/configure it by choosing distinct initial weights of suitable scale and considering layer-specific biases for stable early training.
Deep Learning with JAX
The process of assigning initial values to neural-network weights before training begins.
- How it is used
- Uses randomness to initialize neural-network parameters before training and to establish a JAX model’s functional structure.
- How deep it goes
- Explains the role of initialization and external PRNG keys, but stops short of specifying initialization distributions or algorithms.
- Examples
- Uses a fully connected network with 784 inputs, 512 hidden neurons, and 10 output neurons to illustrate initialization and application.
- What you can do after
- Build it from scratch by initializing layer parameters with externally supplied random state and passing them into a pure forward pass.
Read how Deep Learning with JAX explains weight initialization →
Related concepts
- initializationkind of · Initialization is the process of establishing initial values or a starting state for a program entity, component, resource, or computation before subsequent operation.
- parameterconfigures · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
- random restartused by · Random restart is a search or optimization technique that repeatedly begins from newly selected initial states to escape local optima or other poor regions.
- autoencoderused by · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
- exploding gradientaddresses · An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or numerical and convergence failures.
- vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.