Weight initialization assigns starting values to a neural network’s trainable weights before learning begins. The choice matters most when training deep networks: even ample data and fast GPUs cannot fix an unstable optimization process on their own. section 2.4
Random weights give neurons in the same layer different starting points. If every weight has the same value, those neurons receive the same gradient during backpropagation and update identically. The layer then fails to learn distinct features. Randomness breaks this symmetry, but its scale matters: weights that are too large can produce exploding gradients or saturated neurons; weights that are too small can leave early layers with vanishingly small updates. Both problems intensify with depth. section 2.4
AlexNet’s starting point
AlexNet drew its weights from a zero-mean Gaussian distribution with a small standard deviation of 0.01. It also set biases in specific layers to 1 rather than 0, helping ReLU neurons become active at the start of training instead of getting stuck outputting zero. These choices worked alongside momentum, a decaying learning rate, and other training decisions; initialization was one part of a system, not a standalone cure for difficult optimization. section 2.5.4 section 2.5.5
The practical lesson is to begin with differences between neurons while keeping signals and gradients in a range that permits learning. AlexNet helped demonstrate the value of that combination, though deeper networks still faced vanishing gradients and required further innovations. section 2.4 section 2.5.5
