Initialization assigns starting values to a neural network’s hidden states, weights, and biases before training. In an RNN, these values can help different neurons begin tracking sequence position or other temporal patterns. section 8.3.1
Practical initialization choices include:
- Seed a hidden state with zeros, ones, or a gradient of values between zero and one when the task has meaningful state polarity or position. section 8.3.1
- Keep initialization consistent across training to encourage recognizable weight structure and make some learned roles easier to inspect. section 8.3.1
- Master random sampling and shuffling before using advanced, highly structured seeding; overly fixed sample ordering can encourage local minima. section 8.3.1
Sequence-position seeding
An RNN might initialize part of its hidden state with a progression between zero and one so particular neurons begin with a positional pattern. Consistent initialization and sample ordering can preserve that structure, but alphabetizing equal-length texts may make performance uneven across the data. section 8.3.1
