A dying ReLU is an inactive unit that outputs zero and never recovers. In AlexNet training, setting some layer biases to 1 helped activate ReLU units from the outset and mitigate this failure mode. section 2.5.4
AlexNet initialization
Weights began with a zero-mean Gaussian distribution and a small standard deviation of 0.01; selected biases were set to 1 rather than the typical 0 to encourage initial ReLU activation. section 2.5.4
