Alpha

As explained in Inside Deep Learning: Math, Algorithms, Models

A learned scalar coefficient that scales the residual branch in ReZero and is initialized to zero at the start of training.

Alpha (α) is a learned scalar that controls a residual branch in ReZero. The block computes h = x + α · ReLU(F(x)) and starts with α = 0. section 14.2

Its training behavior follows a simple progression:

• At initialization, α = 0, so h = x; the residual subnetwork contributes nothing and the architecture behaves linearly. section 14.2.1
• Gradient descent changes α, allowing ReLU(F(x)) to contribute gradually as training discovers useful complexity. section 14.2
• Each residual block learns its own value, selecting when to contribute and helping determine the network’s effective depth. section 14.2.1

Figure 14.5 ReZero architecture with several epochs of training. As the model trains, it can learn to change each α independently, slowly adding hidden layers back into the network. This way, it gradually learns to use the depth and select the total effective depth it desires based on the data. In this case, ReZero learned that it only needed one of its layers; for the second, it learned to keep α ≈ 0.

After training, alpha values can differ across blocks: some layers contribute strongly, while values near zero effectively remove others. section 14.2.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains zero initialization, gradient updates, and independently selected effective depth, but defers implementation details.

Examples

Uses h = x + α · ReLU(F(x)) and diagrams showing zero, changing, and near-zero alpha values.

What it assumes

Assumes familiarity with residual connections and gradient descent while defining alpha’s role from its governing equation.

What you can do after

Evaluate how ReZero initializes residual blocks and learns which layers should contribute.

How it connects

Connects ReZero with residual connections, gradient descent, residual blocks, and effective depth.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

• ReZeroused by · A training trick for residual-style networks that gates residual branches with a learnable scalar initialized to zero so the network starts near linear and gradually becomes more expressive.
• identity functionimplements · An identity function is a function that maps every input to itself unchanged.
• gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.