Delta, written δ^(l), is an auxiliary variable measuring how rapidly the loss changes with respect to a layer's pre-activation output. It carries the chain-rule signal backward through a neural network, connecting loss changes to weight and bias gradients. section 8.4.5
For a sigmoid output layer with MSE loss, δ^(L) = (a^(L) − y)a^(L)(1 − a^(L)). Earlier layers propagate it backward: δ^(l) = δ^(l+1)w^(l+1)a^(l)(1 − a^(l)). section 8.4.5
PyTorch implementation
A backward pass initializes the final delta from the prediction and target, then iterates from layer L−1 to 0, multiplying the next delta by the connecting weight and sigmoid derivative. The resulting deltas produce weight and bias gradients. section 8.4.5
Equation 8.20
| δ(L) = (a(L)−ȳi) a(L)(1−a(L)) |
Equation 8.20 displays the simplified output-layer delta for sigmoid activation and MSE loss. section 8.4.5
Listing 8.3 PyTorch code for forward and backward propagation
def forward_backward(x, y, W, b):
L = len(W) - 1
a = []
for l in range(0, L+1):
a_prev = x if l == 0 else a[l-1] #1
z_l = Z(a_prev, W[l], b[l])
a_l = A(z_l)
a.append(a_l)
loss = mse_loss(a[L], y) #2
deltas = [None for _ in range(L + 1)]
W_grads = [None for _ in range(L + 1)] #3
b_grads = [None for _ in range(L + 1)]
a_L = a[L] #4
deltas[L] = (a_L - y) * a_L * (1 - a_L)
W_grads[L] = torch.matmul(deltas[L], a[L - 1].T) #5
b_grads[L] = deltas[L]
for l in range(L-1, -1, -1): #6
a_l = a[l]
deltas[l] = torch.matmul(W[l+1].T, deltas[l + 1]) * a_l * (1 - a_l)
W_grads[l] = torch.matmul(deltas[l], a[l - 1].T)
b_grads[l] = deltas[l]
return loss, W_grads, b_grads
Listing 8.3 shows deltas being initialized, recursively propagated, and converted into parameter gradients. section 8.4.5
