A forward pass generates a neural network’s output from an input by evaluating its layers in order. In a layered network, each layer receives only the preceding layer’s output, so an MLP can compute its prediction without explicitly expanding the entire nested function. section 8.2 section 8.3.2
For layer l, compute the pre-activation z^(l) = W^(l)a^(l−1) + b^(l), then apply an activation function a^(l) = σ(z^(l)). The input serves as a^(−1), and the final activation is the network output. Repeating these two operations turns matrix arithmetic and nonlinearities into a prediction. section 8.3.2 section 8.6
A forward pass has three practical properties:
- It proceeds from the layer nearest the input toward the output layer. section 8.6
- It needs only the current and preceding layer’s values during sequential evaluation. section 8.3.2
- It supplies the prediction used to calculate loss before backpropagation produces parameter gradients. section 8.4.6
A PyTorch classifier
A model with two linear layers computes scores by nesting the layers: scores = linear2(linear1(X)). During training, the forward pass returns these raw scores to cross-entropy loss; prediction applies softmax and argmax afterward to obtain a class. section 9.2.3 section 9.2.3
Listing 8.1 PyTorch code for forward propagation
def Z(x, W_l, b_l): #1
return torch.matmul(W_l, x) + b_l
def A(z_l): #2
return torch.sigmoid(z_l)
def forward(x, W, b): #3
L = len(W) - 1
a_l = x
for l in range(0, L + 1): #4
z_l = Z(a_l, W[l], b[l]) #5
a_l = A(z_l) #6
return a_l
The implementation shows the computation directly: calculate each layer’s affine transform, apply sigmoid, feed the activation into the next iteration, and return the final activation. section 8.3.2
