A forward pass runs a model on its inputs to produce predictions. During training, those predictions are compared with targets to obtain a loss. The pass evaluates operations in order—from inputs through intermediate values to the output and, when included in the computation graph, the loss. It does not itself calculate gradients or update trainable weights; those come next in the training loop. section 2.4 section 2.4.4
Follow the values through a small graph
Take a scalar input x = 2, weight w = 3, bias b = 1, and target y_true = 4. Compute x * w = 6, then y = 6 + 1 = 7, then loss_val = abs(4 - 7) = 3. Each value is available only after its inputs have been evaluated. A backward pass works in the opposite direction to find how the loss changes with w and b. section 2.4.4
Two practical distinctions matter when running a training pass:
- Call a Keras model as model(inputs, training=True). Dropout changes behavior between training and inference, so omitting the training setting can change what the pass computes. section 7.4.1
- When requesting gradients, use model.trainable_weights rather than model.weights. Some non-trainable weights are updated by their layers during the forward pass, not by gradient-based updates. section 7.4.1
A forward pass need not process an input in one step. For a recurrent neural network, start with a zero state, visit each sequence element, combine that element with the current state, and use the resulting output as the next state. The sequence is one input even though the network loops internally over its timesteps. section 13.3.1
Listing 13.15 NumPy implementation of a simple RNN
import numpy as np timesteps = 100 #1 input_features = 32 #2 output_features = 64 #3 inputs = np.random.random((timesteps, input_features)) #4 state_t = np.zeros((output_features,)) #5 W = np.random.random((output_features, input_features)) #6 U = np.random.random((output_features, output_features)) #6 b = np.random.random((output_features,)) #6 successive_outputs = [] for input_t in inputs: #7 output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b) #8 successive_outputs.append(output_t) #9 state_t = output_t #10 final_output_sequence = np.concatenate(successive_outputs, axis=0) #11
A NumPy implementation makes the recurrent pass concrete: the loop applies input and state matrices, a bias, and tanh at each timestep, saves each output, and carries it into the next iteration. section 13.3.1
The arithmetic precision of a forward pass is also a practical choice. Mixed-precision training can use float16 for forward computations while retaining float32 for gradient updates. Float8 is not a drop-in replacement: it requires activation rescaling and other changes, and its overhead may outweigh its benefit on smaller models or less powerful GPUs. section 18.3.5
