A feed-forward network has a direct computational graph from input to output. A multilayer perceptron (MLP) is one such network: it stacks densely connected layers to make classification or regression predictions. section 10.1
For an input vector x, a dense layer computes Wx + b, where W is its weight matrix and b its bias. Stacking only affine operations limits what the network can express, so hidden layers apply a nonlinearity such as ReLU: fₗ(x) = ReLU(Wₗx + bₗ). Composing these layers carries the input toward the output. section 10.1
Figure 10.1 MLP architecture
The MLP architecture diagram makes the sequence of layers easier to trace. section 10.1
Choose the output and loss for the prediction task:
- For classification, softmax converts output values into class probabilities that sum to 1; cross-entropy measures the prediction against the label. section 10.1
- For regression, use an identity output activation and a loss such as mean squared error. section 10.1
A forward pass computes each layer’s output with the current weights, then computes the loss. A backward pass propagates loss gradients through the layers by the chain rule, producing gradients with respect to both earlier layer inputs and current layer parameters. The backward pass trains the network; it does not change the direct input-to-output path used for prediction. section 10.1
MNIST digit classification
A Keras/TensorFlow MLP implementation loads MNIST images, reshapes each image into a vector, and scales its pixel values before training. Training and validation cross-entropy loss and accuracy provide measures of its classification performance. section 10.1
More layers or hidden units can increase capacity when the model underfits. Weight decay, early stopping, and dropout are options for limiting overfitting. Beyond ordinary prediction, an MLP can also parameterize a Gaussian mixture model in a mixture density network. section 10.6 section 11.2
