Chapter 8: Training neural networks: Forward propagation and backpropagation

Introduction

This chapter introduces supervised neural-network training as fitting MLP parameters from labeled input-output examples, then previews the mathematical tools used to do it.

• Supervised training data: Describe supervised neural networks in terms of training data made of sample inputs paired with known correct outputs, and explain the role of manual labeling.
  Why it matters: Training methods in this chapter use known target outputs to measure and improve a network's predictions.
• Training as parameter estimation: Explain that training estimates neural-network parameter values so the network's outputs are as close as possible to the known correct outputs for training inputs, with the aim of producing reasonably correct outputs on unseen inputs.
  Why it matters: This objective connects the network's adjustable parameters to the loss minimization and backpropagation developed in the chapter.
• Chapter training roadmap: State that the chapter develops sigmoid functions as differentiable surrogates for Heaviside steps, layer representation through matrix-vector multiplication, and regression loss with forward and backward propagation mathematics.
  Why it matters: These are the ingredients the chapter combines to explain how an MLP can be trained.

Key terms: training data, supervised neural networks, labeling, input-output pairs, training, parameter values, multilayer perceptron, MLP, neural network, training inputs

8.1 Differentiable step-like functions

This section introduces differentiable step-like activations as trainable replacements for the perceptron’s nondifferentiable Heaviside step function.

• Differentiability for network training: Explain why the Heaviside step function’s discontinuity and lack of differentiability obstruct gradient-based neural-network training. State that training needs the gradient of loss with respect to weights, and that this gradient consists of partial derivatives.
  Why it matters: Differentiability makes it possible to calculate the weight-update directions used to train a neural network.
• Differentiable step-like replacements: Describe the purpose of differentiable step-like functions: they mimic the Heaviside step behavior while remaining differentiable for training. Identify the sigmoid function as the most significant example introduced for this purpose.
  Why it matters: These functions preserve threshold-like behavior while allowing neural networks to be optimized with gradients.

Key terms: Heaviside step function, discontinuity, differentiable, gradient vector, loss function, partial derivatives, differentiable step-like functions, sigmoid function

8.1.1 Sigmoid function

This section introduces the differentiable sigmoid as an S-shaped, parameterized approximation to a step function and explains its key algebraic and derivative properties.

• Sigmoid curve and parameters: Describe the sigmoid function as an S-shaped curve and explain how its parameters w and b respectively control the steepness of its transition and the position of the curve.
  Why it matters: Parameterizing the sigmoid lets a neural network shift and sharpen a decision-like transition while retaining differentiability.
• Differentiable step approximation: Explain why a sigmoid with a large w is useful as a step-like function: it can become virtually indistinguishable from a Heaviside step function while remaining differentiable.
  Why it matters: Neural-network training needs differentiable functions even when the desired behavior resembles a hard threshold.
• Sigmoid identities and derivative: State the positive-input form, negative-input symmetry, and derivative relationship of the sigmoid, and describe where its derivative is largest or nearly zero on the curve.
  Why it matters: The derivative explains where a sigmoid changes most strongly and supplies the quantity needed for gradient-based training.

Key terms: sigmoid function, parameterized sigmoid function, w, b, Heaviside step function, differentiable, large w, sigmoid, sigmoid of negative x, derivative of sigmoid

8.1.2 Tanh function

This section presents tanh as a zero-centered sigmoid-like activation whose stronger near-zero gradients can speed neural-network training on standardized inputs.

• Tanh output range: Describe tanh as a sigmoid-like hyperbolic tangent activation function whose outputs range from −1 to 1 and are centered around zero, unlike sigmoid outputs from 0 to 1.
  Why it matters: Its zero-centered output changes how activations and gradients behave during training.
• Gradient advantage near zero: Explain why tanh is preferred over sigmoid in this section: tanh has a much larger derivative near x = 0, producing stronger gradients and therefore larger weight-update steps and faster convergence.
  Why it matters: Activation gradients directly affect how quickly gradient-based training can adjust network weights.
• Zero-centered input condition: State that tanh's near-zero gradient advantage mainly applies when the input data is centered around zero, which preprocessing commonly achieves by standardizing data to have mean zero before it enters the neural network.
  Why it matters: The benefit attributed to tanh depends on where the network's inputs lie relative to its high-gradient region.

Key terms: hyperbolic tangent, tanh, sigmoid function, derivative, gradient, weight updates, convergence, sigmoid, centered around 0, standardize

8.2 Why layering?

This section explains how successive-layer connectivity simplifies MLPs and why multiple nonlinear layers can model complex functions more efficiently than a single hidden layer.

• Successive-layer network connectivity: Describe the defining connectivity of a layered MLP: neurons in a layer receive input only from the preceding layer, so connections exist only between successive layers. Explain that this restricted structure simplifies network evaluation and training.
  Why it matters: This layer-by-layer structure is what makes forward propagation and backpropagation manageable.
• Depth versus single hidden layer: Explain that a single hidden layer can in theory model all mathematical functions, but it is not necessarily the most efficient architecture. Describe how organizing neurons in more than one layer can model complicated problems with fewer perceptrons.
  Why it matters: Depth can provide a more efficient way to build expressive models than putting all hidden neurons into one layer.
• Nonlinearity and expressive power: Explain that extra layers help primarily because each layer introduces a nonlinear function, such as a sigmoid. Relate a greater number of properly parameterized nonlinear functions to the ability to model more complicated functions and thus to greater expressive power.
  Why it matters: Nonlinearity is the reason added depth can expand what a neural network can represent.

Key terms: layered network, MLP, preceding layer, successive layers, single hidden layer, perceptrons, Cybenko’s theorem, modeling efficiency, nonlinearity, nonlinear function

8.3 Linear layers

This section introduces the linear, or fully connected, layer as the basic MLP layer and establishes its connection and weight notation.

• Fully connected linear layers: Describe a linear layer as a layer in which every neuron in the preceding layer connects to every neuron in the following layer, and state that it is also called a fully connected layer.
  Why it matters: This connection pattern is the basic structure used to express MLP computations algebraically.
• Connection weight count: State that a fully connected layer with m neurons in the previous layer and n neurons in the next layer has mn connections, with a separate weight for each connection.
  Why it matters: The number of connections determines how many weight parameters the layer must learn.
• Layer weight indexing: Describe the notation w_jk^(l) as the weight from source neuron k in layer l−1 to destination neuron j in layer l, with the destination index written before the source index; also describe the perceptron computation as separated into a weighted-sum layer and a sigmoid layer rather than a single unit.
  Why it matters: This notation and separation make it possible to write layer computations compactly and trace them during forward and backward propagation.

Key terms: linear layer, fully connected layer, perceptron, neuron, MLP, m, n, mn connections, weights, w_jk^(l)

8.3.1 Linear layers expressed as matrix-vector multiplication

This section expresses every neuron in an MLP layer, and then the entire layer, as weighted activations computed compactly with matrix-vector multiplication and an elementwise sigmoid.

• Neuron weighted-input computation: Describe how a neuron in layer l receives activations from layer l − 1, forms their weighted sum (with its bias), and produces its activation by applying the sigmoid function.
  Why it matters: This connects the individual perceptron calculation to the computations performed inside a multilayer network.
• Layer matrix-vector formulation: Explain how the computations of all n neurons in layer l are written together as a matrix-vector operation using W^(l), the activation vector from layer l − 1, a bias vector, and an elementwise sigmoid. State that W^(l) has n rows and m columns for connections from m prior-layer neurons to n current-layer neurons.
  Why it matters: The matrix form is the compact representation used to compute an entire neural-network layer at once.

Key terms: activation, perceptron, weighted sum, bias, sigmoid function, layer l, matrix-vector multiplication, weight matrix, W^(l), activation vector

8.3.2 Forward propagation and grand output functions for an MLP of linear layers

This section explains that an MLP of linear layers computes its final, or grand, output by evaluating each layer sequentially from the previous layer's output rather than explicitly expanding the full nested expression.

• Sequential forward propagation: Describe forward propagation through an MLP of fully connected linear layers as repeated application of the single-linear-layer computation. State that layer 0 is computed from the network input, each later layer is computed from the preceding layer's output, and layer L is the grand output.
  Why it matters: This layer-by-layer view is the practical way to evaluate a multilayer network.
• Layerwise evaluation efficiency: Explain why the complicated overall MLP expression need not be explicitly evaluated: repeated layerwise evaluation produces the same result. Describe that only the previous and current layer outputs need to be kept in memory at a given time.
  Why it matters: Computing one layer at a time makes an MLP easier to implement and reason about.

Key terms: forward propagation, linear layers, fully connected layers, grand output, MLP, previous layer, current layer, memory

8.4 Training and backpropagation

This section introduces neural-network training as iterative gradient-based adjustment of weights and biases to minimize loss across held-out-evaluated training data.

• Training objective and parameters: Explain that training estimates an MLP's weights and biases so its outputs on training inputs become as close as possible to the known ground-truth outputs. Describe training as many small parameter adjustments that may converge but are not guaranteed to reach the globally best values.
  Why it matters: This is the fundamental purpose of training a neural network rather than merely running it for inference.
• Loss over training dataset: Describe loss as a quantitative error measure that decreases when a network output agrees better with ground truth, using mean squared error as the representative example. Explain that training seeks low total loss over the entire training dataset rather than perfect performance on only some instances.
  Why it matters: A dataset-wide loss gives training a measurable objective and prevents improvement on a few examples from disguising poor overall performance.
• Gradient-based parameter updates: Trace how gradients determine small updates to the weight and bias parameters: compute the loss gradient with respect to the parameters, then move in the loss-reducing direction with a learning-rate-scaled update. State that weights may be organized as a matrix for forward propagation or rasterized into a vector for discussing updates.
  Why it matters: Gradients supply the practical direction for changing many network parameters so that loss is reduced.

Key terms: training, parameters, weights, biases, ground truth, MLP, loss, error, mean squared error, MSE

8.4.1 Loss and its minimization: Goal of training

This section defines neural-network training as minimizing loss over training input–ground-truth pairs by choosing all layer weights and biases, motivating iterative gradient descent.

• Training loss over data: Describe loss as a quantity evaluated over a training set of input and ground-truth-output pairs, measuring how well the MLP's outputs match the ground truth.
  Why it matters: Training needs a single objective that says how well the network performs on its examples.
• Unified parameter vectors: Explain that each layer's weight matrix can be rasterized into a vector and concatenated with those of the other layers into one giant weight vector; biases can likewise be collected into one giant bias vector.
  Why it matters: Treating all parameters together lets loss be viewed as a function of the entire MLP rather than of one layer at a time.
• Iterative loss minimization: Explain that training seeks the weight and bias vectors that minimize loss; although a minimum is characterized by zero loss gradients with respect to those parameters, directly computing the combined gradient is intractable, so training uses iterative gradient descent.
  Why it matters: This explains why neural-network training adjusts parameters repeatedly rather than solving for the best parameters in one direct calculation.

Key terms: training data set, ground-truth output, loss, MLP, weight matrix, rasterize, weight vector, bias vector, training, minimize loss

8.4.2 Loss surface and gradient descent

This section presents loss minimization as moving through parameter space along the negative loss gradient and gives the iterative weight and bias update rule for gradient descent.

• Loss-surface parameter geometry: Explain how the loss function can be viewed geometrically as a surface whose domain consists of every weight and bias parameter dimension. Describe a low-dimensional drawing as a visualization of this otherwise high-dimensional parameter space.
  Why it matters: This geometric picture makes training interpretable as navigating parameter values toward lower error.
• Negative-gradient descent direction: Explain that the negative gradient of the loss with respect to the parameters gives the best direction for moving toward a loss minimum. Distinguish this descent path from an arbitrary path that may still reach a minimum but is not optimal.
  Why it matters: Choosing the negative-gradient direction is the principle that turns loss minimization into an effective training method.
• Iterative parameter update rule: Describe the gradient-descent update of both weights and biases: subtract an amount proportional to their respective loss gradients, using the constant r. State that the relevant partial derivatives must be recomputed at every iteration because their values change as the parameters change.
  Why it matters: This update-and-recompute cycle is the concrete procedure by which a network's parameters are trained toward lower loss.

Key terms: loss function, loss surface, parameter space, weights, biases, gradient, negative gradient, gradient descent, minimum, partial derivatives

8.4.3 Why a gradient provides the best direction for descent

This section uses a local Taylor approximation to show that the negative gradient is the direction of greatest immediate loss decrease and introduces the learning rate as its step size.

• Negative-gradient descent direction: Explain why moving in the negative gradient direction gives the largest local decrease in the loss function. Relate this result to the dot product between a proposed parameter offset and the loss gradient.
  Why it matters: This explains the central choice of direction used by gradient descent to train neural networks.
• Local Taylor loss approximation: Describe how a multidimensional Taylor expansion approximates loss at a small parameter offset, and explain why the quadratic and higher-order terms can be ignored for a sufficiently small offset.
  Why it matters: The local approximation is the mathematical basis for treating the negative gradient as the best immediate descent direction.
• Learning-rate step size: State that the proportionality constant scaling a negative-gradient update is the learning rate, a tunable hyperparameter. Explain the trade-off that a larger learning rate progresses faster but can overshoot a minimum.
  Why it matters: Choosing the step size determines whether gradient-based training makes useful progress without jumping past a minimum.

Key terms: gradient, negative gradient, gradient descent, loss, dot product, multidimensional Taylor expansion, Hessian matrix, offset, local approximation, quadratic terms

8.4.4 Gradient descent and local minima

This section explains why gradient descent may settle in a local minimum and why modern neural-network training often accepts that result or retrains from a new random start.

• Local minima and retraining: Explain that gradient descent can get stuck in a local minimum rather than reaching the global minimum, and describe the modern neural-network response: accept an accurate-enough local solution or retrain from a different random starting position to possibly reach a better minimum.
  Why it matters: Training does not guarantee the best possible minimum, so repeated training can be a practical way to improve a model.

Key terms: local minimum, global minimum, gradient descent, retrain, random position, simulated annealing

8.4.5 The backpropagation algorithm

This section derives backpropagation as a layer-by-layer reverse procedure for computing loss gradients and updating an MLP's weights and biases.

• Auxiliary variable meaning: Explain that the auxiliary variable δ for a neuron is the rate of change of loss with respect to that neuron's pre-activation output z. Describe how δ packages the chain-rule information needed to obtain parameter derivatives.
  Why it matters: The auxiliary variables make it possible to compute neural-network gradients without deriving one huge closed-form expression for the whole network.
• Reverse layerwise propagation: Trace the simple-network backpropagation procedure: perform a forward pass from layer 0 through L, compute δ at the last layer, then propagate δ backward from L to 0 while calculating derivatives and updating each layer's parameters. Explain why a layer can be handled using adjacent-layer quantities rather than the entire network at once.
  Why it matters: This reverse layerwise organization is what makes gradient calculation practical for multilayer networks.
• Output-layer delta computation: Describe how the output-layer auxiliary variables are computed directly from the output activation, ground-truth output, and activation derivative for MSE loss. State that this direct output-layer calculation supplies the starting point for backward propagation.
  Why it matters: Backpropagation needs an initial loss-sensitivity signal at the network output before it can propagate information to earlier layers.
• Vector backpropagation equations: Describe the arbitrary multilayer network version of backpropagation: each layer has vectors of activations and auxiliary variables, and the previous layer's δ is computed from the next layer's δ, the transposed connecting weight matrix, and elementwise multiplication by the current activation derivative. Relate these δ vectors to the matrix weight gradient and bias gradient for that layer.
  Why it matters: The vector and matrix form is the scalable form of backpropagation used for layers containing many neurons.

Key terms: auxiliary variable, δ, pre-activation output, loss, chain rule, forward pass, forward propagation, backpropagation, last layer, gradient

8.4.6 Putting it all together: Overall training algorithm

This subsection brings forward propagation, MSE loss, and backpropagated parameter gradients together as the components of neural-network training.

• Neural-network training components: Describe how training combines forward propagation to produce a prediction, MSE loss to compare that prediction with ground truth y, and backpropagation to compute loss gradients with respect to W and b.
  Why it matters: These are the connected computations that make it possible to train a neural network rather than merely run it.

Key terms: forward propagation, output prediction, MSE loss, 𝕃, ground truth, y

8.5 Training a neural network in PyTorch

This section implements demand-prediction neural-network training in PyTorch, from preparing data and defining a model through automatic differentiation, optimization, and prediction.

• Demand prediction data formulation: Describe how the mobile-phone demand example is framed as a regression problem, with three input features and a ground-truth sales target for each historical week.
  Why it matters: A clear mapping from real-world variables to inputs and targets is necessary before a neural network can be trained.
• Feature min-max normalization: Explain why the three features are scaled to the common 0–1 range with min-max normalization before training, despite their originally very different numerical ranges.
  Why it matters: Feature scaling can speed training and reduce the chance that training gets stuck at a local minimum.
• PyTorch network definition: Describe the PyTorch structure of the two-layer neural network: a torch.nn.Module with a forward method, chained linear layers and sigmoid activations, and a final linear layer that produces the regression prediction.
  Why it matters: The model definition specifies how an input matrix is transformed into the predictions that training evaluates.
• MSE loss and autograd: Explain that MSELoss compares model demand predictions with ground-truth sales, and that calling backward on the loss uses PyTorch autograd to compute gradients with respect to the model parameters.
  Why it matters: The loss supplies the training objective, while automatic differentiation supplies the parameter gradients needed to reduce it.
• PyTorch training loop: Trace the repeated training loop: run a forward pass, compute loss, call backward to obtain gradients, and call the SGD optimizer's step method to update weights and biases until loss is acceptably low; then use the trained model on new inputs.
  Why it matters: This loop turns a randomly initialized network into a model whose predictions move closer to the ground truth.

Key terms: regression problem, prev_week_sales, discount_fraction, weeks_to_next_holidays, input vector, X, min-max normalization, normalized data, local minimum, torch.nn.Module

8.6 Summary

This summary recaps differentiable perceptrons, layered MLP structure, and training through loss minimization with forward and backward propagation.