Backpropagation is the backward application of the chain rule through a deep neural network. A network composes computational layers, each with parameters such as weights and biases. Backpropagation relates changes in the objective to changes in those parameters so they can be refined during training. section 1.1 section 1.4
Write a layer as hₗ = fₗ(hₗ₋₁; θₗ), where θₗ denotes its parameters, and let J be the objective. Once the effect of hₗ on J is known, the chain rule gives ∂J/∂θₗ = (∂J/∂hₗ)(∂hₗ/∂θₗ). It also carries that effect toward the preceding layer: ∂J/∂hₗ₋₁ = (∂J/∂hₗ)(∂hₗ/∂hₗ₋₁). Repeating this step connects the objective to parameters throughout the composition. Stochastic gradient descent uses the resulting parameter derivatives; these equations describe the dependency, not an implementation of the backward pass. section 1.4
Figure 1.3 Deep neural network (DNN) architecture for sentiment analysis
A sentiment-analysis network provides a concrete layered architecture in which parameter refinement must reach beyond the final layer. section 1.1
A useful practice problem is to derive the backward pass through residual connections and use it to explain their benefit. That requires tracing derivatives through the connection rather than treating the network as a single function. section 11.6
