liveBook · Manning

Backpropagation

As explained in Build a DeepSeek Model (From Scratch)

A backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.

Backpropagation computes loss gradients in reverse through a neural network. An input gradient carries the computation to the preceding layer; a weight gradient tells the optimizer how to adjust that layer’s parameters to reduce the loss. section 5.6.1 section 6.4.1

For a linear operation written with row-oriented tensors as z = xW, let g = dL/dz be the incoming gradient. The chain rule gives dL/dx = gWᵀ and dL/dW = xᵀg. The first result continues the backward pass; the second supports the weight update. section 5.6.1

A mixed-precision backward pass

For the input-gradient calculation, an incoming BF16 gradient and the weights are converted to FP8 for multiplication. The result is accumulated in FP32, then stored in BF16 for the preceding layer. This keeps the heavy multiplication in a low-precision format while retaining more precision in the result passed between layers. section 5.6.1

Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Follow the forward and backward data flows through a linear operator, including the precision formats used for weights and gradients. section 5.6.1

In a PyTorch training step, loss.backward() calculates parameter gradients after the forward pass and loss calculation. optimizer.step() uses them to update the weights, and optimizer.zero_grad() clears old gradients before the next mini-batch. section 6.4.1

The reverse dependency also shapes distributed schedules. In a naive GPipe schedule, all micro-batches finish their forward passes before backward passes begin. Each stage waits for the gradient from the stage after it, leaving GPUs idle as the pipeline drains. With nn.DataParallel, by contrast, GPUs compute gradients for their own data slices in parallel, and the main GPU aggregates them. section 6.5.3 section 6.4.4

Figure 6.5 The backward pass in a naive pipeline schedule. The process moves in reverse, starting from the last GPU. The white area to the right of the bars represents the idle time as the pipeline drains.

See how gradients move from the last GPU toward the first and leave idle time as the pipeline drains. section 6.5.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through linear-layer input and weight gradients, mixed-precision data flow, and distributed scheduling without deriving general backpropagation internals.
Examples
Uses a linear layer’s gradients, a PyTorch training step, and a multi-GPU pipeline drain as concrete examples.
What you can do after
Apply it via PyTorch using loss.backward(), optimizer.step(), and optimizer.zero_grad() in a training step.
How it connects
Connects backpropagation to the chain rule, mixed precision, pipeline parallelism, and data parallelism.

Also explained in

Compare how 25 books explain backpropagation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.