Batching groups training examples for a weight update. A mini-batch gives a less noisy gradient than a single example, but the whole mini-batch must fit in GPU memory. A micro-batch is a smaller chunk of a batch that can be processed as a separate computation unit. section 6.4.1 section 6.5.1
In a standard training step, the model computes predictions for a mini-batch, averages the loss, and backpropagates to calculate gradients. The optimizer updates the weights, then clears the gradients before the next mini-batch. For a batch of B examples, the batch gradient is the average of their individual gradients. section 6.4.1
The batch gradient averages the gradients of the B examples. section 6.4.1
Splitting batches supports different scaling strategies:
- Gradient accumulation processes smaller mini-batches sequentially and accumulates their gradients before one weight update. This simulates a larger effective batch on memory-constrained hardware. section 6.6
- Data parallelism assigns batch shards to GPUs with identical model replicas. Each GPU computes local gradients; the gradients are then aggregated and averaged so replicas apply the same update. section 6.4.3
- Pipeline parallelism splits the model into stages and feeds micro-batches through them. Different stages can work on different micro-batches, but stages must wait for inputs or gradients from their neighbors. section 6.5.1 section 6.5.3
Filling a four-GPU pipeline
A batch of 32 can be split into eight micro-batches of four. In a four-stage pipeline, the first GPU starts the first micro-batch; the next GPU can start it only after receiving its activations. Meanwhile, the first GPU can start the second micro-batch. The initial waiting time forms part of the pipeline bubble. section 6.5.1 section 6.5.3
Figure 6.4 The forward pass in a naive pipeline schedule. Each GPU must wait for the previous one to complete its work on a micro-batch. The white area to the left of the bars represents the initial idle time (the "bubble") as the pipeline fills up.

The forward-pass timeline shows how dependencies stagger the stages as the pipeline fills. section 6.5.3






















