Gradient accumulation enables a large effective batch on a memory-constrained GPU by processing smaller batches sequentially and combining their gradients before updating the weights. Each smaller-batch forward and backward pass is one accumulation step. The trade-off is more computation time to reduce batch-memory requirements. section 6.4.2 section 6.5.8
For equal-sized batches, effective batch size = physical batch size × accumulation steps. Divide each batch’s averaged loss by the number of accumulation steps before calling backward; otherwise, summing the gradients scales the update relative to the average over the effective batch. section 6.4.1 section 6.4.2
A PyTorch accumulation cycle delays the update and gradient clearing:
- Run the forward pass on a small batch, calculate its loss, and divide the loss by accumulation_steps. section 6.4.2
- Call loss.backward(). PyTorch adds the new gradients to those already stored in each parameter’s .grad attribute. section 6.4.2
- After the configured number of steps, call optimizer.step() once, then optimizer.zero_grad() to begin the next cycle. section 6.4.2
Four batches, one update
With a physical batch size of 16 and four accumulation steps, the effective batch size is 64. After each backward pass, gradients remain stored; after the fourth, the optimizer updates the weights and clears them. Only 16 examples are processed at a time. section 6.4.2
Listing 6.26 Gradient Accumulation Loop (Simplified)
accumulation_steps = 4
# Dataloader now provides small batches of 16
for i, batch in enumerate(dataloader):
# 1. Forward pass (on the small batch)
outputs = model(batch)
loss = loss_function(outputs, batch.labels)
# Normalize the loss
loss = loss / accumulation_steps #A
# 2. Backward pass (accumulates gradients for the small batch)
loss.backward() #B
# 3. Update weights only after enough steps
if (i + 1) % accumulation_steps == 0: #C
optimizer.step() #D
optimizer.zero_grad() #EFollow the loss normalization, accumulated backward passes, and conditional optimizer update in the simplified loop. It updates only on complete four-step cycles. section 6.4.2
Distributed data parallelism adds another factor: effective batch size = per-GPU batch size × number of GPUs × accumulation steps. For eight GPUs processing eight samples each, 64 accumulation steps produce an effective batch of 4,096. Data parallelism addresses training time, while accumulation addresses the batch-size memory limit; each GPU still needs to hold the model. section 6.4.7 section 6.5.8


