liveBook · Manning

Gradient accumulation

As explained in Build a DeepSeek Model (From Scratch)

An accumulation step is one smaller-batch computation whose gradients contribute to a later optimizer update.

Gradient accumulation enables a large effective batch on a memory-constrained GPU by processing smaller batches sequentially and combining their gradients before updating the weights. Each smaller-batch forward and backward pass is one accumulation step. The trade-off is more computation time to reduce batch-memory requirements. section 6.4.2 section 6.5.8

For equal-sized batches, effective batch size = physical batch size × accumulation steps. Divide each batch’s averaged loss by the number of accumulation steps before calling backward; otherwise, summing the gradients scales the update relative to the average over the effective batch. section 6.4.1 section 6.4.2

A PyTorch accumulation cycle delays the update and gradient clearing:

  • Run the forward pass on a small batch, calculate its loss, and divide the loss by accumulation_steps. section 6.4.2
  • Call loss.backward(). PyTorch adds the new gradients to those already stored in each parameter’s .grad attribute. section 6.4.2
  • After the configured number of steps, call optimizer.step() once, then optimizer.zero_grad() to begin the next cycle. section 6.4.2

Four batches, one update

With a physical batch size of 16 and four accumulation steps, the effective batch size is 64. After each backward pass, gradients remain stored; after the fourth, the optimizer updates the weights and clears them. Only 16 examples are processed at a time. section 6.4.2

Listing 6.26 Gradient Accumulation Loop (Simplified)
accumulation_steps = 4
# Dataloader now provides small batches of 16
for i, batch in enumerate(dataloader):
    # 1. Forward pass (on the small batch)
    outputs = model(batch)
    loss = loss_function(outputs, batch.labels)
    # Normalize the loss
    loss = loss / accumulation_steps #A
    # 2. Backward pass (accumulates gradients for the small batch)
    loss.backward() #B
    # 3. Update weights only after enough steps
    if (i + 1) % accumulation_steps == 0: #C
        optimizer.step() #D
        optimizer.zero_grad() #E

Follow the loss normalization, accumulated backward passes, and conditional optimizer update in the simplified loop. It updates only on complete four-step cycles. section 6.4.2

Distributed data parallelism adds another factor: effective batch size = per-GPU batch size × number of GPUs × accumulation steps. For eight GPUs processing eight samples each, 64 accumulation steps produce an effective batch of 4,096. Data parallelism addresses training time, while accumulation addresses the batch-size memory limit; each GPU still needs to hold the model. section 6.4.7 section 6.5.8

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces loss normalization, gradient storage, and optimizer scheduling across four iterations, then calculates a distributed batch size.
Examples
Uses batches of 16 accumulated four times and eight GPUs processing eight samples over 64 steps.
What it assumes
Reviews mini-batch forward and backward passes, optimizer updates, and gradient clearing before introducing accumulation.
What you can do after
Apply it via PyTorch by normalizing loss and scheduling backward passes, optimizer updates, and gradient clearing.
How it connects
Combines with distributed data parallelism to address memory and training-time bottlenecks, while distinguishing batch-memory limits from model-size limits.

Also explained in

Compare how 3 books explain gradient accumulation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.