liveBook · Manning

Gradient accumulation

Gradient accumulation is a training technique that combines gradients from multiple smaller batches before applying an optimizer update to model parameters.

How 3 Manning books explain it

Build a DeepSeek Model (From Scratch)

An accumulation step is one smaller-batch computation whose gradients contribute to a later optimizer update.

How it is used
Enables large effective batches when a physical batch would exceed GPU memory.
How deep it goes
Traces loss normalization, gradient storage, and optimizer scheduling across four iterations, then calculates a distributed batch size.
Examples
Uses batches of 16 accumulated four times and eight GPUs processing eight samples over 64 steps.
What you can do after
Apply it via PyTorch by normalizing loss and scheduling backward passes, optimizer updates, and gradient clearing.

Read how Build a DeepSeek Model (From Scratch) explains gradient accumulation →

Rearchitecting LLMs

Gradient accumulation is a training technique that sums gradients across multiple batches before updating model parameters.

How it is used
Uses accumulated gradients to train with larger effective batches when GPU memory is limited.
How deep it goes
Implements loss scaling, delayed optimizer updates, and handling for a final incomplete accumulation group.
Examples
Illustrates batches of 16 accumulated four times and a LoRA setup using batch size 4 with two accumulation steps.
What you can do after
Apply gradient accumulation in a PyTorch training loop or configure it through TRL SFTConfig.

Read how Rearchitecting LLMs explains gradient accumulation →

Building Reliable AI Systems

A technique that sums gradients across multiple mini-batches before updating model parameters.

How it is used
Reduces GPU-memory pressure during fine-tuning by accumulating gradients across mini-batches before parameter updates.
How deep it goes
Defines the memory-saving mechanism and gives a batch-size-one, four-step configuration.
Examples
Uses a per-device batch size of 1 with accumulation over 4 steps to reach an effective batch size of 4.
What you can do after
Apply it through Transformers TrainingArguments by accumulating four steps at a per-device batch size of one.

Read how Building Reliable AI Systems explains gradient accumulation →

Also covered in

What to know first

Related concepts

Questions about gradient accumulation

What is gradient accumulation?

Gradient accumulation is a training technique that combines gradients from multiple smaller batches before applying an optimizer update to model parameters.

Which Manning books explain gradient accumulation?

Build a DeepSeek Model (From Scratch), Rearchitecting LLMs and Building Reliable AI Systems.

What should I know before learning gradient accumulation?

Manning books that teach gradient accumulation usually introduce objective function, backpropagation, forward pass and memory usage first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.