Gradient accumulation
Gradient accumulation is a training technique that combines gradients from multiple smaller batches before applying an optimizer update to model parameters.
Build a DeepSeek Model (From Scratch)
An accumulation step is one smaller-batch computation whose gradients contribute to a later optimizer update.
- How it is used
- Enables large effective batches when a physical batch would exceed GPU memory.
- How deep it goes
- Traces loss normalization, gradient storage, and optimizer scheduling across four iterations, then calculates a distributed batch size.
- Examples
- Uses batches of 16 accumulated four times and eight GPUs processing eight samples over 64 steps.
- What you can do after
- Apply it via PyTorch by normalizing loss and scheduling backward passes, optimizer updates, and gradient clearing.
Read how Build a DeepSeek Model (From Scratch) explains gradient accumulation →
Rearchitecting LLMs
Gradient accumulation is a training technique that sums gradients across multiple batches before updating model parameters.
- How it is used
- Uses accumulated gradients to train with larger effective batches when GPU memory is limited.
- How deep it goes
- Implements loss scaling, delayed optimizer updates, and handling for a final incomplete accumulation group.
- Examples
- Illustrates batches of 16 accumulated four times and a LoRA setup using batch size 4 with two accumulation steps.
- What you can do after
- Apply gradient accumulation in a PyTorch training loop or configure it through TRL SFTConfig.
Read how Rearchitecting LLMs explains gradient accumulation →
Building Reliable AI Systems
A technique that sums gradients across multiple mini-batches before updating model parameters.
- How it is used
- Reduces GPU-memory pressure during fine-tuning by accumulating gradients across mini-batches before parameter updates.
- How deep it goes
- Defines the memory-saving mechanism and gives a batch-size-one, four-step configuration.
- Examples
- Uses a per-device batch size of 1 with accumulation over 4 steps to reach an effective batch size of 4.
- What you can do after
- Apply it through Transformers TrainingArguments by accumulating four steps at a per-device batch size of one.
Read how Building Reliable AI Systems explains gradient accumulation →
Also covered in
- Building Specialized AI SystemsIn early access (MEAP)
- Designing Deep Learning Systems: A guide for software engineers
- LLM Customization and Fine-TuningIn early access (MEAP)
- Domain-Specific Small Language Models
- Crack Any Codebase with AIIn early access (MEAP)
- LLMs in Production
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- backpropagationBackpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- forward passA forward pass is a computation that propagates inputs through a model or dependency structure in the forward direction to calculate outputs or intermediate values.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
Related concepts
- batch sizeproduces · Batch size is the number of items grouped for processing together in a single batch or computational operation.
- GPU memoryaddresses · GPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- memory usageaddresses · Memory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
- backpropagationuses · Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
- data parallelismused by · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
Questions about gradient accumulation
What is gradient accumulation?
Gradient accumulation is a training technique that combines gradients from multiple smaller batches before applying an optimizer update to model parameters.
Which Manning books explain gradient accumulation?
Build a DeepSeek Model (From Scratch), Rearchitecting LLMs and Building Reliable AI Systems.
What should I know before learning gradient accumulation?
Manning books that teach gradient accumulation usually introduce objective function, backpropagation, forward pass and memory usage first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


