liveBook · Manning

Gradient checkpointing

As explained in LLM Customization and Fine-Tuning

A memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.

When a training run exceeds GPU memory, gradient checkpointing can make room for a larger model or longer sequences. It trades additional compute time for lower activation memory rather than changing the model’s trainable parameters. section 5.1.2

Ordinary training retains intermediate activations from the forward pass for gradient computation. Gradient checkpointing discards activations during that pass and recomputes them as needed during backpropagation. One estimate puts the memory reduction at 30–50%, with 20–30% more compute time; another estimates a 40–60% reduction in peak activation memory. Treat these as ballparks, not guarantees. section 5.1.2 section 6.6.1

Choose the tradeoff for the training workload:

  • For LoRA training, enable checkpointing when activation memory limits model size or sequence length on a single GPU. section 5.1.2
  • For full-parameter supervised fine-tuning, combine checkpointing with a small micro-batch and gradient accumulation. Optimizer states for all parameters also consume memory, so reducing activations does not eliminate the full memory burden. section 6.1.2
  • For direct preference optimization, budget for both the trainable policy and frozen reference model. Checkpointing helps but may not make a full-parameter run fit on one GPU. section 8.2.3
Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
Parameter
Recommended
Notes
num_train_epochs
3
Rarely need more than 5; watch val loss
per_device_train_batch_size
1 to 2
Limited by GPU VRAM
gradient_accumulation_steps
8 to 16
An effective batch of 16 is a good default
learning_rate
2e-5
Range: 1e-5 to 5e-5
warmup_ratio
0.03 to 0.10
Or 50 to 200 fixed steps
weight_decay
0.01
Mild regularization
max_length
512 to 1024
Must cover your longest examples
bf16
True
Use fp16 only if bf16 unsupported
gradient_checkpointing
True
Essential for consumer GPUs
save_total_limit
2 to 3
Disk space: intermediate checkpoints are about 24 GB each with optimizer state; the final model is about 8 GB

Starting settings for full SFT on 4B–7B models include gradient_checkpointing=True, small per-device batches, and gradient accumulation. section 6.1.2

A limit of checkpointing in DPO

A full-parameter DPO run on a 4B model peaked near 54 GB across three cards despite checkpointing. At that run’s sequence length, it did not fit on a single 24 GB card; a LoRA variant was used for single-GPU setups. section 8.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains forward-pass activation disposal and backward-pass recomputation, with estimated memory savings and compute costs.
Examples
Illustrates a full-parameter 4B DPO run peaking near 54 GB across three cards despite checkpointing.
What you can do after
Operate training configurations using checkpointing alongside small batches and gradient accumulation when activation memory is limiting.
How it connects
Connects checkpointing with gradient accumulation, LoRA, supervised fine-tuning, and direct preference optimization.

Also explained in

Compare how 2 books explain gradient checkpointing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.