When a training run exceeds GPU memory, gradient checkpointing can make room for a larger model or longer sequences. It trades additional compute time for lower activation memory rather than changing the model’s trainable parameters. section 5.1.2
Ordinary training retains intermediate activations from the forward pass for gradient computation. Gradient checkpointing discards activations during that pass and recomputes them as needed during backpropagation. One estimate puts the memory reduction at 30–50%, with 20–30% more compute time; another estimates a 40–60% reduction in peak activation memory. Treat these as ballparks, not guarantees. section 5.1.2 section 6.6.1
Choose the tradeoff for the training workload:
- For LoRA training, enable checkpointing when activation memory limits model size or sequence length on a single GPU. section 5.1.2
- For full-parameter supervised fine-tuning, combine checkpointing with a small micro-batch and gradient accumulation. Optimizer states for all parameters also consume memory, so reducing activations does not eliminate the full memory burden. section 6.1.2
- For direct preference optimization, budget for both the trainable policy and frozen reference model. Checkpointing helps but may not make a full-parameter run fit on one GPU. section 8.2.3
Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
|
Parameter
|
Recommended
|
Notes
|
|
num_train_epochs
|
3
|
Rarely need more than 5; watch val loss
|
|
per_device_train_batch_size
|
1 to 2
|
Limited by GPU VRAM
|
|
gradient_accumulation_steps
|
8 to 16
|
An effective batch of 16 is a good default
|
|
learning_rate
|
2e-5
|
Range: 1e-5 to 5e-5
|
|
warmup_ratio
|
0.03 to 0.10
|
Or 50 to 200 fixed steps
|
|
weight_decay
|
0.01
|
Mild regularization
|
|
max_length
|
512 to 1024
|
Must cover your longest examples
|
|
bf16
|
True
|
Use fp16 only if bf16 unsupported
|
|
gradient_checkpointing
|
True
|
Essential for consumer GPUs
|
|
save_total_limit
|
2 to 3
|
Disk space: intermediate checkpoints
are about 24 GB each with optimizer state; the final model is about 8 GB
|
Starting settings for full SFT on 4B–7B models include gradient_checkpointing=True, small per-device batches, and gradient accumulation. section 6.1.2
A limit of checkpointing in DPO
A full-parameter DPO run on a 4B model peaked near 54 GB across three cards despite checkpointing. At that run’s sequence length, it did not fit on a single 24 GB card; a LoRA variant was used for single-GPU setups. section 8.2.3

