Memory swap transfers intermediate neural-network data between CPU memory and GPU memory as computation proceeds, allowing training to use a GPU whose memory is smaller than the model or workload requires. section 4.4.1 section 4.5
Use it when a model cannot fit on one available GPU. The technique is easy to implement, but moving data between memories slows training; data scientists generally implement it, while platform developers should recognize when to recommend it. section 4.4.1 section 4.5
Memory swap belongs to a set of alternatives for GPU memory pressure:
- Gradient accumulation reduces per-step memory use by processing configurable minibatches before applying one cumulative parameter update. section 4.4.1
- Model parallelism divides a large model across multiple GPUs, but can result in low GPU utilization. section 4.5
Fitting a model on a smaller GPU
A data science team wants to train a 24 GB BERT model on a GPU with 10 GB of memory. Memory swap is one candidate memory-saving technique; training will fit at the cost of slower execution. section 4.4.1 section 4.5
