Memory limitation

As explained in Designing Deep Learning Systems: A guide for software engineers

A limitation imposed when the memory required by training data or a model exceeds the capacity of a device.

A memory limitation occurs when training data or a neural network exceeds the capacity of a local compute device, terminating training with an out-of-memory error. Data parallelism improves training speed but does not remove this constraint. section 4.2.1

Address the limitation according to what exceeds device memory:

• Reduce the batch size when loading training data causes OOM; ensure each minibatch fits into every worker’s memory. section 4.2.1
• Adopt model parallelism or pipeline parallelism when the neural network itself exceeds one device’s capacity. section 4.2.1

Figure 4.1 A synchronous data parallelism concept graph. (a) Deep learning training on a single GPU. (b) Synchronous data parallel training with three GPUs.

Compare single-device training with synchronous data parallelism and see why adding workers does not by itself solve per-device memory limits. section 4.2.1

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Distinguishes dataset OOM from model OOM and gives remedies, but stops short of implementing parallelism.

Examples

Uses smaller batches for dataset OOM and parallel model placement for networks too large for one device.

What it assumes

Assumes basic familiarity with deep-learning training data, models, batches, and compute devices.

What you can do after

Diagnose/debug with it by identifying whether data or model size causes OOM and selecting an appropriate parallelism response.

How it connects

Connects out-of-memory errors with batch size, data parallelism, model parallelism, and pipeline parallelism.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Related concepts

• model parallelismaddressed by · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
• pipeline parallelismaddressed by · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
• memory boundcontrasts with · A memory-bound workload is one whose performance is limited primarily by the rate of accessing or transferring data rather than by computational throughput.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.