liveBook · Manning

Parallelism

As explained in Graph Neural Networks in Action

Parallel computing is a technique that divides work among multiple processing units so computations can execute concurrently.

When a graph neural network (GNN) takes too long to train, parallelism can spread training work across multiple processors on one machine or across a cluster. Batching makes this practical by splitting data into portions that can be handled separately. Parallelism can reduce execution time, but additional devices take work to set up and configure. section 7.8 section 7.11

For multi-GPU training in PyTorch, two options are:

  • DataParallel runs on multiple GPUs in one machine. Replicating the model during each forward pass adds cost, particularly as the model and data grow. section 7.8
  • DistributedDataParallel works across GPUs and machines. It uses dedicated CUDA buffers for communication between GPUs and generally has less overhead, making it the preferred option here for larger workloads. section 7.8
Table 7.2 Pros and cons of processor choice
Hardware
Recommended Workload
Pros
Cons
CPU
Preprocessing
Suitable for data collection and preprocessing
More affordable than GPUs and TPUs
Slower for training due to lack of accelerated parallel processing
GPU
Training
Excellent for training due to parallel processing
More expensive than CPUs
Surpassed by TPUs for deep learning tasks
TPU
Preprocessing and training
Faster computation time and cost-effectiveness for deep learning tasks
Requires specific software
infrastructure
Limited to Google platforms
NPU
Training
Optimized for deep learning and especially good for on-device AI applications, reducing reliance on cloud services
Limited to specific AI workloads, primarily neural network-based tasks
IPU
Training
Especially good for graph-based tasks such as GNNs
Can be more complex to program and optimize compared to NPUs

Compare processor workloads and constraints before choosing hardware: GPUs suit parallel training, while other processors differ in cost, software requirements, and specialization. section 7.4.2

Parallelism is one response to a scaling problem, not a substitute for choosing an efficient GNN architecture or managing memory. If the full graph is too large, sampling and batching can reduce the data handled per iteration; remote backends and graph coarsening address other size constraints, each with its own trade-offs. section 7.3.1

How Graph Neural Networks in Action covers it

How deep it goes
Compares two PyTorch parallelization options and their overhead, while stopping short of a configuration walkthrough.
Examples
Compares CPU, GPU, TPU, NPU, and IPU workloads and cites NVIDIA GPUs used on Colab.
What it assumes
Builds on batching as a way to split training data and uses PyTorch terminology for multi-GPU training.
What you can do after
Evaluate/understand when DistributedDataParallel may suit multi-GPU GNN training better than DataParallel.
How it connects
Connects parallelism to batching, distributed computing, GNN architecture selection, sampling, and remote backends.
Cover of Graph Neural Networks in Action
liveBookmanning.com
Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

Compare how 38 books explain parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.