liveBook · Manning

GPU

As explained in Graph Neural Networks in Action

A graphics processing unit (GPU) is a massively parallel processor designed to accelerate graphics, numerical, and machine-learning computations.

A graphics processing unit (GPU) is a parallel processor suited to intensive numerical workloads. In GNN workflows, GPUs commonly accelerate training because many computations can run concurrently. section 7.4.1 section 7.4.2

Choose and scale GPU use around workload size and available memory:

  • Use GPU parallelism primarily for model training; CPUs remain useful for preprocessing. section 7.4.2
  • Check processor memory because it limits the workload scale, especially for large data or complex architectures. section 7.4.1
  • Use multiple GPUs with PyTorch DistributedDataParallel to distribute computation and shorten training iterations. section 7.4.1
  • Balance speed against cost: GPUs accelerate training but generally cost more than CPUs and can be surpassed by TPUs for deep-learning workloads. section 7.4.2

Typical training hardware

PyTorch and PyG examples commonly run on NVIDIA GPUs such as the Tesla T4, A100, and V100, including instances available through Colab. section 7.4.2

Table 7.2 Pros and cons of processor choice
Hardware
Recommended Workload
Pros
Cons
CPU
Preprocessing
Suitable for data collection and preprocessing
More affordable than GPUs and TPUs
Slower for training due to lack of accelerated parallel processing
GPU
Training
Excellent for training due to parallel processing
More expensive than CPUs
Surpassed by TPUs for deep learning tasks
TPU
Preprocessing and training
Faster computation time and cost-effectiveness for deep learning tasks
Requires specific software
infrastructure
Limited to Google platforms
NPU
Training
Optimized for deep learning and especially good for on-device AI applications, reducing reliance on cloud services
Limited to specific AI workloads, primarily neural network-based tasks
IPU
Training
Especially good for graph-based tasks such as GNNs
Can be more complex to program and optimize compared to NPUs

Compare GPUs with CPUs, TPUs, NPUs, and IPUs by workload, benefits, and limitations. section 7.4.2

How Graph Neural Networks in Action covers it

How deep it goes
Explains parallelism, memory constraints, multi-GPU distribution, and trade-offs, but does not detail GPU architecture.
Examples
Uses NVIDIA Tesla T4, A100, and V100 GPUs available through Colab as training examples.
What you can do after
Evaluate GPU suitability, memory needs, cost trade-offs, and multi-GPU scaling for GNN training.
How it connects
Connects GPUs to GNNs, PyTorch, PyG, DistributedDataParallel, CPUs, TPUs, NPUs, and IPUs.
Cover of Graph Neural Networks in Action
liveBookmanning.com
Read Graph Neural Networks in Action on liveBook Get the book at manning.com

Also explained in

Compare how 14 books explain GPU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Graph Neural Networks in Action. The section links point to the parts of the book it draws on.