A graphics processing unit (GPU) is hardware optimized for parallel matrix multiplication. That makes it useful for training neural networks and accelerating other numerical workloads. section 7.1.1
GPUs support parallel computation in two main ways:
- Run independent model-training jobs simultaneously, increasing the information gathered from expensive optimization experiments within the same time. section 7.1.1
- Move Gaussian-process data, models, and likelihoods to the GPU with PyTorch-style cuda() calls to speed matrix-heavy training. section 12.1.2
Batch model training
An ML engineer tuning a neural network can train multiple models at the same time on several GPUs instead of waiting for sequential runs. The same batch idea evaluates multiple BayesOpt queries simultaneously. section 7.1.1
Figure 7.1 Illustration of the benefits of parallelism. Three generic processes either run sequentially (left) or in parallel (right). When running in parallel, the three processes only take one third of the total time required if run sequentially.
The parallel-versus-sequential process diagram makes the throughput benefit of GPU-style parallel execution concrete. section 7.1.1
