Parallelism runs independent processes simultaneously, reducing total completion time. In expensive black-box optimization, parallel queries gather more information within the same time budget than sequential evaluations. section 7.1.1
Parallel evaluation in practice
An ML engineer can train several neural-network models simultaneously on multiple GPUs, while a scientist can synthesize several drug recipes at once. Three equal-duration processes can take roughly one third as long in parallel as sequentially. section 7.1.1
Batch BayesOpt uses parallelism by selecting multiple query points and evaluating the objective at them simultaneously. Simply choosing the highest acquisition scores can cluster queries in one region, wasting parallel resources; useful batches spread points across the search space. section 7.1.1 section 7.1.2
Figure 7.1 Illustration of the benefits of parallelism. Three generic processes either run sequentially (left) or in parallel (right). When running in parallel, the three processes only take one third of the total time required if run sequentially.
Compare sequential and parallel execution visually: three independent processes finish in one third of the sequential time. section 7.1.1
Figure 7.2 The batch BayesOpt loop. Compared to sequential BayesOpt, batch BayesOpt requires multiple query points to be identified at step 2 and evaluates the objective function at these points simultaneously at step 3.
Trace the batch BayesOpt loop from selecting multiple query points to simultaneous objective evaluation. section 7.1.1
