8 Scaling up AutoML

 

This chapter covers

  • Loading large datasets into memory batch by batch
  • Using multiple GPUs to speed up search and training
  • Using Hyperband to efficiently schedule model training to make the best use of the available computing resources
  • Using pretrained models and warm-start to accelerate the search process

This chapter introduces various techniques for large-scale training—for example, using large datasets to train large models on multiple GPUs. For datasets that are too big to fit into memory all at once, we’ll show you how to load them batch by batch during the training. We’ll also introduce different parallelization strategies to distribute the training and search processes onto multiple GPUs. In addition, we’ll show you some strategies to accelerate the search process with limited computing resources, using advanced search algorithms and search spaces.

8.1 Handling large-scale datasets

8.1.1 Loading an image-classification dataset

8.1.2 Splitting the loaded dataset

8.1.3 Loading a text-classification dataset

8.1.4 Handling large datasets in general

8.2 Parallelization on multiple GPUs

8.2.1 Data parallelism

8.2.2 Model parallelism

8.2.3 Parallel tuning

8.3 Search speedup strategies

8.3.1 Model scheduling with Hyperband

8.3.2 Faster convergence with pretrained weights in the search space

8.3.3 Warm-starting the search space

Summary