A distributed system divides a large workload across networked devices or cluster nodes so computation can run concurrently. In deep learning, it becomes essential when models no longer fit in one GPU or when training and inference need to move faster. section 3.3 section 3.3.2
Choose a parallelism strategy according to what limits the workload:
- Data parallelism splits the data across multiple copies of a model or pipeline; PyTorch DistributedDataParallel supports this pattern, but each model copy must fit on one GPU. section 3.3.2
- Model parallelism divides the model across devices when replicating the whole model is not practical. section 3.3.2
- Tensor parallelism breaks tensors across dimensions, while sequence parallelism reduces memory overhead; combining them can parallelize more of a transformer. section 3.3.2
- Microbatching reduces the bubble created when GPUs holding model weights wait for other stages to process data. section 3.3.2
Parallelizing a slow task with Ray
A sequential loop calls slow_function ten times, each pausing for one second. Decorating slow_function_ray with @ray.remote submits the calls concurrently; ray.get collects their results. The example uses ray.init() and ray.shutdown() to manage the runtime. section 3.3.2
Listing 3.3 Example Ray parallelization task
import ray
import time
ray.init() #1
def slow_function(x): #2
time.sleep(1)
return x
@ray.remote
def slow_function_ray(x): #3
time.sleep(1)
return x
results = [slow_function(i) for i in range(1, 11)] #4
results_future = [slow_function_ray.remote(i) for i in range(1, 11)] #5
results_ray = ray.get(results_future)
print("Results without Ray: ", results)
print("Results with Ray: ", results_ray)
ray.shutdown()
Inspect the minimal Ray listing to compare sequential function calls with remotely submitted concurrent tasks. section 3.3.2
