Data parallelism splits data across multiple devices or nodes, each running a separate copy of the model or pipeline. It is often the easiest parallelization approach and can reduce training or inference time through concurrent computation. section 3.3.2
Use data parallelism with these practical constraints in mind:
- Configure PyTorch DistributedDataParallel when the model fits on one GPU. section 3.3.2
- Use distributed-computing tools such as Ray to execute concurrent work across a cluster. section 3.3.2
- Combine data, tensor, and pipeline parallelism as 3D parallelism when one approach alone is insufficient. section 3.5
Listing 3.3 Example Ray parallelization task
import ray
import time
ray.init() #1
def slow_function(x): #2
time.sleep(1)
return x
@ray.remote
def slow_function_ray(x): #3
time.sleep(1)
return x
results = [slow_function(i) for i in range(1, 11)] #4
results_future = [slow_function_ray.remote(i) for i in range(1, 11)] #5
results_ray = ray.get(results_future)
print("Results without Ray: ", results)
print("Results with Ray: ", results_ray)
ray.shutdown()
A Ray example contrasts sequential execution with submitting ten remote tasks and collecting their results. section 3.3.2
