Parallelism divides a problem into smaller tasks that execute simultaneously, improving performance and efficiency. In machine learning, it lets a model process different data parts at the same time, accelerating training. section 16.1
Sequential versus concurrent work
A function that sleeps for 0.5 seconds takes about 5 seconds when called ten times sequentially. Running the same calls through concurrent.futures.ThreadPoolExecutor takes about 0.5 seconds. Threads share memory and are lightweight; processes use separate memory and are typically used for distributed training because their isolation simplifies accelerator coordination. section 16.1
Listing 16.1 Sequential execution timing example
# In[]
def some_work(x):
time.sleep(0.5)
return x
start_time = time.time()
results_sequential = sum([some_work(i) for i in range(10)])
sequential_time = time.time() - start_time
print(results_sequential)
print(f"Sequential Time: {sequential_time:.2f} seconds")
# Out[]
45
Sequential Time: 5.04 seconds
The timing listing makes the speed difference between sequential and multithreaded execution concrete. section 16.1
Distributed parallelism coordinates work across processes and devices through collective communication. Broadcast sends a tensor from one rank to every other rank; all-reduce combines values, such as by summing them, and returns the result to every process. section 16.2
Listing 16.6 PyTorch’s broadcast and all_reduce operations
import torch.distributed as dist
...
def perform_broadcast(rank):
if rank == 0:
tensor = torch.tensor([123.0])
else:
tensor = torch.zeros(1)
dist.broadcast(tensor, src=0) #1
def perform_all_reduce(rank):
tensor = torch.tensor([float(rank)])
dist.all_reduce(tensor, op=dist.ReduceOp.SUM) #2
...
# Out[]
Process 1 performing broadcast
Process 2 performing broadcast
Process 0 performing broadcast
Process 2 has tensor tensor([0.])
Process 1 has tensor tensor([0.])
Process 0 has tensor tensor([123.])
Process 0 received tensor: tensor([123.])
Process 1 received tensor: tensor([123.])
Process 2 received tensor: tensor([123.])
Process 0 performing all_reduce
Process 2 performing all_reduce
Process 1 performing all_reduce
Process 0 has tensor tensor([0.])
Process 1 has tensor tensor([1.])
Process 2 has tensor tensor([2.])
Process 0 received tensor: tensor([3.])
Process 2 received tensor: tensor([3.])
Process 1 received tensor: tensor([3.])
The PyTorch listing demonstrates broadcast and all_reduce on tensors, including rank-specific inputs and shared results. section 16.2
Combining parallelism strategies creates engineering challenges around checkpointing, communication, composition, and resource utilization. TorchTitan provides a native-PyTorch reference for distributed large-language-model training with ready-to-use configurations. section 16.9
