A graphics processing unit (GPU) is a parallel compute device that accelerates the tensor-heavy arithmetic used by neural networks. PyTorch treats it as a large tensor calculator, making it especially useful for compute-intensive training and larger applications. section 1.1 section 1.2.1
Use a GPU effectively by managing computation and data placement:
- Check availability with `torch.cuda.is_available()` and select a CUDA device with `torch.device("cuda")`. section 1.2.1
- Move tensors to the selected device with `.to(device)` before performing operations. section 1.2.1
- Keep related tensors on the same device; mixing CPU and CUDA tensors raises a device-mismatch error. section 1.2.1
- Batch enough work to offset CPU-to-GPU transfer and synchronization overhead, and benchmark small workloads because the CPU can be faster. section 1.2.1
- Copy CUDA tensors back with `.cpu()` before converting them to NumPy. section 1.2.1
Accelerating matrix multiplication
For matrices X with shape n×m and Y with shape m×p, multiplication produces C with shape n×p. A practical timing comparison creates a large tensor, measures `x @ x` on the CPU, moves it with `x = x.to(device)`, and measures the GPU version. Small 128×128 work illustrates why transfer costs can dominate. section 1.2.1
def moveTo(obj, device):
"""
obj: the python object to move to a device, or to move its
contents to a device
device: the compute device to move objects to
"""
if isinstance(obj, list):
return [moveTo(x, device) for x in obj]
elif isinstance(obj, tuple):
return tuple(moveTo(list(obj), device))
elif isinstance(obj, set):
return set(moveTo(list(obj), device))
elif isinstance(obj, dict):
to_ret = dict()
for key, value in obj.items():
to_ret[moveTo(key, device)] = moveTo(value, device)
return to_ret
elif hasattr(obj, "to"):
return obj.to(device)
else:
return obj
some_tensors = [torch.tensor(1), torch.tensor(2)]
print(some_tensors) print(moveTo(some_tensors, device))
[tensor(1), tensor(2)]
[tensor(1, device='cuda:0'), tensor(2, device='cuda:0')]
Use the recursive `moveTo` helper to transfer tensors nested inside lists, tuples, sets, and dictionaries to one compute device. section 1.2.1
