Broadcasting lets PyTorch perform element-wise operations on tensors whose shapes differ. Instead of manually making every tensor the same shape, you can use a scalar or a size-one dimension across a larger tensor. This is useful when one value must affect every entry, or when a short set of weights must be applied across an image. section 3.4 section 3.5
To check whether shapes work together, align them on the right and compare dimensions from right to left. Each pair must either be equal or have a size of 1 on one side. Missing leading dimensions act like size-one dimensions. Think of values along a size-one dimension as *virtually repeated* to match the other tensor. The result takes the larger size at each position. If neither size in a pair is 1 and they differ, the shapes are incompatible. section 3.4 section 3.5
A scalar scales and offsets a vector
With a vector `t_u` and scalar tensors `w` and `b`, `w * t_u + b` multiplies every vector element by `w`, then adds `b` to every result. The same expression defines a simple model without writing an element-by-element loop. section 5.3.1
# In[7]:
x = torch.ones(())
y = torch.ones(3,1)
z = torch.ones(1,3)
a = torch.ones(2, 1, 1)
print(f"shapes: x: {x.shape}, y: {y.shape}")
print(f" z: {z.shape}, a: {a.shape}")
print("x * y:", (x * y).shape)
print("y * z:", (y * z).shape)
print("y * z * a:", (y * z * a).shape)
# Out[7]:
shapes: x: torch.Size([]), y: torch.Size([3, 1])
z: torch.Size([1, 3]), a: torch.Size([2, 1, 1])
x * y: torch.Size([3, 1])
y * z: torch.Size([3, 3])
y * z * a: torch.Size([2, 3, 3])
Try the shape checks: a scalar times a `(3, 1)` tensor stays `(3, 1)`; `(3, 1)` times `(1, 3)` becomes `(3, 3)`; adding a `(2, 1, 1)` shape extends the result to `(2, 3, 3)`. section 5.3.1
Apply channel weights across images
An image batch can have shape `(2, 3, 5, 5)`: batch, channels, rows, columns. Give its three channel weights shape `(3, 1, 1)`, then multiply. Broadcasting applies each weight across the corresponding channel’s rows and columns—and across both images—while keeping the batch shape `(2, 3, 5, 5)`. Summing over channels then produces weighted grayscale images. section 3.5
Shape matching can be hard to follow through a longer calculation. Named tensors offer a way to label dimensions; aligning channel weights to an image makes the intended axes explicit. Broadcasting also matters when calculating gradients: summing the per-input contributions to a scalar parameter reverses the way that parameter was applied across the input vector. section 3.5 section 5.4.2
