Matrix-matrix multiplication, also called matmul or tensor product, combines rows from one matrix with columns from another. In neural networks, it transforms an input matrix into a new representation, often before adding a bias and applying an activation. section 2.3 section 2.3.3
A dense-layer transformation
A typical transformation is written as output = relu(matmul(input, W) + b). The matmul combines the input with learned weights W; the bias b is added afterward, and relu replaces negative values with zero. In PyTorch, the same computation is written as torch.matmul(inputs, W) + b. section 2.3 section 3.4.2
Use these rules when checking a matrix product:
- Require x.shape[1] == y.shape[0]; the width of the first matrix must match the height of the second. section 2.3.3
- Produce a result shaped (x.shape[0], y.shape[1]); each result entry is the vector product of one row and one column. section 2.3.3
- Do not swap operands casually: once a tensor has more than one dimension, matmul(x, y) generally differs from matmul(y, x). section 2.3.3
- Extend the same shape logic to higher-rank tensors, where trailing dimensions determine compatibility and the remaining dimensions carry through. section 2.3.3
Figure 2.5 Matrix product box diagram
Use the matrix product box diagram to visualize why the first matrix's width must match the second matrix's height. section 2.3.3
