Matrix-matrix multiplication combines rows from one matrix with columns from another, producing a new matrix. In transformer attention, two large products dominate the computation: the query-key product QKᵀ and the attention-weights-times-value product. Repeating these products for every generated token creates the quadratic inference bottleneck in sequence length. section 2.3.1
The same operation also forms the computational core of neural-network linear layers and their gradients:
- Forward propagation computes output from weights and input, written conceptually as y = W × x. section 5.6.1
- Input and weight values can be converted to FP8 for fast multiplication, while outputs are accumulated with higher precision. section 5.6.1
- The input gradient uses dL/dx = (dL/dz) × Wᵀ, while the weight gradient uses dL/dW = xᵀ × (dL/dz). section 5.6.1
- Master weights remain in FP32 or BF16, and the weight update is performed in FP32 to preserve training stability. section 5.6.1
Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Trace how a linear layer converts inputs and weights to FP8 for multiplication while retaining high-precision weights and gradients. section 5.6.1












