In a transformer encoder block, the feed-forward network (FFN) consists of an up-projection and a down-projection. Their larger weight matrices make FFN layers a priority when diagnosing weight-quantization error. section 5.5
BERT weight quantization
On BERT-base, FFN matrices are 768×3072, compared with 768×768 attention projections. Under per-channel weight fake quantization, FFN sublayers produce 16.2× the aggregate output MSE at INT8 and 2.0× at INT4. section 5.5
Figure 5.15 Sublayer sensitivity to per-channel weight fake quantization on BERT-base-uncased. FFN sublayers dominate—16.2× more aggregate output MSE than all four attention projections at INT8, 2.0× at INT4. Dashed line separates attention (below) from FFN (above).

Compare FFN and attention sublayer sensitivity to per-channel weight fake quantization. section 5.5



















