Batch normalization folding absorbs normalization into the preceding layer’s weights and biases. For a convolution, scale each output channel’s weights by γ/√(running variance + ε), and set its bias to (bias − running mean) × scale + β; use zero for a missing convolution bias. Fold before inserting fake-quantization modules: otherwise, quantization-aware training calibrates against unfolded weights, while export uses folded weights with a different distribution and dynamic range. section 5.3.1
Listing 5.9 Batch normalization folding for QAT
def fold_bn_into_conv(conv, bn):
"""Fold batch norm into preceding Conv2d."""
w = conv.weight.data
b = conv.bias.data if conv.bias is not None else torch.zeros(
w.shape[0], device=w.device)
factor = bn.weight / torch.sqrt(bn.running_var + bn.eps) #A
new_weight = w * factor.view(-1, 1, 1, 1)
new_bias = (b - bn.running_mean) * factor + bn.bias
new_conv = nn.Conv2d(conv.in_channels, conv.out_channels,
conv.kernel_size, conv.stride,
conv.padding, bias=True)
new_conv.weight.data = new_weight
new_conv.bias.data = new_bias
return new_convInspect the PyTorch Conv2d folding function, including its handling of missing bias and construction of the folded layer. section 5.3.1
