BitLinear trains a linear layer with quantization active on every forward pass, helping weights adapt to ternary rounding instead of imposing it after training. section 8.5.2
Weights are scaled by their mean absolute value, rounded and clamped to {−1, 0, +1}. Activations are scaled per final-dimension row and clamped to the int8 range. A detach-based straight-through estimator (STE) uses quantized values in the forward pass while routing gradients to full-precision weights and activations. section 8.5.2
Toy regression comparison
On a two-layer MLP, BitLinear QAT reaches test MSE 0.0070, versus 0.0052 for FP32. Post-training ternarization reaches 0.0199, showing the cost of applying the constraint only after training. section 8.5.2
Listing 8.2 Loading Llama-2-7B in NF4 with QLoRA adapters
def weight_quant(w, eps=1e-5):
scale = 1.0 / w.abs().mean().clamp(min=eps)
return (w * scale).round().clamp(-1, 1) / scale #A
def activation_quant(x, eps=1e-5):
scale = 127.0 / x.abs().amax(dim=-1, keepdim=True).clamp(min=eps)
return (x * scale).round().clamp(-128, 127) / scale #B
class BitLinear(nn.Linear):
def forward(self, x):
x_q = x + (activation_quant(x) - x).detach() #C
w = self.weight
w_q = w + (weight_quant(w) - w).detach() #C
return F.linear(x_q, w_q, self.bias)
The listing provides the weight and activation quantization functions and the STE-backed linear forward pass. section 8.5.2
