liveBook · Manning

BitLinear

As explained in Quantization and Fast Inference

BitLinear is the example's linear-layer implementation that quantizes weights and activations during training while preserving gradient flow.

BitLinear trains a linear layer with quantization active on every forward pass, helping weights adapt to ternary rounding instead of imposing it after training. section 8.5.2

Weights are scaled by their mean absolute value, rounded and clamped to {−1, 0, +1}. Activations are scaled per final-dimension row and clamped to the int8 range. A detach-based straight-through estimator (STE) uses quantized values in the forward pass while routing gradients to full-precision weights and activations. section 8.5.2

Toy regression comparison

On a two-layer MLP, BitLinear QAT reaches test MSE 0.0070, versus 0.0052 for FP32. Post-training ternarization reaches 0.0199, showing the cost of applying the constraint only after training. section 8.5.2

Listing 8.2 Loading Llama-2-7B in NF4 with QLoRA adapters
def weight_quant(w, eps=1e-5):
    scale = 1.0 / w.abs().mean().clamp(min=eps)
    return (w * scale).round().clamp(-1, 1) / scale             #A
 
def activation_quant(x, eps=1e-5):
    scale = 127.0 / x.abs().amax(dim=-1, keepdim=True).clamp(min=eps)
    return (x * scale).round().clamp(-128, 127) / scale         #B
 
class BitLinear(nn.Linear):
    def forward(self, x):
        x_q = x + (activation_quant(x) - x).detach()            #C
        w = self.weight
        w_q = w + (weight_quant(w) - w).detach()                 #C
        return F.linear(x_q, w_q, self.bias)
 

The listing provides the weight and activation quantization functions and the STE-backed linear forward pass. section 8.5.2

How Quantization and Fast Inference covers it

How deep it goes
Implements scaled weight and activation quantization plus STE gradient flow, and verifies gradients and toy-task behavior.
Examples
Compares QAT and post-training ternarization on a toy MLP and cites BitNet b1.58 2B4T as a trained ternary model.
What you can do after
Build it from scratch by quantizing weights and activations in the forward pass and using a detach-based STE to preserve gradients.
How it connects
Connects linear layers to quantization-aware training, post-training quantization, and the straight-through estimator.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.