liveBook · Manning

MSE-optimal range estimation

As explained in Quantization and Fast Inference

A quantization range-estimation method that chooses boundaries to minimize mean squared error between original and reconstructed values.

MSE-optimal range estimation selects a quantization boundary that minimizes point-wise error between original activations and their reconstructed values. It trades some clipping for finer precision across the values that remain in range. section 4.3

During calibration:

  • Collect activation statistics in a histogram and search candidate range boundaries. section 4.3
  • Choose the candidate with the lowest reconstruction MSE; the search has moderate compute cost. section 4.3

Calibration comparison

One calibration result selected a range of 10.5917, with MSE 0.000600 and 0.0236% clipped values. Absmax had no clipping but a higher MSE of 0.000688. section 4.3

Listing 4.6 MSE-optimal observer with histogram acceleration
class MSEOptimalObserver:
    def __init__(self, num_bins=2048, num_candidates=200):
        self.num_bins = num_bins
        self.num_candidates = num_candidates
        self.histogram = None
        self.max_val = 0.0
    
    def observe(self, tensor):
        abs_vals = tensor.abs().float()
        curr_max = abs_vals.max().item()
        if curr_max > self.max_val:
            self.max_val = curr_max
        
        hist = torch.histc(abs_vals.flatten(), bins=self.num_bins,
                          min=0, max=self.max_val)                #A
        if self.histogram is None:
            self.histogram = hist.cpu()
        else:
            self.histogram += hist.cpu()
    
    def compute_range(self):
        candidates = torch.linspace(0.5 * self.max_val, self.max_val,
                                   self.num_candidates)
        bin_width = self.max_val / self.num_bins
        bin_centers = torch.arange(self.num_bins) * bin_width + bin_width / 2
        
        best_mse, best_range = float('inf'), self.max_val
        
        for r in candidates:
            scale = r / 127
            clipped = torch.clamp(bin_centers, 0, r.item())
            quantized = torch.round(clipped / scale) * scale
            squared_error = (bin_centers - quantized) ** 2
            mse = (squared_error * self.histogram).sum() / self.histogram.sum()  #B
            
            if mse < best_mse:
                best_mse, best_range = mse, r.item()
        
        return best_range

Inspect the histogram-accelerated observer and its candidate range setup. section 4.3

How Quantization and Fast Inference covers it

How deep it goes
Defines a point-wise MSE objective and histogram-accelerated candidate search, with moderate grid-search cost.
Examples
Compares MSE and absmax calibration results using range, clipping, MSE, and signal-to-noise ratio.
What it assumes
Assumes post-training calibration data and activation observations; representative calibration samples support production accuracy.
What you can do after
Evaluate activation-range candidates by reconstruction MSE and compare clipping and error against other estimators.
How it connects
Connects activation calibration to absolute-maximum ranges, percentile clipping, KL-divergence, and post-training quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.