MSE-optimal range estimation selects a quantization boundary that minimizes point-wise error between original activations and their reconstructed values. It trades some clipping for finer precision across the values that remain in range. section 4.3
During calibration:
- Collect activation statistics in a histogram and search candidate range boundaries. section 4.3
- Choose the candidate with the lowest reconstruction MSE; the search has moderate compute cost. section 4.3
Calibration comparison
One calibration result selected a range of 10.5917, with MSE 0.000600 and 0.0236% clipped values. Absmax had no clipping but a higher MSE of 0.000688. section 4.3
Listing 4.6 MSE-optimal observer with histogram acceleration
class MSEOptimalObserver:
def __init__(self, num_bins=2048, num_candidates=200):
self.num_bins = num_bins
self.num_candidates = num_candidates
self.histogram = None
self.max_val = 0.0
def observe(self, tensor):
abs_vals = tensor.abs().float()
curr_max = abs_vals.max().item()
if curr_max > self.max_val:
self.max_val = curr_max
hist = torch.histc(abs_vals.flatten(), bins=self.num_bins,
min=0, max=self.max_val) #A
if self.histogram is None:
self.histogram = hist.cpu()
else:
self.histogram += hist.cpu()
def compute_range(self):
candidates = torch.linspace(0.5 * self.max_val, self.max_val,
self.num_candidates)
bin_width = self.max_val / self.num_bins
bin_centers = torch.arange(self.num_bins) * bin_width + bin_width / 2
best_mse, best_range = float('inf'), self.max_val
for r in candidates:
scale = r / 127
clipped = torch.clamp(bin_centers, 0, r.item())
quantized = torch.round(clipped / scale) * scale
squared_error = (bin_centers - quantized) ** 2
mse = (squared_error * self.histogram).sum() / self.histogram.sum() #B
if mse < best_mse:
best_mse, best_range = mse, r.item()
return best_rangeInspect the histogram-accelerated observer and its candidate range setup. section 4.3
