liveBook · Manning

Activation

As explained in Quantization and Fast Inference

A numerical output produced by a neural-network layer or operation and passed to subsequent computation.

An activation is an intermediate tensor produced by a neural-network operation and passed to subsequent computation. Unlike weights, activations are recomputed for every input, so their ranges can shift; transformer hidden states can also develop extreme outliers. These properties make activation quantization a separate precision decision from weight quantization. section 3.0

Affine quantization adapts the integer grid to an activation tensor’s observed range. Calibration includes zero in that range, then calculates a scale and integer zero-point. Quantization divides each value by the scale, rounds, adds the zero-point, and clamps to the integer range. This can use more of the grid than symmetric quantization when activations are predominantly positive. section 2.5

Listing 2.8 AffineQuantizer for activations
class AffineQuantizer:
    def __init__(self, bits=8):
        self.q_min = 0
        self.q_max = (1 << bits) - 1                              #A
        self.scale = 1.0
        self.zero_point = 0
 
    def calibrate(self, tensor):
        r_min = tensor.min().item()
        r_max = tensor.max().item()
        r_min = min(r_min, 0.0)                                    #B
        r_max = max(r_max, 0.0)                                    #B
 
        real_range = r_max - r_min
        int_range = self.q_max - self.q_min
 
        if real_range == 0:
            self.scale, self.zero_point = 1.0, 0
        else:
            self.scale = real_range / int_range
            initial_z = self.q_min - (r_min / self.scale)
            self.zero_point = int(round(initial_z))                #C
            self.zero_point = max(self.q_min, min(self.q_max, self.zero_point))
 
    def quantize(self, tensor):
        scaled = (tensor / self.scale) + self.zero_point
        rounded = torch.round(scaled)
        return torch.clamp(rounded, self.q_min, self.q_max).to(torch.uint8)
 
    def dequantize(self, q_tensor):
        return self.scale * (q_tensor.float() - self.zero_point)  #D
 

Inspect calibration of an 8-bit affine activation quantizer, including the zero-range fallback and zero-point calculation. section 2.5

ReLU activations

For one ReLU activation distribution, symmetric quantization uses codes 0–127 with scale 0.139; asymmetric quantization uses 0–255 with scale 0.069. The latter has approximately 2.01 times finer resolution in this comparison. section 2.5

In a linear layer with symmetric weights and affine activations, the activation zero-point must be corrected in the matrix-product arithmetic. Omitting the correction can introduce substantial error when the zero-point is nonzero. The demonstration multiplies quantized values after converting them to floating point, so it illustrates the arithmetic rather than establishing integer-kernel performance. section 2.5

Shifted activation inputs

With an activation zero-point of 48, the corrected output has mean absolute error 0.0302 against FP32; dropping the correction raises it to 1.4174, or 46.9 times as much. section 2.5

Choose quantization granularity with activation statistics, acceptable accuracy loss, and target hardware in mind. A single scale per tensor is simple but crude; finer schemes trade additional granularity against implementation and kernel constraints. section 3.0

How Quantization and Fast Inference covers it

How deep it goes
Works through affine calibration, quantization, linear-layer zero-point correction, and output error; introduces granularity choices without resolving them.
Examples
Compares ReLU activation resolution and tests shifted inputs with and without zero-point correction.
What it assumes
Assumes familiarity with scale and zero-point equations and distinguishes granular from overload error before applying them to tensors.
What you can do after
Evaluate activation quantization using tensor ranges, integer-grid resolution, and output error against an FP32 baseline.
How it connects
Connects activations to affine quantization, symmetric quantization, ReLU, zero-point correction, and quantization granularity.

Also explained in

Compare how 6 books explain activation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.