An already-trained model may use float32 weights even when it needs to run on a memory-limited device or serve predictions with low latency. Quantization lowers numeric precision for inference. Converting weights to int8 can make an inference-only model four times smaller while keeping accuracy near that of the original; inference may also run faster. section 6.3.2 section 18.3.6
Why scaling matters
Suppose you want to multiply two float32 tensors. Casting their values straight to int8 can erase useful detail: values such as 0.1 and 0.9 become zero. Instead, scale each tensor so its values use the available int8 range, round and cast, multiply, then undo both scales on the output. Rounding still introduces error, but the demonstrated result stays close to the float32 multiplication. section 18.3.6
def abs_max_quantize(value): abs_max = ops.max(ops.abs(value), keepdims=True) #1 scale = ops.divide(127, abs_max + 1e-7) #2 scaled_value = value * scale #3 scaled_value = ops.clip(ops.round(scaled_value), -127, 127) #4 scaled_value = ops.cast(scaled_value, dtype="int8") #5 return scaled_value, scale int_x, x_scale = abs_max_quantize(x) int_kernel, kernel_scale = abs_max_quantize(kernel)
The abs-max example computes a scale, rounds and clips values to −127 through 127, then casts them to int8. section 18.3.6
int_y = ops.matmul(int_x, int_kernel) y = ops.cast(int_y, dtype="float32") / (x_scale * kernel_scale)
The quantized tensors enter matrix multiplication; the result is cast back to float32 and divided by the product of their scales. section 18.3.6
>>> y array([[ 0.9843736, 0.3933239], [-1.0151455, -3.1965137]]) >>> ops.matmul(x, kernel) array([[ 0.98 , 0.40999997], [-1. , -3.2 ]])
Compare the quantized output with the float32 output to see the remaining rounding difference. section 18.3.6
In practice, do not hand-code the scaling pipeline for a model. Call Keras’s built-in int8 quantization API on a trained model, then check whether its predictions remain accurate enough for your application. Quantization differs from weight pruning, which removes less-significant coefficients rather than reducing their precision. section 18.3.6 section 6.3.2
model = ... #1 model.quantize("int8") #2 predictions = model.predict(...) #3
The practical Keras path is to call model.quantize("int8") before making predictions. section 18.3.6
