Token-wise variation means magnitudes differ across token positions. For value-cache quantization, this motivates a separate scale for each token rather than each channel. section 3.4
Value-cache comparison
Per-token quantization wins four of six tests across layers 0, 11, and 21 at INT8 and INT4. At layers 11 and 21, it lowers MSE by 1.2× and 1.4×; layer 0 ties at INT8 and slightly favors per-channel at INT4. section 3.4
Table 3.6 Values granularity comparison (per-channel vs per-token)
|
Layer
|
Bits
|
Per-Channel MSE
|
Per-Token MSE
|
Winner
|
|
0
|
INT8
|
Tie
|
Tie
|
Per-channel (1.0×)
|
|
11
|
INT8
|
Higher
|
Lower
|
Per-token (1.2×)
|
|
21
|
INT8
|
Higher
|
Lower
|
Per-token (1.4×)
|
|
0
|
INT4
|
Lower
|
Higher
|
Per-channel (1.1×)
|
|
11
|
INT4
|
Higher
|
Lower
|
Per-token (1.2×)
|
|
21
|
INT4
|
Higher
|
Lower
|
Per-token (1.4×)
|
Compare per-channel and per-token quantization results for values across layers and bit widths. section 3.4
A practical asymmetric configuration uses per-channel scales for keys and per-token scales for values. section 3.4 section 3.4
