liveBook · Manning

Token-wise structure

As explained in Quantization and Fast Inference

Token-wise variation is a pattern in which numerical magnitudes differ across token positions.

Token-wise variation means magnitudes differ across token positions. For value-cache quantization, this motivates a separate scale for each token rather than each channel. section 3.4

Value-cache comparison

Per-token quantization wins four of six tests across layers 0, 11, and 21 at INT8 and INT4. At layers 11 and 21, it lowers MSE by 1.2× and 1.4×; layer 0 ties at INT8 and slightly favors per-channel at INT4. section 3.4

Table 3.6 Values granularity comparison (per-channel vs per-token)
Layer
Bits
Per-Channel MSE
Per-Token MSE
Winner
0
INT8
Tie
Tie
Per-channel (1.0×)
11
INT8
Higher
Lower
Per-token (1.2×)
21
INT8
Higher
Lower
Per-token (1.4×)
0
INT4
Lower
Higher
Per-channel (1.1×)
11
INT4
Higher
Lower
Per-token (1.2×)
21
INT4
Higher
Lower
Per-token (1.4×)

Compare per-channel and per-token quantization results for values across layers and bit widths. section 3.4

A practical asymmetric configuration uses per-channel scales for keys and per-token scales for values. section 3.4 section 3.4

How Quantization and Fast Inference covers it

How deep it goes
Compares per-token and per-channel value quantization by MSE across three layers at INT8 and INT4.
Examples
Uses value-cache comparisons at layers 0, 11, and 21, where per-token quantization wins four of six tests.
What you can do after
Apply asymmetric KV-cache quantization by scaling values per token and keys per channel.
How it connects
Links token-wise variation to KV-cache quantization, per-token scaling, per-channel scaling, and mean squared error.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.