liveBook · Manning

Sub-scale

As explained in Quantization and Fast Inference

A sub-scale is a quantization scale applied to a smaller chunk of weights within a hierarchical quantization scheme.

A sub-scale adjusts a smaller chunk of weights within hierarchical quantization. In Q6_K, each INT8 sub-scale covers 16 weights and combines with a shared FP16 super-scale and each weight’s signed code. section 10.2.5

Reconstructing a Q6_K weight

For weight 0, the packed bits produce code 29, which becomes −3 after subtracting 32. With sub-scale 94 and super-scale −1.14 × 10⁻⁵, the reconstructed weight is approximately +3.21 × 10⁻³. section 10.2.5

Listing 10.4 Q6_K superblock dump for blk.0.ffn_down.weight in llama-2-7b.Q4_K_M.gguf
─── Q6_K superblock layout (210 bytes / 256 weights) ─────────────────
0x00   ql        (128 B: low 4 bits of 256 weights)    dd fe e8 99 ...               #A
0x80   qh        (64 B: high 2 bits of 256 weights)    e9 aa 79 ae ...               #B
0xc0   scales    (16 B: 16 INT8 sub-scales)            5e 80 64 a0 ...
0xd0   d         (FP16 super-scale)                    bf 80                         #C
 
  Decoded super-scale d        = 0x80bf -> approx -1.14e-5
  scales[0]                    = 94
  Reconstructed weight[0]      = +0.003210
    (formula: w = d * scales[i // 16] * ((low4 | (high2 << 4)) - 32))
 

Inspect a Q6_K superblock’s packed weight bits, sub-scales, super-scale, and reconstructed weight. section 10.2.5

How Quantization and Fast Inference covers it

How deep it goes
Works through Q6_K bit packing and numerical reconstruction, without detailing other k-quant layouts.
Examples
Reconstructs Q6_K weight 0 from packed bits, sub-scale 94, and a super-scale.
What you can do after
Understand Q6_K weight reconstruction using a chunk sub-scale and shared super-scale.
How it connects
Connects sub-scales to quantization, GGUF, and k-quant formats.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.