liveBook · Manning

Q8_0

As explained in Quantization and Fast Inference

Q8_0 is a block quantization format that stores 32 weights in 34 bytes using an FP16 scale and 32 signed INT8 values.

Q8_0 stores 32 weights in 34 bytes: one FP16 scale and 32 signed INT8 values. It reconstructs each weight as the scale multiplied by its quantized value, for 1.0625 bytes per weight including the scale. section 10.2.5

Reconstructing weights

A dumped block has scale 0.000358 and first quantized values [8, -40, 23, -27]. Multiplying gives reconstructed values [0.002861, -0.014305, 0.008225, -0.009656]. section 10.2.5

Listing 10.3 Q8_0 block dump for blk.0.ffn_down.weight in llama-2-7b.Q8_0.gguf
─── Q8_0 block layout (34 bytes / 32 weights) ────────────────────────
0x00   d         (FP16 scale)                          dc 0d                          #A
0x02   qs        (32 INT8 weights, signed)             08 d8 17 e5 3c 33 fd 1d ...    #B
 
  Decoded scale d              = 0x0ddc -> +0.000358
  Quantized weights[0..3]      = [8, -40, 23, -27]
  Reconstructed[0..3]          = [+0.002861, -0.014305, +0.008225, -0.009656]          #C
    (formula: w = d * qs[i])
 

Inspect the complete 34-byte block dump, including its decoded scale and reconstructed values. section 10.2.5

In one fixed-prompt test at temperature 0, Q8_0 and F16 produced identical first 64 tokens; Q8_0 used 47% less storage. section 10.3.4

How Quantization and Fast Inference covers it

How deep it goes
Defines the byte layout and reconstruction formula, then walks through a numerical block dump.
Examples
Reconstructs four weights from a byte dump and compares 64 generated tokens against F16.
What it assumes
Assumes familiarity with symmetric INT8 dequantization rather than establishing that concept from zero.
What you can do after
Understand how to decode a Q8_0 block into approximate weights using its scale and signed INT8 values.
How it connects
Connects GGUF weight blocks with symmetric INT8 dequantization and comparisons against F16 and other quantization formats.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.