Q8_0 stores 32 weights in 34 bytes: one FP16 scale and 32 signed INT8 values. It reconstructs each weight as the scale multiplied by its quantized value, for 1.0625 bytes per weight including the scale. section 10.2.5
Reconstructing weights
A dumped block has scale 0.000358 and first quantized values [8, -40, 23, -27]. Multiplying gives reconstructed values [0.002861, -0.014305, 0.008225, -0.009656]. section 10.2.5
Listing 10.3 Q8_0 block dump for blk.0.ffn_down.weight in llama-2-7b.Q8_0.gguf
─── Q8_0 block layout (34 bytes / 32 weights) ────────────────────────
0x00 d (FP16 scale) dc 0d #A
0x02 qs (32 INT8 weights, signed) 08 d8 17 e5 3c 33 fd 1d ... #B
Decoded scale d = 0x0ddc -> +0.000358
Quantized weights[0..3] = [8, -40, 23, -27]
Reconstructed[0..3] = [+0.002861, -0.014305, +0.008225, -0.009656] #C
(formula: w = d * qs[i])
Inspect the complete 34-byte block dump, including its decoded scale and reconstructed values. section 10.2.5
In one fixed-prompt test at temperature 0, Q8_0 and F16 produced identical first 64 tokens; Q8_0 used 47% less storage. section 10.3.4
