liveBook · Manning

Q6_K

As explained in Quantization and Fast Inference

Q6_K is a k-quant format that stores 256 weights per superblock using 6-bit codes and hierarchical scales.

Q6_K stores 256 weights in a 210-byte superblock. It splits each weight’s 6-bit code between low-nibble and high-pair byte regions, then applies a 16-weight sub-scale and a super-scale to reconstruct the weight. section 10.2.5

To reconstruct a weight:

  • Extract its low 4 bits and high 2 bits, combine them into a 6-bit code, then subtract 32 to obtain a signed value. section 10.2.5
  • Multiply the signed value by its 16-weight chunk’s INT8 sub-scale and the FP16 super-scale. section 10.2.5

Reconstructing weight 0

The example combines low bits 1101 with high bits 01 to get 29; subtracting 32 gives −3. With sub-scale 94 and super-scale about −1.14 × 10⁻⁵, the reconstructed weight is about +3.21 × 10⁻³. section 10.2.5

How Quantization and Fast Inference covers it

How deep it goes
Works through the 210-byte layout and weight reconstruction, including bit extraction and hierarchical scaling.
Examples
Reconstructs weight 0 from sample bytes, yielding approximately +3.21 × 10⁻³.
What it assumes
Assumes familiarity with GGUF tensor blocks and the hierarchical k-quant scale structure.
What you can do after
Apply the script’s --mode tensor option to decode raw GGUF block bytes.
How it connects
Connects Q6_K to k-quant formats, GGUF, and Q4_K_M tensor-format recipes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.