liveBook · Manning

Sub-byte packing

As explained in Quantization and Fast Inference

Sub-byte packing combines multiple values narrower than one byte into byte-aligned storage units.

INT4 halves weight storage versus INT8 by placing two 4-bit values in each byte: one in the high nibble and one in the low. Packing shifts signed values from [-8, 7] into [0, 15]; unpacking uses shifts and masks to recover them. section 3.5

Account for these implementation constraints:

  • Unpacking adds bit shifts and masks, so memory-bandwidth savings must offset the extra work. section 3.5
  • Align INT4 group boundaries to byte boundaries; group sizes divisible by two provide clean alignment. section 3.5

Memory impact

A 4096×4096 projection with INT4 weights and group-128 FP16 scales uses 8.25 MB, versus 32 MB for FP16 weights: 3.88× compression, with 3% scale overhead. section 3.5

Listing 3.28 INT4 packing and memory layout
def pack_int4(values):
    """Pack pairs of INT4 values [-8,7] into bytes."""
    values_unsigned = (values.clamp(-8, 7) + 8).to(torch.uint8)   #A
    paired = values_unsigned.reshape(*values.shape[:-1], -1, 2)
    return (paired[..., 0] << 4) | paired[..., 1]                 #B
 
def unpack_int4(packed):
    """Unpack bytes into INT4 value pairs."""
    high = (packed >> 4) & 0x0F
    low = packed & 0x0F
    unpacked = torch.stack([high, low], dim=-1).reshape(*packed.shape[:-1], -1)
    return unpacked.to(torch.int8) - 8                            #C
 
def compute_memory_layout(out_features, in_features, group_size, bits=4):
    num_groups = in_features // group_size
    quantized_bytes = (out_features * in_features * bits) // 8    #D
    scale_bytes = out_features * num_groups * 2                   #E
    fp16_bytes = out_features * in_features * 2
    total = quantized_bytes + scale_bytes
    return {
        'compression': fp16_bytes / total,
        'overhead_pct': scale_bytes / total * 100
    }
 

See the pack and unpack operations implemented with shifts, masks, and paired values. section 3.5

How Quantization and Fast Inference covers it

How deep it goes
Details signed-value conversion, nibble pairing, unpacking operations, and byte-alignment constraints.
Examples
Quantifies storage for a 4096×4096 projection using INT4 weights and group-128 FP16 scales.
What it assumes
Assumes byte-oriented CPU or GPU storage and defines nibbles and signed INT4 values.
What you can do after
Configure INT4 weights with byte-aligned groups and weigh unpacking cost against memory savings.
How it connects
Connects INT4 quantization and group quantization to memory layout and bandwidth tradeoffs.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.