INT4 halves weight storage versus INT8 by placing two 4-bit values in each byte: one in the high nibble and one in the low. Packing shifts signed values from [-8, 7] into [0, 15]; unpacking uses shifts and masks to recover them. section 3.5
Account for these implementation constraints:
- Unpacking adds bit shifts and masks, so memory-bandwidth savings must offset the extra work. section 3.5
- Align INT4 group boundaries to byte boundaries; group sizes divisible by two provide clean alignment. section 3.5
Memory impact
A 4096×4096 projection with INT4 weights and group-128 FP16 scales uses 8.25 MB, versus 32 MB for FP16 weights: 3.88× compression, with 3% scale overhead. section 3.5
Listing 3.28 INT4 packing and memory layout
def pack_int4(values):
"""Pack pairs of INT4 values [-8,7] into bytes."""
values_unsigned = (values.clamp(-8, 7) + 8).to(torch.uint8) #A
paired = values_unsigned.reshape(*values.shape[:-1], -1, 2)
return (paired[..., 0] << 4) | paired[..., 1] #B
def unpack_int4(packed):
"""Unpack bytes into INT4 value pairs."""
high = (packed >> 4) & 0x0F
low = packed & 0x0F
unpacked = torch.stack([high, low], dim=-1).reshape(*packed.shape[:-1], -1)
return unpacked.to(torch.int8) - 8 #C
def compute_memory_layout(out_features, in_features, group_size, bits=4):
num_groups = in_features // group_size
quantized_bytes = (out_features * in_features * bits) // 8 #D
scale_bytes = out_features * num_groups * 2 #E
fp16_bytes = out_features * in_features * 2
total = quantized_bytes + scale_bytes
return {
'compression': fp16_bytes / total,
'overhead_pct': scale_bytes / total * 100
}
See the pack and unpack operations implemented with shifts, masks, and paired values. section 3.5
