liveBook · Manning

Padding

As explained in Rearchitecting LLMs

Padding is the addition of extra elements to a computational dimension so its size meets a hardware or algorithmic alignment requirement.

GPU matrix-multiplication kernels operate on fixed-size tiles. When a pruned feed-forward width misses a tile boundary, the kernel pads it upward and still allocates, moves, and multiplies the extra columns. Pruning to an aligned width avoids paying for that discarded work. section 12.7.1

A 20% width cut

Cutting a width of 8,192 by 20% leaves 6,553 neurons after truncation. With a 128-column alignment, the kernel serves 6,656 columns, wasting 103; snapping down to 6,528 avoids that padding. section 12.7.1

Listing 12.4 Reporting the padding tax and snapping to an aligned width
def padding_report(target_width, multiple=128): #1
    served = ((target_width + multiple - 1) // multiple) * multiple   
    return served, served - target_width
 
def snap_to_aligned(target_width, multiple=128): #2
    return (target_width // multiple) * multiple                      
 
raw = int(8192 * 0.80)                 
served, waste = padding_report(raw)    
aligned = snap_to_aligned(raw)         
print(f"naive target {raw}: served {served}, wasted columns {waste}")
print(f"aligned target {aligned}: served {aligned}, wasted columns 0")

Use the Python helpers to report a target width’s served width and padding waste, or snap the target down to an aligned boundary. section 12.7.1

How Rearchitecting LLMs covers it

How deep it goes
Explains tile alignment and quantifies its cost with a pruning example and Python helpers.
Examples
Calculates padding for a 6,553-wide target and compares it with snapping to 6,528.
What you can do after
Apply the Python helpers to estimate padding waste or snap a pruned width to an aligned boundary.
How it connects
Connects width pruning to GPU matrix multiplication and tensor-core alignment.

Also explained in

Compare how 22 books explain padding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.