liveBook · Manning

Dimension alignment

As explained in Rearchitecting LLMs

Dimension alignment is the adjustment of tensor or matrix dimensions to values that match hardware processing constraints efficiently.

Dimension alignment adjusts a matrix or tensor dimension to a hardware-friendly multiple, such as 32, 64, 128, or 256. An intermediate size such as 6554 can require GPU padding with zeros. In width pruning, OptiPFair’s expansion_divisor parameter adjusts how many neurons are retained so the resulting intermediate size is divisible by the chosen value. section 5.2.6

Alignment does not guarantee a speedup on every GPU: a T4 may be memory-bandwidth-bound, leaving Tensor Cores idle. It is most effective in compute-bound workloads such as training or high-traffic inference on A100 or H100 GPUs. section 5.2.6

pruned_model, stats = prune_model(
    model=model,
    pruning_type="MLP_GLU",
    neuron_selection_method="PPM", 
    pruning_percentage=20,
    expansion_divisor=64,  #A
    dataloader=dataloaderwiki,
    show_progress=True
)

The pruning call demonstrates setting expansion_divisor to 64 when pruning MLP_GLU neurons. section 5.2.6

How Rearchitecting LLMs covers it

How deep it goes
Explains padding, configures expansion_divisor, and compares hardware bottlenecks without detailing kernel internals.
Examples
Shows an MLP_GLU pruning call with expansion_divisor set to 64 and contrasts T4 with A100/H100 workloads.
What you can do after
Operate OptiPFair pruning by setting expansion_divisor to produce a divisible intermediate dimension.
How it connects
Connects width pruning with GPU padding, Tensor Cores, and memory bandwidth.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.