liveBook · Manning

Tensor Core

As explained in Rearchitecting LLMs

A Tensor Core is a specialized NVIDIA GPU execution unit optimized for high-throughput matrix arithmetic.

Tensor Cores are specialized GPU units that perform matrix multiplication on tiles. For FP16, a Tensor Core can process a 16×16 block in one clock cycle. section 5.2.6

Width pruning can better utilize Tensor Cores when the resulting intermediate size is divisible by values such as 32, 64, 128, or 256. Misaligned dimensions may require padding. Alignment helps most in compute-bound workloads; on a T4, memory-bandwidth limits may leave Tensor Cores idle. section 5.2.6

pruned_model, stats = prune_model(
    model=model,
    pruning_type="MLP_GLU",
    neuron_selection_method="PPM", 
    pruning_percentage=20,
    expansion_divisor=64,  #A
    dataloader=dataloaderwiki,
    show_progress=True
)

The pruning call sets expansion_divisor to 64 to align the resulting intermediate size. section 5.2.6

How Rearchitecting LLMs covers it

How deep it goes
Explains tiled matrix multiplication, alignment, padding, and hardware bottlenecks.
Examples
Uses FP16 16×16 tiles, a misaligned size of 6554, and an expansion divisor of 64.
What you can do after
Configure OptiPFair width pruning with an expansion_divisor to align the retained neuron count.
How it connects
Connects Tensor Cores with width pruning, matrix multiplication, and GPU compute and memory bottlenecks.

Also explained in

Compare how 4 books explain Tensor Core →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.