Throughput measures how many tokens a model generates per second. It is an indicator of inference efficiency: higher throughput means more output in the same time. Measure it alongside total inference time and time to first token (TTFT), which captures the delay before a response begins. section 4.3.5
Pruning can increase throughput by changing the model’s physical architecture. Depth pruning removes Transformer blocks, reducing rounds of data movement between VRAM and compute. Width pruning reduces matrix dimensions. In the reported comparisons, models with the same amount of pruning had similar speeds even when their selection strategies or calibration datasets differed. section 4.6 section 5.2.6
For a useful comparison:
- Benchmark a base model and pruned variants under the same generation and hardware conditions; record batch size and output-length limits. section 4.3.5 section 5.2.6
- Check quality as well as speed. For example, removing 15 attention layers caused benchmark scores to collapse, despite reducing dynamic VRAM. section 8.4.2
- Treat shared Colab T4 measurements as directional: hardware sharing, memory state, and thermal load can make precise comparisons unreliable. section 8.4.2
Depth-pruning benchmark
On a Colab T4 GPU, with warmup and 10 records from each dataset, the base model generated 17.65 tokens/s. Two data-guided pruned models reached 20.59 and 20.61 tokens/s, gains of 16.7% and 16.8%. Their TTFT and total inference time also improved. section 4.3.5
Table 4.4 Inference performance benchmarks
|
Performance
|
Inference Time
|
TTFT
|
Throughput
|
|
Base model
|
2.834s
± 0.041s
|
64.49ms
± 1.0ms
|
17.65 tokens/s
± 0.26
|
|
Data guided
|
2.429s
± 0.009s
(+14.3%)
|
56.30ms
± 4.05ms
(+12.7%)
|
20.59 tokens/s
± 0.08
(+16.7%)
|
|
Data guided protected
|
2.426s
± 0.010s (+14.4%)
|
54.40ms
± 1.78ms (+15.6%)
|
20.61 tokens/s
± 0.09 (+16.8%)
|
Compare throughput with inference time and TTFT for the base and pruned models. section 4.3.5
Width-pruning benchmark
For short generation of at most 50 tokens in batches of four on a Colab T4, the base model produced 18.31 tokens/s on WikiText. The two pruned models reached 21.90 and 21.83 tokens/s, with similar gains on SMS. section 5.2.6
Speed is not the only deployment cost. Energy per generated token can improve even when total energy rises because a model generates more tokens; compare both measures rather than treating throughput as a substitute for energy efficiency. section 5.2.6































