liveBook · Manning

Throughput

As explained in Rearchitecting LLMs

Throughput is the amount of model work or output that a system completes per unit of time.

Throughput measures how many tokens a model generates per second. It is an indicator of inference efficiency: higher throughput means more output in the same time. Measure it alongside total inference time and time to first token (TTFT), which captures the delay before a response begins. section 4.3.5

Pruning can increase throughput by changing the model’s physical architecture. Depth pruning removes Transformer blocks, reducing rounds of data movement between VRAM and compute. Width pruning reduces matrix dimensions. In the reported comparisons, models with the same amount of pruning had similar speeds even when their selection strategies or calibration datasets differed. section 4.6 section 5.2.6

For a useful comparison:

  • Benchmark a base model and pruned variants under the same generation and hardware conditions; record batch size and output-length limits. section 4.3.5 section 5.2.6
  • Check quality as well as speed. For example, removing 15 attention layers caused benchmark scores to collapse, despite reducing dynamic VRAM. section 8.4.2
  • Treat shared Colab T4 measurements as directional: hardware sharing, memory state, and thermal load can make precise comparisons unreliable. section 8.4.2

Depth-pruning benchmark

On a Colab T4 GPU, with warmup and 10 records from each dataset, the base model generated 17.65 tokens/s. Two data-guided pruned models reached 20.59 and 20.61 tokens/s, gains of 16.7% and 16.8%. Their TTFT and total inference time also improved. section 4.3.5

Table 4.4 Inference performance benchmarks
Performance
Inference Time
TTFT
Throughput
Base model
2.834s
± 0.041s
64.49ms
± 1.0ms
17.65 tokens/s
± 0.26
Data guided
2.429s
± 0.009s
(+14.3%)
56.30ms
± 4.05ms
(+12.7%)
20.59 tokens/s
± 0.08
(+16.7%)
Data guided protected
2.426s
± 0.010s (+14.4%)
54.40ms
± 1.78ms (+15.6%)
20.61 tokens/s
± 0.09 (+16.8%)

Compare throughput with inference time and TTFT for the base and pruned models. section 4.3.5

Width-pruning benchmark

For short generation of at most 50 tokens in batches of four on a Colab T4, the base model produced 18.31 tokens/s on WikiText. The two pruned models reached 21.90 and 21.83 tokens/s, with similar gains on SMS. section 5.2.6

Speed is not the only deployment cost. Energy per generated token can improve even when total energy rises because a model generates more tokens; compare both measures rather than treating throughput as a substitute for energy efficiency. section 5.2.6

How Rearchitecting LLMs covers it

How deep it goes
Defines tokens per second and reports benchmark conditions and results, while deferring evaluation-system details and controlled attention-pruning speed analysis.
Examples
Compares base and pruned models on a Colab T4, including depth-pruning and batched width-pruning benchmarks.
What you can do after
Evaluate model inference performance against a baseline while accounting for quality trade-offs and measurement variability.
How it connects
Connects throughput to depth pruning, width pruning, memory bandwidth, inference latency, and energy efficiency.

Also explained in

Compare how 32 books explain throughput →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.