liveBook · Manning

Inference performance

As explained in Rearchitecting LLMs

Inference speed is the rate at which a trained model produces outputs from inputs.

Inference performance describes how quickly a trained model produces outputs and, more broadly, the resources it uses to do so. Common measures include latency and generated tokens per second. Evaluate efficiency alongside capability: a faster model may also be less accurate. section 2.1 section 5.2.6

Establish a baseline before modifying the model. One timing workflow warms up the GPU, runs several test prompts, and reports the mean and standard deviation of response times. Repeat comparisons under consistent conditions, and treat small differences cautiously: shared GPU workload can cause variation even between runs in the same environment. section 2.2 section 5.2.6

Depth pruning can improve speed by removing complete model blocks. Fewer blocks mean fewer operations and memory accesses, as well as fewer attention layers contributing to the key-value cache during generation. The capability cost depends on which blocks are removed, so measure both efficiency and task performance after pruning. section 4.1.2

Width-pruning comparison

On a Colab T4 GPU, with batches of four and a maximum of 50 generated tokens, the base model produced 18.31 tokens per second on WikiText and 18.26 on SMS. Two width-pruned variants reached about 21.8–21.9 tokens per second, gains of roughly 19–20%. Their calibration datasets differed, but their resulting matrix dimensions were the same. section 5.2.6

Table 5.2 Inference performance
Model
Tokens / sec
(WikiText)
Tokens / sec
(SMS)
Gains
Base
18.31
18.26
 
Wiki model
21.90
21.93
+19.6% / +20.1%
SMS model
21.83
21.89
+19.2% / +19.9%

Compare base and pruned-model throughput across both prompt datasets. section 5.2.6

Efficiency work can also reduce input requirements rather than change model structure. In a specialization test, QLoRA and QDoRA replaced a fifteen-line prompt with “Extract:” while achieving full schema compliance on the test cases. General benchmark scores nevertheless declined; this result demonstrates prompt simplification, not a measured speed gain. section 7.3.5

How Rearchitecting LLMs covers it

How deep it goes
Describes timing and throughput comparisons, explains pruning mechanisms, and defers the inference evaluation system.
Examples
Compares baseline and width-pruned model throughput on WikiText and SMS prompts.
What you can do after
Evaluate inference efficiency against a baseline while accounting for capability loss and measurement variability.
How it connects
Connects inference performance to depth pruning, width pruning, key-value caching, and knowledge distillation.

Also explained in

Compare how 2 books explain inference performance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.