Inference performance describes how quickly a trained model produces outputs and, more broadly, the resources it uses to do so. Common measures include latency and generated tokens per second. Evaluate efficiency alongside capability: a faster model may also be less accurate. section 2.1 section 5.2.6
Establish a baseline before modifying the model. One timing workflow warms up the GPU, runs several test prompts, and reports the mean and standard deviation of response times. Repeat comparisons under consistent conditions, and treat small differences cautiously: shared GPU workload can cause variation even between runs in the same environment. section 2.2 section 5.2.6
Depth pruning can improve speed by removing complete model blocks. Fewer blocks mean fewer operations and memory accesses, as well as fewer attention layers contributing to the key-value cache during generation. The capability cost depends on which blocks are removed, so measure both efficiency and task performance after pruning. section 4.1.2
Width-pruning comparison
On a Colab T4 GPU, with batches of four and a maximum of 50 generated tokens, the base model produced 18.31 tokens per second on WikiText and 18.26 on SMS. Two width-pruned variants reached about 21.8–21.9 tokens per second, gains of roughly 19–20%. Their calibration datasets differed, but their resulting matrix dimensions were the same. section 5.2.6
Table 5.2 Inference performance
|
Model
|
Tokens / sec
(WikiText)
|
Tokens / sec
(SMS)
|
Gains
|
|
Base
|
18.31
|
18.26
|
|
|
Wiki model
|
21.90
|
21.93
|
+19.6% / +20.1%
|
|
SMS model
|
21.83
|
21.89
|
+19.2% / +19.9%
|
Compare base and pruned-model throughput across both prompt datasets. section 5.2.6
Efficiency work can also reduce input requirements rather than change model structure. In a specialization test, QLoRA and QDoRA replaced a fifteen-line prompt with “Extract:” while achieving full schema compliance on the test cases. General benchmark scores nevertheless declined; this result demonstrates prompt simplification, not a measured speed gain. section 7.3.5

