Appendix D. Inference performance evaluation
At first glance, measuring how long a model takes to generate text is simple: you only need to time a call to generate. But many variables affect inference performance, and not every application cares about the same thing. Some need low latency; others, a high token generation speed. Some answer short prompts, while others receive prompts of thousands of tokens that must be processed quickly.
The environment matters too. The precision you load the model with, or the state of the GPU at that moment, changes the results. Even two runs of the same model on the same GPU can differ, so you should report that variation alongside your measurements.
This appendix explains how to measure inference so you can interpret the results. We start with what each metric measures and which part of the model’s work it reflects. Then we design a fair measurement, including the particularities of running it in Google Colab. We finish by building, step by step, simplified functions that measure the same quantities as measure_detailed_performance, which you’ve used in earlier chapters. With them, we compare two of the models from the final capstone.