Optimization improves system performance or resource efficiency by changing models, serving processes, code, or hardware. Start by measuring the whole system rather than assuming the model is the bottleneck. section 15.4.1
Use a practical sequence when improving a serving system:
- Profile end-to-end performance and identify where time or resources are actually spent. section 15.4.1
- Choose the most optimizable component, not automatically the slowest one. section 15.4.1
- Prefer models and system designs that already fit requirements; use quantization, pruning, distillation, or scaling when justified. section 15.4.2
- Match the effort to the situation: straightforward cloud scaling may suit an early product, while low-level optimization may pay off at large scale. section 15.4.2
A misleading latency target
A pipeline takes 200 ms: 120 ms comes from a highly optimized GPU model, while 80 ms comes from arbitrary Python preprocessing. Improving preprocessing is more promising than changing the model. In another pipeline, dozens of simple models spend only about 5% of runtime in inference, so batching fails to help; preprocessing improvements deliver a 40% speedup instead. section 15.4.1
