Profiling measures system performance to reveal bottlenecks before optimization. It applies across the whole serving pipeline, not only to the model. section 15.4.1
Use profiling to make optimization decisions deliberately:
- Measure each stage, including model inference, data preprocessing, and network interactions. section 15.4.1
- Target the most optimizable bottleneck, not automatically the slowest component. section 15.4.1
- Test proposed techniques against measurements rather than assuming batching or model changes will help. section 15.4.1
Choosing preprocessing over model optimization
A 200 ms request spent 120 ms in an already highly optimized GPU model and 80 ms in arbitrary Python preprocessing. Optimizing preprocessing offered more opportunity. In another pipeline, inference consumed only 5% of runtime, so batching failed to help; preprocessing changes reduced latency by 40%. section 15.4.1
