Latency is the time between sending a request and receiving a response, including network and processing delays. In real-time applications, that wait can affect whether a deployment is practical, even when its model quality is acceptable. section 7.3
A locally hosted distilled student can reduce latency by avoiding external API round-trips. Illustrative API network delays are 200–500 ms per request, while a local student may respond in 50–200 ms on modest hardware. These figures vary with provider and region; they are not a like-for-like measurement of total API and local response times. section 7.3
Assess latency alongside the other deployment tradeoffs:
- A distilled student requires upfront teacher-data generation and training, even if it reduces ongoing API costs. section 7.3
- Local deployment can improve data control and operational independence, but the business case must also account for build and maintenance effort. section 7.3 section 1.3
- Avoid adaptation when the existing model is already good enough or when the project lacks a baseline or sufficient high-quality data. section 1.11











