liveBook · Manning

Latency

As explained in LLM Customization and Fine-Tuning

The time between sending a request and receiving a response, including network and processing delays.

Latency is the time between sending a request and receiving a response, including network and processing delays. In real-time applications, that wait can affect whether a deployment is practical, even when its model quality is acceptable. section 7.3

A locally hosted distilled student can reduce latency by avoiding external API round-trips. Illustrative API network delays are 200–500 ms per request, while a local student may respond in 50–200 ms on modest hardware. These figures vary with provider and region; they are not a like-for-like measurement of total API and local response times. section 7.3

Assess latency alongside the other deployment tradeoffs:

  • A distilled student requires upfront teacher-data generation and training, even if it reduces ongoing API costs. section 7.3
  • Local deployment can improve data control and operational independence, but the business case must also account for build and maintenance effort. section 7.3 section 1.3
  • Avoid adaptation when the existing model is already good enough or when the project lacks a baseline or sufficient high-quality data. section 1.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Compares illustrative API network delays with local student response times without specifying a measurement procedure.
Examples
Uses customer support chat, interactive code assistants, and clinical decision support as latency-sensitive applications.
What you can do after
Evaluate whether avoiding API round-trips could justify a locally deployed student for a real-time application.
How it connects
Connects latency reduction to knowledge distillation, self-hosting, data sovereignty, and operational independence.

Also explained in

Compare how 12 books explain latency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.