liveBook · Manning

Latency

As explained in Quantization and Fast Inference

The elapsed time required for a model or system to produce a result.

Latency is the elapsed time required to produce a result. For a user-facing target, measure the end-to-end path, not just model inference: decoding, preprocessing and postprocessing can cost more than the forward pass they surround. Inference-only latency remains useful for comparing runtimes and diagnosing bottlenecks. section 11.5 section 12.0

Instrument each pipeline stage and record the end-to-end distribution. Use the stage breakdown to identify where optimization will matter, and report end-to-end p95 for a user-facing service-level agreement. Stage means and end-to-end percentiles answer different questions and should not be treated as interchangeable. section 11.6 section 12.2

A vision pipeline on M3

After 20 warm-up frames, a Core ML INT8 pipeline timed 200 images at 9.00 ms end-to-end p95. Its 6.95 ms per-image mean comprised 4.205 ms JPEG decode, 0.809 ms resize, 0.705 ms normalization, 1.228 ms inference and 0.004 ms argmax. Decode alone consumed about 60% of the mean; speeding up inference would not address the largest stage. section 12.2

Listing 12.4 Timing each stage and the end-to-end path
import time
import numpy as np
from dataclasses import dataclass
from _common import percentile           # shared helper, lives next to the script
 
@dataclass
class StageTimes:
    decode: float = 0.0
    resize: float = 0.0
    normalize: float = 0.0
    infer: float = 0.0
    post: float = 0.0
    def total(self) -> float:
        return self.decode + self.resize + self.normalize + self.infer + self.post
 
def run_once(predict, path):
    t = StageTimes()
    t0 = time.perf_counter()
    img = _decode(path);     t1 = time.perf_counter()   # JPEG bytes -> RGB
    img = _resize(img);      t2 = time.perf_counter()   # 224x224 bilinear
    x = _normalize(img);     t3 = time.perf_counter()   # 1CHW, ImageNet mean/std
    logits = predict(x);     t4 = time.perf_counter()   # Core ML forward pass
    _ = _post(logits);       t5 = time.perf_counter()   # argmax
    t.decode    = (t1 - t0) * 1000.0
    t.resize    = (t2 - t1) * 1000.0
    t.normalize = (t3 - t2) * 1000.0
    t.infer     = (t4 - t3) * 1000.0
    t.post      = (t5 - t4) * 1000.0
    return t                                            #A
 
predict = _build_predictor("coreml", model_path)        #B
for p in paths[:warmup]:                                #C
    run_once(predict, p)
samples = [run_once(predict, p) for p in paths[warmup:warmup + n]]
 
stage_mean = {f: float(np.mean([getattr(s, f) for s in samples]))
              for f in ("decode", "resize", "normalize", "infer", "post")}
e2e_p95 = percentile([s.total() for s in samples], 0.95)  #D

The timing listing shows separate stage timers alongside the end-to-end measurement. section 12.2

The bottleneck can change with the device and implementation. On M3, the measured Accelerate vDSP log-mel stage took about 22 ms alongside 14 ms encoder inference. On Pixel, a portable Kotlin log-mel implementation took 243 ms alongside 290 ms inference. Use native tuned primitives where available; otherwise consider a tuned FFT binding or placing the operation in the graph. section 11.5 section 11.6

For language-model serving, distinguish time to first token (TTFT) from inter-token latency and throughput. On the same L4, model and request load, single-request TTFT p95 was 858.6 ms for FP16 and 37.8 ms for AWQ; inter-token latency p95 was 48.8 ms and 9.7 ms respectively. At concurrency 64, AWQ inter-token latency rose to 24.7 ms while throughput increased. Compare settings at matched concurrency rather than treating peak throughput as a latency guarantee. section 12.3.1

Table 12.1 Generation throughput, time-to-first-token p95, and inter-token latency p95 across the concurrency sweep on the same L4. ITL is inter-token latency
Concurrency
FP16 tokens/s
AWQ tokens/s
FP16 TTFT p95 (ms)
AWQ TTFT p95 (ms)
FP16 ITL p95 (ms)
AWQ ITL p95 (ms)
1
22.6
58.7
858.6
37.8
48.8
9.7
2
57.7
166.9
217.0
39.7
48.8
10.1
4
127.8
305.5
98.0
39.3
48.8
20.9
8
236.8
545.2
97.2
55.5
48.8
23.5
16
494.2
1,161.9
98.0
58.4
48.8
24.1
32
858.2
1,854.6
220.0
78.2
48.8
24.3
64
1,587.3
2,925.8
236.9
132.8
48.9
24.7

The sweep shows how throughput and both latency measures change with concurrency for FP16 and AWQ. section 12.3.1

Validate latency on the intended hardware and monitor it after deployment: queue buildup and thermal throttling can cause drift. Quantization can reduce latency, but its benefit depends on the model, hardware and load, and must be weighed against measured quality changes. section 12.0 section 12.4

How Quantization and Fast Inference covers it

How deep it goes
Explains stage timing, end-to-end p95 and concurrency-dependent serving latency with measured device and GPU comparisons.
Examples
Compares an M3 image-classification pipeline and FP16 versus AWQ language-model serving on an L4.
What you can do after
Diagnose latency bottlenecks by timing stages, reporting end-to-end p95 and comparing serving configurations at matched load.
How it connects
Connects latency to throughput, quantization, preprocessing, observability and thermal throttling.

Also explained in

Compare how 12 books explain latency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.