liveBook · Manning

Latency

As explained in Rearchitecting LLMs

Latency is elapsed time measured for a language model request or for producing an individual output token.

Latency here is generation time per token. The unmodified Llama-3.2-3B baseline produces 33.55 milliseconds per token at batch size 1 on a Colab T4; this is a comparison point, not a universal performance figure. section 12.6.2

(EngineCore pid=4098) INFO 10-04 18:24:45 [gpu_worker.py:640]
Available KV cache memory: 5.63 GiB
(EngineCore pid=4098) INFO 10-04 18:24:45 [kv_cache_utils.py:2395]
GPU KV cache size: 52,720 tokens, Maximum concurrency for 2,048 tokens
per request: 25.74x
RESULT ms_per_token=33.55 text=' London, Paris and Rome. These cities
are the most popular tourist destinations in Europe. The most visited
city in Europe is London. London is the capital'
(log: ch12_vllm/logs/control.log)

Inspect the probe output for the measured latency and the associated KV cache results. section 12.6.2

How Rearchitecting LLMs covers it

How deep it goes
Measures milliseconds per generated token with a fixed probe, but does not establish performance across other setups.
Examples
Benchmarks Llama-3.2-3B with a fixed prompt and reports 33.55 milliseconds per token on a Colab T4.
What you can do after
Evaluate generation speed against a baseline measured under a specified configuration.
How it connects
Ties token-generation timing to vLLM inference and KV cache measurements.

Also explained in

Compare how 12 books explain latency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.