Latency here is generation time per token. The unmodified Llama-3.2-3B baseline produces 33.55 milliseconds per token at batch size 1 on a Colab T4; this is a comparison point, not a universal performance figure. section 12.6.2
(EngineCore pid=4098) INFO 10-04 18:24:45 [gpu_worker.py:640] Available KV cache memory: 5.63 GiB (EngineCore pid=4098) INFO 10-04 18:24:45 [kv_cache_utils.py:2395] GPU KV cache size: 52,720 tokens, Maximum concurrency for 2,048 tokens per request: 25.74x RESULT ms_per_token=33.55 text=' London, Paris and Rome. These cities are the most popular tourist destinations in Europe. The most visited city in Europe is London. London is the capital' (log: ch12_vllm/logs/control.log)
Inspect the probe output for the measured latency and the associated KV cache results. section 12.6.2











