liveBook · Manning

Time to first token

Also called TTFT

Time to first token is the elapsed time from submitting a generation request until its first generated token is produced or available.

How 6 Manning books explain it

Quantization and Fast Inference

The elapsed time from submitting a language-model request until its first generated token is available.

How it is used
Uses TTFT as the user-visible response-latency measure for language-model deployment budgets and service-level agreements.
How deep it goes
Explains prefill and decode separately, with a prompt-length sweep and guidance on latency and power budgets.
Examples
Compares Llama-3.2-1B TTFT on a Pixel across 32- and 1,024-token prompts, alongside decode rate and power.
What you can do after
Evaluate language-model configurations against initial-response latency and sustained-power budgets while distinguishing requested from observed compute backends.

Read how Quantization and Fast Inference explains time to first token →

AI Model Evaluation

Time to first token is the elapsed time from a request until the first generated token becomes available to the user.

How it is used
Guides evaluation of perceived responsiveness in LLM-powered chat and streaming experiences.
How deep it goes
Defines TTFT and situates it among prompt, retrieval, generation, completion, cost, and failure metrics without prescribing an implementation.
Examples
Uses a six-second chatbot response and early token streaming to contrast blank-screen delay with visible progress.
What you can do after
Evaluate LLM experiences by measuring first output, continued generation, completion time, and surrounding production bottlenecks.

Read how AI Model Evaluation explains time to first token →

Machine Learning Platform Engineering

Time-to-first-token is the elapsed time from a request until the first generated token is delivered to the user.

How it is used
Evaluates user-perceived responsiveness in streaming LLM applications rather than relying only on total response latency.
How deep it goes
Defines the metric and contrasts it with total generation time, but does not detail its internal timing instrumentation.
Examples
Uses streaming-versus-batch comparisons and Langfuse monitoring of time to first output token.
What you can do after
Evaluate initial LLM responsiveness and monitor it through Langfuse instrumentation.

Read how Machine Learning Platform Engineering explains time to first token →

Generative AI in Action

First token latency is the elapsed time from a request until the first generated token is returned by a generative model.

How it is used
Frames first-token responsiveness as a core operational metric for developing and deploying generative AI applications.
How deep it goes
Explains the timing boundary and major influences, then stops short of detailed model-internal performance analysis.
Examples
Uses Azure OpenAI REST measurement, streaming requests, concurrent calls, and contrasting short and long poem-generation workloads.
What you can do after
Measure it via a streaming Azure OpenAI API test and interpret results alongside prompt size, model size, TPOT, and throughput.

Read how Generative AI in Action explains time to first token →

Rearchitecting LLMs

Time to first token is the elapsed time from a generation request until the model produces its first output token.

How it is used
Measures first-token latency for perceived responsiveness in interactive applications and latency-sensitive agent calls.
How deep it goes
Defines TTFT and reports benchmark comparisons, but does not detail timing instrumentation.
Examples
Compares base and depth-pruned model TTFT on a T4 GPU and describes repeated short agent calls.
What you can do after
Evaluate first-token responsiveness alongside total inference time and throughput when comparing serving configurations.

Read how Rearchitecting LLMs explains time to first token →

LLMs in Production

Time to first token is the elapsed time between submitting a language-model request and receiving its first generated token.

How it is used
Measures perceived responsiveness for streaming language-model services during load testing.
How deep it goes
Defines TTFT as delay until feedback and computes it in a Locust streaming test, without prescribing a TTFT target.
Examples
Uses a Locust streaming request to record timestamps and export TTFT, latency, and TPS to CSV.
What you can do after
Diagnose perceived responsiveness by measuring TTFT alongside latency and tokens per second in a Locust load test.

Read how LLMs in Production explains time to first token →

Also covered in

What to know first

Related concepts

Questions about time to first token

What is time to first token?

Time to first token is the elapsed time from submitting a generation request until its first generated token is produced or available.

Which Manning books explain time to first token?

Quantization and Fast Inference, AI Model Evaluation, Machine Learning Platform Engineering, Generative AI in Action, Rearchitecting LLMs and LLMs in Production.

What should I know before learning time to first token?

Manning books that teach time to first token usually introduce latency, stream processing, mean and centroid first.

What is the difference between time to first token and time-to-complete?

Time to first token is the elapsed time from submitting a generation request until its first generated token is produced or available. Time-to-complete is the elapsed time from submitting a generation request until the complete generated response is received.

What is time to first token a kind of?

Time to first token is a kind of latency. Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.

What is time to first token also called?

Time to first token is also called TTFT.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.