Time to first token measures the elapsed time between submitting a request and delivering the first generated token. It captures initial responsiveness rather than the time required to complete generation. section 12.4.4
Streaming applications make this distinction operationally important: users often perceive output that begins immediately as more responsive than a completed batch response that arrives later. section 12.4.4
Use the metric to inspect the first-response experience:
- Compare initial display delay with total generation time; the two measurements describe different parts of the interaction. section 12.4.4
- Track the metric in LLM observability alongside retrieval behavior, query cost, and other application timings. section 12.1.3
Perceived responsiveness
A response that displays source material immediately and begins generating after three seconds can feel faster than a two-second response that provides no feedback until completion. section 12.4.4
from langchain.callbacks import LangfuseCallbackHandler
langfuse_handler = LangfuseCallbackHandler()
result = chain.invoke(
{"query": "How do I rotate API keys?"},
config={"callbacks": [langfuse_handler]}
)
A Langfuse callback attached to a LangChain invocation provides a concrete integration point for tracing application activity and monitoring first-token timing. section 12.1.3
