liveBook · Manning

Throughput

As explained in AI Model Evaluation

Tokens per second is the rate at which a generative model produces output tokens after generation begins.

Tokens per second measures how quickly a generative model emits output after generation begins. In streaming chat, it captures the pace users see once the first token arrives; it complements time to first token and time to final token, which mark the start and completion. section 4.2.3

Treat it as one part of system performance, not a stand-alone quality signal. Increasing batch size may improve throughput while worsening latency for individual users, so assess the trade-off alongside model quality and product impact. section 4.2.4 section 4.6

How AI Model Evaluation covers it

How deep it goes
Defines output rate after generation begins and distinguishes it from initial wait and completion time.
Examples
Uses chat and streaming interfaces to frame visible output flow after generation starts.
What you can do after
Evaluate response flow by pairing tokens per second with first-token and completion timing.
How it connects
Connects output rate to time to first token, time to final token, and latency.

Also explained in

Compare how 32 books explain throughput →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.