liveBook · Manning

Throughput

As explained in Vibe Engineering

Throughput is the rate at which a system produces outputs, measured here as generated tokens per second.

Throughput measures how quickly a system produces useful output. For local language-model inference, the practical unit is generated tokens per second; for services and agent fleets, request frequency, concurrent work, and parallel builds reveal the same capacity question. section 9.3.2 section 5.5.3

Use throughput as an operational metric rather than a speed claim:

  • Compare output rate with the workload: a 70B model producing 15–20 tokens/sec may suit offline generation but not interactive completion, while a 7B model can reach 50–70 tok/s on an RTX 4060/4070. section 9.3.2
  • Weight optimization by demand, not by the worst latency alone: reducing a frequent endpoint's mean latency can save more total processing time than optimizing a slower endpoint called rarely. section 7.2
  • Treat parallelism as an architectural concern: independent modules reduce rebuild scope, cache contention, and I/O conflict when many agents work concurrently. section 10.3.5

Interactive versus offline generation

An M4 Max with 64GB or 128GB can run Llama 3.1 70B, but its 15–20 tokens/sec rate is described as too slow for interactive completions. A smaller model on an RTX 4060/4070 reaches 50–70 tok/s and is better aligned with responsive use. section 9.3.2

Figure 9.3 LLMfit software interface.

Use the LLMfit interface to compare model size, quantization, available memory, headroom, and expected throughput before downloading a model. section 9.3.2

How Vibe Engineering covers it

How deep it goes
Quantifies tokens per second, request-weighted processing, and parallel build effects without developing a general throughput formula.
Examples
Uses token-rate comparisons, request-frequency calculations, and parallel module-build timings as concrete performance evidence.
What it assumes
Assumes familiarity with latency, request frequency, model quantization, hardware memory, and concurrent builds.
What you can do after
Evaluate hardware, model, endpoint, or module choices against workload demand and expected parallel capacity.
How it connects
Connects throughput with latency, scalability, concurrency, quantization, hardware memory, and modular architecture.

Also explained in

Compare how 32 books explain throughput →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.