Throughput measures how quickly a system produces useful output. For local language-model inference, the practical unit is generated tokens per second; for services and agent fleets, request frequency, concurrent work, and parallel builds reveal the same capacity question. section 9.3.2 section 5.5.3
Use throughput as an operational metric rather than a speed claim:
- Compare output rate with the workload: a 70B model producing 15–20 tokens/sec may suit offline generation but not interactive completion, while a 7B model can reach 50–70 tok/s on an RTX 4060/4070. section 9.3.2
- Weight optimization by demand, not by the worst latency alone: reducing a frequent endpoint's mean latency can save more total processing time than optimizing a slower endpoint called rarely. section 7.2
- Treat parallelism as an architectural concern: independent modules reduce rebuild scope, cache contention, and I/O conflict when many agents work concurrently. section 10.3.5
Interactive versus offline generation
An M4 Max with 64GB or 128GB can run Llama 3.1 70B, but its 15–20 tokens/sec rate is described as too slow for interactive completions. A smaller model on an RTX 4060/4070 reaches 50–70 tok/s and is better aligned with responsive use. section 9.3.2
Figure 9.3 LLMfit software interface.

Use the LLMfit interface to compare model size, quantization, available memory, headroom, and expected throughput before downloading a model. section 9.3.2































