liveBook · Manning

Throughput

Also called tokens per second

Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.

How 32 Manning books explain it

Quantization and Fast Inference

The amount of computation or output completed per unit of time, including its retention during sustained operation.

How it is used
Uses throughput to size inference capacity and compare deployment cost under real workloads.
How deep it goes
Explains bandwidth and batching effects, then measures concurrency, thread-count, and sustained-operation tradeoffs on specific hardware.
Examples
Compares FP16 and AWQ across L4 concurrency levels and tracks five-minute throughput on a fanless M3.
What you can do after
Evaluate throughput results against latency, sustained operation, hardware constraints, and cost when choosing an inference deployment.

Read how Quantization and Fast Inference explains throughput →

Generative AI in Action

Throughput is the amount of work or number of requests a system processes during a specified period.

How it is used
Uses throughput to assess whether production LLM deployments can handle incoming demand.
How deep it goes
Defines throughput in relation to RPS and provides capacity-planning and monitoring guidance without a detailed throughput calculation.
Examples
Illustrates capacity allocation with a 240,000 TPM regional quota split across deployments.
What you can do after
Operate/configure deployment capacity by allocating TPM, managing rate limits, and testing throughput under load.

Read how Generative AI in Action explains throughput →

Rearchitecting LLMs

Throughput is the amount of model work or output that a system completes per unit of time.

How it is used
Measures inference efficiency when comparing base and pruned language models.
How deep it goes
Defines tokens per second and reports benchmark conditions and results, while deferring evaluation-system details and controlled attention-pruning speed analysis.
Examples
Compares base and pruned models on a Colab T4, including depth-pruning and batched width-pruning benchmarks.
What you can do after
Evaluate model inference performance against a baseline while accounting for quality trade-offs and measurement variability.

Read how Rearchitecting LLMs explains throughput →

Apache Kafka in Action

Throughput is the amount of data or number of messages processed or transferred per unit of time.

How it is used
Measures Kafka message or data rates when benchmarking producers, planning capacity, and monitoring workloads.
How deep it goes
Works through a producer benchmark and configuration comparison, then gives partition and broker sizing guidance without prescribing a universal target rate.
Examples
Uses a million-record producer test, a batching comparison, and a replicated-storage calculation for broker planning.
What you can do after
Diagnose/debug with it by comparing measured rates and latency before adjusting producer settings or scaling partitions and brokers.

Read how Apache Kafka in Action explains throughput →

Architecting an Apache Iceberg Lakehouse

The maximum volume or rate of data that an ingestion system can process within a specified period.

How it is used
Guides ingestion and storage choices by comparing data volume capacity with freshness, latency, reliability, and operational demands.
How deep it goes
Defines the metric, compares workload patterns, and details batching, compaction, partitioning, commits, and metadata safeguards.
Examples
Uses dashboards, overnight ETL, machine-learning features, real-time analytics, and a 100-billion-event media pipeline.
What you can do after
Evaluate ingestion models and tools, then design high-volume Iceberg pipelines with controlled commits and metadata maintenance.

Read how Architecting an Apache Iceberg Lakehouse explains throughput →

LLMs in Production

Throughput is the rate at which a system completes requests or produces outputs over a period of time.

How it is used
Uses throughput to assess LLM service capacity and the responsiveness of streamed output.
How deep it goes
Defines QPS and TPS, outlines load-testing steps, and explains batching’s latency trade-off.
Examples
Illustrates throughput with a Locust streaming test and an LLM running on a Raspberry Pi.
What you can do after
Evaluate service capacity and streaming responsiveness using QPS, TPS, latency, and time to first token.

Read how LLMs in Production explains throughput →

Kafka Streams in Action, Second Edition

Throughput is the amount of data or number of records processed or transferred during a unit of time.

How it is used
Optimizes Kafka record processing by distributing work across partitions, consumers, tasks, and threads.
How deep it goes
Explains partition and concurrency scaling, resource tradeoffs, and key skew, but does not quantify throughput with a measurement formula.
Examples
Uses a ten-partition Kafka Streams repartition and a Spring Kafka loan pipeline with three input and two output partitions.
What you can do after
Operate/configure Kafka applications by sizing partitions, setting listener concurrency, repartitioning streams, and checking key balance.

Read how Kafka Streams in Action, Second Edition explains throughput →

PostgreSQL Mistakes and How to Avoid Them

TPS is the number of transactions a database system completes per second.

How it is used
Measures PostgreSQL benchmark capacity while evaluating memory allocation and connection concurrency.
How deep it goes
Interprets repeated pgbench measurements through TPS, latency, client counts, and overload behavior rather than deriving the metric internally.
Examples
Compares 100-client memory configurations, a failed 1,000-client stress test, and a 16-to-2,000-client pgbench sweep.
What you can do after
Evaluate/understand benchmark throughput alongside latency to identify underutilization, overload, and unsafe configuration changes.

Read how PostgreSQL Mistakes and How to Avoid Them explains throughput →

Effective Conversational AI

Throughput is the amount of work or number of requests a system processes per unit of time.

How it is used
Uses throughput to judge whether RAG configurations can handle high query volumes.
How deep it goes
Measures throughput in queries per second and compares it alongside accuracy, latency, and resource use.
Examples
Compares PharmaBot configurations at 20, 18, and 16 queries per second alongside accuracy, latency, and resource-use metrics.
What you can do after
Evaluate RAG configurations by balancing query capacity against response quality, latency, and resource utilization.

Read how Effective Conversational AI explains throughput →

Vibe Engineering

Throughput is the rate at which a system produces outputs, measured here as generated tokens per second.

How it is used
Guides capacity decisions for local inference, service optimization, and concurrent agent or build workloads.
How deep it goes
Quantifies tokens per second, request-weighted processing, and parallel build effects without developing a general throughput formula.
Examples
Uses token-rate comparisons, request-frequency calculations, and parallel module-build timings as concrete performance evidence.
What you can do after
Evaluate hardware, model, endpoint, or module choices against workload demand and expected parallel capacity.

Read how Vibe Engineering explains throughput →

Kubernetes for Developers

Requests per second is a throughput metric measuring the number of requests received or processed during one second.

How it is used
Guides capacity planning and horizontal pod autoscaling for HTTP workloads.
How deep it goes
Explains measurement, replica-capacity calculation, Ingress requirements, and external-metric configuration without detailing monitoring-provider internals.
Examples
Uses a timeserver Deployment, NodePort Service, Ingress, ApacheBench traffic, and an external request-count HPA.
What you can do after
Operate/configure Kubernetes autoscaling with an external RPS metric and evaluate replica changes under generated HTTP load.

Read how Kubernetes for Developers explains throughput →

Machine Learning System Design

Throughput is the amount of work or number of results processed by a system per unit of time.

How it is used
Frames inference requirements around the volume of predictions processed over time, especially for large batch workloads.
How deep it goes
Defines throughput, contrasts it with latency, and connects it to batching, scalability, tradeoffs, and operational monitoring.
Examples
Uses batch demand forecasting with AWS Batch, Docker, EC2, S3, scheduled jobs, and rows-processed monitoring.
What you can do after
Operate/configure batch inference by prioritizing processing capacity, selecting scalable infrastructure, and monitoring workload performance.

Read how Machine Learning System Design explains throughput →

Kafka for Architects

Throughput is the rate at which a system processes or transfers data during a specified period.

How it is used
Guides Kafka capacity planning, configuration, cost evaluation, and operational monitoring for event-processing systems.
How deep it goes
Defines volume, peak rate, traffic pattern, and burst recovery, then links them to settings without detailing implementation procedures.
Examples
Uses streaming-platform events, small e-commerce traffic, and a 300,000-message-per-day burst profile.
What you can do after
Evaluate throughput requirements and select configuration and monitoring priorities for Kafka-based systems.

Read how Kafka for Architects explains throughput →

JavaScript in Depth

The amount of useful data or work a system completes per unit of time.

How it is used
Uses throughput to compare stream models and diagnose performance problems alongside latency, memory, and allocation behavior.
How deep it goes
Explains controlled measurement, allocation costs, backpressure, buffering, and conversion overhead without deriving a general performance model.
Examples
Compares 4 KB chunks across stream models and examines fast-producer, slow-consumer memory behavior and repeated stream conversion.
What you can do after
Diagnose performance bottlenecks by measuring comparable workloads and interpreting throughput with allocation, memory, latency, and garbage-collection signals.

Read how JavaScript in Depth explains throughput →

Build High Value Software Systems

The rate at which completed work passes through a production or delivery system.

How it is used
Diagnoses architectural components that constrain request throughput and guides where capacity improvements belong.
How deep it goes
Applies five constraint-management steps, from identifying a shared database bottleneck through increasing capacity and finding the next constraint.
Examples
Compares food-delivery service latencies and mitigations for a shared PostgreSQL bottleneck.
What you can do after
Diagnose a limiting component by comparing peak response times, then prioritize exploiting or elevating that constraint.

Read how Build High Value Software Systems explains throughput →

Domain-Specific Small Language Models

The rate at which a processor can perform floating-point operations over time.

How it is used
Uses GPU throughput and bandwidth to analyze inference speed, accuracy, compute cost, and optimization choices.
How deep it goes
Explains ops-per-byte bottleneck analysis and generation-time estimates, while noting simplifying assumptions in the timing model.
Examples
Uses GPT-Neo on an NVIDIA Tesla T4 to compare hardware specifications, estimate generation time, and measure observed inference duration.
What you can do after
Diagnose whether inference is memory- or compute-bound and estimate generation time from model size, precision, bandwidth, and prefill cost.

Read how Domain-Specific Small Language Models explains throughput →

Grokking GPU Programming

Throughput is the amount of work completed per unit of time or over an overall execution period.

How it is used
Positions throughput as the GPU's design goal in the CPU–GPU partnership.
How deep it goes
Defines throughput and latency, then explains how GPU cores and memory bandwidth support high throughput.
Examples
Uses image filtering and matrix math as examples of GPU-suited work.
What you can do after
Evaluate workload suitability for GPUs by checking for abundant independent work and regular memory access.

Read how Grokking GPU Programming explains throughput →

Transformers in Action

Throughput is the amount of inference work completed per unit of time, such as generated tokens or served requests per second.

How it is used
Frames inference optimization around reducing latency and maximizing generated-token or request throughput during deployment.
How deep it goes
Explains throughput through KV caching, memory growth, scheduling, layouts, tiling, and coalesced GPU access, but stops short of a throughput formula.
Examples
Demonstrates cache-enabled generation benchmarking, vLLM serving, tiled matrix multiplication, and row-major versus column-major layouts.
What you can do after
Diagnose throughput bottlenecks by comparing KV-cache behavior and examining GPU memory layout, bandwidth, tiling, and access coalescing.

Read how Transformers in Action explains throughput →

Starting Data Analytics with Generative AI and Python

Throughput is the rate at which a system processes data, commonly expressed as transactions or records per unit of time.

How it is used
Measures data-processing capacity when evaluating performance, access patterns, workload baselines, and future expansion.
How deep it goes
Defines the rate, gives its division formula, and explains practical influences without detailing implementation internals.
Examples
Compares execution time and throughput for a bootstrapping function across DataFrame sizes.
What you can do after
Evaluate processing capacity and estimate resource requirements as the amount of data increases.

Read how Starting Data Analytics with Generative AI and Python explains throughput →

GPU Programming with Triton

Throughput is the amount of computation or data processing completed per unit of time.

How it is used
Frames GPU performance analysis around the rates of computation and data movement.
How deep it goes
Defines compute and memory ceilings, then classifies kernels by arithmetic intensity and distance from those ceilings.
Examples
Uses H100 peak-rate estimates, representative roofline kernels, and matrix multiplication in transformer workloads.
What you can do after
Evaluate whether a kernel is memory-bound or compute-bound and identify its distance from the corresponding performance ceiling.

Read how GPU Programming with Triton explains throughput →

Refactoring to Rust

Throughput is the amount of work a system completes per unit of time.

How it is used
Uses throughput to identify performance limits and motivate refactoring or horizontal and vertical scaling.
How deep it goes
Explains bottlenecks and scaling strategies, but stops short of measuring throughput or detailing optimization techniques.
Examples
Uses concurrent Python processes, shared image output, and a Mandelbrot loop to illustrate scaling behavior.
What you can do after
Evaluate whether a bottleneck or scaling strategy is limiting system capacity and recognize unsafe multi-process behavior.

Read how Refactoring to Rust explains throughput →

AI Model Evaluation

Tokens per second is the rate at which a generative model produces output tokens after generation begins.

How it is used
Tracks how quickly generated output reaches users after generation starts, complementing first-token and completion timing.
How deep it goes
Defines output rate after generation begins and distinguishes it from initial wait and completion time.
Examples
Uses chat and streaming interfaces to frame visible output flow after generation starts.
What you can do after
Evaluate response flow by pairing tokens per second with first-token and completion timing.

Read how AI Model Evaluation explains throughput →

Amazon Web Services in Action, Third Edition: An in-depth guide to AWS

Throughput is the rate at which a storage service processes read or write operations, commonly measured per second.

How it is used
Uses throughput to size S3 object-key layouts, select cache sharding, and scale EC2 workloads against network demand.
How deep it goes
Explains per-prefix limits, naming effects, cache sharding, and proportional scaling, but stops short of general performance measurement methods.
Examples
Uses archive year prefixes, Redis shard counts, and EC2 network-byte metrics as concrete throughput examples.
What you can do after
Apply throughput reasoning by redesigning S3 prefixes, choosing cache sharding, or configuring proportional autoscaling metrics.

Read how Amazon Web Services in Action, Third Edition: An in-depth guide to AWS explains throughput →

CUDA for LLMs

Throughput is the amount of work a system completes per unit of time.

How it is used
Measures performance for GPU deep-learning workloads, including token generation, quantized inference, and distributed execution.
How deep it goes
Defines throughput against latency and connects the metric to token rates, request rates, and batch pipelines.
Examples
Uses token and request rates, quantized inference measurements, PyTorch–CUDA comparisons, and multi-GPU batch pipelines.
What you can do after
Evaluate end-to-end performance by measuring throughput alongside latency, memory use, and output correctness.

Read how CUDA for LLMs explains throughput →

Microsoft Azure in Action

Throughput is the amount of data or request processing successfully handled by a system per unit of time.

How it is used
Uses throughput to size Cosmos DB capacity for ordinary workloads and traffic spikes.
How deep it goes
Explains request-unit configuration and autoscaling limits, but stops short of calculating workload-specific capacity.
Examples
Uses a holiday-traffic scenario, a 400-request-unit container, and a 10,000-request-unit autoscale ceiling.
What you can do after
Operate/configure Cosmos DB throughput with Azure CLI, including fixed capacity and autoscaling limits.

Read how Microsoft Azure in Action explains throughput →

Latency

Throughput is the rate at which a system processes or transfers units of work or data over time.

How it is used
Measures how much data an application transfers or how many requests it processes per time unit.
How deep it goes
Explains measurement, bandwidth limits, concurrency, pipelining, bottlenecks, and latency trade-offs with a worked example.
Examples
Uses network transfer rates, request processing, and serial versus pipelined laundry processing.
What you can do after
Evaluate application performance, compare capacity with delivered work, and reason about concurrency and pipeline bottlenecks.

Read how Latency explains throughput →

Decision Engineering

Throughput is the amount of work or number of operations a system completes per unit of time.

How it is used
Quantifies processing capacity and helps distinguish system handling from the amount and kind of demanded work.
How deep it goes
Defines throughput as requests or data volume per second and distinguishes it from load and scalability.
Examples
Compares storefront and backoffice search, and gives throughput tiers for storefront, B2B, backoffice, and background processing.
What you can do after
Evaluate requirements by comparing component demand and peak scenarios rather than relying on one request-rate figure.

Read how Decision Engineering explains throughput →

Automating API Delivery

Throughput is the amount of work or number of requests processed by a system within a specified time interval.

How it is used
Measures API request activity as part of RED monitoring for microservice performance.
How deep it goes
Defines requests processed per time window and relates measurement periods to usage trends.
Examples
Uses per-second, per-minute, per-week, and per-month request measurements.
What you can do after
Evaluate API usage trends by measuring requests across selected time windows.

Read how Automating API Delivery explains throughput →

Delivering Machine Learning Projects: From design to deployment

Throughput is the amount of work or data that a system processes within a specified time interval.

How it is used
Defines a production nonfunctional requirement for the number of model executions or service requests completed within a timeframe.
How deep it goes
Explains throughput, contrasts it with latency, and covers cold starts, parallelism, bottlenecks, and production-like measurement.
Examples
Uses cold starts, ten parallel models, production-mirroring test harnesses, and service-request targets as evaluation examples.
What you can do after
Evaluate throughput against end-to-end targets using production-like hardware, workloads, and serving conditions.

Read how Delivering Machine Learning Projects: From design to deployment explains throughput →

Grokking Concurrency

Throughput is the number of tasks a system completes or handles during a given period of time.

How it is used
Measures system performance by counting how much work completes during a period.
How deep it goes
Defines throughput and contrasts it with latency, stopping short of implementation or optimization procedures.
Examples
Uses motorcycle and bus transportation to contrast individual travel speed with people transported per hour.
What you can do after
Evaluate/understand performance by distinguishing completed work per time from the duration of one task.

Read how Grokking Concurrency explains throughput →

Fast by Default

Throughput is the number of requests or other work items a service handles per unit of time.

How it is used
Uses requests per second to plan backend capacity and assess whether services can handle peak user traffic.
How deep it goes
Defines throughput, contrasts it with response speed, and specifies peak-load targets with surge headroom.
Examples
Uses a service handling 500 requests per second at peak and contrasts fast responses with insufficient request capacity.
What you can do after
Evaluate backend capacity by measuring requests per second against peak traffic, response speed, errors, and planned headroom.

Read how Fast by Default explains throughput →

ScyllaDB in Action

Throughput is the rate at which a system completes operations or processes data over time.

How it is used
Guides ScyllaDB load testing and migration-rate decisions by showing how much traffic a database can handle.
How deep it goes
Compares operations-per-second results with latency during progressively heavier tests, without developing a formal capacity model.
Examples
Uses cassandra-stress results, partition and row rates, Grafana dashboards, and a source-to-destination migration scenario.
What you can do after
Diagnose database capacity with cassandra-stress and Grafana while balancing throughput against latency and degradation.

Read how ScyllaDB in Action explains throughput →

Also covered in

What to know first

Related concepts

Questions about throughput

What is throughput?

Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.

Which Manning books explain throughput?

Quantization and Fast Inference, Generative AI in Action, Rearchitecting LLMs, Apache Kafka in Action, Architecting an Apache Iceberg Lakehouse, LLMs in Production, Kafka Streams in Action, Second Edition, PostgreSQL Mistakes and How to Avoid Them and 24 more.

What should I know before learning throughput?

Manning books that teach throughput usually introduce performance, language model, mean and batching first.

What is the difference between throughput and latency?

Throughput is the rate at which a system processes or completes work, data, or requests over a specified period. Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.

What is throughput also called?

Throughput is also called tokens per second.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.