Throughput
The amount of work, data, or output a system processes or completes per unit of time.
Apache Kafka in Action
Throughput is the amount of data or number of messages processed or transferred per unit of time.
- How it is used
- Measures Kafka message or data rates when benchmarking producers, planning capacity, and monitoring workloads.
- How deep it goes
- Works through a producer benchmark and configuration comparison, then gives partition and broker sizing guidance without prescribing a universal target rate.
- Examples
- Uses a million-record producer test, a batching comparison, and a replicated-storage calculation for broker planning.
- What you can do after
- Diagnose/debug with it by comparing measured rates and latency before adjusting producer settings or scaling partitions and brokers.
Generative AI in Action
Throughput is the amount of work or number of requests a system processes during a specified period.
- How it is used
- Measures whether an LLM deployment successfully handles incoming demand over time.
- How deep it goes
- Distinguishes handled load from incoming RPS and connects measurement to capacity planning and load tests, without a worked throughput calculation.
- Examples
- Uses Azure OpenAI PTU sizing and stress tests with Apache JMeter or Locust as practical applications.
- What you can do after
- Operate/configure an LLM deployment by monitoring throughput under load and using the results to guide capacity and traffic controls.
Architecting an Apache Iceberg Lakehouse
The maximum volume or rate of data that an ingestion system can process within a specified period.
- How it is used
- Guides ingestion and storage choices by comparing data volume capacity with freshness, latency, reliability, and operational demands.
- How deep it goes
- Defines the metric, compares workload patterns, and details batching, compaction, partitioning, commits, and metadata safeguards.
- Examples
- Uses dashboards, overnight ETL, machine-learning features, real-time analytics, and a 100-billion-event media pipeline.
- What you can do after
- Evaluate ingestion models and tools, then design high-volume Iceberg pipelines with controlled commits and metadata maintenance.
Read how Architecting an Apache Iceberg Lakehouse explains throughput →
LLMs in Production
Throughput is the rate at which a system completes requests or produces outputs over a period of time.
- How it is used
- Measures LLM service capacity as QPS and streamed output pace as TPS during performance decisions.
- How deep it goes
- Defines QPS and TPS, relates them to user experience, and gives load-testing steps without deriving a capacity model.
- Examples
- Uses a Locust streaming test and a Raspberry Pi deployment to show measured and hardware-limited output rates.
- What you can do after
- Diagnose/debug with throughput metrics to find load limits and judge whether streamed output is fast enough for users.
Kafka Streams in Action, Second Edition
Throughput is the amount of data or number of records processed or transferred during a unit of time.
- How it is used
- Optimizes Kafka record processing by distributing work across partitions, consumers, tasks, and threads.
- How deep it goes
- Explains partition and concurrency scaling, resource tradeoffs, and key skew, but does not quantify throughput with a measurement formula.
- Examples
- Uses a ten-partition Kafka Streams repartition and a Spring Kafka loan pipeline with three input and two output partitions.
- What you can do after
- Operate/configure Kafka applications by sizing partitions, setting listener concurrency, repartitioning streams, and checking key balance.
Read how Kafka Streams in Action, Second Edition explains throughput →
PostgreSQL Mistakes and How to Avoid Them
TPS is the number of transactions a database system completes per second.
- How it is used
- Measures PostgreSQL benchmark capacity while evaluating memory allocation and connection concurrency.
- How deep it goes
- Interprets repeated pgbench measurements through TPS, latency, client counts, and overload behavior rather than deriving the metric internally.
- Examples
- Compares 100-client memory configurations, a failed 1,000-client stress test, and a 16-to-2,000-client pgbench sweep.
- What you can do after
- Evaluate/understand benchmark throughput alongside latency to identify underutilization, overload, and unsafe configuration changes.
Read how PostgreSQL Mistakes and How to Avoid Them explains throughput →
Effective Conversational AI
Throughput is the amount of work or number of requests a system processes per unit of time.
- How it is used
- Uses throughput to judge whether RAG configurations can handle high query volumes.
- How deep it goes
- Measures throughput in queries per second and compares it alongside accuracy, latency, and resource use.
- Examples
- Compares PharmaBot configurations at 20, 18, and 16 queries per second alongside accuracy, latency, and resource-use metrics.
- What you can do after
- Evaluate RAG configurations by balancing query capacity against response quality, latency, and resource utilization.
Kubernetes for Developers
Requests per second is a throughput metric measuring the number of requests received or processed during one second.
- How it is used
- Guides capacity planning and horizontal pod autoscaling for HTTP workloads.
- How deep it goes
- Explains measurement, replica-capacity calculation, Ingress requirements, and external-metric configuration without detailing monitoring-provider internals.
- Examples
- Uses a timeserver Deployment, NodePort Service, Ingress, ApacheBench traffic, and an external request-count HPA.
- What you can do after
- Operate/configure Kubernetes autoscaling with an external RPS metric and evaluate replica changes under generated HTTP load.
Machine Learning System Design
Throughput is the amount of work or number of results processed by a system per unit of time.
- How it is used
- Frames inference requirements around the volume of predictions processed over time, especially for large batch workloads.
- How deep it goes
- Defines throughput, contrasts it with latency, and connects it to batching, scalability, tradeoffs, and operational monitoring.
- Examples
- Uses batch demand forecasting with AWS Batch, Docker, EC2, S3, scheduled jobs, and rows-processed monitoring.
- What you can do after
- Operate/configure batch inference by prioritizing processing capacity, selecting scalable infrastructure, and monitoring workload performance.
Read how Machine Learning System Design explains throughput →
Kafka for Architects
Throughput is the rate at which a system processes or transfers data during a specified period.
- How it is used
- Guides Kafka capacity planning, configuration, cost evaluation, and operational monitoring for event-processing systems.
- How deep it goes
- Defines volume, peak rate, traffic pattern, and burst recovery, then links them to settings without detailing implementation procedures.
- Examples
- Uses streaming-platform events, small e-commerce traffic, and a 300,000-message-per-day burst profile.
- What you can do after
- Evaluate throughput requirements and select configuration and monitoring priorities for Kafka-based systems.
Transformers in Action
Throughput is the amount of inference work completed per unit of time, such as generated tokens or served requests per second.
- How it is used
- Frames inference optimization around reducing latency and maximizing generated-token or request throughput during deployment.
- How deep it goes
- Explains throughput through KV caching, memory growth, scheduling, layouts, tiling, and coalesced GPU access, but stops short of a throughput formula.
- Examples
- Demonstrates cache-enabled generation benchmarking, vLLM serving, tiled matrix multiplication, and row-major versus column-major layouts.
- What you can do after
- Diagnose throughput bottlenecks by comparing KV-cache behavior and examining GPU memory layout, bandwidth, tiling, and access coalescing.
Starting Data Analytics with Generative AI and Python
Throughput is the rate at which a system processes data, commonly expressed as transactions or records per unit of time.
- How it is used
- Measures data-processing capacity when evaluating performance, access patterns, workload baselines, and future expansion.
- How deep it goes
- Defines the rate, gives its division formula, and explains practical influences without detailing implementation internals.
- Examples
- Compares execution time and throughput for a bootstrapping function across DataFrame sizes.
- What you can do after
- Evaluate processing capacity and estimate resource requirements as the amount of data increases.
Read how Starting Data Analytics with Generative AI and Python explains throughput →
Refactoring to Rust
Throughput is the amount of work a system completes per unit of time.
- How it is used
- Uses throughput to identify performance limits and motivate refactoring or horizontal and vertical scaling.
- How deep it goes
- Explains bottlenecks and scaling strategies, but stops short of measuring throughput or detailing optimization techniques.
- Examples
- Uses concurrent Python processes, shared image output, and a Mandelbrot loop to illustrate scaling behavior.
- What you can do after
- Evaluate whether a bottleneck or scaling strategy is limiting system capacity and recognize unsafe multi-process behavior.
Amazon Web Services in Action, Third Edition: An in-depth guide to AWS
Throughput is the rate at which a storage service processes read or write operations, commonly measured per second.
- How it is used
- Uses throughput to size S3 object-key layouts, select cache sharding, and scale EC2 workloads against network demand.
- How deep it goes
- Explains per-prefix limits, naming effects, cache sharding, and proportional scaling, but stops short of general performance measurement methods.
- Examples
- Uses archive year prefixes, Redis shard counts, and EC2 network-byte metrics as concrete throughput examples.
- What you can do after
- Apply throughput reasoning by redesigning S3 prefixes, choosing cache sharding, or configuring proportional autoscaling metrics.
Microsoft Azure in Action
Throughput is the amount of data or request processing successfully handled by a system per unit of time.
- How it is used
- Uses throughput to size Cosmos DB capacity for ordinary workloads and traffic spikes.
- How deep it goes
- Explains request-unit configuration and autoscaling limits, but stops short of calculating workload-specific capacity.
- Examples
- Uses a holiday-traffic scenario, a 400-request-unit container, and a 10,000-request-unit autoscale ceiling.
- What you can do after
- Operate/configure Cosmos DB throughput with Azure CLI, including fixed capacity and autoscaling limits.
Latency
Throughput is the rate at which a system processes or transfers units of work or data over time.
- How it is used
- Measures how much data an application transfers or how many requests it processes per time unit.
- How deep it goes
- Explains measurement, bandwidth limits, concurrency, pipelining, bottlenecks, and latency trade-offs with a worked example.
- Examples
- Uses network transfer rates, request processing, and serial versus pipelined laundry processing.
- What you can do after
- Evaluate application performance, compare capacity with delivered work, and reason about concurrency and pipeline bottlenecks.
Automating API Delivery
Throughput is the amount of work or number of requests processed by a system within a specified time interval.
- How it is used
- Measures API request activity as part of RED monitoring for microservice performance.
- How deep it goes
- Defines requests processed per time window and relates measurement periods to usage trends.
- Examples
- Uses per-second, per-minute, per-week, and per-month request measurements.
- What you can do after
- Evaluate API usage trends by measuring requests across selected time windows.
Delivering Machine Learning Projects: From design to deployment
Throughput is the amount of work or data that a system processes within a specified time interval.
- How it is used
- Defines a production nonfunctional requirement for the number of model executions or service requests completed within a timeframe.
- How deep it goes
- Explains throughput, contrasts it with latency, and covers cold starts, parallelism, bottlenecks, and production-like measurement.
- Examples
- Uses cold starts, ten parallel models, production-mirroring test harnesses, and service-request targets as evaluation examples.
- What you can do after
- Evaluate throughput against end-to-end targets using production-like hardware, workloads, and serving conditions.
Read how Delivering Machine Learning Projects: From design to deployment explains throughput →
Grokking Concurrency
Throughput is the number of tasks a system completes or handles during a given period of time.
- How it is used
- Measures system performance by counting how much work completes during a period.
- How deep it goes
- Defines throughput and contrasts it with latency, stopping short of implementation or optimization procedures.
- Examples
- Uses motorcycle and bus transportation to contrast individual travel speed with people transported per hour.
- What you can do after
- Evaluate/understand performance by distinguishing completed work per time from the duration of one task.
ScyllaDB in Action
Throughput is the rate at which a system completes operations or processes data over time.
- How it is used
- Guides ScyllaDB load testing and migration-rate decisions by showing how much traffic a database can handle.
- How deep it goes
- Compares operations-per-second results with latency during progressively heavier tests, without developing a formal capacity model.
- Examples
- Uses cassandra-stress results, partition and row rates, Grafana dashboards, and a source-to-destination migration scenario.
- What you can do after
- Diagnose database capacity with cassandra-stress and Grafana while balancing throughput against latency and degradation.
Related concepts
- meancontrasts with · A mean is a measure of central tendency, calculated as the arithmetic average of values or, for a random variable, as the probability-weighted average of its possible values.
- performancemeasures · Performance is the measurable effectiveness or operational behavior of a system under specified conditions, assessed through relevant qualities such as task results, speed, responsiveness, throughput, or resource use.
- parallelismproduced by · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
- monitoringmeasured by · Monitoring is the ongoing observation and measurement of a system’s behavior, state, or operational signals to assess its condition and detect changes or problems.
- network bandwidthconfigured by · Network bandwidth is the maximum rate at which a network connection can transfer data, typically measured in bits per second.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.