Throughput
Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.
Quantization and Fast Inference
The amount of computation or output completed per unit of time, including its retention during sustained operation.
- How it is used
- Uses throughput to size inference capacity and compare deployment cost under real workloads.
- How deep it goes
- Explains bandwidth and batching effects, then measures concurrency, thread-count, and sustained-operation tradeoffs on specific hardware.
- Examples
- Compares FP16 and AWQ across L4 concurrency levels and tracks five-minute throughput on a fanless M3.
- What you can do after
- Evaluate throughput results against latency, sustained operation, hardware constraints, and cost when choosing an inference deployment.
Read how Quantization and Fast Inference explains throughput →
Generative AI in Action
Throughput is the amount of work or number of requests a system processes during a specified period.
- How it is used
- Uses throughput to assess whether production LLM deployments can handle incoming demand.
- How deep it goes
- Defines throughput in relation to RPS and provides capacity-planning and monitoring guidance without a detailed throughput calculation.
- Examples
- Illustrates capacity allocation with a 240,000 TPM regional quota split across deployments.
- What you can do after
- Operate/configure deployment capacity by allocating TPM, managing rate limits, and testing throughput under load.
Rearchitecting LLMs
Throughput is the amount of model work or output that a system completes per unit of time.
- How it is used
- Measures inference efficiency when comparing base and pruned language models.
- How deep it goes
- Defines tokens per second and reports benchmark conditions and results, while deferring evaluation-system details and controlled attention-pruning speed analysis.
- Examples
- Compares base and pruned models on a Colab T4, including depth-pruning and batched width-pruning benchmarks.
- What you can do after
- Evaluate model inference performance against a baseline while accounting for quality trade-offs and measurement variability.
Apache Kafka in Action
Throughput is the amount of data or number of messages processed or transferred per unit of time.
- How it is used
- Measures Kafka message or data rates when benchmarking producers, planning capacity, and monitoring workloads.
- How deep it goes
- Works through a producer benchmark and configuration comparison, then gives partition and broker sizing guidance without prescribing a universal target rate.
- Examples
- Uses a million-record producer test, a batching comparison, and a replicated-storage calculation for broker planning.
- What you can do after
- Diagnose/debug with it by comparing measured rates and latency before adjusting producer settings or scaling partitions and brokers.
Architecting an Apache Iceberg Lakehouse
The maximum volume or rate of data that an ingestion system can process within a specified period.
- How it is used
- Guides ingestion and storage choices by comparing data volume capacity with freshness, latency, reliability, and operational demands.
- How deep it goes
- Defines the metric, compares workload patterns, and details batching, compaction, partitioning, commits, and metadata safeguards.
- Examples
- Uses dashboards, overnight ETL, machine-learning features, real-time analytics, and a 100-billion-event media pipeline.
- What you can do after
- Evaluate ingestion models and tools, then design high-volume Iceberg pipelines with controlled commits and metadata maintenance.
Read how Architecting an Apache Iceberg Lakehouse explains throughput →
LLMs in Production
Throughput is the rate at which a system completes requests or produces outputs over a period of time.
- How it is used
- Uses throughput to assess LLM service capacity and the responsiveness of streamed output.
- How deep it goes
- Defines QPS and TPS, outlines load-testing steps, and explains batching’s latency trade-off.
- Examples
- Illustrates throughput with a Locust streaming test and an LLM running on a Raspberry Pi.
- What you can do after
- Evaluate service capacity and streaming responsiveness using QPS, TPS, latency, and time to first token.
Kafka Streams in Action, Second Edition
Throughput is the amount of data or number of records processed or transferred during a unit of time.
- How it is used
- Optimizes Kafka record processing by distributing work across partitions, consumers, tasks, and threads.
- How deep it goes
- Explains partition and concurrency scaling, resource tradeoffs, and key skew, but does not quantify throughput with a measurement formula.
- Examples
- Uses a ten-partition Kafka Streams repartition and a Spring Kafka loan pipeline with three input and two output partitions.
- What you can do after
- Operate/configure Kafka applications by sizing partitions, setting listener concurrency, repartitioning streams, and checking key balance.
Read how Kafka Streams in Action, Second Edition explains throughput →
PostgreSQL Mistakes and How to Avoid Them
TPS is the number of transactions a database system completes per second.
- How it is used
- Measures PostgreSQL benchmark capacity while evaluating memory allocation and connection concurrency.
- How deep it goes
- Interprets repeated pgbench measurements through TPS, latency, client counts, and overload behavior rather than deriving the metric internally.
- Examples
- Compares 100-client memory configurations, a failed 1,000-client stress test, and a 16-to-2,000-client pgbench sweep.
- What you can do after
- Evaluate/understand benchmark throughput alongside latency to identify underutilization, overload, and unsafe configuration changes.
Read how PostgreSQL Mistakes and How to Avoid Them explains throughput →
Effective Conversational AI
Throughput is the amount of work or number of requests a system processes per unit of time.
- How it is used
- Uses throughput to judge whether RAG configurations can handle high query volumes.
- How deep it goes
- Measures throughput in queries per second and compares it alongside accuracy, latency, and resource use.
- Examples
- Compares PharmaBot configurations at 20, 18, and 16 queries per second alongside accuracy, latency, and resource-use metrics.
- What you can do after
- Evaluate RAG configurations by balancing query capacity against response quality, latency, and resource utilization.
Vibe Engineering
Throughput is the rate at which a system produces outputs, measured here as generated tokens per second.
- How it is used
- Guides capacity decisions for local inference, service optimization, and concurrent agent or build workloads.
- How deep it goes
- Quantifies tokens per second, request-weighted processing, and parallel build effects without developing a general throughput formula.
- Examples
- Uses token-rate comparisons, request-frequency calculations, and parallel module-build timings as concrete performance evidence.
- What you can do after
- Evaluate hardware, model, endpoint, or module choices against workload demand and expected parallel capacity.
Kubernetes for Developers
Requests per second is a throughput metric measuring the number of requests received or processed during one second.
- How it is used
- Guides capacity planning and horizontal pod autoscaling for HTTP workloads.
- How deep it goes
- Explains measurement, replica-capacity calculation, Ingress requirements, and external-metric configuration without detailing monitoring-provider internals.
- Examples
- Uses a timeserver Deployment, NodePort Service, Ingress, ApacheBench traffic, and an external request-count HPA.
- What you can do after
- Operate/configure Kubernetes autoscaling with an external RPS metric and evaluate replica changes under generated HTTP load.
Machine Learning System Design
Throughput is the amount of work or number of results processed by a system per unit of time.
- How it is used
- Frames inference requirements around the volume of predictions processed over time, especially for large batch workloads.
- How deep it goes
- Defines throughput, contrasts it with latency, and connects it to batching, scalability, tradeoffs, and operational monitoring.
- Examples
- Uses batch demand forecasting with AWS Batch, Docker, EC2, S3, scheduled jobs, and rows-processed monitoring.
- What you can do after
- Operate/configure batch inference by prioritizing processing capacity, selecting scalable infrastructure, and monitoring workload performance.
Read how Machine Learning System Design explains throughput →
Kafka for Architects
Throughput is the rate at which a system processes or transfers data during a specified period.
- How it is used
- Guides Kafka capacity planning, configuration, cost evaluation, and operational monitoring for event-processing systems.
- How deep it goes
- Defines volume, peak rate, traffic pattern, and burst recovery, then links them to settings without detailing implementation procedures.
- Examples
- Uses streaming-platform events, small e-commerce traffic, and a 300,000-message-per-day burst profile.
- What you can do after
- Evaluate throughput requirements and select configuration and monitoring priorities for Kafka-based systems.
JavaScript in Depth
The amount of useful data or work a system completes per unit of time.
- How it is used
- Uses throughput to compare stream models and diagnose performance problems alongside latency, memory, and allocation behavior.
- How deep it goes
- Explains controlled measurement, allocation costs, backpressure, buffering, and conversion overhead without deriving a general performance model.
- Examples
- Compares 4 KB chunks across stream models and examines fast-producer, slow-consumer memory behavior and repeated stream conversion.
- What you can do after
- Diagnose performance bottlenecks by measuring comparable workloads and interpreting throughput with allocation, memory, latency, and garbage-collection signals.
Build High Value Software Systems
The rate at which completed work passes through a production or delivery system.
- How it is used
- Diagnoses architectural components that constrain request throughput and guides where capacity improvements belong.
- How deep it goes
- Applies five constraint-management steps, from identifying a shared database bottleneck through increasing capacity and finding the next constraint.
- Examples
- Compares food-delivery service latencies and mitigations for a shared PostgreSQL bottleneck.
- What you can do after
- Diagnose a limiting component by comparing peak response times, then prioritize exploiting or elevating that constraint.
Read how Build High Value Software Systems explains throughput →
Domain-Specific Small Language Models
The rate at which a processor can perform floating-point operations over time.
- How it is used
- Uses GPU throughput and bandwidth to analyze inference speed, accuracy, compute cost, and optimization choices.
- How deep it goes
- Explains ops-per-byte bottleneck analysis and generation-time estimates, while noting simplifying assumptions in the timing model.
- Examples
- Uses GPT-Neo on an NVIDIA Tesla T4 to compare hardware specifications, estimate generation time, and measure observed inference duration.
- What you can do after
- Diagnose whether inference is memory- or compute-bound and estimate generation time from model size, precision, bandwidth, and prefill cost.
Read how Domain-Specific Small Language Models explains throughput →
Grokking GPU Programming
Throughput is the amount of work completed per unit of time or over an overall execution period.
- How it is used
- Positions throughput as the GPU's design goal in the CPU–GPU partnership.
- How deep it goes
- Defines throughput and latency, then explains how GPU cores and memory bandwidth support high throughput.
- Examples
- Uses image filtering and matrix math as examples of GPU-suited work.
- What you can do after
- Evaluate workload suitability for GPUs by checking for abundant independent work and regular memory access.
Transformers in Action
Throughput is the amount of inference work completed per unit of time, such as generated tokens or served requests per second.
- How it is used
- Frames inference optimization around reducing latency and maximizing generated-token or request throughput during deployment.
- How deep it goes
- Explains throughput through KV caching, memory growth, scheduling, layouts, tiling, and coalesced GPU access, but stops short of a throughput formula.
- Examples
- Demonstrates cache-enabled generation benchmarking, vLLM serving, tiled matrix multiplication, and row-major versus column-major layouts.
- What you can do after
- Diagnose throughput bottlenecks by comparing KV-cache behavior and examining GPU memory layout, bandwidth, tiling, and access coalescing.
Starting Data Analytics with Generative AI and Python
Throughput is the rate at which a system processes data, commonly expressed as transactions or records per unit of time.
- How it is used
- Measures data-processing capacity when evaluating performance, access patterns, workload baselines, and future expansion.
- How deep it goes
- Defines the rate, gives its division formula, and explains practical influences without detailing implementation internals.
- Examples
- Compares execution time and throughput for a bootstrapping function across DataFrame sizes.
- What you can do after
- Evaluate processing capacity and estimate resource requirements as the amount of data increases.
Read how Starting Data Analytics with Generative AI and Python explains throughput →
GPU Programming with Triton
Throughput is the amount of computation or data processing completed per unit of time.
- How it is used
- Frames GPU performance analysis around the rates of computation and data movement.
- How deep it goes
- Defines compute and memory ceilings, then classifies kernels by arithmetic intensity and distance from those ceilings.
- Examples
- Uses H100 peak-rate estimates, representative roofline kernels, and matrix multiplication in transformer workloads.
- What you can do after
- Evaluate whether a kernel is memory-bound or compute-bound and identify its distance from the corresponding performance ceiling.
Refactoring to Rust
Throughput is the amount of work a system completes per unit of time.
- How it is used
- Uses throughput to identify performance limits and motivate refactoring or horizontal and vertical scaling.
- How deep it goes
- Explains bottlenecks and scaling strategies, but stops short of measuring throughput or detailing optimization techniques.
- Examples
- Uses concurrent Python processes, shared image output, and a Mandelbrot loop to illustrate scaling behavior.
- What you can do after
- Evaluate whether a bottleneck or scaling strategy is limiting system capacity and recognize unsafe multi-process behavior.
AI Model Evaluation
Tokens per second is the rate at which a generative model produces output tokens after generation begins.
- How it is used
- Tracks how quickly generated output reaches users after generation starts, complementing first-token and completion timing.
- How deep it goes
- Defines output rate after generation begins and distinguishes it from initial wait and completion time.
- Examples
- Uses chat and streaming interfaces to frame visible output flow after generation starts.
- What you can do after
- Evaluate response flow by pairing tokens per second with first-token and completion timing.
Amazon Web Services in Action, Third Edition: An in-depth guide to AWS
Throughput is the rate at which a storage service processes read or write operations, commonly measured per second.
- How it is used
- Uses throughput to size S3 object-key layouts, select cache sharding, and scale EC2 workloads against network demand.
- How deep it goes
- Explains per-prefix limits, naming effects, cache sharding, and proportional scaling, but stops short of general performance measurement methods.
- Examples
- Uses archive year prefixes, Redis shard counts, and EC2 network-byte metrics as concrete throughput examples.
- What you can do after
- Apply throughput reasoning by redesigning S3 prefixes, choosing cache sharding, or configuring proportional autoscaling metrics.
CUDA for LLMs
Throughput is the amount of work a system completes per unit of time.
- How it is used
- Measures performance for GPU deep-learning workloads, including token generation, quantized inference, and distributed execution.
- How deep it goes
- Defines throughput against latency and connects the metric to token rates, request rates, and batch pipelines.
- Examples
- Uses token and request rates, quantized inference measurements, PyTorch–CUDA comparisons, and multi-GPU batch pipelines.
- What you can do after
- Evaluate end-to-end performance by measuring throughput alongside latency, memory use, and output correctness.
Microsoft Azure in Action
Throughput is the amount of data or request processing successfully handled by a system per unit of time.
- How it is used
- Uses throughput to size Cosmos DB capacity for ordinary workloads and traffic spikes.
- How deep it goes
- Explains request-unit configuration and autoscaling limits, but stops short of calculating workload-specific capacity.
- Examples
- Uses a holiday-traffic scenario, a 400-request-unit container, and a 10,000-request-unit autoscale ceiling.
- What you can do after
- Operate/configure Cosmos DB throughput with Azure CLI, including fixed capacity and autoscaling limits.
Latency
Throughput is the rate at which a system processes or transfers units of work or data over time.
- How it is used
- Measures how much data an application transfers or how many requests it processes per time unit.
- How deep it goes
- Explains measurement, bandwidth limits, concurrency, pipelining, bottlenecks, and latency trade-offs with a worked example.
- Examples
- Uses network transfer rates, request processing, and serial versus pipelined laundry processing.
- What you can do after
- Evaluate application performance, compare capacity with delivered work, and reason about concurrency and pipeline bottlenecks.
Decision Engineering
Throughput is the amount of work or number of operations a system completes per unit of time.
- How it is used
- Quantifies processing capacity and helps distinguish system handling from the amount and kind of demanded work.
- How deep it goes
- Defines throughput as requests or data volume per second and distinguishes it from load and scalability.
- Examples
- Compares storefront and backoffice search, and gives throughput tiers for storefront, B2B, backoffice, and background processing.
- What you can do after
- Evaluate requirements by comparing component demand and peak scenarios rather than relying on one request-rate figure.
Automating API Delivery
Throughput is the amount of work or number of requests processed by a system within a specified time interval.
- How it is used
- Measures API request activity as part of RED monitoring for microservice performance.
- How deep it goes
- Defines requests processed per time window and relates measurement periods to usage trends.
- Examples
- Uses per-second, per-minute, per-week, and per-month request measurements.
- What you can do after
- Evaluate API usage trends by measuring requests across selected time windows.
Delivering Machine Learning Projects: From design to deployment
Throughput is the amount of work or data that a system processes within a specified time interval.
- How it is used
- Defines a production nonfunctional requirement for the number of model executions or service requests completed within a timeframe.
- How deep it goes
- Explains throughput, contrasts it with latency, and covers cold starts, parallelism, bottlenecks, and production-like measurement.
- Examples
- Uses cold starts, ten parallel models, production-mirroring test harnesses, and service-request targets as evaluation examples.
- What you can do after
- Evaluate throughput against end-to-end targets using production-like hardware, workloads, and serving conditions.
Read how Delivering Machine Learning Projects: From design to deployment explains throughput →
Grokking Concurrency
Throughput is the number of tasks a system completes or handles during a given period of time.
- How it is used
- Measures system performance by counting how much work completes during a period.
- How deep it goes
- Defines throughput and contrasts it with latency, stopping short of implementation or optimization procedures.
- Examples
- Uses motorcycle and bus transportation to contrast individual travel speed with people transported per hour.
- What you can do after
- Evaluate/understand performance by distinguishing completed work per time from the duration of one task.
Fast by Default
Throughput is the number of requests or other work items a service handles per unit of time.
- How it is used
- Uses requests per second to plan backend capacity and assess whether services can handle peak user traffic.
- How deep it goes
- Defines throughput, contrasts it with response speed, and specifies peak-load targets with surge headroom.
- Examples
- Uses a service handling 500 requests per second at peak and contrasts fast responses with insufficient request capacity.
- What you can do after
- Evaluate backend capacity by measuring requests per second against peak traffic, response speed, errors, and planned headroom.
ScyllaDB in Action
Throughput is the rate at which a system completes operations or processes data over time.
- How it is used
- Guides ScyllaDB load testing and migration-rate decisions by showing how much traffic a database can handle.
- How deep it goes
- Compares operations-per-second results with latency during progressively heavier tests, without developing a formal capacity model.
- Examples
- Uses cassandra-stress results, partition and row rates, Grafana dashboards, and a source-to-destination migration scenario.
- What you can do after
- Diagnose database capacity with cassandra-stress and Grafana while balancing throughput against latency and degradation.
Also covered in
- Practices of the Python Pro
- Software Architecture Decision-MakingIn early access (MEAP)
- Go by Example
- Shipping Go: Develop, deliver, discuss, design, and go again
- gRPC Microservices in Go
- F# in Action: A Revised Edition of Get Programming with F#
- Learn Concurrent Programming with Go
- Python Concurrency with asyncio
- Machine Learning Platform Engineering
- LLM Evaluation and Alignment, The Foundational IdeasIn early access (MEAP)
- Quarkus in Action
- AI Agents for Offensive SecurityIn early access (MEAP)
What to know first
- performancePerformance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- meanThe mean is a measure of central location equal to the arithmetic average of numerical observations, or the probability-weighted average of a random variable’s possible values.
- batchingBatching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
Related concepts
- latencycontrasts with · Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
- memory bandwidthcontrasts with · Memory bandwidth is the rate at which data can be transferred between a memory system and a processor or other computing component.
- throughput retentionmeasured by · Throughput retention is the ratio of throughput over a later measurement interval to throughput over an initial interval, expressed as a percentage.
- token generationmeasures · Token generation is the iterative process by which a language model produces a sequence of output tokens, typically one token at a time.
- meancontrasts with · The mean is a measure of central location equal to the arithmetic average of numerical observations, or the probability-weighted average of a random variable’s possible values.
- parallelismproduced by · Parallelism is the execution of multiple independent tasks or computational operations at the same time rather than sequentially.
Questions about throughput
What is throughput?
Throughput is the rate at which a system processes or completes work, data, or requests over a specified period.
Which Manning books explain throughput?
Quantization and Fast Inference, Generative AI in Action, Rearchitecting LLMs, Apache Kafka in Action, Architecting an Apache Iceberg Lakehouse, LLMs in Production, Kafka Streams in Action, Second Edition, PostgreSQL Mistakes and How to Avoid Them and 24 more.
What should I know before learning throughput?
Manning books that teach throughput usually introduce performance, language model, mean and batching first.
What is the difference between throughput and latency?
Throughput is the rate at which a system processes or completes work, data, or requests over a specified period. Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
What is throughput also called?
Throughput is also called tokens per second.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.































