Throughput measures how much data a system processes or transfers over a period. In Kafka architectures, it is a design and operating concern: high-throughput systems handle large volumes of small messages at high rates while preserving acceptable latency. section 5.2.7
Treat throughput as a nonfunctional requirement rather than an isolated performance number. Expected volume, peak rate, traffic pattern, and burst recovery determine how producers, brokers, and consumers should be configured. Message size, batching, compression, partitioning, buffer capacity, and memory usage all influence the achievable rate and operating cost. section 10.1.4
Architectural decisions should account for both capacity and operational consequences:
- Size for normal traffic and bursts, including the time allowed to process accumulated backlog. section 10.1.4
- Balance compression and batching gains against CPU overhead, memory needs, message size limits, and network usage. section 10.1.4
- Monitor throughput with lag, resource usage, logging, and tracing; chained processors can hide bottlenecks and errors. section 8.5.6
- Question whether Kafka’s operational complexity is justified when traffic is small and real-time processing is not important. section 5.2.7
Burst-oriented requirement
A streaming system might expect 300,000 messages per day, receive 50 events per second during business hours, reach 100 events per second in bursts, and need to clear the resulting backlog within five minutes. section 10.1.4
Table 10.2 Defining throughput requirements
| Throughput requirement |
Example |
| Expected volume |
300,000 messages per day |
| Peak rate |
50 events/sec during business hours; up to 100/sec during bursts |
| Pattern |
Real-time events, with high activity from 9 AM to 5 PM on weekdays |
| Burst handling |
System must buffer and process backlog within 5 minutes during peak load |
Use the throughput-requirements table to record expected volume, peak rate, traffic pattern, and burst-handling targets before selecting Kafka settings. section 10.1.4
