A Kafka system may need to move more data without making users wait longer. Throughput measures how much it handles per second: bytes per second for data bandwidth, or records per second for messages. It is not the same as latency. High throughput does not guarantee a fast response, and capacity must be weighed against resource use. section 6.0 section 6.5
Start by measuring, rather than changing Kafka settings on a guess. The producer performance tool can send a fixed number of records of a chosen size to an existing topic. Its output reports both records per second and MB per second alongside latency. Run several repetitions before treating a result as representative; a tool-only test does not replace an end-to-end test with real producers. section 6.2.2
$ kafka-producer-perf-test.sh \
--topic performance-test \
--num-records 1000000 \
--record-size 10000 \
--throughput -1 \
--producer-props bootstrap.servers=localhost:9092
This command sends one million 10,000-byte records without a throughput limit. It requires the performance-test topic and a reachable bootstrap server, and can write multiple gigabytes to disk. section 6.2.2
Read the result, then change one configuration
One run reported about 12,317 records/s, 117.47 MB/s, and 165 ms average latency. With a larger batch.size and linger.ms, the illustrated run reported about 45,880 records/s and 437.55 MB/s. These are test results, not promised rates: batching generally improves throughput but can also increase latency. section 6.2.2 section 6.5
$ kafka-producer-perf-test.sh \
--topic performance-test \
--num-records 1000000 \
--record-size 10000 \
--throughput -1 \
--producer-props bootstrap.servers=localhost:9092 \
batch.size=100000 linger.ms=100
...
1000000 records sent, 45879.977978 records/sec (437.55 MB/sec),
64.61 ms avg latency, 279.00 ms max latency, 58 ms 50th,
101 ms 95th, 126 ms 99th, 178 ms 99.9th.
Compare this command and its output with the starting test to see the changed batch.size and linger.ms settings and the reported throughput and latency. section 6.2.2
If throughput is still insufficient, locate the constraint before scaling:
- Partitions distribute work across consumers. More partitions can support more parallel consumers, but consume client and broker resources and complicate recovery. A starting point of 12 can be increased when demand warrants it; partition counts cannot be decreased. section 6.1.2 section 6.5
- Broker sizing must account for expected data rates, network capacity, storage duration, and replication. Measure before tuning an already performant default configuration. section 6.3.2
- Producer acknowledgments and compression affect performance, reliability, bandwidth, or CPU use. Do not choose a higher rate without checking the accompanying trade-off. section 6.2.2 section 6.5
Keep measuring after deployment. In cloud environments, monitor network throughput alongside latency and autoscaling during load spikes; for managed Kafka, message throughput remains an application-level metric relevant to service targets. section 15.5.3 section 15.6
