Throughput measures how much data or how many records a system processes or transfers during a unit of time. In Kafka applications, partition count, consumer processing time, task count, and thread count determine how much work can proceed in parallel. section 2.4.2 section 7.2.5
A practical Kafka strategy is to increase partitions so consumers or stream tasks can process records concurrently. More partitions can improve throughput, but they also increase TCP connections, open file handles, and broker resource usage. Heavyweight consumer processing can still limit performance. section 2.4.2 section 12.2.2
Tune throughput by checking the following:
- Choose enough partitions for high-throughput periods without exceeding broker limits. section 2.4.2
- Use repartitioning when increasing task and thread capacity without changing a shared source topic. section 7.2.5
- Verify that keys are distributed evenly; concentrated keys can leave one partition and task carrying most of the workload. section 7.2.5
- Match consumer concurrency to the available partitions so each partition can have a dedicated consumer. section 12.2.2
Scaling a loan-application pipeline
Suppose a loan application input topic needs three partitions for expected traffic, while approval and rejection topics need two partitions each because each path receives roughly half the applications. A Spring Kafka listener can set its container concurrency to the partition count, and the application can record incoming keys and timestamps while tracking produced offsets and timestamps. section 12.2.2
