chapter six

6 Performance

 

This chapter covers

  • Topic settings to increase performance
  • Determining the best number of partitions for a topic
  • Kafka Broker settings which influence performance
  • Producer and consumer performance tuning

In the last chapter, we saw how we can reliably produce messages with Kafka. In this chapter, we'll address the unfinished business of optimizing Kafka. How do we achieve the promised performance with Kafka?

Similar to reliability, when we talk about performance, we first need to clarify what this means to us. We can define performance in different ways. Are we concerned with bandwidth? The bandwidth that the system offers us is often very crucial. We measure bandwidth in bytes per second. How many KB/MB/GB per second of data throughput do we achieve with our system? When we only talk about bandwidth, we often neglect latency, although it is usually of greater importance to us.

6.1 Configuring topics for performance

6.1.1 Scaling and load balancing

6.1.2 How many partitions should we have?

6.1.3 Changing the number of partitions

6.2 Producer performance

6.2.1 Producer configuration

6.2.2 Producer performance test

6.3 Broker configuration and optimization

6.3.1 Optimizing brokers

6.3.2 Determining broker count and sizing

6.4 Consumer performance

6.4.1 Consumer configuration

6.4.2 Consumer performance test

6.5 Summary