6 Performance

 

This chapter covers

  • Topic settings to increase performance
  • Determining the best number of partitions for a topic
  • Kafka broker settings that influence performance
  • Producer and consumer performance tuning

In the previous chapter, we saw how to reliably produce messages with Kafka. In this chapter, we’ll address the unfinished business of optimizing Kafka. How do we achieve the promised performance with Kafka?

Similar to reliability, when we talk about performance, we first need to clarify what this means to us. We can define performance in different ways. Are we concerned with bandwidth? The bandwidth that the system offers us is often crucial. We measure bandwidth in bytes per second. How many kilobytes/megabytes/gigabytes per second of data throughput do we achieve with our system? When we only talk about bandwidth, we often neglect latency, although it’s usually of greater importance to us.

6.1 Configuring topics for performance

6.1.1 Scaling and load balancing

6.1.2 Determining how many partitions are needed

6.1.3 Changing the number of partitions

6.2 Producer performance

6.2.1 Producer configuration

6.2.2 Producer performance test

6.3 Broker configuration and optimization

6.3.1 Optimizing brokers

6.3.2 Determining broker count and sizing

6.4 Consumer performance

6.4.1 Consumer configuration

6.4.2 Consumer performance test

Summary