This chapter covers
- Kafka cluster management with KRaft and ZooKeeper
- Migrating from ZooKeeper to KRaft
- How clients connect to Kafka
In the first chapters, we got to know Kafka as a distributed system and addressed multiple times that coordination in such systems is always associated with challenges. These problems affect not only Kafka itself but also our consumers when using consumer groups. Fortunately, Kafka takes a lot of work off our hands, and we don’t have to worry about reliably distributing partitions to our consumers.
But how is Kafka actually managed? And what do we need cluster management for? The core challenge that cluster management systems solve is finding consensus in distributed systems. As long as we only have one broker, this is simple. We don’t have to reach an agreement with anyone; this broker is the leader for all partitions, always in sync, the group coordinator for all consumer groups, and so on. But how do we ensure a consistent state when adding more brokers? Who then distributes the partitions to the brokers, monitors the health status of the brokers, and removes brokers from the cluster if they don’t report? Who stores the current configuration of the cluster?