ZooKeeper

As explained in Apache Kafka in Action

A ZooKeeper ensemble is a group of ZooKeeper nodes that collectively provide distributed coordination and metadata services.

In a Kafka cluster with several brokers, someone must keep track of partition assignments, leaders, and the broker acting as controller. Older Kafka deployments use a separate ZooKeeper ensemble for this coordination. The ensemble stores cluster metadata and helps the brokers agree on a consistent state; it does not store the messages in Kafka topics. section 4.3.1 section 7.2

ZooKeeper exposes a hierarchical store of ZNodes that clients can create, read, and modify. Its key property is agreement: when clients compete to change a ZNode, they observe one consistent result. ZooKeeper Atomic Broadcast (ZAB) reaches consensus through agreement by a majority of ZooKeeper nodes. An ensemble usually has three or five nodes. section 7.2 section 4.3.1

Electing a controller

Suppose the controller fails. If the controller ZNode is absent, brokers try to write their own IDs to it. One succeeds and becomes the new controller. ZooKeeper also tracks connected clients, helping Kafka detect failed brokers. The controller can then reassign partitions and select new leaders to keep the cluster available. section 7.2 section 4.3.1

Figure 7.2 Before KRaft, a Kafka cluster consisted of multiple brokers and a ZooKeeper ensemble consisting usually of three or five ZooKeeper nodes. The controller was simply one of the brokers. If the controller failed, another broker took over these tasks.

The diagram separates the Kafka brokers from their ZooKeeper ensemble and shows that the controller is a broker, not a ZooKeeper node. section 7.2

For an existing ZooKeeper-based cluster, two operational concerns matter:

• Secure the connection. ZooKeeper holds metadata, ACLs, and potentially SASL-SCRAM credentials; traffic is unencrypted by default. Kafka supports TLS connections to ZooKeeper from version 2.5, and mutual TLS is recommended. Keep clients from accessing ZooKeeper directly, and watch encryption-related latency because ZooKeeper is latency-sensitive. section 13.2.6
• Plan a move to KRaft. ZooKeeper is a separate system to maintain, and its general-purpose metadata model can become a bottleneck. KRaft brings coordination into Kafka; ZooKeeper-based cluster management is slated for removal in Kafka 4.0. Migration calls for provisioning KRaft controllers, configuring migration mode, verifying metadata transfer, and transitioning brokers. section 4.3.1 section 7.2 section 7.5

How Apache Kafka in Action covers it

How deep it goes

Explains ZNodes, competing controller writes, failure detection, and majority-based ZAB consensus without detailing the protocol’s internals.

Examples

Uses brokers competing to write a controller ZNode and a diagram of a Kafka cluster with a three-node ZooKeeper ensemble.

What it assumes

Assumes familiarity with Kafka brokers and partitions while introducing ZooKeeper’s ZNodes and consensus role.

What you can do after

Operate/configure it by securing broker connections and planning migration of an existing Kafka cluster to KRaft.

How it connects

Links ZooKeeper to Kafka brokers, controllers, partition leadership, access control lists, consensus, and KRaft.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka for ArchitectsApache ZooKeeper is a distributed coordination service historically used by Kafka to manage cluster metadata and controller coordination.
• 

  liveBookmanning.com
  Quarkus in ActionApache ZooKeeper is a distributed coordination service that provides shared configuration, naming, synchronization, and metadata-management primitives.

Compare how 3 books explain ZooKeeper →

Related concepts

• Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• KRaftalternative to · KRaft is Apache Kafka’s Raft-based metadata-management mode, using an internal controller quorum instead of ZooKeeper.
• Zabused by · ZooKeeper Atomic Broadcast is a distributed coordination protocol that orders and replicates state changes consistently among ZooKeeper servers.
• Strimziused by · Strimzi is an open-source Kubernetes operator and set of resources for deploying and managing Apache Kafka clusters and related components.
• leader electionuses · Leader election is a distributed coordination process that selects one eligible participant to act as leader or coordinator.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.