ZooKeeper

As explained in Kafka for Architects

Apache ZooKeeper is a distributed coordination service historically used by Kafka to manage cluster metadata and controller coordination.

ZooKeeper is a separate coordination cluster historically used by Kafka to store cluster metadata, monitor broker liveness, and coordinate the active controller. Kafka brokers manage topics, partitions, and client communication, while ZooKeeper provides the coordination layer. section 9.1.4

A ZooKeeper-managed Kafka deployment divides responsibilities across two cooperating systems:

• Metadata management: brokers send updates such as topic creation and partition assignments to the ZooKeeper leader, which replicates them to followers through the ZooKeeper Atomic Broadcast protocol. section 9.1.4
• Broker liveness: brokers send heartbeats; when a heartbeat times out, ZooKeeper marks the broker unavailable and partition leadership is reassigned. section 9.1.4
• Controller coordination: an active Kafka controller communicates with the ZooKeeper leader, loads metadata into memory, and distributes cluster-state updates to other brokers. section 9.1.4

Migration pressure

A Kafka cluster using ZooKeeper must account for an architectural transition: KRaft became experimental in Kafka 2.8.0, the default metadata system in 3.7.0, and Kafka 4.0 requires ZooKeeper to be removed. Upgrades therefore require compatibility checks across brokers, Kafka Connect, Schema Registry, REST Proxy, client libraries, the operating system, and the Java runtime. section 11.1

Figure 9.6 Various types of communication within a Kafka cluster using ZooKeeper

Use the architecture diagram to trace communication among ZooKeeper servers, Kafka brokers, the active controller, and clients. section 9.1.4

How Kafka for Architects covers it

How deep it goes

Explains the two-cluster architecture, ZAB replication, heartbeats, controller coordination, and migration away from ZooKeeper.

Examples

Uses topic creation, partition assignment, broker heartbeats, controller updates, and a Kafka-ZooKeeper communication diagram.

What it assumes

Assumes familiarity with Kafka brokers, topics, partitions, controllers, and supporting ecosystem components.

What you can do after

Evaluate ZooKeeper-based Kafka architecture and assess its removal and ecosystem compatibility during upgrades.

How it connects

Connects Kafka brokers with metadata management, broker liveness, active controllers, ZAB, and KRaft.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA ZooKeeper ensemble is a group of ZooKeeper nodes that collectively provide distributed coordination and metadata services.
• 

  liveBookmanning.com
  Quarkus in ActionApache ZooKeeper is a distributed coordination service that provides shared configuration, naming, synchronization, and metadata-management primitives.

Compare how 3 books explain ZooKeeper →

Related concepts

• Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• KRaftalternative to · KRaft is Apache Kafka’s Raft-based metadata-management mode, using an internal controller quorum instead of ZooKeeper.
• Zabused by · ZooKeeper Atomic Broadcast is a distributed coordination protocol that orders and replicates state changes consistently among ZooKeeper servers.
• Strimziused by · Strimzi is an open-source Kubernetes operator and set of resources for deploying and managing Apache Kafka clusters and related components.
• leader electionuses · Leader election is a distributed coordination process that selects one eligible participant to act as leader or coordinator.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.