A dual-cluster architecture operates Kafka clusters in separate environments—commonly on premises and in the cloud—and synchronizes data between them. It supports hybrid deployments that combine on-premises control and low-latency processing with cloud scalability, flexibility, and disaster-recovery options. section 9.2.2 section 9.5
Replication tools connect the clusters while introducing operational trade-offs:
- Replication tools consume from the source cluster and produce to the target cluster, often using Kafka Connect. section 9.2.2
- Kafka Connect-based replication can rename topics, fan in or fan out data, transform records, and support exactly-once delivery guarantees. section 9.2.2
- Deploying the replicator near the target cluster keeps producer writes local, while source fetches cross the WAN and may add replication lag. section 9.2.2
- Operating two clusters increases infrastructure, storage, networking, synchronization overhead, and total operational complexity. section 9.2.2 section 9.5
On-premises and cloud replication
An organization keeps an on-premises Kafka cluster for reliable, low-latency local clients and replicates data to a cloud Kafka cluster for scalable services, analytics, resilience, and disaster recovery. section 9.2.2 section 9.5
Figure 9.7 Replicating data from a local Kafka cluster to a Kafka cluster in the cloud
Use the architecture diagram to trace replication from a local Kafka cluster to a cloud Kafka cluster. section 9.2.2
