A hybrid Kafka deployment keeps core operational systems on-premises while using cloud services for analytics, machine learning, or business intelligence. The architectural question is how to move the data those services need without moving every operational workload to the cloud. section 9.2.2
A common design uses two Kafka clusters, one in each environment:
- Replicate selected data from the on-premises cluster to the cloud cluster with Apache MirrorMaker or Confluent Replicator. Data sent to the cloud can be anonymized or transformed for cloud applications. section 9.2.2
- Choose replicator placement with the network in mind. Placement near the target cluster makes producer writes local and acknowledgments more predictable, but source fetches still cross the WAN and can increase replication lag. section 9.2.2
- Budget for infrastructure, storage, networking, and operations in both environments. Weigh those costs against latency, fault tolerance, scalability, and compliance requirements. section 9.2.2 section 9.2
Figure 9.7 Replicating data from a local Kafka cluster to a Kafka cluster in the cloud
The diagram shows replication from a local Kafka cluster to a separate cloud Kafka cluster. section 9.2.2
Customer 360 operational data store
For a Customer 360 ODS, core systems that generate data can remain on-premises while managed cloud services support analytics. This preserves access to the core systems' internals while leaving room to expand cloud integration later. section 9.2.3
Hybrid does not remove governance decisions. Assess which data may leave the premises: cloud deployments can create data-locality and compliance risks involving metadata, logs, or data stored outside an organization's control. A hybrid design must also account for the network and security constraints on replication. section 9.2.1 section 9.2.2
