Self-hosting Kafka means installing and operating the cluster on premises rather than buying it as a managed cloud service. It gives the organization control over brokers and configuration, but makes its team responsible for the infrastructure throughout its lifecycle. section 1.6.2
Before choosing that control, account for:
- Hardware procurement, infrastructure setup, physical security, monitoring, maintenance, and troubleshooting. section 1.6.2
- Broker configuration, authentication, observability, and disaster recovery—work that needs people with the right operational skills. section 9.2.1
- The alternative’s limits: a managed service reduces administration but may restrict broker settings, version changes, and available ecosystem components. section 1.6.2
A local Customer 360 cluster
A team that needs control starts with three local Kafka brokers. It records a replication factor of 3 and min.insync.replicas of 2 for its topics, making redundancy an explicit design decision alongside the hosting choice. section 2.3.3
asyncapi: 3.0.0
info:
title: Customer 360 Project
version: 1.0.0
description: >
This project aims to aggregate data from various sources, providing a
comprehensive view of customer information.
channels:
customerProfile:
description: Storing profile information of customers.
address: CUSTOMER_PROFILES
bindings:
kafka:
partitions: 9 #1
replicas: 3
topicConfiguration:
min.insync.replicas: 2 #2
transactions:
description: Information about customer transactions.
address: TRANSACTIONS
bindings:
kafka:
partitions: 9
replicas: 3
topicConfiguration:
min.insync.replicas: 2
The AsyncAPI listing records the topics’ partitions, replicas, and minimum in-sync replicas so the local cluster design can be reviewed and refined. section 2.3.3
Self-hosting is a fit decision, not a default. Compare it with cloud and hybrid deployment against cost, scalability, latency, security, compliance, and the team’s ability to keep the cluster operable. section 1.6 section 9.2
