This chapter covers
- Handling configuration, leadership assignments, and state coordination
- Exploring Kafka deployment strategies: on-premises, cloud-based, and hybrid solutions
- Best practices for authentication, authorization, encryption, and protecting data
When you get close to launching a prototype into production, it’s time to think about the concrete operational details. How will your system manage metadata and coordination so you can size and place controllers, anticipate behavior during incidents, and plan migrations from older architectures (such as ZooKeeper) to KRaft? Here is where another key actor in the Kafka ecosystem, the controller quorum, comes into play. Controllers manage metadata and ensure that clusters remain operational; they use quorum-based decisions to maintain fault tolerance. All of this needs to be configured.
Then there is the question of deployment. On-premises, cloud, and hybrid deployment models are all viable, and it’s important to compare them to match latency, cost, and operability.
Finally, you’ll want to make security actionable from end to end, including authentication (mTLS/SASL), authorization (ACLs), encryption in transit (TLS), data-at-rest protection, and even optional end-to-end encryption. We’ll look at all these operational details in this chapter.