9 Managing Kafka within the Enterprise
This chapter covers
- How Kafka handles configuration, leadership assignments, and state coordination
- Exploring Kafka deployment strategies, including on-premise, cloud-based, and hybrid solutions
- Best practices for authentication, authorization, encryption, and protecting data
The team gathered once again in the meeting room, ready for another round of discussion. Max arrived with a sense of urgency.
Max: Alright, team, how are things going? Don’t you think it’s time we stop playing around and finally move this prototype into some sort of environment? You know, make it visible, accessible, so other teams can start working with it?
Rob: You’re absolutely right, Max. And that’s exactly what we want to discuss today. But before we can do that, we need to talk about something important: the budget.
Max: The budget? Didn’t we already settle this? Start on-prem, get the brokers running, and keep it simple.
Eva: True, but brokers alone won’t be enough. We’ll also need a cluster of controllers to manage the metadata.
Max: Metadata? What are you talking about?
Rob: All the topics, configurations, partition details, and other operational data—Kafka needs to store that somewhere, and that’s the job of the controller cluster.
Max: Another cluster? Really? Look, databases also manage metadata, but they handle it silently, behind the scenes. Why does Kafka need this extra setup?