9 Managing Kafka within the Enterprise

 

This chapter covers

  • How Kafka handles configuration, leadership assignments, and state coordination
  • Exploring Kafka deployment strategies, including on-premise, cloud-based, and hybrid solutions
  • Best practices for authentication, authorization, encryption, and protecting data

The team gathered once again in the meeting room, ready for another round of discussion. Max arrived with a sense of urgency.

Max: Alright, team, how are things going? Don’t you think it’s time we stop playing around and finally move this prototype into some sort of environment? You know, make it visible, accessible, so other teams can start working with it?

Rob: You’re absolutely right, Max. And that’s exactly what we want to discuss today. But before we can do that, we need to talk about something important: the budget.

Max: The budget? Didn’t we already settle this? Start on-prem, get the brokers running, and keep it simple.

Eva: True, but brokers alone won’t be enough. We’ll also need a cluster of controllers to manage the metadata.

Max: Metadata? What are you talking about?

Rob: All the topics, configurations, partition details, and other operational data—Kafka needs to store that somewhere, and that’s the job of the controller cluster.

Max: Another cluster? Really? Look, databases also manage metadata, but they handle it silently, behind the scenes. Why does Kafka need this extra setup?

9.1 Managing metadata

9.1.1 Introducing KRaft controllers

9.1.2 Example of cluster configuration

9.1.3 Failover scenarios

9.1.4 Using Zookeeper

9.2 Choosing a deployment solution

9.2.1 Choosing between on-premises and cloud Kafka deployment

9.2.2 Hybrid Approach

9.2.3 Choosing the Right Deployment for a Customer 360 Project

9.3 Protecting Kafka

9.3.1 Kafka security overview

9.3.2 Encrypting using TLS

9.3.3 Authentication

9.3.4 Authorization

9.3.5 Protecting data at rest

9.3.6 Enabling security in the Customer360 project

9.4 Online resources

9.5 Summary