10 Organizing a Kafka Project
This chapter covers
- Defining project requirements across environment setup, non-functional requirements, infrastructure sizing, and resource quotas.
- Maintaining Kafka cluster using tools, GitOps, and the Kafka Admin API.
- Testing Kafka applications.
When a team considers adopting a new technology for a project, they often focus intensely on technical considerations—prototyping, assessing feasibility, estimating costs, and reviewing infrastructure demands. Yet one critical factor frequently underestimated is how the technology aligns with the organization’s project workflow. Key questions arise: What specific requirements must be met? Who owns gathering them? How will deliverables be tracked? Can the technology seamlessly integrate into the CI/CD pipeline? Are QA teams equipped to test for integration risks?
In this chapter, we’ll address these challenges and present best practices for structuring a Kafka project to ensure smooth adoption, governance, and long-term success.
10.1 Defining Kafka Project Requirements
Max sighed as he sat down, shaking his head.
Max: Alright, team. Our project is starting to attract a lot of attention. Every time I go for lunch, someone stops me and asks if they can also use Kafka for their use case. It’s like we’ve created a new buzzword in the company.
Rob: Well, before we let everyone jump on board, they need to understand whether Kafka is actually suitable for their use case.