3 Kafka Clients and Message Production
This chapter covers
- Connecting to Kafka
- Configuring client applications
- How to send messages using Kafka
- Typical challenges faced by producer applications.
Here we shift from broker internals to how clients talk to Kafka. We focus on producer basics: configuring clients, connecting, and choosing serializers. Then we see how messages are sent and confirmed, including partitioning, acknowledgments, batching, timeouts, and timestamps. Along the way, we note common pitfalls and how to avoid them. The goal here is to help you send data to Kafka reliably and efficiently, with settings you can justify in production.
3.1 Field notes: Patterns and pitfalls for Kafka clients
The conference room buzzed with anticipation as Max Sellington, Eva Catalyst, and Rob Routine convened to discuss the intricacies of developing client applications for Kafka integration.
Max (leaning forward): Alright team, let's explore the realm of client applications. What are the requirements for our client teams, and how well-versed should they be in Kafka to effectively integrate with it? Additionally, what are the associated costs we need to consider?
Eva (nodding): Max, integrating with Kafka can be quite direct. Frameworks can simplify many of Kafka's complexities, letting teams concentrate on their specific needs. That said, mastering Kafka does demand time and commitment.