3 Kafka Clients and Message Production

 

This chapter covers

  • Connecting to Kafka
  • Configuring client applications
  • How to send messages using Kafka
  • Typical challenges faced by producer applications.

The conference room buzzed with anticipation as Max Sellington, Eva Catalyst, and Rob Routine convened to discuss the intricacies of developing client applications for Kafka integration.

MAX (leaning forward): Alright team, let's explore the realm of client applications. What are the requirements for our client teams, and how well-versed should they be in Kafka to effectively integrate with it? Additionally, what are the associated costs we need to consider?

EVA (nodding): Max, integrating with Kafka can be quite direct. Frameworks can simplify many of Kafka's complexities, letting teams concentrate on their specific needs. That said, mastering Kafka does demand time and commitment.

MAX: I'm not entirely convinced it's as easy as everyone claims it to be, Eva. Let's consider our concrete proof-of-concept, the Customer 360 project. How do we go about creating the client applications? Will they be structured as microservices? Is programming necessary, or can we rely on configuration? How dependent is this setup on Kafka?

3.1 Communicating with Kafka

3.1.1 Configuring clients

3.1.2 Connecting to Kafka

3.1.3 Serializing and deserializing data

3.1.4 Setting quotas

3.1.5 Setting up Customer360 project

3.1.6 Architecture Points: Initial Producer Configuration

3.2 Sending a message

3.2.1 Partitioning strategy

3.2.2 Partitioning strategy for Customer360

3.2.3 Acknowledgement strategies

3.2.4 Acknowledgment strategy for Customer360

3.2.5 Batches and timeouts

3.2.6 Configuring Producer for Customer360

3.2.7 Common producer issues

3.2.8 Architecture Points: advanced producer configuration

3.3 Online resources

3.4 Summary