3 Kafka clients and message production

Let’s shift from the broker internals to how clients talk to Kafka. Client configurations play a significant role in the data-transfer process to and from brokers, so we need to look at the options available to optimize application performance and reliability. First we want to understand the basics of producers: configuring clients, connecting to Kafka, and choosing serializers. Then we can focus on how messages are sent and confirmed, including partitioning, acknowledgments, batching, timeouts, and timestamps. Along the way, we’ll note some common pitfalls and how to avoid them. The goal here is to help you send data to Kafka reliably and efficiently, with settings you can justify in production. Then, in the next chapter, we’ll examine the intricacies of developing consumer applications.

3.1 Communicating with Kafka

3.1.1 How producers send messages to brokers

3.1.2 Configuring clients

3.1.3 Connecting to Kafka

3.1.4 Serializing and deserializing data

3.1.5 Setting quotas

3.1.6 Field notes: Setting up the Customer 360 operational data store

3.2 Sending a message

3.2.1 Partitioning strategy

3.2.2 Field notes: Partitioning strategy for the Customer 360 ODS

3.2.3 Acknowledgment strategies

3.2.4 Field notes: Implementing an acknowledgment strategy

3.2.5 Batches and timeouts

3.2.6 Common producer challenges

3.3 Online resources

Summary