This chapter covers:
- Exploring the production write path and how it works
- Creating custom producer serializers, partitioners, and interceptors
- Examining various configuration options that we can apply to solve our company’s requirements
In the previous chapter, we started to look at some of the requirements that an organization could have in regards to their data. Some of those discussions that were made have practical impacts in how we need to send our data into Kafka. To enter into the world of Kafka, the portal gate is through the producer. Please take note that when we are looking at how things work below in detail, I will try to lean towards the defaults. Since the producer can be in different programming languages (different client implementations), the details provided will be specific to the Java client that we used in our previous examples. While some clients (C/C++, Python, and, Go) will likely have similar configuration, it is not guaranteed. Defects and features can also be unique per client.
After reading this chapter, we will be on our way to solving our requirements by producing data in a couple of different ways. The producer, despite its importance, is only one part of this system. It is true that some of the producer configuration options can be changed or set at the broker topic level. We will discover those options as we move further along. However, getting data into Kafka is our first concern in this chapter.