A producer is an application that publishes messages to Kafka brokers. Instead of sending an update separately to every interested service, it publishes once so independent consumers can react. The producer typically does not know which consumers will read the message. An application can also be a consumer if it both reads and publishes messages. section 1.3.1 section 1.4.1
One address change, several uses
When a customer changes an address, a customer service stores the new value and publishes a notification. Billing, reporting, and order services can each subscribe and process the change independently. Business requirements determine which notifications are worth emitting; the service that owns the data remains responsible for making the event structure reusable. section 1.4.1
A producer names the topic that will receive a message. To send it, the client serializes the data, chooses a partition, assembles messages into batches, and forwards those batches to the appropriate broker. It can wait for an acknowledgment and retransmit after errors or lost acknowledgments. Client libraries handle much of this routine; applications still need defensible configuration choices. section 3.1.1 section 3.2
Figure 3.1 Simplified workflow of message transmission by a producer, including serialization, partition selection, batch assembly, and data transmission to the server, with an optional confirmation step
Follow the producer workflow from serialization and partition selection through batching, transmission, and optional confirmation. section 3.1.1
The main design choices have operational consequences:
- Choose a partition key when related messages must stay in sequence; balance that need against distributing work evenly across partitions. section 3.2.6
- Size client memory and batches for messages awaiting acknowledgment. Insufficient buffer capacity can impede sending. section 3.2.6
- Keep the producer’s clock accurate: incorrect message timestamps can affect retention and storage costs. section 3.2.6
Producers obtain and cache cluster metadata to find partition leaders. If leadership changes, the client library can refresh that metadata and resend the batch. Frameworks can simplify integration, while lower-level clients offer finer control; either choice leaves the publishing service responsible for the events it emits. section 3.2 section 3.1 section 1.4.1
