liveBook · Manning

Kafka producer

As explained in Apache Kafka in Action

A producer is an application or service that publishes records to Kafka topics.

When a product price changes, an application can publish the update to Kafka without sending it separately to inventory management and sales analysis. That application is a producer: a Kafka client that writes records to a topic. Consumers read those records independently, including later while the data is retained. section 2.2 section 2.4 section 4.3.3

Publish a price update

With Kafka running locally and the topic already created, run `echo "coffee pads 10" | kafka-console-producer.sh --topic products.prices.changelog --bootstrap-server localhost:9092`. The tool takes the input and sends it to the named topic. It prints no success confirmation for this piped command. section 2.2

For ongoing updates, start `kafka-console-producer.sh --topic products.prices.changelog --bootstrap-server localhost:9092` and type one message per line; Ctrl-D ends input. Multiple producers can write to the same topic—for example, departments updating different product categories. Multiple consumers can then read those updates without coordinating with the producers or with one another. section 2.4 section 17.2.1

A production client does more than submit text:

  • Serialize keys and values into byte arrays. The data format is the application's choice, not Kafka's. section 8.1.2 section 17.2.1
  • Choose a topic partition with a partitioner, then buffer records into batches and send them to that partition's leader broker. A fixed partition is possible, but ordinarily the provided partitioning method is preferable. section 8.1.2 section 1.3
  • Set acknowledgments, timeouts, and retries to suit reliability needs. Batching and buffer management also affect performance and resource use. section 8.5 section 6.2

Plan for failures outside the happy path. If a producer cannot reach a broker, messages queue up; the application must decide whether to block, buffer, drop messages, or report unavailability. For third-party integration, Kafka Connect may avoid custom producer code. For non-Java producers, check partitioning compatibility when keys must map consistently with Java producers. section 16.1.1 section 14.0

How Apache Kafka in Action covers it

How deep it goes
Traces serialization, partitioning, buffering, and sending, then identifies acknowledgment and failure-handling decisions.
Examples
Uses console commands to publish product-price updates and parallel producer and consumer windows to observe their flow.
What it assumes
Starts with a local Kafka cluster and created topic; introduces producing through commands rather than requiring Kafka client code.
What you can do after
Apply it via kafka-console-producer.sh to publish updates, then evaluate partitioning and reliability choices for application producers.
How it connects
Connects producers to topics, partitions, serializers, brokers, consumers, and Kafka Connect.

Also explained in

Compare how 2 books explain Kafka producer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.