When an online store changes a product price, inventory and sales analysis both need the update—but they need not call each other. Apache Kafka lets a producer publish the change once and lets independent consumers read it. Kafka is an open-source, distributed event-streaming platform built around persistent logs, so consumers can also revisit retained events after a failure or for later analysis. section 1.1 section 2.4
A producer writes records to a topic, which organizes a stream of related data. Kafka divides topics into partitions across brokers. A record has a value and may have a key, timestamp, and headers; records with the same key go to the same partition. An offset identifies a record’s position within its partition. Consumers read records, while a consumer group shares partitions among its members to process work in parallel. section 1.3 section 3.3 section 4.5
Figure 1.2 The components and data flow of Kafka
Follow a record from its producer through a topic partition and broker to a consumer; the diagram also shows the components involved in replication. section 1.3
Replication places partition copies on multiple brokers. A leader handles a partition’s reads and writes, and followers provide resilience if a broker fails. Adding partitions supports parallel work; adding brokers distributes storage and load. Those gains depend on topic design and a healthy cluster, not merely on publishing more records. section 1.3 section 4.5 section 1.4
Watch a price stream from two terminals
With Kafka running and the topic `products.prices.changelog` available, start `kafka-console-consumer.sh --topic products.prices.changelog --bootstrap-server localhost:9092` in one terminal. In another, run `kafka-console-producer.sh --topic products.prices.changelog --bootstrap-server localhost:9092` and enter `coffee pads 11`. The consumer displays the update. Start a second consumer with `--from-beginning` to read retained earlier prices independently. Stop the producer with Ctrl-D and consumers with Ctrl-C. section 2.0 section 2.4
Kafka is an exchange between systems, not a replacement for their databases or search engines. Its asynchronous model loosens service dependencies, but consumers must decide how to use the data, and retention or cleanup limits how far back they can read. For production, plan reliable brokers and networking, monitoring, schemas, access controls, and capacity. Managed hosting reduces infrastructure work but does not remove the need for Kafka expertise. section 4.5 section 17.2.1 section 1.4 section 13.4
