An online store needs order and inventory events to reach several systems without a tangle of point-to-point integrations. A persistent distributed log provides a shared place for those events: producers write them, and consumers read them when ready. If a consumer fails, it can resume; a new service can also read earlier events. section 1.1
Start with a log: a sequential list that accepts new entries at the end and can be read from a chosen position. In Kafka, a topic’s data is spread across partitions on multiple brokers. An offset identifies a message’s position within its partition—not a position across the whole topic. section 4.5
Following an order event
A producer sends an order event to a topic. Its partitioner selects a partition; messages with the same key go to the same partition. A consumer group divides partitions among its consumers, with each partition assigned to one consumer in that group. Another system can later read retained events for its own work. section 4.5 section 1.1
Distribution addresses two limits of a single log. Partitions let brokers share storage and consumers process in parallel; replicated partitions improve reliability when machines fail. Each partition has a leader replica that coordinates it. This adds operational structure, but it does not turn Kafka into a replacement for a database, key-value store, or search engine. section 4.2 section 4.5
Figure 4.3 Scaling vertically means adding more resources to a single instance. Scaling horizontally means adding more instances to a system.
The scaling diagram contrasts a bigger single machine with adding machines—the reason a high-volume log is distributed rather than merely given a larger server. section 4.2
