Apache Kafka is a distributed platform for moving and retaining streams of events. Its core job is to transport messages between applications, but its persistent event log also supports replay, data integration, and real-time processing. It is most useful when many systems need to react independently to the same changes, or when high volumes of events must arrive with low latency. section 1.3 section 1.7.3 section 1.1.2
Consider an order workflow built from chained request-response calls. Each service may have to wait for another, and a failure can disrupt the chain. With Kafka, a producer publishes an event and interested consumers process it independently. This reduces direct coupling, but it shifts design work toward eventual consistency, idempotency, and events that arrive out of order. section 1.1 section 1.1.1
Three roles define the basic flow:
- Producers send messages without needing direct integrations with their consumers. section 1.4.1
- Brokers form a cluster that stores messages and makes them available to consumers; replication helps preserve data when a broker fails. section 1.3.1 section 1.4.2
- Consumers subscribe to topics and request messages. Retained messages let them resume after a disconnection or replay earlier events. section 1.3.1 section 1.4.2
Figure 1.3 The key components in the Kafka ecosystem are producers, brokers, and consumers.
The producer–broker–consumer diagram shows the intermediary that lets applications exchange events without calling one another directly. section 1.3.1
One change, several reactions
When a customer changes address, the customer service can publish a notification. Billing, ordering, and analytics services can each receive it and update their own work independently. The event should reflect a business need, and the service that owns the data remains responsible for its structure. section 1.4.1
Retention changes what Kafka can do. For message delivery, events remain available long enough for subscribers to recover or reprocess them. Retaining a complete change history can instead support event sourcing: a consumer replays changes to reconstruct state. That does not make Kafka a general replacement for a database; queries requiring random access to particular records are a poor fit. section 1.7.1 section 1.7.2
Adoption is an architectural and operational choice, not a default. Kafka’s throughput and replication suit demanding event pipelines, including logs, metrics, and real-time reactions. A smaller system may not need that capacity, while strict ordering, batch transfers, or random data access may favor another design. Teams must also account for client reliability settings, monitoring, security, infrastructure costs, and the control they give up with a managed service. section 5.2.7 section 5.5 section 1.6.1 section 1.6.2
