liveBook · Manning

Distributed commit log

As explained in Kafka for Architects

A distributed commit log is a replicated, append-oriented record of events maintained across multiple nodes.

A distributed commit log is a replicated, append-oriented record of events maintained across multiple nodes. Kafka stores messages as an immutable event sequence rather than treating them only as transient broker traffic. section 1.7.3

This design shifts integration from point-to-point data movement toward a central, durable, replayable stream. Persistence, replication, and historical retention support high-throughput workloads, fault tolerance, recovery, and event-sourcing use cases. section 1.7.3 section 12.0

Architectural consequences include:

  • Consumers can recover data after downtime instead of depending solely on an in-flight message exchange. section 12.0
  • Historical events can be retained and reprocessed, making the log suitable for event-sourcing scenarios. section 1.7.3
  • Event modeling becomes an architectural concern: teams identify sources, define captured events, and establish schemas for compatibility and scalability. section 1.7.3

Enterprise event foundation

A customer-data architecture can capture transactions, customer interactions, and social-media mentions as events, then expose them to connectors and real-time stream-processing applications through Kafka’s durable log. section 1.7.3

How Kafka for Architects covers it

How deep it goes
Defines the architecture and consequences, but defers implementation mechanics and detailed operational procedures.
Examples
Uses transactions, customer interactions, social-media mentions, consumer recovery, and replayable event streams as concrete examples.
What you can do after
Evaluate Kafka architectures by relating replication, retention, schemas, and replay to scalable event-driven integration.
How it connects
Connects distributed commit logs with Kafka, event sourcing, durable storage, event schemas, and real-time stream processing.

Also explained in

Compare how 2 books explain distributed commit log →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.