Append-only system

As explained in Kafka for Architects

An append-only log is a sequential record structure in which new entries are added without modifying prior entries.

An append-only system records new entries sequentially without modifying earlier entries. Kafka can provide this log for event-driven services, but it is not a database replacement. section 5.1.1

Treating Kafka as an append-only system requires architectural trade-offs:

• Retain older messages in cheaper object storage while keeping recent data on high-performance disks. section 5.1.1
• Replay events to reconstruct service state when historical data is important for analytics, auditing, or regulatory compliance. section 5.1.1
• Preserve a database or read store as the system of record because the log lacks referential integrity, strong consistency, secondary indexes, and full SQL querying. section 5.1.1

Reconstructing customer state

Customer360Service receives profile-change events through Kafka and can rebuild its state by replaying the retained event history. Compaction and retention choices affect how much history remains and how long rebuilding takes. section 5.1.1

Figure 5.2 Using compacted topics in Kafka: you always retain the latest version of each event, allowing Kafka to act as a source of truth.

Use the compacted-topic diagram to compare retaining the latest event version with preserving a complete historical log. section 5.1.1

How Kafka for Architects covers it

How deep it goes

Explains sequential retention, replay, storage tiers, compaction, and database trade-offs without detailing implementation.

Examples

Uses Customer360Service profile notifications, event replay, object-storage offload, and compacted topics.

What it assumes

Assumes familiarity with Kafka and microservice communication while defining the log’s role and limitations.

What you can do after

Evaluate append-only designs against databases by weighing replay latency, retention, compaction, querying, and consistency requirements.

How it connects

Connects Kafka with event-driven microservices, event storage, databases, read stores, retention, and compaction.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Snowflake Data EngineeringAn append-only strategy is a data-loading technique that adds new records without updating or deleting existing records.

Compare how 2 books explain append-only system →

Related concepts

• event sourcingused by · Event sourcing is a persistence pattern in which state changes are recorded as an ordered sequence of events from which application state can be derived.
• source of truthequivalent to · A source of truth is the authoritative source or representation from which the accepted value or state of information is determined.
• lead functionuses · A lead function returns a value from a subsequent position in an ordered sequence, optionally within a partition and with a fallback when unavailable.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.