Append-only system
A system that records new data by adding entries while preserving existing entries without modifying or removing them.
Kafka for Architects
An append-only log is a sequential record structure in which new entries are added without modifying prior entries.
- How it is used
- Supports event-driven services that distribute changes, retain history, and reconstruct state through replay.
- How deep it goes
- Explains sequential retention, replay, storage tiers, compaction, and database trade-offs without detailing implementation.
- Examples
- Uses Customer360Service profile notifications, event replay, object-storage offload, and compacted topics.
- What you can do after
- Evaluate append-only designs against databases by weighing replay latency, retention, compaction, querying, and consistency requirements.
Snowflake Data Engineering
An append-only strategy is a data-loading technique that adds new records without updating or deleting existing records.
- How it is used
- Supports incremental pipelines that retain historical changes while avoiding update-based SCD type 2 maintenance.
- How deep it goes
- Works through row insertion and derives validity ends with LEAD and NVL, while deferring the illustrative query from exercises.
- Examples
- Uses PARTNER rows where a rating changes from B to A between June 1 and August 8, 2023.
- What you can do after
- Apply it via Snowflake SQL to retain versions and calculate validity intervals for downstream consumers.
Read how Snowflake Data Engineering explains append-only system →
Also covered in
- Real-World Cryptography
- Terraform in Depth: Infrastructure as Code with Terraform and OpenTofu
- Functional Programming in C#, Second Edition
- Architecting an Apache Iceberg Lakehouse
- Rust in Action: Systems programming concepts and techniques
What to know first
- consistencyConsistency is the degree to which related data, behavior, or results agree with one another and conform to applicable rules or expectations.
- pipelineA pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
Related concepts
- event sourcingused by · Event sourcing is a persistence pattern in which state changes are recorded as an ordered sequence of events from which current state can be derived or reconstructed.
- lead functionuses · A lead function returns a value from a subsequent position in an ordered sequence, optionally within a partition and with a fallback when unavailable.
- resource contentionaddresses · Resource contention is competition among concurrent operations, processes, or workloads for access to a limited shared resource.
Questions about append-only system
What is append-only system?
A system that records new data by adding entries while preserving existing entries without modifying or removing them.
Which Manning books explain append-only system?
Kafka for Architects and Snowflake Data Engineering.
What should I know before learning append-only system?
Manning books that teach append-only system usually introduce consistency and pipeline first.
What is the difference between append-only system and slowly changing dimension?
A system that records new data by adding entries while preserving existing entries without modifying or removing them. A slowly changing dimension is a dimensional-modeling pattern that preserves controlled historical changes to dimension attributes over time.
What is append-only system also called?
Append-only system is also called append-only storage.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

