Append-only system
An append-only system preserves existing data and records changes by adding new entries without modifying or removing prior entries.
Kafka for Architects
An append-only log is a sequential record structure in which new entries are added without modifying prior entries.
- How it is used
- Supports event-driven services that distribute changes, retain history, and reconstruct state through replay.
- How deep it goes
- Explains sequential retention, replay, storage tiers, compaction, and database trade-offs without detailing implementation.
- Examples
- Uses Customer360Service profile notifications, event replay, object-storage offload, and compacted topics.
- What you can do after
- Evaluate append-only designs against databases by weighing replay latency, retention, compaction, querying, and consistency requirements.
Snowflake Data Engineering
An append-only strategy is a data-loading technique that adds new records without updating or deleting existing records.
- How it is used
- Supports incremental pipelines that retain historical changes while avoiding update-based SCD type 2 maintenance.
- How deep it goes
- Works through row insertion and derives validity ends with LEAD and NVL, while deferring the illustrative query from exercises.
- Examples
- Uses PARTNER rows where a rating changes from B to A between June 1 and August 8, 2023.
- What you can do after
- Apply it via Snowflake SQL to retain versions and calculate validity intervals for downstream consumers.
Read how Snowflake Data Engineering explains append-only system →
Related concepts
- lead functionuses · A lead function returns a value from a subsequent position in an ordered sequence, optionally within a partition and with a fallback when unavailable.
- event sourcingused by · Event sourcing is a persistence pattern in which state changes are recorded as an ordered sequence of events from which application state can be derived.
- source of truthequivalent to · A source of truth is the authoritative source or representation from which the accepted value or state of information is determined.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.