Snapshot retention policy

As explained in Architecting an Apache Iceberg Lakehouse

A snapshot retention policy defines which table snapshots remain available and the conditions under which older snapshots are expired.

A snapshot retention policy controls which Iceberg table snapshots remain available and when older history expires. Each commit creates a snapshot; unmanaged growth can inflate metadata, slow query planning, and strain catalog performance. section 10.3.1 section 2.4

Tune retention against workload behavior and recovery requirements:

• Use time-based windows for high-frequency ingestion, CDC, or streaming tables; a fixed retain_last count may represent only minutes or hours. section 10.3.1
• Use snapshot tags to preserve audit points, backfill states, or controlled rollback targets while pruning routine history aggressively. section 10.3.1
• Balance metadata and storage savings against reduced time-travel coverage; retention cannot recover expired historical states. section 10.3.1 section 2.4

Expiring snapshots

CALL catalog_name.system.expire_snapshots(table =>'db.sales', older_than =>TIMESTAMP '2024-06-01 00:00:00.000', retain_last =>20); section 10.3.1

Figure 10.9 When Snapshot 1 expires, only metadata or data files that aren’t associated with the remaining snapshots are deleted.

Shows that expiration removes only metadata or data files no longer associated with remaining snapshots. section 10.3.1

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes

Explains expiration tradeoffs, time windows, snapshot counts, tagging, and file cleanup without detailing implementation internals.

Examples

Demonstrates expiration through expire_snapshots SQL with an older_than timestamp and retain_last count.

What you can do after

Operate/configure snapshot expiration and preserve selected recovery points with time windows, counts, and tags.

How it connects

Connects snapshot retention with time travel, snapshot tagging, metadata management, catalog performance, auditing, and rollback.

liveBookmanning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

• snapshot expirationconfigures · Snapshot expiration is the removal of references to older Iceberg table snapshots so their unneeded files can become eligible for cleanup.
• change-data-captureconfigured by · Change-data capture is a technique for detecting and recording changes to a data source, such as insertions, updates, and deletions, so they can be propagated or processed.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.