A retention policy defines how long Iceberg snapshots, table states, data files, and metadata remain available before expiration or deletion. It balances time-travel and rollback capabilities against storage cost, table performance, and regulatory requirements. Without regular cleanup, obsolete snapshots and unreferenced files can accumulate even after data is logically deleted. section 10.3 section 11.3.4
Build the policy around these operational distinctions:
- Use snapshot expiration as the primary lifecycle mechanism. Expiring snapshots removes obsolete history and makes data or metadata files unreferenced by remaining snapshots, tags, or branches eligible for deletion. section 11.2.3 section 11.2.4
- Use orphan-file removal only as corrective maintenance for files left by failed deletes, crashed writers, aborted transactions, or external processes. Avoid treating it as a routine follow-up because object-store listings can be expensive at scale. section 11.2.3 section 11.2.4
- Account for write mode. Copy-on-Write replaces data files during updates and ties obsolete-file eligibility closely to snapshot expiration; Merge-on-Read can retain delete files that require additional compaction and lifecycle management. section 10.3.2 section 11.4
- Protect snapshots needed for recovery, reproducibility, auditing, or regulatory workflows through suitable retention windows, tags, and branches. Expiration can remove rollback targets once their references and snapshots age out. section 11.2.2 section 11.3.4
Daily snapshot expiration
An Airflow DAG can run an Iceberg expiration job daily and remove snapshots older than seven days. The workflow should be paired with workload-aware safeguards and should invoke orphan-file cleanup only when an operational incident justifies it. section 11.2.3 section 11.2.4
Figure 10.8 Iceberg tables have snapshot isolation, allowing you to query older versions of the table. The catch is that all the historical data is still stored along with the current data.
Snapshot isolation preserves queryable historical states, while retaining those states also preserves their associated physical files until expiration. section 10.3
