Snapshot expiration is an Apache Iceberg maintenance procedure for controlling the storage and metadata costs of historical table states. Time travel and rollback preserve snapshots, but accumulated history enlarges metadata.json, increases planning work, and retains files that are no longer needed. section 10.1.3 section 10.3
Expiration removes selected snapshots and deletes data or metadata files uniquely associated with them. Iceberg protects files referenced by active or retained snapshots, tags, or branches, so remaining table states stay queryable and reproducible. section 10.3.1 section 11.2.4
Production use depends on explicit retention and operational controls:
- Set both an age boundary and a minimum number of recent snapshots to balance storage reclamation against rollback and time-travel needs. section 10.3.1 section 11.3.4
- Run expiration periodically or from metadata-aware workflows; high-velocity CDC tables may need daily expiration, while monthly reporting tables may need only monthly maintenance. section 11.1.3 section 11.1.2
- Treat remove_orphan_files as corrective cleanup for failed deletes or abandoned writes, not as the routine follow-on to successful expiration. section 11.2.3 section 11.2.4
- Validate parameters, state, age limits, file counts, and data-volume bounds before automating irreversible maintenance. section 11.3.2
Spark and Dremio retention calls
Spark can retain the latest 20 snapshots while expiring snapshots older than June 1, 2024: CALL catalog_name.system.expire_snapshots(table =>'db.sales', older_than =>TIMESTAMP '2024-06-01 00:00:00.000', retain_last =>20); Dremio provides the equivalent VACUUM TABLE s3.sales EXPIRE SNAPSHOTS older_than '2024-06-01 00:00:00.000' retain_last 20; procedure. section 10.3.1
Expiration alone may not satisfy secure deletion. Logical deletes remain physically present while retained snapshots reference their files; Merge-on-Read tables additionally require compaction before old base files can be removed. Compliance workflows therefore coordinate compaction, snapshot expiration, and orphan cleanup. section 10.3.3 section 11.2.5
