Orphan file cleanup removes storage files that are not referenced by an Iceberg table’s metadata. These artifacts commonly result from failed or aborted writes, crashed writers, incomplete deletes, or external processes writing directly to a table path. They remain invisible to table queries but increase storage costs and complicate audits. section 11.2.4 section 11.3.2
Treat remove_orphan_files as a corrective operation, not the routine successor to snapshot expiration. expire_snapshots normally reclaims valid files no longer referenced by active snapshots, tags, or branches; orphan cleanup addresses files that never became part of a valid table state. section 11.2.3 section 11.2.4
Apply the procedure with operational safeguards:
- Run it sparingly and with a tight time window because object-store listings can be expensive at scale. section 11.2.3
- Use an age threshold so recently written files remain protected from in-flight or recently failed operations; a common default is 10 days, with 24 hours as a minimum. section 11.3.2
- Restrict delete privileges and isolate maintenance workflows because misapplied cleanup can have irreversible effects in poorly separated storage layouts. section 11.3.2
- Schedule cleanup through Airflow, dbt, Dagster, or another workflow engine, using table-specific policies and workload-aware cadence. section 11.1 section 11.1.1
Retention workflow
A daily maintenance DAG can expire snapshots according to retention policy and invoke cleanup only when failed jobs or untracked files justify storage listing. For secure deletion in merge-on-read tables, compact delete files and base files first, expire every snapshot retaining the old files, then clean up remaining unreferenced artifacts. section 10.3.3 section 11.1.1
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
import subprocess
default_args = {
'owner': 'dataops',
'start_date': datetime(2025, 1, 1),
'retries': 1,
'retry_delay': timedelta(minutes=5),
}
def expire_snapshots():
subprocess.run([
"spark-submit ",
"--class ", "org.apache.iceberg.actions.ExpireSnapshots ",
"--master ", "local ",
"maintenance.jar ",
"--table ", "db.sales ",
"--older-than ", "7d "]) #1
with DAG('iceberg_retention_policy',#2
default_args=default_args,
schedule_interval='@daily',
catchup=False) as dag:
expire = PythonOperator(
task_id='expire_old_snapshots',
python_callable=expire_snapshots
)
expire >>cleanup
Use the retention DAG as an implementation pattern for scheduling snapshot expiration and the subsequent maintenance stage. section 11.2.3
