RewriteDataFiles is an Apache Iceberg maintenance procedure that rewrites physical data files to improve layout, reduce metadata overhead, and accelerate reads. It can merge small or oversized files, apply compression, sort records, consolidate delete files, and refresh file statistics. section 10.2.1
Choose the rewrite strategy according to workload and operational constraints:
- Bin-pack, the default, combines small files into target-sized outputs and reduces file-open and metadata costs. section 10.2.1
- Sort rewrites records by selected columns to improve locality and predicate pruning; Z-order can help multidimensional access patterns but may increase compaction cost and is workload dependent. section 10.2.1
- Merge-on-Read tables particularly benefit when delete files accumulate, because rewriting applies deletes to base files and restores read performance. section 10.1.4
Operational guardrails
Run compaction incrementally by partition, file size, or timestamp, and validate parameters before orchestration. Bound data volume and file counts, use planning or dry-run modes where available, and monitor query latency, planning time, delete-file growth, and table-size expansion. section 10.1.4 section 11.3.2
CALL catalog_name.system.rewrite_data_files(
table =>'db.sales',
strategy =>'sort',
sort_order =>'zorder(customer_id, product_id)'
);
A Spark SQL example shows how to invoke RewriteDataFiles with sort and Z-order options. section 10.2.1
