Compaction reorganizes fragmented Iceberg data and metadata by rewriting files and manifests. It consolidates small files, applies delete files, improves data colocation, and restores more predictable planning and query performance. section 10.2 section 10.2.1
Frequent micro-batch or streaming commits improve freshness but create file fragmentation and metadata growth. Compaction balances that ingestion benefit against read efficiency, compute use, concurrent-workload impact, and maintenance cost. section 6.1.1 section 6.2.3
A practical compaction strategy makes several implementation decisions:
- Choose bin-pack to combine files toward a target size, or sort compaction to reorder records for better locality and filtered-query pruning. section 10.2.1
- Select eligible files with size and count thresholds, then group them into independently processed file sets for parallelism and controlled resource use. section 10.2.3
- Scope rewrites by partition, time window, or business predicate to prioritize hot or degraded areas and keep jobs within operational limits. section 10.2.4
- Align sorting, partition evolution, and file layout with query access patterns rather than applying one physical organization uniformly. section 10.1.2 section 10.2.4
- Treat merge-on-read maintenance as essential: applying accumulated delete files during rewrites reduces read-time reconciliation overhead, while copy-on-write shifts cost toward writes. section 10.1.4 section 10.5
- Automate threshold-driven jobs with metadata-aware orchestration, and coordinate compaction with snapshot expiration and orphan-file cleanup. section 11.1 section 11.1.1
Scoped rewrite for a hot partition
A Spark maintenance workflow can rewrite only files matching region = 'us-west', or combine that predicate with a sale_date range. A metadata check can trigger the job when small-file or delete-file thresholds are breached, leaving other partitions untouched. section 10.2.4 section 11.1.1
Figure 10.7 During compaction, data files are written into fewer, larger files, and manifests are rewritten so that fewer manifests track more files. This improves table performance that has degraded after many writes.
Shows how compaction produces fewer, larger data files and rewrites manifests so metadata tracks the table more efficiently. section 10.2
