Metadata sprawl is inefficient growth in an Iceberg table’s metadata layer. Fragmented or oversized manifest files and long snapshot histories increase query-planning time and coordination overhead for reads and writes. section 10.1.3
Iceberg organizes metadata through snapshots, manifest lists, manifest files, and metadata.json. Snapshots point to manifest lists; manifests track data files, partition summaries, column statistics, and paths; metadata.json records snapshot history, schemas, and partition specifications. section 10.1.3
Metadata sprawl commonly develops through:
- High-frequency streaming or near-real-time ingestion that creates many snapshots, each committing only a few data files. section 10.1.3
- Write paths that do not merge manifests, producing many manifest files with few data-file entries. section 10.1.3
- Growing snapshot history that makes metadata.json more expensive to read and rewrite during queries and commits. section 10.1.3
Streaming ingestion pattern
A near-real-time pipeline repeatedly commits small batches. Each commit adds a snapshot and new manifest files, so metadata expands even when the underlying data files remain reasonably organized. section 10.1.3
Figure 10.4 Metadata bloat occurs when you have more manifests than necessary for the number of data files, or you have manifests tracking too many small data files. Both will delay query planning.
Shows how unnecessary manifests and manifests tracking too many small data files delay query planning. section 10.1.3
Figure 10.5 Ideally, you’ll have fewer manifest files tracking fewer data files containing a greater number of records per file, minimizing the cost of metadata traversal and data file I/O.
Contrasts sprawl with a healthier layout using fewer manifests, fewer data files, and more records per file. section 10.1.3
