Snapshot history is the ordered record of prior versioned table states. In Apache Iceberg, each snapshot points to one manifest list, while metadata.json records the table’s snapshot history alongside schemas and partition specifications. section 10.1.3
Snapshot history participates in table metadata growth:
- High-frequency or streaming ingestion can create many snapshots, each committing only a small number of data files. section 10.1.3
- A longer snapshot list increases the cost of reading current metadata and rewriting metadata.json during commits. section 10.1.3
- Snapshot accumulation contributes to metadata bloat even when the underlying data layout remains efficient. section 10.1.3
Near-real-time ingestion
A streaming job that repeatedly commits a few data files creates a new snapshot and manifest files for each commit, gradually enlarging the table’s metadata layer. section 10.1.3
Figure 10.4 Metadata bloat occurs when you have more manifests than necessary for the number of data files, or you have manifests tracking too many small data files. Both will delay query planning.
Relates excessive manifests and small data-file groups to metadata bloat and delayed query planning. section 10.1.3
