An Iceberg table is a logical table whose data and metadata are coordinated through Apache Iceberg. A catalog resolves its name to metadata, while query and processing engines use that metadata to discover schemas, partitions, snapshots, manifests, and data files. This separates table identity and version history from any single engine, enabling Spark, Flink, Trino, Dremio, and other compatible tools to work from a shared view. section 7.1.2 section 9.1.1
An Iceberg table supports several operational responsibilities:
- Stores schema, partitioning, snapshot history, manifests, and file-level statistics in managed metadata. section 7.1.2
- Coordinates writes through atomic catalog-pointer updates, causing conflicting commits to fail instead of silently overwriting one another. section 7.1.2
- Enables time travel and rollback by retaining snapshot history and the schema associated with each version. section 7.1.2
- Supports schema evolution, snapshot isolation, and interoperable access across ingestion, query, BI, notebook, and machine-learning workloads. section 6.6 section 9.1.1
- Requires lifecycle attention to file generation, commit frequency, compaction, partition sizing, snapshot expiration, and metadata growth at scale. section 6.5.2
From ingestion to analysis
A high-volume streaming pipeline can use Flink to consume Kafka events, deduplicate and partition them, then commit batches to an Iceberg table. Query engines inspect the catalog, plan scans using file statistics, and read the appropriate snapshot. Superset or another BI tool can expose the resulting table as a dataset for charts and dashboards. section 6.5.2 section 3.5.3
Figure 7.4 Engines request tables from catalogs, and the catalogs return the metadata locations.
Follow the catalog-to-engine interaction showing how a table name resolves to a metadata location. section 7.1.2
Figure 7.5 The metadata provides different levels of data so an engine can identify a snapshot and determine which data files in that snapshot should be scanned.
Inspect how metadata identifies a snapshot and narrows the data files scanned by an engine. section 7.1.2
