An open table format defines a logical table and its metadata independently of physical files and execution engines. It adds a standard metadata layer over data stored on low-cost object storage, allowing file collections to behave as coherent, relational tables across tools. section 2.0 section 2.1
A table format coordinates:
- Schemas, partitions, snapshots, and file-level statistics for consistent discovery, pruning, and versioning. section 2.1
- Shared access for query engines, BI tools, processing frameworks, and other consumers without requiring separate proprietary copies. section 1.6 section 2.1
- Transactional behavior such as atomic commits, time travel, concurrent-write isolation, and safe updates to metadata. section 6.3 section 1.7
Apache Iceberg over Parquet
Apache Iceberg illustrates an open table format: Parquet stores the analytical data efficiently at the file level, while Iceberg organizes, versions, and describes those files through metadata. Spark, Flink, Trino, Snowflake, Dremio, Qlik, and other tools can work from the same table. section 2.1 section 2.5
Figure 2.1 Table formats let data lakehouses store data warehouse-like tables on the data lake with ACID guarantees, so the data lake can function like a data warehouse.
Visualizes how table formats place a logical, ACID-capable table abstraction over data-lake storage. section 2.1
