Apache Iceberg is an open table format that adds a standard metadata layer to files on low-cost object storage. It separates logical table structure from physical layout, allowing distributed files—typically Apache Parquet—to behave as coherent analytical tables across engines. section 2.0 section 2.1
Its architecture addresses data-lake weaknesses around metadata scalability, concurrent writes, schema changes, query planning, and fragmented tooling. Layered metadata tracks schemas, partitions, snapshots, files, and statistics so engines can prune data before execution rather than discovering it through broad directory scans. section 1.5 section 2.3
Core capabilities include:
- Full ACID transactions make concurrent reads and writes atomic, isolated, durable, and protected against stale-write conflicts. section 2.6.1
- Schema evolution supports adding, renaming, reordering, and deleting columns without rewriting the entire table or breaking existing queries. section 2.4 section 2.6.2
- Time travel queries historical snapshots for auditing, debugging, recovery, and versioned analysis; retention policies limit how far back queries can reach. section 2.6.3 section 2.4
- Partition evolution changes physical partitioning without migrating existing data, while hidden partitioning conceals transformation details from users. section 2.4
- Branches and tags isolate experiments, stage changes, and control promotion of table states. section 2.4
Iceberg supports a modular lakehouse: object storage holds data, ingestion frameworks write tables, catalogs track metadata, federation layers serve analytics, and consumption tools support BI, AI, and real-time applications. Its open contract lets teams combine engines and reduce vendor lock-in and data duplication. section 2.7 section 2.5
Local retail lakehouse workflow
A Docker Compose environment can combine PostgreSQL as the operational source, Apache Spark for ingestion, MinIO for object storage, Nessie as the catalog, Dremio for SQL, and Apache Superset for dashboards. Spark reads fashion-sales data from PostgreSQL and writes Iceberg tables; Dremio creates analytical views for regional, customer, and campaign trends. section 3.1 section 3.2 section 3.2.3
Figure 2.1 Table formats let data lakehouses store data warehouse-like tables on the data lake with ACID guarantees, so the data lake can function like a data warehouse.
Visualizes how a table format turns files on a data lake into warehouse-like tables with metadata and ACID guarantees. section 2.1
