The storage layer is the foundation of an Apache Iceberg lakehouse: it stores data cost-effectively while shaping reliability, scalability, performance, security, and operating cost. A modular architecture keeps storage independent from ingestion, cataloging, federation, and consumption so each component can evolve separately. section 2.7 section 5.0
Evaluate a storage layer against these requirements:
- Performance: measure how quickly query engines retrieve and scan files. section 5.1
- Security: provide appropriate access controls and encryption for sensitive data. section 5.1
- Integrity: support recovery from hardware failures and human mistakes. section 5.1
- Operations and cost: account for scalability, cloud expense, maintenance effort, compliance, and deployment constraints. section 5.1 section 5.5
Choose between colocated block storage and separate object storage by weighing their tradeoffs. Block storage, such as HDFS, keeps data near compute for low latency but couples capacity growth to compute provisioning and increases recovery responsibility. Object storage, such as S3 or MinIO, separates compute from storage and provides scalable, lower-overhead operation, while network access can add latency and variable write visibility. section 5.4
Apache Parquet and the S3 API provide the dominant interoperability standards. Iceberg commonly stores analytical data as Parquet files and uses the S3 API to access object-storage platforms across cloud and on-premises environments. section 5.3 section 5.6
Local lakehouse deployment
A Docker Compose environment can use MinIO as the object storage layer for Iceberg data and metadata files. Spark writes tables to MinIO, Nessie tracks table metadata, Dremio queries the catalog and storage, and Superset provides dashboards. section 3.2 section 3.2.3
Figure 2.6 Components of a complete data lakehouse implementation
Use the lakehouse component diagram to see the storage layer alongside ingestion, catalog, federation, and consumption. section 2.7
