Chapter 1: The world of the data lakehouse
The introduction explains the tradeoffs that motivated the data lakehouse and positions Apache Iceberg as an open way to make lake data usable across analytical tools.
After this chapter you will be able to
- Traditional architecture tradeoffs. Explain why warehouses, lakes, and hybrid approaches left organizations seeking a different architecture for analytics.
- Iceberg tables and shared access. Describe how Iceberg's open table format changes the way tools use data stored in a lake.
- Lakehouse design aims. Explain what the data lakehouse aims to balance as data-management needs grow.
- Transactional and analytical access patterns. Explain why row-based OLTP databases suited individual record operations but struggled with historical analytics, and how column-based layouts better fit queries that read many rows but few columns.
- Warehouse response and tradeoffs. Describe why organizations moved analytics into enterprise data warehouses and state the limitations identified here.
- On-premises warehouse constraints. Explain why traditional enterprise data warehouses were expensive and difficult to scale, including the effects of coupled storage and compute and warehouse-specific ETL.
- Cloud warehouse flexibility. Describe how separating compute and storage changed scaling, costs, and access to analytics in cloud data warehouses.
- Remaining cloud warehouse limitations. Explain why cloud warehouses still created inefficiency and lock-in despite their scaling improvements.
- Data lake origins. Explain why data lakes emerged and describe their basic approach to storage and analytics.
- Hadoop storage and compute economics. Explain how early Hadoop lakes made large-scale batch processing and raw-data retention economical.
- Loose files and data swamps. Explain how loosely organized lake files made datasets difficult to manage and trust.
- Parallel lake and warehouse tradeoffs. Explain why organizations maintained both lakes and cloud warehouses and what gap remained.
- Scalable table metadata. Explain why Hive metadata became a bottleneck as analytical tables grew and how Iceberg’s metadata model addresses that limitation.
- Safer schema evolution. Describe the Hive schema limitation and why separating logical table structure from physical file layout matters.
- Planning-time data pruning. Explain how Iceberg’s metadata lets query engines prune earlier than Hive often did, and why that helps on cloud object storage.
- Concurrent table correctness. Explain what Iceberg’s ACID guarantees provide when readers and writers share a dataset, and identify the basis for those guarantees given here.
- Lakehouse analytical foundation. Describe the overall shift Iceberg enables in how a data lake can be used.
- Shared Open Data Foundation. Explain how an Iceberg-based lakehouse lets different teams and tools use the same managed datasets while reducing duplicated data and conversions.
- Iceberg Ecosystem Flexibility. Explain why the section emphasizes Iceberg’s ecosystem when comparing it with other table formats.
Section by section
Introduction
The introduction explains the tradeoffs that motivated the data lakehouse and positions Apache Iceberg as an open way to make lake data usable across analytical tools.
- Traditional architecture tradeoffs: Explain why warehouses, lakes, and hybrid approaches left organizations seeking a different architecture for analytics.
Why it matters: These tradeoffs explain why the lakehouse emerged rather than being just another storage option. - Iceberg tables and shared access: Describe how Iceberg's open table format changes the way tools use data stored in a lake.
Why it matters: A shared table representation lets the lake serve as a common source for analytics tools. - Open lakehouse capabilities: Explain the combination of openness and warehouse-like capabilities attributed to an Apache Iceberg lakehouse.
Why it matters: This combination states the promise of the architecture without requiring commitment to one proprietary warehouse system.
Key terms: data warehouses, data lakes, hybrid approaches, data lakehouse, Apache Iceberg, open table format, distributed storage, canonical copy, ETL, Apache Iceberg lakehouse
1.1 Evolution from database to data lakehouse
Introduces the lakehouse as the latest response to growing data-management complexity and competing platform needs.
- Lakehouse design aims: Explain what the data lakehouse aims to balance as data-management needs grow.
Why it matters: These aims provide a starting point for evaluating why organizations adopt a lakehouse.
Key terms: data lakehouse, cost, scalability, accessibility, open, interoperable
1.2 The rise of data warehouses
This section explains why transactional databases struggled with growing analytical workloads and how enterprise data warehouses addressed that mismatch, at a cost.
- Transactional and analytical access patterns: Explain why row-based OLTP databases suited individual record operations but struggled with historical analytics, and how column-based layouts better fit queries that read many rows but few columns.
Why it matters: The different access patterns explain why one storage layout does not serve both workloads equally well. - Warehouse response and tradeoffs: Describe why organizations moved analytics into enterprise data warehouses and state the limitations identified here.
Why it matters: Separating analytics from operational databases explains both the rise of warehouses and the motivation to seek alternatives.
Key terms: OLTP, row-based layout, column-based layouts, analytical queries, enterprise data warehouse (EDW), online analytical processing (OLAP), optimized schemas, cost, rigidity
1.3 The move to cloud data warehouses
This section explains how cloud data warehouses eased the cost and scaling problems of on-premises warehouses while retaining data-loading and vendor lock-in limitations.
- On-premises warehouse constraints: Explain why traditional enterprise data warehouses were expensive and difficult to scale, including the effects of coupled storage and compute and warehouse-specific ETL.
Why it matters: These constraints explain why a more flexible warehouse model appealed to organizations. - Cloud warehouse flexibility: Describe how separating compute and storage changed scaling, costs, and access to analytics in cloud data warehouses.
Why it matters: This distinction captures the main improvement cloud warehouses made over on-premises systems. - Remaining cloud warehouse limitations: Explain why cloud warehouses still created inefficiency and lock-in despite their scaling improvements.
Why it matters: These unresolved problems motivate the search for a different foundation for large-scale analytics.
Key terms: storage and compute, on-premises hardware, overprovision, ETL pipelines, cloud data warehouses, compute and storage, self-service analytics, proprietary serialization formats, query semantics, duplicated storage
1.4 The data lake and the Hadoop era
The section explains why Hadoop-era data lakes emerged, what made them economical, and why their limitations left organizations needing a better way to combine lake and warehouse capabilities.
- Data lake origins: Explain why data lakes emerged and describe their basic approach to storage and analytics.
Why it matters: This explains the need that warehouses alone did not meet. - Hadoop storage and compute economics: Explain how early Hadoop lakes made large-scale batch processing and raw-data retention economical.
Why it matters: Hadoop’s cost advantage helps explain why organizations adopted lakes despite their later difficulties. - Hadoop analytics performance limits: Describe why Hadoop-era lakes struggled with interactive analytics and how later engines improved performance without removing every difficulty.
Why it matters: Cheap large-scale processing did not automatically provide responsive, easy-to-operate analytics. - Loose files and data swamps: Explain how loosely organized lake files made datasets difficult to manage and trust.
Why it matters: Having abundant data is of limited value when teams cannot reliably use or govern it. - Parallel lake and warehouse tradeoffs: Explain why organizations maintained both lakes and cloud warehouses and what gap remained.
Why it matters: The burden of two systems sets up the reason to seek a lakehouse approach.
Key terms: data lake, distributed storage, semistructured and unstructured data, Hadoop Distributed File System (HDFS), MapReduce, colocated, commodity hardware, raw data, Hive, Impala
1.5 Apache Iceberg: Giving data lakes data warehouse capabilities
The section explains how Apache Iceberg addresses Hive-era limitations to give data lakes reliable, warehouse-like table behavior.
- Scalable table metadata: Explain why Hive metadata became a bottleneck as analytical tables grew and how Iceberg’s metadata model addresses that limitation.
Why it matters: Metadata must remain manageable as an analytical platform grows. - Safer schema evolution: Describe the Hive schema limitation and why separating logical table structure from physical file layout matters.
Why it matters: Long-lived datasets need to change as business needs change without making common schema edits unsafe. - Planning-time data pruning: Explain how Iceberg’s metadata lets query engines prune earlier than Hive often did, and why that helps on cloud object storage.
Why it matters: Avoiding unnecessary storage operations is especially valuable when querying data in cloud object stores. - Concurrent table correctness: Explain what Iceberg’s ACID guarantees provide when readers and writers share a dataset, and identify the basis for those guarantees given here.
Why it matters: A shared analytical table must remain correct when workloads read and update it concurrently. - Lakehouse analytical foundation: Describe the overall shift Iceberg enables in how a data lake can be used.
Why it matters: This connects Iceberg’s table capabilities to the book’s broader reason for building a lakehouse.
Key terms: Hive Metastore, metadata scalability, scalable table format, schema management, logical table structure, physical file layout, prune, query planning, directory listings, file-footer reads
1.6 The data lakehouse: Best of both worlds
The section explains how an Iceberg-based lakehouse combines shared, low-cost data storage with warehouse-like usability, then compares Iceberg’s flexibility and ecosystem with other table formats.
- Shared Open Data Foundation: Explain how an Iceberg-based lakehouse lets different teams and tools use the same managed datasets while reducing duplicated data and conversions.
Why it matters: A shared data foundation helps teams choose tools by workload without maintaining a separate version of the data for each one. - Snapshot Based Table History: Describe what Iceberg’s snapshot-based architecture lets users do with earlier table states.
Why it matters: Earlier table states can help investigate problems without a separate historical copy of each dataset. - Evolving And Hidden Partitions: Distinguish partition evolution from hidden partitioning and explain their stated benefits.
Why it matters: These features let partitioning adapt as data changes without making queries manage partition fields. - Iceberg Ecosystem Flexibility: Explain why the section emphasizes Iceberg’s ecosystem when comparing it with other table formats.
Why it matters: Broad tool support gives organizations more room to evolve their architecture without committing to one vendor’s execution model.
Key terms: data lakehouse, shared data products, open table formats, Apache Iceberg, snapshot-based architecture, time travel, roll back, partition evolution, hidden partitioning, transforms
1.7 Summary
Where this is used later
- What does it mean that Iceberg is a table format? (chapter 2) · Develops what it means for Iceberg to be a table format.
- ACID transactions (chapter 2) · Explains the ACID transactions mentioned here.
- Why you need a table format (chapter 2) · Provides the file-management problems that motivate a table format.
- How Apache Iceberg manages metadata (chapter 2) · Develops the explanation of how Iceberg manages metadata.
- (chapter intro) (chapter 2) · Introduces Iceberg in the context of adding warehouse capabilities to data lakes.
- How tables evolve (chapter 2) · Explores table evolution beyond the distinction introduced here.
- Cost efficiency and query performance (chapter 2) · Develops the cost and query-performance benefits introduced here.
- Commit protocols and conflict handling (chapter 6) · Examines commit protocols and conflict handling in greater depth.
Concepts explained in this chapter
- data lakehouseA data lakehouse is a data architecture that combines the openness and flexible storage of a data lake with the governance and analytical capabilities of a data warehouse.
- Apache IcebergAn open table format for managing large analytic datasets with metadata-driven structure, snapshots, schema evolution, and transactional write capabilities.
- data warehouseA data warehouse is a centralized repository optimized for analytical querying and reporting over integrated historical data.
- data lakeA data lake is a storage-centered data platform that retains data in flexible formats on relatively inexpensive storage for varied downstream processing.
- open table formatA table format is a specification for representing a logical table and its metadata independently of the underlying physical data storage.
- ACIDThe transaction guarantees of atomicity, consistency, isolation, and durability that preserve reliable database state changes.
- HiveApache Hive is a data warehouse infrastructure that provides SQL-based querying and table management over distributed data stored in file systems or object stores.
- hidden partitioningA partitioning method in which partition values are derived from transforms and are not exposed as query columns.
- metadata designThe organization and representation of metadata used to describe, locate, version, and plan access to table data.
- partition evolutionThe ability to change a table's partition strategy over time without rewriting existing data.
- query planningQuery planning is the process of determining an execution strategy for a query, including which files or partitions must be read.
- snapshot-based architectureAn architecture that represents table state through immutable or versioned snapshots of metadata and data references.
- Hadoop Distributed File SystemHadoop Distributed File System is a distributed file system designed to store very large datasets across clusters of commodity machines.
- partition pruningPartition pruning is the elimination of data partitions that cannot satisfy a query's filters before or during data access.
- vendor lock-inTool lock-in is dependence on a particular software tool or vendor that makes changing tools difficult or costly.
- columnar dataA columnar format stores values from the same field together to support efficient analytical reading and compression.
- schema evolutionSchema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
- ETLA workflow that extracts data from sources, transforms it, and loads it into a target data system.
- SparkApache Spark is an open-source distributed processing framework for large-scale batch, streaming, and analytical workloads.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.