What does section 2.1, What does it mean that Iceberg is a table format?, cover?

The section explains Iceberg’s table-format abstraction, why it works across engines, and how it complements Parquet’s file-level storage. Explain that a table format is a logical wrapper that defines how data files, schemas, partitions, and snapshots are stored, using metadata to give engines a consistent view of a dataset. Explain why Iceberg’s table abstraction lets different execution engines work with the same dataset while teams retain their preferred tools. Describe how Parquet stores values by column and why that layout benefits analytical queries.

What does section 2.2, Why you need a table format, cover?

This section explains why files in a data lake need a table format to become consistent, manageable, and efficient analytical tables. Distinguish a file format from a table format: Parquet, ORC, and CSV describe records within files, but do not define a dataset spanning many files as one table. Explain that Hive added a table abstraction for SQL access, though its transactional guarantees and metadata scalability were limited. Explain why managing a dataset as files alone makes concurrent writes and schema changes difficult. Describe the possible results of conflicting writes and the lack of built-in tracking for dataset evolution. Explain why query engines need table-level metadata to locate relevant data efficiently. Describe the section’s basic lakehouse table picture: data files hold the dataset, while metadata files help engines scan it.

What does section 2.3, How Apache Iceberg manages metadata, cover?

The section explains how Iceberg’s layered metadata helps query engines locate table data and skip irrelevant files before scanning. Describe the roles of table-level metadata, manifest lists, and manifests, and explain how the catalog gives query engines a stable entry point to them. Trace how Iceberg uses metadata statistics to rule out irrelevant data before scanning, and explain why that improves query efficiency compared with searching through lake files.

What does section 2.4, Key features of Apache Iceberg, cover?

The section explains Iceberg’s transactions, schema changes, snapshots, partitioning, and table-state controls for reliable lakehouse tables. Explain how Iceberg’s ACID transactions let multiple jobs and tools read and write the same datasets safely. Describe which schema changes the section says Iceberg supports and how they affect existing queries and downstream consumers. Explain what Iceberg’s snapshots let queries revisit, why that is useful, and what limits time travel.

What does section 2.5, Apache Iceberg: An open source standard, cover?

This section explains how Iceberg’s vendor-neutral, community-governed standard lets organizations use different tools without tying their lakehouse to one vendor. Explain why Iceberg’s status as an open standard helps organizations build long-lived data architectures. Describe how a common contract lets independent systems work with the format as tools and vendors change. Describe how Iceberg’s broad contributor community and tool integrations support an interoperable lakehouse. Give examples of different kinds of work that integrated tools can perform.

What does section 2.6, Benefits of Apache Iceberg, cover?

Introduces Iceberg’s shift from treating lake data as raw files to treating it as managed tables, while noting that its benefits depend on integration with query engines and catalogs. Explain how Iceberg changes the way organizations work with data lake data, from identifying a dataset as a folder of Parquet files to treating it as a managed table visible in a catalog. State what the section says is needed to unlock Iceberg’s full potential for performance and flexibility.

What does section 2.6.1, ACID transactions, cover?

This section explains how Iceberg’s ACID guarantees keep shared tables correct when multiple jobs read and write concurrently. Explain what atomicity, consistency, isolation, and durability each guarantee for changes to an Iceberg table, including what happens to a conflicting write based on a stale table view. Explain why concurrent writes are problematic without transactional guarantees and how Iceberg’s ACID guarantees reduce the need for external coordination.

What does section 2.6.2, How tables evolve, cover?

This section explains how Iceberg lets long-lived tables change their schemas and partitioning without the costly rewrites associated with traditional data lakes. Explain how Iceberg lets a table’s schema change as requirements evolve, without costly dataset rewrites or breaking existing queries. Distinguish Iceberg’s changeable partitioning strategies from the fixed partition schemes described for Hive. Describe what Iceberg’s schema history records and why the section says it is useful.

What does section 2.6.3, Time travel and snapshot-based queries, cover?

Explains how Iceberg uses snapshots to query earlier table versions and why that helps with recovery and historical analysis. Explain that Iceberg can query a table as it was at a specific point in time because scanning an older snapshot uses that snapshot’s data file listing. Describe how earlier Iceberg snapshots help recover from bad or incomplete ingestion and support historical analysis without duplicate datasets, in contrast to overwritten data in traditional data lakes.

What does section 2.6.5, Cost efficiency and query performance, cover?

This section explains how Apache Iceberg can lower lakehouse costs by reducing data copies, simplifying ingestion, and limiting the data scanned by queries. Explain why keeping copies of data across lakes, warehouses, and data marts becomes costly, especially in the cloud. Describe how querying Iceberg tables directly in the data lake and pruning unnecessary files reduce analytics costs. Explain how putting batch and streaming data into one Iceberg table can simplify ETL and reduce data movement.

What does section 2.7, Apache Iceberg lakehouse components, cover?

Introduces the modular Iceberg lakehouse, why its components can be chosen independently, and the basic role of each of its five layers. Explain why an Iceberg lakehouse is modular and how that differs from a monolithic data warehouse. Describe the benefits the section attributes to choosing tools for each component. Describe the basic job of each of the five components named in the section: storage, ingestion, catalog, federation, and consumption.

What should I know before reading chapter 2?

Apache Iceberg: Giving data lakes data warehouse capabilities (chapter 1): Introduces Iceberg in the context of adding warehouse capabilities to data lakes.

Chapter 2: Apache Iceberg and the lakehouse

What you will learn in chapter 2 of Architecting an Apache Iceberg Lakehouse

Introduces Apache Iceberg as a table format that uses metadata to make files in a data lake behave as relational tables.

After this chapter you will be able to

Iceberg table format role. Explain that Apache Iceberg is a community-driven table format that adds a standard metadata layer over data files, allowing collections of files in a data lake to be treated as coherent relational tables without changing file-level storage.
Table Format and Metadata. Explain that a table format is a logical wrapper that defines how data files, schemas, partitions, and snapshots are stored, using metadata to give engines a consistent view of a dataset.
Engine Independent Table Access. Explain why Iceberg’s table abstraction lets different execution engines work with the same dataset while teams retain their preferred tools.
Iceberg and Parquet Responsibilities. Distinguish Iceberg’s table-level responsibilities from Parquet’s file-level responsibilities and explain why the two complement each other.
Files versus table formats. Distinguish a file format from a table format: Parquet, ORC, and CSV describe records within files, but do not define a dataset spanning many files as one table. Explain that Hive added a table abstraction for SQL access, though its transactional guarantees and metadata scalability were limited.
Table-level consistency needs. Explain why managing a dataset as files alone makes concurrent writes and schema changes difficult. Describe the possible results of conflicting writes and the lack of built-in tracking for dataset evolution.
Metadata and query efficiency. Explain why query engines need table-level metadata to locate relevant data efficiently. Describe the section’s basic lakehouse table picture: data files hold the dataset, while metadata files help engines scan it.
Layered Iceberg Metadata Structure. Describe the roles of table-level metadata, manifest lists, and manifests, and explain how the catalog gives query engines a stable entry point to them.
Metadata Guided File Pruning. Trace how Iceberg uses metadata statistics to rule out irrelevant data before scanning, and explain why that improves query efficiency compared with searching through lake files.
Concurrent table consistency. Explain how Iceberg’s ACID transactions let multiple jobs and tools read and write the same datasets safely.
Controlled schema changes. Describe which schema changes the section says Iceberg supports and how they affect existing queries and downstream consumers.
Snapshot time travel limits. Explain what Iceberg’s snapshots let queries revisit, why that is useful, and what limits time travel.
Evolving and hidden partitions. Distinguish partition evolution from hidden partitioning and explain their effect on data rewrites and queries.
Vendor-neutral table format. Explain why Iceberg’s status as an open standard helps organizations build long-lived data architectures. Describe how a common contract lets independent systems work with the format as tools and vendors change.
Managed tables over raw files. Explain how Iceberg changes the way organizations work with data lake data, from identifying a dataset as a folder of Parquet files to treating it as a managed table visible in a catalog.
Four ACID guarantees. Explain what atomicity, consistency, isolation, and durability each guarantee for changes to an Iceberg table, including what happens to a conflicting write based on a stale table view.
Schema evolution without rewrites. Explain how Iceberg lets a table’s schema change as requirements evolve, without costly dataset rewrites or breaking existing queries.
Partition strategy evolution. Distinguish Iceberg’s changeable partitioning strategies from the fixed partition schemes described for Hive.
Snapshot-based time travel. Explain that Iceberg can query a table as it was at a specific point in time because scanning an older snapshot uses that snapshot’s data file listing.
Metadata-driven partition pruning. Explain how a filter on a data column lets Iceberg identify and prune relevant partitions, even when the query does not name a partition key, and contrast this with the manual filters required by traditional Hive-style tables.
Costs of duplicate data. Explain why keeping copies of data across lakes, warehouses, and data marts becomes costly, especially in the cloud.
Direct queries and scan pruning. Describe how querying Iceberg tables directly in the data lake and pruning unnecessary files reduce analytics costs.
Modular lakehouse benefits. Explain why an Iceberg lakehouse is modular and how that differs from a monolithic data warehouse. Describe the benefits the section attributes to choosing tools for each component.
Five lakehouse layer roles. Describe the basic job of each of the five components named in the section: storage, ingestion, catalog, federation, and consumption.
Storage Layer Contents and Locations. Describe the storage layer as a scalable place for data files and Iceberg metadata, and identify cloud, on-premises, and hybrid deployment options.
Independent Storage and Compute Scaling. Explain why separating storage and compute is beneficial in the lakehouse model described here.
Ingestion layer role and modes. Describe how the ingestion layer transforms, cleans, and loads data from different systems into Iceberg tables, either in batches or continuously through streaming.
Catalog as shared entry point. Explain how a lakehouse catalog connects storage and query engines by tracking Iceberg tables and the locations of their metadata.
Shared catalog operational benefits. Explain how a shared catalog supports consistent access to Iceberg tables across tools, including atomic updates, interoperability, and portable access controls.
Cross-source data unification. Explain how the federation layer lets teams query Iceberg alongside external data sources and create unified business views.
Consumption layer use cases. Describe how the consumption layer uses data for analytics, AI, and operational workflows, including BI dashboards, model training, embedded analytics, and data APIs or applications.
Shared data across tools. Explain why a modular lakehouse and Iceberg’s open standard let teams use data products across tools without duplicating ETL, and what problems that avoids.

Cover of Architecting an Apache Iceberg Lakehouse

liveBook manning.com

Read chapter 2 on liveBook Get the book at manning.com

What it builds on

Apache Iceberg: Giving data lakes data warehouse capabilities (chapter 1) · Introduces Iceberg in the context of adding warehouse capabilities to data lakes.

Section by section

Introduction

Introduces Apache Iceberg as a table format that uses metadata to make files in a data lake behave as relational tables.

Iceberg table format role: Explain that Apache Iceberg is a community-driven table format that adds a standard metadata layer over data files, allowing collections of files in a data lake to be treated as coherent relational tables without changing file-level storage.
Why it matters: This distinction explains what Iceberg adds to a data lake without replacing its underlying files or storage.

Key terms: table format, standard metadata layer, Apache Parquet, relational tables, object storage

2.1 What does it mean that Iceberg is a table format?

The section explains Iceberg’s table-format abstraction, why it works across engines, and how it complements Parquet’s file-level storage.

Table Format and Metadata: Explain that a table format is a logical wrapper that defines how data files, schemas, partitions, and snapshots are stored, using metadata to give engines a consistent view of a dataset.
Why it matters: This distinction explains what Iceberg adds to files stored in a data lake.
Engine Independent Table Access: Explain why Iceberg’s table abstraction lets different execution engines work with the same dataset while teams retain their preferred tools.
Why it matters: A shared table view makes the lakehouse usable by teams with different processing and query tools.
Parquet Columnar Efficiency: Describe how Parquet stores values by column and why that layout benefits analytical queries.
Why it matters: Parquet explains where file-level scan efficiency comes from in the Iceberg–Parquet pairing.
Iceberg and Parquet Responsibilities: Distinguish Iceberg’s table-level responsibilities from Parquet’s file-level responsibilities and explain why the two complement each other.
Why it matters: Separating file storage from table management prevents confusing Parquet’s capabilities with Iceberg’s.

Key terms: table format, logical wrapper, metadata, schemas, partitions, snapshots, table abstraction, execution engine, shared view, Apache Parquet

2.2 Why you need a table format

This section explains why files in a data lake need a table format to become consistent, manageable, and efficient analytical tables.

Files versus table formats: Distinguish a file format from a table format: Parquet, ORC, and CSV describe records within files, but do not define a dataset spanning many files as one table. Explain that Hive added a table abstraction for SQL access, though its transactional guarantees and metadata scalability were limited.
Why it matters: This distinction explains why choosing Parquet alone does not create a lakehouse table.
Table-level consistency needs: Explain why managing a dataset as files alone makes concurrent writes and schema changes difficult. Describe the possible results of conflicting writes and the lack of built-in tracking for dataset evolution.
Why it matters: Reliable analytics requires consistency across the dataset, not just a suitable format for each file.
Metadata and query efficiency: Explain why query engines need table-level metadata to locate relevant data efficiently. Describe the section’s basic lakehouse table picture: data files hold the dataset, while metadata files help engines scan it.
Why it matters: Metadata helps an engine avoid unnecessary file inspection when planning a query.

Key terms: file formats, table format, table abstraction, Hive Metastore, consistency, partial updates, duplicate records, schema changes, centralized, query-aware metadata, directory listings

2.3 How Apache Iceberg manages metadata

The section explains how Iceberg’s layered metadata helps query engines locate table data and skip irrelevant files before scanning.

Layered Iceberg Metadata Structure: Describe the roles of table-level metadata, manifest lists, and manifests, and explain how the catalog gives query engines a stable entry point to them.
Why it matters: These layers explain how an engine finds a table’s data without treating every file as equally relevant.
Metadata Guided File Pruning: Trace how Iceberg uses metadata statistics to rule out irrelevant data before scanning, and explain why that improves query efficiency compared with searching through lake files.
Why it matters: Skipping files before a scan is the performance benefit of the metadata structure described here.

Key terms: metadata.json, manifest lists, manifests, catalog, snapshots, metadata pruning, manifest list statistics, query planning, data files

2.4 Key features of Apache Iceberg

The section explains Iceberg’s transactions, schema changes, snapshots, partitioning, and table-state controls for reliable lakehouse tables.

Concurrent table consistency: Explain how Iceberg’s ACID transactions let multiple jobs and tools read and write the same datasets safely.
Why it matters: This is why multiple engines can safely share an Iceberg table.
Controlled schema changes: Describe which schema changes the section says Iceberg supports and how they affect existing queries and downstream consumers.
Why it matters: A table’s data model can change without forcing a full rewrite or disrupting consumers.
Snapshot time travel limits: Explain what Iceberg’s snapshots let queries revisit, why that is useful, and what limits time travel.
Why it matters: Historical data is useful only when you understand what snapshots retain and what they cannot restore.
Evolving and hidden partitions: Distinguish partition evolution from hidden partitioning and explain their effect on data rewrites and queries.
Why it matters: Partitioning can serve query performance without making users manage the table’s physical layout.
Branching and tagging table states: Describe what branching and tagging table states allow teams to do with changes.
Why it matters: Teams can prepare and test table changes before promoting them.

Key terms: ACID transactions, concurrent reads and writes, schema evolution, table level, downstream consumers, snapshots, time travel, snapshot retention policies, partition evolution, hidden partitioning

2.5 Apache Iceberg: An open source standard

This section explains how Iceberg’s vendor-neutral, community-governed standard lets organizations use different tools without tying their lakehouse to one vendor.

Vendor-neutral table format: Explain why Iceberg’s status as an open standard helps organizations build long-lived data architectures. Describe how a common contract lets independent systems work with the format as tools and vendors change.
Why it matters: A vendor-neutral format gives you more freedom to change the tools around your data over time.
Community support and interoperability: Describe how Iceberg’s broad contributor community and tool integrations support an interoperable lakehouse. Give examples of different kinds of work that integrated tools can perform.
Why it matters: Broad support makes it practical to combine tools for different lakehouse workloads.

Key terms: open source standard, common contract, vendor neutrality, Apache Software Foundation, interoperable, tool integrations

2.6 Benefits of Apache Iceberg

Introduces Iceberg’s shift from treating lake data as raw files to treating it as managed tables, while noting that its benefits depend on integration with query engines and catalogs.

Managed tables over raw files: Explain how Iceberg changes the way organizations work with data lake data, from identifying a dataset as a folder of Parquet files to treating it as a managed table visible in a catalog.
Why it matters: This shift explains why adopting Iceberg involves more than continuing to manage folders of files as before.
Query engine and catalog integration: State what the section says is needed to unlock Iceberg’s full potential for performance and flexibility.
Why it matters: The potential benefits depend on how Iceberg fits into the surrounding lakehouse systems.

Key terms: raw files, managed tables, catalog, query engines, catalog systems, performance, flexibility

2.6.1 ACID transactions

This section explains how Iceberg’s ACID guarantees keep shared tables correct when multiple jobs read and write concurrently.

Four ACID guarantees: Explain what atomicity, consistency, isolation, and durability each guarantee for changes to an Iceberg table, including what happens to a conflicting write based on a stale table view.
Why it matters: These guarantees explain why concurrent changes need not leave an Iceberg table partially updated or silently overwritten.
Safe shared table access: Explain why concurrent writes are problematic without transactional guarantees and how Iceberg’s ACID guarantees reduce the need for external coordination.
Why it matters: Commit-time correctness makes a shared data lake more reliable and easier to operate.

Key terms: ACID, atomicity, consistency, isolation, durability, stale view, transactional guarantees, external locking, coordination, commit time

2.6.2 How tables evolve

This section explains how Iceberg lets long-lived tables change their schemas and partitioning without the costly rewrites associated with traditional data lakes.

Schema evolution without rewrites: Explain how Iceberg lets a table’s schema change as requirements evolve, without costly dataset rewrites or breaking existing queries.
Why it matters: Schema flexibility helps long-lived datasets keep pace with changing business requirements.
Partition strategy evolution: Distinguish Iceberg’s changeable partitioning strategies from the fixed partition schemes described for Hive.
Why it matters: Teams need not lock a long-lived table into its original partitioning choice.
Recorded schema history: Describe what Iceberg’s schema history records and why the section says it is useful.
Why it matters: A record of structural changes helps teams manage and audit tables over the long term.

Key terms: schema evolution, backward compatibility, partition evolution, partitioning strategies, schema history

2.6.3 Time travel and snapshot-based queries

Explains how Iceberg uses snapshots to query earlier table versions and why that helps with recovery and historical analysis.

Snapshot-based time travel: Explain that Iceberg can query a table as it was at a specific point in time because scanning an older snapshot uses that snapshot’s data file listing.
Why it matters: Snapshot selection is the basis for seeing earlier versions of an Iceberg table.
Recovery and historical analysis: Describe how earlier Iceberg snapshots help recover from bad or incomplete ingestion and support historical analysis without duplicate datasets, in contrast to overwritten data in traditional data lakes.
Why it matters: Earlier versions reduce the need for manual recovery and duplicate datasets.

Key terms: time travel, snapshot, data file listing, roll back, previous snapshot, versioned analytics

2.6.4 Hidden partitioning to reduce accidental full-table scans

This section explains how Iceberg’s hidden partitioning prunes files from ordinary column filters without exposing partition keys to query writers.

Metadata-driven partition pruning: Explain how a filter on a data column lets Iceberg identify and prune relevant partitions, even when the query does not name a partition key, and contrast this with the manual filters required by traditional Hive-style tables.
Why it matters: A reader can see why an ordinary timestamp filter need not trigger an unnecessary full-table scan.
Partition strategy separation: Describe how keeping partition transforms in metadata, rather than adding schema columns or reflecting partitions in directories, preserves a clean query-facing schema and lets partitioning strategies change without breaking queries or rewriting data.
Why it matters: Queries can remain simple and stable even as a table’s partitioning strategy changes.

Key terms: hidden partitioning, partition transforms, partition pruning, table metadata, logical schema, storage layout

2.6.5 Cost efficiency and query performance

This section explains how Apache Iceberg can lower lakehouse costs by reducing data copies, simplifying ingestion, and limiting the data scanned by queries.

Costs of duplicate data: Explain why keeping copies of data across lakes, warehouses, and data marts becomes costly, especially in the cloud.
Why it matters: These duplicated steps explain the costs that Iceberg is intended to reduce.
Direct queries and scan pruning: Describe how querying Iceberg tables directly in the data lake and pruning unnecessary files reduce analytics costs.
Why it matters: It connects fewer data copies and less scanning to the section’s cost and performance claim.
Unified batch and streaming ingestion: Explain how putting batch and streaming data into one Iceberg table can simplify ETL and reduce data movement.
Why it matters: Less movement and fewer copies can mean less work to operate the analytics pipeline.

Key terms: data lakes, data warehouses, data marts, ETL, Iceberg tables, query engines, prune unnecessary files, data scans, ETL workflows, streaming

2.7 Apache Iceberg lakehouse components

Introduces the modular Iceberg lakehouse, why its components can be chosen independently, and the basic role of each of its five layers.

Modular lakehouse benefits: Explain why an Iceberg lakehouse is modular and how that differs from a monolithic data warehouse. Describe the benefits the section attributes to choosing tools for each component.
Why it matters: Modularity explains why a lakehouse architect can evaluate its components separately rather than commit to one vendor’s stack.
Five lakehouse layer roles: Describe the basic job of each of the five components named in the section: storage, ingestion, catalog, federation, and consumption.
Why it matters: These five roles provide a map for discussing the lakehouse without confusing where data is stored, managed, prepared, or used.

Key terms: modular lakehouse architecture, interoperability, vendor lock-in, storage layer, ingestion layer, catalog layer, federation layer, consumption layer

2.7.1 Storage layer: Foundation of your lakehouse

This section describes what the lakehouse storage layer holds, where it can run, and why separating storage from compute matters.

Storage Layer Contents and Locations: Describe the storage layer as a scalable place for data files and Iceberg metadata, and identify cloud, on-premises, and hybrid deployment options.
Why it matters: The storage layer provides the durable foundation for both the lakehouse’s data and its table metadata.
Independent Storage and Compute Scaling: Explain why separating storage and compute is beneficial in the lakehouse model described here.
Why it matters: Independent scaling gives teams more flexibility in how they run workloads without tying those choices to storage.

Key terms: storage layer, data files, Apache Parquet, metadata files, object store, filesystem, separation of storage and compute, compute workloads

2.7.2 Ingestion layer: Feeding data into Iceberg tables

This section defines the ingestion layer, illustrates batch and streaming ways to load Iceberg tables, and explains why flexibility matters.

Ingestion layer role and modes: Describe how the ingestion layer transforms, cleans, and loads data from different systems into Iceberg tables, either in batches or continuously through streaming.
Why it matters: This identifies how data reaches Iceberg tables and the two ingestion modes the architecture supports.
Flexible ingestion benefits: Explain why supporting both batch and streaming ingestion matters for analytics and AI use cases.
Why it matters: Flexibility lets the lakehouse accommodate different arrival patterns while making data available sooner.

Key terms: ingestion layer, Iceberg tables, batch ingestion, streaming ingestion, flexible ingestion layer, modular, latency

2.7.3 Catalog layer: Your entry point to the lakehouse

This section explains the catalog’s role as the shared entry point to Iceberg tables, the benefits it provides across tools, and the kinds of catalog options available.

Catalog as shared entry point: Explain how a lakehouse catalog connects storage and query engines by tracking Iceberg tables and the locations of their metadata.
Why it matters: A shared catalog gives engines a consistent way to find and manage the same tables.
Shared catalog operational benefits: Explain how a shared catalog supports consistent access to Iceberg tables across tools, including atomic updates, interoperability, and portable access controls.
Why it matters: These shared capabilities help teams use the same datasets securely without creating tool-specific silos.
Variety of catalog options: Describe how the listed catalog options differ in emphasis, using examples from the section rather than treating them as interchangeable.
Why it matters: Recognizing their different emphases helps frame a later catalog choice.

Key terms: catalog layer, storage layer, query engines, metadata, single source of truth, atomic table updates, cross-tool interoperability, portable access controls, data silos, Hive Metastore

2.7.4 Federation layer: Modeling and accelerating data

This section explains how the federation layer connects Iceberg with other data sources and makes the combined data useful for analytics.

Cross-source data unification: Explain how the federation layer lets teams query Iceberg alongside external data sources and create unified business views.
Why it matters: Organizations can use Iceberg data alongside data that remains in databases, warehouses, and other lakehouse formats.
Consistent business metrics: Describe the semantic layer’s role in defining consistent metrics and business logic across an organization.
Why it matters: Shared metric definitions help people interpret combined data consistently.
Federation tools and acceleration: Distinguish the section’s examples of Dremio and Trino, and state how query acceleration fits into federation.
Why it matters: The examples show that federation tools can offer different amounts of built-in modeling and acceleration.

Key terms: federation layer, federated query engines, unified business views, semantic layer, business logic, consistent metrics, query acceleration, Dremio, Reflections, Trino

2.7.5 Consumption layer: Delivering business value

This section shows how the consumption layer turns Iceberg data into business value and why an open, modular lakehouse can serve multiple uses from shared data.

Consumption layer use cases: Describe how the consumption layer uses data for analytics, AI, and operational workflows, including BI dashboards, model training, embedded analytics, and data APIs or applications.
Why it matters: It connects the lakehouse architecture to the ways a business actually uses its data.
Shared data across tools: Explain why a modular lakehouse and Iceberg’s open standard let teams use data products across tools without duplicating ETL, and what problems that avoids.
Why it matters: Shared access helps different teams use the same trusted data without building separate copies for every workload.

Key terms: consumption layer, business intelligence (BI), AI and machine learning (ML), embedded analytics, data APIs, modular, open standard, data products, single, trusted source of truth, duplicating ETL

2.8 Summary

Where this is used later

Apache Parquet (chapter 5) · The distinction between Iceberg as a table format and the files holding table data.
Hadoop (chapter 5) · The distinction between the Iceberg table format and its underlying storage.
The role of the catalog in Apache Iceberg lakehouses (chapter 7) · Provides the basic catalog role that the later catalog discussion develops.
Verifying data storage in MinIO (chapter 3) · Iceberg’s metadata structure
Reading Iceberg tables with Dremio (chapter 3) · Introduces Iceberg table metadata.
Defining ingestion requirements (chapter 4) · Iceberg’s capabilities provide context for supporting different ingestion needs.
Using branching and tagging for governance (chapter 11) · Applies branching and tagging to governance.
Databricks Unity Catalog (chapter 7) · Iceberg’s role as an open standard that multiple tools can support.

Concepts explained in this chapter

Apache IcebergAn open table format for managing large analytic datasets with metadata-driven structure, snapshots, schema evolution, and transactional write capabilities.
partition pruningPartition pruning is the elimination of data partitions that cannot satisfy a query's filters before or during data access.
open table formatA table format is a specification for representing a logical table and its metadata independently of the underlying physical data storage.
hidden partitioningA partitioning method in which partition values are derived from transforms and are not exposed as query columns.
ACIDThe transaction guarantees of atomicity, consistency, isolation, and durability that preserve reliable database state changes.
metadataMetadata is structured information that describes the location, structure, contents, versions, or other properties of data.
partition evolutionThe ability to change a table's partition strategy over time without rewriting existing data.
schema evolutionSchema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
file-level metadataMetadata describing individual data files, including their locations and query-relevant statistics.
manifest listA manifest list is an Apache Iceberg metadata structure that represents a snapshot by listing its manifest files and summary statistics.
metadata designThe organization and representation of metadata used to describe, locate, version, and plan access to table data.
metadata.jsonmetadata.json is an Apache Iceberg table metadata file that records table versions, schemas, partition specifications, and related table state.
table metadataTable metadata is information describing a table's schema, data files, versions, partitioning, and related structural state.
consumption layerA lakehouse architectural layer that exposes data to business-intelligence, artificial-intelligence, and real-time applications.
ParquetApache Parquet is an open columnar file format designed for efficient storage and analytical processing of large datasets.
lakehouse catalogA lakehouse architectural component that tracks and governs table metadata and enables data systems to discover and access tables.
ETLA workflow that extracts data from sources, transforms it, and loads it into a target data system.
ingestion layerAn ingestion layer is a system layer that collects data from source systems, transforms and cleans it, and loads it into storage tables.
data lakeA data lake is a storage-centered data platform that retains data in flexible formats on relatively inexpensive storage for varied downstream processing.
snapshot-based architectureAn architecture that represents table state through immutable or versioned snapshots of metadata and data references.
snapshotA consistent versioned representation of a table's state, including the data files that constitute that state.
data lakehouseA data lakehouse is a data architecture that combines the openness and flexible storage of a data lake with the governance and analytical capabilities of a data warehouse.
storage abstractionA storage layer is a lakehouse architectural layer that stores data on cost-effective storage systems.
vendor lock-inTool lock-in is dependence on a particular software tool or vendor that makes changing tools difficult or costly.
federated query engineA federated query engine is a query engine that executes queries across multiple heterogeneous data sources without requiring all data to be stored in one system.
semantic layerA semantic layer is a governed modeling layer that defines consistent business entities, dimensions, and metrics across data sources.
batch pipelineBatch ingestion is the process of loading accumulated data into a target system in discrete processing runs.
columnar dataA columnar format stores values from the same field together to support efficient analytical reading and compression.
HiveApache Hive is a data warehouse infrastructure that provides SQL-based querying and table management over distributed data stored in file systems or object stores.
query planningQuery planning is the process of determining an execution strategy for a query, including which files or partitions must be read.
snapshot retention policyA snapshot retention policy defines which table snapshots remain available and the conditions under which older snapshots are expired.
FivetranFivetran is a managed data-integration platform that automates movement of data from sources to destinations.
FlinkApache Flink is an open-source distributed processing framework for stateful batch and stream computations.
Kafka ConnectKafka Connect is a framework for continuously transferring data between Apache Kafka and external systems through connectors.
SparkApache Spark is an open-source distributed processing framework for large-scale batch, streaming, and analytical workloads.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.