liveBook · Manning

Chapter 9: Understanding the consumption layer

What you will learn in chapter 9 of Architecting an Apache Iceberg Lakehouse

The introduction explains how the consumption layer turns a lakehouse’s technical foundation into useful outcomes and why Iceberg’s openness changes how tools access data.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains how the consumption layer turns a lakehouse’s technical foundation into useful outcomes and why Iceberg’s openness changes how tools access data.

Key terms: consumption layer, openness, table-format portability, tool lock-in

9.1 Revisiting the benefits of the lakehouse for consumption

This section explains how an Iceberg lakehouse can reduce fragmented data consumption by providing shared data and definitions across tools.

Key terms: data silos, Apache Iceberg, Iceberg tables, consumption layer, shared semantic layer, business metrics, shared definitions, query federation, long tail, ETL jobs

9.1.1 Connecting the lakehouse to the people

The section explains how curated Iceberg data, shared business definitions, and federation make a lakehouse useful to people and tools without duplicating data or business rules.

Key terms: bronze, silver, and gold layers, semantic layer, business-ready datasets, authoritative, federation layer, standardized interfaces, shared semantics, tool-specific silos

9.2 Revisiting requirements from our audit

This section explains why consumption-layer choices must reflect the requirements and operational priorities identified in the platform audit.

Key terms: consumption layer, platform audit, low-latency query paths, semantic layers, data masking, audit trails

9.2.1 Interpreting requirements for consumption

This section translates audit findings into performance, access, governance, consistency, and cost requirements for the lakehouse consumption layer.

Key terms: query latency, predicate pushdown, caching, incremental results, access patterns, AI-native interfaces, Model Context Protocol (MCP) servers, curated agent skills, curated Iceberg datasets, row-level security

9.2.2 Requirements for BI tools

The section identifies the performance, semantic, connectivity, governance, and workflow requirements a lakehouse must meet to support effective BI tools.

Key terms: performance and responsiveness, caching, materialized views, federation engines, semantic consistency, centralized semantic layer, metrics, interoperability, open interfaces, JDBC

9.2.3 Requirements for interactive notebook environments

This section identifies what a lakehouse consumption layer must provide for interactive notebooks to support exploration and repeatable workflows.

Key terms: low-latency access, predicate pushdown, column pruning, query planning, JDBC, ODBC, Arrow Flight, Arrow, native Iceberg connectors, multiple compute engines

9.2.4 Requirements for AI and specialized data consumption tools

This section identifies the distinct access, performance, security, and consistency requirements that ML, AI, real-time applications, embedded analytics, and external sharing place on the lakehouse consumption layer.

Key terms: ML, high-throughput access, stable snapshots, schema evolution, AI agents, context-aware queries, low latency, high concurrency, user- and role-level permissions, streaming applications

9.3 Open interfaces for seamless consumption

Open interfaces let varied consumption tools access governed lakehouse data while common access layers keep definitions and policies consistent.

Key terms: open interfaces, JDBC, ODBC, Arrow Flight, Model Context Protocol (MCP), consumption tools, common access layers, federation layer, semantic layer, business logic

9.3.1 JDBC and ODBC

This section explains how JDBC and ODBC connect consumption tools to Iceberg-capable query engines, why they remain useful, and where they fall short.

Key terms: JDBC, ODBC, consumption tools, query engines, federation layer, connectors, enterprise access controls, centralized federation engine, access policies, metric definitions

9.3.2 Arrow Flight

This section explains how Arrow Flight provides fast, columnar data access for lakehouse consumption tools and how it fits alongside traditional interfaces.

Key terms: Arrow Flight, Apache Arrow, columnar, gRPC, JDBC, ODBC, Arrow Database Connectivity (ADBC), query engine, Iceberg tables, governed datasets

9.3.3 Model Context Protocol (MCP)

The section explains how MCP lets AI assistants consume live lakehouse data and invoke governed tools, extending beyond traditional data-access interfaces.

Key terms: Model Context Protocol (MCP), LLMs, two-way interaction, structured responses, callable tools, MCP servers, federation engine, semantic definitions, role-based access controls

9.4 Business intelligence tools in the lakehouse

Explains how BI tools consume Iceberg lakehouse data through federated interfaces while shared semantics and centralized controls keep dashboards consistent and manageable.

Key terms: Iceberg tables, open, federated interfaces, federation layer, BI dashboards, semantic layers, shared definitions of metrics, access controls, caching, query optimization

9.4.1 Open source BI tools

The section introduces open source BI tools for a lakehouse and identifies what to check when choosing one.

Key terms: federation layer, query engines, Iceberg tables, caching, centralized metric definitions, Apache Superset, Metabase, Lightdash, Redash, RBAC

9.4.2 Commercial BI tools

This section explains how commercial BI tools access governed Iceberg data through lakehouse interfaces and what to consider when choosing among them.

Key terms: commercial BI tools, open interfaces, federation layer, governed Iceberg tables, Tableau, Power BI, Looker, LookML, semantic modeling, row-level security

9.4.3 Tools for AI and machine learning workloads

This section explains how an Iceberg lakehouse supports AI and machine learning workflows, the tools that consume its data, and the capabilities those tools require.

Key terms: lakehouse, Iceberg tables, data versioning, schema evolution, time travel, notebook platforms, feature stores, model training and orchestration platforms, lineage, curated Iceberg data

9.5 Choosing the right consumption tools: Ten illustrated scenarios

This section frames consumption-tool selection as a context-dependent decision that must fit the broader lakehouse architecture.

Key terms: consumption tools, semantic layer, federation layer, catalog layer

9.5.1 Startup with a data science focus

This scenario shows how a small, Python-focused AI startup can choose a lightweight notebook-based consumption layer for its Iceberg lakehouse.

Key terms: consumption layer, JupyterLab, DuckDB, PyIceberg, MLflow, catalog-based reads, Spark sessions, snapshot capabilities, versioned access

9.5.2 Large financial institution with strict governance

This scenario shows how a global bank can let teams keep their preferred BI tools while governing metrics and data access centrally.

Key terms: federated semantic layer, JDBC/ODBC, metric governance, Iceberg tables, query engine, RBAC, row- and column-level security, catalog policies, audit logs

9.5.3 Mid-sized e-commerce platform building embedded analytics

This section shows how a mid-sized e-commerce company can deliver responsive, governed analytics inside merchant dashboards using an embedded BI tool and an Iceberg lakehouse.

Key terms: embedded analytics, Metabase, iframe or API-based embedding, white-label customization, multi-tenant data isolation, partitioning, metadata tagging, row-level security controls, access filters, JDBC connectors

9.5.4 Decentralized media organization enabling self-service analytics

This scenario shows how a media company can give its brands a choice of BI tools while keeping metrics, access, and query visibility centrally governed.

Key terms: BI tool, semantic layer, dbt, canonical metrics, federation engines, Iceberg, federation layer, brand-specific access policies, JDBC, ODBC

9.5.5 Government agency balancing public transparency and internal control

This scenario shows how a government agency can offer public access to selected lakehouse data while keeping sensitive data controlled and historical results reproducible.

Key terms: Apache Superset, Looker Studio, cached extracts, controlled query endpoints, REST API, internal, public, restricted, metadata tagging, Trino

9.5.6 Healthcare provider with compliance and data locality constraints

This scenario shows how a healthcare provider can support clinical dashboards and research notebooks while keeping patient data local, access restricted, and analytics governed.

Key terms: Iceberg tables, region-aware catalog layer, locality requirements, federation engine, query planning, Power BI, clinical dashboards, JupyterHub, research notebooks, region bound

9.5.7 Logistics company unifying real-time operations and historical analysis

This scenario shows how a logistics company combines live operational dashboards and historical planning tools over a consistent Iceberg lakehouse.

Key terms: live-query capabilities, Looker, semantic layer, Arrow Flight, DuckDB, streaming ingestion, Iceberg tables, older partitions, federation layer, single source of truth

9.5.8 SaaS company offering customizable data access to clients

This section illustrates how a SaaS company can give clients dashboards and API access to shared Iceberg data while preserving tenant isolation, governance, and scalability.

Key terms: Superset, RESTful API, federated query engine, Iceberg tables, tenant-aware partitioning, metadata tagging, central catalog, tenant ID, secure tokens, query result caching

9.5.9 Nonprofit organization supporting collaborative research

This section shows how a climate-research nonprofit combines consumption tools and lakehouse controls to let partners collaborate on shared data while preserving reproducibility and governance.

Key terms: JupyterHub, Apache Superset, Arrow Flight, DuckDB, read-only access, hosted notebooks, views, snapshots, time-based queries, data lineage

9.5.10 Manufacturing company enabling predictive maintenance

This section illustrates how a manufacturer combines consumption tools with a shared Iceberg lakehouse to support predictive maintenance.

Key terms: predictive maintenance, Grafana, Apache Superset, JupyterHub, Iceberg tables, federated engine, catalog, governed access, feature engineering

9.6 Summary

Concepts explained in this chapter

← Chapter 8: Designing the federation layer Chapter 10: Maintaining an Iceberg lakehouse →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.