Chapter 9: Understanding the consumption layer
The introduction explains how the consumption layer turns a lakehouse’s technical foundation into useful outcomes and why Iceberg’s openness changes how tools access data.
After this chapter you will be able to
- Consumption layer business value. Explain how the consumption layer connects the lakehouse’s storage, ingestion, catalog, and federation foundation to practical uses such as dashboards, exploration, queries, and machine learning.
- Tools coming to data. Distinguish traditional consumption through data replication and tool-specific constraints from the Iceberg lakehouse approach, where portable, open tables let tools access data in place.
- Shared lakehouse data foundation. Explain how fragmented departmental tools lead to repeated data copies, complex ETL, and inconsistent results, and how an Iceberg lakehouse changes that pattern for consumers.
- Curated data and shared meaning. Explain how tiered curation and a semantic layer turn Iceberg tables into trusted, business-ready datasets with consistent meanings across an organization.
- Federated reuse across consumers. Describe how federation lets different consumers use the same curated datasets and shared semantics, and explain why this reduces copies and repeated business-rule definitions.
- Audit requirements in consumption design. Explain how audit findings should guide the choice of tools for analyzing, visualizing, or modeling lakehouse data. Relate fast access to curated datasets to low-latency query paths and reliable semantic layers, and compliance needs to enforced masking or audit trails.
- Interactive performance requirements. Explain why dashboards and interactive analyses need low query latency, and describe the query-engine capabilities and cost trade-off the section identifies.
- Human and AI access patterns. Describe why the consumption layer needs multiple access paths for people and AI agents, including what AI-friendly interfaces make possible.
- Consistent consumption governance. Explain what governance controls must apply when data is consumed and why inheriting controls from the catalog is preferable.
- Shared metric definitions. Explain how a shared semantic layer affects reports and decisions made through different consumption tools.
- Responsive BI dashboards. Explain why BI dashboards need fast, up-to-date results and how the lakehouse can support efficient queries against Iceberg tables.
- Shared metric definitions. Explain why BI metric definitions should live in a centralized semantic layer rather than being recreated separately in each tool.
- Open BI connectivity. Describe how open interfaces let an organization use multiple BI platforms against the same lakehouse datasets.
- Governed BI data access. Explain how the consumption layer should limit BI users to authorized data.
- Responsive notebook data access. Explain why notebook users need direct, low-latency access to raw and transformed data, and name the access optimizations the section identifies.
- Open notebook data interfaces. Describe why a lakehouse should expose standard data access interfaces for notebook libraries, including the performance-related options mentioned here.
- Flexible notebook compute. Explain why notebook access must work with multiple compute frameworks without unnecessary data movement, and how notebook work can extend beyond exploration.
- Secure reproducible notebook work. Describe the security safeguards notebooks require and why versioned, snapshot-queryable data matters when notebook prototypes become operational workflows.
- Reproducible ML training access. Explain why ML training needs high-throughput access to stable, versioned data and how an Iceberg snapshot keeps features and labels consistent during a training run.
- Low-latency governed AI queries. Describe the kinds of data AI agents consume and the query performance and permission requirements their interactions impose.
- Recent and historical operational access. Explain why real-time applications need both fresh updates and historical context, and describe the consumption approaches the section gives for exposing them.
- Standardized connections for consumption. Explain how open interfaces connect dashboards, notebooks, and machine learning pipelines to lakehouse data without bespoke integrations or brittle extracts.
- Consistent governed access. Explain why routing consumption through common access layers helps users get consistent results across tools.
- Consumption to query engine path. Trace how a BI or other consumption tool uses JDBC or ODBC to query Iceberg tables through a query engine, sometimes via a federation layer.
- Columnar data transfer. Explain how Arrow Flight transfers data and why it can outperform JDBC and ODBC when tools retrieve large analytical datasets.
- Lakehouse access through Flight. Describe how Flight and ADBC connect consumption applications to lakehouse query engines and Iceberg data.
- MCP for AI consumption. Explain why MCP is useful alongside JDBC and Arrow Flight: it gives LLMs a standardized way to interact with external data, tools, and services rather than only access structured data.
- Governed lakehouse tools. Describe how a lakehouse can support MCP by exposing queries, metrics, or actions as callable tools, and illustrate how an assistant could use them while respecting access controls and shared definitions.
- BI access without storage coupling. Explain how Iceberg tables and open, federated interfaces let BI dashboards query data in place without depending on its physical storage location.
- Federated BI access. Explain how an open source BI tool can reach Iceberg data through a query engine or federation layer rather than accessing the tables on its own.
- Open source BI selection. Describe the checks the section recommends when evaluating an open source BI tool for a lakehouse.
- Commercial BI lakehouse access. Explain how commercial BI tools can use open interfaces and federated query engines to access governed Iceberg tables without duplicating data.
- Centralized AI data foundation. Explain how a lakehouse supports AI experimentation and production without repeatedly exporting data to separate ML platforms.
- Notebook feature and training tools. Describe the roles of notebooks, feature stores, and training or orchestration platforms when they use lakehouse data.
- Requirements for AI consumption. State the lakehouse capabilities needed to support the AI tools described in this section.
- Contextual consumption tool selection. Explain why no single consumption tool fits every organization and how business goals, technical requirements, team preferences, and integration with existing lakehouse layers shape the choice.
- Notebook-centered consumption choice. Justify why the startup builds its consumption layer around JupyterLab rather than a traditional BI tool. Describe how DuckDB, PyIceberg, and MLflow support its data science work.
- Reproducible Iceberg data access. Explain how the startup's notebooks access Iceberg data and how its training workflows remain reproducible.
- Shared metrics across BI tools. Explain why the bank chooses a federated semantic layer instead of one mandatory BI tool, and how it provides consistent metrics across teams’ tools.
- Governed and traceable access. Trace how the semantic layer reaches Iceberg data and describe where access restrictions and auditability fit in the architecture.
- Embedded BI tool choice. Justify the choice of Metabase for merchant-facing dashboards in terms of embedding, customization, and operational needs.
- Merchant data isolation. Trace how the architecture supports tenant isolation from Iceberg tables through dashboard queries.
- Flexible tools and shared metrics. Explain how the company preserves each brand’s choice of BI tool while keeping metric definitions consistent across brands.
- Governed access and query visibility. Describe how the shared lakehouse supports brand-specific access and centralized visibility into BI queries.
- Public and internal access paths. Describe how the agency uses different consumption paths for internal analytics and public access. Explain that public dashboards and APIs expose curated, nonsensitive data rather than unrestricted access to internal tables.
- Data domains and access enforcement. Explain how the agency separates data into domains and enforces access policies for internal and public queries.
- Region-aware patient data access. Trace how the lakehouse supports data locality and restricted access: patient data is stored in Iceberg tables partitioned by geography and department, a region-aware catalog tags locality requirements, and Dremio enforces those policies during query planning.
- Clinical and research tool choices. Describe why the provider uses Power BI for clinical dashboards and JupyterHub for research notebooks, and how each is deployed to support compliant access.
- Dual-horizon consumption tool choices. Explain how the company matches consumption tools to live operations, historical planning, and specialized analysis.
- Shared lakehouse across time horizons. Trace how streamed Iceberg data and the federation layer support both recent operational queries and historical analysis while keeping views consistent.
- Shared dashboard and API access. Describe how the company serves business users through embedded Superset dashboards and developers through an API, with both interfaces connecting to a federated query engine over Iceberg tables.
- Tenant isolation across layers. Explain how the design limits clients to their own data through tenant-aware tables, a central catalog, authenticated tenant-based query restrictions, and secure API tokens.
- Researcher access and tools. Describe how the nonprofit gives internal and external researchers different ways to work with shared datasets. Explain how those choices accommodate varied partner tools while limiting external access.
- Reproducible governed research workflows. Explain how snapshots, catalog access policies, lineage tracking, and audit information support reproducible collaboration while protecting sensitive findings.
- Predictive maintenance tool roles. Explain why the manufacturer uses different tools for equipment monitoring, exploratory analysis, and failure-prediction modeling.
- Shared governed data foundation. Describe how the tools draw on a common lakehouse data layer and why that helps operational monitoring and longer-term modeling stay aligned.
What it builds on
- Consumption layer: Delivering business value (chapter 2) · Introduces consumption as a lakehouse layer.
- Apache Iceberg: An open source standard (chapter 2) · Establishes Iceberg’s role as an open standard.
- Ensuring consistent semantics and business logic (chapter 8) · Introduces consistent semantics and business logic as a federation requirement.
- What data federation is and why it matters (chapter 8) · Introduces data federation and why it matters.
- Defining consumption requirements (chapter 4) · The consumption requirements established during the earlier requirements work.
- Catalog-based access control (chapter 7) · Assumes the catalog can participate in access control.
- Providing seamless connectivity for analytics tools (chapter 8) · The need for seamless connectivity for analytics tools.
- Semantic layer considerations in Trino (chapter 8) · The earlier discussion of semantic layers provides context for canonical definitions.
Section by section
Introduction
The introduction explains how the consumption layer turns a lakehouse’s technical foundation into useful outcomes and why Iceberg’s openness changes how tools access data.
- Consumption layer business value: Explain how the consumption layer connects the lakehouse’s storage, ingestion, catalog, and federation foundation to practical uses such as dashboards, exploration, queries, and machine learning.
Why it matters: It keeps the architecture focused on what people can do with the data. - Tools coming to data: Distinguish traditional consumption through data replication and tool-specific constraints from the Iceberg lakehouse approach, where portable, open tables let tools access data in place.
Why it matters: This explains why an open lakehouse can support varied tool choices without making a separate copy for each one.
Key terms: consumption layer, openness, table-format portability, tool lock-in
9.1 Revisiting the benefits of the lakehouse for consumption
This section explains how an Iceberg lakehouse can reduce fragmented data consumption by providing shared data and definitions across tools.
- Shared lakehouse data foundation: Explain how fragmented departmental tools lead to repeated data copies, complex ETL, and inconsistent results, and how an Iceberg lakehouse changes that pattern for consumers.
Why it matters: A shared foundation lets teams use different tools without repeatedly copying and reshaping the same data. - Shared business metric definitions: Explain why sharing data alone does not guarantee consistent reports and what shared definitions contribute to consumption.
Why it matters: Reports can disagree even when they draw on shared data if their metrics mean different things. - Access to external data: Explain why data outside the primary warehouse creates a consumption challenge and what query federation avoids.
Why it matters: Access to data beyond the primary warehouse need not depend on another chain of copying jobs.
Key terms: data silos, Apache Iceberg, Iceberg tables, consumption layer, shared semantic layer, business metrics, shared definitions, query federation, long tail, ETL jobs
9.1.1 Connecting the lakehouse to the people
The section explains how curated Iceberg data, shared business definitions, and federation make a lakehouse useful to people and tools without duplicating data or business rules.
- Curated data and shared meaning: Explain how tiered curation and a semantic layer turn Iceberg tables into trusted, business-ready datasets with consistent meanings across an organization.
Why it matters: People and systems need both trustworthy data and a shared interpretation of it. - Federated reuse across consumers: Describe how federation lets different consumers use the same curated datasets and shared semantics, and explain why this reduces copies and repeated business-rule definitions.
Why it matters: Shared data and definitions make analytics more accessible without sacrificing consistent governance.
Key terms: bronze, silver, and gold layers, semantic layer, business-ready datasets, authoritative, federation layer, standardized interfaces, shared semantics, tool-specific silos
9.2 Revisiting requirements from our audit
This section explains why consumption-layer choices must reflect the requirements and operational priorities identified in the platform audit.
- Audit requirements in consumption design: Explain how audit findings should guide the choice of tools for analyzing, visualizing, or modeling lakehouse data. Relate fast access to curated datasets to low-latency query paths and reliable semantic layers, and compliance needs to enforced masking or audit trails.
Why it matters: A consumption tool delivers value only if it meets the access and governance needs the organization actually identified.
Key terms: consumption layer, platform audit, low-latency query paths, semantic layers, data masking, audit trails
9.2.1 Interpreting requirements for consumption
This section translates audit findings into performance, access, governance, consistency, and cost requirements for the lakehouse consumption layer.
- Interactive performance requirements: Explain why dashboards and interactive analyses need low query latency, and describe the query-engine capabilities and cost trade-off the section identifies.
Why it matters: Fast answers make interactive tools useful, but the speed must remain affordable. - Human and AI access patterns: Describe why the consumption layer needs multiple access paths for people and AI agents, including what AI-friendly interfaces make possible.
Why it matters: Different consumers can use the same curated data without being forced into one interface. - Consistent consumption governance: Explain what governance controls must apply when data is consumed and why inheriting controls from the catalog is preferable.
Why it matters: A useful access path must not become a way around the lakehouse's policies. - Shared metric definitions: Explain how a shared semantic layer affects reports and decisions made through different consumption tools.
Why it matters: Teams can spend less time reconciling reports when their tools use the same definitions. - Compute usage management: Describe how the consumption layer should help manage resource-heavy workloads and their costs.
Why it matters: Seeing and separating expensive workloads helps teams use shared compute responsibly.
Key terms: query latency, predicate pushdown, caching, incremental results, access patterns, AI-native interfaces, Model Context Protocol (MCP) servers, curated agent skills, curated Iceberg datasets, row-level security
9.2.2 Requirements for BI tools
The section identifies the performance, semantic, connectivity, governance, and workflow requirements a lakehouse must meet to support effective BI tools.
- Responsive BI dashboards: Explain why BI dashboards need fast, up-to-date results and how the lakehouse can support efficient queries against Iceberg tables.
Why it matters: A dashboard is less useful when users must wait for it or cannot rely on its timeliness. - Shared metric definitions: Explain why BI metric definitions should live in a centralized semantic layer rather than being recreated separately in each tool.
Why it matters: Shared definitions prevent departments from making decisions based on conflicting versions of the same metric. - Open BI connectivity: Describe how open interfaces let an organization use multiple BI platforms against the same lakehouse datasets.
Why it matters: Open connectivity accommodates different teams’ BI tools without creating separate copies of their data. - Governed BI data access: Explain how the consumption layer should limit BI users to authorized data.
Why it matters: Users can trust shared BI access only when it does not expose data they should not see. - Flexible collaborative BI workflows: Describe the BI activities beyond viewing static dashboards and the lakehouse support that makes them easier.
Why it matters: Clear schemas help BI teams create and share useful analyses more independently.
Key terms: performance and responsiveness, caching, materialized views, federation engines, semantic consistency, centralized semantic layer, metrics, interoperability, open interfaces, JDBC
9.2.3 Requirements for interactive notebook environments
This section identifies what a lakehouse consumption layer must provide for interactive notebooks to support exploration and repeatable workflows.
- Responsive notebook data access: Explain why notebook users need direct, low-latency access to raw and transformed data, and name the access optimizations the section identifies.
Why it matters: Responsive queries let practitioners explore and revise their work without waiting on every notebook cell. - Open notebook data interfaces: Describe why a lakehouse should expose standard data access interfaces for notebook libraries, including the performance-related options mentioned here.
Why it matters: Open interfaces allow practitioners to use familiar notebook libraries against lakehouse data. - Flexible notebook compute: Explain why notebook access must work with multiple compute frameworks without unnecessary data movement, and how notebook work can extend beyond exploration.
Why it matters: Compute flexibility lets a notebook move from experimentation toward a production workflow without needless data copying. - Reusable semantic definitions: Explain how notebook users can reuse canonical metrics while retaining the freedom to write their own transformations.
Why it matters: Shared metric definitions reduce inconsistencies between notebooks and other analytical tools. - Secure reproducible notebook work: Describe the security safeguards notebooks require and why versioned, snapshot-queryable data matters when notebook prototypes become operational workflows.
Why it matters: Notebook experiments are more useful in production when sensitive access is controlled and results can be reproduced.
Key terms: low-latency access, predicate pushdown, column pruning, query planning, JDBC, ODBC, Arrow Flight, Arrow, native Iceberg connectors, multiple compute engines
9.2.4 Requirements for AI and specialized data consumption tools
This section identifies the distinct access, performance, security, and consistency requirements that ML, AI, real-time applications, embedded analytics, and external sharing place on the lakehouse consumption layer.
- Reproducible ML training access: Explain why ML training needs high-throughput access to stable, versioned data and how an Iceberg snapshot keeps features and labels consistent during a training run.
Why it matters: Stable training data makes model development repeatable without freezing the evolving table. - Low-latency governed AI queries: Describe the kinds of data AI agents consume and the query performance and permission requirements their interactions impose.
Why it matters: An AI-facing consumption layer has to serve relevant results quickly without bypassing access controls. - Recent and historical operational access: Explain why real-time applications need both fresh updates and historical context, and describe the consumption approaches the section gives for exposing them.
Why it matters: Fraud detection and recommendations need to consider both what just happened and what happened before. - Secure embedded analytics endpoints: Describe how dashboards and customer-facing applications consume structured insights and what their endpoints must enforce.
Why it matters: Application access is useful only if each consumer sees the data they are authorized to see. - Portable controlled external sharing: Describe the access and portability requirements for sharing lakehouse data with external organizations.
Why it matters: Partners can use shared data while the organization retains control and avoids tool lock-in.
Key terms: ML, high-throughput access, stable snapshots, schema evolution, AI agents, context-aware queries, low latency, high concurrency, user- and role-level permissions, streaming applications
9.3 Open interfaces for seamless consumption
Open interfaces let varied consumption tools access governed lakehouse data while common access layers keep definitions and policies consistent.
- Standardized connections for consumption: Explain how open interfaces connect dashboards, notebooks, and machine learning pipelines to lakehouse data without bespoke integrations or brittle extracts.
Why it matters: Standard connections make it easier to support current tools and adopt new ones. - Consistent governed access: Explain why routing consumption through common access layers helps users get consistent results across tools.
Why it matters: A shared access path helps users trust that definitions and policies remain consistent across tools.
Key terms: open interfaces, JDBC, ODBC, Arrow Flight, Model Context Protocol (MCP), consumption tools, common access layers, federation layer, semantic layer, business logic
9.3.1 JDBC and ODBC
This section explains how JDBC and ODBC connect consumption tools to Iceberg-capable query engines, why they remain useful, and where they fall short.
- Consumption to query engine path: Trace how a BI or other consumption tool uses JDBC or ODBC to query Iceberg tables through a query engine, sometimes via a federation layer.
Why it matters: It clarifies how familiar applications reach lakehouse data without connecting directly to Iceberg tables. - Standard connector advantages: Explain why JDBC and ODBC remain common choices for lakehouse consumption and how a centralized federation engine can extend their benefits.
Why it matters: Standard connectors make lakehouse data accessible to existing tools while supporting consistent governance across them. - Synchronous interface limitations: Describe the workloads and deployment conditions where JDBC and ODBC can be less suitable.
Why it matters: These limits help explain why standard database connectors may not meet every consumption need.
Key terms: JDBC, ODBC, consumption tools, query engines, federation layer, connectors, enterprise access controls, centralized federation engine, access policies, metric definitions
9.3.2 Arrow Flight
This section explains how Arrow Flight provides fast, columnar data access for lakehouse consumption tools and how it fits alongside traditional interfaces.
- Columnar data transfer: Explain how Arrow Flight transfers data and why it can outperform JDBC and ODBC when tools retrieve large analytical datasets.
Why it matters: The transfer format helps explain why Flight suits data-intensive analytics and interactive applications. - Lakehouse access through Flight: Describe how Flight and ADBC connect consumption applications to lakehouse query engines and Iceberg data.
Why it matters: This places the fast transport in the consumption architecture without confusing it with the lakehouse's governance controls. - Complementary consumption interfaces: Distinguish Flight's role from that of JDBC and ODBC in a lakehouse.
Why it matters: Choosing a faster path for demanding tools need not remove broadly compatible interfaces.
Key terms: Arrow Flight, Apache Arrow, columnar, gRPC, JDBC, ODBC, Arrow Database Connectivity (ADBC), query engine, Iceberg tables, governed datasets
9.3.3 Model Context Protocol (MCP)
The section explains how MCP lets AI assistants consume live lakehouse data and invoke governed tools, extending beyond traditional data-access interfaces.
- MCP for AI consumption: Explain why MCP is useful alongside JDBC and Arrow Flight: it gives LLMs a standardized way to interact with external data, tools, and services rather than only access structured data.
Why it matters: This distinction explains why an AI-facing lakehouse may need more than a conventional query interface. - Governed lakehouse tools: Describe how a lakehouse can support MCP by exposing queries, metrics, or actions as callable tools, and illustrate how an assistant could use them while respecting access controls and shared definitions.
Why it matters: Making capabilities callable is useful only if AI access also preserves the lakehouse’s governance boundaries.
Key terms: Model Context Protocol (MCP), LLMs, two-way interaction, structured responses, callable tools, MCP servers, federation engine, semantic definitions, role-based access controls
9.4 Business intelligence tools in the lakehouse
Explains how BI tools consume Iceberg lakehouse data through federated interfaces while shared semantics and centralized controls keep dashboards consistent and manageable.
- BI access without storage coupling: Explain how Iceberg tables and open, federated interfaces let BI dashboards query data in place without depending on its physical storage location.
Why it matters: It explains how dashboards can remain stable as the lakehouse’s storage changes. - Shared metrics and centralized controls: Describe how semantic layers and centralized controls support consistent, governed BI consumption across visualization platforms.
Why it matters: Teams can use different visualizations without giving up consistent metrics or centralized management.
Key terms: Iceberg tables, open, federated interfaces, federation layer, BI dashboards, semantic layers, shared definitions of metrics, access controls, caching, query optimization
9.4.1 Open source BI tools
The section introduces open source BI tools for a lakehouse and identifies what to check when choosing one.
- Federated BI access: Explain how an open source BI tool can reach Iceberg data through a query engine or federation layer rather than accessing the tables on its own.
Why it matters: It clarifies where a BI tool fits in the lakehouse rather than treating it as the query engine. - Open source tool strengths: Distinguish the strengths the section attributes to Superset, Metabase, and Lightdash or Redash.
Why it matters: The examples show why different teams might prefer different open source BI tools. - Open source BI selection: Describe the checks the section recommends when evaluating an open source BI tool for a lakehouse.
Why it matters: These checks turn a tool comparison into an assessment of fit with the existing lakehouse.
Key terms: federation layer, query engines, Iceberg tables, caching, centralized metric definitions, Apache Superset, Metabase, Lightdash, Redash, RBAC
9.4.2 Commercial BI tools
This section explains how commercial BI tools access governed Iceberg data through lakehouse interfaces and what to consider when choosing among them.
- Commercial BI lakehouse access: Explain how commercial BI tools can use open interfaces and federated query engines to access governed Iceberg tables without duplicating data.
Why it matters: This connection lets organizations use familiar BI tools while keeping data and governance in the lakehouse. - Platform strengths and semantics: Distinguish the strengths this section highlights for Tableau, Power BI, and Looker, including their approaches to permissions, governance, or shared metrics.
Why it matters: These distinctions help match a BI platform to users’ experience and governance needs. - Commercial BI selection criteria: Describe the factors to check when incorporating a commercial BI tool into a lakehouse.
Why it matters: A capable dashboard is useful only if it also fits the lakehouse’s connections, governance, deployment, and workload needs.
Key terms: commercial BI tools, open interfaces, federation layer, governed Iceberg tables, Tableau, Power BI, Looker, LookML, semantic modeling, row-level security
9.4.3 Tools for AI and machine learning workloads
This section explains how an Iceberg lakehouse supports AI and machine learning workflows, the tools that consume its data, and the capabilities those tools require.
- Centralized AI data foundation: Explain how a lakehouse supports AI experimentation and production without repeatedly exporting data to separate ML platforms.
Why it matters: This is why the lakehouse can serve AI work without becoming another source of duplicated data. - Notebook feature and training tools: Describe the roles of notebooks, feature stores, and training or orchestration platforms when they use lakehouse data.
Why it matters: These tool categories show how shared lakehouse data can serve successive stages of an ML workflow. - Vector search and agent access: Distinguish how vector stores and AI agents use lakehouse data for AI applications.
Why it matters: It shows that lakehouse consumption extends beyond conventional dashboards and model training. - Requirements for AI consumption: State the lakehouse capabilities needed to support the AI tools described in this section.
Why it matters: Tool connectivity alone is not enough when AI work must also be secure, reproducible, and usable across compute environments.
Key terms: lakehouse, Iceberg tables, data versioning, schema evolution, time travel, notebook platforms, feature stores, model training and orchestration platforms, lineage, curated Iceberg data
9.5 Choosing the right consumption tools: Ten illustrated scenarios
This section frames consumption-tool selection as a context-dependent decision that must fit the broader lakehouse architecture.
- Contextual consumption tool selection: Explain why no single consumption tool fits every organization and how business goals, technical requirements, team preferences, and integration with existing lakehouse layers shape the choice.
Why it matters: A tool that suits a team’s users and goals still needs to work with the rest of its lakehouse.
Key terms: consumption tools, semantic layer, federation layer, catalog layer
9.5.1 Startup with a data science focus
This scenario shows how a small, Python-focused AI startup can choose a lightweight notebook-based consumption layer for its Iceberg lakehouse.
- Notebook-centered consumption choice: Justify why the startup builds its consumption layer around JupyterLab rather than a traditional BI tool. Describe how DuckDB, PyIceberg, and MLflow support its data science work.
Why it matters: Consumption tools should fit the team's work and constraints rather than add infrastructure it does not yet need. - Reproducible Iceberg data access: Explain how the startup's notebooks access Iceberg data and how its training workflows remain reproducible.
Why it matters: Lightweight notebook access can still give model training a consistent, discoverable, and governed data foundation.
Key terms: consumption layer, JupyterLab, DuckDB, PyIceberg, MLflow, catalog-based reads, Spark sessions, snapshot capabilities, versioned access
9.5.2 Large financial institution with strict governance
This scenario shows how a global bank can let teams keep their preferred BI tools while governing metrics and data access centrally.
- Shared metrics across BI tools: Explain why the bank chooses a federated semantic layer instead of one mandatory BI tool, and how it provides consistent metrics across teams’ tools.
Why it matters: Teams can work in familiar tools without giving the same KPI different meanings. - Governed and traceable access: Trace how the semantic layer reaches Iceberg data and describe where access restrictions and auditability fit in the architecture.
Why it matters: The bank can offer flexible analytics while restricting and tracing access to regulated data.
Key terms: federated semantic layer, JDBC/ODBC, metric governance, Iceberg tables, query engine, RBAC, row- and column-level security, catalog policies, audit logs
9.5.3 Mid-sized e-commerce platform building embedded analytics
This section shows how a mid-sized e-commerce company can deliver responsive, governed analytics inside merchant dashboards using an embedded BI tool and an Iceberg lakehouse.
- Embedded BI tool choice: Justify the choice of Metabase for merchant-facing dashboards in terms of embedding, customization, and operational needs.
Why it matters: The BI choice has to fit both the merchant experience and the company's need to govern and scale it. - Merchant data isolation: Trace how the architecture supports tenant isolation from Iceberg tables through dashboard queries.
Why it matters: Embedded dashboards must not expose one merchant's data to another. - Responsive consistent dashboard queries: Describe how Metabase connects to the lakehouse and how the design supports dashboard performance and consistent metrics.
Why it matters: Merchants need dashboards that respond well and report the same metrics consistently.
Key terms: embedded analytics, Metabase, iframe or API-based embedding, white-label customization, multi-tenant data isolation, partitioning, metadata tagging, row-level security controls, access filters, JDBC connectors
9.5.4 Decentralized media organization enabling self-service analytics
This scenario shows how a media company can give its brands a choice of BI tools while keeping metrics, access, and query visibility centrally governed.
- Flexible tools and shared metrics: Explain how the company preserves each brand’s choice of BI tool while keeping metric definitions consistent across brands.
Why it matters: Shared metric definitions let brands work independently without reporting on incompatible measures. - Governed access and query visibility: Describe how the shared lakehouse supports brand-specific access and centralized visibility into BI queries.
Why it matters: The company can offer self-service analytics without giving every brand unrestricted access or losing sight of query activity.
Key terms: BI tool, semantic layer, dbt, canonical metrics, federation engines, Iceberg, federation layer, brand-specific access policies, JDBC, ODBC
9.5.5 Government agency balancing public transparency and internal control
This scenario shows how a government agency can offer public access to selected lakehouse data while keeping sensitive data controlled and historical results reproducible.
- Public and internal access paths: Describe how the agency uses different consumption paths for internal analytics and public access. Explain that public dashboards and APIs expose curated, nonsensitive data rather than unrestricted access to internal tables.
Why it matters: Separate access paths let the agency publish useful data without opening its internal analytics environment. - Data domains and access enforcement: Explain how the agency separates data into domains and enforces access policies for internal and public queries.
Why it matters: These controls make public transparency compatible with protection of sensitive information. - Historical reporting and reproducibility: Explain why the agency uses Iceberg time travel and snapshot isolation for historical views.
Why it matters: Preserving past views helps the agency substantiate reports and repeat research against historical data.
Key terms: Apache Superset, Looker Studio, cached extracts, controlled query endpoints, REST API, internal, public, restricted, metadata tagging, Trino
9.5.6 Healthcare provider with compliance and data locality constraints
This scenario shows how a healthcare provider can support clinical dashboards and research notebooks while keeping patient data local, access restricted, and analytics governed.
- Region-aware patient data access: Trace how the lakehouse supports data locality and restricted access: patient data is stored in Iceberg tables partitioned by geography and department, a region-aware catalog tags locality requirements, and Dremio enforces those policies during query planning.
Why it matters: This is how the provider makes patient data usable for analytics without setting aside locality requirements. - Clinical and research tool choices: Describe why the provider uses Power BI for clinical dashboards and JupyterHub for research notebooks, and how each is deployed to support compliant access.
Why it matters: The tool choices support distinct clinical and research workflows without bypassing governance. - Shared metrics and auditability: Explain how federated semantic definitions give both tools consistent clinical and operational KPIs while restricting sensitive metrics, and describe how logging, masking, and snapshot tracking support an audit trail.
Why it matters: Shared definitions and audit records help users trust results across clinical analytics and research.
Key terms: Iceberg tables, region-aware catalog layer, locality requirements, federation engine, query planning, Power BI, clinical dashboards, JupyterHub, research notebooks, region bound
9.5.7 Logistics company unifying real-time operations and historical analysis
This scenario shows how a logistics company combines live operational dashboards and historical planning tools over a consistent Iceberg lakehouse.
- Dual-horizon consumption tool choices: Explain how the company matches consumption tools to live operations, historical planning, and specialized analysis.
Why it matters: The tool choices follow the distinct needs of operators, planners, and specialized users without requiring separate data foundations. - Shared lakehouse across time horizons: Trace how streamed Iceberg data and the federation layer support both recent operational queries and historical analysis while keeping views consistent.
Why it matters: Both immediate decisions and longer-term plans can draw on the same consistently presented data.
Key terms: live-query capabilities, Looker, semantic layer, Arrow Flight, DuckDB, streaming ingestion, Iceberg tables, older partitions, federation layer, single source of truth
9.5.8 SaaS company offering customizable data access to clients
This section illustrates how a SaaS company can give clients dashboards and API access to shared Iceberg data while preserving tenant isolation, governance, and scalability.
- Shared dashboard and API access: Describe how the company serves business users through embedded Superset dashboards and developers through an API, with both interfaces connecting to a federated query engine over Iceberg tables.
Why it matters: Clients can choose visual or programmatic access while the company maintains one data-serving architecture. - Tenant isolation across layers: Explain how the design limits clients to their own data through tenant-aware tables, a central catalog, authenticated tenant-based query restrictions, and secure API tokens.
Why it matters: Shared data access is only viable if one customer cannot query another customer’s data. - Governed large result delivery: Describe how the company handles larger exports and protects the platform through caching or cloud-stored results, API audit logs, and rate limits.
Why it matters: Client self-service includes sizable exports, but those requests still need accountable and stable delivery.
Key terms: Superset, RESTful API, federated query engine, Iceberg tables, tenant-aware partitioning, metadata tagging, central catalog, tenant ID, secure tokens, query result caching
9.5.9 Nonprofit organization supporting collaborative research
This section shows how a climate-research nonprofit combines consumption tools and lakehouse controls to let partners collaborate on shared data while preserving reproducibility and governance.
- Researcher access and tools: Describe how the nonprofit gives internal and external researchers different ways to work with shared datasets. Explain how those choices accommodate varied partner tools while limiting external access.
Why it matters: The tool choices let partners work in different environments without giving everyone unrestricted access. - Reproducible governed research workflows: Explain how snapshots, catalog access policies, lineage tracking, and audit information support reproducible collaboration while protecting sensitive findings.
Why it matters: Researchers can revisit the data behind a finding while the organization retains oversight of access and sensitive results.
Key terms: JupyterHub, Apache Superset, Arrow Flight, DuckDB, read-only access, hosted notebooks, views, snapshots, time-based queries, data lineage
9.5.10 Manufacturing company enabling predictive maintenance
This section illustrates how a manufacturer combines consumption tools with a shared Iceberg lakehouse to support predictive maintenance.
- Predictive maintenance tool roles: Explain why the manufacturer uses different tools for equipment monitoring, exploratory analysis, and failure-prediction modeling.
Why it matters: The right consumption tool depends on what each team needs to do with the data. - Shared governed data foundation: Describe how the tools draw on a common lakehouse data layer and why that helps operational monitoring and longer-term modeling stay aligned.
Why it matters: A shared, governed data foundation lets teams investigate equipment issues and develop models without working from disconnected datasets.
Key terms: predictive maintenance, Grafana, Apache Superset, JupyterHub, Iceberg tables, federated engine, catalog, governed access, feature engineering
9.6 Summary
Concepts explained in this chapter
- semantic layerA semantic layer is a governed modeling layer that defines consistent business entities, dimensions, and metrics across data sources.
- row-level securityA row-level filter restricts access to records according to predicates applied at query or policy evaluation time.
- Iceberg tableAn Iceberg table is a logical table whose data and metadata are managed using the Apache Iceberg table format.
- consumption layerA lakehouse architectural layer that exposes data to business-intelligence, artificial-intelligence, and real-time applications.
- business intelligence toolA BI tool is a business-intelligence software application for querying, analyzing, and visualizing organizational data.
- Apache SupersetApache Superset is an open-source web application for data exploration, dashboarding, and business-intelligence visualization.
- data lakehouseA data lakehouse is a data architecture that combines the openness and flexible storage of a data lake with the governance and analytical capabilities of a data warehouse.
- federated query engineA federated query engine is a query engine that executes queries across multiple heterogeneous data sources without requiring all data to be stored in one system.
- MCPThe Model Context Protocol is a standardized protocol that lets large language models interact with external data, tools, and services through governed calls.
- computational notebookJupyter is an open-source interactive computing environment for combining executable code, narrative text, data, and visualizations.
- query pushdownFilter pushdown is a query optimization that evaluates filtering conditions as close as possible to the source data.
- dashboardA dashboard is a visual interface that combines charts and other summaries for monitoring or analyzing data.
- observabilityObservability is the capability to infer a system's internal behavior and resource usage from its emitted telemetry and operational data.
- role-based access controlRole-based access control is an authorization technique that grants permissions according to assigned user or service roles.
- auditabilityAuditability is the extent to which data access, changes, and control operations can be recorded and examined for verification or compliance.
- query engineA query engine is a software system that parses, plans, and executes queries against data sources.
- reproducibilityReproducibility is the ability to obtain consistent analytical or computational results by repeating a process with the same inputs, versions, and conditions.
- ODBCODBC is a language-independent API standard for connecting applications to database management systems and executing SQL statements.
- workload separationWorkload isolation is the separation of computational workloads so that resource use or failures in one workload do not adversely affect another.
- TableauTableau is a business-intelligence and data-visualization software platform for exploring data and creating interactive reports and dashboards.
- requirementsA storage requirement is an explicit, testable condition that a storage system or architecture must satisfy.
- Arrow Database ConnectivityADBC is an Arrow-based database connectivity interface for high-performance access to analytical data systems.
- semantic consistencySemantic consistency is the preservation of the same meanings, entities, measures, and business definitions across data-access tools and interfaces.
- vendor lock-inTool lock-in is dependence on a particular software tool or vendor that makes changing tools difficult or costly.
- Dremio CloudDremio is a data lakehouse platform that provides SQL query, semantic, and data-management capabilities over diverse data sources.
- snapshot isolationSnapshot isolation is a concurrency-control technique in which a transaction reads from a consistent version of data while concurrent changes are isolated from that view.
- authorizationAccess control is the restriction of data or system operations to authorized users or services according to defined permissions.
- HIPAAHIPAA is a United States law establishing requirements for protecting the privacy and security of individually identifiable health information.
- complianceA compliance posture is an organization's current condition of adherence to applicable laws, regulations, policies, and data-handling requirements.
- data residencyData residency is the requirement that data be stored or processed within specified geographic or jurisdictional boundaries.
- lakehouse catalogA lakehouse architectural component that tracks and governs table metadata and enables data systems to discover and access tables.
- query planningQuery planning is the process of determining an execution strategy for a query, including which files or partitions must be read.
- data integrityData integrity is the preservation of data's correctness, completeness, consistency, and protection from unauthorized alteration or loss.
- PyIcebergA software development kit that reads and parses Apache Iceberg metadata files and exposes their contents programmatically.
- schema evolutionSchema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
- Apache IcebergAn open table format for managing large analytic datasets with metadata-driven structure, snapshots, schema evolution, and transactional write capabilities.
- cachingCaching is the technique of storing reusable data or computation results in a faster-access location to reduce subsequent processing or retrieval cost.
- partition pruningPartition pruning is the elimination of data partitions that cannot satisfy a query's filters before or during data access.
- ETLA workflow that extracts data from sources, transforms it, and loads it into a target data system.
- data qualityData quality is the degree to which data is accurate, consistent, complete, timely, and fit for its intended use.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
