Chapter 4: Preparing for your move to Apache Iceberg

Introduction

The chapter introduction explains why an Iceberg transition should begin with an audit of the existing platform, input from stakeholders, and requirements that guide later tool choices.

• Current platform audit: Explain why an organization should audit its existing data landscape before integrating Apache Iceberg.
  Why it matters: Understanding the current platform keeps the transition grounded in actual constraints rather than assumptions.
• Stakeholder perspectives and support: Describe what different stakeholders contribute to an audit and why they should be interviewed early.
  Why it matters: A platform plan is more useful when it reflects both its users' problems and the organization's priorities.
• Requirements as selection filters: Explain how clarified requirements should guide later evaluations of lakehouse tools.
  Why it matters: Requirements make tool evaluations relevant to the organization rather than to a vendor's pitch.

Key terms: infrastructure audit, data landscape, stakeholders, data engineers, analysts, compliance teams, business leaders, requirements, filter, lakehouse ecosystem

4.1 Auditing your data platform

This section explains why an Iceberg lakehouse design should begin with a focused audit that turns organizational needs into architectural requirements.

• Audit to architecture path: Explain how an audit connects stakeholder discovery to requirements and then to a technical blueprint, and why skipping it risks designing for the wrong problem.
  Why it matters: An audit gives architectural decisions a basis in the organization’s actual needs.
• Business-focused audit scope: Distinguish the audit’s purpose from an exhaustive inventory, and describe how clarity about business goals helps evaluate technology options.
  Why it matters: A focused audit makes technology choices easier to judge against business priorities.

Key terms: stakeholder discovery, requirements, technical blueprint, data estate, business goals, vendors, components, deployment models

4.1.1 Who are the stakeholders?

Identifies the stakeholder groups an Iceberg lakehouse audit should involve and the distinct needs each brings to the design.

• Stakeholder involvement and alignment: Explain why the audit should involve stakeholders across the organization early, rather than treating the lakehouse as a platform for one team.
  Why it matters: Early alignment makes a move to Iceberg more likely to work for the organization as well as technically.
• Builders and data consumers: Distinguish what data engineers can reveal about the current platform from what analysts and data scientists can reveal.
  Why it matters: The people who build data flows and the people who use their output reveal different failures.
• Early compliance and security input: Explain why compliance and security professionals must participate early and what constraints they contribute.
  Why it matters: A technically sound design can still be blocked by unresolved risk or regulatory concerns.
• Business outcomes and priorities: Describe the role of business leaders and product owners in setting platform outcomes and priorities.
  Why it matters: Business priorities give technical decisions a measurable purpose.
• Infrastructure constraints and foundations: Describe what IT and infrastructure teams contribute to the audit and why the section recommends starting the conversation with them.
  Why it matters: The platform design must fit the infrastructure that will run and support it.

Key terms: stakeholders, audit, alignment, shared ownership, data engineering team, data analysts, data scientists, technical debt, data value chain, compliance

4.1.2 What should you ask stakeholders?

Stakeholder interviews turn current pain points and future ambitions into requirements and constraints for an Iceberg lakehouse.

• Engineering pipeline pain points: Describe what to ask data engineers about pipeline maintenance and how their answers inform technical requirements.
  Why it matters: The people maintaining pipelines can reveal constraints that a system inventory alone would miss.
• Analyst access and trust: Explain what to ask analysts and data scientists about getting and trusting data, and what their answers reveal.
  Why it matters: A platform must make data both accessible and trustworthy to avoid delays and fragmented copies.
• Compliance and governance constraints: Describe what to ask compliance and security teams and why their constraints should be identified early.
  Why it matters: Early governance constraints can shape platform choices before they become costly to change.
• Business value targets: Explain how interviews with business leaders connect platform capabilities to strategic outcomes.
  Why it matters: Technical improvements need a clear connection to outcomes the organization values.
• Infrastructure deployment boundaries: Describe what IT and infrastructure teams should clarify about operational and deployment constraints.
  Why it matters: A proposed architecture must fit the organization's actual operating boundaries.
• Interviews into architectural requirements: Explain why stakeholder discussions go beyond a checklist and how their findings support deliberate design.
  Why it matters: Connecting stakeholder findings prevents isolated, reactive architecture decisions.

Key terms: data engineers, manual workarounds, scaling bottlenecks, technical requirements, analysts, data scientists, data quality, freshness, usability, personal silos

4.1.3 Conducting a technological audit

This section explains how to audit an existing data platform so an Iceberg lakehouse design reflects its systems, constraints, and organizational needs.

• Technical audit as reality check: Explain how a technology audit complements stakeholder interviews and informs an Iceberg lakehouse design.
  Why it matters: The audit keeps the new architecture grounded in actual constraints rather than repeating old problems.
• Storage and format readiness: Describe what to record about storage systems and file formats, and how those findings affect an Iceberg move.
  Why it matters: Existing data locations and formats determine how feasible a migration is and where preparation is needed.
• Processing pipeline migration needs: Explain how reviewing ETL and ELT frameworks identifies workloads that can move easily and those needing changes.
  Why it matters: Existing pipelines can substantially change the effort and sequence of an Iceberg migration.
• Catalog compatibility and governance: Describe what to examine in the existing metadata setup and why Iceberg catalog compatibility matters.
  Why it matters: A catalog choice affects whether the existing governance setup and desired Iceberg features can work together.
• Query and BI access patterns: Explain why the audit examines query engines and BI tools and their connections to existing tables.
  Why it matters: The new platform must account for the ways people already query data and the performance they rely on.

Key terms: technology audit, current state, architectural reality check, storage architecture, file formats, Parquet, conversion pipelines, staging layers, ETL and ELT frameworks, batch and streaming jobs

4.2 Hamerliva Bank’s audit in action

Introduces Hamerliva Bank as an illustrative audit case and explains how its constraints shape the purpose of the audit.

• Purpose of the bank audit: Explain how Hamerliva Bank’s audit helps it move from fragmented systems and competing demands toward a prioritized understanding of what its future Iceberg lakehouse must deliver.
  Why it matters: The audit establishes what the lakehouse needs to accomplish before the bank designs it.

Key terms: audit, Iceberg adoption, lakehouse

4.2.1 Hamerliva Bank interviews its stakeholders

Hamerliva Bank interviews stakeholders before designing its Iceberg architecture, revealing technical problems, governance obligations, operational constraints, and business opportunities.

• Stakeholder First Audit Approach: Explain why Hamerliva Bank began its audit by interviewing people who use the data rather than letting technology dictate the architecture.
  Why it matters: The audit gives the architecture a basis in organizational needs, not just technology choices.
• Pipeline And Data Access Problems: Describe the problems data engineers, analysts, and data scientists reported with the existing platform.
  Why it matters: These accounts connect platform weaknesses to the daily work of producing and using data.
• Compliance And Security Obligations: Describe the controls compliance and security teams said the platform needed and why gaps posed risks.
  Why it matters: Regulatory and security needs are architectural constraints, not optional additions after deployment.
• Business And Infrastructure Priorities: Distinguish business leadership’s desired outcomes from IT and infrastructure teams’ deployment constraints.
  Why it matters: The interviews show that the proposed platform must address business ambitions within real deployment constraints.

Key terms: stakeholders, audit, buy-in, legacy ETL tools, schema changes, streaming data, audit trails, data residency, encryption, data access logs

4.2.2 Hamerliva Bank audits its technology stack

Hamerliva Bank’s technical audit identifies fragmented systems and data formats, then uses those findings to show why its move to Iceberg requires an integrated architectural redesign.

• Fragmented storage and file formats: Describe where Hamerliva Bank’s data was stored, how duplication arose, and why its file formats posed a migration challenge.
  Why it matters: Storage locations and existing file formats shape the scope of the move to Iceberg.
• Processing and metadata fragmentation: Explain how the bank’s mix of processing tools and metadata systems created integration and operational difficulties.
  Why it matters: The audit shows that connecting data flows and governing changes matter alongside choosing Iceberg tables.
• Analytics fragmentation and redesign: Explain what the query and BI audit revealed and why the overall findings called for more than incremental improvements.
  Why it matters: The audit’s conclusion establishes the scale of architectural change the bank needs to plan.

Key terms: HDFS, NAS, Amazon S3, CSV, JSON, Parquet, Apache Spark, Informatica, NiFi, Kafka

4.2.3 Hamerliva Bank summarizes the audit findings

Hamerliva Bank turns stakeholder interviews and a technical audit into a shared account of platform problems and open questions, without choosing solutions yet.

• Audit report purpose: Explain why the bank compiled a structured audit report and how it organized the findings.
  Why it matters: A shared account of the problems lets teams define requirements before committing to a design.
• Fragmented storage findings: Describe the storage problems and unresolved questions recorded in the audit.
  Why it matters: The storage findings distinguish observed costs of fragmentation from questions the design still has to resolve.
• Mixed processing workflows: Describe what the audit found about processing tools and ingestion workflows.
  Why it matters: The mix of tools and flow types makes ingestion requirements a question to define, not a tool choice already settled.
• Governance and metadata gaps: Explain how inconsistent metadata practices affected governance at the bank.
  Why it matters: These findings show why metadata and governance requirements must address more than where data is stored.
• Inconsistent data consumption: Describe the consumption-layer problems found in the audit and their effect on business users.
  Why it matters: Tools that work individually can still leave users unable to agree on or trace the data they consume.

Key terms: structured report, four domains, current state, HDFS, on-premises NAS, cloud data warehouses, data residency, Spark, Kafka, legacy ETL

4.3 From audit to requirements: Laying the groundwork for design

The section explains how to turn audit findings and use cases into outcome-focused requirements that guide Iceberg lakehouse design without choosing tools prematurely.

• Use cases as design bridge: Explain how use cases translate audit findings into requirements by linking business goals to the operational scenarios the lakehouse must support.
  Why it matters: Starting with use cases keeps requirements tied to the work the platform must actually support.
• Outcome focused component requirements: Describe how to write requirements as specific outcomes for the five lakehouse components, then group and prioritize them by business value and implementation effort.
  Why it matters: Outcome-focused requirements make later architectural tradeoffs easier to explain and justify.

Key terms: use cases, audit, operational scenarios, requirements, storage, ingestion, catalog, federation, consumption

4.3.1 Defining storage requirements

This section shows how to turn a storage audit into testable, prioritized requirements that guide a cloud, on-premises, or hybrid storage decision.

• Audit-based storage requirements: Explain how to turn audit findings into specific, testable storage requirements covering compliance, performance, cost, and operations without prescribing a vendor or technology.
  Why it matters: Clear statements of need let you evaluate storage options against your constraints rather than against assumptions.
• Storage deployment model trade-offs: Distinguish the benefits, constraints, and resulting requirement concerns of cloud, on-premises, and hybrid storage.
  Why it matters: The right storage model depends on documented constraints and trade-offs, not a default preference for cloud or on-premises infrastructure.
• Storage requirement priorities: Describe how to prioritize documented storage requirements and why that helps architectural and stakeholder decisions.
  Why it matters: Priorities make trade-offs explicit when no storage option satisfies every ambition equally.

Key terms: storage requirements, audit results, outcome-focused, specific and testable, cloud-based storage, on-premises storage, hybrid storage, consistent object store interface, critical, nice to have

4.3.2 Defining ingestion requirements

This section shows how to turn source-system needs, freshness targets, data-change behavior, and operational constraints into prioritized ingestion requirements before choosing tools.

• Source systems and ingestion patterns: Describe how to identify source systems, map batch and real-time ingestion patterns, and find weaknesses in existing pipelines before selecting tools.
  Why it matters: The sources and existing pain points are a better starting point for tool selection than a generic feature list.
• Business-driven data freshness: Explain how business decision times determine ingestion latency requirements and when batch, streaming, or a hybrid model is appropriate.
  Why it matters: A precise freshness requirement avoids paying for streaming complexity where timely batch updates suffice.
• Changing data and integrity guarantees: Describe requirements for schema changes, late data, corrections, and integrity guarantees in ingestion pipelines.
  Why it matters: Explicit change-handling and integrity requirements prevent reliability problems from surfacing only after pipelines are running.
• Ingestion ownership and observability: Explain which operational responsibilities and integrations should be captured in ingestion requirements.
  Why it matters: Pipelines need people and operational systems that can maintain them and respond when they fail.
• Prioritized framework selection criteria: Explain how to prioritize documented ingestion requirements and use them to evaluate frameworks.
  Why it matters: Priorities turn an audit into practical selection criteria without overengineering the lakehouse.

Key terms: source systems, ingestion patterns, schema drift, batch ingestion, streaming ingestion, latency targets, hybrid ingestion model, schema evolution, late-arriving data, corrections

4.3.3 Defining catalog requirements

This section explains how to turn an Iceberg lakehouse audit into catalog requirements for metadata coordination, interoperability, governance, resilience, and deployment.

• Catalog coordination and metadata: Explain how the catalog helps engines locate Iceberg tables, resolve metadata, and manage updates, and distinguish traditional metadata-file references from newer catalogs that serve metadata from managed stores.
  Why it matters: The catalog design affects how reliably and flexibly the lakehouse serves multiple tools.
• REST catalog interoperability: Justify when Iceberg REST catalog support should be a requirement and explain why individual tools’ read and write capabilities still need checking.
  Why it matters: A standard API is useful only if the tools you intend to use support the operations you need.
• Catalog governance and engine enforcement: Describe the catalog’s role in shared access policies and dataset discovery, and distinguish it from the query engine’s enforcement of row- or column-level security.
  Why it matters: Security requirements must account for both policy management and the engine that applies the rules.
• Metadata resilience and traceability: Describe catalog requirements for preserving a useful history of table changes and making activity auditable, including the section’s examples of snapshot expiration and access logging.
  Why it matters: These requirements make table history and access activity usable for operations and oversight.
• Catalog deployment and operating requirements: Distinguish self-hosted from managed catalogs and explain why selection must also address functional needs and operational qualities such as availability and scalability.
  Why it matters: The right catalog must fit both the team’s operating capacity and the lakehouse’s long-term demands.

Key terms: catalog, Iceberg table metadata files, metadata, Iceberg REST catalog specification, interoperability, read operations, writes, governance, RBAC, ABAC

4.3.4 Defining federation requirements

This section turns audit findings into federation requirements for accessing distributed data through an Iceberg lakehouse without requiring every dataset to be moved.

• Federated access and source coverage: Explain when federated access is needed and how to define connectivity requirements from the audit. Describe federation as querying and joining data across systems without first copying it, and require coverage of the relevant sources.
  Why it matters: Source coverage lets teams analyze distributed data without centralizing every dataset.
• Consistent federated data meaning: Explain why source connectivity alone is insufficient and describe the semantic consistency requirement across federated datasets.
  Why it matters: Users need consistent business meaning, not merely a connection to each data source.
• Federated performance outcomes: Describe the performance outcomes federation requirements should specify and the acceleration capabilities to assess, rather than mandating a particular technology.
  Why it matters: Cross-system access is useful only when query performance remains practical.
• Standard consumption interfaces: Describe the interface requirements that let BI tools, notebooks, and applications use federated datasets.
  Why it matters: Federated data must be reachable by the tools people already use.
• Federation operational controls: Describe operational requirements that help manage the complexity added by federation.
  Why it matters: Operational controls keep cross-system querying manageable as workloads and sources change.

Key terms: federation, connectivity, coverage, federated access, semantic consistency, semantic layer, unified semantic model, metric definitions, interactive performance, minimal data movement

4.3.5 Defining consumption requirements

This section shows how to turn an audit of how people use data into consumption requirements for access, performance, security, and interoperability.

• Audit-based consumption requirements: Explain how an audit of users’ tools, workflows, and difficulties becomes requirements for how people access and use lakehouse data.
  Why it matters: Requirements grounded in users’ actual workflows can address obstacles that a tool inventory alone would miss.
• Cross-engine semantic consistency: Describe the interoperability requirement for users who work across multiple query engines.
  Why it matters: Users working across engines need consistent meanings for the data they use.

Key terms: consumption requirements, audit, user experience, query performance, interoperability, semantic consistency, engines

4.4 Hamerliva Bank defines its requirements

Hamerliva Bank turns its audit findings into technology-neutral criteria for designing the five lakehouse components.

• Technology-neutral architectural requirements: Explain how Hamerliva Bank uses its audit findings to define architectural requirements, and distinguish those requirements from choices of specific tools or technologies.
  Why it matters: Keeping requirements separate from tool choices lets the bank evaluate designs against what its audit revealed.

Key terms: architectural requirements, stakeholder priorities, storage, ingestion, catalog, federation

4.4.1 Storage requirements

Hamerliva Bank turns its fragmented storage audit into requirements for compliance, interoperability, and cost-conscious data retention.

• Hamerliva Bank storage requirements: Explain how Hamerliva Bank’s storage audit shaped its requirements and how those requirements guide storage choices.
  Why it matters: The bank can compare storage options against its actual compliance, integration, and cost needs rather than choosing on features alone.

Key terms: regionally bound, on-premises storage, S3-compatible APIs, archival tiering, Iceberg metadata

4.4.2 Ingestion requirements

Hamerliva Bank turns its ingestion audit findings into requirements for streaming, batch, and shared pipeline capabilities.

• Workload-specific ingestion requirements: Explain why Hamerliva Bank chose dual-mode ingestion and describe what its streaming and batch pipelines must do.
  Why it matters: The ingestion design must serve workloads with different latency needs without sacrificing consistent results.
• Shared pipeline governance requirements: Describe the requirements that apply to every Hamerliva Bank ingestion pipeline and why they matter.
  Why it matters: Shared requirements keep changing workloads manageable and ingested data traceable.

Key terms: dual-mode ingestion strategy, late-arriving events, deterministic results, Apache Spark, Airflow DAGs, schema evolution, lineage metadata, catalog registration

4.4.3 Catalog requirements

Hamerliva Bank turns its fragmented metadata and governance problems into catalog requirements for evaluating Iceberg catalog options.

• Hamerliva catalog selection criteria: Explain the catalog requirements Hamerliva Bank derived from its audit and how it would use them to shortlist options.
  Why it matters: The criteria give the bank a way to compare catalogs against its actual interoperability and governance needs.

Key terms: Iceberg REST Catalog specification, versioned schema evolution, rollback, user attribution, RBAC, corporate identity providers

4.4.4 Federation requirements

Hamerliva Bank turns its scattered datasets and manual workarounds into requirements for federated access, shared business logic, and consistent governance.

• Unified access without duplication: Explain why Hamerliva Bank requires query engines to access Iceberg tables, S3 buckets, and on-premises HDFS together without duplicating data.
  Why it matters: This requirement lets the bank evaluate engines for access to both Iceberg and its existing data.
• Shared logic and access controls: Describe the common semantic layer and consistent access controls Hamerliva Bank requires for federated queries.
  Why it matters: Unified queries are useful only if business definitions and access restrictions remain consistent across sources.

Key terms: federated access, Iceberg tables, S3 buckets, on-premises HDFS, data duplication, common semantic layer, reusable business logic, row- and column-level access controls, governance

4.4.5 Consumption requirements

Hamerliva Bank turns problems found in its consumption audit into requirements for fast, consistent, governed access to Iceberg data.

• Consumption problems and performance: Explain how dashboard latency spikes, inconsistent semantics, and analysts’ shadow pipelines shaped the bank’s consumption requirements, including its dashboard response-time target.
  Why it matters: The requirements address problems users encountered, rather than merely confirming that a tool can query Iceberg.
• Consistent governed access: Describe the bank’s requirement for consistent semantics and security across tools, including how query engines should apply access policies.
  Why it matters: A tool’s ability to read Iceberg tables alone does not satisfy the bank’s governance and consistency needs.
• Downstream and Python access: State the access requirements for downstream tools and data science users, including standard interfaces, conditional columnar exchange, and Python-based workflows.
  Why it matters: The bank’s consumption layer must fit the tools its downstream users already rely on.

Key terms: consumption requirements, latency spikes, semantics, shadow pipelines, interactive dashboards, consistent semantics, row-level access policies, catalog, enterprise roles, query engines

4.4.6 From requirements to design

Hamerliva Bank turns its requirements into a shared basis for architectural decisions.

• Requirements as design criteria: Explain how Hamerliva Bank’s requirements guide its architectural design without resolving every open question.
  Why it matters: Shared criteria help teams make architectural choices that serve the same goals.

Key terms: requirements, testable criteria, architectural design process, stakeholder expectations

4.5 Architectural plan and road show

The section explains how to turn Iceberg lakehouse requirements into an architectural plan and validate it with stakeholders before implementation.

• Requirements-based architectural blueprint: Describe how an architectural plan translates requirements into technology choices, federation strategies, and a blueprint showing how the five lakehouse components work together. Explain that the plan should reflect audit findings, organizational constraints, tradeoffs, and stakeholder priorities.
  Why it matters: A plan grounded in requirements makes the proposed architecture explainable and relevant to the organization.
• Stakeholder road show: Explain why the completed plan is presented to each stakeholder group before implementation and how their feedback is used.
  Why it matters: Early stakeholder validation can reveal gaps in the design and make adoption easier.

Key terms: architectural plan, storage, ingestion, catalog, federation, consumption, stakeholder road show, stakeholder groups, pain points, feedback

4.5.1 Hamerliva Bank creates its architectural plan

Hamerliva Bank turns its requirements into a reviewable Iceberg lakehouse plan, explaining how each architectural layer serves operational, regulatory, and analytical needs.

• Requirements Led Architecture Planning: Explain how Hamerliva Bank developed and presented its architectural plan from its requirements. Describe why the plan remained open to stakeholder review.
  Why it matters: The architecture has to meet real requirements and earn support from the people who will use and govern it.
• Hybrid Storage Placement: Explain why the bank combined Amazon S3 with on-premises MinIO in its storage design.
  Why it matters: The storage choice balances cloud-scale analytics with control over regulated data.
• Split Ingestion Workloads: Describe how the bank divides ingestion between batch and near-real-time pipelines, including where updates and corrections belong.
  Why it matters: Separating fresh, append-heavy data from corrections gives the bank low latency without making consistency harder to manage.
• Central Catalog Choice: Explain why the bank selected Apache Polaris as its central catalog.
  Why it matters: A shared catalog helps the bank manage tables and access consistently across environments and engines.
• Federated Query Design: Explain Dremio's role in querying data across the bank's systems and serving analytical workloads.
  Why it matters: Federation lets users analyze data spanning existing systems without first copying it all into one place.
• Governed Consumption Paths: Describe how business users and data scientists access the planned lakehouse while sharing consistent definitions and controls.
  Why it matters: Different users can work with the same curated data without giving up consistent meaning and governance.

Key terms: cross-functional group, five core Iceberg lakehouse components, architecture guide, staged migration plan, hybrid storage model, Amazon S3, MinIO, data residency, S3-compatible object storage, dual-mode ingestion strategy

4.5.2 Hamerliva Bank conducts a road show

Hamerliva Bank uses a role-specific road show to test and refine its Iceberg architecture while building support for implementation.

• Role-specific architecture briefings: Explain how Hamerliva Bank tailored road-show briefings to stakeholder responsibilities while grounding them in the audit, requirements, and business motivation.
  Why it matters: People can assess a proposed architecture more usefully when it is connected to their own responsibilities.
• Feedback-driven plan refinements: Describe how stakeholders used the architecture guide to challenge the proposal and how their feedback changed the rollout or design.
  Why it matters: A road show can reveal practical gaps before implementation makes them costly to fix.
• Cross-functional implementation buy-in: Explain why the road show sought more than approval of a technical design and what participation achieved for implementation.
  Why it matters: Shared ownership helps carry a cross-functional platform change from planning into rollout.

Key terms: road show, role-specific briefings, architectural plan, architecture guide, living reference, migration phases, cross-functional alignment, buy-in, transition liaisons

4.6 Summary