Chapter 7: Implementing the catalog layer

Introduction

Introduces the catalog as the metadata and coordination layer of an Iceberg lakehouse and explains why catalog selection must fit organizational needs.

• Catalog layer responsibilities: Explain how the catalog differs from storage and ingestion: it registers and organizes Iceberg tables, tracks their metadata, manages namespaces, and coordinates data operations.
  Why it matters: This distinction clarifies what the catalog contributes to the lakehouse architecture.
• Strategic catalog selection: Explain why choosing an Iceberg catalog is more than a technical choice: it affects governance, interoperability, scalability, and ecosystem integration, so it must fit an organization’s constraints and goals.
  Why it matters: A catalog that fits one organization may not fit another’s governance or operational needs.

Key terms: catalog layer, table metadata, namespaces, coordination, governance, interoperability, scalability, architectural constraints, compliance needs, operational goals

7.1 The role of the catalog in Apache Iceberg lakehouses

The section explains how Iceberg’s versioned metadata and catalog work together to give multiple engines a consistent way to find and update tables.

• Versioned table metadata: Explain how committed table changes create immutable metadata versions and how atomic replacement of metadata supports consistent table views.
  Why it matters: The catalog’s role makes more sense once you see what its current table state points to.
• Catalog coordination of table state: Describe how the catalog tracks tables and identifies their current metadata version so multiple compute engines can work with the same table state.
  Why it matters: This shared coordination is what lets different tools operate on the same Iceberg tables.
• Catalog organization and interoperability: Describe the catalog’s additional responsibilities for table locations, namespaces, and transactional metadata updates, and explain why separating metadata coordination from compute and storage matters.
  Why it matters: These responsibilities help explain what a catalog must provide beyond a list of table names.

Key terms: metadata versions, atomic replacement, consistent views, catalog, coordination layer, current state, compute engines, table locations, namespaces, transactional updates

7.1.1 Responsibilities of the catalog

This section explains how the Iceberg catalog helps engines find tables, organize them, identify their current metadata, and coordinate safe commits.

• Table registration and discovery: Explain how an engine uses the catalog to find a registered Iceberg table and why a shared catalog matters to multiple engines.
  Why it matters: A shared catalog lets different processing engines find the same registered tables consistently.
• Current metadata association: Describe how the catalog identifies the metadata associated with a table and why changing that association supports versioned table history.
  Why it matters: Keeping the table identifier separate from its changing metadata association makes versioned table states possible.
• Namespace organization: Explain how namespaces organize registered tables and what that organization helps with.
  Why it matters: Namespaces keep large collections of tables navigable and help organize access control.
• Optimistic commit coordination: Trace how the catalog handles a proposed write when another operation may have changed the table.
  Why it matters: Catalog-mediated commit checks protect table metadata when writers operate concurrently.

Key terms: table registration, discovery, metadata location, metadata location management, table identifier, transactionally, versioned table history, namespaces, table discovery, access control

7.1.2 Catalog interactions with query and processing engines

This section traces how engines use the catalog and Iceberg metadata to resolve tables, plan work, commit writes, and access table history.

• Catalog based table resolution: Trace how an engine begins working with an Iceberg table: it requests the table by name from the catalog, then uses the returned reference to load the table’s latest metadata.
  Why it matters: This is the starting point for understanding how different engines find the same Iceberg table.
• Metadata driven snapshot planning: Explain how an engine uses table metadata to determine which files to read or write, and how statistics and file-level metadata help reduce scanning.
  Why it matters: Metadata lets an engine narrow its work instead of scanning unnecessary data.
• Atomic write commit coordination: Trace how an engine commits a write by constructing new metadata and attempting an atomic update to the catalog’s pointer, including what happens if another operation updates the table first.
  Why it matters: The atomic pointer update explains how concurrent engines can avoid silently overwriting one another’s changes.
• Shared schema and snapshot history: Describe how engines use discovered metadata for current and previous schemas and for requests involving earlier snapshots, and why a unified metadata view helps different engines stay aligned.
  Why it matters: A shared view of schemas and snapshots supports consistent work across ETL, streaming, and analytics engines.

Key terms: catalog, table resolution, metadata file, snapshot planning, statistics, file-level metadata, data-skipping, predicate pushdown, commit coordination, catalog’s pointer

7.2 Evaluating catalog requirements

The section explains why catalog selection should begin with organizational requirements rather than technical features alone.

• Requirements-led catalog selection: Explain why an organization should clarify its catalog requirements before selecting a catalog, and describe the consequences of a poor fit.
  Why it matters: Starting with your organization’s needs helps you avoid choosing a catalog that creates problems for the teams using it.

Key terms: catalog layer, organizational requirements, stakeholder interviews, technical audits

7.2.1 Performance, availability, and scale

This section explains how to evaluate an Iceberg catalog for performance, availability, and scalability under expected lakehouse workloads.

• Catalog latency and operations: Explain why catalog response time affects Iceberg reads and writes, particularly when workloads require frequent metadata lookups or concurrent writes.
  Why it matters: A slow catalog can make the lakehouse feel slow even when its data storage is working well.
• Catalog uptime dependency: Explain why catalog availability matters even when storage remains operational, and describe the fault-tolerant deployment capabilities to consider.
  Why it matters: Catalog uptime is a dependency for accessing tables, not just an operational concern for the catalog service.
• Growing metadata workload capacity: Describe the kinds of growth a catalog must handle without becoming a bottleneck, including metadata reads and concurrent writes.
  Why it matters: A catalog needs capacity for the full metadata workload, not just fast lookups for a small number of tables.
• Workload based catalog evaluation: Describe how to evaluate a candidate catalog against expected workloads rather than relying on one readiness metric.
  Why it matters: Testing against your own workload makes a catalog comparison more useful than relying on a headline capacity figure.

Key terms: catalog, metadata lookups, table planning, query execution, availability, catalog uptime, clustering, multiregion replication, managed failover, scalability

7.2.2 Metadata governance and lineage

This section explains how to evaluate an Iceberg catalog’s governance capabilities, from organizing and securing tables to tracing and auditing metadata changes.

• Catalog governance foundation: Explain why catalog selection must consider governance as well as table locations, and describe how namespaces help organize tables.
  Why it matters: It frames governance as a catalog-selection requirement rather than an optional addition to table lookup.
• Identity and fine-grained authorization: Distinguish external identity integration from fine-grained authorization, and explain the difference between RBAC and ABAC for catalog access.
  Why it matters: It helps distinguish how a catalog recognizes a user from how it decides what that user may do.
• Scoped storage credentials: Describe credential vending and why it limits a querying client’s storage access.
  Why it matters: It connects catalog governance to restricted access at the storage level.
• Lineage and metadata audit trails: Explain what lineage and metadata-operation logs contribute to governance, including what catalog changes should be recorded.
  Why it matters: Traceable changes support external obligations while making internal investigations and debugging easier.

Key terms: catalog, system of record, metadata governance, namespaces, identity providers, RBAC, ABAC, fine-grained authorization, tables, credential vending

7.2.3 Security and compliance

This section explains how to evaluate a catalog’s access controls, metadata protection, and regulatory compliance capabilities.

• Catalog access control: Distinguish Iceberg’s schema and transaction guarantees from the catalog’s responsibility for controlling table visibility and permitted actions. Explain how integration with existing identity systems supports role-appropriate permissions.
  Why it matters: Choosing a catalog means checking whether it can enforce access rules through the identity systems your organization already uses.
• Catalog metadata encryption: Explain why catalog metadata needs protection in transit and at rest, even when the storage layer encrypts data files.
  Why it matters: Encrypted data files do not, by themselves, protect the metadata a catalog transmits or stores.
• Catalog compliance controls: Describe how regulatory requirements can affect catalog architecture, including metadata location, administrative logging, and metadata retention or deletion.
  Why it matters: A catalog must be evaluated against the organization’s regulatory obligations, not just its access-control features.

Key terms: catalog, identity and access management (IAM), fine-grained permissions, catalog metadata, encryption in transit, encryption at rest, data residency, audit trails, metadata retention and deletion, metadata lifecycle policies

7.2.4 Deployment flexibility and ecosystem compatibility

This section explains how to evaluate an Iceberg catalog’s deployment options, network placement, and compatibility with the tools around it.

• Catalog deployment flexibility: Explain how deployment flexibility affects a catalog’s fit with on-premises, cloud, and hybrid infrastructure, and why a REST interface helps tools communicate with it across environments.
  Why it matters: Deployment and interface choices determine whether a catalog fits your infrastructure without tying every tool to one environment.
• Catalog placement and throughput: Explain why catalog network placement should usually be judged by reliability, bandwidth, and concurrent throughput rather than round-trip latency alone, and when network distance may still matter.
  Why it matters: It keeps catalog placement decisions focused on the workload bottlenecks most likely to affect a lakehouse.
• Cross-engine catalog compatibility: Describe what compatibility with processing and query engines requires, including consistent table resolution and transaction behavior, and why Iceberg REST compliance is useful.
  Why it matters: Teams using different engines need them to find and transact on tables consistently.
• Automation integration capabilities: Describe the catalog capabilities that help it integrate with CI/CD pipelines, schema registries, orchestration frameworks, and metadata governance tools.
  Why it matters: Integration features reduce manual work and friction as the lakehouse ecosystem grows.

Key terms: deployment flexibility, standalone services, managed services, REST-based catalogs, network placement, throughput, concurrency, round-trip latency, metadata operations, table resolution

7.2.5 Cost and operational overhead

This section explains how to evaluate a catalog’s long-term cost and operational burden alongside its features.

• Catalog cost and management tradeoffs: Explain why catalog cost includes both direct expenses and ongoing engineering effort, and contrast the operational tradeoffs of self-managed and managed catalogs.
  Why it matters: A catalog that looks inexpensive to acquire may be costly to operate over time.
• Operational support and catalog fit: Describe how observability, documentation, and support affect catalog operating effort, and explain why feature depth must be balanced against simplicity and budget.
  Why it matters: Day-to-day visibility and support can matter as much as an impressive feature list.

Key terms: self-managed catalogs, managed services, operational overhead, vendor lock-in, monitoring and observability, structured logs, alerting systems, documentation, community support, operational simplicity

7.2.6 Catalog federation and mesh architectures

Explains how federated catalogs give independently governed Iceberg environments a unified view, why organizations adopt them, and what capabilities to evaluate.

• Federated catalog operating model: Explain how multiple catalogs can operate autonomously while presenting a unified view of Iceberg tables. Describe what each catalog controls locally and how a common interface enables discovery and access across the mesh.
  Why it matters: This distinction explains how a lakehouse can offer a global view without requiring one central catalog to control everything.
• Catalog mesh adoption drivers: Justify catalog federation in terms of regulatory boundaries, decentralized governance, cloud-specific deployments, and the limits of a single catalog. Explain how local control can coexist with enterprise-wide analytics and governance.
  Why it matters: These drivers help determine when distributed coordination is preferable to a single catalog.
• Federation capability requirements: Describe the capabilities catalogs need to participate in federation and the questions to ask when evaluating them. Include interfaces, namespace resolution, security and identity propagation, and keeping changes aligned without conflicts or stale states.
  Why it matters: A global view is only useful if resources can be found and updates and permissions remain trustworthy across catalogs.

Key terms: federated catalog architecture, catalog mesh, namespace, unified, global view, data sovereignty, data residency, decentralized teams, single point of failure, consistent APIs, namespace resolution

7.3 Apache Iceberg REST Catalog specification

This section explains why client-specific catalog integrations made multi-engine Iceberg use fragile and how a common REST interface improves interoperability.

• Client-side catalog fragmentation: Explain why early Iceberg catalog integrations made multi-engine interoperability fragile and how moving commit responsibility into the catalog service addresses that problem.
  Why it matters: It explains why sharing a table format alone did not guarantee reliable cooperation between engines.
• REST catalog interoperability: Describe how the Apache Iceberg REST Catalog specification helps engines and catalog services work together.
  Why it matters: A common catalog interface gives architects more freedom to change engines or catalog services.

Key terms: engine-specific client libraries, commit coordination, metadata updates, catalog service, Apache Iceberg REST Catalog specification, HTTP-based interface, catalog operations, interoperability

7.3.1 Before the Apache Iceberg REST Catalog specification

Explains why pre-REST Iceberg catalog integrations were fragile: language-specific clients duplicated catalog and transaction logic, creating correctness risks and slowing development.

• Client-side transactional correctness risks: Explain why pre-REST catalog integrations could threaten data integrity in multi-engine environments, even though engines could communicate with catalogs through systems such as the Hive Metastore.
  Why it matters: Catalog connectivity alone does not ensure that different engines will commit changes consistently.
• Duplicated catalog client development: Describe how language-specific catalog libraries created inconsistent feature support and repeated work for catalog providers.
  Why it matters: Duplicated client work made consistent catalog behavior harder to deliver and slowed new features.

Key terms: commit coordination, conflict detection, metadata updates, transactional guarantees, Hive Metastore (HMS), language-specific client libraries, feature completeness

7.3.2 The solution

This section explains how the Iceberg REST Catalog specification replaces catalog-specific client integrations with a common interface and enables more responsibility to move into the catalog.

• Common catalog interface: Explain how the REST Catalog specification lets engines and applications interact with compliant catalogs through a shared, engine-neutral HTTP API rather than catalog-specific client libraries. Describe the kinds of catalog operations it standardizes.
  Why it matters: A common interface lets organizations choose or change catalogs without rebuilding integrations around each engine.
• Catalog-centered responsibilities: Explain how the REST-based model can move scan-planning and commit responsibilities from individual engines into catalog APIs, and why that supports consistent correctness and governance.
  Why it matters: Centralizing these responsibilities can make behavior more consistent across engines while simplifying their clients.

Key terms: Apache Iceberg REST Catalog specification, engine-neutral HTTP-based API, compliant catalog, language-independent protocol, scan planning, manifest parsing, centralized commit APIs, conflict resolution, policy enforcement

7.4 Catalog options: Exploring the ecosystem

Introduces the main categories of Apache Iceberg catalogs and the deployment tradeoffs that distinguish them.

• Catalog deployment categories: Distinguish built-in, open source self-managed, and vendor-managed Iceberg catalogs by their deployment needs and tradeoffs.
  Why it matters: The deployment model helps narrow catalog choices to those that fit an organization’s operational capacity and constraints.
• Shared catalog responsibilities: Explain what the different catalog implementations all aim to do for an Iceberg lakehouse.
  Why it matters: These shared responsibilities provide a common baseline for comparing otherwise different catalog options.

Key terms: Hadoop catalog, JDBC catalog, self-managed, managed services, table metadata, coordinating transactions, multi-engine access

7.4.1 Hadoop catalog

This section explains how the Hadoop catalog manages Iceberg table metadata through a filesystem and why it is mainly a legacy or transitional choice.

• Filesystem-based metadata management: Describe how the Hadoop catalog maps table names to metadata stored in directories under a warehouse root, and how it performs catalog operations without a separate service.
  Why it matters: Its filesystem-based design explains both its simplicity and its limits.
• Hadoop catalog suitability: Explain why the Hadoop catalog is easy to use in local or controlled settings but discouraged for modern shared production deployments.
  Why it matters: A catalog that is convenient for an isolated setup may not provide what multiple tools and writers need.

Key terms: Hadoop catalog, warehouse root, metadata references, namespaces, centralized transaction coordination, concurrent writes, remote access, service-oriented catalog

7.4.2 Hive catalog

This section explains how the Hive catalog supports gradual Iceberg adoption through an existing Hive Metastore, while arguing against it for new deployments.

• Hive Metastore migration path: Explain how the Hive catalog lets an organization adopt Iceberg alongside legacy Hive tables without replacing its existing metadata infrastructure.
  Why it matters: It explains why a team with substantial Hive infrastructure might choose this catalog during a transition.
• Hive catalog deployment tradeoffs: Distinguish the Hive catalog’s migration advantages from its limitations, and justify why the section favors REST API-based catalogs for new Iceberg deployments.
  Why it matters: It helps distinguish a useful migration bridge from a sound default for a new lakehouse.

Key terms: Hive catalog, Hive Metastore (HMS), Hive integration layers, namespace operations, fine-grained role-based access control, REST Catalog API, Apache Iceberg Hive Catalog plugin, service-oriented catalogs

7.4.3 JDBC catalog

Explains how Iceberg’s JDBC catalog stores shared catalog state in a relational database and when its benefits and limits make it a practical choice.

• SQL-backed catalog state: Explain that the JDBC catalog stores table metadata pointers and namespace information in relational database tables, and performs catalog operations through SQL transactions rather than filesystem metadata pointers.
  Why it matters: This explains both where the catalog keeps its state and how changes to that state are made.
• Shared state and transactional coordination: Describe how a database-backed catalog lets multiple compute environments share metadata state and how SQL transactions reduce inconsistencies during concurrent updates.
  Why it matters: Shared, transactional metadata makes the catalog more suitable for teams with multiple engines or concurrent writers.
• Embedded clients and selection tradeoffs: Distinguish the JDBC catalog’s embedded-client architecture from a separate catalog service, and explain when its database-backed approach is useful and what limits its scale and remote access.
  Why it matters: The deployment model helps determine whether JDBC is a simpler fit than a dedicated catalog service.

Key terms: JDBC catalog, metadata pointers, namespace, SQL transactions, ACID guarantees, atomic updates, centralized metadata store, concurrent writes, single-node architecture, backing store

7.4.4 Apache Polaris

This section explains how Apache Polaris coordinates Iceberg metadata across engines, manages internal and external catalogs, and controls access, along with its deployment tradeoff.

• Centralized REST metadata coordination: Explain how Polaris provides a shared metadata access point for multiple Iceberg engines and how its catalogs relate to cloud storage.
  Why it matters: A shared catalog gives different engines a consistent entry point to Iceberg tables.
• Internal and external catalogs: Distinguish Polaris-managed internal catalogs from external catalogs, including who controls updates and why an external catalog might be useful.
  Why it matters: The distinction determines whether Polaris manages table updates or only provides a common view of tables managed elsewhere.
• Layered role-based permissions: Trace how service principals, principal roles, and catalog roles work together to delegate access at catalog, namespace, and table levels.
  Why it matters: The role hierarchy lets teams give different people and workloads only the catalog privileges they need.
• Temporary storage credential vending: Explain the security purpose of Polaris supplying temporary credentials to query engines at runtime.
  Why it matters: Temporary credentials reduce the need to distribute cloud storage credentials to engines.
• Production fit and overhead: Justify when Polaris is a strong catalog candidate and state the operational cost identified in this section.
  Why it matters: A catalog choice should weigh Polaris’s shared governance benefits against the work of running it.

Key terms: Apache Polaris, Iceberg REST Catalog specification, metadata pointers, internal catalogs, external catalogs, read-only, service principals, principal roles, catalog roles, RBAC

7.4.5 Project Nessie

This section explains how Project Nessie adds versioned catalog coordination to Iceberg and when its benefits justify a separate service.

• Versioned metadata coordination: Explain that Nessie records Iceberg table changes as commits in a version log while Iceberg remains responsible for table metadata and data files.
  Why it matters: This distinction clarifies what adding Nessie changes in an Iceberg architecture.
• Branches tags and atomic merges: Describe how branches let jobs make isolated changes, tags identify fixed points in time, and a successful job’s changes can be merged atomically into the mainline.
  Why it matters: Isolated branches make experimentation and production updates safer.
• Cross-table transaction consistency: Explain how Nessie can group changes to multiple related tables into one commit or merge so they become consistent and atomic together.
  Why it matters: Related ingestion or schema changes may need to appear together across datasets.
• REST credential vending: Describe how Nessie, through Iceberg’s REST protocol, can provide configuration and temporary credentials to engines instead of requiring directly configured cloud-storage credentials.
  Why it matters: It shows how a catalog choice can affect both security and deployment complexity.
• Nessie deployment tradeoff: Justify when Nessie is a good catalog choice by weighing its reproducibility, safe experimentation, and governance benefits against the need to operate a separate service.
  Why it matters: Catalog selection involves operational costs as well as features.

Key terms: metadata coordination layer, commit, version log, branches, tags, merge, mainline, cross-table transactions, single commit, atomicity

7.4.6 Apache Gravitino

This subsection explains how Apache Gravitino unifies metadata and governance across Iceberg and other data systems, and where its breadth brings benefits and tradeoffs.

• Gravitino metadata hierarchy: Describe how Gravitino organizes different kinds of metadata under a shared governance domain. Identify the roles of metalakes, catalogs, and the objects they expose.
  Why it matters: This hierarchy explains how one governance domain can encompass more than Iceberg tables.
• Iceberg and cross-system integration: Explain how Gravitino connects Iceberg-aware engines and brings metadata from other systems into a centralized hub.
  Why it matters: The REST interface and connectors explain Gravitino's role when Iceberg coexists with other systems.
• Shared metadata governance: Describe the governance capabilities Gravitino applies across cataloged assets and distributed metadata domains.
  Why it matters: Governance is part of Gravitino's value as a shared metadata layer, not just a way to locate assets.
• Gravitino fit and tradeoffs: Justify when Gravitino is an attractive catalog choice and identify the cautions stated in this subsection.
  Why it matters: Its broad reach is most useful when that reach outweighs the added operational and feature-maturity concerns.

Key terms: metalake, catalogs, schemas, tables, filesets, models, Apache Iceberg REST Catalog interface, connectors, centralized metadata hub, role-based policies

7.4.7 Lakekeeper

This section describes Lakekeeper’s deployment model, catalog APIs, tenant isolation, and governance features as an Apache Iceberg REST Catalog option.

• Lakekeeper deployment and interfaces: Describe Lakekeeper as a Rust-based, JVM-free Iceberg REST Catalog service and distinguish its client-facing API from its configuration API. State its PostgreSQL backend requirement.
  Why it matters: The two APIs and backend requirement help determine how Lakekeeper fits into a deployment.
• Project and warehouse isolation: Explain how Lakekeeper organizes multiple tenants with projects and warehouses, and why it prevents warehouse storage locations from overlapping.
  Why it matters: Storage-location isolation is central to using one catalog service safely across tenants.
• External identity and permissions: Describe how Lakekeeper delegates authentication and enforces fine-grained permissions across catalog objects.
  Why it matters: This division of responsibilities shows how Lakekeeper fits into an existing identity and governance setup.
• Operational governance safeguards: Explain how soft deletion, change approval hooks, and temporary storage credentials make Lakekeeper operations safer.
  Why it matters: These safeguards address accidental deletion, unsuitable changes, and storage-credential exposure.

Key terms: standalone binary, Iceberg REST API, Management API, PostgreSQL, multi-tenant architectures, projects, warehouses, object store location, OpenFGA, custom authorizers

7.4.8 AWS Glue Data Catalog

This section explains how AWS Glue Data Catalog supports Iceberg tables, how AWS Glue jobs use Iceberg, and when the managed AWS catalog is a good fit.

• Glue catalog role: Explain that AWS Glue Data Catalog is a managed metadata repository for Iceberg tables and that compute engines can connect to its REST endpoint to read and write those tables.
  Why it matters: This distinguishes the shared metadata service from the engines that operate on Iceberg tables.
• Glue Spark Iceberg configuration: Describe the separate native Iceberg support in AWS Glue Spark engine versions 3.0 and 4.0 and the typical settings for an Iceberg job.
  Why it matters: Native engine support reduces setup, but it is distinct from choosing Glue as the metadata catalog.
• Catalog governance integrations: Explain how Glue integrates with Lake Formation, IAM, and CloudTrail for controlled access and auditing.
  Why it matters: These integrations help organizations share data while restricting sensitive information and tracking changes.
• AWS catalog selection tradeoffs: Justify when Glue Data Catalog is a good catalog choice and when another option may fit better.
  Why it matters: A catalog’s operational convenience must be weighed against portability, regional reach, and control.

Key terms: AWS Glue Data Catalog, metadata catalog, REST endpoint, Iceberg tables, AWS Glue Spark engine, --datalake-formats iceberg, GlueCatalog, S3FileIO, custom JARs, AWS Lake Formation

7.4.9 Dremio catalog

This section describes Dremio catalog’s Polaris-based interoperability, enterprise features, managed operation, and fit for organizations using Dremio.

• Polaris Foundation and Interoperability: Explain how Dremio catalog uses Apache Polaris to provide centrally managed Iceberg metadata while allowing compatible engines to read and write the tables.
  Why it matters: You can evaluate Dremio as a managed catalog without assuming every workload must run through Dremio.
• Enterprise Governance and Discovery: Describe the access-control and metadata-discovery features Dremio adds for enterprise use.
  Why it matters: Catalog selection also affects who can access data and how easily people can find and understand it.
• Automated Maintenance and Integration: Explain which Iceberg maintenance tasks Dremio automates and how its interface and storage mapping simplify table management.
  Why it matters: These built-in functions can reduce the separate tooling and routine work needed to operate Iceberg tables.
• Managed Deployment Tradeoff: Justify when Dremio catalog is a good fit and identify the platform dependency that limits its flexibility.
  Why it matters: The operational convenience is most valuable when its Dremio platform dependency fits the organization’s plans.

Key terms: Dremio catalog, Apache Polaris, Iceberg REST Catalog specification, compatible engine, RBAC, row-level filters, column masking, lineage graphs, natural language discovery, compaction

7.4.10 Snowflake Open Catalog

This section describes how Snowflake Open Catalog provides managed, REST-compatible access to Iceberg tables across engines, and explains its catalog types, security controls, and tradeoffs.

• Managed multi-engine catalog access: Explain how Snowflake Open Catalog lets engines other than Snowflake access Iceberg tables without moving compute to Snowflake.
  Why it matters: A managed catalog need not lock your Iceberg tables to one compute engine.
• Internal and external catalog behavior: Distinguish internal catalogs from external catalogs, especially how non-Snowflake engines can use each.
  Why it matters: The catalog type determines whether an external engine can write, not merely whether it can connect.
• Scoped storage and role access: Describe how storage configuration, roles, and credential vending constrain engine access.
  Why it matters: Shared metadata across engines still requires controlled access to catalog resources and storage locations.
• Adoption strengths and constraints: Justify when Open Catalog is an attractive choice and identify the stated limits to consider.
  Why it matters: Its interoperability and managed operation must be weighed against deployment, write-access, and request-cost constraints.

Key terms: Snowflake Open Catalog, Apache Iceberg REST Catalog specification, Apache Polaris, multi-engine access, internal catalogs, external catalogs, credential-vended access, base directory hierarchy, RBAC, principal roles

7.4.11 Databricks Unity Catalog

This section explains how Databricks Unity Catalog combines governance with Iceberg support and when it may fit an Iceberg lakehouse.

• Unified governance across assets: Explain how Unity Catalog provides a shared governance layer for assets across Databricks workspaces, including access control, auditing, lineage, and discovery.
  Why it matters: The shared catalog lets different Databricks teams work under consistent governance.
• Iceberg support and interoperability: Describe the Iceberg operations Unity Catalog supports and why they matter to Databricks users adopting an Iceberg lakehouse.
  Why it matters: Iceberg support offers Databricks users a way to adopt an open table format without giving up their existing working environment.
• Unity Catalog selection fit: Explain the circumstances the section suggests considering when evaluating Unity Catalog for Iceberg.
  Why it matters: A catalog’s governance benefits matter most when they fit the organization’s platform and deployment choices.

Key terms: Unity Catalog, shared catalog layer, fine-grained access control, lineage, native Iceberg support, external Iceberg tables, cloud object storage, interoperability, Databricks, Delta

7.5 Choosing the right catalog: Evaluating options through scenarios

The section frames catalog selection as a context-dependent evaluation rather than a search for one universally best option.

• Context-dependent catalog selection: Explain why an Iceberg catalog should be chosen against an organization’s requirements rather than by a feature checklist or a universal recommendation. Describe the main questions the section suggests asking when comparing options.
  Why it matters: A catalog that looks strong on paper may not fit your team’s infrastructure, governance needs, or capacity to operate it.

Key terms: Iceberg REST Catalog specification, governance and access control, operational overhead, performance and scale, community and support

7.5.1 Scenario: A mid-sized data team migrating from Hive

This scenario shows why a team migrating gradually from Hive chooses the Hive catalog first while leaving room to change catalogs as its lakehouse evolves.

• Phased catalog choice for Hive migration: Justify the team's initial choice of the Hive catalog over the Hadoop catalog, and explain why it plans to reevaluate that choice during its move to a cloud-based lakehouse.
  Why it matters: A catalog that fits the current environment need not be the final choice for the lakehouse.

Key terms: Hive catalog, Hadoop catalog, Hive Metastore (HMS), Nessie, Polaris

7.5.2 Scenario: A rapidly scaling cloud-native startup

This scenario explains why an AWS-focused, fast-growing startup chooses AWS Glue Data Catalog now while keeping a cloud-agnostic catalog in mind for future expansion.

• AWS Glue catalog fit: Explain why the startup selects AWS Glue Data Catalog for its Iceberg lakehouse, connecting the choice to its AWS-based stack and preference for managed services.
  Why it matters: A catalog choice should fit both the platform’s ecosystem and the team’s capacity to operate it.
• Portability tradeoff and revisit: Distinguish Glue’s current AWS-centered advantages from the cloud-agnostic, REST-compliant appeal of Polaris-based catalogs, and describe why the team may revisit them.
  Why it matters: The best catalog for today’s environment may need reevaluation when portability becomes more important.

Key terms: AWS Glue Data Catalog, Apache Iceberg, fully managed, serverless, Polaris, Iceberg REST specification, cloud-agnostic, cloud portability, Lake Formation

7.5.3 Scenario: A multinational enterprise with strict data governance

This scenario explains why a multinational healthcare enterprise selects Apache Gravitino for federated metadata governance across regions, clouds, and on-premises systems.

• Gravitino fit for federated governance: Justify the enterprise’s choice of Apache Gravitino by connecting its distributed, multi-engine governance requirements to Gravitino’s metadata model and federated capabilities. Describe how the choice supports consistent policies and Iceberg tables.
  Why it matters: The scenario shows how catalog selection follows an organization’s governance and interoperability needs rather than a single feature.

Key terms: Apache Gravitino, federated governance, unified governance layer, REST Catalog specification

7.5.4 Scenario: A SaaS startup prioritizing operational simplicity

This scenario shows why a small AWS-based SaaS team chooses AWS Glue Data Catalog for its Iceberg lakehouse.

• Managed AWS catalog choice: Justify the startup’s choice of AWS Glue Data Catalog by connecting its small team, AWS-based Spark and Athena workloads, and need for low operational overhead to Glue’s managed service and AWS integrations. Contrast that choice with the additional infrastructure management required by the alternatives considered.
  Why it matters: The choice illustrates how a team’s existing tools and capacity to run infrastructure can determine which catalog fits best.

Key terms: AWS Glue Data Catalog, Apache Iceberg, Athena, Glue ETL, Lakekeeper, JDBC catalog

7.5.5 Scenario: A large enterprise with multicloud and federated governance needs

This scenario explains why a global financial firm chooses Apache Gravitino for a multicloud Iceberg lakehouse with federated metadata and consistent governance.

• Gravitino selection for federated governance: Justify the firm’s choice of Apache Gravitino over Nessie and Polaris by connecting its federated metadata model, governance features, REST compatibility, and extensibility to the firm’s multiregion, multicloud, and multi-engine requirements.
  Why it matters: The scenario shows how catalog selection follows an organization’s governance and interoperability needs, not just its table format.

Key terms: Apache Gravitino, federated metadata lake, unified governance, Iceberg REST, extensibility

7.5.6 Scenario: A financial firm requiring daily environment cloning for stress testing

This scenario explains why a financial firm chooses Nessie to create reproducible, isolated stress-test environments without copying production data.

• Branch-Based Stress-Test Environments: Explain how the firm uses Nessie branches to run daily stress tests on a snapshot of production data without duplicating datasets or affecting live operations.
  Why it matters: Branch-based isolation makes frequent stress tests reproducible without the time and cost of copying large datasets.
• Nessie Catalog Selection Rationale: Justify the firm’s choice of Nessie over the other catalog options considered, and describe how daily branches fit its testing workflow.
  Why it matters: The catalog choice follows the firm’s need for repeatable, isolated tests rather than REST compatibility alone.

Key terms: Nessie, Git-like branching, isolated metadata views, zero-copy cloning, catalog-level branching, tagging, multitable zero-copy cloning, CI/CD workflows

7.5.7 Scenario: Phased Iceberg migration with query federation across legacy systems

This scenario explains why an enterprise migrating in phases from legacy systems to Iceberg selects the Dremio catalog to preserve analytics access and governance during the transition.

• Federated phased migration: Explain how the Dremio catalog supports a phased Iceberg migration while analysts continue querying legacy data.
  Why it matters: The migration can proceed gradually without interrupting analysts’ access to existing data.
• Catalog choice and tradeoffs: Justify the Dremio catalog choice against the other evaluated options and describe the governance benefits cited for the transition.
  Why it matters: Catalog selection depends on the enterprise’s combined access and governance needs, not Iceberg compatibility alone.

Key terms: Dremio catalog, query federation, Iceberg tables, single SQL interface, Hadoop catalog, JDBC catalog, Polaris, Gravitino, AWS Glue Data Catalog, row-level security

7.5.8 Scenario: A lightweight lakehouse adoption with Hadoop catalog and Python

This scenario shows why a Python-focused startup chooses a lightweight Hadoop catalog and PyIceberg setup to begin using Iceberg without adding catalog infrastructure.

• Lightweight catalog selection rationale: Justify the startup’s choice of the Hadoop catalog over a cloud data warehouse or a more feature-rich catalog, using its stated requirements and the tradeoffs it evaluated.
  Why it matters: The right catalog depends on the team’s current needs, not simply on which option has the most features.
• Python and filesystem workflow: Describe how the startup uses the Hadoop catalog with PyIceberg to create and manage Iceberg tables in S3 from local Python scripts without provisioning another service.
  Why it matters: This explains how the team can start using Iceberg while keeping its initial operational setup small.
• Incremental catalog migration: Explain how the chosen starting point leaves room for the startup to adopt a REST-based or enterprise-grade catalog later.
  Why it matters: A lightweight initial choice need not commit the team to that catalog forever.

Key terms: Hadoop catalog, REST catalogs, PyIceberg, filesystem paths, Amazon S3, REST-based, enterprise-grade catalog, table layout, pipelines

7.6 Catalog-based access control

Explains how Iceberg catalogs can centralize lakehouse access control, use scoped storage credentials, and accommodate current limits in fine-grained security.

• Catalog as security authority: Explain how a catalog-centric model centralizes access decisions for Iceberg tables and why that helps multi-engine lakehouses.
  Why it matters: A shared authority makes governance more consistent as teams and query engines multiply.
• Scoped credential vending: Describe how credential vending lets authorized work reach data files without giving users or engines long-lived storage access.
  Why it matters: Temporary, limited credentials reduce the exposure and management burden of storage access.
• Current access control limits: Distinguish the catalog controls available today from fine-grained policies that may still require other layers, and explain the recommended transitional approach.
  Why it matters: Recognizing these limits helps avoid assuming that centralizing security eliminates every storage or engine policy today.

Key terms: catalog-centric model, RBAC, namespaces, control plane, credential vending, short-lived, scoped credentials, object storage, coarse-grained controls, fine-grained access control, FGAC

7.7 Summary