Catalog

As explained in Architecting an Apache Iceberg Lakehouse

A catalog is a service or system that records, governs, and exposes metadata about data assets such as tables, schemas, locations, and access permissions.

A catalog is the coordination layer that connects data assets, metadata, storage, compute engines, and access policies. In an Apache Iceberg lakehouse, it records table identifiers, namespaces, locations, and the metadata version representing each table’s current state. Engines use it to discover tables, plan queries, and commit updates consistently across shared environments. section 7.1 section 7.1.1

A catalog typically performs four related jobs:

• Registers tables under namespaces so engines can resolve names such as analytics.marketing.sales_data to metadata locations. section 7.1.1
• Associates each table identifier with its current metadata objects, preserving versioned history for time travel and rollback. section 7.1 section 7.1.1
• Coordinates optimistic transactional commits by validating proposed metadata changes against concurrent updates and requiring retries when the commit sequence changes. section 7.1.1
• Governs metadata and storage access through IAM or SSO integration, RBAC or ABAC, credential vending, encryption, audit trails, and lineage. section 7.2.2 section 7.2.3

Catalog design separates storage, compute, and metadata coordination, enabling multiple engines such as Spark, Flink, and Trino to share table state. The tradeoff is operational: lightweight Hadoop or JDBC catalogs minimize dependencies, while self-managed REST catalogs add flexibility and advanced capabilities; managed services simplify operations but can reduce customization or portability. section 7.1 section 7.4

Choosing across environments

A simple development deployment may use a Hadoop or JDBC catalog to avoid additional services. A multi-tenant enterprise deployment may instead evaluate a REST catalog for interoperability, governance, concurrency, operational overhead, cloud strategy, and cost. A cross-region recovery design must replicate catalog state alongside data and metadata, then address absolute file paths when reconstructing tables. section 7.4 section 7.5 section 11.3.3

Figure 7.2 An Apache Iceberg catalog maps each table name to the location of its latest metadata file, which serves as the entry point for tools to begin scanning and interacting with the table.

Shows how a table name maps to the latest metadata file, the entry point engines use to scan and update the table. section 7.1

Table 7.1 A comparison of Apache Iceberg catalogs

Name	OSS	REST catalog	Features
Hadoop	Yes	No	No service required; potential consistency issues
JDBC	Yes	No	Can use any JDBC database as a catalog
Hive	Yes	No	Leverage existing Hive catalog
Apache Polaris	Yes	Yes	RBAC; catalog federation; multi-tenancy
Project Nessie	Yes	Yes	Git-like catalog versioning
Apache Gravitino	Yes	Yes	Federation with Hive; geodistributed catalog
Lakekeeper	Yes	Yes	Rust-based; supports data contracts
AWS Glue Data Catalog	No	Yes	Integrated in AWS tooling
Dremio catalog	Polaris-based	Yes	Commercial service built on Apache Polaris with automated table maintenance; charges based on compute used for maintenance; external requests not charged
Snowflake Open Catalog	Polaris-based	Yes	Commercial service built on Polaris; charges on requests to the catalog
Databricks Unity	No (OSS version different codebase)	Yes	Supports Iceberg and Delta Lake

Compares Iceberg catalog implementations by service model, REST support, governance features, and operational tradeoffs. section 7.4

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes

Explains table resolution, metadata versioning, namespaces, optimistic commits, governance, implementation categories, and recovery constraints.

Examples

Uses namespaces, optimistic commit retries, catalog implementation comparisons, Trino integrations, and cross-region replication scenarios.

What it assumes

Assumes familiarity with data platforms and Iceberg tables while grounding catalog responsibilities in storage, metadata, and compute separation.

What you can do after

Evaluate catalog options against interoperability, governance, operational overhead, performance, support, compliance, cloud strategy, and growth.

How it connects

Connects Apache Iceberg tables with metadata, namespaces, snapshots, schema evolution, transactions, query engines, access control, lineage, and object storage.

liveBookmanning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

• producthas part · A product is a good, service, software offering, or other managed offering made available to customers or users to provide value.
• Hadoop catalogincludes · A catalog implementation that resolves Apache Iceberg tables from filesystem paths and stores their metadata without requiring a separate catalog service.
• bookhas part · A book is a written or printed work, typically organized into pages or sections and published as a physical volume or digital equivalent.
• namespaceuses · A namespace is a naming context or identifier space in which names are defined and resolved, distinguishing them from names in other contexts.
• optimistic concurrency controluses · Optimistic concurrency control is a concurrency-control technique that allows operations to proceed with limited locking and detects conflicting changes through validation before committing updates.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.