liveBook · Manning

Catalog requirements

As explained in Architecting an Apache Iceberg Lakehouse

Catalog requirements are the functional and nonfunctional capabilities specified for selecting, deploying, governing, and operating a lakehouse catalog.

Catalog requirements define how an Iceberg catalog selects, serves, governs, and coordinates table metadata. The design influences scalability, operational complexity, interoperability, and lakehouse evolution. section 4.3.3

Evaluate the catalog against these architectural requirements:

  • Support the Iceberg REST catalog specification when multiple engines or orchestration tools must read and write metadata through a consistent API. section 4.3.3
  • Verify read and write capabilities for every engine because REST support may be partial. section 4.3.3
  • Provide governance controls, including table-level policies and support for role-based or attribute-based access control. section 4.3.3
  • Clarify how metadata is tracked, including file locations or managed relational and service-backed metadata stores. section 4.3.3
  • Account for concurrency, advanced features such as multi-table transactions, and the operational cost of custom adapters. section 4.3.3
Table 4.4 Iceberg catalogs tracking metadata by tracking the locations of metadata files

Table

Metadata location

sales_data_2024

s3://company-data/warehouse/sales_data_2024/metadata/00004-abcdefg.metadata.json

customer_profiles

s3://company-data/warehouse/customer_profiles/metadata/00012-hijklm.metadata.json

inventory_snapshots

s3://company-data/warehouse/inventory_snapshots/metadata/00003-nopqrs.metadata.json

web_traffic_logs

s3://company-data/warehouse/web_traffic_logs/metadata/00008-tuvwxy.metadata.json

Illustrates a file-tracking catalog by mapping table names to Iceberg metadata locations. section 4.3.3

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes
Compares REST compatibility, metadata models, governance, and adapter tradeoffs but stops short of implementation configurations.
Examples
Uses a table mapping names such as sales_data_2024 to object-storage metadata locations.
What you can do after
Evaluate catalog options against engine compatibility, governance controls, metadata storage, concurrency, and operational complexity.
How it connects
Connects catalogs with Apache Iceberg, metadata management, governance, RBAC, ABAC, concurrency, and multi-table transactions.
Cover of Architecting an Apache Iceberg Lakehouse
liveBookmanning.com
Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.