A self-managed catalog is a deployment in which an organization installs, operates, secures, upgrades, and maintains catalog software in infrastructure it controls. It avoids licensing fees in many open-source implementations but transfers infrastructure and engineering responsibility to the organization. section 7.2.5 section 7.4
Evaluate the pattern across its main operational responsibilities and tradeoffs:
- Plan infrastructure for deployment, security, upgrades, high availability, and potentially multiregion operation. section 7.2.5 section 7.4
- Provide monitoring, structured logging, alerting integration, and recovery procedures to limit manual troubleshooting. section 7.2.5
- Balance customization and portability against personnel costs, maintenance effort, support availability, and operational complexity. section 7.2.5 section 7.7
Open-source catalog deployment
Apache Polaris, Project Nessie, Apache Gravitino, and Lakekeeper illustrate self-managed catalog choices. They provide capabilities such as REST access, versioning, federation, or multitenancy, while requiring infrastructure planning and ongoing oversight. section 7.4
Table 7.1 A comparison of Apache Iceberg catalogs
| Name |
OSS |
REST catalog |
Features |
|---|---|---|---|
| Hadoop |
Yes |
No |
No service required; potential consistency issues |
| JDBC |
Yes |
No |
Can use any JDBC database as a catalog |
| Hive |
Yes |
No |
Leverage existing Hive catalog |
| Apache Polaris |
Yes |
Yes |
RBAC; catalog federation; multi-tenancy |
| Project Nessie |
Yes |
Yes |
Git-like catalog versioning |
| Apache Gravitino |
Yes |
Yes |
Federation with Hive; geodistributed catalog |
| Lakekeeper |
Yes |
Yes |
Rust-based; supports data contracts |
| AWS Glue Data Catalog |
No |
Yes |
Integrated in AWS tooling |
| Dremio catalog |
Polaris-based |
Yes |
Commercial service built on Apache Polaris with automated table maintenance; charges based on compute used for maintenance; external requests not charged |
| Snowflake Open Catalog |
Polaris-based |
Yes |
Commercial service built on Polaris; charges on requests to the catalog |
| Databricks Unity |
No (OSS version different codebase) |
Yes |
Supports Iceberg and Delta Lake |
Compare self-managed and managed Apache Iceberg catalogs by service requirements, REST support, features, portability, and operational implications. section 7.4
