Project Nessie is an open-source catalog and metadata coordination layer for Apache Iceberg. It adds Git-like branches, tags, commits, and merges to data-lake metadata while leaving table data and Iceberg metadata files in object storage. section 7.4.5 section 3.3
Nessie supports lakehouse workflows through several related mechanisms:
- Records table changes as commits and groups them into branches, allowing development, staging, and production states to coexist. section 7.4.5
- Uses immutable tags as references to known points in time and mutable branches for evolving work. section 7.4.5 section 11.2.2
- Supports isolated ingestion or analytics work on a branch, then merges successful changes atomically into the mainline. section 7.4.5
- Coordinates transactions spanning multiple tables, which helps keep related schema or ingestion changes consistent. section 7.4.5
- Provides a REST service with multilingual client libraries, integrations for Spark, Flink, and Trino, and support for the Iceberg REST Catalog interface. section 7.4.5
Nessie coordinates catalog state but does not directly manage table data or Iceberg metadata files. Iceberg handles table operations and storage, while Nessie maintains a consistent version log. This separation supports scalable storage designs but makes catalog availability and backup operational concerns. section 7.4.5 section 11.3.1
Local Iceberg lakehouse
A Docker Compose environment can run Nessie beside MinIO, Apache Spark, Dremio, PostgreSQL, and Apache Superset. Spark reads sales data from PostgreSQL and writes Iceberg tables to MinIO through Nessie. Dremio connects to the same catalog and storage, discovers the tables, and serves SQL views to Superset dashboards. The example uses Nessie's in-memory backend, making it appropriate for development or testing rather than persistence across restarts. section 3.2 section 3.2.2 section 3.2.3
version: "3" #1
services: #2
nessie: #3
image: projectnessie/nessie:latest
container_name: nessie
networks:
lakehouse-net:
ports:
- 19120:19120
Use the Docker Compose service definition to inspect a minimal Nessie deployment, including its image, network, and REST API port. section 3.2.2
For production, enable authentication and governance controls, choose a persistent backend, and export or back up catalog state. Branching can isolate experiments and support rollback, but recovery still depends on preserving the correct Iceberg metadata locations in storage. section 3.4.2 section 11.3.1
Daily stress-test environments
A financial firm can branch production metadata at the start of each stress-test cycle, run simulations without duplicating data, and merge or discard the results afterward. Automating branch lifecycles supports reproducible environments, auditability, and CI/CD integration without exposing incomplete changes to production. section 7.5.6
