A metadata catalog is the control plane and gateway for Apache Iceberg tables. It maps logical table names to current metadata locations, allowing engines to query, write, and maintain tables while the data and metadata files remain decoupled from the catalog. section 11.3.1
A catalog supports several architectural and operational responsibilities:
- Coordinate access between users, compute engines, and object storage while enforcing identity, role, row, and column policies. section 11.2.6
- Provide a recovery point for table registration: a lost catalog entry can hide intact files, while a known metadata.json location can restore visibility. section 11.3.1
- Centralize metadata across Iceberg, Hive, Kafka, HDFS, filesets, models, and topics when a federated platform is required. section 7.4.6
Notebook-based data science platform
A small data science team can combine JupyterLab, DuckDB, PyIceberg, Spark, and MLflow. Catalog-based reads support dataset discovery and security enforcement, while Iceberg snapshots keep training workflows reproducible without requiring a full BI layer. section 9.5.1
from openlineage.spark import OpenLineageSparkListener
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("iceberg-lineage-tracking ") \
.config("spark.sql.catalog.my_catalog ", "org.apache.iceberg.spark.SparkCatalog ") \
.config("spark.sql.catalog.my_catalog.type ", "hadoop ") \
.config("spark.sql.catalog.my_catalog.warehouse ", "s3://my-bucket/warehouse ") \
.config("spark.extraListeners ", "openlineage.spark.agent.OpenLineageSparkListener ") \ #1
.config("openlineage.transport.type ", "http ") \
.config("openlineage.transport.url ", "http://openlineage-collector:5000 ") \ #2
.getOrCreate()
df = spark.read.table("my_catalog.db.critical_fact_table ")
df.filter("region = 'US'").writeTo("my_catalog.db.reporting_fact_table ").overwrite() #3
A Spark configuration shows how a catalog, warehouse, and OpenLineage listener work together while reading and writing Iceberg tables. section 11.2.6
