Provenance records how data moves through sources, transformations, storage locations, and outputs. It creates a chain of custody for auditability, compliance, debugging, and data trust. section 7.2.2 section 4.2.1
A production provenance capability typically connects:
- Users or services with metadata operations, including table creation, schema updates, and snapshot rollbacks. section 7.2.2
- Jobs with input tables, output tables, query plans, timestamps, and the resulting table versions. section 11.2.6 section 11.2.7
- Catalog policies with identities, permissions, storage access, and audit records across engines. section 7.2.2 section 11.2.6
- Ingestion flows with observable movement, retries, backpressure, and processor-level history. section 6.4.3
Tracking an Iceberg table update
A Spark job can register the OpenLineage listener, read an Iceberg table, filter records, and overwrite a reporting table. The resulting event records the job, inputs, outputs, query metadata, and timestamp; Iceberg snapshots can associate that event with a specific table version. section 11.2.6 section 11.2.7
from openlineage.spark import OpenLineageSparkListener
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("iceberg-lineage-tracking ") \
.config("spark.sql.catalog.my_catalog ", "org.apache.iceberg.spark.SparkCatalog ") \
.config("spark.sql.catalog.my_catalog.type ", "hadoop ") \
.config("spark.sql.catalog.my_catalog.warehouse ", "s3://my-bucket/warehouse ") \
.config("spark.extraListeners ", "openlineage.spark.agent.OpenLineageSparkListener ") \ #1
.config("openlineage.transport.type ", "http ") \
.config("openlineage.transport.url ", "http://openlineage-collector:5000 ") \ #2
.getOrCreate()
df = spark.read.table("my_catalog.db.critical_fact_table ")
df.filter("region = 'US'").writeTo("my_catalog.db.reporting_fact_table ").overwrite() #3
Inspect a Spark configuration and write operation that emits provenance to a centralized OpenLineage collector. section 11.2.6
