Provenance records where data came from and how it became a derived dataset. In an ETL pipeline, traceability includes the transformations applied to raw data, their order, and their parameters. Clear, reproducible lineage helps establish trust in analytical results and model outputs. section 4.4.4
For an industrial digital twin, a decision may require lineage to the source system, transforming pipeline, validator, and validation time. Extracted values may also need links to a document revision, validation workflow, and accountable owner. This information supports audits and investigations; it does not replace expert validation. section 4.5.4 section 2.6
Weekly power consumption
Minute-by-minute meter data in a CSV can be transformed into weekly consumption and cost totals for a dashboard. Traceability would identify the CSV and preserve the aggregation steps and parameters, so the derived totals can be understood and reproduced. The Spark example performs the aggregation; it does not demonstrate a complete lineage-recording system. section 4.4.4
Listing 4.9 An example of using Apache Spark to extract minute-by-minute power
from pyspark.sql import SparkSession
from pyspark.sql.functions import date_trunc, sum as spark_sum, avg, col
from pyspark.sql.types import *
spark = SparkSession.builder.appName("WeeklyUsage").getOrCreate()
schema = StructType([ #1
StructField("datetime_utc", TimestampType(), True),
StructField("datetime_local", TimestampType(), True),
StructField("watt_hours", DoubleType(), True),
StructField("cost_dollars", DoubleType(), True),
StructField("is_peak", BooleanType(), True)
])
df = spark.read.option("header", "true").schema(schema).csv(
"powerpal_data.csv")
weekly_usage = df.groupBy( #2
date_trunc("week", col("datetime_local")).alias("week_start")
).agg(
(spark_sum("watt_hours") / 1000).alias("total_kwh"),
spark_sum("cost_dollars").alias("total_cost_dollars")
).orderBy("week_start")
print("Weekly Usage (kWh) and Cost:")
weekly_usage.show()
spark.stop()
Inspect the CSV schema and weekly aggregation to see the input and transformations that lineage would need to describe. section 4.4.4
Provenance becomes more demanding when vendors, contractors, and internal teams supply conflicting data. Clear source ownership helps establish who can resolve disagreements, while complete lineage helps explain decisions such as extending a maintenance interval. Feature stores also use lineage tracking alongside versioning to manage engineered features. section 4.5.4 section 4.3.6











