Provenance is the traceable history of data used by a digital twin: where a value came from, how it changed, and how it reached a derived dataset or decision. A list of sources is a starting point, not a complete lineage record. Trustworthy outputs require transformations to be clear and reproducible. section 2.6 section 4.4.4
For data that feeds a decision, retain enough context to answer:
- Which source system or document revision supplied the value? section 2.6 section 4.5.4
- Which pipeline transformed it, in what order, and with which parameters? section 4.4.4 section 4.5.4
- Who validated it, when, and who owns the source if systems disagree? section 4.5.4
From meter readings to a dashboard
A meter exports minute-by-minute power readings as CSV. An Apache Spark job groups readings by local week, sums watt-hours into kilowatt-hours, and sums costs for a dashboard. The resulting weekly value is easier to trust when its raw input and transformation can be traced. The aggregation code shows the transformation; it does not, by itself, demonstrate that lineage metadata is recorded. section 4.4.4
Listing 4.9 An example of using Apache Spark to extract minute-by-minute power
from pyspark.sql import SparkSession
from pyspark.sql.functions import date_trunc, sum as spark_sum, avg, col
from pyspark.sql.types import *
spark = SparkSession.builder.appName("WeeklyUsage").getOrCreate()
schema = StructType([ #1
StructField("datetime_utc", TimestampType(), True),
StructField("datetime_local", TimestampType(), True),
StructField("watt_hours", DoubleType(), True),
StructField("cost_dollars", DoubleType(), True),
StructField("is_peak", BooleanType(), True)
])
df = spark.read.option("header", "true").schema(schema).csv(
"powerpal_data.csv")
weekly_usage = df.groupBy( #2
date_trunc("week", col("datetime_local")).alias("week_start")
).agg(
(spark_sum("watt_hours") / 1000).alias("total_kwh"),
spark_sum("cost_dollars").alias("total_cost_dollars")
).orderBy("week_start")
print("Weekly Usage (kWh) and Cost:")
weekly_usage.show()
spark.stop()
Inspect the input schema, weekly grouping, and summed fields to identify the steps a lineage record would need to describe. section 4.4.4
At home, provenance may be informal. In an industrial twin, a faulty reading may need to be traced during an incident investigation, or a maintenance decision justified to a regulator. Multiple owners, validation workflows, and high data volumes make complete, auditable lineage an operational requirement rather than a convenience. section 2.6 section 4.5.4
