liveBook · Manning

Provenance

As explained in Digital Twins in Action

Lineage tracking records the origins, transformations, dependencies, and destinations of data or derived artifacts.

Provenance records where data came from and how it became a derived dataset. In an ETL pipeline, traceability includes the transformations applied to raw data, their order, and their parameters. Clear, reproducible lineage helps establish trust in analytical results and model outputs. section 4.4.4

For an industrial digital twin, a decision may require lineage to the source system, transforming pipeline, validator, and validation time. Extracted values may also need links to a document revision, validation workflow, and accountable owner. This information supports audits and investigations; it does not replace expert validation. section 4.5.4 section 2.6

Weekly power consumption

Minute-by-minute meter data in a CSV can be transformed into weekly consumption and cost totals for a dashboard. Traceability would identify the CSV and preserve the aggregation steps and parameters, so the derived totals can be understood and reproduced. The Spark example performs the aggregation; it does not demonstrate a complete lineage-recording system. section 4.4.4

Listing 4.9 An example of using Apache Spark to extract minute-by-minute power
from pyspark.sql import SparkSession
from pyspark.sql.functions import date_trunc, sum as spark_sum, avg, col
from pyspark.sql.types import *

spark = SparkSession.builder.appName("WeeklyUsage").getOrCreate()

schema = StructType([  #1
    StructField("datetime_utc", TimestampType(), True),
    StructField("datetime_local", TimestampType(), True),
    StructField("watt_hours", DoubleType(), True),
    StructField("cost_dollars", DoubleType(), True),
    StructField("is_peak", BooleanType(), True)
])

df = spark.read.option("header", "true").schema(schema).csv(
  "powerpal_data.csv")

weekly_usage = df.groupBy(  #2
    date_trunc("week", col("datetime_local")).alias("week_start")
).agg(
    (spark_sum("watt_hours") / 1000).alias("total_kwh"),
    spark_sum("cost_dollars").alias("total_cost_dollars")
).orderBy("week_start")

print("Weekly Usage (kWh) and Cost:")
weekly_usage.show()

spark.stop()

Inspect the CSV schema and weekly aggregation to see the input and transformations that lineage would need to describe. section 4.4.4

Provenance becomes more demanding when vendors, contractors, and internal teams supply conflicting data. Clear source ownership helps establish who can resolve disagreements, while complete lineage helps explain decisions such as extending a maintenance interval. Feature stores also use lineage tracking alongside versioning to manage engineered features. section 4.5.4 section 4.3.6

How Digital Twins in Action covers it

How deep it goes
Defines lineage requirements and illustrates transformations with Spark, but stops short of implementing a complete lineage-recording system.
Examples
Uses weekly power consumption and cost aggregates, document revisions, and maintenance-interval investigations to illustrate lineage needs.
What you can do after
Evaluate and understand the lineage information needed to reproduce derived data and audit industrial decisions.
How it connects
Connects provenance to ETL, data quality, digital twins, and feature stores.

Also explained in

Compare how 12 books explain provenance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Digital Twins in Action. The section links point to the parts of the book it draws on.