liveBook · Manning

Time travel

As explained in Designing Deep Learning Systems: A guide for software engineers

Time travel is a data-versioning capability that retrieves a table as it existed at a specified historical point or version.

Time travel retrieves a Delta Lake table as it existed at a historical timestamp or table version. By tracking table history, it supports reproducible datasets and rollback to earlier stages. section 2.3.1

A practical workflow uses Delta Lake history to:

  • Inspect the complete table history or only the most recent operation. section 2.3.1
  • Load a historical snapshot with timestampAsOf, such as 2021-07-01. section 2.3.1
  • Load an exact table state with versionAsOf, such as version 12. section 2.3.1
Listing 2.12 Delta Lake time travel commands
pathToTable = "/my/sample/text/intent/dataset/A"
 
deltaTable = DeltaTable.forPath(spark, pathToTable)    #1
fullHistoryDF = deltaTable.history()                   #2
lastOperationDF = deltaTable.history(1)                #3
 
 
df = spark.read.format("delta")                        #4
       .option("timestampAsOf", "2021-07-01")          #4
       .load(pathToTable)                              #4
 
df = spark.read.format("delta")                        #5
      .option("versionAsOf", "12")                     #5
      .load(pathToTable)                               #5

Runnable Spark commands show how to obtain DeltaTable history and read snapshots by timestamp or version. section 2.3.1

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes
Explains historical queries, table history, and rollback, then demonstrates timestamp- and version-based reads with Spark.
Examples
Uses DeltaTable.history(), timestampAsOf with 2021-07-01, and versionAsOf with version 12.
What it assumes
Assumes Delta Lake tables and Spark data loading, while defining table history and historical snapshots through commands.
What you can do after
Diagnose dataset changes and reproduce historical table states using Delta Lake and Spark read options.
How it connects
Connects Delta Lake tables with ACID transactions, schema enforcement, object storage, Apache Spark, and dataset reproducibility.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.