Delta Lake is an open-source storage layer for large tabular datasets in cloud object storage. It adds scalable ACID transactions where key-value-oriented stores such as Amazon S3 make consistency and metadata operations difficult. Delta Lake tables hold batch and streaming data while managing table metadata, caching, indexing, and schema enforcement. section 2.3.1
A Delta Lake table is the core working unit. Like a SQL table, it supports queries, inserts, updates, and merges. Write-time schema validation protects tables from data pollution, while table history enables time travel: readers can load a prior timestamp or version and roll a table back to an earlier state. section 2.3.1
Figure 2.16 Delta Lake data ingestion and processing workflow. Both stream data and batch data can be saved as Delta Lake tables, and the Delta Lake tables are stored in the cloud object storage, such as Amazon S3.
Batch and streaming inputs can enter Delta Lake tables stored in cloud object storage, providing a common processing layer for both ingestion modes. section 2.3.1
A practical dataset-management flow organizes tables by refinement level:
- Bronze tables retain raw inputs from multiple sources, including imperfect data. section 2.3.1
- Silver tables receive cleaned and transformed data through ETL. section 2.3.1
- Gold tables hold filtered and purified data for downstream use. section 2.3.1
For Spark-based machine-learning pipelines, Petastorm converts data from Delta Lake tables into dataset objects consumable by PyTorch or TensorFlow training code. This offers an adoptable alternative to building a dataset-management service from scratch when the data pipeline already runs on Apache Spark. section 2.3.1 section 2.0
Recovering a table snapshot
Use DeltaTable.forPath to inspect full or recent table history. Read a historical snapshot with Spark’s delta format by setting timestampAsOf to a date or versionAsOf to a table version, then loading the table path. section 2.3.1
Figure 2.17 Petastorm converts the Delta Lake table to datasets that can be read by the PyTorch or Tensorflow framework.
Petastorm bridges a Delta Lake table and framework-readable PyTorch or TensorFlow datasets. section 2.3.1
