Petastorm bridges Spark-managed Delta Lake data and deep-learning input pipelines. It converts selected Delta Lake table data into dataset objects that training code can consume through PyTorch or TensorFlow, avoiding a separate hand-built dataset-serving layer when the data pipeline already uses Apache Spark. section 2.3.1 section 2.0
Figure 2.17 Petastorm converts the Delta Lake table to datasets that can be read by the PyTorch or Tensorflow framework.
The conversion boundary is the key architectural role: Delta Lake remains the managed table store, while Petastorm exposes data in forms readable by training frameworks. section 2.3.1
A practical flow starts by ingesting and transforming data into Delta Lake tables, then reading the training table as a Spark DataFrame. Select only the model inputs and labels, split the DataFrame for training and validation, and create a Petastorm Spark converter for each split. Configure the converter cache directory before creating converters. section 2.3.1
Image training pipeline
An image ETL reads JPEG files with Spark, derives labels from file paths, extracts image dimensions, and writes a Delta Lake image dataset. A PyTorch consumer then loads content and label_index columns, makes a 90/10 train-validation split, creates converters, and reads the training data through the converter’s PyTorch interface. section 2.3.1
Petastorm fits a larger dataset-management design with clear responsibilities:
- Delta Lake stores batch or streaming tables in cloud object storage and supplies metadata handling, caching, indexing, schema enforcement, transactions, and table history. section 2.3.1
- Spark performs ingestion, ETL, selection, and splitting over the table data before conversion. section 2.3.1
- Petastorm adapts prepared Spark data for PyTorch or TensorFlow readers so model code can consume the dataset directly. section 2.3.1
This approach is a pragmatic adopt-versus-build choice for teams whose existing pipeline is built on Apache Spark. It keeps managed dataset concerns in Delta Lake while placing framework-specific reading at the training boundary. Image preparation demonstrates that the same arrangement can handle unstructured source files after they are organized into a table. section 2.3.1 section 2.0
