PyG InMemoryDataset packages graph preprocessing into a repeatable pipeline when the processed dataset fits in memory. It preserves raw inputs, filtering and transformations, and produces graph objects for training workflows. section 8.3.4
A custom dataset typically follows this pipeline:
- Subclass InMemoryDataset and initialize it with a root directory plus optional transform and pre_transform arguments. section 8.3.4
- Declare raw and processed file locations, then implement download when external acquisition is needed. section 8.3.4
- Implement process to convert source graphs into Data objects, append them to a list, collate them, and save the resulting data and slices. section 8.3.4
- Load the processed data and slices in the constructor for later graph access and batching. section 8.3.4
Social-network edge list
A process method can read a NetworkX edge list, map node names to integer indices, create node features, labels, edge_index, and train/test masks, then wrap the graph in Data. section 8.3.4
Figure 8.15 Steps to preprocess data in PyG. From raw files, there are essentially two paths to prep data for ingestion by a PyG algorithm. The first path, shown here, directly creates an iterator of data instances, which is used by the dataloader. The second path mimics the first but performs this process within the dataloader class.
The preprocessing diagram contrasts direct on-the-fly graph iteration with the dataset-based path that preserves preprocessing before DataLoader ingestion. section 8.3.4
