ETL (extract, transform, load) turns raw data from one or more sources into a schema-compliant representation ready for preprocessing or training. In graph projects, it is the bridge between nongraph databases or files and usable nodes, edges, and relationships. section 8.3.2
A practical graph ETL pipeline proceeds as follows:
- Extract data from databases, CSV files, JSON files, or other established data sources. section 8.3.2
- Join records as needed and apply the chosen schema and graph data model. section 8.3.2
- Transform relationships into graph representations such as an edge list or adjacency list, then save them in a manipulable format such as CSV or text. section 8.3.2
- Load the resulting files into NetworkX for exploratory analysis and visualization, or into PyG for preprocessing and training. section 8.3.2
Recruitment relationships
A recruitment dataset stored in CSV and JSON can be read into a Python dictionary, converted into an adjacency list and an edge list, and written to files. The graph can then be inspected in NetworkX and prepared in PyG. For an undirected graph, transformation also ensures that relationships are represented symmetrically and avoids duplicate edges. section 8.3.2
Figure 8.9 A graph with nodes and edges marked (top). An edge list representation (middle); each entry contains the edge number and the pair of nodes connected. An adjacency list representation in a dictionary (bottom); each key is a node, and the values are its adjacent nodes.
Compare the same graph as nodes and edges, an edge list, and an adjacency-list dictionary. section 8.3.2
A schema reduces ambiguity during transformation. When suitable graph data already exists, published datasets, semantic models, and benchmarks can avoid building the pipeline from scratch; however, public datasets may omit explicit schemas, and small benchmarks may not scale to large data. section 8.4
