Data engineering prepares data for analysis and machine learning by carrying it from capture through pipelines, transformation, storage, and delivery. section 1.2.4 section 1.3
A practical data engineering workflow typically:
- Captures, loads, cleanses, conforms, reshapes, transforms, and ingests data. section 1.2.4
- Uses an extract-transform-load process to place prepared data on an accessible server. section 1.2.4
- Creates variables, treats null values, enriches records, and makes the resulting data ready for analysis or model building. section 1.2.4
From capture to analysis
A pipeline can load captured data, clean and reshape it, enrich it with new variables, handle missing values, and deliver it for analysis, reporting, or machine learning. section 1.2.4 section 1.3
Figure 1.6 Data engineering paves the way for data analysis. It involves data loading, transformation, enrichment, cleaning, preparation, etc., which leads to the creation of data ready for analysis.
The workflow diagram shows how loading, transformation, enrichment, cleaning, and preparation produce data ready for analysis. section 1.2.4
