Full ingestion reloads every source record into the target repository on each pipeline execution, replacing the existing data. It fits low-volume workloads where users do not need historical change tracking. section 12.1.1
A full-ingestion pipeline typically has these operational characteristics:
- Replace the target with a view, or truncate the existing table and insert the current source data. section 12.1.1
- Keep implementation simple because the pipeline needs no change-detection or merge logic. section 12.1.1
- Accept slower execution and greater storage and compute consumption as data volumes grow. section 12.1.1 section 12.1.2
- Lose deleted source records and other historical values when the target is overwritten. section 12.1.1
Initial warehouse load
A bakery pipeline can initially copy all order data from JSON files and operational-system details about business partners and baked goods into the warehouse. A later scheduled reload again replaces the target with the current source data. section 12.1 section 12.0
Table 12.1 Comparison between full and incremental ingestion
|
Full ingestion
|
Incremental ingestion
|
|
|---|---|---|
|
Data scope
|
Ingests all data and inserts into the target tables
|
Ingests only new or changed data and appends or updates the target tables
|
|
Performance
|
Executes longer because all data is processed
|
Executes faster because fewer records are processed
|
|
Resource utilization
|
Consumes more resources, both in storage and compute
|
Consumes fewer resources
|
|
Code design
|
Simple because all data is ingested without additional logic
|
Complex because incremental changes are applied to the data in the target tables, and there is a greater chance of data inconsistencies if a pipeline fails
|
|
Storing historical changes in data
|
Historical changes are not preserved because data is overwritten.
|
Can preserve historical changes, depending on the design
|
|
Maintenance
|
Easy because data is overwritten every time
|
Can be challenging to troubleshoot, depending on the design
|
|
Ingestion frequency
|
Usually scheduled less frequently because it takes longer to execute
|
Can be scheduled more frequently because it takes less time to execute
|
Compare full and incremental ingestion across data scope, performance, resource use, code complexity, historical preservation, and maintenance. section 12.1.2
