ETL is the pipeline pattern of ingesting data, transforming it through one or more stages, and loading it into a warehouse or data mart. It enables analytics but can multiply storage, compute, latency, and maintenance when each target requires a separate copy. section 2.6.5 section 1.3
Architectural tradeoffs shape how ETL is used:
- Traditional pipelines repeatedly scan, rewrite, and materialize data for different systems, increasing operational complexity. section 2.6.5
- Lakehouse architectures reduce ETL and replication by keeping a canonical copy in interoperable Iceberg tables. section 1.0
- Iceberg can combine streaming and batch ingestion in one table, reducing movement between environments. section 2.6.5
Lake-to-warehouse pattern
A common arrangement ingests raw data into a lake, runs ETL stages, and loads the results into a warehouse or data mart for analytics. At scale, querying Iceberg tables directly can avoid the final duplication step. section 2.6.5
Figure 9.1 In traditional architectures, different departments replicate data not only into their departmental warehouses but also between them, increasing latency and inconsistency in the data.
Use the architecture diagram to trace how departmental replication creates latency, inconsistency, and additional ETL work. section 9.1
