An ingestion layer is the set of tools and frameworks that brings raw data from operational databases, message queues, cloud services, or vendors into Iceberg tables. It transforms and cleans data, then loads it through batch, micro-batch, or streaming flows. Unlike the storage layer, which preserves data, ingestion determines how quickly, cleanly, and reliably data arrives. section 2.7.2 section 6.0
Design the layer around explicit requirements:
- Match batch, micro-batch, or streaming ingestion to latency and throughput needs. section 6.0 section 6.1
- Account for reliability, fault recovery, schema evolution, data quality, auditability, and operational complexity. section 6.0 section 6.1
- Select tools according to catalog integration, transformation needs, deployment maturity, scaling, and cloud environment. section 6.4.10
Choosing an ingestion pattern
A high-frequency transaction pipeline may require streaming with low-latency writes and exactly-once guarantees, while nightly historical logs can use batch ingestion with stronger handling for schema drift and file optimization. Spark suits large-scale ETL; Flink suits real-time processing; managed platforms can reduce operational work at the cost of customization. section 2.7.2 section 6.1 section 6.4.10
Figure 2.6 Components of a complete data lakehouse implementation
Use the lakehouse component diagram to locate ingestion between source systems and Iceberg storage, alongside catalog, federation, and consumption layers. section 2.7
