A data lake is a large storage area for files of different structures and sizes. It can hold incoming data before that data has a permanent home, preserve files for backup, or keep them available for future analysis. Data can be parked first and transformed later. section 3.4.2 section 8.5.1
Choose the storage role before choosing a service:
- Use a data lake when large volumes of varied files need a place to land, especially when their later use is not yet settled. section 8.5.1
- Consider a relational database for similarly structured data in linked tables, or a NoSQL store when documents may differ substantially. section 8.5.1
- Consider a data warehouse when the main job is reporting and analysis over large datasets. section 8.5.1
Park the data, then decide
Suppose collecting and managing data is central to an organization’s technology vision, but the eventual transformations are not yet clear. A data lake provides a starting place for the incoming data; AWS S3 is one possible starting point. The storage and management approach can be treated as a supporting service rather than built from scratch. section 3.4.2
Keeping data for possible future needs has a business rationale. When storage was more expensive to run, organizations often retained only what they needed immediately; if requirements changed, discarded historical data could not be recovered. Ask whether incoming files need a permanent home now, later transformation, backup, or a store designed specifically for reporting. section 8.5.1
