Data infrastructure

As explained in Effective Data Analysis

Data infrastructure is the collection of systems, tools, and processes that gathers, stores, processes, retrieves, and presents organizational data for analysis.

Data infrastructure is the collection of systems, tools, and processes that gathers, stores, processes, retrieves, and presents organizational data for analysis. Its scale ranges from a shared drive with daily exports to many tools processing billions of records. section 11.2 section 11.5

A functioning infrastructure typically supports these connected activities:

• Gather information from multiple sources on a regular cadence. section 11.2
• Store structured, semistructured, or unstructured data in suitable repositories, including data warehouses, document databases, or data lakes. section 11.5
• Process and structure data through ETL or ELT before analysis or within the target warehouse. section 11.5
• Retrieve accessible, coherent data so analysts can generate insights for stakeholders and support decisions. section 11.5

Renewable-energy startup proposal

Recommend gathering, storage, and processing tools for utility usage, website activity, advertising interactions, customer tiers, and renewable-energy purchases. Compare Snowflake, MongoDB, Redshift, BigQuery, and Azure, then justify whether a data lake fits the company’s needs over the next one to two years. section 11.2.4

Figure 11.4 When data reaches your doorstep, it often looks nothing like it did at the source.

Use the figure to see how source data can change substantially before becoming an analytics resource. section 11.2

How Effective Data Analysis covers it

How deep it goes

Explains collection, storage, processing, retrieval, ETL, and ELT at overview level, while deferring detailed tool implementation to documentation.

Examples

Uses transformed source data, renewable-energy customer data, technology comparisons, and a data-lake recommendation exercise.

What you can do after

Evaluate and recommend storage, collection, processing, and data-lake options for an organization’s anticipated analytical needs.

How it connects

Connects data warehouses, document databases, ETL, ELT, data lakes, big data, and distributed computing frameworks.

liveBookmanning.com

Read Effective Data Analysis on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentData infrastructure is the hardware, software, storage, networking, and operational environment that supports data access and processing.

Compare how 2 books explain data infrastructure →

Related concepts

• pipelineused by · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• platformuses · A platform is a foundational computing system, environment, or online service that provides capabilities or infrastructure for building, running, delivering, or interacting with applications and services.
• data streamproduces · A data stream is an ordered sequence of data items or events made available incrementally for consumption or processing.
• data collectionhas part · Data collection is the process of systematically acquiring or gathering information, observations, or measurements from one or more sources for a defined purpose.
• data analysisused by · Data analysis is the systematic examination, transformation, and interpretation of data to derive information, identify patterns, or support conclusions and decisions.

This explainer was generated by Manning's knowledge graph from the text of Effective Data Analysis. The section links point to the parts of the book it draws on.