1 What is data engineering?

Data collection is on the rise. More and more systems are generating more and more data every day.1

More than 30,000 gigabytes of data are generated every second, and the rate of data creation is only accelerating.

--Nathan Marz

Increased connectivity has led to increased sophistication and user interaction in software systems. New deployments of connected “smart” electronics also rely on increased connectivity. In response, businesses now collect and store data from all aspects of their products. This has led to an enormous increase in compute and storage infrastructure. Writing for Gartner, Mark Beyer defines “Big Data.”2

Big Data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery, and process optimization.

--Mark A. Beyer

The scale of data collection and processing requires a change in strategy.

1.1 What is data engineering?

1.2 What do data engineers do?

1.3 How does Microsoft define data engineering?

1.3.1 Data acquisition

1.3.2 Data storage

1.3.3 Data processing

1.3.4 Data queries

1.3.5 Orchestration

1.3.6 Data retrieval

1.4 What tools does Azure provide for data engineering?

1.5 Azure Data Engineers

1.6 Example application

Summary