Data engineering
Data engineering is the discipline of designing and operating systems and processes that collect, transform, integrate, store, and deliver data for computational use.
Data Without Labels
Data engineering is the discipline of designing and operating systems and processes that collect, transform, store, and deliver data.
- How it is used
- Prepares reliable, accessible data for analysis, reporting, machine learning, and other data-driven functions.
- How deep it goes
- Traces capture through loading, ETL, cleaning, enrichment, and preparation, but stops short of implementation-specific tools or deployment procedures.
- Examples
- Uses ETL pipelines, hosted datasets, derived variables, null-value treatment, cleaned records, reports, and dashboards.
- What you can do after
- Operate data-preparation workflows that deliver cleansed, transformed, enriched, analysis-ready datasets.
Architecture Modernization: Socio-technical alignment of software, strategy, and structure
Data engineering is the discipline of designing, building, and operating systems and processes that collect, transform, store, and deliver data.
- How it is used
- Connects data collection and delivery to business decisions, analytical products, and the operational realities of architecture and team ownership.
- How deep it goes
- Traces databases, warehouses, and lakes through ingestion trade-offs, but stops short of implementation details.
- Examples
- Uses an auto-parts retailer’s receipts, loyalty data, B2B and B2C flows, warehouses, and lakes to expose ingestion complexity.
- What you can do after
- Evaluate data architectures by comparing ingestion simplicity, consumer burden, domain expertise, and operational cost.
Related concepts
- Pythonused by · Python is a high-level, general-purpose, dynamically typed programming language known for readable syntax and automatic memory management.
- analytics engineeringused by · Analytics engineering is the function of developing and maintaining transformed data models and analytical data infrastructure.
- ETLused by · ETL is a data-integration process that extracts data from source systems, transforms it, and loads the result into a target system.
- data-centric paradigmused by · A machine-learning development approach that prioritizes improving the quality and quantity of training data.
- Snowflakeuses · Snowflake is a cloud data platform providing managed storage, computing, data warehousing, processing, analytics, and related data services.
- pipelinecontrasts with · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.