A data engineer gets, moves, and manipulates the data resources an ML project needs. The role brings practical knowledge of platforms, tooling, programming languages, security, costs, and efficiency—especially when the team must work with awkward data resources. section 3.2.2
Do not assume the data engineer can explain what every data store means to the business. Previous projects may have given them that knowledge, but they may instead rely on a business translator or product owner for context. Keep this responsibility distinct from the ML engineer’s work on model management, testing, evaluation, and serving. section 3.2.2
When staffing a setup sprint, make the assignment concrete:
- Allocate a data engineer to infrastructure setup and the first data investigations. section 4.9
- Start system access requests early so the engineer can begin working with the available systems. section 4.9
Staffing a project setup
In a retail revenue-and-churn project, the delivery lead brings in a data engineer for the setup sprint to support a colleague, set up infrastructure, and start data investigations. The team then reviews its work plan before assigning additional specialists for later work. section 4.9
