2 Your data science could use some engineering

 

This chapter covers

  • Elucidating the differences between a data scientist and an ML engineer
  • Focusing on simplicity in all project work to reduce risk
  • Applying Agile fundamentals to ML project work
  • Illustrating the differences and similarities between DevOps and MLOps

In the preceding chapter, we covered the components of ML engineering from the perspective of project work. Explaining what this approach to DS work entails from a project-level perspective tells only part of the story. Taking a view from a higher level, ML engineering can be thought of as a recipe involving a trinity of core concepts:

  • Technology (tools, frameworks, algorithms)
  • People (collaborative work, communication)
  • Process (software development standards, experimentation rigor, Agile methodology)

The simple truth of this profession is that project work that focuses on each of these elements are generally successful, while those that omit one or many of them tend to fail. This is the very reason for the hyperbolic and oft-quoted failure rates of ML projects in industry (which I find to be rather self-serving and panic-fueled when coming from vendor marketing materials).

2.1 Augmenting a complex profession with processes to increase project success

2.2 A foundation of simplicity

2.3 Co-opting principles of Agile software engineering

2.3.1 Communication and cooperation

2.3.2 Embracing and expecting change

2.4 The foundation of ML engineering

Summary