5 Experimentation in action: Planning and researching an ML project

We spent the preceding two chapters focusing on the processes surrounding planning, scoping of work, and communication among a team working on an ML project. This chapter and the next two focus on the next most critical aspects of ML work as it pertains to data scientists: research, experimentation, prototyping, and MVP development.

Once a project’s requirements have been thoroughly captured from planning meetings (as much as can be realistically achieved) and the goal of the modeling solution has been defined, the next phase of creating an ML solution is to begin experimentation and research. These processes, conducted without an appropriate level of structure, can easily result in a cancelled project.

Projects may be cancelled because of a seemingly endless experimentation phase, wherein no clear direction for finalizing an approach to a solution is decided on. Stalled projects may also be the result of poor predictive capabilities. Whether due to indecision or an inability to meet accuracy expectations, the prevention of stalled and cancelled projects that have data and algorithm issues starts in the experimentation phase.

5.1 Planning experiments

5.1.1 Perform basic research and planning

5.1.2 Forget the blogs—read the API docs

5.1.3 Draw straws for an internal hackathon

5.1.4 Level the playing field

5.2 Performing experimental prep work

5.2.1 Performing data analysis

5.2.2 Moving from script to reusable code

5.2.3 One last note on building reusable code for experimentation

Summary