Software engineering applies organized practices to designing, developing, deploying, and maintaining software. For data work, it turns exploratory code into software that is easier to read, test, share, reuse, and maintain. Its goal is reliable, scalable, efficient applications rather than code that only works once on a local dataset. section 1.0 section 1.1
A common starting point is a collection of Jupyter Notebooks: useful for trying ideas, visualizing data, and recording results, but difficult to manage as a project grows. Code may be spread across notebooks and languages, duplicate functions, lack a shared structure, and become hard to integrate into another application. Engineering practices provide structure without removing the value of experimentation. section 1.0 section 1.1
Software engineering addresses recurring data science problems with practical techniques:
- Write robust, structured code and use object-oriented programming to reduce scattered or spaghetti code. section 1.1
- Use source control so several contributors can work on one codebase and recover earlier changes. section 1.1
- Use chunking or distributed parallelization when model-training data does not fit in memory. section 1.1
- Test code against varied inputs and edge cases rather than assuming a prototype always works. section 1.1
- Schedule extraction or prediction work when data must be retrieved or models must run repeatedly. section 1.1 section 1.2.5
From a churn prototype to a working service
A churn model has little operational value if it remains in a data scientist’s files and never sees new customer data. Putting it into production means arranging recurring or real-time predictions so others can use the result. The workflow also needs to anticipate failures: a scheduled training query might return zero rows, which could trigger an alert to an on-call engineer instead of silently producing a bad update. section 1.2 section 1.2.8 section 1.2.5
These practices also support collaboration among data scientists, data engineers, and software engineers. Responsibilities can overlap: data engineers build data workflows and manage databases, data scientists analyze and model data, and software engineers build reliable applications. Shared engineering habits make their code and operational work easier to connect. section 1.1 section 1.2.8
Table 1.1 An overview of data science-related problems and corresponding software engineering solutions
| Data science problem | Software engineering solution |
Messy (spaghetti) code scattered across Jupyter Notebook files |
Robust code-writing and object-oriented programming |
Difficult to collaborate on the same codebase |
Source control |
Training models on data that doesn’t fit into memory |
Scaling techniques, such as chunking or distributed parallelization |
Turning a model from a prototype into one actively used by others |
Putting the model into production |
Hard to know whether your code works for a variety of inputs or edge cases |
Testing principles |
Repeatedly retrieving data from a database or new source files |
Scheduling data extraction code to run automatically |
Compare familiar data science workflow problems with the engineering techniques that address them, from messy notebooks to production models and scheduled extraction. section 1.1

