liveBook · Manning

Software engineering

As explained in Software Engineering for Data Scientists

Software engineering is the systematic application of engineering principles to the design, implementation, testing, deployment, and maintenance of software.

Software engineering applies organized practices to designing, developing, deploying, and maintaining software. For data work, it turns exploratory code into software that is easier to read, test, share, reuse, and maintain. Its goal is reliable, scalable, efficient applications rather than code that only works once on a local dataset. section 1.0 section 1.1

A common starting point is a collection of Jupyter Notebooks: useful for trying ideas, visualizing data, and recording results, but difficult to manage as a project grows. Code may be spread across notebooks and languages, duplicate functions, lack a shared structure, and become hard to integrate into another application. Engineering practices provide structure without removing the value of experimentation. section 1.0 section 1.1

Software engineering addresses recurring data science problems with practical techniques:

  • Write robust, structured code and use object-oriented programming to reduce scattered or spaghetti code. section 1.1
  • Use source control so several contributors can work on one codebase and recover earlier changes. section 1.1
  • Use chunking or distributed parallelization when model-training data does not fit in memory. section 1.1
  • Test code against varied inputs and edge cases rather than assuming a prototype always works. section 1.1
  • Schedule extraction or prediction work when data must be retrieved or models must run repeatedly. section 1.1 section 1.2.5

From a churn prototype to a working service

A churn model has little operational value if it remains in a data scientist’s files and never sees new customer data. Putting it into production means arranging recurring or real-time predictions so others can use the result. The workflow also needs to anticipate failures: a scheduled training query might return zero rows, which could trigger an alert to an on-call engineer instead of silently producing a bad update. section 1.2 section 1.2.8 section 1.2.5

These practices also support collaboration among data scientists, data engineers, and software engineers. Responsibilities can overlap: data engineers build data workflows and manage databases, data scientists analyze and model data, and software engineers build reliable applications. Shared engineering habits make their code and operational work easier to connect. section 1.1 section 1.2.8

Table 1.1 An overview of data science-related problems and corresponding software engineering solutions
Data science problem Software engineering solution

Messy (spaghetti) code scattered across Jupyter Notebook files

Robust code-writing and object-oriented programming

Difficult to collaborate on the same codebase

Source control

Training models on data that doesn’t fit into memory

Scaling techniques, such as chunking or distributed parallelization

Turning a model from a prototype into one actively used by others

Putting the model into production

Hard to know whether your code works for a variety of inputs or edge cases

Testing principles

Repeatedly retrieving data from a database or new source files

Scheduling data extraction code to run automatically

Compare familiar data science workflow problems with the engineering techniques that address them, from messy notebooks to production models and scheduled extraction. section 1.1

How Software Engineering for Data Scientists covers it

How deep it goes
Defines the discipline and maps common data workflow problems to techniques, while deferring detailed treatment of those techniques.
Examples
Uses scattered Jupyter notebooks, churn prediction, failed web requests, and zero-row training queries as practical cases.
What you can do after
Evaluate/understand how engineering practices address collaboration, scale, testing, recurring workflows, and production model use.
How it connects
Connects source control, object-oriented programming, testing, scaling, scheduling, model deployment, and exception handling.

Also explained in

Compare how 2 books explain software engineering →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.