14 Getting to production

 

This chapter covers

  • Addressing concerns, risks, and problems when taking your data pipeline to production
  • Employing strategies for building a production-ready application

Our data-wrangling journey together is coming to a close, although it’s at this stage where your real work is about to begin. Although it may seem that exploratory coding, development, and testing are a pile of work, you ain’t seen nothing yet. Building and testing your data pipeline are often only small parts of the project lifecycle.

An ugly truth of software development is that most developers will spend the majority of their time maintaining existing applications after they’ve entered production. Getting to production is a big deal: we need to deploy our application, monitor it, and understand its behavior.

We then need to update our app so that we can deploy bug fixes or upgrade its feature set. At the same time, we need a solid testing regime to ensure that it doesn’t explode in a smoldering mess. These are several of the things we must deal with after our application enters the production phase.

14.1 Production concerns

14.2 Taking our early warning system to production

14.3 Deployment

14.4 Monitoring

14.5 Reliability

14.5.1 System longevity

14.5.2 Practice defensive programming

14.5.3 Data protection

14.5.4 Testing and automation

14.5.5 Handling unexpected errors

14.5.6 Designing for process restart

14.5.7 Dealing with an ever-growing database

14.6 Security

14.6.1 Authentication and authorization

14.6.2 Privacy and confidentiality

14.6.3 Secret configuration

14.7 Scaling

14.7.1 Measurement before optimization

14.7.2 Vertical scaling

14.7.3 Horizontal scaling

Summary