12 Live data
This chapter covers
- Working with a real-time data feed
- Receiving data through HTTP POST and sockets
- Decoupling modules in your server with an event-based architecture
- Triggering SMS alerts and generating automated reports
- Sending new data to a live chart through socket.io
In this chapter we bring together multiple aspects of data wrangling that we’ve already learned and combine them into a real-time data pipeline. We’re going to build something that’s almost a real production system. It’s a data pipeline that will do all the usual things: acquire and store data (chapter 3), clean and transform the data (chapter 6), and, in addition, perform on-the-fly data analysis (chapter 9).
Output from the system will take several forms. The most exciting will be a browser-based visualization, based on our work from chapter 10, but with live data feeding in and updating as we watch. It will automatically generate a daily report (using techniques from chapter 11) that’s emailed to interested parties. It will also issue SMS text message alerts about unusual data points arriving in the system.
To be sure, the system we’ll build now will be something of a toy project, but besides that, it will demonstrate many of the features you’d want to see in a real system of this nature, and on a small scale, it could work in a real production environment.