concept pipeline in category deep learning

This is an excerpt from Manning's book Deep Learning Design Patterns.
Typically, the computer system works as a pipeline process, where cumulative information, transformations and state transitions are the inputs to a model at various points in the pipeline. The output (prediction) of each model would be used to perform the next information transformation and/or decide the next state transition. Typically, each model is trained and deployed independently, usually as a microservice and a backend application drives the pipeline process.

This is an excerpt from Manning's book Deep Learning with Structured Data.
As you can see, it takes several steps (including initial setup of the Google geocoding API) to finally get latitude and longitude values added to the dataset. Understanding how to manipulate latitude and longitude values in the context of a Python program is a useful skill for many common business problems that have a spatial dimension, and latitude and longitude values make it possible to create visualizations (such as the one in Figure 4.5) to identify hotspots for delays. As it turns out, the deep learning model described in chapters 5 and 6 does not use location data, either the free-form text or latitude and longitude. The free-form text locations cannot be used in the refactored dataset, and the process for converting free-form text locations into latitude and longitude is complicated and difficult to integrate into a pipeline. However, Chapter 9 includes a section on how you could augment the model to incorporate location data to identify which subsection of a streetcar route a delay occurs in.
Figure 8.17 shows how new data points to be scored by the trained model get entered by the user, interpreted by Rasa, and then processed by the pipelines so that they are in the correct format for the trained Keras model to make a prediction on.
Figure 8.17 new data points: the complete flow from user input to what the model expects
![]()
The pipelines that are applied to new data points entered by the user prior to the trained model being applied to the new data points are identical to the pipelines that are applied to the dataset prior to training the model, as shown in Figure 8.18:
Figure 8.18 training data flows through the same pipeline as data that is being scored
![]()
In this section we have introduced the idea of pipelines and shown at a high level where they fit in the training and deployment processes. In the next section we will dig into the code that defines the pipelines.
sc_delay_pipeline = Pipeline([('fill_empty',fe),('encode_categorical',ec)]) sc_delay_pipeline_keras_prep = Pipeline([('prep_for_keras',pk)]) Next, we set the parameters for the class instantiations in the pipeline: sc_delay_pipeline.set_params(fill_empty__collist = collist, fill_empty__continuouscols = continuouscols, fill_empty__textcols = textcols,encode_categorical__col_list = collist) sc_delay_pipeline_keras_prep.set_params(prep_for_keras__collist = collist, prep_for_keras__continuouscols = continuouscols, prep_for_keras__textcols = textcols)