concept datum discovery in category spark

appears as: data discovery, data discovery
Spark in Action, Second Edition

This is an excerpt from Manning's book Spark in Action, Second Edition.

As with every data transformation project, you start by looking at the data and its structure (also called data discovery ). After that, you will build the mapping, write the transformation code, run the code, and finally analyze the result.

Figure 12.11 The last step of the data transformation includes the review of the transformed data to ensure it is matching your goal. You may have to go back to the drawing board if the transformation does not produce the expected results. Any good process is iterative by nature, so you will certainly have to go back to data discovery.