5. Building a simple app for deployment

 

This chapter covers

  • Building a simple application that will not require data ingestion
  • Using Java lambdas with Spark
  • Building an application with or without lambdas
  • Interacting with Spark in local mode, cluster mode, and interactively
  • Calculating an approximation of π by using Spark

In the previous chapters, you discovered what Apache Spark is and how to build simple applications, and, hopefully, understood key concepts including the dataframe and laziness. Chapters 5 and 6 are linked: you will build an application in this chapter and deploy it in chapter 6.

In this chapter, you will start from scratch by building an application. You built applications previously in this book, but they always needed to ingest data at the very beginning of the process. Your lab will generate data within and by Spark, avoiding the need to ingest data. Ingesting data in a cluster is a bit more complex than creating a self-generated dataset. The goal of this application is to approximate a value of π (pi).

You will then learn about the three ways to interact with Spark:

  • Local mode, which you are already familiar with through the examples in the previous chapters
  • Cluster mode
  • Interactive mode
Lab

Examples from this chapter are available in GitHub at https://github .com/jgperrin/net.jgp.books.spark.ch05.

5.1 An ingestionless example

5.1.1 Calculating π

5.1.2 The code to approximate π

5.1.3 What are lambda functions in Java?

5.1.4 Approximating π by using lambda functions

5.2 Interacting with Spark

5.2.1 Local mode

5.2.2 Cluster mode

5.2.3 Interactive mode in Scala and Python

Summary