concept local mode in category spark

This is an excerpt from Manning's book Spark in Action, Second Edition. Login to get full access to this book.

Local mode is not a cluster, but it’s much easier

In order for you to be able to run the example in this chapter without setting up a full cluster, I specified local as the value for the master because you are running Spark in local mode. If you had a cluster, you would give the address of this cluster. You will learn more about clusters in chapters 5 and 6.

For the sake of building your mental model, you will assume you have a cluster instead of the local mode.

In this section, you’ll walk through the code, which you will use throughout this chapter in various examples. You will first run the code in local mode. You will then modify this version of the code to use Java lambda functions.

Figure 5.6 In local mode, all the components are running on a single node.

Behind the scenes, Spark starts the required machinery to stand up a master and worker. In local mode, you will not have to submit a JAR file; Spark will set the right class path, so you do not even have to deal with JAR hell .

To start Spark in local mode, as you have done in previous examples, you simply start a session by specifying the master node to be local:

.master( "local" )

copy

Here is the full code to get or create a session in local mode: