concept lambda function in category spark

This is an excerpt from Manning's book Spark in Action, Second Edition. Login to get full access to this book.

Java 8 introduced lambda functions, which can exist without belonging to a class and can be passed around as parameters, and are executed on demand. You will discover how lambda functions can help you (or not) to write transformation code.

You may be familiar with lambda functions in Java. If you are not, Java 8 introduced a new type of function, which can be created without belonging to a class. Lambda functions can be passed around as a parameter and are executed on demand. It’s Java’s first step toward functional programming. The notation for a lambda function is <variable> -> <function> .

In the source code repository, net.jgp.books.spark.ch05.lab900_simple_lambda.SimpleLambdaApp contains an example of lambda functions using a list. It will iterate twice over a list of French first names and build their composed form, such as the following:

Georges and Jean-Georges are different French first names!
Claude and Jean-Claude are different French first names!
...
Louis and Jean-Louis are different French first names!
-----
Georges and Jean-Georges are different French first names!
...
Luc and Jean-Luc are different French first names!
 Louis and Jean-Louis are different French first names!

copy

The first iteration is done in one line, and the second iteration is done in several lines of code, showing the block syntax in lambda functions. The following listing illustrates the process.

Listing 5.4 A basic lambda function

package net.jgp.books.spark.ch05.lab900_simple_lambda;
 
import java.util.ArrayList;
import java.util.List;
 
public class SimpleLambdaApp {
 
     public static void main(String[] args ) {
      List<String> frenchFirstNameList = new ArrayList<>();
       frenchFirstNameList .add( "Georges" ); #1
       frenchFirstNameList .add( "Claude" ); #1
... #1
       frenchFirstNameList .add( "Luc" ); #1
       frenchFirstNameList .add( "Louis" ); #1
 
       frenchFirstNameList .forEach( #2
             name -> System. out .println( name + " and Jean-" + name
                + " are different French first names!" )); #3
 
      System. out .println( "-----" );
 
       frenchFirstNameList .forEach( #2
             name -> { #4
              String message = name + " and Jean-" ;
               message += name ;
               message += " are different French first names!" ;
              System. out .println( message );
            });
    }
 }

copy

Lambda functions allow you to write more-compact code and avoid repeating some of the boring statements; you did not need a loop in listing 5.4. However, readability might suffer. Nevertheless, whether you like or hate lambda functions, you will see more and more of them in the code you will deal with, both in this book and in your professional life. In the next section, you will rewrite the code to approximate π by using a couple of lambda functions.

Figure 5.5 Computing π by using lambda functions in Java: the code is more compact, but the process is the same as the class-based process in figure 5.3.

Listing 5.5 Computing π by using lambda functions

package net.jgp.books.spark.ch05.lab101_pi_compute_lambda;
...
public class PiComputeLambdaApp implements Serializable {
...
     private void start( int slices ) {
...
       long t2 = System.currentTimeMillis();
      System. out .println( "Initial dataframe built in " + ( t2 - t1 ) + " ms" );
      Dataset<Integer> dotsDs = incrementalDf
            .map((MapFunction<Row, Integer>) status -> { #1
               double x = Math.random() * 2 - 1;
               double y = Math.random() * 2 - 1;
               counter ++;
               if ( counter % 100000 == 0) {
                System. out .println( "" + counter + " darts thrown so far" );
              }
               return ( x * x + y * y <= 1) ? 1 : 0;
            }, Encoders.INT());
       long t3 = System.currentTimeMillis();
      System. out .println( "Throwing darts done in " + ( t3 - t2 ) + " ms" );
       int dartsInCircle = #2
             dotsDs .reduce((ReduceFunction<Integer>) ( x , y ) -> x + y );
       long t4 = System.currentTimeMillis();
      System. out .println( "Analyzing result in " + ( t4 - t3 ) + " ms" );
 ...

copy

The lambda functions are doing the same operations as the classes in listing 5.3. The first lambda throws the darts randomly, doing the mapping operation. The second lambda function performs a reduce operation by adding the results together.

The source code is definitely more compact (I also kept only the major differences). However, the code may be a little more difficult to read for a software engineer whose Java skills do not include lambda functions.