8 Architecting serverless parallel computing

 

This chapter covers

  • Principles of MapReduce
  • Serverless solution with Step Functions and EFS

There’s a secret about AWS Lambda that we like to tell people: it’s a supercomputer that can perform faster than the largest EC2 instance. The trick is to think about Lambda in terms of parallel computation. If you can divide your problem into hundreds or thousands of smaller problems and solve them in parallel, you will get to a result faster than if you try to solve the same problem by moving through it sequentially.

Parallel computing is an important topic in computer science and is often talked about in the undergraduate computer science curriculum. Interestingly, Lambda, by its very nature, predisposes us to think and apply concepts from parallel computing. Services like Step Functions and DynamoDB make it easier to build parallel applications.

In this chapter, we’ll illustrate how to build a serverless video transcoder in Lambda that outperforms bigger and more expensive EC2 servers.

8.1 Introduction to MapReduce

8.1.1 How to transcode a video

8.1.2 Architecture overview

8.2 Architecture deep dive

8.2.1 Maintaining state

8.2.2 Step Functions

8.3 An alternative architecture

Summary