8 Architecting serverless parallel computing
This chapter covers
- Principles of MapReduce
- Architecting a Serverless solution with Step Functions and EFS
There’s a secret about AWS Lambda that I like to tell people: It’s a supercomputer that can perform faster than the largest EC2 instance. The trick is to think about Lambda in terms of parallel computation. If you can divide your problem into hundreds or thousands of smaller problems, and solve them in parallel, you will get to a result faster than if you try to solve the same problem by moving through it sequentially. Parallel computing is an important topic in computer science and is often talked about in the undergraduate computer science curriculum. Interestingly, Lambda, by its very nature, predisposes us to think and apply concepts from parallel computing. Services like Step Functions and DynamoDB make it easier to build parallel applications. In this chapter we’ll illustrate how to build a Serverless video transcoder in Lambda that will outperform bigger and more-expensive EC2 servers. We’ll look at the implementation, cost, performance, and discuss the pros and cons of the approach for decomposing and solving large problems using Lambda.