Chapter 10. Writing a YARN application
This chapter covers
- Understanding key capabilities of a YARN application
- How to write a basic YARN application
- An examination of YARN frameworks and applications
Looking at the source code for any reasonably sized YARN application typically results in words like “complex” and “low-level” being thrown around. At its core, writing a YARN application isn’t that complex, as you’ll discover in this chapter. The complexity with YARN is typically introduced once you need to build more advanced features into your application, such as supporting secure Hadoop clusters or handling failure scenarios, which are complicated in distributed systems regardless of the framework. That being said, there are emerging frameworks that abstract away the YARN APIs and provide common features that you’ll require.
In this chapter, you’ll write a simple YARN application that will run a Linux command on a node in the cluster. Once you’ve run your application, you’ll be introduced to some of the more advanced features that you may need in your YARN application. Finally, this chapter looks at some of the open source YARN abstractions and examine their features.
Before we get started, let’s ease into YARN programming by looking at the building blocks of a YARN application.