Chapter 8. Railway-oriented processing
This chapter covers
- Handling failure within Unix programs, Java exceptions, and error logging
- Designing for failure inside and across stream processing applications
- Composing failures inside work steps with the Scalaz Validation
- Failing fast across work step boundaries with Scala’s map and flatMap
So far, we have focused on what you might call the happy path within our unified log. On the happy path, events successfully validate against their schemas, inputs are never accidentally null, and Java exceptions are so rare that we don’t mind them crashing our application.
The problem with focusing exclusively on the happy path is that failures do happen. More than this: if you implement a unified log across your department or company, failures will happen extremely frequently, because of the sheer volume of events flowing through, and the complexity of your event stream processing. Linus’s law states, “Given enough eyeballs, all bugs are shallow.”[1] Adapting this, a law of unified log processing might be as follows:
1Linus’s law is further explained at Wikipedia, https://en.wikipedia.org/wiki/Linus%27s_Law.
Given enough events, all bugs are inevitable.