12 Parallel workflow and agent programming with TPL Dataflow
This chapter covers
- Using TPL Dataflow blocks
- Constructing a highly concurrent workflow
- Implementing a sophisticated Producer/Consumer pattern
- Integrating Reactive Extensions with TPL Dataflow
Today’s global market requires that businesses and industries be agile enough to respond to a constant flow of changing data. These workflows are frequently large, and sometimes infinite or unknown in size. Often, the data requires complex processing, leading to high throughput demands and potentially immense computational loads. To cope with these requirements, the key is to use parallelism to exploit system resources and multiple cores.
But today’s .NET Framework’s concurrent programming models weren’t designed with dataflow in mind. When designing a reactive application, it’s fundamental to build and treat the system components as units of work. These units react to messages, which are propagated by other components in the chain of processing. These reactive models emphasize a push-based model for applications to work, rather than a pull-based model (see chapter 6). This push-based strategy ensures that the individual components are easy to test and link, and, most importantly, easy to understand.
This new focus on push-based constructions is changing how programmers design applications. A single task can quickly grow complex, and even simple-looking requirements can lead to complicated code.