concept TDF in category .net

appears as: TDF, TDF
Concurrency in .NET

This is an excerpt from Manning's book Concurrency in .NET.

Compositionality is the main strength of TPL Dataflow, because its set of independent containers, known as blocks, is designed to be combined. These blocks can be a chain of different tasks to construct a parallel workflow, and are easily swapped, reordered, reused, or even removed. TDF emphasizes a component’s architectural approach to ease the restructure of the design. These dataflow components are useful when you have multiple operations that must communicate with one another asynchronously or when you want to process data as it becomes available, as shown in figure 12.2.

Figure 12.2 TDF embraces the concepts of reusable components. In this figure, each step of the workflow acts as reusable components. TDF brings a few core primitives that allow you to express computations based on Dataflow graphs.

c12-02.png

The Producer/Consumer pattern is one of the most widely used patterns in parallel programming. Developers use it to isolate work to be executed from the processing of that work. In a typical Producer/Consumer pattern, at least two separated threads run concurrently: one produces and pushes the data to process into a queue, and the other verifies the presence of the new incoming piece of data and processes it. The queue that holds the tasks is shared among these threads, which requires care for accessing tasks safely. TDF is a great tool for implementing this pattern, because it has intrinsic support for multiple readers and multiple writers concurrently, and it encourages a pipeline pattern of programming with producers sending messages to decoupled consumers (figure 12.6).

Figure 12.6 Multiple-producers/one-consumer pattern using the TDF BufferBlock, which can manage and throttle the pressure of multiple producers

c12-06.png

In the case of a multiple-Producer/single-Consumer pattern, it’s important to enforce a restriction between the number of items generated and the number of items consumed. This constraint aims to balance the work between the producers when the consumer cannot handle the load. This technique is called throttling. Throttling protects the program from running out of memory if the producers are faster than the consumer. Fortunately, TDF has built-in support for throttling, which is achieved by setting the maximum size of the buffer through the property BoundedCapacity, part of the DataFlowBlockOptions. In Listing 12.4, this property ensures that there will never be more than 10 items in the BufferBlock queue. Also, in combination with enforcing the limit of the buffer size, it’s important to use the function SendAsync, which waits without blocking for the buffer to have available space to place a new item.

Listing 12.6 Agent based on TDF in action
List<string> urls = new List<string> {  
                @"http://www.google.com",
                @"http://www.microsoft.com",
                @"http://www.bing.com",
                @"http://www.google.com"
            };
var agentStateful = Agent.Start(ImmutableDictionary<string,string>.Empty, 
 async (ImmutableDictionary<string,string> state, string url) => {   #1  
       if (!state.TryGetValue(url, out string content))
       using (var webClient = new WebClient()){
         content = await webClient.DownloadStringTaskAsync(url);  
         await File.WriteAllTextAsync(createFileNameFromUrl(url), content);
 return state.Add(url, content);   #2  
      }
 return state;   #2  
   });
urls.ForEach(url => agentStateful.Post(url));

#1   Uses an asynchronous anonymous function to construct the agent. This function performs the current state and input message received.
#2   The function, which acts as the behavior of the agent, returns the updated state to keep track of any changes available to the next message processing.


!@%STYLE%@!
{"css":"{\"css\": \"font-weight: bold;\"}","target":"[[{\"line\":6,\"ch\":20},{\"line\":6,\"ch\":31}],[{\"line\":7,\"ch\":1},{\"line\":7,\"ch\":6}],[{\"line\":18,\"ch\":12},{\"line\":18,\"ch\":17}],[{\"line\":6,\"ch\":32},{\"line\":6,\"ch\":51}],[{\"line\":7,\"ch\":8},{\"line\":7,\"ch\":27}],[{\"line\":10,\"ch\":19},{\"line\":10,\"ch\":24}],[{\"line\":11,\"ch\":9},{\"line\":11,\"ch\":14}],[{\"line\":10,\"ch\":19},{\"line\":10,\"ch\":24}],[{\"line\":11,\"ch\":9},{\"line\":11,\"ch\":14}],[{\"line\":16,\"ch\":34},{\"line\":16,\"ch\":38}]]"}
!@%STYLE%@!