Real time data processing using Asp.net Web Api, TPL Dataflow, SignalR and AngularJS

TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency.  https://msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx

we want to just write the code, and the way we structure it results in no synchronization issues.  So  we don’t have to think about synchronization. In this world each object has its own private thread of execution, and only ever manipulates its own internal state.

Instead of one single thread executing through many objects by calling object methods, objects send asynchronous messages to each other.

If the object is busy processing a previous message, the message is queued. When the object is no longer busy it then processes the next message.
Fundamentally, if each object only has one thread of execution, then updating its own internal state is perfectly safe.

Blocks are essentially a message source, target, or both. In addition to receiving and sending messages, a block represents an element of concurrency for processing the messages it receives.

Multiple blocks are linked together to produce networks of blocks. Messages are then posted asynchronously into the network for processing.

Consider the following use case.  Several orders  are extracted from a source, The data contained in each order need to be transformed into a orderDetail object ready to be sent to the  Sales web service as a SalesOrderDetail object.

The  process could be structured into a series of blocks, where each block is responsible for doing some part of the overall processing.

We want to improve system performance but also we want to monitor the process in real time.

To achieve this goal, we suggest to build a distributed architecture consisting of a REST web server ( ASP.NET WEB API,  SIGNALR),  a WEB client (ASP.NET MVC and Angular JS ) and a web service that processes files ( WCF or Other).

To follow this tutorial, you must undertand ASP.NET WEB API , SIGNALR and TPL Dataflow.

TECHNOLOGY ARCHITECTURE

• HUB Server  : ASP.NET WEB API and SIGNALR
• Monitoring Client   : ASP.NET MVC and AngularJS
• Processing Server : TPL DataFlow

1. HUB SERVER.

To Build our Hub Server, we will use ASP.NET Web API because clients must connect to the hub by uploading json data

For more information about SIGNALR please take a look at http://www.asp.net/signalr

So, let Create an ASP.NET WEB API Project  as follow :

• install Microsoft ASP.NET SignalR

Lets define a HubName dashboads , to get notified, clients (web browsers, mobiles, desktop app, ……) must connect to the defined Hub

Lets define  a ApiController (DashboardController)  derived from SignalRBase and 2 action PostProcessor  and PostSalesOrderDetailEntity

SignalRBase implement IHub and allow us to access our Hub inside DashboardController.

We use SignalR  as it allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available and supports Web Sockets.

• Hub.Clients.All.LoadBalance(item) ==>  Send message to all connected clients (Notify all connected client to invoke LoadBalance function of the Hub)
• Hub.Clients.Client(id).LoadBalance(item) ==> Send Message to a specific client.
• Hub.Clients.Group(groupId).LoadBalance(item) ==> Send Message to all clients connected to a specific group.

When Server is invoked, Hub.Clients.All.LoadBalance(item) ( where item is Processor), data is pushed to Hub and be available for clients as follow

2.PROCESSING SERVER

I will implement the processing server using TPL Dataflow

There are several blocks like ActionBlock<TInput>, ActionBlock<TInput>,  BroadcastBlock<T>, BufferBlock<T> etc….

https://msdn.microsoft.com/en-us/library/system.threading.tasks.dataflow(v=vs.110).aspx

So create a class library project and add the following :

• _blockConfiguration provide options to configure Dataflow network.  Here we configure our block to be parallelized using the number of processor available in the current machine (MaxDegreeOfParallelism = Environment.ProcessorCount)
• _bufferBlock has the responsibility to extract orders from the source  as they arrive
• _receptorBlockOne , _receptorBlockTwo and _receptorBlockThree has the responsibility to load balance fetched orders and calculate OrderDetails from Orders
• _transformBlockCalculateSalesOrderDetailhas responsability to calculate SalesOrderDetails from OrderDetails
• _loggerBlock  has responsability to log informations

For better performance , we want to load balance our process. So our next step is to create 3 load balanced receptors,  if _receptorBlockOne is busy, _receptorBlockTwo or _receptorBlockThee will process the item,… ReceptorBlockOne, ReceptorBlockTwo and ReceptorBlockThree are blocks. So, if a message is refused by one block, the next linked block will be offered the message. If all blocks refuse the message, the first block to become available to process the message will do so. To achieve this goal, we have to make a block non-greedy, simply set the queue length to 1.

Lets define a flowComplete variable so as to propagate completion and faulting notification  automatically.

Now, its time to build our network by linking our Dataflow blocks

The following example shows the complete code for this walkthrough.

3.MONITORING CLIENTS

Lets create a asp.net project and add the following

Reference the following packages

Define a common.services module

Define an app function

Define a dashboardSvc service

Define a dashboardCtrl controller

Define an Index view

The result in action

 

I hope this post will help you.

Sample code is available here   DataflowSignalrAngularDemo

Regards