Real time data processing using Asp.net Web Api, TPL Dataflow, SignalR and AngularJS
TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency. https://msdn.microsoft.com/en-us/library/hh228603(v=vs.110).aspx
we want to just write the code, and the way we structure it results in no synchronization issues. So we don’t have to think about synchronization. In this world each object has its own private thread of execution, and only ever manipulates its own internal state.
Instead of one single thread executing through many objects by calling object methods, objects send asynchronous messages to each other.
If the object is busy processing a previous message, the message is queued. When the object is no longer busy it then processes the next message.
Fundamentally, if each object only has one thread of execution, then updating its own internal state is perfectly safe.
Blocks are essentially a message source, target, or both. In addition to receiving and sending messages, a block represents an element of concurrency for processing the messages it receives.
Multiple blocks are linked together to produce networks of blocks. Messages are then posted asynchronously into the network for processing.
Consider the following use case. Several orders are extracted from a source, The data contained in each order need to be transformed into a orderDetail object ready to be sent to the Sales web service as a SalesOrderDetail object.
The process could be structured into a series of blocks, where each block is responsible for doing some part of the overall processing.
We want to improve system performance but also we want to monitor the process in real time.
To achieve this goal, we suggest to build a distributed architecture consisting of a REST web server ( ASP.NET WEB API, SIGNALR), a WEB client (ASP.NET MVC and Angular JS ) and a web service that processes files ( WCF or Other).
To follow this tutorial, you must undertand ASP.NET WEB API , SIGNALR and TPL Dataflow.
- HUB Server : ASP.NET WEB API and SIGNALR
- Monitoring Client : ASP.NET MVC and AngularJS
- Processing Server : TPL DataFlow
- HUB SERVER.
To Build our Hub Server, we will use ASP.NET Web API because clients must connect to the hub by uploading json data
For more information about SIGNALR please take a look at http://www.asp.net/signalr
So, let Create an ASP.NET WEB API Project as follow :
- install Microsoft ASP.NET SignalR
Lets define a HubName dashboads , to get notified, clients (web browsers, mobiles, desktop app, ……) must connect to the defined Hub
Lets define a ApiController (DashboardController) derived from SignalRBase and 2 action PostProcessor and PostSalesOrderDetailEntity
SignalRBase implement IHub and allow us to access our Hub inside DashboardController.
We use SignalR as it allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available and supports Web Sockets.
- Hub.Clients.All.LoadBalance(item) ==> Send message to all connected clients (Notify all connected client to invoke LoadBalance function of the Hub)
- Hub.Clients.Client(id).LoadBalance(item) ==> Send Message to a specific client.
- Hub.Clients.Group(groupId).LoadBalance(item) ==> Send Message to all clients connected to a specific group.
When Server is invoked, Hub.Clients.All.LoadBalance(item) ( where item is Processor), data is pushed to Hub and be available for clients as follow
I will implement the processing server using TPL Dataflow
There are several blocks like ActionBlock<TInput>, ActionBlock<TInput>, BroadcastBlock<T>, BufferBlock<T> etc….
So create a class library project and add the following :
- _blockConfiguration provide options to configure Dataflow network. Here we configure our block to be parallelized using the number of processor available in the current machine (MaxDegreeOfParallelism = Environment.ProcessorCount)
- _bufferBlock has the responsibility to extract orders from the source as they arrive
- _receptorBlockOne , _receptorBlockTwo and _receptorBlockThree has the responsibility to load balance fetched orders and calculate OrderDetails from Orders
- _transformBlockCalculateSalesOrderDetailhas responsability to calculate SalesOrderDetails from OrderDetails
- _loggerBlock has responsability to log informations
For better performance , we want to load balance our process. So our next step is to create 3 load balanced receptors, if _receptorBlockOne is busy, _receptorBlockTwo or _receptorBlockThee will process the item,… ReceptorBlockOne, ReceptorBlockTwo and ReceptorBlockThree are blocks. So, if a message is refused by one block, the next linked block will be offered the message. If all blocks refuse the message, the first block to become available to process the message will do so. To achieve this goal, we have to make a block non-greedy, simply set the queue length to 1.
Lets define a flowComplete variable so as to propagate completion and faulting notification automatically.
Now, its time to build our network by linking our Dataflow blocks
The following example shows the complete code for this walkthrough.
Lets create a asp.net project and add the following
Reference the following packages
Define a common.services module
Define an app function
Define a dashboardSvc service
Define a dashboardCtrl controller
Define an Index view
The result in action
I hope this post will help you.
Sample code is available here DataflowSignalrAngularDemo