I am creating a new ASP MVC order application in the Amazon (AWS) cloud with the persistence layer at my local datacenter. I will be using the CQRS pattern. The goal of the project is high availability using Queue(s) to store and forward writes (commands/events) that can be picked up and handled asynchronously at my local datacenter. Then, ff the WAN or my local datacenter fails, my cloud MVC app can still take orders and just queue them up until processing can resume.
My first thought was to use AWS SQS for the queuing and create my own queue consumer/dispatcher/handler in my own c# application to process the incoming messages/events.
MVC (# Amazon) --> Event/POCO --> SQS --> QueueReader (# my datacenter) --> DB
Then I found NServiceBus. NSB seems to handle lots of details very nicely: message handling, retries, error handling, etc. I hate to reinvent the wheel, and NServiceBus seems like a full featured and mature product that would be perfect for me.
But on further research, it does NOT look like NServiceBus is really meant to be used over the WAN in physically separated environments (Cloud to my Datacenter). Google and SO don't really paint a good picture of using NServiceBus across the WAN like I need.
Can I do this?
MVC (# Amazon) --> Event/POCO --> NServiceBus over WAN --> NServiceBus Handler(s) --> DB
How can I use NServiceBus across the WAN? Or is there a better solution to handle queuing and message handling between Amazon an my local datacenter?
Using SQS as a transport for NServiceBus is an option, however you have to be aware of the trade offs as described here. This has been done with Azure queue storage, though I'm not aware of any great SQS implementations.
Another option is to create a VPN between your datacenter and an AWS VPC. This would allow direct MSMQ communication between AWS servers and your data center, provided you open the appropriate ports in the corresponding security group. There are some caveats with this approach. First, is regarding endpoint names. NServiceBus version 2.6 and below uses Environment.MachineName as the name of the endpoint, for which you would have to setup a proper DNS. I believe later versions use the machine's IP address. Perhaps a more important caveat is that a VPN makes your systems more coupled.
Yet another way, is to use the NServiceBus notion of a gateway. This however should be a logical business decision. A gateway is very similar to the regular transport but is usually has a different business context behind it.
NServiceBus includes a Gateway component that handles bridging physically separated data centers.
http://docs.particular.net/nservicebus/gateway/
It basically moves the messaging to an HTTP channel and handles the retry logic and deduplication issues that you'd normally have with a web service.
If you download the full NServiceBus package (not just include it via NuGet) then you will see a folder full of samples and one of those covers usage of the Gateway, and that is a great way to get started.
Related
I would like to find a solution to create a pub/sub medium for 2 microservices to talk to each other,
I am aware i can use some third parties E.g Redis, RabbitMQ
Implementing event-based communication between microservices (integration events)
The challenge lies on the client is unable to allow install any third parties tool due to security reason.
The messageQueue server in Windows won't be allowed to use too.
I can only use the applications that is only existed in the server.
Therefore i am asking if there is anyway that i can create one simple app using windows service.
It is a one-to-many relationship. I have one service that will be dealing with data, once if there is any update, it will publish to those services that is subsribed to it.
It seems my problem could be similar with
.NET Scalable Pub/Sub service implementation
WCF Pub/Sub with subscriber caching(link is dead on the WCF pub-sub)
but i dont see any critical solutions.
I was thinking to use data notifications that MSSQL offers as last alternatives, but seems like it could cause a bottle neck when the applications get scale up.
The internet is so much flooded with articles using third parties tool.
Thanks
Check out Rebus library, that allows using different transport methods to send end receive messages in just a line of code (so in the future you can change it without effort).
You could use SQL Server or try to develop your own transport method
I have an ASP.NET MVC application hosted in Azure.
This application is complemented with a desktop application that also has WCF services for communicating with III party interfaces. WCF are hosted locally.
There are thousands of clients using the desktop application at different geographical locations.
Till now, every desktop application used to talk to web app using api with the help of WCF.
This was limited to on demand from the desktop application.
Whenever desktop application feels the need to talk to web app, it used the way of web api from WCF.
Now, what I want is:-
To access the different desktop applications(typically called sites), from azure depending upon the need.
This is required on account of an online ordering system that is through web app/mobile app.
I do not want to keep polling from desktop application to know about if any new order is there for this site.
I feel it would be better if I can play from other side.
Also, keeping in mind that IP of sites will not be fixed. There may be issue with firewall. NAT may translate resource identifier differently.
Can service bus in azure may be of any help, but what confuses me is that every desktop application is having its own WCF service and order should reach the respective site only.
Any type of ideas on this would be appreciated.
According to your description, Service Bus messaging is a perfect way to achieve this.
More information about Service Bus Messaging, we can refer to: Service Bus queues, topics, and subscriptions
In addition, We can also use RabbitMQ or ZeroMQ which is similar with Service Bus Messaging because both of them are free. You can choose an best way to realize your requirements.
About differences between ZeroMQ and RabbitMQ:
ZeroMQ has better performance, but it is built in the case of allowing message data loss to apply to high throughput / low latency applications. Unlike ZeroMQ, RabbitMQ fully implements the AMQP protocol, which is similar to mailbox services, supporting message persistence, transaction, congestion control, load balancing and so on, making RabbitMQ have a more extensive application scenario.
Function RabbitMQ ZeroMQ
Message persistence Support Not Support
Transaction Support Not Support
performance Low High
stability High Low
Support for AMQP protocol Support Not Support
Application scenario Data loss is not allowed High throughput
More information about RabbitMQ and ZeroMQ, we can refer to:
RabbitMQ
ZeroMQ
If you are able to modify the desktop applications, implementing a websockets connection with SignalR might be worth a look. The desktop applications sign up with a SignalR hub you provide.
You can then push data to the clients from, for example an ASP.NET MVC app. It works very reliable and handles lots of connections well. It is typically used for realtime web communication but might be useful in your case, too.
The downside is probably, that the desktop app needs to initially sign up to a hub to receive push messages.
We currently have a solution, using web services (implement via WCF) where client software periodically call a service to retrieve a list of new items waiting for them, then for each item, calls a separate service to do the actual download. This is the typical message polling scenario, and is relatively simple in implementation and trouble free. It does not give a near real time messaging solution however. For real time messaging you'd want more of a push notification architecture.
Because of security concerns we do not want clients to expose interfaces that our system can call because then each client needs to secure that exposed interface among other concerns.
Id like to explore the option of the client still initiating the connection to the server as we do today, but instead of polling, we maintain the connection and once the client has completed the connection, the server is now able to push messages to the client.
Our clients base may be implementing their solution on a variety of platforms, from Linux to Windows, .Net to PHP, etc. Additionally, clients vendors have a varying degree of technical capabilities so complexity of the solution is a factor here. Can SOAP or REST services be used in this type of architecture? Are there other technologies I should be looking at? Our server side part of the solution would need to be a .net one.
I have a straightforward, existing ASP.NET MVC web solution. The server-based stuff writes information to a database. I am now going to integrate/synchronize this system with a number of other 3rd-party systems. I want to separate the integration processing from the existing core processing, leaving the existing system as untouched as possible.
My plan is as follows:
whenever a database write occurs on the core system server I will write a message to an MSMQ Queue.
an entirely separate server-based windows service will poll the MSMQ, look at the message and will write messages to one or more 'outbound' sync MSMQ queues.
other windows services will monitor the 'outbound' sync queues, and will talk to the 3rd-party systems as necessary, managing the outbound synchronization.
I have a couple of questions:
Should I have a single windows service doing all this, or should I have several services, one central 'routing' one and one for each 3rd-party system?
Should I use WCF for any of this. Does that buy me anything, given that the 'trigger' for writing to the initial queue is already 'happening' on a server-based process?
Thanks very much.
To answer your questions:
Should I have a single windows service doing all this
Definitely not. What if you want to scale out the routing service, or relocate it?
Should I use WCF
If you have your heart set on msmq then the only advantage WCF gives you is it provides a convenient, proven way to design and host your service endpoints, and an alternative to mucking around in System.Messaging. I would say at this stage it doesn't matter that much.
Does that buy me anything
Not sure what you mean, but as Wiktor says in his post, you could chose not to use vanilla .Net or WCF and choose a service bus type framework such as masstransit or nservicebus.
The benefit here is it abstracts you away from the messaging sub-system so you could in theory move away from msmq in the future to rabbitmq or azure queues.
First, a separate windows service is always safer than any attempt to integrate this with your asp.net runtime.
Second, do not write anything by yourself. Use
http://code.google.com/p/masstransit/
It is straightforward and does everything you need. Reference the library from their nuget package, read some tutorials and you will love it.
I need to build a system that is similar to a pub/sub system. It is composed of multiple sub-systems or services running in separate executables or as a Windows Services.
The sub-systems are:
The pub/sub service
A pub/sub service managing communications between the internal sub-systems and the users.
A user can have multiple channels open (A web page connected to a SignalR service, a mobile device connected to a duplex WCF service, etc.).
The service should manage all the channels of an user and be able to send information to them on demand based on topics or specific users.
The service must support multiple transports like SignalR, WCF, or others ...
Worker services
A worker that runs as a Windows Service and sends information to the users using the pub/sub service.
The SignalR and WCF host
The SignalR service and WCF service will be hosted on IIS
My questions are
As the sub-systems run in separate processes, how do I communicate between the pub/sub service and the other sub-systems like (the workers and IIS). The communication must be really fast. Do I use named-pipes, is it fast enough ?
An example; The worker tells the pub/sub system to send a message to a user, the pub/sub systems checks the channels opened for the user (let's say a SignalR channel), then in turn it must notify the SignalR service running in IIS to send the message to the user's browser.
Do you know of implementations of similar systems ?
Observations
I cannot use third-party service-bus services (Azure ..). And even with that .. I can't see a solutions to the problems above.
The service must be very scalable and high-demand proof.
If the question is how to bridge SignalR with other transports there are several solutions.
On a single server you could just connect them up with the Reactive framework's own pubsub mechanism which is neatly encapsulated in the Subject class.
If you need to scale out to multiple servers you'll want to use an existing service bus or perhaps roll your own simple one using SQL server and a SqlDependency.
You could also use SignalR as a client on one server communicating with the other servers to copy messages between them.
I recommend you look into some of the existing Service Bus technologies for .NET.
There is an article which clearly explains the possible mechanism of how to incorporate a pub/sub design pattern in your .NET application. The answer lies in using a .NET In-Memory distributed cache and use its clustering capabilities as a publish subscribe medium. Since it's clustered therefore you won't have to worry about down-times as well.
Basically you'll be using the Application Initiated Custom Events
Register your events
public void OnApplicationEvent(object notifId, object data)
{
...
}
_cache.CustomEvent += new CustomEventCallback(this.OnApplicationEvent);
And fire those events whenever you need to
_cache.RaiseCustomEvent("NotificationID", DateTime.Now);
Pub/Sub design pattern in a .NET distributed cache
Full disclosure: I work for Alachisoft