I have a Microservice (Web API), that publish messages to a Topic (Topic A).
Now I have another Microservice (Web API) that should subscribe to this topic and act upon the messages.
My question is simply: How should I do this since my Microservice that should subscribe to the Topic is a WebApi? In my web api, I want in somehow instantly know when a new message is available in the Topic. Should I poll the service bus via an endpoint?
I'm uncertain about the best practices about this.
All examples that I have seen using console applications to subscribe. But that's not my case since I have an web api.
There are different ways of doing this.
1. Using Azure Functions
This way you create two applications. Your standard web api, and separately you create an Azure Function that will handle the messages from the queue. There multiple benefits of this approach, on of them is that you are isolating the code handling the queue, so if you have many messages, it will not affect the performance of your API
2. Using a Singleton service inside your web application
The idea here is that your API application is handling queue messages in the background. This has the advantage that you have only one application doing everything, simpler to maintain for example. It has the disadvantage that a very big inflow of messages will slow down your APIs.
(Note, in the link above look for Consuming messaging from the Queue
Whether is a WebAPI or a console, it is the responsibility of the consumer to communicates and collect records. Being a WebAPI doesn't mean that it should only have public endpoints. Typical WebAPI, might have public endpoint (for external world) or can have private endpoints (for internal communications) or can have a combination of both. The responsibility of private endpoints could be reading data from service data-store, consuming external services via adapter services etc. In your case, upon initialization of your WebAPI, you might want to create a consumer object and start reading data and process as you want. Hope this help.
You can poll in a Web Job or background task. But the built-in way to do this is with an Azure Function triggered from the Topic, or with Azure Event Grid.
For listening in the background you can use IHostedService. Inside Method StartAsync
you can register message processors;
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions)
And on StopAsync you can stop processing messages and close the client.
Related
Domain Driven Design Passing Events to separate Bounded Contexts
A user action in MVC should generate an Event which is passed to a remote (same LAN) Event handler.
What I've tested:
MVC: fire and forget service call (asynchronous) ->
(IIS hosted) WCF which gathers data and populates a message ->
Sent via EasyNetQ/RabbitMQ ServiceBus ->
The event is consumed by a Subscriber (using a DI container initialized from a WCF service endpoint) which handles the event & it's data.
I did some testing to see how it works if the service is called fairly quickly by looping in the MVC side
for (int i = 0; i < 200; i++)
{
...
client.MyServiceMethod(someId, startDate);
...
}
The MessageQueue part is quick, based on the timestamps it is sent to the queue and received by the subscriber within the same second. Looping through the WCF service calls is very slow. It takes many seconds to loop through them. I tried switching from wsHttpBinding to netTcpBinding, and playing with the serviceThrottling in WCF.
WCF isn't compulsory, but it seems like a separate event handling project (on the publisher end) would be beneficial and could be physically located elsewhere from the MVC app (load reduction etc.). Is WCF plausible for a situation like this, or should I try using Windows services or some other self-hosted e.g. console app etc, or potentially using a thread in MVC to generate the event data, or are there better scenarios? What are the best practices in this type of Event handling system? Basically it seems like it would be beneficial to have something generating the Event data since it has to be handled somewhere while not slowing down the UI that the end user is using.
Instead of trying to roll your own infrastructure like this, I think you would do well to employ a tool like NServiceBus (not free) or MassTransit (free). (I would consider this best practice.)
I can't speak for MassTransit, but my experience with NServiceBus has been very good. You only need to specify which messages go to which queue. You can use several different queueing technologies, but I would recommend starting with the default MSMQ implementation. No WCF configuration nightmares necessary. ;)
All of your message handlers will also be automatically wrapped in a distributed transaction so that if a DB interaction fails, the entire message will be rolled back and you'll be able to try the message again in the future.
If I undertood well, your event creation process is "heavy" and you want to avoid to be created in the MVC process. I guess you are sending some information to the WCF service in order to let him prepare the event.
You could think of a 2 consumers scenario avoiding the WCF step:
Your MVC application creates and publish a "light" event with all data required in order to create the "heavy" event (basically with the input data you would pass to WCF)
An EventCreator subscriber consumes this message and prepares the heavy event
Your already existing consumer will then consume the heavy event
EasyNetQ already provides simple functions to publish and consume the message.
Most of the tutorials you find online suggest using TopShelf for hosting your consumers in a console application (debug) or windows service (production). EasyNetQ has an example here: EasyNetQ with TopShelf
If you want to "hide" the EasyNetQ dependecy on your MVC project, you could wrap the EasyNetQ IBus to a custom Bus and use an IoC container in order to inject a specific implementation of your bus. The example provided above uses Castle.Windsor as IoC container
My worker role expose a WCF service and have multiple instances.
I want my client to call this service and make all instances working concurrently.
I'm trying to figure out what is the best way to do this scatter-gather task.
(And I'm trying to avoid service bus and use WCF only)
I can't think of any good way to do this without something like service bus topics. Or using custom functionality that does nearly the same thing. Why are you trying to avoid Service Bus?
There's really no way to make a client-side call to multiple server instances simultaneously, just using Azure's built-in services. Even using Service Bus topics, there's no way to guarantee that multiple subscribers will consume a message at the same time and execute at the same time (even with a message embargo time, you still cannot absolutely guarantee each subscriber will consume + process a message at an exact time).
This will need to be an application-side action. For example: You can queue up your wcf requests. Your queue-reader can then direct-connect to an internal endpoint on each instance, triggering an action to run in parallel. This won't give you exact parallel operation, but it will be pretty close. As another option, you can have several threads available per instance, and you could run the same request on each thread (again, managed by you).
In essence, this is an architectural facet of your app. Azure won't be able to facilitate a parallel-call across instances; you can take advantage of queues, internal services, etc. to accomplish this.
I am unsure of whether this is possible and have conducted a handful of quick searches on the web and I don't think it is but here goes.
I have a web service (old school .asmx) which retrieves a list of items from a database. I want my client app to be notified if there are any items to be processed INSTEAD of the client app every so often making a request to the web service. Is this possible?
I know this is possible via WCF (duplex communication) however due to complications I can't use that approach.
Oops I just realised that I would also require the web service to poll itself or get something to call it asynchronously? Don't think this is a good solution...
Thanks in advance, Onam.
It is either polling (initiated by the client) or by maintaining a session like Exchange, Imap etc does. Classic .asmx communication is initiated by the client. Therefore it provides no events.
You can implement duplex yourself by developing a callback service hosted by your client. Your webservice can then invoke the callback client. However you must find a good solution to register and unregister the clients to the webservice.
Actually, I would not be a good idea to let the database invoke the items on each poll. You could cache the results in the logic layer referenced by the webservice. If you prefer the classic .asmx approach, you can create a bool method that checks whether new data is in the cache that is implemented by the logic. This would reduce the performance inpact, and then it won't be that bad to poll the server.
I need to build a system that is similar to a pub/sub system. It is composed of multiple sub-systems or services running in separate executables or as a Windows Services.
The sub-systems are:
The pub/sub service
A pub/sub service managing communications between the internal sub-systems and the users.
A user can have multiple channels open (A web page connected to a SignalR service, a mobile device connected to a duplex WCF service, etc.).
The service should manage all the channels of an user and be able to send information to them on demand based on topics or specific users.
The service must support multiple transports like SignalR, WCF, or others ...
Worker services
A worker that runs as a Windows Service and sends information to the users using the pub/sub service.
The SignalR and WCF host
The SignalR service and WCF service will be hosted on IIS
My questions are
As the sub-systems run in separate processes, how do I communicate between the pub/sub service and the other sub-systems like (the workers and IIS). The communication must be really fast. Do I use named-pipes, is it fast enough ?
An example; The worker tells the pub/sub system to send a message to a user, the pub/sub systems checks the channels opened for the user (let's say a SignalR channel), then in turn it must notify the SignalR service running in IIS to send the message to the user's browser.
Do you know of implementations of similar systems ?
Observations
I cannot use third-party service-bus services (Azure ..). And even with that .. I can't see a solutions to the problems above.
The service must be very scalable and high-demand proof.
If the question is how to bridge SignalR with other transports there are several solutions.
On a single server you could just connect them up with the Reactive framework's own pubsub mechanism which is neatly encapsulated in the Subject class.
If you need to scale out to multiple servers you'll want to use an existing service bus or perhaps roll your own simple one using SQL server and a SqlDependency.
You could also use SignalR as a client on one server communicating with the other servers to copy messages between them.
I recommend you look into some of the existing Service Bus technologies for .NET.
There is an article which clearly explains the possible mechanism of how to incorporate a pub/sub design pattern in your .NET application. The answer lies in using a .NET In-Memory distributed cache and use its clustering capabilities as a publish subscribe medium. Since it's clustered therefore you won't have to worry about down-times as well.
Basically you'll be using the Application Initiated Custom Events
Register your events
public void OnApplicationEvent(object notifId, object data)
{
...
}
_cache.CustomEvent += new CustomEventCallback(this.OnApplicationEvent);
And fire those events whenever you need to
_cache.RaiseCustomEvent("NotificationID", DateTime.Now);
Pub/Sub design pattern in a .NET distributed cache
Full disclosure: I work for Alachisoft
Suppose, I want to scale out (add more boxes) some WCF service. This looks pretty easy, set up load balancer that calls WCF services on multiple boxes using for example round robin algorithm.
However how to deal with situation when a WCF service have callback contract. When a client connects to some particular box, it receives events only raised by this computer WCF service instance. And I want client to receive events that were raised by any WCF service instance in group (cluster).
What is the best way to make WCF service know about events raised by other WCF service instances?
Some ideas: Multicast, broadcast, WCF NetPeerTcpBinding, Single server that subscribes to all WCF services in cluster (acting as event aggregate).
UPDATE: I have managed to create test system, using NetPeerTCPBinding as a mechanism to share events across servers. I haven't made a benchmark yet, but I feel that WCF P2P is to heavy for this tusk, I'm gonna implement UDP broadcast based event sharing system.
I would implement this by setting up a MSMQ queue that each server can subscribe to, and when an event occurs that the other servers need to know about, the service can publish it.
I use a library called NServiceBus to make this entire process simple. NServiceBus is a full-featured library that uses MSMQ (among other transports) to create pub/sub messaging buses, which would exactly solve your problem. It is easy to use and has a fluent interface for configuration, subscription, and publishing.
I will come back and edit this post later with an example, but the NServiceBus website has plenty of documentation to get you started until then.
Have you considered messaging? Sounds ideal.