I am relatively new to MassTransit in .NET and its integration with AWS SNS/SQS. The documentation and tutorials are fine, but as usual, the devil is in the details.
I am especially unsure on what the best way would be to configure MassTransit in an environment with multiple envs, e.g., development, staging, production, and multiple application instances in production environment.
Multi-environment sketch
Here is what I know so far:
For different environments, create a dedicated SNS topic for each message and each environment, e.g., development-my-event and production-my-event.
Endpoints need to be unique. Therefore, create a separate endpoint for each consumer in the application, e.g., development-consumer1-my-event. A second consumer in a different application will get an endpoint development-consumer2-my-event.
Now assuming that there are multiple people working simultaneously on the applications, in order to avoid messaging conflicts, they will need their own topics and queues as well, e.g., development-user123-my-event. Is that a valid way to go?
The production code runs in a clustered environment, i.e., there will be multiple instances of any application. Do we have to make the endpoints of each applications then even more unique, like, addings another identifier so that the queue names do not conflict?
Hoping that this is not the case, I would assume that I can start many identical consumers that listen to the same endpoint, which would (hopefully) also solve that I would only one of those to process e.g. a command message.
Any insights to my thoughts highly appreciated, thanks!
So far:
Implemented custom entity and endpoint name formatters to distinguish different environments.
Formatter diversy further for different dev machines
Specified to use ".fifo" for command queues and topics
Yes, you'd need to add yet another discriminator for each developer. Not sure I would ever do this, I'd likely just use localstack and develop locally.
If by multiple instances, you mean scaled out to load balance by competing consumer on the same queue, that's the default behavior. If you need to fan out events to the same consumer in multiple applications, you'll need to either scope them or use some prefix on the endpoint name formatter.
As mentioned above, competing consumer is the default if a service is scaled out to multiple instances.
Related
Basically I have a requirement to connect to an http streaming API in my app - it works fine, but what I want to do is to connect to multiple instances of this API for different accounts with the streaming provider. For whatever reason this doesn't work as the connections interfere with each other, however if I run multiple instances of the app separately, each connecting to the streaming API for a different account, then it is fine.
So I wanted to know what the best way would be to isolate / sandbox these connections so they might work within the same app. I know it is possible to use separate AppDomains but I wondered if anyone might suggest a better/easier strategy.
I'd like to use MassTransit with Amazon MQ (maybe Amazon SQS/SNS?). When creating the MassTransit Bus is there some means to give the bus a name or prefix? This would allow a team of developers to share the same instance as each queue, route, bridge, etc. name would be prefixed (and thus unique).
Digging thru the MassTransit configuration code for both brokers, it doesn't look like that is possible. Though I would think this would be a broker-independent setting.
RabbitMQ supports virtual hosts which could be used for this purpose, but I think that is specific to RabbitMQ.
You can specify your own EntityNameFormatter, which is used to generate the name for published events. To send messages, you can format the queue names yourself, so you could add a prefix at that level.
This isn't something that is handled at the host level today, though it continues to come up and might be worth further investigation.
Lately we have setup easynetq queues in publish-subscribe and request-response way and common queue connection / endpoint. The goal is setup easynetq in way that developing logic using queue must be independent between developers - currently we have configured easynetq that every developer machine gets his own prefixed set of queues but it looks like that if 2 ot more developers starts subscribers they read messages from owned queues and also other developers queueus.
How configure easynetq using code to resolve this issue?
An easier way to isolate the activities of different developers, as you describe, would be to use a different virtual host (vhost in the connection string) for each developer. vhosts are completely isolated from each other.
Queues could then have the same name for each developer. The fact that they are on a different vhost would separate them.
https://www.rabbitmq.com/vhosts.html
My worker role expose a WCF service and have multiple instances.
I want my client to call this service and make all instances working concurrently.
I'm trying to figure out what is the best way to do this scatter-gather task.
(And I'm trying to avoid service bus and use WCF only)
I can't think of any good way to do this without something like service bus topics. Or using custom functionality that does nearly the same thing. Why are you trying to avoid Service Bus?
There's really no way to make a client-side call to multiple server instances simultaneously, just using Azure's built-in services. Even using Service Bus topics, there's no way to guarantee that multiple subscribers will consume a message at the same time and execute at the same time (even with a message embargo time, you still cannot absolutely guarantee each subscriber will consume + process a message at an exact time).
This will need to be an application-side action. For example: You can queue up your wcf requests. Your queue-reader can then direct-connect to an internal endpoint on each instance, triggering an action to run in parallel. This won't give you exact parallel operation, but it will be pretty close. As another option, you can have several threads available per instance, and you could run the same request on each thread (again, managed by you).
In essence, this is an architectural facet of your app. Azure won't be able to facilitate a parallel-call across instances; you can take advantage of queues, internal services, etc. to accomplish this.
I have a straightforward, existing ASP.NET MVC web solution. The server-based stuff writes information to a database. I am now going to integrate/synchronize this system with a number of other 3rd-party systems. I want to separate the integration processing from the existing core processing, leaving the existing system as untouched as possible.
My plan is as follows:
whenever a database write occurs on the core system server I will write a message to an MSMQ Queue.
an entirely separate server-based windows service will poll the MSMQ, look at the message and will write messages to one or more 'outbound' sync MSMQ queues.
other windows services will monitor the 'outbound' sync queues, and will talk to the 3rd-party systems as necessary, managing the outbound synchronization.
I have a couple of questions:
Should I have a single windows service doing all this, or should I have several services, one central 'routing' one and one for each 3rd-party system?
Should I use WCF for any of this. Does that buy me anything, given that the 'trigger' for writing to the initial queue is already 'happening' on a server-based process?
Thanks very much.
To answer your questions:
Should I have a single windows service doing all this
Definitely not. What if you want to scale out the routing service, or relocate it?
Should I use WCF
If you have your heart set on msmq then the only advantage WCF gives you is it provides a convenient, proven way to design and host your service endpoints, and an alternative to mucking around in System.Messaging. I would say at this stage it doesn't matter that much.
Does that buy me anything
Not sure what you mean, but as Wiktor says in his post, you could chose not to use vanilla .Net or WCF and choose a service bus type framework such as masstransit or nservicebus.
The benefit here is it abstracts you away from the messaging sub-system so you could in theory move away from msmq in the future to rabbitmq or azure queues.
First, a separate windows service is always safer than any attempt to integrate this with your asp.net runtime.
Second, do not write anything by yourself. Use
http://code.google.com/p/masstransit/
It is straightforward and does everything you need. Reference the library from their nuget package, read some tutorials and you will love it.