I've been working on sending and receiving ActiveMQ messages between .net applications and everything work great when there is a active Consumer Connection.
However there are scenarios that I won't be able to have a active Consumer, example: web application consuming active messages.
I notice that any message not delivered to a consumer is place in a "Messages Enqueued" bucket.
Is there a way I can consume an already send and Enqueued message for a particular topic?
I am workign with Topic only, as shown on the example below
any tips from .net or even java might help me out thanks
Topics (except for durable topics) are inherently going to throw messages away if you don't have a consumer connected at the time the message is sent. That's just how topics work. My guess is that the web UI's column header is misleading- it probably should say "messages sent". You cannot retrieve them.
More on durable topics and queues here: http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
Related
I'm using RabbitMQ for the following scenario. When a user uses a premium search feature, I send a message via RabbitMQ to one of a few server instances. They run the same routine (DB queries and billing). I want to make sure I don't process the same message more than once.
I've come across this great tutorial but the exchange type presented in it is "Topic", which does not work for me, because I process the same message more than once.
How can I implement the request-response pattern with worker queues in RabbitMQ so that each message is handled only once and there's load balancing?
Anton Gogolev's comment above is correct. You cannot guarantee a message will be processed only once, for many reasons. But, this is often a requirement of systems - to only produce the desired result once.
The way to do that is through idempotence - the idea that no matter how many times a given message is processed, it will only make the desired change once.
There are a lot of ways to do this. One simple example is to use a shared database that tracks which messages have been processed. When you receive a message, you check to see if it has been processed already. If not, you process it. If it has, you just ignore it and move on.
In your case, if you are doing request/response and want load balancing, you probably want multiple consumers on the same queue. You could have 2 or 10 or 300 instances of your request handler listening to the same queue, and you won't have too much worry about duplicate processing.
RabbitMQ will send a given message to a single consumer. It will wait for that consumer to say it is done processing, or if the consumer crashes or rejects the message, it will requeue the message for another consumer to try again.
In this way, you will generally have only 1 request handler per request. But it will always be possible for more than one to handle the same message, which is why idempotence is important.
Regarding the use of a topic exchange vs any other type of exchange - it doesn't make much difference. There will always be the possibility of more than one queue receiving the message that you are sending, because you can have multiple queues bound to the same exchange with the same binding keys.
I am working to port an application which was designed to work in a non-Azure environment. One of the elements of the architecture is a singleton which does not scale, and which I'm hoping to replace w/ multiple worker processes serving the resource that the singleton currently provides.
I have the necessary changes in place to replace the singleton, and am at the point of constructing the communications framework to provide interconnection from the UI servers to the resource workers and I'm wondering if I should just use a TCP binding on a WCF service or whether using the Azure Service Bus would make more sense. The TCP/WCF is easy, but doesn't solve the complete problem: how do I ensure that only one worker processes a UI request?
From reading the available documentation, it sounds like the service bus will solve this, but I've yet to see a concrete example of implementation. I'm hoping someone here can assist and/or point me in the right direction.
Seems that Azure Service Bus queues are the right solution for you.
Azure Service Bus can be used in 3 different ways:
Queues
Topics
Relays
From windows azure site:
Service Bus queues provide one-way asynchronous queuing. A sender sends a message to a Service Bus queue, and a receiver picks up that message at some later time. A queue can have just a single receiver
You can find more info at:
http://www.windowsazure.com/en-us/develop/net/fundamentals/hybrid-solutions/
Adding to Davide's answer.
Another alternative would be to use Windows Azure Queues. They are designed to facilitate asynchronous communication between web and worker roles. From your web role you push messages into a queue which is polled by your worker roles.
Your worker role can "Get" one or more messages from a queue and work on those messages. When you get a message from a queue, you can instruct the queue service to make those messages invisible to other callers for a certain amount of time (known as message visibility timeout). That would ensure that only worker role instance get to work on a message.
Once the worker role has completed the work, it can simply delete the message. If there's an error in processing the message, the message automatically reappears in the queue once the visibility timeout period has expired. You may find this link helpful: http://www.windowsazure.com/en-us/develop/net/how-to-guides/queue-service/.
Azure queues are not designed for inter process communication, but inter-application communication. The message delivery latency is substantial, and delivery timing cannot be guaranteed. Websockets or NetTcpBinding is more suitable for applications that talk to eachother in realtime. Although must admit, you get some free stuff with queuez, especially the locking mechanisms. Just my 2 cents
I am looking for a message broker API to use it with c#.
Normally the things are quite simple. I have a server that knows what jobs are to do and I have some clients that need to get these jobs.
And here are the special requirements I have:
If a client got a job but fails to answer within a specific time, then another client should do the work.
More than one queue and priorities
If possible it needs to work with big message queues (this way I could just load all jobs sometimes a month and forget about it
secured communications would be good.
API for talking with the broker from c#. How much work is done? What is still to do?
Delete some jobs...
If available replication to another broker would be good.
The broker needs to run on windows
What is not an issue:
low latency (there is no problem when a message needs minutes)
Do you know such a message broker that is free to use?
RabbitMQ and several other AMQP implementations satisfy most of (if not all of) these requirements.
RabbitMQ allows clients to acknowledge receipt and/or processing of messages. As per http://www.rabbitmq.com/tutorials/amqp-concepts.html#message-acknowledge:
If a consumer dies without sending an acknowledgement the AMQP broker
will redeliver it to another consumer or, if none are available at the
time, the broker will wait until at least one consumer is registered
for the same queue before attempting redelivery.
Many queues (and in fact many brokers) are supported, in a variety of different configurations
It scales particularly well, even for very large message queues: http://www.rabbitmq.com/faq.html#performance
Encryption is supported: http://www.rabbitmq.com/faq.html#channel-encryption
There is a .NET Client Users Guide and API docs: http://www.rabbitmq.com/documentation.html
There is live failover if a broker dies: http://www.rabbitmq.com/clustering.html
It runs on Windows, Linux, and probably anything else that has an Erlang implementation
I have Created a RabbitMQ Producer and a RabbitMQ Consumer....
suppose my producer produces 10 messages. How can i get a particular message from those 10 messages.
I want to know how can i uniquely identify a message and read that or consume that message.
There are several ways to do this, but the one I use most is to use a routing key that is unique to the type of message. Consumers, then, bind to that exchange using a specific routing key, which causes messages to go only to those consumers.
If you can avoid it, you should never just dump messages into a single queue and let the consumers sort them out. The routing keys and exchanges are powerful tools made specifically for routing messages. You should leverage that.
I have an example that shows how to do a topic queue in C# which appears to be what your looking for RabbitMQ Tutorial I also have one that shows how to use the EventingBasicConsumer to avoid blocking when getting messages RabbitMQ EventingBasicConsumer
I'm designing a service for sending out emails for our eCommerce site (order confirmations, alerts etc...)
The plan is to have a "SendEmail" method, that generates a chunk of XML representing the email to be sent, and sticks it on an Amazon SQS queue.
My web app(s) and other applications will use this to "send" emails.
I then require a way of checking the queue, and physically sending out the email messages.
(I know how I'm going to be dispatching emails)
I'm curious as to what the best way to "poll" the queue would be?
Should I create a windows service, and use something like Quartz.net to schedule it to check the queue every x number of minutes for example?
Is there a better way of doing this?
Sounds more like Amazon SNS is your huckleberry. (not really sure what that means, but I saw it in a movie once).
SQS is more of a "hopefully someone comes looking for this message at some point before it expires!" where SNS seems more like a "I need to make sure this gets to whoever needs it right away!"
It even includes email as a pre-built transport. (not even sure if that's the right word)
Amazon SNS provides a simple web services interface that can be used to create topics you want to notify applications (or people) about, subscribe clients to these topics, publish messages, and have these messages delivered over clients’ protocol of choice (i.e. HTTP, email, etc.). Amazon SNS delivers notifications to clients using a “push” mechanism that eliminates the need to periodically check or “poll” for new information and updates.
To be sure, they have the same freemium model as the rest of the services with limitations on email:
You can get started with Amazon SNS for free. Each month, Amazon SNS customers pay no charges for the first 100,000 Amazon SNS Requests, no charges for the first 100,000 Notifications over HTTP and no charges for the first 1,000 Notifications over Email.
If I were you, and if I REALLY wanted to make SQS work for this scenario, I would create a windows service that retrieves any messages from the queue every 10 minutes and then dispatches them.
If I could potentially use a different service, I would seriously consider using something like Postmark (Which I just found out about today thanks to a comment on Jeff Atwood’s blog post). You would just submit your email message info to their api, and they would do the rest. They even have a api for checking for bounced emails. I have never used this service, but I think it sounds great and would seriously consider using it in the future.
Looks like SES is what you need. http://aws.amazon.com/ses/ It is very inexpensive too compared to other similar services or building other systems.
You can use the AWS Beanstalk service to consume the tasks in the Queue.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
Windows service is the best way, but you must not take it. To run background job in ASP.NET environment there is HostingEnvironment.QueueBackgroundWorkItem. A good starting point to implement continues polling (via long polling) is sqs-consumer.