It is possible in RabbitMq, I am using MassTransit to consume messages
messages to be retained?
I will consume a message, and through it I will make a search and depending on the result I want to delete it or not, is it possible to do that?
I want to know if it is possible to read a message and not remove it in RabbitMq, so that I can read it a second or third time.
Related
We are using RABBITMQ Queues with C# API to perform distributed work where we have different windows application running, subscribed to a one rabbitmq queue and is working fine but we have a situation where we require to perform some operation only if its a last message in a queue. Is there any way in c# api to know whether the receiving message is the last message or not in a queue.? Something like if an application consumes a message from a queue and we get to know this is the last message and perform some operation.
You can check this http://docs.celeryproject.org/en/latest/userguide/monitoring.html#inspecting-queues
It gives you list of messages in queue and you can easily find last message.
I am evaluating using RabbitMQ as message queue/message bus and have been looking at the example tutorials on the RabbitMQ page.
I am looking for a specific scenario not covered by the tutorials and I am not sure if and how it would be possible to do via RabbitMQ.
The setup:
Let's assume I got a service, let's call it "purchase orders" and I have to other services called "logistics" and "accounting".
When an order is sent, I want to send it as a message via RabbitMQ.
There 2 "account" and 3 "logistic" services
What would be the correct way to ensure that "account" and "logistic" will process the message only once? Using pub/sub will cause the messages to be processed twice (account) or trice (logistics) if i understand it correctly.
With work queues and prefetch=1 it would assure that only one gets it, but I have 2 services and want each type of service to get one.
Is there a way to combine both and have a work queues for each of the service, without sending 2 separate events/messages to two different exchanges?
Using pub/sub will cause the messages to be processed twice (account) or trice (logistics) if i understand it correctly.
you probably have 1 queue per worker, based on your description, and you are routing the message to all worker queues. therefore, each worker gets a copy of the message, because you routed the message to all of the queues.
what you want is a single "account" queue and a single "logistic" queue. you will have multiple account services reading from the single account queue; same for the logistic service / queue.
setting prefetch=1 is important as well. this prevents you from reading too many messages in to a single worker, at once.
Is there a way to combine both and have a work queues for each of the service, without sending 2 separate events/messages to two different exchanges?
yes - don't use a fanout exchange. use a topic or direct exchange, and use multiple routing keys to route a single message to both the account and logistics queues.
What would be the correct way to ensure that "account" and "logistic" will process the message only once?
there is no way to guarantee this, 100%. at some point, even with a proper setup like I've described, you will have a network failure or a worker crash or some other problem and a message will get processed twice. you have to account for this in you design, using some form of idempotence in your message processing.
hope that helps!
I'm using RabbitMQ for the following scenario. When a user uses a premium search feature, I send a message via RabbitMQ to one of a few server instances. They run the same routine (DB queries and billing). I want to make sure I don't process the same message more than once.
I've come across this great tutorial but the exchange type presented in it is "Topic", which does not work for me, because I process the same message more than once.
How can I implement the request-response pattern with worker queues in RabbitMQ so that each message is handled only once and there's load balancing?
Anton Gogolev's comment above is correct. You cannot guarantee a message will be processed only once, for many reasons. But, this is often a requirement of systems - to only produce the desired result once.
The way to do that is through idempotence - the idea that no matter how many times a given message is processed, it will only make the desired change once.
There are a lot of ways to do this. One simple example is to use a shared database that tracks which messages have been processed. When you receive a message, you check to see if it has been processed already. If not, you process it. If it has, you just ignore it and move on.
In your case, if you are doing request/response and want load balancing, you probably want multiple consumers on the same queue. You could have 2 or 10 or 300 instances of your request handler listening to the same queue, and you won't have too much worry about duplicate processing.
RabbitMQ will send a given message to a single consumer. It will wait for that consumer to say it is done processing, or if the consumer crashes or rejects the message, it will requeue the message for another consumer to try again.
In this way, you will generally have only 1 request handler per request. But it will always be possible for more than one to handle the same message, which is why idempotence is important.
Regarding the use of a topic exchange vs any other type of exchange - it doesn't make much difference. There will always be the possibility of more than one queue receiving the message that you are sending, because you can have multiple queues bound to the same exchange with the same binding keys.
I'm working on a task where I'm trying to insure delivery of data to a database in the order in which it is written. The database will be located on another machine, and it's possible that the database machine could fail while the sending computer continues to queue up database messages.
I'm trying to use WCF for this task and from my initial reading believed that this might be possible using WCF with message queue and a ReceiveContext enabled channel. The documentation states that ReceiveContext 'enables an application to decide whether to access the message or leave it in the queue for further processing.'
The problem that I've encountered is that if I call the Abandon method indicating that the message has not been successful processed and should be left on the queue, WCF appears to place the message at the back of the queue instead of leaving it at the front of the queue. Since I need to write the messages to the database in the order they were originally sent this solution will not work.
Is there any way to force WCF to 'peek' at a message before removing it or cause an abandon message to be placed at the front of the queue? If not could anyone suggest a method of accomplishing what I'm attempting to do without having to write a lot of code from scratch.
Thanks,
Al
I have Created a RabbitMQ Producer and a RabbitMQ Consumer....
suppose my producer produces 10 messages. How can i get a particular message from those 10 messages.
I want to know how can i uniquely identify a message and read that or consume that message.
There are several ways to do this, but the one I use most is to use a routing key that is unique to the type of message. Consumers, then, bind to that exchange using a specific routing key, which causes messages to go only to those consumers.
If you can avoid it, you should never just dump messages into a single queue and let the consumers sort them out. The routing keys and exchanges are powerful tools made specifically for routing messages. You should leverage that.
I have an example that shows how to do a topic queue in C# which appears to be what your looking for RabbitMQ Tutorial I also have one that shows how to use the EventingBasicConsumer to avoid blocking when getting messages RabbitMQ EventingBasicConsumer