I am experiencing a racing condition issue with my rabbitmq client. My service has multiple instances listening on a single queue, storing received messages into a db.
When they all get restarted at once, i sometimes see messages being redelivered and stored in the db twice. This is normally handled on client side by checking if the correlationid has already been stored in the db. This works 99.9% of the time (i am processing 5mill messages a day, it happens once or twice a day).
So as i said, i suspect a racing condition being responsible for this. I think i receive the message again while my first message is still being processed. So when i check i dont see it stored in the db, and in the end, store it twice.
I should not that this is a non-issue, but has been bothering me because i can't really explain what happens.
I suspect that it happens when i restart the services. I think i disconnect from the queue, while i am still processing the message, triggering rabbitmq to redeliver again to another instance that is not shutdown yet.
What i want to do is when i am stopping the service is to
tell rabbitmq that i dont want to receive further messages
wait for all currently processing messages to finish
send acks / nacks
shutdown
Right now i am first deregistering the received event
_consumerServer.Received -= MessageReceived;
then i am disposing the channel and the server
if (_channel != null)
{
_channel.Close();
_channel.Dispose();
}
if (_connectionServer != null)
{
_connectionServer.Close();
_connectionServer.Dispose();
}
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Rather than try and shut down a consumer so that messages won't be redelivered, you should handle redelivery correctly. Check for and handle the case where the redelivered flag is set on a message, and act appropriately. You should also try store your messages in such a way that the store operation is idempotent - i.e. it can happen multiple times and you will only have one record in your database.
Please see the guidelines that the team have provided here:
https://www.rabbitmq.com/reliability.html#consumer
Related
So I'm trying to get MSMQ messages forwarded from one machine to another (which is dead easy - I was surprised), but one of the requirements from the ops side of the house is that we need to be able to see a log entry somewhere when the remote server decides not to accept a message. For example, if I try to send to a nonexistent queue, like so:
MessageQueue remoteQueue = new MessageQueue(#"FormatName:Direct=OS:machinename\private$\notarealqueue");
remoteQueue.Send("Test", MessageQueueTransactionType.Single);
The message goes into the local delivery queue, and appears to get sent across the network, but because the queue doesn't exist, the remote MSMQ manager discards the message. However, there's no entry in the Event Log that I can find about the message being dropped on the floor, and that makes people nervous. The Microsoft/Windows/MSMQ/EndToEnd log only seems to involve successful messages, which doesn't seem particularly useful. Is there a log I'm not seeing somewhere?
You can use MSMQ dead letter queues for that.
message.UseDeadLetterQueue = true;
With that enabled, if message can't be delivered it will be sent to one of two system dead letter queues - one for transactional and one for non transactional messages. You'll also find there the reason why message was not delivered, which was the original destination queue, full message body, label, etc.
You can use one of tools for managing queues to resend or recover these messages.
The event log is solely for the health state of MSMQ. What happens to a single message is trivial and not logged in the event log. Imagine what would happen if a million messages were discarded and had to be logged in the event log.
I'm using RabbitMQ to deliver messages to worker processes (using the official C# client). I have been running simple tests during the implementation, and all has been going swimmingly until now.
I ran a test where I queued messages for a worker process that was not listening (no connection). Once I had queued several hundred messages, I started that process. It created its IModel, declared its queue (which already existed), and began consuming messages (with BasicConsume). This went great. This process, as it processed messages, created messages for other queues. There were processes already listening to these queues (with BasicConsume), and so the messages were immediately delivered to those clients (or so the server thought...). The messages are never processed.
The server definitely believes that the messages have been delivered (the messages are all in the "unacked" bucket, not the "ready" bucket), but
IBasicConsumer.HandleBasicDeliver never got called on the client. I have tried several different techniques (using a Subscription, using QueueingBasicConsumer as well as my own custom consumer), and the outcome is exactly the same. I'm at a complete loss. If I close the connection (there is only one connection here), then the messages immediately move from the "unacked" bucket to the "ready" bucket".
Why doesn't the client get notified when messages are delivered?
Looking into the code, ModelBase.Close() calls ConsumerDispatcher.Shutdown() (ModelBase.cs line 301), and from there, it calls workService.StopWork() (ConcurrentConsumerDispatcher.cs line 27). It seems to me (by a cursory view of the code) that this stops ALL work in the connection's ConsumerWorkService. Instead, should ConcurrentConsumerDispatcher.Shutdown() be calling workService.StopWork(this) on line 27?
It's a bug in the RabbitMQ client, and a fix has already been merged in.
It should be available in the next nightly build, on 4/18/2015.
If your BasicConsume defines noAck = false, after you Dequeues a message needs to run the next code:channel.BasicAck(result.DeliveryTag, false);
If your BasicConsume defines noAck = true, after you Dequeues a message it's removed from the server automatically.
I'm sending a message to a private queue via c# :
MessageQueue msgQ = new MessageQueue(#".\private$\aaa");
msgQ.Formatter = new XmlMessageFormatter(new[] { typeof (String) });
msgQ.Send(msg);
It does work and I do see the message in the queue.
However, is there any way to get an ACK whether the message got to the queue with success ?
ps
BeginPeek and PeekCompleted is an event which is raised when a message becomes available in the queue or when the specified interval of time has expired. it is not helping me because I need to know if the message that I sent was received by msmq. BeginPeek will be raised also if someone else entered a message to the queue. and the last thing I want is to check via BeginPeek - from who this message comes from.
How can I do that?
ps2
Or maybe I don't have to worry since msgQ.Send(msg); will raise an exception if a message wasn't inserted....?
I think what you are trying to do should not be handled in code. When you send the message, it is placed in the outgoing queue. There are numerous reasons why it would not reach the destination, such as a network partition or the destination queue being full. But this should not matter to your application - as far as it is concerned, it sent the message, it committed transaction, it received no error. It is a responsibility of the underlying infrastructure to do the rest, and that infrastructure should be monitored to make sure there are no technical issues.
Now what should really be important to your application is the delivery guarantees. I assume from the scenario that you are describing that you need durable transactional queues to ensure that the message is not lost. More about the options available can be read here
Also, if you need some identifier to display to the user as a confirmation, a common practice is to generate it in the sending code and place it in the message itself. Then the handling code would use the id to do the required work.
Using transactional queues and having all your machines enroll in DTC transactions likely would provide what you're looking for. However, it's kinda a pain in the butt and DTC has side effects - like all transactions are enrolled together, including DB transactions.
Perhaps a better solution would to be use a framework like MassTransit or NServiceBus and do a request-response, allowing the reviecer to respond with actual confirmation message say not only "this has been delivered" but also "I acknowledge this" with timeout options.
As Oleksii have explained about reliable delivery.
However this can effect on performance.
What I can suggest is:
Why not create a MSMQ server on the machine that is sending MSG to other system.
What I am thinking is
Server 1 sends MSMSQ to Server 2
Server 2 receives adds to queue
Server 2 process queue/fire your code here to send a MSMQ msg to Server 1
Server 1 receives MSG (any successful msg with MSGId)
Do your further task
This approach can be an extra mile, but will keep your servers out of performance Lag.
We have pub/sub application that involves an external client subscribing to a Web Role publisher via an Azure Service Bus Topic. Our current billing cycle indicates we've sent/received >25K messages, while our dashboard indicates we've sent <100. We're investigating our implementation and checking our assumptions in order to understand the disparity.
As part of our investigation we've gathered wireshark captures of client<=>service bus traffic on the client machine. We've noticed a regular pattern of communication that we haven't seen documented and would like to better understand. The following exchange occurs once every 50s when there is otherwise no activity on the bus:
The client pushes ~200B to the service bus.
10s later, the service bus pushes ~800B to the client. The client registers the receipt of an empty message (determined via breakpoint.)
The client immediately responds by pushing ~1000B to the service bus.
Some relevant information:
This occurs when our web role is not actively pushing data to the service bus.
Upon receiving a legit message from the Web Role, the pattern described above will not occur again until a full 50s has passed.
Both client and server connect to sb://namespace.servicebus.windows.net via TCP.
Our application messages are <64 KB
Questions
What is responsible for the regular, 3-packet message exchange we're seeing? Is it some sort of keep-alive?
Do each of the 3 packets count as a separately billable message?
Is this behavior configurable or otherwise documented?
EDIT:
This is the code the receives the messages:
private void Listen()
{
_subscriptionClient.ReceiveAsync().ContinueWith(MessageReceived);
}
private void MessageReceived(Task<BrokeredMessage> task)
{
if (task.Status != TaskStatus.Faulted && task.Result != null)
{
task.Result.CompleteAsync();
// Do some things...
}
Listen();
}
I think what you are seeing is the Receive call in the background. Behind the scenes the Receive calls are all using long polling. Which means they call out to the Service Bus endpoint and ask for a message. The Service Bus service gets that request and if it has a message it will return it immediately. If it doesn't have a message it will hold the connection open for a time period in case a message arrives. If a message arrives within that time frame it will be returned to the client. If a message is not available by the end of the time frame a response is sent to the client indicating that no message was there (aka, your null BrokeredMessage). If you call Receive with no overloads (like you've done here) it will immediately make another request. This loop continues to happend until a message is received.
Thus, what you are seeing are the number of times the client requests a message but there isn't one there. The long polling makes it nicer than what the Windows Azure Storage Queues have because they will just immediately return a null result if there is no message. For both technologies it is common to implement an exponential back off for requests. There are lots of examples out there of how to do this. This cuts back on how often you need to go check the queue and can reduce your transaction count.
To answer your questions:
Yes, this is normal expected behaviour.
No, this is only one transaction. For Service Bus you get charged a transaction each time you put a message on a queue and each time a message is requested (which can be a little opaque given that Recieve makes calls multiple times in the background). Note that the docs point out that you get charged for each idle transaction (meaning a null result from a Receive call).
Again, you can implement a back off methodology so that you aren't hitting the queue so often. Another suggestion I've recently heard was if you have a queue that isn't seeing a lot of traffic you could also check the queue depth to see if it was > 0 before entering the loop for processing and if you get no messages back from a receive call you could go back to watching the queue depth. I've not tried that and it is possible that you could get throttled if you did the queue depth check too often I'd think.
If these are your production numbers then your subscription isn't really processing a lot of messages. It would likely be a really good idea to have a back off policy to a time that is acceptable to wait before it is processed. Like, if it is okay that a message sits for more than 10 minutes then create a back off approach that will eventually just be checking for a message every 10 minutes, then when it gets one process it and immediately check again.
Oh, there is a Receive overload that takes a timeout, but I'm not 100% that is a server timeout or a local timeout. If it is local then it could still be making the calls every X seconds to the service. I think this is based on the OperationTimeout value set on the Messaging Factory Settings when creating the SubscriptionClient. You'd have to test that.
I have an NServiceBus application for which a given message may not be processed due to some external event not having taken place. Because this other event is not an NSB event I can't implement sagas properly.
However, rather than just re-queuing the message (which would cause a loop until that external event has occurred), I'm wrapping the message in another message (DelayMessage) and queuing that instead. The DelayMessage is picked up by a different service and placed in a database until the retry interval expires. At which point, the delay service re-queues the message on the original queue so another attempt can be made.
However, this can happen more than once if that external event still hasn't taken place, and in the case where that even never happens, I want to limit the number of round trips the message takes. This means the DelayMessage has a MaxRetries property, but that is lost when the delay service queues the original message for the retry.
What other options am I missing? I'm happy to accept that there's a totally different solution to this problem.
Consider implementing a saga which stores that first message, holding on to it until the second message arrives. You might also want the saga to open a timeout as well so that your process won't wait indefinitely if that second message got lost or something.