We are using MassTransit(3.5.2) together with RabbitMQ(3.6.6). We are having a problem with a message that never gets removed from the queue (even if we have read and consumed the message).
To read from the queue we have implemented an IConsumer class.
The only thing we are doing is closing the sender (abrupt).
After that the message never gets acknowledged and rabbitmq is continuing to send the same message to our consumer all the time.
Have anyone else got the same problem and how did you solve this issue?
:: Update from comments ::
We had already checked the log files and it says: "closing AMQP connection <0.18285.1> ([::1]:57008 -> [::1]:5672): client unexpectedly closed TCP connection ".
That doesnt feel that wierd since i am actually closing the tcp connection unexpectedly with killing the .exe file :)
Regarding the log files from masstransit we have also done that and we do not get any error, we only gets two debug messages.
One that we have received and one that we are sending the result.
DEBUG 47 MassTransit.Messages - RECEIVE rabbitmq://localhost/[VirtualHost]/[ConsumerName] N/A ContractCommand CommandConsumer(00:00:00.0364932)
DEBUG 30 MassTransit.Messages - SEND rabbitmq://localhost/[VirtualHost]/bus-[ComputerName]-[SenderName].Server.vshost-4bayyydsf9rfs3qzbdkgx8bbr1?durable=false&autodelete=true d0700000-762f-c85b-f03a-08d4679c39d4 Result
One observation that I have made in my consumer is that at the same time as I am force closing my sender I get an MessageNotConfirmedException followed by some AlreadyClosedException from RabbitMQ.
And it's after that we get in the infinite loop when MT does not set the ACK/NACK. (and in the infinite loop I do not get any MessagenNotConfirmedException).
Also for my consumer to properly work again I need to restart my consumer then it will be ACK/NACKed.
MessageNotConfirmedMessage: "'MassTransit.RabbitMqTransport.MessageNotConfirmedException' in mscorlib.dll Additional information: rabbitmq://localhost/[VirtualHost]/bus-[ComputerName]-[Service].Server.vshost-4bayyydsf9rfsf3ybdkgxg5h8b => The message was not confirmed by RabbitMQ
'RabbitMQ.Client.Exceptions.AlreadyClosedException' Additional information: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=404, text="NOT_FOUND - no exchange 'bus-[ComputerName]-[ServiceName].Server.vshost-4bayyydsf9rfs3qzbdkgx8bbr1' in vhost '[VirtualHost]'", classId=60, methodId=40, cause=
I have updated our code to use MassTransit 3.5.7 and after that I have not been able to reproduce this issue.
When you are done processing your queue object, you can mark it as successful by BasicAck(e.DeliveryTag, false)
Related
I have a WebJob getting messages from Service bus which is working fine. If the webjob fails to finish or throws exception the message is sent back in the queue which is fine in itself.
I have set MaxDequeueCount to 10 and it is not a problem if it fails in some cases as it will try to process it again. But the problem is that the message seems to be sent to the bottom of the queue and other messages are handled before we get back to the first failed one.
I would like to handle one message at a time because I might have multiple updates on the same entity coming in a row. If the order changes it would update the entity in wrong order.
Is if it is possible to send the message back infront of the queue on error or continue working on the same message until we reach the MaxDequeueCount?
Ideally, you should not be relying on message order.
Given your requirement, you could potentially go with the FIFO semantics of Azure Service Bus Sessions. When a message is handled within a session and message is aborted, it will be handled once again rather than go to the end of the queue. You need to keep in mind the following:
Can only process one message at a time
Requires session to be used
If message is not completed and not aborted, it will be left hanging in the queue and will be picked up when a session is restarted.
I'm integrating RMQ into my project, in order to implement work queues.
I understand what if module succeeded, it calls the ack method so RMQ will know about it.
What about failures?
I read that only when connection or channel are closed, RMQ knows we've failed and re-push the message to the queue.
I'd like however to make the RMQ re-push messages whenever I have an internal error, regardless of wheter I crash or not (e.g. failure to insert to DB, I handle that gracefully without crashing however I want the whole job to be re-tried).
Do I have to manually close and open the channel again in order to trigger that?
You can use negative ACK, or rejects. Info here.
The AMQP specification defines the basic.reject method that allows
clients to reject individual, delivered messages, instructing the
broker to either discard them or requeue them.
When I start consuming from a queue that is already locked by another exclusive consumer then no exception is thrown by the consume method, although I can see the error being logged. The ExclusiveConsumer keeps trying to start consuming every 10 seconds, which is great, but I would like to check (from code) when it hasn't started.
Consume failed. queue='Queue1', consumer tag='cb439965-72d4-4d52-a248-91077b796703', message='The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=403, text="ACCESS_REFUSED - queue '1' in vhost '/' in exclusive use", classId=60, methodId=20, cause='
Is there any way I can detect the Consumer status without modifying the code?
I'm using RabbitMQ to deliver messages to worker processes (using the official C# client). I have been running simple tests during the implementation, and all has been going swimmingly until now.
I ran a test where I queued messages for a worker process that was not listening (no connection). Once I had queued several hundred messages, I started that process. It created its IModel, declared its queue (which already existed), and began consuming messages (with BasicConsume). This went great. This process, as it processed messages, created messages for other queues. There were processes already listening to these queues (with BasicConsume), and so the messages were immediately delivered to those clients (or so the server thought...). The messages are never processed.
The server definitely believes that the messages have been delivered (the messages are all in the "unacked" bucket, not the "ready" bucket), but
IBasicConsumer.HandleBasicDeliver never got called on the client. I have tried several different techniques (using a Subscription, using QueueingBasicConsumer as well as my own custom consumer), and the outcome is exactly the same. I'm at a complete loss. If I close the connection (there is only one connection here), then the messages immediately move from the "unacked" bucket to the "ready" bucket".
Why doesn't the client get notified when messages are delivered?
Looking into the code, ModelBase.Close() calls ConsumerDispatcher.Shutdown() (ModelBase.cs line 301), and from there, it calls workService.StopWork() (ConcurrentConsumerDispatcher.cs line 27). It seems to me (by a cursory view of the code) that this stops ALL work in the connection's ConsumerWorkService. Instead, should ConcurrentConsumerDispatcher.Shutdown() be calling workService.StopWork(this) on line 27?
It's a bug in the RabbitMQ client, and a fix has already been merged in.
It should be available in the next nightly build, on 4/18/2015.
If your BasicConsume defines noAck = false, after you Dequeues a message needs to run the next code:channel.BasicAck(result.DeliveryTag, false);
If your BasicConsume defines noAck = true, after you Dequeues a message it's removed from the server automatically.
I am using the C# UdpClient class to to UDP networking. There is one UdpClient object, bound to a fixed local port, but not to any remote endpoint, because it needs to be able to send/receive to/from multiple different endpoints.
I have two threads: One for sending, one for receiving. Now, when I send data to an endpoint that exists, but doesn't listen on that port, I expect a SocketException. And I do get one. Unfortunately, it is not my Send call that returns the exception, but the Receive call. So on my sending thread, I send data to an "invalid" endpoint, and my receiving thread gets the exception. Unfortunately, at that point, I have of course no idea what endpoint caused that exception to happen.
Storing the endpoint before sending, then accessing that in the receiving thread is just a race condition error waiting to happen.
Unfortunately, the SocketException does not give me the endpoint that caused the error.
Any ideas? Is it somehow possible to make the exception be thrown on the sending thread?
Help is greatly appreciated.
When you send() a UDP packet, it goes out on the wire and effectively disappears. You should not assume that you will get any feedback at all.
Sometimes, if there is no listener at the destination, the destination might be kind enough to send back an ICMP_UNREACH_PORT message. The routers in between then might be kind enough to deliver that message to your operating system. If that happens, it will be long after your original send() call returned. For ICMP_UNREACH_PORT, the OS typically caches it and reports an error the next time yo do a send() to the same destination. Other ICMP messages (you didn't mention which exception you are getting) could affect other calls.
So the bottom line is that there's no telling when, or if, UDP errors will be reported. It depends on a lot of variables. So be prepared to handle exceptions on any call, and be prepared for packets to just disappear without any error reported.
I think this is expected behavior for UDP. A UDP send() is not a blocking operation, so it won't wait for a potential error. (Not to mention the fact that you can't rely on the error messages being reliably received when sending to an active host with a closed port - it could be firewalled, rate-limited or otherwise dropped due to congestion, etc.)
You can connect() the UDP socket to a specific remote endpoint, which would allocate a unique port number and allow the OS to [most likely] distinguish errors from that specific endpoint from any other random host. But again, you should not rely on the ability to handle these errors.
It's too bad there isn't more information in the exception. This seems like an oversight in the way .NET handles UDP sockets. According to the documentation, you need to check the exception's ErrorCode and handle the error appropriately. (which, in your case, could likely mean ignoring the error.)