How can I send a message (or publish an event) when a message runs out of retries and moves to the error queue?
When a request comes into my system, I create a Saga to track it. The Saga sends commands to Handlers to do async work. If the handler fails, I want to both move that command to the error queue (the default behavior) and send a message to the Saga to alert the client that originally requested the work.
I have tried customizing the recoverability behavior to use the Saga as the error queue, which sends the command back but does not get it into the error queue:
recoverability.CustomPolicy((config, context) =>
{
// invocation of default recoverability policy
var action = DefaultRecoverabilityPolicy.Invoke(config, context);
if (action is MoveToError)
{
return RecoverabilityAction.MoveToError("SagaEndpoint");
}
return action;
});
Another thing I tried was using a Behavior to hook into the Pipeline, but there does not seem to be a a way to override the "move to error queue" step. I can create an IIncomingLogicalMessageContext and try/catch around the await next();, but that triggers for each retry instead of just the final one. I also tried an IOutgoingLogicalMessageContext, but that does not get invoked when a message moves to the error queue. If I missed something, that could be a solution.
I also know I can use a timeout in the Saga to guess when the Handler fails. But I would rather not wait for a timeout if the failure is quick or risk timing out if the work takes longer than expected.
I found this older question that sounds like it's asking the same thing, but the answer is incomplete and uses the older EventHandler Notifications instead of the newer Task-based Notifications. If there is a way to access an IMessageSession or IEndpointInstance from the Notification callback, I think that would work for me as well.
There's not an "easy" way to do that because at the moment when recoverability is happening, any transaction related to the incoming message (this is different for each transport) is in doubt, so you can't really do anything else within the scope of what's going on right at that moment.
Once you start your endpoint, you can cast the IEndpointInstance to an IMessageSession (same thing without things like the Stop method) and then assign that to a place where your "error queue notifier" will be able to find it. Then any operation you do with the IMessageSession will basically be a separate context, disconnected from the processing of the incoming message.
Just understand that if the message is failing processing because of an underlying problem with the queue, that's not going to report correctly. That's why most people would be doing some sort of call to a reporting/diagnostics service in those callbacks.
Related
I have a Azure Function v1 triggered via Service Bus topic. If there happens any error, I put the BrokeredMessage to dead letter. It seems to work but after that I see following in Function's log streaming:
2019-11-19T10:49:31.382 [Error] MessageReceiver error
(Action=Complete) :
Microsoft.ServiceBus.Messaging.MessageLockLostException: The lock
supplied is invalid. Either the lock expired, or the message has
already been removed from the queue.
Here is what how I am putting BrokeredMessage to dead letter:
myBrokeredMessage.DeadLetter(deadLetterReason, exception.Message);
// after this I have tried following but doesn't work:
// 1. do nothing
// 2. myBrokeredMessage.Complete();
// 3. myBrokeredMessage.Abandon();
My Function is running fine. But after it has run and executed above code, that error appears to log streaming. It seems doing what I want (putting BrokeredMessage to dead letter queue), but that error doesn't seem nice and I want to fix it. I guess there is some kind of lock that I'm not handling correctly.
What should I do to fix that error?
What should I do to fix that error?
This is more of a warning than an error. The way Functions are designed is that by default the function completes or dead-letters the message. If you take control over what happens to the incoming message, Functions runtime doesn't it and still tries to apply the logic of completion as from its perspective there was no exception thrown from the user code and therefore the incoming message should be considered successfully processed and be completed.
With Functions 2.x there's a host setting that you could turn on to allow manual completion and disable the automatic completion. That is not available in v1.0, so you'll have to ignore the logged error. Or, alternatively, upgrade to 2.x.
I am experiencing a racing condition issue with my rabbitmq client. My service has multiple instances listening on a single queue, storing received messages into a db.
When they all get restarted at once, i sometimes see messages being redelivered and stored in the db twice. This is normally handled on client side by checking if the correlationid has already been stored in the db. This works 99.9% of the time (i am processing 5mill messages a day, it happens once or twice a day).
So as i said, i suspect a racing condition being responsible for this. I think i receive the message again while my first message is still being processed. So when i check i dont see it stored in the db, and in the end, store it twice.
I should not that this is a non-issue, but has been bothering me because i can't really explain what happens.
I suspect that it happens when i restart the services. I think i disconnect from the queue, while i am still processing the message, triggering rabbitmq to redeliver again to another instance that is not shutdown yet.
What i want to do is when i am stopping the service is to
tell rabbitmq that i dont want to receive further messages
wait for all currently processing messages to finish
send acks / nacks
shutdown
Right now i am first deregistering the received event
_consumerServer.Received -= MessageReceived;
then i am disposing the channel and the server
if (_channel != null)
{
_channel.Close();
_channel.Dispose();
}
if (_connectionServer != null)
{
_connectionServer.Close();
_connectionServer.Dispose();
}
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Rather than try and shut down a consumer so that messages won't be redelivered, you should handle redelivery correctly. Check for and handle the case where the redelivered flag is set on a message, and act appropriately. You should also try store your messages in such a way that the store operation is idempotent - i.e. it can happen multiple times and you will only have one record in your database.
Please see the guidelines that the team have provided here:
https://www.rabbitmq.com/reliability.html#consumer
I'm new to service bus and not able to figure this out.
Basically i'm using Azure function app which is hooked onto the service bus queue. Let's say a trigger is fired from the service bus and I receive a message from the queue, and in the processing of that message something goes wrong in my code. In such cases how do I make sure to put that message back in the queue again? Currently its just disappearing into thin air and when I restart my function app on VS, the next message from the queue is taken.
Ideally only when all my data processing is done and when i hit myMsg.Success() do I want it to be removed from the queue.
public static async Task RunAsync([ServiceBusTrigger("xx", "yy", AccessRights.Manage)]BrokeredMessage mySbMsg, TraceWriter log)
{
try{ // do something with mySbMsg }
catch{ // put that mySbMsg back in the queue so it doesn't disappear. and throw exception}
}
I was reading up on mySbMsg.Abandon() but it looks like that puts the message in the dead letter queue and I am not sure how to access it? and if there is a better way to error handle?
Cloud queues are a bit different than in-memory queues because they need to be robust to the possibility of the client crashing after it received the queue message but before it finished processing the message.
When a queue message is received, the message becomes "invisible" so that other clients can't pick it up. This gives the client a chance to process it and the client must mark it as completed when it is done (Azure Functions will do this automatically when you return from the function). That way, if the client were to crash in the middle of processing the message (we're on the cloud, so be robust to random machine crashes due to powerloss, etc), the server will see the absence of the completed message, assume the client crashed, and eventually resend the message.
Practically, this means that if you receive a queue message, and throw an exception (and thus we don't mark the message as completed), it will be invisible for a few minutes, but then it will show up again after a few minutes and another client can attempt to handle it. Put another way, in Azure functions, queue messages are automatically retried after exceptions, but the message will be invisible for a few minutes inbetween retries.
If you want the message to remain on the queue to be retried, the function should not swallow exception and rather throw. That way Function will not auto-complete the message and retry it.
Keep in mind that this will cause message to be retried and eventually, if exception persists, to be moved into dead-letter queue.
As per my understanding, I think what you are for is if there is an error in processing the message it needs to retry the execution instead of swallowing it. If you are using Azure Functions V2.0 you define the message handler options in the host.json
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": false,
"maxConcurrentCalls": 1
}
}
}
prefetchCount - Gets or sets the number of messages that the message receiver can simultaneously request.
autoComplete - Whether the trigger should automatically call complete after processing, or if the function code will manually call complete.
After retrying the message n(defaults to 10) number of times it will transfer the message to DLQ.
If I have a service bus brokered message receiver configured. and it fails for any reason. I call on it
message.abandon();
however this means the message will be back again in the queue/subscription.
can i configure a timeout after which the same message is available in the queue for processing.
For example: if there is only one message in the queue. and it's failing, then it is not good to keep processing it every second or every minute. if i configure something, that can make sure, the failed/abandoned message only reappears after 30 mins . then it is useful.
Any suggestions?
When you abandon a message, you cannot supply a "cool off" time. The message will be available right away. It won't be dead-lettered until MaxDeliveryCount attempts have been exhausted. Once all those processing attempts have been used up, the message will go to the dead-letter queue.
If you need to postpone message processing, there are several options.
Deferring a message
You could defer a message using BrokeredMessage.DeferAsync(). The message will go back to the queue for future processing and a SequenceNumber of the message will be returned. The caveat with this approach is the need to hold on to the SequenceNumber in order to retrieve the message later. If you happened to lose SequenceNumber, it is still possible to browse for deferred messages and retrieve those. More information here.
Scheduling a new future message
Another option would be to clone an incoming failing message, schedule it for some time in future using BrokeredMessage.ScheduledEnqueueTimeUTC and completing the original message. With this option, I'd recommend to send the new message scheduled in future to be dispatch using send-via feature, also known as Transaction Processing, to leverage atomic operation of completing the incoming message and sending the outgoing one. This way the code will not produce "ghost" message if completion fails. More information here.
Scheduling using client, not message
Another option is to schedule using a client and not BrokeredMessage using client.ScheduleMessageAsync(). It will return aSequenceNumber` you need to hold on to, but using this API a message can be cancelled at any point in time w/o waiting for the schedule time to arrive or receiving the message. More information here.
I have an NServiceBus application for which a given message may not be processed due to some external event not having taken place. Because this other event is not an NSB event I can't implement sagas properly.
However, rather than just re-queuing the message (which would cause a loop until that external event has occurred), I'm wrapping the message in another message (DelayMessage) and queuing that instead. The DelayMessage is picked up by a different service and placed in a database until the retry interval expires. At which point, the delay service re-queues the message on the original queue so another attempt can be made.
However, this can happen more than once if that external event still hasn't taken place, and in the case where that even never happens, I want to limit the number of round trips the message takes. This means the DelayMessage has a MaxRetries property, but that is lost when the delay service queues the original message for the retry.
What other options am I missing? I'm happy to accept that there's a totally different solution to this problem.
Consider implementing a saga which stores that first message, holding on to it until the second message arrives. You might also want the saga to open a timeout as well so that your process won't wait indefinitely if that second message got lost or something.