Have encounted a strange issue with poisoned messages in an MSMQ queue. When a poisoned message is detected I'm using the code below to handle the exception and move the message to the poison queue, but this fails because the message is not found even though I get its lookupId from the thrown exception. See relevant code below.
public bool HandleError(Exception error)
{
var poisonException = error as MsmqPoisonMessageException;
if (null == poisonException) return false;
var lookupId = poisonException.MessageLookupId;
var queuePath = Environment.MachineName + "\\" + ConfigurationManager.AppSettings["QueuePath"];
var poisonQueuePath = Environment.MachineName + "\\" + ConfigurationManager.AppSettings["PoisonQueuePath"];
var orderQueue = new System.Messaging.MessageQueue(queuePath);
var poisonMessageQueue = new System.Messaging.MessageQueue(poisonQueuePath);
// Use a new transaction scope to remove the message from the main queue and add it to the poison queue.
using (var txScope = new TransactionScope(TransactionScopeOption.RequiresNew))
{
int retryCount = 0;
while (retryCount < 3)
{
retryCount++;
try
{
// Try to get the poison message using the look up id. This line throws InvalidOperationException
var message = orderQueue.ReceiveByLookupId(lookupId);
// Send the message to the poison message queue.
poisonMessageQueue.Send(message, System.Messaging.MessageQueueTransactionType.Automatic);
txScope.Complete();
Logger.Debug("Moved poisoned message with look up id: " + lookupId + " to poison queue: " + ConfigurationManager.AppSettings["PoisonQueuePath"]);
break;
}
catch (InvalidOperationException e)
{
if (retryCount < 3)
{
Logger.Debug("Trying to move message to poison queue but message is not available, sleeping for 10 seconds before retrying", e);
Thread.Sleep(TimeSpan.FromSeconds(10));
}
else
{
Logger.Debug("Giving up on trying to move the message", e);
}
}
}
}
Logger.Info("Restarting the service to process rest of the messages in the queue");
WaitCallback restartCallback = new WaitCallback(Start);
ThreadPool.QueueUserWorkItem(restartCallback);
return true;
}
This code is basically copied from Microsoft's example code here.
The error thrown is of the correct type:
System.ServiceModel.MsmqPoisonMessageException: The transport channel detected a poison message.
But when attempting to get the message from the queue I get:
System.InvalidOperationException: Message requested was not found in the queue specified.
My first thought was that the queues might not have the correct permissions set but I've double checked that the Network Service user has all the necessary rights to read and write messages to both queues.
It's worth mentioning that this code has been working perfectly in production for months, and has survived many poisoned messages in the past. Any input on what might have caused this issue is greatly appreciated.
This will happen when you have more than one retry cycle specified. If your maxRetryCycles is greater than zero and your retryCycleDelay is greater than 30 seconds, you will see the problem you describe. The message is actually sitting in a subqueue called "retry" as it waits the retryCycleDelay between cycles. So when your IErrorHandler looks for it in the "main" queue, it won't find it. For some reason, WCF throws the MsmqPoisonMessageException at the end of each retry cycle, NOT just once at the end of all the retry cycles. Which means that your IErrorHandler will get called at the end of each cycle. Seems really strange to me but that's the way it is.
A better approach now days (if you can guarantee that your code will have MSMQ 4.0) is to change your receiveErrorHandling from "Fault" to "Move" and then get rid of your IErrorHandler. With that approach the messages will be moved for you after all the retries and retry cycles have completed. It is moved to a subqueue called "poison".
See here for more details:
Poison Message Handling in MSMQ 4.0
Related
Below code I have for one of the Azure Function Service Bus topic trigger, where I am receiving the service bus messages in batch and each message I am putting into one Task.
I have below service bus settings as well,
"serviceBus": {
"prefetchCount": 0,
"messageHandlerOptions": {
"autoComplete": false,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
}
}
Things working as expected, but 2 issues I am seeing randomly,
When I am putting prefetchCount = 4 (example) I am getting warning like,
WRN] Prefetch count for receiver with Identifier ingestion-topic/Subscriptions/ingestion-sub-80c010ae-2078-4bdf-b7e2-c51672e668d6 is less than the max messages requested. When using prefetch, it isn't possible to receive more than the prefetch count in any single Receive call: PrefetchCount: 10; MaxMessages: 1000
Question - What this mean? and what's the perfect settings above different settings?
Plus randomly I m seeing below error, however messages are not dead-letters, what this error meaning and what could be potential fix here ?
function: RunAsync The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue
Can I set PrefetchCount = 0.
Thanks and appreciate!!!
[FunctionName(nameof(RunAsync))]
public async Task RunAsync([ServiceBusTrigger("%InputTopic%", "%InputSubscription%", Connection = "ServiceBusConnection", AutoCompleteMessages = false)]
ServiceBusReceivedMessage[] messages, ServiceBusMessageActions messageActions)
{
_logger.LogInformation($"Number of orders: {messages.Length}");
var taskList = new List<Task<Tuple<bool, ServiceBusReceivedMessage>>>();
foreach (var message in messages)
{
try
{
var order = message.Body.ToObjectFromJson<Order>();
//process each messages in parallel with dedicated task
taskList.Add(Task.Run(() => _messageProcessor.Process(order.ArticleNumber, message)));
//If the code execution makes it here, then you are good to go
await messageActions.CompleteMessageAsync(message);
}
catch (TimeoutException toex)
{
//Wait a couple of seconds
//Let's assume the retry fails again, so we want abandon it
//This will put the message back into the queue and increment DeliveryCount by 1
_logger.LogInformation($"A transient exception happened: {toex.Message}");
await messageActions.AbandonMessageAsync(message);
}
catch (FormatException fex)
{
if (message.DeliveryCount > 10)
{
_logger.LogInformation($"Sending message; {message.MessageId} to DLQ");
await messageActions.DeadLetterMessageAsync(message, fex.Message + " sending to DLQ");
}
else
{
_logger.LogInformation($"An format exception happened: {fex.Message}, DeliveryCount: {message.DeliveryCount}");
await messageActions.AbandonMessageAsync(message);
}
}
catch (Exception ex)
{
_logger.LogInformation($"An exception happened: {ex.Message}");
//Comment out this CompleteAsync and the message will get processed when Lock Duration is breached
await messageActions.CompleteMessageAsync(message);
}
}
//get responses for all the task
var responses = await Task.WhenAll(taskList);
//make decision to complete or DeadLetter
foreach (var (flag, message) in responses)
{
switch (flag)
{
case false:
_logger.LogError("Error processing message");
break;
default:
var order = message.Body.ToObjectFromJson<Order>();
_logger.LogInformation($"OrderID: {order.Id}, ArticleNumber: {order.ArticleNumber}, Amount: {order.Amount}, Customer First Name: {order.Customer.FirstName}, Customer Last Name: {order.Customer.LastName}");
break;
}
}
}
you can set PrefetchCount to 0 it is and optional parameter. It is
available if you want high speed and want message to be ready after
the maximum number of messages are already fetch.
That is why you are getting the warning because it seems that the
number of messages available is less that the max count and you are
setting up the prefetch count on top of it.
Regarding the error try to setup retry policy so that every time
something fails the function will try to retry the task again. This is done by setting up a parameter called maximum retry count.
Reference:
prefetch azure service bus messages.
Retry policy for function.
So I am busy writing an Watchdog for message queue installed on one of my servers. I have a application ( 3rd party ) that is listening on the queue and processing the messages. I now want to do a count on it and if message reach example 1500 I send a email. So all my code works except that I need to close the 3rd party app to use the message queue. What I do is I get all the queue names that exist. Work fine.
public void GetPrivateQueues()
{
MessageQueue[] QueueList =
MessageQueue.GetPrivateQueuesByMachine(".");
foreach (MessageQueue queueItem in QueueList)
{
i++;
myPrivateQueues.Add(queueItem.Path);
Count(queueItem.Path);
}
return;
}
So when I do the count of the queue like this
public void Count(String path)
{
MessageQueue queue = new MessageQueue(path);
MessageEnumerator messageEnumerator = queue.GetMessageEnumerator2();
int iii = 0;
while (messageEnumerator.MoveNext())
{
iii++;
}
myPrivateQueuesCount.Add(iii);
return;//i;
}
I get the error.
System.Messaging.MessageQueueException (0x80004005): Sharing violation resulted from queue being open already for exclusive receive.
How can I go about reading the queue to do a count without trying to get exclusive access on it. I just need to count it.
Thank you
Regards
I used performance Counter to read the queue.
Working like a dream now!
Added the catch exception. This is for when the queue is blank. I write a 0. Performance counter gives error on blank queue.
public void Count(String path)
{
path = path.Remove(0, 21);
try
{
PerformanceCounter queueCounter = new PerformanceCounter(
"MSMQ Queue",
"Messages in Queue",
#path);
Console.WriteLine("Queue contains {0} messages",
queueCounter.NextValue().ToString());
myPrivateQueuesCount.Add((int)queueCounter.NextValue());
}
catch (Exception exc)
{
myPrivateQueuesCount.Add(0);
}
return;
}
I have a legacy Windows service running on Server 2008 that reads messages from a Transactional MSMQ Queue. This is not configured as a WCF service.
We are wanting to improve the handling of failed and poison messages in code (C# 4.0) by catching custom exceptions and sending the related message to a separate 'failed' or 'poison' queue depending upon the type of exception thrown.
I can't get the Catch code to send the message to the separate queue - it disappears from the original queue (as desired!) but doesn't show up in the 'failed' queue.
For testing all of the queues have no Authentication required and permissions are set to allow Everyone to do everything.
Clearly something is missing or wrong and I suspect it is transaction related, but I can't see it. Or perhaps this is not possible the way I am trying to do it ?
Any guidance / suggestions appreciated!
Our simplified PeekCompleted Event code:
private void MessageReceived(object sender, PeekCompletedEventArgs e)
{
using (TransactionScope txnScope = new TransactionScope())
{
MyMessageType currentMessage = null;
MessageQueue q = ((MessageQueue)sender);
try
{
Message queueMessage = q.EndPeek(e.AsyncResult);
currentMessage = (FormMessage)queueMessage.Body;
Processor distributor = new Processor();
Processor.Process(currentMessage); // this will throw if need be
q.ReceiveById(e.Message.Id);
txnScope.Complete();
q.BeginPeek();
}
catch (MyCustomException ex)
{
string qname = ".\private$\failed";
if (!MessageQueue.Exists(qname)){
MessageQueue.Create(qname , true);
}
MessageQueue fq = new MessageQueue(qname){
Formatter = new BinaryMessageFormatter()
};
System.Messaging.Message message2 = new System.Messaging.Message{
Formatter = new BinaryMessageFormatter(),
Body = currentMessage,
Label = "My Failed Message";
};
fq.Send(message2); //send to failed queue
q.ReceiveById(e.Message.Id); //off of original queue
txnScope.Complete(); // complete the trx
_queue.BeginPeek(); // next or wait
}
//other catches handle any cases where we want to tnxScope.Dispose()
EDIT : October 8, 2013
Hugh's answer below got us on the right track. Inside the Catch block the Failed Queue was already created as transactional
MessageQueue.Create(qname , true);
but the Send needed a TransactionType parameter
fq.Send(message2,MessageQueueTransactionType.Single);
That did the trick. Thanks Hugh!
If the message is disappearing from your original queue then that means your code is reaching the second scope.Complete() (in your catch block).
This means the problem has to do with your send to the error queue.
I would suggest that you need to create the queue as transactional because you are sending from within a scope.
MessageQueue fq = new MessageQueue(qname, true){
Formatter = new BinaryMessageFormatter()
};
Then you need to do a transactional send:
fq.Send(message2, Transaction.Current);
See if this works.
We have a service that receives messages from n message queues. However, if the Message Queuing service is restarted, the message retrieval service stops receiving messages even after the Message Queuing service has restarted successfully.
I have tried to specifically catch the MessageQueueException that is thrown in the message retrieval service and invoke the queue's BeginReceive method again. However, in the 2 seconds or so that it takes the Message Queuing service to restart, I get about 1875 instances of the exception and then the service stops functioning when another MessageQueueException is thrown in our StartListening method.
Is there an elegant way to recover from a Message Queuing service restart?
private void OnReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
try
{
Message message = queue.EndReceive(e.AsyncResult);
this.StartListening(queue);
if (this.MessageReceived != null)
this.MessageReceived(this, new MessageReceivedEventArgs(message));
}
catch (MessageQueueException)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, StringResource.LogMessage_QueueManager_MessageQueueException, queue.MachineName, queue.QueueName, queue.Path));
this.StartListening(queue);
}
}
public void StartListening(MessageQueue queue)
{
queue.BeginReceive();
}
I need to deal with the infinite loop issue this causes and clean it up a bit but you get the idea.
When the MessageQueueException occurs, invoke the RecoverQueue method.
private void RecoverQueue(MessageQueue queue)
{
string queuePath = queue.Path;
bool queueRecovered = false;
while (!queueRecovered)
{
try
{
this.StopListening(queue);
queue.Close();
queue.Dispose();
Thread.Sleep(2000);
MessageQueue newQueue = this.CreateQueue(queuePath);
newQueue.ReceiveCompleted += new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
this.StartListening(newQueue);
LogUtility.LogInformation(String.Format(CultureInfo.InvariantCulture, "Message queue {0} recovered successfully.", newQueue.QueueName));
queueRecovered = true;
}
catch (Exception ex)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, "The following error occurred while trying to recover queue: {0} error: {1}", queue.QueueName, ex.Message));
}
}
}
public void StopListening(MessageQueue queue)
{
queue.ReceiveCompleted -= new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
}
Upon receiving the exception that is the result of the service restarting, you have to release the old MessageQueue, i.e. unwiring your ReceiveCompleted event, disposing the MessageQueue, etc. Then create a new instance of the MessageQueue and hook up to the ReceiveCompleted event again on the new MessageQueue instance.
Alternatively, you can use a polling method that creates a new instance on a certain interval, calls MessageQueue.Receive(TimeSpan), will wait for an incoming message or until the timeout occurs. In which case you handle the message and destroy the MessageQueue instance and start the iteration again.
By recreating the MessageQueue each time, you ensure a built in recovery. Also, the overhead of creating the MessageQueue is minimal due to internal caching of the underlying queue.
Pseudo-code...
while (!notDone)// or use a timer or periodic task of some sort...
{
try
{
using (MessageQueue queue = new MessageQueue(queuePath))
{
Message message = queue.Receive(TimeSpan.FromMilliseconds(500));
// process message
}
}
catch (MessageQueueException ex)
{
// handle exceptions
}
}
I have inherited a windows service that processes a large number of e-mails in a queue. Sounds simple, Grab queue, send e-mail, if SmtpClient.SendAsync does not return an error from the call back then flag the e-mail in the DB as being sent.. I am using a Semaphore to waitone on the thread so multiple calls can be made to the Async Send method of the SMTP Client. This is the only way I can get the status and per Microsoft docs it has to finish the operation before another call can be made async. So now for the fun part. I decided to use a Parallel.ForEach to get he queue like so. This method is called in the Windows Service OnStart. Please note I have tried calling this method on a separate Thread and get the same results.
I am thinking that either A, I am missing something obvious, due to my lack of knowledge on threading, or something is flat bugged. Most likely A.
private static void ProcessEmailQueue()
{
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
Parallel.ForEach(emailQueue, message =>
{
_smtpMail.FromAddress = message.FromAddress;
_smtpMail.ToAddress = message.ToAddress;
_smtpMail.Subject = message.Subject;
_smtpMail.SendAsHtml = message.IsHtml > 0;
_smtpMail.MessageBody = message.MessageBody;
_smtpMail.UserToken = message.EmailQueueID;
bool sendStatus = _smtpMail.SendMessage();
// THIS BLOWS UP with InvalidOperation Exception
});
}
Here is the SMTP Method being called from withing the loop.
public bool SendMessage()
{
mailSendSemaphore = new Semaphore(0, 10); // This is defined as private static Semaphore mailSendSemaphore;
try
{
var fromAddress = new MailAddress(FromAddress);
var toAddress = new MailAddress(ToAddress);
using (var mailMessage = new MailMessage(fromAddress, toAddress))
{
mailMessage.Subject = Subject;
mailMessage.IsBodyHtml = SendAsHtml;
mailMessage.Body = MessageBody;
Envelope = mailMessage;
smtp.SendCompleted += smtp_SendCompleted;
smtp.SendAsync(mailMessage, UserToken);
mailSendSemaphore.WaitOne();
return _mailSent;
}
}
catch (Exception exception)
{
_logger.Error(exception);
return _mailSent;
}
}
CALLBACK For Smtp Send
private void smtp_SendCompleted(object sender, AsyncCompletedEventArgs e)
{
if (e.Cancelled)
{
}
if (e.Error != null)
{
}
else
{
_mailSent = true;
}
mailSendSemaphore.Release(2);
}
Here is the Exception, took a few to get it for some odd reason.
System.InvalidOperationException was unhandled by user code
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
Source=System
StackTrace:
at System.Net.Mail.SmtpClient.SendAsync(MailMessage message, Object userToken)
at DFW.Infrastructure.Communications.SmtpMail.SendMessage() in SmtpMail.cs:line 71
at EmaiProcessorService.EmailQueueService.b_0(EmailQueue message) in Service1.cs:line 57
at System.Threading.Tasks.Parallel.<>c_DisplayClass2d2.<ForEachWorker>b__23(Int32 i)
at System.Threading.Tasks.Parallel.<>c__DisplayClassf1.b__c()
InnerException:
Seems my waitone is getting obliterated by System.Threading.Tasks.Parallel
Okay, now that we've got the error text, it seems fairly clear:
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
This concurs with the documentation:
Two simple options:
Create a fixed number of clients, and a queue of messages to send. Make each client take a message from the queue each time it finishes, until the queue is empty. BlockingCollection<T> is good for this.
Create a new SmtpClient per message. This could cause you to effectively launch a DOS attack on your SMTP server, which isn't ideal.
To be honest, it's not really clear why you're using SendAsync when you're then just waiting for the message to be sent anyway...
I'm not clear on why you're using a Semaphore here, but you're almost certainly using it incorrectly. You're creating a new semaphore instance for each call to SendMessage. Also, you're calling WaitOne on it once, and then calling Release(2), so eventually you'll have more releases than acquires. That's probably what causes your InvalidOperationException.
It doesn't do you any good to parallelize processing of the email queue, since you can only send one message at a time. And trying to do it asynchronously inside of the Parallel.Foreach is just more needless complication.
You're better off using something like ThreadPool.QueueUserWorkItem, and having a simple loop that sends one message at a time.
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
ThreadPool.QueueUserWorkItem(ProcessEmailQueue, emailQueue);
void ProcessEmailQueue(object state)
{
List<EmailQueue> emailQueue = (List<EmailQueue>)state;
foreach (var message in EmailQueue)
{
// Format and send message here.
}
}
Alternatively, you can do the same thing with a Task. The point is that you just need a single thread to process the queue sequentially. Since you can't send more than one message at a time, Parallel.ForEach doesn't do you any good.
EDIT:
If you need to do multiple sends at a time, you can probably modify your original code. First, initialize the semaphore at class scope:
private static Semaphore mailSendSemaphore = new Semaphore(10, 10);
Then, in your SendMessage method:
bool SendMessage()
{
// acquire semaphore. This will block until there's a slot available.
mailSendSemaphore.WaitOne();
try
{
// do all your processing here, including sending the message.
// use Send rather than SendAsync
}
finally
{
mailSendSemaphore.Release();
}
}
There's no need to use SendAsync.