We get this weird error after running our Azure worker role (Event Processor Host) for some time. Initially I thought it was due to us queueing too many cloud to device messages via ServiceClient (Microsoft.Azure.Devices) but surely we will see a "Device Queue depth cannot exceed 50 messages" ERROR. We are also very careful about closing and disposing as well.
Code below:
public async Task<AzureDevices.Message> SendCloudToDeviceAsync(string deviceId, string message, bool ack = false, string MessageId = null)
{
AzureDevices.ServiceClient ServiceClient = null;
try
{
ServiceClient = AzureDevices.ServiceClient.CreateFromConnectionString(Configuration.IOTHubConnectionString);
await ServiceClient.OpenAsync();
logger.Info(string.Format("Encoding & Sending message {0} for Device {1}", message, deviceId));
var commandMessage = new AzureDevices.Message(Encoding.ASCII.GetBytes(message))
{
//Whether we require feedback from the hub...
Ack = ack == true ? AzureDevices.DeliveryAcknowledgement.Full : AzureDevices.DeliveryAcknowledgement.None,
MessageId = MessageId == null ? Guid.NewGuid().ToString() : MessageId,
//ExpiryTimeUtc = expiry,
To = deviceId
};
await ServiceClient.SendAsync(deviceId, commandMessage);
Common.ExtensionMethods.WriteHighlightedMessage(string.Format("Sent message {0} with MessageId {1}", message, commandMessage.MessageId), ConsoleColor.Green);
logger.Info(string.Format("Sent message {0} with MessageId {1}", message, commandMessage.MessageId));
return commandMessage;
}
catch (Exception e)
{
Common.ExtensionMethods.WriteHighlightedMessage(string.Format("SendCloudToDeviceMessageAsync: {0}", e.Message), ConsoleColor.Red);
if (e.Message.Contains("Device Queue depth cannot exceed 50 messages"))
{
logger.Warn("SendCloudToDeviceMessageAsync for device {0}: {1}", deviceId, e.Message);
if (e.InnerException != null) logger.Warn(e.InnerException);
}
else
{
logger.Error("SendCloudToDeviceMessageAsync for device {0}: {1}", deviceId, e.Message);
logger.Error(e.StackTrace);
if (e.InnerException != null) logger.Error(e.InnerException);
}
}
finally
{
if(ServiceClient != null)
{
await ServiceClient.CloseAsync();
ServiceClient.Dispose();
}
}
return new AzureDevices.Message();
}
Here is part of the InnerException:
System.IO.IOException: The encryption operation failed, see inner exception. ---> System.ComponentModel.Win32Exception: The context has expired and can no longer be used
The Exact line the error is triggered is on OpenAsync().
I very well may doing this on multiple threads, is there a limit?
In a future release, we wont be doing an "application layer ACK" but rather let the decide think the server is happy via an MQTT ACK. This will drop the amount of cloud to device messages we are sending.
Device-To-Cloud commands are queued on IoT Hub and delivered asynchronously only when the device is connected. Only 50 commands can be queued per device.Please see here.Even though you have closed the service client,the command is still in the queue in service-end until the device client receive the messages completely.So i think you should make sure that the command in the queue is not over the limitation.
Related
I am new to Azure Service Bus and appreciate any help I can get.
In my current project, using c# and Azure.Messaging.ServiceBus we use an On-Premise server running a "Task Engine" windows service that listens to various queues (including MSMQ) to receive and process messages. We are migrating to Azure Service Bus Queue now.
I implemented ReceiveMessageAsync() method to read and process messages. The connection is persistent because the base class of the Task Engine service is already running in loop. While the below code works fine from my local pc (connected to VPN), it fails with the following error as soon as it's deployed to the on-premise server. The server also uses up all memory and shuts down causing other queues to terminate.
Error messages:
Azure.Messaging.ServiceBus.ServiceBusException: Creation of ReceivingAmqpLink did not complete in 30000 milliseconds. (ServiceTimeout)
Azure.Messaging.ServiceBus.ServiceBusException: 'receiver31' is closed (GeneralError)
Note:
Private Endpoint is enabled on Azure Service Bus and we use token and client credentials to connect to Azure.
All code below works fine locally when run for more than 2 hours and processes messages as soon as they are manually sent to queue using Azure Portal.
Code:
public **override **void StartUp(ContextBase context)
{
// Save the thread context
base.StartUp(context);
//Get values from Config
_tenantId = System.Configuration.ConfigurationManager.AppSettings["tenant-id"];
_clientId = System.Configuration.ConfigurationManager.AppSettings["client-id"];
_clientSecret = System.Configuration.ConfigurationManager.AppSettings["client-secret"];
_servicebusNamespace = System.Configuration.ConfigurationManager.AppSettings["servicebus-namespace"];
_messageQueueName = System.Configuration.ConfigurationManager.AppSettings["servicebus-inbound-queue"];
getAzureServiceBusAccess();
// Set the running flag
_isRunning = true;
}
//Called when service is initialized and then when Reset Connection happens due to error
private static void getAzureServiceBusAccess()
{
var _token = new ClientSecretCredential(_tenantId, _clientId, _clientSecret);
var clientOptions = new ServiceBusClientOptions()
{
TransportType = ServiceBusTransportType.AmqpWebSockets
};
_serviceBusClient = new ServiceBusClient(_servicebusNamespace, _token, clientOptions);
_serviceBusReceiver = _serviceBusClient.CreateReceiver(_messageQueueName, new ServiceBusReceiverOptions());
}
public **override **void DoAction()
{
// Make sure we haven't shut down
if (_isRunning)
{
// Wait next message
tryReceiveMessages();
}
}
private async void tryReceiveMessages()
{
try
{
ServiceBusReceivedMessage message = null;
message = await _serviceBusReceiver.ReceiveMessageAsync();
if (message != null && _isRunning)
{
try
{
string _messageBody = message.Body.ToString();
// <<Send message body to Task Adapter that adds it to the database and processes the job>>
await _serviceBusReceiver.CompleteMessageAsync(message);
}
catch (ServiceBusException s)
{
Tracer.RaiseError(Source.AzureSB, "Azure Service Bus Queue resulted in exception when processing message.", s);
throw;
}
catch (Exception ex)
{
Tracer.RaiseError(Source.AzureSB, "Unexpected error occurred moving task from Azure Service Bus to database; attempting to re-queue message.", ex);
if (message != null)
await _serviceBusReceiver.AbandonMessageAsync(message);
}
}
}
catch (ServiceBusException s)
{
tryResetConnections(s);
}
catch(Exception ex)
{
Tracer.RaiseError(Source.AzureSB, "Azure Service Bus Queue reset connection error.", ex);
throw;
}
}
private void tryResetConnections(Exception exception)
{
try
{
if (DateTime.Now.Subtract(LastQueueReset).TotalSeconds > 1800)
{
LastQueueReset = DateTime.Now;
getAzureServiceBusAccess();
}
else
{
//Send notification email to dev group
}
}
catch (Exception ex)
{
throw;
}
}
private async void closeAndDisposeConnectionAsync()
{
try
{
await _serviceBusReceiver.DisposeAsync();
}
catch(Exception ex)
{
//Do not throw and eat exception - Receiver may have been already disposed
}
try
{
await _serviceBusClient.DisposeAsync();
}
catch (Exception ex)
{
//Do not throw and eat exception - Client may have been already disposed
}
}
We tried to open the network settings on Azure Service Bus to public but that didn't resolve the issue.
I have requested the DevOps team to open ports 443, 5671 and 5672 for AMQPWebSockets and still waiting to hear back to test.
Below code I have for one of the Azure Function Service Bus topic trigger, where I am receiving the service bus messages in batch and each message I am putting into one Task.
I have below service bus settings as well,
"serviceBus": {
"prefetchCount": 0,
"messageHandlerOptions": {
"autoComplete": false,
"maxConcurrentCalls": 32,
"maxAutoRenewDuration": "00:05:00"
}
}
Things working as expected, but 2 issues I am seeing randomly,
When I am putting prefetchCount = 4 (example) I am getting warning like,
WRN] Prefetch count for receiver with Identifier ingestion-topic/Subscriptions/ingestion-sub-80c010ae-2078-4bdf-b7e2-c51672e668d6 is less than the max messages requested. When using prefetch, it isn't possible to receive more than the prefetch count in any single Receive call: PrefetchCount: 10; MaxMessages: 1000
Question - What this mean? and what's the perfect settings above different settings?
Plus randomly I m seeing below error, however messages are not dead-letters, what this error meaning and what could be potential fix here ?
function: RunAsync The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue
Can I set PrefetchCount = 0.
Thanks and appreciate!!!
[FunctionName(nameof(RunAsync))]
public async Task RunAsync([ServiceBusTrigger("%InputTopic%", "%InputSubscription%", Connection = "ServiceBusConnection", AutoCompleteMessages = false)]
ServiceBusReceivedMessage[] messages, ServiceBusMessageActions messageActions)
{
_logger.LogInformation($"Number of orders: {messages.Length}");
var taskList = new List<Task<Tuple<bool, ServiceBusReceivedMessage>>>();
foreach (var message in messages)
{
try
{
var order = message.Body.ToObjectFromJson<Order>();
//process each messages in parallel with dedicated task
taskList.Add(Task.Run(() => _messageProcessor.Process(order.ArticleNumber, message)));
//If the code execution makes it here, then you are good to go
await messageActions.CompleteMessageAsync(message);
}
catch (TimeoutException toex)
{
//Wait a couple of seconds
//Let's assume the retry fails again, so we want abandon it
//This will put the message back into the queue and increment DeliveryCount by 1
_logger.LogInformation($"A transient exception happened: {toex.Message}");
await messageActions.AbandonMessageAsync(message);
}
catch (FormatException fex)
{
if (message.DeliveryCount > 10)
{
_logger.LogInformation($"Sending message; {message.MessageId} to DLQ");
await messageActions.DeadLetterMessageAsync(message, fex.Message + " sending to DLQ");
}
else
{
_logger.LogInformation($"An format exception happened: {fex.Message}, DeliveryCount: {message.DeliveryCount}");
await messageActions.AbandonMessageAsync(message);
}
}
catch (Exception ex)
{
_logger.LogInformation($"An exception happened: {ex.Message}");
//Comment out this CompleteAsync and the message will get processed when Lock Duration is breached
await messageActions.CompleteMessageAsync(message);
}
}
//get responses for all the task
var responses = await Task.WhenAll(taskList);
//make decision to complete or DeadLetter
foreach (var (flag, message) in responses)
{
switch (flag)
{
case false:
_logger.LogError("Error processing message");
break;
default:
var order = message.Body.ToObjectFromJson<Order>();
_logger.LogInformation($"OrderID: {order.Id}, ArticleNumber: {order.ArticleNumber}, Amount: {order.Amount}, Customer First Name: {order.Customer.FirstName}, Customer Last Name: {order.Customer.LastName}");
break;
}
}
}
you can set PrefetchCount to 0 it is and optional parameter. It is
available if you want high speed and want message to be ready after
the maximum number of messages are already fetch.
That is why you are getting the warning because it seems that the
number of messages available is less that the max count and you are
setting up the prefetch count on top of it.
Regarding the error try to setup retry policy so that every time
something fails the function will try to retry the task again. This is done by setting up a parameter called maximum retry count.
Reference:
prefetch azure service bus messages.
Retry policy for function.
So I am busy writing an Watchdog for message queue installed on one of my servers. I have a application ( 3rd party ) that is listening on the queue and processing the messages. I now want to do a count on it and if message reach example 1500 I send a email. So all my code works except that I need to close the 3rd party app to use the message queue. What I do is I get all the queue names that exist. Work fine.
public void GetPrivateQueues()
{
MessageQueue[] QueueList =
MessageQueue.GetPrivateQueuesByMachine(".");
foreach (MessageQueue queueItem in QueueList)
{
i++;
myPrivateQueues.Add(queueItem.Path);
Count(queueItem.Path);
}
return;
}
So when I do the count of the queue like this
public void Count(String path)
{
MessageQueue queue = new MessageQueue(path);
MessageEnumerator messageEnumerator = queue.GetMessageEnumerator2();
int iii = 0;
while (messageEnumerator.MoveNext())
{
iii++;
}
myPrivateQueuesCount.Add(iii);
return;//i;
}
I get the error.
System.Messaging.MessageQueueException (0x80004005): Sharing violation resulted from queue being open already for exclusive receive.
How can I go about reading the queue to do a count without trying to get exclusive access on it. I just need to count it.
Thank you
Regards
I used performance Counter to read the queue.
Working like a dream now!
Added the catch exception. This is for when the queue is blank. I write a 0. Performance counter gives error on blank queue.
public void Count(String path)
{
path = path.Remove(0, 21);
try
{
PerformanceCounter queueCounter = new PerformanceCounter(
"MSMQ Queue",
"Messages in Queue",
#path);
Console.WriteLine("Queue contains {0} messages",
queueCounter.NextValue().ToString());
myPrivateQueuesCount.Add((int)queueCounter.NextValue());
}
catch (Exception exc)
{
myPrivateQueuesCount.Add(0);
}
return;
}
Have encounted a strange issue with poisoned messages in an MSMQ queue. When a poisoned message is detected I'm using the code below to handle the exception and move the message to the poison queue, but this fails because the message is not found even though I get its lookupId from the thrown exception. See relevant code below.
public bool HandleError(Exception error)
{
var poisonException = error as MsmqPoisonMessageException;
if (null == poisonException) return false;
var lookupId = poisonException.MessageLookupId;
var queuePath = Environment.MachineName + "\\" + ConfigurationManager.AppSettings["QueuePath"];
var poisonQueuePath = Environment.MachineName + "\\" + ConfigurationManager.AppSettings["PoisonQueuePath"];
var orderQueue = new System.Messaging.MessageQueue(queuePath);
var poisonMessageQueue = new System.Messaging.MessageQueue(poisonQueuePath);
// Use a new transaction scope to remove the message from the main queue and add it to the poison queue.
using (var txScope = new TransactionScope(TransactionScopeOption.RequiresNew))
{
int retryCount = 0;
while (retryCount < 3)
{
retryCount++;
try
{
// Try to get the poison message using the look up id. This line throws InvalidOperationException
var message = orderQueue.ReceiveByLookupId(lookupId);
// Send the message to the poison message queue.
poisonMessageQueue.Send(message, System.Messaging.MessageQueueTransactionType.Automatic);
txScope.Complete();
Logger.Debug("Moved poisoned message with look up id: " + lookupId + " to poison queue: " + ConfigurationManager.AppSettings["PoisonQueuePath"]);
break;
}
catch (InvalidOperationException e)
{
if (retryCount < 3)
{
Logger.Debug("Trying to move message to poison queue but message is not available, sleeping for 10 seconds before retrying", e);
Thread.Sleep(TimeSpan.FromSeconds(10));
}
else
{
Logger.Debug("Giving up on trying to move the message", e);
}
}
}
}
Logger.Info("Restarting the service to process rest of the messages in the queue");
WaitCallback restartCallback = new WaitCallback(Start);
ThreadPool.QueueUserWorkItem(restartCallback);
return true;
}
This code is basically copied from Microsoft's example code here.
The error thrown is of the correct type:
System.ServiceModel.MsmqPoisonMessageException: The transport channel detected a poison message.
But when attempting to get the message from the queue I get:
System.InvalidOperationException: Message requested was not found in the queue specified.
My first thought was that the queues might not have the correct permissions set but I've double checked that the Network Service user has all the necessary rights to read and write messages to both queues.
It's worth mentioning that this code has been working perfectly in production for months, and has survived many poisoned messages in the past. Any input on what might have caused this issue is greatly appreciated.
This will happen when you have more than one retry cycle specified. If your maxRetryCycles is greater than zero and your retryCycleDelay is greater than 30 seconds, you will see the problem you describe. The message is actually sitting in a subqueue called "retry" as it waits the retryCycleDelay between cycles. So when your IErrorHandler looks for it in the "main" queue, it won't find it. For some reason, WCF throws the MsmqPoisonMessageException at the end of each retry cycle, NOT just once at the end of all the retry cycles. Which means that your IErrorHandler will get called at the end of each cycle. Seems really strange to me but that's the way it is.
A better approach now days (if you can guarantee that your code will have MSMQ 4.0) is to change your receiveErrorHandling from "Fault" to "Move" and then get rid of your IErrorHandler. With that approach the messages will be moved for you after all the retries and retry cycles have completed. It is moved to a subqueue called "poison".
See here for more details:
Poison Message Handling in MSMQ 4.0
library: clrzmq4 (https://github.com/zeromq/clrzmq4) in a c# project.
I am using zmq router-dealer configuration. The server is written in python and runs on linux. My dealer client written in c# runs on a windows machine. It sends messages and waits from the response
public Boolean sendMessage(Dictionary<String, String> msgDict)
{
ZError err;
String errStr;
var reqFrame = new ZFrame(JsonConvert.SerializeObject(msgDict));
retval = socket.Send(reqFrame, out err);
if (err != null)
{
errStr = String.Format("Error while sending command {3} {0} {1}", err.Text, err.Number, err.Name);
return false;
}
err = null;
respFrame = socket.ReceiveFrame(out err);
if (err != null)
{
errStr = String.Format("Error while receiving response data {0} {1} {2} {3}", err.Text, err.Number, err.Name, num_messages);
return false;
}
return true;
}
I set the sendTimeout and receiveTimeout on the socket to 2 min each.
When I keep calling sendMessage, exactly at the 255th time, receiveFrame timesout . On the server I see the message being processed and response being sent like everytime. And after this point, my send also timesout with the same error "EAGAIN" Resource temporarily unavailable.
There are the things I tried
Data with different lengths from 2 KB to 20 MB
set the sendhighwatermark and receivehighwatermark to different values: 10, 1000, 10000
Tried polling on the socket instead of ReceiveFrame
Tried making the sockets completely blocking.
In each of the above cases the failure occured at exactly the 255th time. In case of blocking sockets, it got blocked at the 255th time too.
I can't use netmq as much as I would like to because it doesn't have curvezmq and the server needs it.
I also tried a dealer client from another linux machine and it had no issues 255th time or even later.