I have a job that imports files into a system. Everytime a file is imported, we create a blob in azure and we send a message with instructions to a queue so that the data is persisted in SQL accordingly. We do this using azure-webjobs and azure-webjobssdk.
We experienced an issue in which after the messages failed more than 7 times, they didn't move to the poision queue as expected. The code is the following:
Program.cs
public class Program
{
static void Main()
{
//Set up DI
var module = new CustomModule();
var kernel = new StandardKernel(module);
//Configure JobHost
var storageConnectionString = AppSettingsHelper.Get("StorageConnectionString");
var config = new JobHostConfiguration(storageConnectionString) { JobActivator = new JobActivator(kernel), NameResolver = new QueueNameResolver() };
config.Queues.MaxDequeueCount = 7;
config.UseTimers();
//Pass configuration to JobJost
var host = new JobHost(config);
host.RunAndBlock();
}
}
Functions.cs
public class Functions
{
private readonly IMessageProcessor _fileImportQueueProcessor;
public Functions(IMessageProcessor fileImportQueueProcessor)
{
_fileImportQueueProcessor = fileImportQueueProcessor;
}
public async void FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
{
await _fileImportQueueProcessor.ProcessAsync(item);
}
}
_fileImportQueueProcessor.ProcessAsync(item) threw an exception and the message's dequeue count was properly increased and re-processed. However, it was never moved to the poison-queue. I attached a screenshot of the queues with the dequeue counts at over 50.
After multiple failures the webjob was stuck in a Pending Restart state and I was unable to either stop or start and I ended up deleting it completely. After running the webjob locally, I saw messages being processed (I assumed that the one with a dequeue count of over 7 should've been moved to the poison queue).
Any ideas on why this is happening and what can be done to have the desired behavior.
Thanks,
Update
Vivien's solution below worked.
Matthew has kind enough to do a PR that will address this. You can check out the PR here.
Fred,
The FileImportQueue method being an async void is the source of your problem.
Update it to return a Task:
public class Functions
{
private readonly IMessageProcessor _fileImportQueueProcessor;
public Functions(IMessageProcessor fileImportQueueProcessor)
{
_fileImportQueueProcessor = fileImportQueueProcessor;
}
public async Task FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
{
await _fileImportQueueProcessor.ProcessAsync(item);
}
}
The reason for the dequeue count to be over 50 is because when _fileImportQueueProcessor.ProcessAsync(item) threw an exception it will crash the whole process. Meaning the WebJobs SDK can't execute the next task that will move the message to the poison queue.
When the message is available again in the queue the SDK will process it again and so on.
Related
I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.
To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.
As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
I have a MQTT calls inside a loop and in each iteration, it should return a response from the subscriber so that I could use the value being forwarded after I published. But the problem is I don't know how would I do it.
I hope you have an idea there or maybe if I'm just not implementing it right, may you guide me through this. Thanks.
Here's my code:
// MyClientMgr
class MyClientMgr{
public long CurrentOutput { get; set; }
public void GetCurrentOutput(MyObjectParameters parameters, MqttClient client)
{
MyMessageObject msg = new MyMessageObject
{
Action = MyEnum.GetOutput,
Data = JsonConvert.SerializeObject(parameters)
}
mq_GetCurrentOutput(msg, client);
}
private void mq_GetCurrentOutput(MyMessageObject msg, MqttClient client)
{
string msgStr = JsonConvert.SerializeObject(msg);
client.Publish("getOutput", Encoding.UTF8.GetBytes(msgStr),
MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
client.MqttMsgPublishReceived += (sender, e) =>{
MyObjectOutput output = JsonConvert.DeserializeObject<MyObjectOutput>(Encoding.UTF8.GetString(e.Message));
CurrentOutput = output;
};
}
}
// MyServerMgr
class MyServerMgr
{
public void InitSubscriptions()
{
mq_GetOutput();
}
private void mq_GetOutput()
{
MqttClient clientSubscribe = new MqttClient(host);
string clientId = Guid.NewGuid().ToString();
clientSubscribe.Connect(clientId);
clientSubscribe.Subscribe(new string[] { "getOutput" }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
MqttClient clientPublish = new MqttClient(host);
string clientIdPub = Guid.NewGuid().ToString();
clientPublish.Connect(clientIdPub);
clientSubscribe.MqttMsgPublishReceived += (sender, e) => {
MyMessageObj msg = JsonConvert.DeserializeObject<MyMessageObj>(Encoding.UTF8.GetString(e.Message));
var output = msg.Output;
clientPublish.Publish("getOutput", Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(output)), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
}
}
}
// MyCallerClass
class MyCallerClass
{
var host = "test.mqtt.org";
var myServer = new MyServerMgr(host);
var myClient = new MyClientMgr();
myServer.InitSubscriptions();
MqttClient client = new MqttClient(host);
for(int i = 0; i < 10; i++)
{
long output = 0;
MyObjectParameters parameters = {};
myClient.GetCurrentOutput(parameters, client) // here I call the method from my client manager
// to publish the getting of the output and assigned
// below for use, but the problem is the value doesn't
// being passed to the output variable because it is not
// yet returned by the server.
// Is there a way I could wait the process to
// get the response before assigning the output?
output = myClient.CurrentOutput; // output here will always be null
// because the response is not yet forwarded by the server
}
}
I have a loop in my caller class to call the mqtt publish for getting the output, but I have no idea how to get the output before it was assigned, I want to wait for the response first before going to the next.
I've already tried doing a while loop inside like this:
while(output == 0)
{
output = myClient.CurrentOutput;
}
Yes, I can get the output here, but it will slow down the process that much. And sometimes it will fail.
Please help me. Thanks.
It looks like you are trying to do synchronous communication over an asynchronous protocol (MQTT).
By this I mean you want to send a message and then wait for a response, this is not how MQTT works as there is no concept of a reply to a message at the protocol level.
I'm not that familiar with C# so I'll just give an abstract description of possible solution.
My suggestion would be to use a publishing thread, wait/pulse (Look at the Monitor class) to have this block after each publish and have the message handler call pulse when it has received the response.
If the response doesn't contain a wait to identify the original request you will also need a state machine variable to record which request is in progress.
You may want to look at putting a time out on the wait in case the other end does not respond for some reasons.
You can use AutoResetEvent class that has WaitOne() and Set() methods. Using WaitOne() after publish will wait until the message is published and using Set() under client_MqttMsgPublishReceived event will release the wait when the subscriber received the message he subscribed for.
I have created a azure webjob which will send a strongly typed message to a service bus queue and it successfully sends.
I want to create another webjob which should be triggered whenever there is a message in the servicebus queue. Please find below the code i am trying. For some reason, though there are messages in the servicebus queue, the webjob is not getting triggered and getting an error when i run the webjob locally.
Error:
System.InvalidOperationException
{"Missing value for trigger parameter 'blobIinfo'."}
Code:
public static void Main()
{
var config = new JobHostConfiguration
{
NameResolver = new QueueNameResolver(),
ServiceBusConnectionString = ApplicationSettings.ServiceBusConnectionString
};
var host = new JobHost(config);
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
}
[NoAutomaticTrigger]
public static void ProcessQueueMessage(
TextWriter log,
[ServiceBusTrigger("testsftppollingqueue")] SftpQueueMessage blobIinfo
)
{
while (true)
{
log.WriteLine("Queue message refers to blob: " + blobIinfo.BlobUri);
Thread.Sleep(TimeSpan.FromMinutes(PollingInterval));
}
}
Can anyone help me how to solve this?
Thanks
You have to use
host.RunAndBlock();
instead of
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
Also, please take out the NoAutomaticTrigger attribute.
I'm using RabbitMQ to provide consume items from queue. My application handled by Windows Service that most of the time running.
On service START (OnStart) I using the method
while (true)
{
var ea =(BasicDeliverEventArgs)consumer.Queue.Dequeue();
// ... Handle this item
}
In order to consume item from the queue. The function 'Dequeue()' is blocking. The Thread will be blocked in this line until some item will arrive.
My problem start when I'm trying to implement the OnStop method of my service. My target is to stop waiting for new items when OnStop signal arrived. So, I'm modified my code to something like this:
while (true)
{
if (this.IsStopping)
return; // OnStop signal arrived. Stop waiting.
var ea =(BasicDeliverEventArgs)consumer.Queue.Dequeue();
// ... Handle this item
}
In some cases, the code above is working fine. BUT, if the queue is empty, the execution of the Windows Service won't happen.
How do you recommend me to solve this problem?
you have two ways to do that, the first one is called "poison message", during the stop service you send a message to the queue:
while (true)
{
var ea =(BasicDeliverEventArgs)consumer.Queue.Dequeue();
// ... Handle this item
if (mymessage is MyPoisonMessage)
break;
}
Actually I don't i like, but it is a quick solution.
Another one is extend the class DefaultBasicConsumer and then use the consumer tag to close the consumer, something like that:
class SampleConsumer : DefaultBasicConsumer
{
public SampleConsumer(IModel channel) : base(channel)
{
}
public override void HandleBasicDeliver(string consumerTag, ulong deliveryTag, bool redelivered, string exchange, string routingKey,
IBasicProperties properties, byte[] body)
{
.....
Then instance the class, get the consumerTag and close it in this way:
channel.BasicCancel(consumerTag)
hope it helps.
At this line of code i am getting the error as i mentioned
I declared MSMQ_NAME as string as follows
private const string MSMQ_NAME = ".\\private$\\ASPNETService";
private void DoSomeMSMQStuff()
{
using (MessageQueue queue = new MessageQueue(MSMQ_NAME))
{
queue.Send(DateTime.Now); //Exception raises
queue.Close();
}
}
Can you first verify the queue is existing with the name 'ASPNETService' at below location?
Computer Management -> Services and Applications -> Message Queuing -> Private Queues
I had a similar problem. I was confused because my code worked on my local development machine, but not in production. Even stranger, the queues were created the exact same way.
It turns out that IIS doesn't have access to them by default. I just opened up the permissions.
Computer Management -> Private Queues -> right-click queue name -> Properties -> Security Tab -> click "Everyone" user -> click Full Control/Allow checkbox -> click OK
This fixed it for me, and in my case it's not an issue, but you may want to think about the ramifications of just opening it up for all users.
Also, I had to do this across all queues on all servers. There doesn't seem to be a way to multi-select queues or folders in order to set permissions for multiple queues simultaneously.
I was having the same problem.
I had created a new private queue and gave Full Permission to Everyone.
But I was still catching a "Queue does not exist or you do not have sufficient permissions to perform the operation" when trying to Send() to the queue. And I was able to verify that MessageQueue.Exists(".\\private$\\myqueue") was returning true.
Restarting the Message Queuing Service resolved my the problem for me.
I had same problem and I did like below where I check whether queue exists or not. If yes send message else create queue and then send message
MessageQueue msgQueue = null;
string queuePath = ".\\Private$\\billpay";
Payment newPayment = new Payment()
{
Payee = txtPayee.Text,
Payor = txtPayor.Text,
Amount = Convert.ToInt32(txtAmount.Text),
DueDate = dpDueDate.SelectedDate.Value.ToShortDateString()
};
Message msg = new Message();
msg.Body = newPayment;
msg.Label = "Gopala - Learning Message Queue";
if (MessageQueue.Exists(queuePath) == false)
{
//Queue doesnot exist so create it
msgQueue = MessageQueue.Create(queuePath);
}
else
{
msgQueue = new MessageQueue(queuePath);
}
msgQueue.Send(msg);
I was facing the same problem, I had resolved it using the following class to create queue
private MessageQueue messageQueue;
public const string DEFAULT_QUEUE_NAME = "newQueue";
public const string QUEUENAME_PREFIX = ".\\Private$\\";
public static string QueueName
{
get
{
string result = string.Format("{0}{1}", QUEUENAME_PREFIX, DEFAULT_QUEUE_NAME);
return result;
}
}
public void SendMessage()
{
string queuePath = QueueName;
MessageQueue messageQueue = MessageQueue.Create(queuePath);
messageQueue.Send("msg");
}
Create message queue in same manner for receiving the message.
For others struggling with this and pulling their hair out like I have been, I finally found something that works when all of the upvoted suggestions failed.
Even if you think the host name of your target queue's hosting system is being resolved correctly, don't believe it. Try replacing the host name with an IP address and see if it works. It does for me. I can WRITE to a public queue using a host name on my remote server without problems, but trying to READ from it produces exactly the error listed for this question.
For example, if I declare the following:
private static string QueueName = #"FormatName:DIRECT=TCP:SOMEHOST\MyQueue";
private static System.Messaging.MessageQueue Queue = new System.Messaging.MessageQueue(QueueName);
Where "MyQueue" is a public queue on server SOMEHOST, the following code will successfully insert messages to the queue, but always fails on the Receive():
Queue.Formatter = new XmlMessageFormatter(new Type[] { typeof(String) });
// The Receive() call here is a blocking call. We'll wait if there is no message in the queue, and processing
// is halted until there IS a message in the queue.
//
try
{
Queue.Send("hello world", System.Messaging.MessageQueueTransactionType.Single);
var msg = Queue.Receive(MessageQueueTransactionType.Single);
}
catch (Exception ex)
{
// todo error handling
}
One simple change in how I specify the queue location is all that's needed to make the Receive() stop failing with the dreaded "queue does not exist or you do not have sufficient permissions" error:
private static string QueueName = #"FormatName:DIRECT=TCP:192.168.1.100\MyQueue";
(Obviously I've obfuscated IP addresses and other sensitive info). Using the IP address is not obviously a production-worthy scenario, but it did point me to some type of name resolution problem as being the possible cause of the error. I cannot explain why Send() works but Receive() does not when I am using a host name instead of IP, but I can reproduce these results consistently. Until I can figure out what's going on with the name resolution, I'm no longer wasting a day trying to read messages from a queue.