Our application dequeues the message from Azure queue and makes it invisible for a given period of time.
Worker than processes it and when all work is done it deletes it from the queue.
But sometimes delete fails with 404 error not found. The problem might be that message's pop receipt has been changed.
Because when message is dequeued separate thread also runs and increases invisibility of a message to prevent it to be picked by other consumer. But it calls UpdateMessage which actually changes pop receipt.
Because UpdateMesasge and DeleteMessage might run at the same time, DeleteMessage sometimes fails because its PopReceipt is no longer valid.
Is there any way to avoid PopReceipt change on UpdateMessage?
Code sample:
TimerCallback extenderHandler = new TimerCallback(async state =>
{
try
{
var oldNextVisibleTime = queueMessage.NextVisibleTime.Value;
// extend message lease to prevent it from being picked by another worker instances
await returnMessage();
}
catch (Exception ex)
{
// NOTE: exceptions on Timer are not propagated to main thread; the error is only logged, because operation will be retried;
}
});
// start message extender timer which extends message lease time when it's nearing its max hold time, timer runs until it's disposed
using (var messageExtenderTimer = new System.Threading.Timer(extenderHandler, null, 0, (int)MessageLeaseCheckInterval.TotalMilliseconds))
{
processMessage();
}
In returnMessage method UpdateMessageAsync from Microsoft.WindowsAzure.Storage.Queue is called.
In processMessage method processing itself is done and at the end message is deleted using DeleteMessage method from Microsoft.WindowsAzure.Storage.Queue
And sometimes fails UpdateMessageAsync and sometimes DeleteMessage. Because of that I wonder that when these two concurrent threads make changes to the message - message is changed in the queue before PopReceipt is updated on message itself.
Is there any way to avoid PopReceipt change on UpdateMessage?
Unfortunately no. Whenever a message is updated, a new PopReceipt will be returned. From the documentation link (#4 item):
The message has been updated with a new visibility timeout. When the
message is updated, a new pop receipt will be returned.
Related
I have a long running process which performs matches between millions of records I call this code using a Service Bus, However when my process passes the 5 minute limit Azure starts processing the already processed records from the start again.
How can I avoid this
Here is my code:
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
long receivedMessageTrasactionId = 0;
try
{
IQueueClient queueClient = new QueueClient(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
// Process the message
receivedMessageTrasactionId = Convert.ToInt64(Encoding.UTF8.GetString(message.Body));
// My Very Long Running Method
await DataCleanse.PerformDataCleanse(receivedMessageTrasactionId);
//Get Transaction and Metric details
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
Log4NetErrorLogger(ex);
throw ex;
}
}
Messages are intended for notifications and not long running processing.
You've got a fewoptions:
Receive the message and rely on receiver's RenewLock() operation to extend the lock.
Use user-callback API and specify maximum processing time, if known, via MessageHandlerOptions.MaxAutoRenewDuration setting to auto-renew message's lock.
Record the processing started but do not complete the incoming message. Rather leverage message deferral feature, sending yourself a new delayed message with the reference to the deferred message SequenceNumber. This will allow you to periodically receive a "reminder" message to see if the work is finished. If it is, complete the deferred message by its SequenceNumber. Otherise, complete the "reminder" message along with sending a new one. This approach would require some level of your architecture redesign.
Similar to option 3, but offload processing to an external process that will report the status later. There are frameworks that can help you with that. MassTransit or NServiceBus. The latter has a sample you can download and play with.
Note that option 1 and 2 are not guaranteed as those are client-side initiated operations.
I am using the SteamBot application.
I have a command that is supposed to act when a specific event occurs. In this particular case, I, as the administrator, issue the command and the bot waits for a trade request and then continue on with my logic. What I can't figure out is how to make it wait until the trade request has occurred and then proceed.
Assume that I have issued the command and am now in the function that handles that command's behavior. This is a very simple example of what I'm trying to do.
private void giveItemsToUser()
{
Bot.GetInventory();
// Wait here
var items = Trade.CurrentSchema.GetItemsByCraftingMaterial('hat');
foreach (var item in items)
{
added += Trade.AddAllItemsByDefindex(item.Defindex, amount);
}
Trade.SetReady(true);
Trade.AcceptTrade();
}
In the // Wait here section, I want the bot to stop until the trade has started. Then, once it has started, it can proceed and add items as expected.
I've tried adding an inTrade (initialized to false) variable that is set to true in the OnTradeInit function and putting the bot to sleep until that is true.
while (!inTrade)
{
Console.WriteLine("Not in trade. Sleeping for 10 seconds.");
Thread.Sleep(10000);
}
This, however, blocks the main thread and throws it into a tight loop. When it wakes up, it immediately checks if the variable is true and if not puts it back to sleep. There isn't time for the HandleSteamMessage function in Bot.cs to handle anything - even if the bot did receive an event while sleeping.
I've tried to use an EventWaitHandle, but that never seems to fire.
EventWaitHandle waitForInTrade = new EventWaitHandle(false, EventResetMode.AutoReset, "OnTradeRequest");
waitForInTrade.WaitOne(30000);
Console.WriteLine("In trade? " + inTrade.ToString());
This pauses the main thread for 30 seconds (due to the timeout passed in WaitOne) and then continues as expected. However, inTrade is still false, even if a trade request had been received. Since I'm not in a trade at that point, and subsequent code would be invalid.
So, what is the proper way to handle this type of interaction?
I cross posted this to the /r/steambot subreddit as well. I hope that is not a problem, especially since some of the comments on Github mentioned posting here for better assistance.
Your approach is fundamentally incorrect. Steambot is event-based, so your logic needs to be based on responding to events in the event-handlers, not waiting for the events to occur.
So in other words, when the admin issues the Give Items command, you can't simply call a GiveItemsToUser() method. Instead, you'll need something more like QueueItemsToBeGivenToUser() which creates a list of items to give to the user and stores it in a private variable. Then when OnTradeInit() is called, you can check if that user has any items queued up, and if they do, add those items to the trade.
I have created a scheduled task with a master and two listener worker roles:
http://msdn.microsoft.com/en-us/library/windowsazure/hh697709.aspx
My listener project has two instances. What prevents two worker roles from getting the same message twice?
Here is my code:
public override void Run()
{
// This is a sample worker implementation. Replace with your logic.
//Trace.TraceInformation("CloudCartConnector.TaskRole entry point called", "Information");
while (true)
{
ExecuteTask();
Thread.Sleep(30000);
Trace.TraceInformation("Working", "Information");
}
}
private void ExecuteTask()
{
try
{
CloudQueueMessage message = queue.GetMessage();
if (message != null)
{
JMATask task = GetTask(message.AsString);
queue.DeleteMessage(message);
PerformTask(task);
}
}
catch (Exception ex)
{
Trace.TraceInformation("Unable to get messages: " + ex.ToString());
}
}
Getting a message (or set of messages) from a queue will make the message(s) invisible to other clients of the queue for a period of time (NextVisibleTime) and as long as you delete the message within that window of time, no other client should ever see it. Therefore it's important to make sure you complete work within the visibility timeout or you could have another worker pick up the same logical message and begin processing it again.
If you know the average amount of time it takes for a worker to process a particular type of message then you can call the GetMessage overload that takes a TimeSpan that specifies exactly how long that visibility timeout should be. The default, for the parameterless overload, is 30 seconds.
Additionally, you can dynamically update the visibility timeout as the processing of a message progresses within a worker role using UpdateMessage. This way if something might take two hours to process, but there are various steps along the way, you can update visibility timeout as you progress through the steps. This way if processing of a message failed ten minutes into the worker role, it will re-appear and can be picked up sooner.
How do you stop receiving messages from a subscription client set as a an event-driven message pump? I currently have some code that works however when I run two tests consecutively they second breaks. I'm fairly sure messages are still being pulled off the subscription from the first instance i created.
http://msdn.microsoft.com/en-us/library/dn130336.aspx
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = true; // Indicates if the message-pump should call complete on messages after the callback has completed processing.
options.MaxConcurrentCalls = 1; // Indicates the maximum number of concurrent calls to the callback the pump should initiate
options.ExceptionReceived += LogErrors; // Enables you to be notified of any errors encountered by the message pump
// Start receiveing messages
Client.OnMessage((receivedMessage) => // Initiates the message pump and callback is invoked for each message that is received. Calling Close() on the client will stop the pump.
{
// Process the message
Trace.WriteLine("Processing", receivedMessage.SequenceNumber.ToString());
}, options);
You need to do two things.
First, call subscriptionClient.Close(), which will eventually (but not immediately) stop the message pump.
Second, on your message received callback, check if the client is closed, like so:
if (subscriptionClient.IsClosed)
return;
You can call SubscriptionClient.Close() to stop further messages from being processed.
Also indicated in the comment in the code:
Calling Close() on the client will stop the pump.
I looked and see the exact same behavior. The message pump does NOT stop trying to process messages even when you close the subscription client. If your message processing handler attempts to do a .Complete or .Abandon on the message, it will throw an exception though, because the client is closed. Need a way to stop the pump guys.
I currently have a background thread. In this thread is a infinite loop.
This loop once in a while updates some values in a database, and then listens 1 second on the MessageQueue (with queue.Receive(TimeSpan.FromSeconds(1)) ).
As long as no message comes in, this call then internally throws a MessageQueueException (Timeout) which is caught and then the loop continues. If there is a message the call normally returns and the message is processed, after which the loop continues.
This leads to a lot of First chance exceptions (every second, except there is a message to process) and this spams the debug output and also breaks in the debugger when I forgot to exclude MessageQueueExceptions.
So how is the async handling of the MessageQueue meant to be done correctly, while still ensuring that, as long as my application runs, the queue is monitored and the database is updated too once in a while. Of course the thread here should not use up 100% CPU.
I just need the big picture or a hint to some correctly done async processing.
Rather than looping in a thread, I would recommend registering a delegate for the ReceiveCompleted event of your MessageQueue, as described here:
using System;
using System.Messaging;
namespace MyProject
{
///
/// Provides a container class for the example.
///
public class MyNewQueue
{
//**************************************************
// Provides an entry point into the application.
//
// This example performs asynchronous receive operation
// processing.
//**************************************************
public static void Main()
{
// Create an instance of MessageQueue. Set its formatter.
MessageQueue myQueue = new MessageQueue(".\\myQueue");
myQueue.Formatter = new XmlMessageFormatter(new Type[]
{typeof(String)});
// Add an event handler for the ReceiveCompleted event.
myQueue.ReceiveCompleted += new
ReceiveCompletedEventHandler(MyReceiveCompleted);
// Begin the asynchronous receive operation.
myQueue.BeginReceive();
// Do other work on the current thread.
return;
}
//**************************************************
// Provides an event handler for the ReceiveCompleted
// event.
//**************************************************
private static void MyReceiveCompleted(Object source,
ReceiveCompletedEventArgs asyncResult)
{
// Connect to the queue.
MessageQueue mq = (MessageQueue)source;
// End the asynchronous Receive operation.
Message m = mq.EndReceive(asyncResult.AsyncResult);
// Display message information on the screen.
Console.WriteLine("Message: " + (string)m.Body);
// Restart the asynchronous Receive operation.
mq.BeginReceive();
return;
}
}
}
Source: https://learn.microsoft.com/en-us/dotnet/api/system.messaging.messagequeue.receivecompleted?view=netframework-4.7.2
Have you considered a MessageEnumerator which is returned from the MessageQueue.GetMessageEnumerator2 ?
You get a dynamic content of the queue to examine and remove messages from a queue during the iteration.
If there are no messages then MoveNext() will return false and you don't need to catch first-chance exceptions
If there are new messages after you started iteration then they will be iterated over (if they are put after a cursor).
If there are new messages before a cursor then you can just reset an iterator or continue (if you don't need messages with lower priority at the moment).
Contrary to the comment by Jamie Dixon, the scenario IS exceptional. Note the naming of the method and its parameters: BeginReceive(TimeSpan timeout)
Had the method been named BeginTryReceive, it would've been perfectly normal if no message was received. Naming it BeginReceive (or Receive, for the sync version) implies that a message is expected to enter the queue. That the TimeSpan parameter is named timeout is also significant, because a timeout IS exceptional. A timeout means that a response was expected, but none was given, and the caller chooses to stop waiting and assumes that an error has occured. When you call BeginReceive/Receive with a 1 second timeout, you are stating that if no message has entered the queue by that time, something must have gone wrong and we need to handle it.
The way I would implement this, if I understand what you want to do correctly, is this:
Call BeginReceive either with a very large timeout, or even without a timeout if I don't see an empty queue as an error.
Attach an event handler to the ReceiveCompleted event, which 1) processes the message, and 2) calls BeginReceive again.
I would NOT use an infinite loop. This is both bad practice and completely redundant when using asynchronous methods like BeginReceive.
edit: To abandon a queue which isn't being read by any client, have the queue writers peek into the queue to determine if it is 'dead'.
edit: I have another suggestion. Since I don't know the details of your application I have no idea if it is either feasible or appropriate. It seems to me that you're basically establishing a connection between client and server, with the message queue as the communication channel. Why is this a 'connection'? Because the queue won't be written to if no one is listening. That's pretty much what a connection is, I think. Wouldn't it be more appropriate to use sockets or named pipes to transfer the messages? That way, the clients simply close the Stream objects when they are done reading, and the servers are immediately notified. As I said, I don't know if it can work for what you're doing, but it feels like a more appropriate communication channel.