Develop Reference

Pop receipt changed on update message

Our application dequeues the message from Azure queue and makes it invisible for a given period of time.
Worker than processes it and when all work is done it deletes it from the queue.
But sometimes delete fails with 404 error not found. The problem might be that message's pop receipt has been changed.
Because when message is dequeued separate thread also runs and increases invisibility of a message to prevent it to be picked by other consumer. But it calls UpdateMessage which actually changes pop receipt.
Because UpdateMesasge and DeleteMessage might run at the same time, DeleteMessage sometimes fails because its PopReceipt is no longer valid.
Is there any way to avoid PopReceipt change on UpdateMessage?
Code sample:
TimerCallback extenderHandler = new TimerCallback(async state =>
{
try
{
var oldNextVisibleTime = queueMessage.NextVisibleTime.Value;
// extend message lease to prevent it from being picked by another worker instances
await returnMessage();
}
catch (Exception ex)
{
// NOTE: exceptions on Timer are not propagated to main thread; the error is only logged, because operation will be retried;
}
});
// start message extender timer which extends message lease time when it's nearing its max hold time, timer runs until it's disposed
using (var messageExtenderTimer = new System.Threading.Timer(extenderHandler, null, 0, (int)MessageLeaseCheckInterval.TotalMilliseconds))
{
processMessage();
}
In returnMessage method UpdateMessageAsync from Microsoft.WindowsAzure.Storage.Queue is called.
In processMessage method processing itself is done and at the end message is deleted using DeleteMessage method from Microsoft.WindowsAzure.Storage.Queue
And sometimes fails UpdateMessageAsync and sometimes DeleteMessage. Because of that I wonder that when these two concurrent threads make changes to the message - message is changed in the queue before PopReceipt is updated on message itself.

Is there any way to avoid PopReceipt change on UpdateMessage?
Unfortunately no. Whenever a message is updated, a new PopReceipt will be returned. From the documentation link (#4 item):
The message has been updated with a new visibility timeout. When the
message is updated, a new pop receipt will be returned.

Wait for RabbitMQ Threads to finish in Windows Service OnStop()

I am working on a windows service written in C# (.NET 4.5, VS2012), which uses RabbitMQ (receiving messages by subscription). There is a class which derives from DefaultBasicConsumer, and in this class are two actual consumers (so two channels). Because there are two channels, two threads handle incoming messages (from two different queues/routing keys) and both call the same HandleBasicDeliver(...) function.
Now, when the windows service OnStop() is called (when someone is stopping the service), I want to let both those threads finish handling their messages (if they are currently processing a message), sending the ack to the server, and then stop the service (abort the threads and so on).
I have thought of multiple solutions, but none of them seem to be really good. Here's what I tried:
using one mutex; each thread tries to take it when entering HandleBasicDeliver, then releases it afterwards. When OnStop() is called, the main thread tries to grab the same mutex, effectively preventing the RabbitMQ threads to actually process any more messages. The disadvantage is, only one consumer thread can process a message at a time.
using two mutexes: each RabbitMQ thread has uses a different mutex, so they won't block each other in the HandleBasicDeliver() - I can differentiate which
thread is actually handling the current message based on the routing key. Something like:
HandleBasicDeliver(...)
{
if(routingKey == firstConsumerRoutingKey)
{
// Try to grab the mutex of the first consumer
}
else
{
// Try to grab the mutex of the second consumer
}
}
When OnStop() is called, the main thread will try to grab both mutexes; once both mutexes are "in the hands" of the main thread, it can proceed with stopping the service. The problem: if another consumer would be added to this class, I'd need to change a lot of code.
using a counter, or CountdownEvent. Counter starts off at 0, and each time HandleBasicDeliver() is entered, counter is safely incremented using the Interlocked class. After the message is processed, counter is decremented. When OnStop() is called, the main thread checks if the counter is 0. Should this condition be fulfilled, it will continue. However, after it checks if counter is 0, some RabbitMQ thread might begin to process a message.
When OnStop() is called, closing the connection to the RabbitMQ (to make sure no new messages will arrive), and then waiting a few seconds ( in case there are any messages being processed, to finish processing) before closing the application. The problem is, the exact number of seconds I should wait before shutting down the apllication is unknown, so this isn't an elegant or exact solution.
I realize the design does not conform to the Single Responsibility Principle, and that may contribute to the lack of solutions. However, could there be a good solution to this problem without having to redesign the project?

We do this in our application, The main idea is to use a CancellationTokenSource
On your windows service add this:
private static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();
Then in your rabbit consumers do this:
1. change from using Dequeue to DequeueNoWait
2. have your rabbit consumer check the cancellation token
Here is our code:
public async Task StartConsuming(IMessageBusConsumer consumer, MessageBusConsumerName fullConsumerName, CancellationToken cancellationToken)
{
var queueName = GetQueueName(consumer.MessageBusConsumerEnum);
using (var model = _rabbitConnection.CreateModel())
{
// Configure the Quality of service for the model. Below is how what each setting means.
// BasicQos(0="Don't send me a new message until I’ve finished", _fetchSize = "Send me N messages at a time", false ="Apply to this Model only")
model.BasicQos(0, consumer.FetchCount.Value, false);
var queueingConsumer = new QueueingBasicConsumer(model);
model.BasicConsume(queueName, false, fullConsumerName, queueingConsumer);
var queueEmpty = new BasicDeliverEventArgs(); //This is what gets returned if nothing in the queue is found.
while (!cancellationToken.IsCancellationRequested)
{
var deliverEventArgs = queueingConsumer.Queue.DequeueNoWait(queueEmpty);
if (deliverEventArgs == queueEmpty)
{
// This 100ms wait allows the processor to go do other work.
// No sense in going back to an empty queue immediately.
// CancellationToken intentionally not used!
// ReSharper disable once MethodSupportsCancellation
await Task.Delay(100);
continue;
}
//DO YOUR WORK HERE!
}
}

Usually, how we ensure a windows service not stop before processing completes is to use some code like below. Hope that help.
protected override void OnStart(string[] args)
{
// start the worker thread
_workerThread = new Thread(WorkMethod)
{
// !!!set to foreground to block windows service be stopped
// until thread is exited when all pending tasks complete
IsBackground = false
};
_workerThread.Start();
}
protected override void OnStop()
{
// notify the worker thread to stop accepting new migration requests
// and exit when all tasks are completed
// some code to notify worker thread to stop accepting new tasks internally
// wait for worker thread to stop
_workerThread.Join();
}

Why thread enters WaitSleepJoin state?

I'm coding singleton class that runs a background thread. Here is how it started and maintained:
private void EnsureBackgroundThread()
{
try
{
if (this.RunnerThread == null)
this.RunnerThread = new Thread(this.BackgroundRunner) { IsBackground = true };
if (this.RunnerThread.ThreadState != ThreadState.Running)
{
Debug.WriteLine("----ApplePushNotificatorService.EnsureBackgroundThread ThreadState: " + this.RunnerThread.ThreadState);
this.RunnerThread.Start();
}
}
catch (Exception ex)
{
this.LoggerService.Log(null, ex);
}
}
I call my method in this class from TestClass like so:
apns.Send("dev", devices, "Testing...", out badIds);
// Wait 5 seconds to let it send stuff through.
Thread.Sleep(5000);
devices.Clear();
devices.Add("some bad id");
// Now let's call this again, but this time we should get back some bad Ids
apns.Send("dev", devices, "Testing...", out badIds);
// Wait 5 seconds to let it send stuff through.
Thread.Sleep(5000);
devices.Clear();
devices.Add("9edc21d0d4e369f50040c5d2c94f2ea29c7d596090e4ddae253712cd406391df");
apns.Send("dev", devices, "Test message for Andrew's phone", out badIds);
// Wait 5 seconds to let it send stuff through.
Thread.Sleep(5000);
I checked error logs and I see exceptions:
Thread is running or terminated; it cannot restart.
In debug it says:
----ApplePushNotificatorService.EnsureBackgroundThread ThreadState: Background, WaitSleepJoin
Why does it enter "WaitSleepJoin" state? Is it because I do "Thread.Sleep" in my test?
Does my code to keep thread alive look correct? How do I work around this? The idea is when "Send" method called on singleton - we need to make sure background thread is running.
EDIT:
This is re-worked code that is working properly
private void EnsureBackgroundThread()
{
try
{
if (this.RunnerThread != null && this.RunnerThread.IsAlive) return;
this.RunnerThread = new Thread(this.BackgroundRunner) { IsBackground = true };
this.RunnerThread.Start();
}
catch (Exception ex)
{
this.LoggerService.Log(null, ex);
}
}

The state tells us that the thread is currently sleeping, likely in one of your calls to Sleep. This means it's still running. Because it's still running, you cannot start it. You're trying to start the same thread multiple times. You can't do that. You start it once, then it's started, and that's that. Trying to start it a second time, either while it's running, or after it is done, is not possible.

Quite simply, as the error states: your thread has been created and is running or terminated - and then you try to start it again.
Presumably in your TestClass you have multiple calls to your singleton class (I'm guessing this may be somewhere under the apns.Send call). The first time you call EnsureBackgroundThread, a single thread will be created and started. The next time you call EnsureBackgroundThread will call Thread.Start on the same thread, thus causing the same error.
It's perhaps important to note here that, when a thread completes, the variable referencing it isn't set to null - but more likely you're just calling the EnsureBackgroundThread method more than once and the code you've written doesn't support that.

Usage multithreading could lead to excessive memory use

I'm having a windows service project that logs messages to a database (or other place). The frequency of these messages could go up to ten per second. Since sending and processing the messages shouldn't delay the main process of the service I start a new thread for the processing of every message. This means that if the main process needs to send 100 log messages, 100 threads are started that process each message. I learned that when a thread is done, it will be cleaned so I don't have to dispose it. As long as I dispose all used objects in the thread everything should be working fine.
The service could go into a exception that leads to shutting down the service. Before the service shuts down it should wait for all threads that were logging messages. To achieve this it adds the thread to a list every time a thread is started. When the wait-for-threads method is called, all threads in the list are checked if it is still alive and if so, it uses join to wait for it.
The code:
Creating the thread:
/// <summary>
/// Creates a new thread and sends the message
/// </summary>
/// <param name="logMessage"></param>
private static void ThreadSend(IMessage logMessage)
{
ParameterizedThreadStart threadStart = new ParameterizedThreadStart(MessageHandler.HandleMessage);
Thread messageThread = new Thread(threadStart);
messageThread.Name = "LogMessageThread";
messageThread.Start(logMessage);
threads.Add(messageThread);
}
The waiting for threads to end:
/// <summary>
/// Waits for threads that are still being processed
/// </summary>
public static void WaitForThreads()
{
int i = 0;
foreach (Thread thread in threads)
{
i++;
if (thread.IsAlive)
{
Debug.Print("waiting for {0} - {1} to end...", thread.Name, i);
thread.Join();
}
}
}
Now my main concern is if this service runs for a month it will still have all threads (millions) in the list (most of them dead). This will eat memory and I don't know how much. This in whole doesn't seem to be a good practice to me, I want to clean up finished threads but I can't find out how to do it. Does any one have a good or best practice for this?

Remove the threads from the list if they are dead?
/// <summary>
/// Waits for threads that are still being processed
/// </summary>
public static void WaitForThreads()
{
List<Thread> toRemove = new List<int>();
int i = 0;
foreach (Thread thread in threads)
{
i++;
if (thread.IsAlive)
{
Debug.Print("waiting for {0} - {1} to end...", thread.Name, i);
thread.Join();
}
else
{
toRemove.Add(thread);
}
}
threads.RemoveAll(x => toRemove.Contains(x));
}
Have a look at Task Parallelism

First of all: Creating one thread per log message is not a good idea. Either use ThreadPool or create a limited number of worker threads which handle the log items from a common queue (producer/consumer).
Second: Of course you need to also remove the thread references from the list! Either when the thread method ends, it can remove itself, or you can even do it on a regular basis. For example, have a timer run every half and hour that checks the list for dead threads and removes them.

If all you're doing in those threads is logging, you should probably have a single logging thread and a shared queue that the main thread puts messages on. The logging thread can then read the queue and log. This is incredibly easy with the BlockingCollection.
Create the queue in the service's main thread:
BlockingCollection<IMessage> LogMessageQueue = new BlockingCollection<IMessage>();
Your service's main thread creates a Logger (see below) instance, which starts a thread to process log messages. The main thread adds items to the LogMessageQueue. The logger thread reads them from the queue. When the main thread wants to shut down, it calls LogMessageQueue.CompleteAdding. The logger will empty the queue and exit.
Main thread would look like this:
// start the logger
Logger _loggingThread = new Logger(LogMessageQueue);
// to log a message:
LogMessageQueue.Add(logMessage);
// when the program needs to shut down:
LogMessageQueue.CompleteAdding();
And the logger class:
class Logger
{
BlockingCollection<IMessage> _queue;
Thread _loggingThread;
public Logger(BlockingCollection<IMessage> queue)
{
_queue = queue;
_loggingThread = new Thread(LoggingThreadProc);
}
private void LoggingThreadProc(object state)
{
IMessage msg;
while (_queue.TryTake(out msg, TimeSpan.Infinite))
{
// log the item
}
}
}
This way you have just one additional thread, messages are guaranteed to be processed in the order they're sent (not true of your current approach), and you don't have to worry about keeping track of thread shutdown, etc.
Update
If some of your log messages will take time to process (the email you described, for example), you can process them asynchronously. For example:
while (_queue.TryTake(out msg, TimeSpan.Infinite))
{
if (msg.Type == Email)
{
// start asynchronous task to send email
}
else
{
// write to log file
}
}
This way, only those messages that potentially take lots of time will run asynchronously. You can also have a secondary queue there if you want, for the email messages. That way you won't get bogged down with a bunch of email threads. Rather, you limit it to one or two, or perhaps a handful.
Note that you can also have multiple Logger instances if you want, all reading from the same message queue. Just make sure they're each writing to a different log file. The queue itself will support multiple consumers.

I think in general the approach to solve your issue is maybe not the best practice.
I mean, instead of creating 1000s of threads, you just want to store 1000s of messages in a database right? And it seems you want to do this asynchronously.
But creating a thread for each message is not really a good idea and actually does not solve that issue...
Instead I would try to implement something like message queues. You can have multiple queues and each queue has its own thread. If messages are coming in, you send them to one of the queues (alternating)...
The queue either waits for a certain amount of messages, or always waits a certain amount of time (e.g. 1 second, depends of how long it takes to store e.g. 100 messages within the database) until it tries to store the queued messages in the database.
This way you should actually always have a constant number of threads and you shouldn't see any performance issues...
Also it would enable you to batch insert data and not only one by one with the overhead of db connections etc...
Of cause, if your database is slower then the tasks are able to store the messages, more and more messages will be queued... But that's true for your current solution, also.

Since multiple answers and comments led to my solution I will post the complete code here.
I used threadpool to manage the threads and code from this page for the wating function.
Creating the thread:
private static void ThreadSend(IMessage logMessage)
{
ThreadPool.QueueUserWorkItem(MessageHandler.HandleMessage, logMessage);
}
Waiting for the threads to finish:
public static bool WaitForThreads(int maxWaitingTime)
{
int maxThreads = 0;
int placeHolder = 0;
int availableThreads = 0;
while (maxWaitingTime > 0)
{
System.Threading.ThreadPool.GetMaxThreads(out maxThreads, out placeHolder);
System.Threading.ThreadPool.GetAvailableThreads(out availableThreads, out placeHolder);
//Stop if all threads are available
if (availableThreads == maxThreads)
{
return true;
}
System.Threading.Thread.Sleep(TimeSpan.FromMilliseconds(1000));
--maxWaitingTime;
}
return false;
}
Optionally you can add this somewhere outside these methods to limit the amount of threads in the pool.
System.Threading.ThreadPool.SetMaxThreads(MaxWorkerThreads, MaxCompletionPortThreads);

Processing Messages from a Queue on a Delay using C# and VS.NET 2005

This is the setup for the queue:
The queue is public and marked as transactional.
A web service is responsible for inserting messages into the queue.
A Windows service is listening in on the queue
Given that, how can I only process messages that are at least 20 minutes old?

Given that a queue is first-in, first-out, you have to assume that all messages that are in a queue arrive at a time equal to or later than the time than the first message in the queue.
You can use that to your advantage.
Basically, you would call the Peek method on your MessageQueue instance and look at the ArrivedTime property. If the time between now, and the arrived time is greater than 20 minutes, then you would process it, otherwise, you would continue to wait (until the next time you process the messages on the queue).
Here is an example:
static Message GetQueueMessage(MessageQueue queue)
{
// Store the current time.
DateTime now = DateTime.Now;
// Get the message on top of the queue.
Message message = queue.Peek();
// If the current time is greater than or equal to 20 minutes, then process it,
// otherwise, get out and return false. Generate 20 minutes first.
// You can refactor this to have it passed in.
TimeSpan delay = TimeSpan.FromMinutes(20);
// If the delay is greater than the arrived time and now, then get out.
if (delay > (now - message.ArrivedTime))
{
// Return null.
return null;
}
// Pop the message from the queue to remove it.
return queue.ReceiveById(message.Id);
}
You can then run this in a loop, waiting for a bit in between (every five seconds, ten, whatever you feel is appropriate for the frequency of processing):
// Have some other reasonable check here.
do while (true)
{
// The message to process.
Message message = GetQueueMessage(queue);
// Continue to process while there is a message.
do while (message != null)
{
// Process the message.
// Get the next message.
message = GetQueueMessage(queue);
}
// Wait a little bit, five seconds for example.
Thread.Sleep(5000);
}
Ideally, based on Charles Conway's comment, you would not process this in a loop, but rather, just call GetQueueMessage from an event handler on a timer which would fire on your designated interval.

// Have some other reasonable check here.
do while (true)
{
...
}
Having "true" in a boolean condition is a code smell... Have you considered using a timer the wakes every 5 seconds?
Timer timer = new Timer();
timer.Elapsed +=new ElapsedEventHandler(timer_Elapsed);
static void timer_Elapsed(object sender, ElapsedEventArgs e)
{
// The message to process.
Message message = GetQueueMessage(queue);
// Continue to process while there is a message.
do while (message != null)
{
// Process the message.
// Get the next message.
message = GetQueueMessage(queue);
}
}

The caveat of this is at what point are you defining 20 minutes old?
Case 1: If defined based upon when the message was generated and sent, then you're going to have to use the Message.SentTime. You can then use Peek to look at the message, and delay processing until the 20 minute threshold has been met. This assumes that messages will arrive in a linear order, which is not guaranteed! (See below.)
Case 2: If defined based upon when the message arrives in the queue, take note of the time of Message.ArrivedTime (IE process is "listening" on the message queue) and consume 20 minutes later.
But... Both cases cause a problem as you can only peek on the first message. As such, you will have to consume the message into a queue or other structure for processing at the designated time.