Develop Reference

How to solve producer/consumer race condition with BlockingCollection<>

I am implementing a logger which writes records to the database. In order to prevent the database writes from blocking the code which is calling the logger, I've moved the DB access to a separate thread, implemented using a producer/consumer model based on BlockingCollection<string>.
Here's the simplified implementation:
abstract class DbLogger : TraceListener
{
private readonly BlockingCollection<string> _buffer;
private readonly Task _writerTask;
DbLogger()
{
this._buffer = new BlockingCollection<string>(new ConcurrentQueue<string>(), 1000);
this._writerTask = Task.Factory.StartNew(this.ProcessBuffer, TaskCreationOptions.LongRunning);
}
// Enqueue the msg.
public void LogMessage(string msg) { this._buffer.Add(msg); }
private void ProcessBuffer()
{
foreach (string msg in this._buffer.GetConsumingEnumerable())
{
this.WriteToDb(msg);
}
}
protected abstract void WriteToDb(string msg);
protected override void Dispose(bool disposing)
{
if (disposing)
{
// Signal to the blocking collection that the enumerator is done.
this._buffer.CompleteAdding();
// Wait for any in-progress writes to finish.
this._writerTask.Wait(timeout);
this._buffer.Dispose();
}
base.Dispose(disposing);
}
}
Now, when my application shuts down, I need to make sure that the buffer is flushed before the database connection goes down. Otherwise, WriteToDb will throw an exception.
So, here's my naive Flush implementation:
public void Flush()
{
// Sleep until the buffer is empty.
while(this._buffer.Count > 0)
{
Thread.Sleep(50);
}
}
The problem with this implementation is the following sequence of events:
There is one entry in the buffer.
In the logging thread, MoveNext() is called on the enumerator, so we're now in the body of ProcessBuffer's foreach loop.
Flush() is called by the main thread. It sees that the collection is empty, so returns immediately.
The main thread closes the DB connection.
Back in the logging thread, the body of the foreach loop starts executing. WriteToDb is called, and fails because the DB connection has been closed.
So, my next try was adding some flags, like so:
private volatile bool _isWritingBuffer = false;
private void ProcessBuffer()
{
foreach (string msg in this._buffer.GetConsumingEnumerable())
{
lock (something) this._isWritingBuffer = true;
this.WriteToDb(msg);
lock (something) this._isWritingBuffer = false;
}
}
public void Flush()
{
// Sleep until the buffer is empty.
bool isWritingBuffer;
lock(something) isWritingBuffer = this._isWritingBuffer;
while(this._buffer.Count > 0 || isWritingBuffer)
{
Thread.Sleep(50);
}
}
However, there's still a race condition, since the entire Flush() method could execute after the collection is empty but before _isWritingBuffer gets set to true.
How can I fix my Flush implementation to avoid this race condition?
Note: For various reasons, I must write the logger from scratch, so please don't answer with a suggestion that I use some existing logging framework.

First never ever lock on a public object, especially this.
Moreover never ever use bare booleans for synchronization: see my blog if you want to have a glimpse in what can go wrong: Synchronization, memory visibility and leaky abstractions :)
Concerning the issue itself I must be missing something but why do you need such a Flush method?
Indeed when you're done with your logging you will dispose of the logger by calling its Dispose method from the main thread.
And you have implemented it in such a way that it will wait for the "write to DB" task.
If I'm wrong and you really need to synchronize with another primitive then you should use an event:
In the DbLogger:
public ManualResetEvent finalizing { get; set; }
public void Flush()
{
finalizing.WaitOne();
}
And somewhere, e.g. in ProcessBuffer you notify when you're done with writing to DB:
finalizing.Set();

ExcelAsyncUtil.Observe - to create a running clock in Excel

I am trying out the ExcelAsyncUtil.Observe function. I made the following code that shows a running clock in Excel. It works fine but I am not sure what I am doing. Two questions:
Should I add functionality for observer.OnCompleted() and observer.OnError()? What does these calls do?
What should I do in the IDisposible class? Why is it there?
Here is my sample code:
[ExcelFunction]
public static object MyExcelTicker()
{
return ExcelAsyncUtil.Observe("MyExcelTicker", new object[] { }, TickerFunction());
}
public static ExcelObservableSource TickerFunction()
{
ExcelObservableSource source = new ExcelObservableSource(() => new TickerObservable());
return source;
}
public class TickerObservable : IExcelObservable
{
public IDisposable Subscribe(IExcelObserver observer)
{
var timer = new System.Timers.Timer();
timer.Interval = 1000;
timer.Elapsed += (s, e) => observer.OnNext(DateTime.Now.ToString());
timer.Start();
// What about observer.OnCompleted() and observer.OnError()?
return new TickerDisposable();
}
}
public class TickerDisposable : IDisposable
{
public void Dispose()
{
// What to do here?
}
}

It has been a while and at least one thing is still left not covered, so let me add to what Govert said.
You've asked:
public class TickerDisposable : IDisposable
{
public void Dispose()
{
// What to do here?
}
}
Let's summarize:
For each new subscriber to your clock-ticker, a Subscribe will be called on the TickerObservable. Therefore, for each subscriber, your code will create a new System.Timers.Timer and a new timer.Elapsed event handler - to get your intended effect. And this is actually all that you need to get your effect.
However, you are also required to return an IDisposable, therefore you've created a dummy TickerDisposable solely for that purpose, and you are not sure what it is for.
Answer:
The IDisposable that the library requires you to return from the Subscribe is there just to allow you to cleanup after your glitter stops shining. Timers are a "system thing". Once you create them and start them, they run. After an hour they cannot be GC'ed, because they are meant to be run until you stop them. Surely, you've +='ed an event hander, the observer (if weakly-reference'd) might be already dead, but your timer does not know! You must stop it at some point.
Hence, IDisposable-related pattern, borrowed from RX: whatever heavy or long-living you allocate, reserve, build, etc in the Subscribe method, put some note about it into that (yours!) IDisposable. Then, when the observer unsubscribes, your IDisposable will get cleaned too, and your custom Dispose method will be run, that will be able to look at your IDiposable's contents and .. cleanup the garbage, or rather, unlock it, so the GC can flush them.
Completing your example:
public class TickerObservable : IExcelObservable
{
public IDisposable Subscribe(IExcelObserver observer)
{
var timer = new System.Timers.Timer();
timer.Interval = 1000;
timer.Elapsed += (s, e) => observer.OnNext(DateTime.Now.ToString());
timer.Start();
return new TickerDisposable(timer);
}
}
public class TickerDisposable : IDisposable
{
private Timer ticky;
public TickerDisposable(Timer timer)
{
ticky = timer;
}
public void Dispose()
{
if(ticky != null)
ticky.Dispose(); // or Stop, or etc..
}
}
The above example is actually most-obvious usage of the returned IDisposable. However, you can use it for any register-unregister notification. For example, with single shared timer, it might look like this:
public class TickerObservable : IExcelObservable
{
private static Timer timer = ..... ; // assume it is up & running & shared
public IDisposable Subscribe(IExcelObserver observer)
{
ElapsedEventHander hd = (s, e) => observer.OnNext(DateTime.Now.ToString());
timer.Elapsed += hd;
return new TickerDisposable(timer, hd);
}
}
public class TickerDisposable : IDisposable
{
private Timer ticky;
private ElapsedEventHander handler;
public TickerDisposable(Timer timer, ElapsedEventHander hd)
{
ticky = timer;
handler = hd;
}
public void Dispose()
{
if(ticky != null && handler != null)
ticky.Elapsed -= handler;
}
}
And now you are perfectly sure that no dead-handlers are lingering at the long-living-shared-timer. (of course the cleanup of the timer is missing here, but that's another thing..). Probably you already got the idea, so, have fun!

The IExcelObserver interface matches the semantics of the IObserver interface from the Reactive Extensions library (http://msdn.microsoft.com/en-us/library/dd783449.aspx).
You function can call OnNext zero or more times, and then call OnError if an error occurs, or OnCompleted if no further events will be raised. Excel-DNA will handle OnError as it would an exception thrown by a regular UDF, and will return #VALUE to the cell or process the exception via the registered UnhandledExceptionHandler. OnCompleted is not so useful in the Excel context - it just indicates that no further values will be raised.
For your example, error don't seem to be a problem, and there is no end to the stream of events, so you need never call OnError or OnCompleted.
The Excel-DNA infrastructure will call the IDisposable.Dispose when the observable is no longer hooked up to a cell formula. For example, if the formula with the MyExcelTicker() call is deleted from the cell. You can use this as a notification to clean up any back-end resources, or ignore the notification if you're not interested.

Invoke/Call an even in a different thread

I have to make a Instant Messenger server in C#. The server is a ConsoleApplication project. And I want to make a server that runs in 3 threads. I will explain in the code below.
The question is how can I invoke a function from a separate thread, or make an event on a separate thread?
This is the main server class.
public class GiNetServer
{
public void Start()
{
netServer = new NetServer();
msgHandler = new NetMsgHandler();
netServer.NewNetMsg += msgHandler.HandleMsg;
Work();
}
private void Work()
{
while(true) //This runs in the MainThread
sleep(1);
}
}
The NetServer class creates 2 Threads: acceptClientsThread and receiveMessagesThread.
The receive thread calls the NewNetMsg.
public class NetServer
{
public event NewNetMsgEventHandler NewNetMsg;
public NetServer()
{
acceptClientsThread = new Thread(ListenForClients);
receiveMessageThread = new Thread(Receive);
//and of course starts them here...
}
private void Receive()
{
while(true)
{
Heartbeat();
}
}
private void Heartbeat()
{
foreach(netClient in clientsList)
{
if (netClient.DataAvalible)
{
netClient.Recive();
}
if (!netClient.IsBufferEmpty())
{
nextMsg = netClient.NextMessage();
if (nextMsg != null)
NewNetMsg(netClient, nextMsg); //Call the event!
}
}
}
}
How can I make the msgHandler.HandleMsg function run in a separate thread or in the MainThread?
Like this, HandleMsg runs in the receiveMessagesThread.
The code above is pseudocode-ish. If there is anything ambigous please let me know.

There are a lot of different ways to move the HandleMsg call onto a different thread, depending on what your requirements are. The simplest way would be to raise the NewNetMsg event on a different thread using the ThreadPool:
ThreadPool.QueueUserWorkItem(s => NewNetMsg(netClient, nextMsg));
You could also use the Task Parallel Library (TPL). You could also add the event to a ConcurrentQueue that is processed by a dedicated background thread. And there are more options. Without more details it is impossible to give a more specific recommendation.

Putting synchronous wrapper over asynchronous methods

I'm integrating with a PIN device with an api containing asynchronous methods. For example one of them is called GetStatus and it raises a DeviceStateChangedEvent with the state passed into it as a parameter.
I'd like to have an interface that is not asynchronous over it though, so that when I call GetStatus on my interface it will actually return the status rather than raising an event to pass that data to me.
I'm thinking I could do something like this:
public class MSRDevice
{
StatusInfo _status;
bool _stateChangedEventCompleted = false;
IPAD _ipad; // <-- the device
public MSRDevice()
{
//Initialize device, wire up events, etc.
}
public StatusInfo GetStatus()
{
_ipad.GetStatus() // <- raises StatusChangedEvent
while(!_stateChangedEventCompleted);
_stateChangedEventCompleted = false;
return _status;
}
void StateChangedEvent(object sender, DeviceStateChangeEventArgs e)
{
_status = e.StatusInfo;
}
}
Is this a good way to address this or this there a better solution?

What you’re doing in your example is called “busy-waiting” (or “spinning”), which is unrecommended in most scenarios since it wastes a lot of CPU power. Preferably, you should use a signalling mechanism, such as the WaitHandle class, for synchronizing when an event of interest (in your case, StatusChangedEvent) has occurred:
public class MSRDevice
{
StatusInfo _status;
IPAD _ipad; // <-- the device
private EventWaitHandle waitHandle = new AutoResetEvent(false);
public MSRDevice()
{
//Initialize device, wire up events, etc.
}
public StatusInfo GetStatus()
{
_ipad.GetStatus() // <- raises StatusChangedEvent asynchronously
waitHandle.WaitOne(); // <- waits for signal
return _status;
}
void StateChangedEvent(object sender, DeviceStateChangeEventArgs e)
{
_status = e.StatusInfo;
waitHandle.Set(); // <- sets signal
}
}

The best option: code it async.
No; that is a hot loop. It will hammer the CPU. It also isn't guaranteed to exit due to register caching (this is trivial to demonstrate on x86 in particular).
If you need it sync, you should use something like an AutoResetEvent.

Timer within Thread within Windows Service

I cant figure out how to proceed the best way with this problem.
Right now I have a windows service which only task is to gather data from a database with a specific DSN and then send out an email if the data is valid. The service contains a timer which ticks every 5 minuts and performs the tasks above.
Now I need to re-write the windows service to be able to run on more than 1 DSN.
I was thinking of making several threads inside the windows service and then again have a seperat timer inside each thread.
Is this a good idea and how can this be done? I want to avoid having a windows service for each DSN.
Ill try to draw it if I dont make any sense
Windows Service
Thread1(DSN1)-----------------------------Thread2(DSN2)----------------------Thread3(DSN3)
Timer(ticks every X minuts)-----------------Timer(same)-------------------------Timer(same)
Logic()---------------------------------------------Logic---------------------------------Logic()
Hope my problem makes sense :)

As far as I Know each timer represents a thread on its own. Knowing this, I would try to dynamically create timer objects for each given dsn.
public partial class Service1 : ServiceBase
{
public Service1()
{
InitializeComponent();
}
private List<GetDataFromDSN> list = null;
protected override void OnStart(string[] args)
{
list = new List<GetDataFromDSN>();
// assume args contains your given dsn values
foreach (string dsn in args)
{
GetDataFromDSN newObj = new GetDataFromDSN();
newObj.DSN = dsn;
list.Add(newObj);
newObj.Start();
}
}
}
public class GetDataFromDSN
{
public string DSN { get; set; }
private Timer timer = null;
private double interval = 1000*60*5; // 5 minutes interval
public GetDataFromDSN()
{
// init your object
timer = new Timer(interval);
timer.Elapsed +=new ElapsedEventHandler(timer_Elapsed);
}
private void timer_Elapsed(object sender, ElapsedEventArgs e)
{
// do what ever you want
}
public void Start() // or even make timer public
{
timer.Start();
}
public void Stop()
{
timer.Stop();
}
}

Do each of the DSNs need to be on a separate Thread?
If you were to encapsulate the Email retrieval and validation logic within some sort of service that the Thread invoked, the fact that there were multiple DSNs could be hidden from the scheduling thread. For instance, an IEmailService might have the following contract:
public interface IEmailService
{
void SendEmailsToValidAddresses();
}
and the implementation might look something like this:
public class MultipleSourcesEmailService : IEmailService
{
private IEnumerable<IDatabaseSource> databases;
public EmailService(params IDatabaseSource[] sources)
{
databases = new List<IDatabaseSource>(sources);
}
public void SendEmailsToValidAddresses()
{
foreach(var database in databases)
{
var emailAddresses = database.SelectAllEmailAddresses();
ValidateAndSendEmailsTo(emailAddresses);
}
}
public void ValidateAndSendEmailsTo(IEnumerable<string> emailAddresses)
{
// Perform appropriate logic
...
}
}
In this way, your timer logic can remain the same and on a single Thread whilst the concern of sending emails is separated into the IEmailService. This also means that you could implement a SingleSourceEmailService and a MultipleSourceEmailService and swap the multiple sources in when you're code complete and the consumer of the service need never know.
Of course, the EmailService as implemented above will SendEmails from multiple sources sequentially - if you need it to run in parallel you could change the EmailService to kick off a new Thread for each of the DSNs that you have, you could even call it the: MultiThreadedMultipleSourceEmailService but as a consumer of the IEmailService your scheduling will never know the difference.

Use a backgroundworker.
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
Just start one for each DSN, very smooth and easy to use.

Try using System.Threading.Timer
Here is sample code from my project, Hope this helps
public void StartDSNTimers()
{
_tmr1 = new Timer(CheckMessages, dsn1, 0, 60000);
_tmr2 = new Timer(CheckMessages, dsn2, 0, 60000);
_tmr3 = new Timer(CheckMessages, dsn3, 0, 60000);
}
private void CheckMessages(object obj)
{
//Logic
}