Develop Reference

Creating a buffer for Consumer and Producer threads using Queue c# .NET

I am writing a windows service application that is capable of collecting data from sensors like temperature, pressure volume etc...
The frequency at which the data is read is pretty high, there could be a hundred sensors and the data being received could be at a frequency could be one per second per sensor..
I need to store this data to an oracle database, for obvious reasons i dont want to hit the database at such a high rate.
Hence i want to create a Buffer.
My plan is to create a Buffer using the standard .NET Queue, a few threads keep Enqueue data into the queue and another timer driven thread can keep writing into the database at regular intervals.
What i want to know is..?? Is This thread safe
If this is not, what is the best way of creating a in memory buffer

To answer your question, as long as you lock accesses, you can have multiple threads access a regular queue.
For me though, I didn't use that and wanted to use queues with locks to keep them thread safe. I have been doing this in c# for one of my programs. I just use a regular queue, and then put a locker on accesses to it (enqueue, dequeue, count). It is completely thread safe if you just lock the accesses.
My setup comes from the tutorial/example here: http://www.albahari.com/threading/part2.aspx#_ProducerConsumerQWaitHandle
My situation is a little different than yours, but pretty similar. For me, my data can come in very fast, and if I don't queue it I lose the data if multiple come in at the same time. Then I have a thread running that slowly takes items off the queue and processes them. This hand-off uses an AutoResetEvent to hold my working-thread until data is ready to be processed. In your case you would use a timer or something that happens regularly.
I copy/pasted my code and tried to change the names. Hopefully I didn't completely break it by missing some name changes, but you should be able to get the gist.
public class MyClass : IDisposable
{
private Thread sensorProcessingThread = null;
private Queue<SensorData> sensorQueue = new Queue<SensorData>();
private readonly object _sensorQueueLocker = new object();
private EventWaitHandle _whSensorEvent = new AutoResetEvent(false);
public MyClass () {
sensorProcessingThread = new Thread(sensorProcessingThread_DoWork);
sensorProcessingThread.Start();
}
public void Dispose()
{
// Signal the end by sending 'null'
EnqueueSensorEvent(null);
sensorProcessingThread.Join();
_whSensorEvent.Close();
}
// The fast sensor data comes in, locks queue, and then
// enqueues the data, and releases the EventWaitHandle
private void EnqueueSensorEvent( SensorData wd )
{
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
_whSensorEvent.Set();
}
}
// When asynchronous events come in, I just throw them into queue
private void OnSensorEvent( object sender, MySensorArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// I have several types of events that can come in,
// they just get packaged up into the same "SensorData"
// struct, and I worry about the contents later
private void FileSystem_Changed( object sender, System.IO.FileSystemEventArgs e )
{
EnqueueSensorEvent(new SensorData(sender, e));
}
// This is the slower process that waits for new SensorData,
// and processes it. Note, if it sees 'null' as data,
// then it knows it should quit the while(true) loop.
private void sensorProcessingThread_DoWork( object obj )
{
while ( true )
{
SensorData wd = null;
lock ( _sensorQueueLocker )
{
if ( sensorQueue.Count > 0 )
{
wd = sensorQueue.Dequeue();
if ( wd == null )
{
// Quit the loop, thread finishes
return;
}
}
}
if ( wd != null )
{
try
{
// Call specific handlers for the type of SensorData that was received
if ( wd.isSensorDataType1 )
{
SensorDataType1_handler(wd.sender, wd.SensorDataType1Content);
}
else
{
FileSystemChanged_handler(wd.sender, wd.FileSystemChangedContent);
}
}
catch ( Exception exc )
{
// My sensor processing also has a chance of failing to process completely, so I have a retry
// methodology that gives up after 5 attempts
if ( wd.NumFailedUpdateAttempts < 5 )
{
wd.NumFailedUpdateAttempts++;
lock ( _sensorQueueLocker )
{
sensorQueue.Enqueue(wd);
}
}
else
{
log.Fatal("Can no longer try processing data", exc);
}
}
}
else
_whWatchEvent.WaitOne(); // No more tasks, wait for a signal
}
}
Something you could maybe look at is Reactive (Rx) for .net from Microsoft. Check out: https://msdn.microsoft.com/en-us/data/gg577611.aspx and at the bottom of page is a pdf tutorial "Curing the asynchronous blues": http://go.microsoft.com/fwlink/?LinkId=208528 This is something very different but maybe you will see something you like.

AccessViolationException on sp_session_player_load()

I'm trying to create a stream application based on Spotify's libspotify SDK.
To achieve this in C# I'm using the ohLibspotify bindings and wrapper. This is only a thin abstraction layer so most of it will be a 1:1 mapping to the libspotify SDK. To play the incoming PCM data I'm using the NAudio library.
Most of the times I can play the first track. Then when I load the second one I get a AccessViolationException whilst trying to call sp_session_player_load(). Also this sometimes happens the first time I try to play a track and sometimes it happens the third time.
This is the function I use to play a track.
public void playTrack(string track, string juke)
{
new Thread(new ThreadStart(() =>
{
var playable = Link.CreateFromString(string.Format("spotify:track:{0}", track)).AsTrack();
if (playing)
{
player.Pause();
App.Logic.spotify.sp_session.PlayerUnload();
}
buffer = new BufferedWaveProvider(new WaveFormat())
{
BufferDuration = TimeSpan.FromSeconds(120),
DiscardOnBufferOverflow = false
};
var waitEvent = new AutoResetEvent(false);
while (!playable.IsLoaded())
{
waitEvent.WaitOne(30);
}
App.Logic.spotify.sp_session.PlayerLoad(playable);
App.Logic.spotify.sp_session.PlayerPlay(true);
player = new WaveOut();
player.Init(buffer);
player.Play();
playing = true;
})).Start();
}
The AccessViolationException occurs on line 6 of the following piece of code within the wrapper library.
[DllImport("libspotify")]
internal static extern SpotifyError sp_session_player_load(IntPtr #session, IntPtr #track);
public void PlayerLoad(Track #track)
{
SpotifyError errorValue;
errorValue = NativeMethods.sp_session_player_load(this._handle, track._handle);
SpotifyMarshalling.CheckError(errorValue);
}
The streaming callbacks:
public override void GetAudioBufferStats(SpotifySession session, out AudioBufferStats stats)
{
stats = new AudioBufferStats()
{
samples = App.Logic.spotify.player.buffer.BufferedBytes / 2,
stutter = 0
};
}
public override int MusicDelivery(SpotifySession session, AudioFormat format, IntPtr frames, int num_frames) {
int incoming_size = num_frames * format.channels * 2;
try
{
if (incoming_size > sample_buffer.Length)
{
short rendered_frames = Convert.ToInt16(Math.Floor((sample_buffer.Length / format.channels / 2d)));
short rendered_size = Convert.ToInt16(rendered_frames * format.channels * 2);
Marshal.Copy(frames, sample_buffer, 0, rendered_size);
App.Logic.spotify.player.buffer.AddSamples(sample_buffer, 0, rendered_size);
return rendered_frames;
}
else
{
Marshal.Copy(frames, sample_buffer, 0, incoming_size);
App.Logic.spotify.player.buffer.AddSamples(sample_buffer, 0, incoming_size);
return num_frames;
}
}
catch (Exception e)
{
return 0;
}
}

As #Weeble said in his comment to my question. It is very hard to diagnose the possible source of an AccessViolationException. In my case it was threading, there were multiple threads accessing the Spotify session object at once.
If this could be your problem as well you might want to look at this article. It talks about thread synchronization in C# and VB.Net. It's very easy really.
lock(sessionLock)
{
App.Logic.spotify.sp_session.PlayerLoad(playable);
App.Logic.spotify.sp_session.PlayerPlay(true);
}
The sessionLock object can be a simple instantiation of the Object type. Though it's not recommended to use any "actual" objects for this. In my case I did the following:
public class Player
{
private object sessionLock = new object();
// Rest of code with locks
}
This way you can lock the sessionLock object every time you want to access the, in this case, SpotifySession object. So that when thread 1 is loading a song and thread 2 wants to process the events at the same time, thread 2 will be blocked until the lock on sessionLock has been lifted.
As #Weeble suggested there are possibly some other things you might want to look into if threading is not the issue.
You might not be handling spotify's ref-counting correctly, and accidentally decreasing a ref-count too early or forgetting to increment one somewhere it is necessary.
You might be corrupting memory in the music callbacks that deal with native pointers.
There might be a bug in ohLibspotify or libspotify. If you think this is the case then go to the ohLibspotify repo or the openhome forum to report your issue.

Some cases when necessary to call GC.Collect manualy

I've read many articles about GC, and about "do no care about objects" paradigm, but i did a test for proove it.
So idea is: i'm creating a lot of large objects stored in local functions, and I suspect that after all tasks are done it will clean the memory itself. But GC didn't. So test code:
class Program
{
static void Main()
{
var allDone = new ManualResetEvent(false);
int completed = 0;
long sum = 0; //just to prevent optimizer to remove cycle etc.
const int count = int.MaxValue/10000000;
for (int i = 0; i < count; i++)
{
ThreadPool.QueueUserWorkItem(delegate
{
unchecked
{
var dumb = new Dumb();
var localSum = 0;
foreach (int x in dumb.Arr)
{
localSum += x;
}
sum += localSum;
}
if (Interlocked.Increment(ref completed) == count)
allDone.Set();
if (completed%(count/100) == 0)
Console.WriteLine("Progress = {0:N2}%", 100.0*completed/count);
});
}
allDone.WaitOne();
Console.WriteLine("Done. Result : {0}", sum);
Console.ReadKey();
GC.Collect();
Console.WriteLine("GC Collected!");
Console.WriteLine("GC CollectionsCount 0 = {0}, 1 = {1}, 2 = {2}", GC.CollectionCount(0), GC.CollectionCount(1),GC.CollectionCount(2));
Console.ReadKey();
}
}
class Dumb
{
public int[] Arr = Enumerable.Range(1,10*1024*1024).ToArray(); // 50MB
}
so in my case app eat ~2GB of RAM, but when I'm clicking on keyboard and launching GC.Collect it free occuped memory up to normal size of 20mb.
I've read that manual calls of GC etc is bad practice, but i cannot avoid it in this case.

In your example there is no need to explicitly call GC.Collect()
If you bring it up in the task manager or Performance Monitor you will see the GC working as it runs. GC is called when needed by the OS (when it is trying to allocate and doesn't have memory it will call GC to free some up).
That being said since your objects ( greater than 85000 bytes) are going onto the large object heap, LOH, you need to watch out for large object heap fragmentation. I've modified your code so show how you can fragment the LOH. Which will give an out of memory exception even though the memory is available, just not contiguous memory. As of .NET 4.5.1 you can set a flag to request that LOH to be compacted.
I modified your code to show an example of this here:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace GCTesting
{
class Program
{
static int fragLOHbyIncrementing = 1000;
static void Main()
{
var allDone = new ManualResetEvent(false);
int completed = 0;
long sum = 0; //just to prevent optimizer to remove cycle etc.
const int count = 2000;
for (int i = 0; i < count; i++)
{
ThreadPool.QueueUserWorkItem(delegate
{
unchecked
{
var dumb = new Dumb( fragLOHbyIncrementing++ );
var localSum = 0;
foreach (int x in dumb.Arr)
{
localSum += x;
}
sum += localSum;
}
if (Interlocked.Increment(ref completed) == count)
allDone.Set();
if (completed % (count / 100) == 0)
Console.WriteLine("Progress = {0:N2}%", 100.0 * completed / count);
});
}
allDone.WaitOne();
Console.WriteLine("Done. Result : {0}", sum);
Console.ReadKey();
GC.Collect();
Console.WriteLine("GC Collected!");
Console.WriteLine("GC CollectionsCount 0 = {0}, 1 = {1}, 2 = {2}", GC.CollectionCount(0), GC.CollectionCount(1), GC.CollectionCount(2));
Console.ReadKey();
}
}
class Dumb
{
public Dumb(int incr)
{
try
{
DumbAllocation(incr);
}
catch (OutOfMemoryException)
{
Console.WriteLine("Out of memory, trying to compact the LOH.");
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
try // try again
{
DumbAllocation(incr);
Console.WriteLine("compacting the LOH worked to free up memory.");
}
catch (OutOfMemoryException)
{
Console.WriteLine("compaction of LOH failed to free memory.");
throw;
}
}
}
private void DumbAllocation(int incr)
{
Arr = Enumerable.Range(1, (10 * 1024 * 1024) + incr).ToArray();
}
public int[] Arr;
}
}

The .NET runtime will garbage collect without your call to the GC. However, the GC methods are exposed so that GC collections can be timed with the user experience (load screens, waiting for downloads, etc).
Use GC methods isn't always a bad idea, but if you need to ask then it likely is. :-)

I've read that manual calls of GC etc is bad practice, but i cannot avoid it in this case.
You can avoid it. Just don't call it. The next time you try to do an allocation, the GC will likely kick in and take care of this for you.

Few things I can think of that may be influencing this, but none for sure :(
One possible effect is that GC doesn't kick in right away... the large objects are on the collection queue - but haven't been cleaned up yet. Specifically calling GC.Collect forces collection right there and that's where you see the difference. Otherwise it would've just happened at some point later.
Second reason i can think of is that GC may collect objects, but not necessarily release memory to OS. Hence you'd continue seeing high memory usage even though it's free internally and available for allocation.

The garbage collection is clever and decide when the time right to collect your objects. This is done by heuristics and you must read about that. The garbage collection makes his job very good. Are the 2GB a problem for yout system or you just wondering about the behaviour?
Whenever you call GC.Collect() don't forget the call GC.WaitingForPendingFinalizer. This avoids unwanted aging of objects with finalizer.

Memory leak issue, private data is increasing .Net Framework 4

In the production windows service I had a memory leak in an application.
Service OnStart event: This how my service kick off. I suspect it is because of the events. All this service does is, it subscribes couple of events from third party library and execute those events when the events occurred.
In this code Aggregator is defined as class property
public EventAggregator Aggregator{ get; set;}
Task.Factory
.StartNew(ConnectionManager.AttemptXXXXXXConnection)
.ContinueWith(task => Aggregator.StartAggregatePublishing(), TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith(task =>
{
if (task.Exception == null) return;
Logger.Log("Failed to start Events aggregation due to:");
foreach (var exception in task.Exception.InnerExceptions)
Logger.Log(exception.Message);
}, TaskContinuationOptions.OnlyOnFaulted );
And the StartAggregatePublishing method is defined as follows
public IEventHandler<ISystemEvent3> SystemEventHandler
{
get; set;
}
public IEventHandler<DeviceEvents2EventArgs> DeviceEventHandler
{
get; set;
}
Here systemEventsManagement and deviceEventManagement belong to third party library events so we can provide our own implementation.
public void StartAggregatePublishing()
{
Logger.Log("Start event publishing");
XXXData.Events.SystemEvent += SystemEventHandler.HandleEvent;
XXXdata.Events.DeviceEvent += DeviceEventHandler.HandleEvent;
systemEventsManagement.Events += systemEventsManagement_Events;
deviceEventsManagement.Event += deviceEventsManagement_Event;
processor.Start();
Logger.Log("System and Device events aggregation started.");
}
Now the game start here with EventAggregator
public EventAggregator( int sleepMills = 1000)
{
processor = new ThreadPausableProcessor( CheckDeviceStates, HandleException, sleepMills );
}
and the method which runs for every minute is like this
private void CheckDeviceStates()
{
if ( Device.IsConnected )
{
foreach (var deviceState in Skidata.Events.GetAllDeviceStates())
{
Logger.Log(string.Format("Checking Handlers for Device id = {0} with Overall state = {1}", deviceState.DeviceId, deviceState.OverallState));
// Here I am handling pertucular event and logging and reporting stuff.. Nothing major
DeviceStateHandler.HandleEvent(deviceState);
}
// Once I handled all the events I am unhooking all the events and disposing everything.
Dispose(true);
int mills;
ConfigurationManager.RefreshSection("appSettings");
if ( Int32.TryParse( ConfigurationManager.AppSettings["PollFrequencyInMills"], out mills ) )
{
if ( processor.SleepMills != mills )
{
processor.SleepMills = mills;
Logger.Log(string.Format("New polling frequency for DeviceStates processor thread = {0} mills", mills));
}
}
}
else
{
Logger.Log("CheckDeviceStates => Not connected to facility.");
}
}
What I am missing here I don't know I hope I gave enough details.
This is code was written by previous programmer in my place now this code is in production.
Can anyone has any clue where exactly it could potentially go wrong.
I have used DotTrace Memory profiler and had couple of snapshots but I couldn't find the problem everything looks ok to me. I can supply those snapshots if needed.
when service re started Private bytes: 31,383,552
after three days of running Private Bytes: 65,584,648

Have you tried Redgate ANTS Memory Profiler ?
It can visualize the relationship between objects and show most memory-hungry objects

Serially process ConcurrentQueue and limit to one message processor. Correct pattern?

I'm building a multithreaded app in .net.
I have a thread that listens to a connection (abstract, serial, tcp...).
When it receives a new message, it adds it to via AddMessage. Which then call startSpool. startSpool checks to see if the spool is already running and if it is, returns, otherwise, starts it in a new thread. The reason for this is, the messages HAVE to be processed serially, FIFO.
So, my questions are...
Am I going about this the right way?
Are there better, faster, cheaper patterns out there?
My apologies if there is a typo in my code, I was having problems copying and pasting.
ConcurrentQueue<IMyMessage > messages = new ConcurrentQueue<IMyMessage>();
const int maxSpoolInstances = 1;
object lcurrentSpoolInstances;
int currentSpoolInstances = 0;
Thread spoolThread;
public void AddMessage(IMyMessage message)
{
this.messages.Add(message);
this.startSpool();
}
private void startSpool()
{
bool run = false;
lock (lcurrentSpoolInstances)
{
if (currentSpoolInstances <= maxSpoolInstances)
{
this.currentSpoolInstances++;
run = true;
}
else
{
return;
}
}
if (run)
{
this.spoolThread = new Thread(new ThreadStart(spool));
this.spoolThread.Start();
}
}
private void spool()
{
Message.ITimingMessage message;
while (this.messages.Count > 0)
{
// TODO: Is this below line necessary or does the TryDequeue cover this?
message = null;
this.messages.TryDequeue(out message);
if (message != null)
{
// My long running thing that does something with this message.
}
}
lock (lcurrentSpoolInstances)
{
this.currentSpoolInstances--;
}
}

This would be easier using BlockingCollection<T> instead of ConcurrentQueue<T>.
Something like this should work:
class MessageProcessor : IDisposable
{
BlockingCollection<IMyMessage> messages = new BlockingCollection<IMyMessage>();
public MessageProcessor()
{
// Move this to constructor to prevent race condition in existing code (you could start multiple threads...
Task.Factory.StartNew(this.spool, TaskCreationOptions.LongRunning);
}
public void AddMessage(IMyMessage message)
{
this.messages.Add(message);
}
private void Spool()
{
foreach(IMyMessage message in this.messages.GetConsumingEnumerable())
{
// long running thing that does something with this message.
}
}
public void FinishProcessing()
{
// This will tell the spooling you're done adding, so it shuts down
this.messages.CompleteAdding();
}
void IDisposable.Dispose()
{
this.FinishProcessing();
}
}
Edit: If you wanted to support multiple consumers, you could handle that via a separate constructor. I'd refactor this to:
public MessageProcessor(int numberOfConsumers = 1)
{
for (int i=0;i<numberOfConsumers;++i)
StartConsumer();
}
private void StartConsumer()
{
// Move this to constructor to prevent race condition in existing code (you could start multiple threads...
Task.Factory.StartNew(this.spool, TaskCreationOptions.LongRunning);
}
This would allow you to start any number of consumers. Note that this breaks the rule of having it be strictly FIFO - the processing will potentially process "numberOfConsumer" elements in blocks with this change.
Multiple producers are already supported. The above is thread safe, so any number of threads can call Add(message) in parallel, with no changes.

I think that Reed's answer is the best way to go, but for the sake of academics, here is an example using the concurrent queue -- you had some races in the code that you posted (depending upon how you handle incrementing currnetSpoolInstances)
The changes I made (below) were:
Switched to a Task instead of a Thread (uses thread pool instead of incurring the cost of creating a new thread)
added the code to increment/decrement your spool instance count
changed the "if currentSpoolInstances <= max ... to just < to avoid having one too many workers (probably just a typo)
changed the way that empty queues were handled to avoid a race: I think you had a race, where your while loop could have tested false, (you thread begins to exit), but at that moment, a new item is added (so your spool thread is exiting, but your spool count > 0, so your queue stalls).
private ConcurrentQueue<IMyMessage> messages = new ConcurrentQueue<IMyMessage>();
const int maxSpoolInstances = 1;
object lcurrentSpoolInstances = new object();
int currentSpoolInstances = 0;
public void AddMessage(IMyMessage message)
{
this.messages.Enqueue(message);
this.startSpool();
}
private void startSpool()
{
lock (lcurrentSpoolInstances)
{
if (currentSpoolInstances < maxSpoolInstances)
{
this.currentSpoolInstances++;
Task.Factory.StartNew(spool, TaskCreationOptions.LongRunning);
}
}
}
private void spool()
{
IMyMessage message;
while (true)
{
// you do not need to null message because it is an "out" parameter, had it been a "ref" parameter, you would want to null it.
if(this.messages.TryDequeue(out message))
{
// My long running thing that does something with this message.
}
else
{
lock (lcurrentSpoolInstances)
{
if (this.messages.IsEmpty)
{
this.currentSpoolInstances--;
return;
}
}
}
}
}

Check 'Pipelines pattern': http://msdn.microsoft.com/en-us/library/ff963548.aspx
Use BlockingCollection for the 'buffers'.
Each Processor (e.g. ReadStrings, CorrectCase, ..), should run in a Task.
HTH..