Develop Reference

Multi Thread c# application System.OutOfMemoryException after 1~5 minutes of runtime

Here is my Timer Elapsed Event, I am receiving the System.OutOfMemoryException on the line Thread thread = new Thread(threadStart);
I am receiving the error fairly fast (1~5 minutes, randomly), and it does not cause unexpected results in my program. I am just wondering what is causing this error, and I am afraid it may cause unexpected results if it is left unchecked. I have searched on the internet and am comming no where near the number of max threads.
readList contains about 46 enteries.
Any help would be appreciated.
private void glob_loopTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
ParameterizedThreadStart threadStart = new ParameterizedThreadStart(readHoldingRegisters);
foreach (readwriteDataGridRow.Read row in readList)
{
Thread thread = new Thread(threadStart);
thread.IsBackground = true;
thread.Start(System.Convert.ToInt32(row.Address));
}
}
catch (Exception ex)
{
UpdateConsole(new object[] { ex.Message.ToString() + " " + ex.StackTrace.ToString(), Color.Red });
Thread.CurrentThread.Abort(); // maybe?
}
}
EDIT:
Here is a bit more information.
My program is reading registers from a Serial Device using the Modbus RTU protocol.
A single register takes less than a tenth of a second to retrieve from readHoldingRegisters
I am open to suggestions on what else to use rather than threads.
note: I need to call readHoldingRegisters 40 - 100 times in a single 'pass'. The passes start when the user hits connect and end when he hits disconnect. Timers are not needed, they just offered a simple way for me to maintain the loop with a start and stop button.
EDIT: Solved
private void glob_loopTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
foreach (readwriteDataGridRow.Read row in readList)
{
readHoldingRegisters(row.Address);
}
}
catch (Exception ex)
{
UpdateConsole(new object[] { ex.Message.ToString() + " " + ex.StackTrace.ToString(), Color.Red });
}
}
The additional Threads were the problem and were not needed.

Ughh, do not, ever (well almost ever) abort threads. There are many preferable ways to make a System.Thread stop. Look around SO, you will find plenty of examples on why doing this is a bad idea and alternative approaches.
On with your question: The problem doesn't seem to be the number of rows in readList. It is more likely that your glob_looperTimer_Elapsed event handler is being executed many times and you are basically starting more and more threads.
What is the interval of your glob_loopTimer?

So how many times is glob_loopTimer_Elapsed called? The name implies that it is run on a periodic timer interval. If so, and if the 46 threads that get created on each invocation do not terminate about as quickly as the timer interval fires, then you could easily be spawning too many threads and running out of memory space as a result. Perhaps you could try logging when each thread starts and when each one finishes to get an idea about how many are in flight at once?
Keep in mind that every thread you allocate will have a certain amount of stack space allocated to it. Depending upon your runtime configuration, this amount of stack space may not be negligible (as in, it may be 1 MB per thread or more) and it may quickly consume your available memory even if you're not close to approaching the theoretical maximum number of threads supported by the OS.

Besides your problem I'll consider using ThreadPool or the TPL.
When using System.Thread there is no automisn to manage the threads...
Also each Thread allocates some memory which could lead to you problem.
The Threadpool and the TPL manage this resources by themselves
see also: -> Thread vs ThreadPool
Reusing threads that have already been created instead of creating new ones (an expensive process)
...
If you queue 100 thread pool tasks, it will only use as many threads as have already been created to service these requests (say 10
for example). The thread pool will make frequent checks (I believe
every 500ms in 3.5 SP1) and if there are queued tasks, it will make
one new thread. If your tasks are quick, then the number of new
threads will be small and reusing the 10 or so threads for the short
tasks will be faster than creating 100 threads up front.
If your workload consistently has large numbers of thread pool requests coming in, then the thread pool will tune itself to your
workload by creating more threads in the pool by the above process so
that there are a larger number of thread available to process requests
check Here for more in depth info on how the thread pool functions under the hood

I just know
Each thread also consumes (by default) around 1 MB of memory.

thread get 100% CPU very fast

I am implementing a very basic thread in C#:
private Thread listenThread;
public void startParser()
{
this.listenThread = new Thread(new ThreadStart(checkingData));
this.listenThread.IsBackground = true;
this.listenThread.Start();
}
private void checkingData()
{
while (true)
{
}
}
Then I immediately get 100% CPU. I want to check if sensor data is read inside the while(true) loop. Why it is like this?
Thanks in advance.

while (true) is what killing your CPU.
You can add Thread.Sleep(X) to you while to give CPU some rest before checking again.
Also, seems like you actually need a Timer.
Look at one of the Timer classes here http://msdn.microsoft.com/en-us/library/system.threading.timer.aspx.
Use Timer with as high pulling interval as you can afford, 1 sec, half a sec.
You need to tradeoff between CPU usage and the maximum delay you can afford between checks.

Let your loop sleep. It's running around and around and getting tired. At the very least, let it take a break eventually.

Because your function isn't doing anything inside the while block, it grabs the CPU, and, for all practical purposes, never lets go of it, so other threads can do their work
private void checkingData()
{
while (true)
{
// executes, immediately
}
}
If you change it to the following, you should see more reasonable CPU consumption:
private void checkingData()
{
while (true)
{
// read your sensor data
Thread.Sleep(1000);
}
}

you can use blocking queue. take a item from blocking queue will block the thread until there is a item put into the queue. that doesn't cost any cpu.
with .net4, you can use BlockingCollection http://msdn.microsoft.com/en-us/library/dd267312.aspx
under version 4, there is not blocking queue int .net framework.
you can find many implements of blocking queue if you google it.
here is a implementation
http://www.codeproject.com/KB/recipes/boundedblockingqueue.aspx
by the way. where does the data you wait come from?
EDIT
if you want to check file. you can use FileSystemWatcher to check it with thread block.
if your data comes from external API and the api doesn't block the thread, there is no way to block the thread except use Thread.Sleep

If you're polling for a condition, definitely do as others suggested and put in a sleep. I'd also add that if you need maximum performance, you can use a statistical trick to avoid sleeping when sensor data has been read. When you detect sensor data is idle, say, 10 times in a row, then start to sleep on each iteration again.

Aborting non working thread

I have a multi thread application written by c#, my max thread number is 256 and this application gets the performance counters of the computers in an Ip interval(192.168.1.0 -192.168.205.255)
it works fine and turns many times in a day. because I have to get reports.
But the problem is some times one machine keeps a thread and never finishes its work so my loop doesnt turn...
Are there any way to create threads with a countdown parameter. when I start the threads in foreach?
foreach(Thread t in threads)
{
t.start(); -----> t.start(countdownParameter) etc....
}
coundown parameter is the max life of each threads. This mean if a thread cant reach a machine it have to be abort. for example 60 seconds.. no not 256 machines, I meant 256 threads... there are about 5000 ip and 600 of them are alive. soo I am using 256 threads to read their values. and the other thing is loop. my loop is working as while all off the ipies finish it starts from beginning.

You can't specify a timeout for thread execution. However, you can try to Join each thread with a timeout, and abort it if it doesn't exit.
foreach(Thread t in threads)
{
t.Start();
}
TimeSpan timeOut = TimeSpan.FromSeconds(10);
foreach(Thread t in threads)
{
if (!t.Join(timeOut))
{
// Still not complete after 10 seconds, abort
t.Abort();
}
}
There are of course more elegant ways to do it, like using WaitHandles with the WaitAll method (note that WaitAll is limited to 64 handles at a time on most implementations, and doesn't work on STA threads, like the UI thread)

You should not terminate the thread from the outside. (Never kill a thread, make it commit suicide). Killing a thread can easily corrupt the state of an appdomain if you're not very careful.
You should rewrite the network code in the threads to either time out once the time-limit has been reached, or use asynchronous network code.

Usually a thread gets stuck on a blocking call (unless of course you have a bug causing an infinite loop). You need to identify which call is blocking and "poke" it to get it to unblock. It could be that your thread is waiting inside one of the .NET BCL waiting calls (WaitHandle.WaitOne, etc.) in which case you could use Thread.Interrupt to unblock it. But, in your case it is more likely that the API managing the communication with the remote computers is hung. Sometimes you can simply close the connection from a separate thread and that will unblock the hung method (as is the case with the Socket class). If all else fails then you really might have to fall back on the method of last of calling Thread.Abort. Just keep in mind that if you abort a thread it might corrupt the state of the app domain in which the abort originated or even the entire process itself. There were a lot of provisions added in .NET 2.0 that make aborts a lot safer than they were before, but there is still some risk.

You can use smth like this:
public static T Exec<T>(Func<t> F, int Timeout, out bool Completed)
{
T result = default(T);
Thread thread = new Thread(() => result = F());
thread.Start();
Completed = thread.Join(Timeout);
if(!Completed) thread.Abort();
return result;
}

Proper way to have an endless worker thread?

I have an object that requires a lot of initialization (1-2 seconds on a beefy machine). Though once it is initialized it only takes about 20 miliseconds to do a typical "job"
In order to prevent it from being re-initialized every time an app wants to use it (which could be 50 times a second or not at all for minutes in typical usage), I decided to give it a job que, and have it run on its own thread, checking to see if there is any work for it in the que. However I'm not entirely sure how to make a thread that runs indefinetly with or without work.
Here's what I have so far, any critique is welcomed
private void DoWork()
{
while (true)
{
if (JobQue.Count > 0)
{
// do work on JobQue.Dequeue()
}
else
{
System.Threading.Thread.Sleep(50);
}
}
}
After thought: I was thinking I may need to kill this thread gracefully insead of letting it run forever, so I think I will add a Job type that tells the thread to end. Any thoughts on how to end a thread like this also appreciated.

You need to lock anyway, so you can Wait and Pulse:
while(true) {
SomeType item;
lock(queue) {
while(queue.Count == 0) {
Monitor.Wait(queue); // releases lock, waits for a Pulse,
// and re-acquires the lock
}
item = queue.Dequeue(); // we have the lock, and there's data
}
// process item **outside** of the lock
}
with add like:
lock(queue) {
queue.Enqueue(item);
// if the queue was empty, the worker may be waiting - wake it up
if(queue.Count == 1) { Monitor.PulseAll(queue); }
}
You might also want to look at this question, which limits the size of the queue (blocking if it is too full).

You need a synchronization primitive, like a WaitHandle (look at the static methods) . This way you can 'signal' the worker thread that there is work. It checks the queue and keeps on working until the queue is empty, at which time it waits for the mutex to signal it again.
Make one of the job items be a quit command too, so that you can signal the worker thread when it's time to exit the thread

In most cases, I've done this quite similar to how you've set up -- but not in the same language. I had the advantage of working with a data structure (in Python) which will block the thread until an item is put into the queue, negating the need for the sleep call.
If .NET provides a class like that, I'd look into using it. A thread blocking is much better than a thread spinning on sleep calls.
The job you can pass could be as simple as a "null"; if the code receives a null, it knows it's time to break out of the while and go home.

If you don't really need to have the thread exit (and just want it to keep from keeping your application running) you can set Thread.IsBackground to true and it will end when all non background threads end. Will and Marc both have good solutions for handling the queue.

Grab the Parallel Framework. It has a BlockingCollection<T> which you can use as a job queue. How you'd use it is:
Create the BlockingCollection<T> that will hold your tasks/jobs.
Create some Threads which have a never-ending loop (while(true){ // get job off the queue)
Set the threads going
Add jobs to the collection when they come available
The threads will be blocked until an item appears in the collection. Whoever's turn it is will get it (depends on the CPU). I'm using this now and it works great.
It also has the advantage of relying on MS to write that particularly nasty bit of code where multiple threads access the same resource. And whenever you can get somebody else to write that you should go for it. Assuming, of course, they have more technical/testing resources and combined experience than you.

I've implemented a background-task queue without using any kind of while loop, or pulsing, or waiting, or, indeed, touching Thread objects at all. And it seems to work. (By which I mean it's been in production environments handling thousands of tasks a day for the last 18 months without any unexpected behavior.) It's a class with two significant properties, a Queue<Task> and a BackgroundWorker. There are three significant methods, abbreviated here:
private void BackgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
if (TaskQueue.Count > 0)
{
TaskQueue[0].Execute();
}
}
private void BackgroundWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
Task t = TaskQueue[0];
lock (TaskQueue)
{
TaskQueue.Remove(t);
}
if (TaskQueue.Count > 0 && !BackgroundWorker.IsBusy)
{
BackgroundWorker.RunWorkerAsync();
}
}
public void Enqueue(Task t)
{
lock (TaskQueue)
{
TaskQueue.Add(t);
}
if (!BackgroundWorker.IsBusy)
{
BackgroundWorker.RunWorkerAsync();
}
}
It's not that there's no waiting and pulsing. But that all happens inside the BackgroundWorker. This just wakes up whenever a task is dropped in the queue, runs until the queue is empty, and then goes back to sleep.
I am far from an expert on threading. Is there a reason to mess around with System.Threading for a problem like this if using a BackgroundWorker will do?

Is Thread.Sleep the proper way to implement my own Timer in C#?

I am aware that System.Threading.Timer exists, but I already have a Thread. This thread is supposed to stay alive all the time, but only execute every X seconds. The test implementation looks like this:
public class MailClass
{
private Action<string> LoggerAction;
private bool _exit;
public MailClass(Action<string> loggerAction)
{
LoggerAction = loggerAction;
}
public void Run()
{
LoggerAction("Run called");
_exit = false;
while(!_exit)
{
Thread.Sleep(TimeSpan.FromSeconds(300));
LoggerAction("Waking up");
}
LoggerAction("Run ended");
}
public void Stop()
{
LoggerAction("Stop called");
_exit = true;
}
}
The Run method executes, then sleeps for 5 Minutes, then executes again. So it's basically a timer that fires every 5 Minutes + the time it takes to execute the action. (and yes, I should cache the TimeSpan instead of re-creating it over and over)
Is this the proper way to do it? (In the real app, the Run action checks a Web Service, so I have no way to signal my Thread to wake up earlier)
Or should I use some other concept to have the thread? One problem I see is the implementation of Stop. The Run Thread runs a loop that checks a bool every time, but if I call Stop() I have to wait until the Sleep Interval is over, which is inconvenient.
Thread.Abort would be harsh, so I guess Thread.Interrupt would work somehow? The Stop() Method should allow Run to finish it's current iteration, so no hard abort. AutoResetEvent looks a bit like what I could need, but I don't fully understand what it does.
Edit: One way I would see this possible is to add a Timer (so a separate thread) and then have Run() end not with Thread.Sleep but with some "Wait until some object changes". I would then change that object either from the second Thread (when the 5 minutes expire) or from the Stop action. But that seems excessive? Essentially, Run needs to react to two conditions: 5 Minutes expire or some external signal (like the change of the _exit flag). Something tells me there should be something built-in, but maybe having another Timer Thread solely focused on sending a signal every 5 minutes is the way to go?

If you're forced to poll, then you're forced to poll. Thread.Sleep() is fine for that.
However with regards to you're interrupt concerns...
I'd re-write your solution a bit to use Monitor.Wait/Pulse. That does require you keep an object around solely to lock(...){} on it, but it strikes me as a cleaner solution.
I say cleaner because using Thread.Interrupt() is effectively using exceptions for "normal" control flow. Stopping a Timer is in no way unexpected. But its a design smell really (if such things exist), nothing more.
Quicky outline:
//Instead of Thread.Sleep(FIVE_MIN) in Run()...
lock(some_obj)
{
if(Monitor.Wait(some_obj, FIVE_MIN)) //Wait for 5 min (or whatever) or until some_obj is Pulse'd
{
//Got Pulse
}
else
{
//Timeout expired
}
}
//And in Stop()...
_exit = true;
lock(some_obj)
{
Monitor.Pulse(some_obj); //Wakeup the thread in Run() if it's currently Wait'ing
}

yeah that's cool, you can also call Thread.Interrupt() to interrupt the sleep, rather than waiting for sleep to return normally.
in the case the thread is not blocking when you interrupt it, it will continue processing normally until it tries to sleep again.

Is there a reason you couldn't just use a timer inside the thread? You'd get what you want, a thread that stays alive forever while firing off your method, plus you could just stop the timer at any point without waiting for 5 minutes or interrupting threads?
(I'm not very experienced in threading, so I might be missing something obvious?)

If time interval is critical then prefer high resolution timers provided in windows which will trigger with higher accuracy.

Seems like a good solution to me. If you're worried about stopping sooner, you can set the sleep time to be less and keep a count so you only run the actual code every 5 minutes. That way it's checking the boolean more often and can break out sooner.

You could look into System.Timers.Timer as well, though truthfully just sleeping is not a bad solution.