Should you code in case the threads die or freeze? - c#

I have an app that starts 10 threads.
Each thread does it's work and waits (using the producer consumer model).
So when more work comes along, one of the threads is unblocked and it does the work.
A colleague at work insists I should write some code that monitors the threads "just in case" they freeze/do not respond or die.
So far in my testing, they work fine and close down correctly.
My question is "should I put code in to do this"? If so, "how do I monitor a thread and check it's status"?
Thanks.
JD

It really depends on the situation. First of all, you should focus on correctness so that it doesn't freeze or die but if you need more reliability, you should first think about how you can gracefully recover from such a situation. You should think about why it might freeze or die and if it did, now what you can do. If you can't do anything good that reliably recovers from such a situation, you shouldn't even try. If can do it without making the situation worse, then you can go and try doing so.
Obviously, if you made such a choice, you have to be careful not to mess things up and introduce some bugs that actually make the bad stuff happen yourself.

Ideally no, your threads should be able to finish properly. Monitoring them is not worth the complexity and processing time. If you do it correctly you won't need monitoring.

Assuming you're coding against the .NET framework 2 or above, an unhandled exception in a worker thread will kill the entire app.
Unless you're absolutely sure your thread code won't throw an exception (or reserve memory, or grab a handle, or call framework code you didn't write...) the first thing you'll want to write in your thread entry method is a try/catch.

If your threads spend a lot of time waiting for work, you could consider refactoring your app to use the System.Threading.ThreadPool class. Doing that would offload much of your thread management to the framework.

Threads don't just hang or die unless there is a bug in the user code. The chances of the OS/Framework screwing up is so negligible that it's, for practical purposes, impossible. It's like worry about the hard disk not writing a file even though File.WriteAllBytes() succeeds. It just doesn't happen.
That said, wildcard brings up a good point that an unhandled exception in a worker thread populates up as an unhandled exception through the entire AppDomain.

Best way to do is use "ThreadPool" and call ThreadPool.QueueUserWorkItem and .Net manages the ThreadPool very efficiently.
You can set maximum number of threads to allocate and you dont need to do any wait operation etc. You can also get WaitHandle to wait upon completion. And you dont need to manage the blocking/unblocking of threads.

You should always code for the chance that the thread may hang or die, even if all you do is forcibly restart the thread.

Related

Correct way to close a foreground thread

When closing an application, what is the best way to close foreground threads at the same time?
From all my research, it seems to be that the best way is to set IsBackground to true..
Is this correct or is there a better way? I'm skeptical as it doesn't seem to follow Microsofts normally descriptive method name pattern.
It will certainly work, the CLR will abort a thread like that when your main startup exits. Pretty similar to using Thread.Abort(), minus the drastic failure modes you'd normally suffer from using Abort(). It is a rude abort, there's nothing that the thread itself can do to stop it. And there will be no more code that runs afterwards that could fail due to the typical problems you'd get from Abort(), like a deadlock. Other than finalizers.
Calling it "correct" is however a bit of a stretch, there's also nothing that the thread can do to terminate cleanly. Which might be detrimental if it has externally observable behavior. Like updating a dbase, talking over a socket or writing a file. That's rudely interrupted as well, potentially leaving a confused server or a half-written file that can cause trouble later. A mitigating circumstance is that this will also happen when your program dies on an unhandled exception, you'd expect it to not cause trouble either. It depends :)

Looking at what happens when a c#/ASP.NET thread is terminated and how to get around problems

I'm working on a ASP.NET website that on some requests will run a very lengthy caching process. I'm wondering what happens exactly if the execution timeout is reached while it is still running in terms of how the code handles it.
Particularly I am wondering about things like if the code is in the try of a try/finally block will the finally still be run?
Also given I am not sure I want the caching to terminate even if it goes on that long is there a way with spawning new threads, etc. that I can circumvent this execution timeout? I am thinking it would be much nicer to return to the user immediately and say "a cache build is happening" rather than just letting them time out. I have recently started playing with some locking code to make sure only one cache build happens at a time but am thinking about extending this to make it run out of sync.
I've not really played with creating threads and such like myself so am not sure exactly how they work, particularly in terms of interacting with ASP.NET. eg if the parent thread that launched it is terminated will that have any effect on the spawned thread?
I know there is kind of a lot of different questions in here and I can split them if that is deemed best but they all seem to go together... I'll try to summarise the questions though:
Will a finally block still be executed if a thread is terminated by ASP.NET while in the try block
Would newly created threads be subject to the same timeouts as the original thread?
Would newly created threads die at the same time as the parent thread that created them?
And the general one of what is the best way to do long running background processes on an ASP.NET site?
Sorry for some noobish questions, I've never really played with threads and they still intimidate me a bit (my brain says they are hard). I could probably test the answer to a lot of tehse questions but I wouldn't be confident enough of my tests. :)
Edit to add:
In response to Capital G:
The problem I have is that the ASp.NET execution timeout is currently set to one hour which is not always long enough for some of these processes I reckon. I've put some stuff in with locks to prevent more than one person setting off these long processes and I was worried the locks might not be released (which if finally blocks aren't always run might happen I guess).
Your comments on not running long processes in ASP.NET is why I was thinking of moving them to other threads rather than blocking the request thread but I don't know if that still counts as running within the ASP.NET architecture that you said was bad.
The code is not actually mine so I'm not allowed (and not sure I 100% understand it enough) to rework it into a service though that is certainly where it would best live.
Would using a BackgroundWorker process for something that could take an hour be feasible in this situation (with respect to comments on long running processes in ASP.NET). I would then make request return a "Cache is building" page until its finished and then go back to serving normally... Its all a bit of a nightmare but its my job so I've got to find a way to improve it. :)
Interesting question, just tested and no it's not guaranteed to execute the code in the finally block, if a thread is aborted it could stop at any point in the processing. You can design some sanity checking and other mechanisms to handle special cleanup routines and such but it has a lot to do with your thread handling as well.
Not necessarily, it depends on how your implementing your threads. If you are working with threads yourself, then you can easily get into situations where the parent thread is killed while it's child threads are still out there processing, you generally want to do some cleanup in the parent thread that ends the child threads as well. Some objects might do a lot of this for you as well, so it's a tough call to say one way or the other. Never assume this at the very least.
No, not necessarily, don't assume this at least, again has to do with your design and whether your doing threading yourself or using some higher level threading object/pattern. I would never assume this regardless.
I don't recommend long running processes within the ASP.NET architecture, unless its within the typical timeout, if it's 10-20s okay but if it's minutes, no, the reason is resource usage within ASP.NET and it's awfully bad on a user. That being said you could perform asynchronous operations where you hand off the work to the server, then you return back to the user when the processing is finished, (this is great for those 10-20s+ processes), the user can be given a little animation or otherwise not have their browser all stuck for that long waiting for whatever is happening on the server to happen.
If it is a long running process, things that take greater than 30-60s+, unless it absolutely has to be done in ASP.NET due to the nature of the process, I suggest moving it to a windows service and schedule it in some way to occur when required.
Note: Threading CAN be complicated, it's not that it's hard so much as that you have to be very aware of what your doing, which requires a firm understanding of what threads are and how they work, I'm no expert, but I'm also not completely new and I'll tell you that in most situations you don't need to get into the realm of threading, even when it seems like you do, if you must however, I would suggest looking into the BackgroundWorker object as they are simplified for the purposes of doing batched processing etc. (honestly for many situations that DO need threads, this is usually a very simple solution).
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
Long or time consuming processes to be started behind the web-page; it should not hit the ASP.NET execution time out; the user page should be freed; running the requests under lock etc. All these situation points towards using async services. In one of the products, where I architected, used services for such scenarios. The service exposes some async method to initiate. The status of the progress can be queried using another method. Every request is given some id and no duplicate requests are fired ever. The progress proceeds even if the user logs out. The user can see the results at a later time.
If you have looked at such options already, let me know if there is any issue. Or if you are yet to look in this direction, please get it this way. For any help, just send in your comments.

Why is the explicit management of threads a bad thing?

In a previous question, I made a bit of a faux pas. You see, I'd been reading about threads and had got the impression that they were the tastiest things since kiwi jello.
Imagine my confusion then, when I read stuff like this:
[T]hreads are A Very Bad Thing. Or, at least, explicit management of threads is a bad thing
and
Updating the UI across threads is usually a sign that you are abusing threads.
Since I kill a puppy every time something confuses me, consider this your chance get your karma back in the black...
How should I be using thread?
Enthusiam for learning about threading is great; don't get me wrong. Enthusiasm for using lots of threads, by contrast, is symptomatic of what I call Thread Happiness Disease.
Developers who have just learned about the power of threads start asking questions like "how many threads can I possible create in one program?" This is rather like an English major asking "how many words can I use in a sentence?" Typical advice for writers is to keep your sentences short and to the point, rather than trying to cram as many words and ideas into one sentence as possible. Threads are the same way; the right question is not "how many can I get away with creating?" but rather "how can I write this program so that the number of threads is the minimum necessary to get the job done?"
Threads solve a lot of problems, it's true, but they also introduce huge problems:
Performance analysis of multi-threaded programs is often extremely difficult and deeply counterintuitive. I've seen real-world examples in heavily multi-threaded programs in which making a function faster without slowing down any other function or using more memory makes the total throughput of the system smaller. Why? Because threads are often like streets downtown. Imagine taking every street and magically making it shorter without re-timing the traffic lights. Would traffic jams get better, or worse? Writing faster functions in multi-threaded programs drives the processors towards congestion faster.
What you want is for threads to be like interstate highways: no traffic lights, highly parallel, intersecting at a small number of very well-defined, carefully engineered points. That is very hard to do. Most heavily multi-threaded programs are more like dense urban cores with stoplights everywhere.
Writing your own custom management of threads is insanely difficult to get right. The reason is because when you are writing a regular single-threaded program in a well-designed program, the amount of "global state" you have to reason about is typically small. Ideally you write objects that have well-defined boundaries, and that do not care about the control flow that invokes their members. You want to invoke an object in a loop, or a switch, or whatever, you go right ahead.
Multi-threaded programs with custom thread management require global understanding of everything that a thread is going to do that could possibly affect data that is visible from another thread. You pretty much have to have the entire program in your head, and understand all the possible ways that two threads could be interacting in order to get it right and prevent deadlocks or data corruption. That is a large cost to pay, and highly prone to bugs.
Essentially, threads make your methods lie. Let me give you an example. Suppose you have:
if (!queue.IsEmpty) queue.RemoveWorkItem().Execute();
Is that code correct? If it is single threaded, probably. If it is multi-threaded, what is stopping another thread from removing the last remaining item after the call to IsEmpty is executed? Nothing, that's what. This code, which locally looks just fine, is a bomb waiting to go off in a multi-threaded program. Basically that code is actually:
if (queue.WasNotEmptyAtSomePointInThePast) ...
which obviously is pretty useless.
So suppose you decide to fix the problem by locking the queue. Is this right?
lock(queue) {if (!queue.IsEmpty) queue.RemoveWorkItem().Execute(); }
That's not right either, necessarily. Suppose the execution causes code to run which waits on a resource currently locked by another thread, but that thread is waiting on the lock for queue - what happens? Both threads wait forever. Putting a lock around a hunk of code requires you to know everything that code could possibly do with any shared resource, so that you can work out whether there will be any deadlocks. Again, that is an extremely heavy burden to put on someone writing what ought to be very simple code. (The right thing to do here is probably to extract the work item in the lock and then execute it outside the lock. But... what if the items are in a queue because they have to be executed in a particular order? Now that code is wrong too because other threads can then execute later jobs first.)
It gets worse. The C# language spec guarantees that a single-threaded program will have observable behaviour that is exactly as the program is specified. That is, if you have something like "if (M(ref x)) b = 10;" then you know that the code generated will behave as though x is accessed by M before b is written. Now, the compiler, jitter and CPU are all free to optimize that. If one of them can determine that M is going to be true and if we know that on this thread, the value of b is not read after the call to M, then b can be assigned before x is accessed. All that is guaranteed is that the single-threaded program seems to work like it was written.
Multi-threaded programs do not make that guarantee. If you are examining b and x on a different thread while this one is running then you can see b change before x is accessed, if that optimization is performed. Reads and writes can logically be moved forwards and backwards in time with respect to each other in single threaded programs, and those moves can be observed in multi-threaded programs.
This means that in order to write multi-threaded programs where there is a dependency in the logic on things being observed to happen in the same order as the code is actually written, you have to have a detailed understanding of the "memory model" of the language and the runtime. You have to know precisely what guarantees are made about how accesses can move around in time. And you cannot simply test on your x86 box and hope for the best; the x86 chips have pretty conservative optimizations compared to some other chips out there.
That's just a brief overview of just a few of the problems you run into when writing your own multithreaded logic. There are plenty more. So, some advice:
Do learn about threading.
Do not attempt to write your own thread management in production code.
Use higher-level libraries written by experts to solve problems with threads. If you have a bunch of work that needs to be done in the background and want to farm it out to worker threads, use a thread pool rather than writing your own thread creation logic. If you have a problem that is amenable to solution by multiple processors at once, use the Task Parallel Library. If you want to lazily initialize a resource, use the lazy initialization class rather than trying to write lock free code yourself.
Avoid shared state.
If you can't avoid shared state, share immutable state.
If you have to share mutable state, prefer using locks to lock-free techniques.
Explicit management of threads is not intrinsically a bad thing, but it's frought with dangers and shouldn't be done unless absolutely necessary.
Saying threads are absolutely a good thing would be like saying a propeller is absolutely a good thing: propellers work great on airplanes (when jet engines aren't a better alternative), but wouldn't be a good idea on a car.
You cannot appreciate what kind of problems threading can cause unless you've debugged a three-way deadlock. Or spent a month chasing a race condition that happens only once a day. So, go ahead and jump in with both feet and make all the kind of mistakes you need to make to learn to fear the Beast and what to do to stay out of trouble.
There's no way I could offer a better answer than what's already here. But I can offer a concrete example of some multithreaded code that we actually had at my work that was disastrous.
One of my coworkers, like you, was very enthusiastic about threads when he first learned about them. So there started to be code like this throughout the program:
Thread t = new Thread(LongRunningMethod);
t.Start(GetThreadParameters());
Basically, he was creating threads all over the place.
So eventually another coworker discovered this and told the developer responsible: don't do that! Creating threads is expensive, you should use the thread pool, etc. etc. So a lot of places in the code that originally looked like the above snippet started getting rewritten as:
ThreadPool.QueueUserWorkItem(LongRunningMethod, GetThreadParameters());
Big improvement, right? Everything's sane again?
Well, except that there was a particular call in that LongRunningMethod that could potentially block -- for a long time. Suddenly every now and then we started seeing it happen that something our software should have reacted to right away... it just didn't. In fact, it might not have reacted for several seconds (clarification: I work for a trading firm, so this was a complete catastrophe).
What had ended up happening was that the thread pool was actually filling up with long-blocking calls, leading to other code that was supposed to happen very quickly getting queued up and not running until significantly later than it should have.
The moral of this story is not, of course, that the first approach of creating your own threads is the right thing to do (it isn't). It's really just that using threads is tough, and error-prone, and that, as others have already said, you should be very careful when you use them.
In our particular situation, many mistakes were made:
Creating new threads in the first place was wrong because it was far more costly than the developer realized.
Queuing all background work on the thread pool was wrong because it treated all background tasks indiscriminately and did not account for the possibility of asynchronous calls actually being blocked.
Having a long-blocking method by itself was the result of some careless and very lazy use of the lock keyword.
Insufficient attention was given to ensuring that the code that was being run on background threads was thread-safe (it wasn't).
Insufficient thought was given to the question of whether making a lot of the affected code multithreaded was even worth doing to begin with. In plenty of cases, the answer was no: multithreading just introduced complexity and bugs, made the code less comprehensible, and (here's the kicker): hurt performance.
I'm happy to say that today, we're still alive and our code is in a much healthier state than it once was. And we do use multithreading in plenty of places where we've decided it's appropriate and have measured performance gains (such as reduced latency between receiving a market data tick and having an outgoing quote confirmed by the exchange). But we learned some pretty important lessons the hard way. Chances are, if you ever work on a large, highly multithreaded system, you will too.
Unless you are on the level of being able to write a fully-fledged kernel scheduler, you will get explicit thread management always wrong.
Threads can be the most awesome thing since hot chocolate, but parallel programming is incredibly complex. However, if you design your threads to be independent then you can't shoot yourself in the foot.
As fore rule of the thumb, if a problem is decomposed into threads, they should be as independent as possible, with as few but well defined shared resources as possible, with the most minimalistic management concept.
I think the first statement is best explained as such: with the many advanced APIs now available, manually writing your own thread code is almost never necessary. The new APIs are a lot easier to use, and a lot harder to mess up!. Whereas, with the old-style threading, you have to be quite good to not mess up. The old-style APIs (Thread et. al.) are still available, but the new APIs (Task Parallel Library, Parallel LINQ, and Reactive Extensions) are the way of the future.
The second statement is from more of a design perspective, IMO. In a design that has a clean separation of concerns, a background task should not really be reaching directly into the UI to report updates. There should be some separation there, using a pattern like MVVM or MVC.
I would start by questioning this perception:
I'd been reading about threads and had got the impression that they were the tastiest things since kiwi jello.
Don’t get me wrong – threads are a very versatile tool – but this degree of enthusiasm seems weird. In particular, it indicates that you might be using threads in a lot of situations where they simply don’t make sense (but then again, I might just mistake your enthusiasm).
As others have indicated, thread handling is additionally quite complex and complicated. Wrappers for threads exist and only in rare occasions do they have to be handled explicitly. For most applications, threads can be implied.
For example, if you just want to push a computation to the background while leaving the GUI responsive, a better solution is often to either use callback (that makes it seem as though the computation is done in the background while really being executed on the same thread), or by using a convenience wrapper such as the BackgroundWorker that takes and hides all the explicit thread handling.
A last thing, creating a thread is actually very expensive. Using a thread pool mitigates this cost because here, the runtime creates a number of threads that are subsequently reused. When people say that explicit management of threads is bad, this is all they might be referring to.
Many advanced GUI Applications usually consist of two threads, one for the UI, one (or sometimes more) for Processing of data (copying files, making heavy calculations, loading data from a database, etc).
The processing threads shouldn't update the UI directly, the UI should be a black box to them (check Wikipedia for Encapsulation).
They just say "I'm done processing" or "I completed task 7 of 9" and call an Event or other callback method. The UI subscribes to the event, checks what has changed and updates the UI accordingly.
If you update the UI from the Processing Thread you won't be able to reuse your code and you will have bigger problems if you want to change parts of your code.
I think you should experiement as much as possible with Threads and get to know the benefits and pitfalls of using them. Only by experimentation and usage will your understanding of them grow. Read as much as you can on the subject.
When it comes to C# and the userinterface (which is single threaded and you can only modify userinterface elements on code executed on the UI thread). I use the following utility to keep myself sane and sleep soundly at night.
public static class UIThreadSafe {
public static void Perform(Control c, MethodInvoker inv) {
if(c == null)
return;
if(c.InvokeRequired) {
c.Invoke(inv, null);
}
else {
inv();
}
}
}
You can use this in any thread that needs to change a UI element, like thus:
UIThreadSafe.Perform(myForm, delegate() {
myForm.Title = "I Love Threads!";
});
A huge reason to try to keep the UI thread and the processing thread as independent as possible is that if the UI thread freezes, the user will notice and be unhappy. Having the UI thread be blazing fast is important. If you start moving UI stuff out of the UI thread or moving processing stuff into the UI thread, you run a higher risk of having your application become unresponsive.
Also, a lot of the framework code is deliberately written with the expectation that you will separate the UI and processing; programs will just work better when you separate the two out, and will hit errors and problems when you don't. I don't recall any specifics issues that I encountered as a result of this, though I have vague recollections of in the past trying to set certain properties of stuff the UI was responsible for outside of the UI and having the code refuse to work; I don't recall whether it didn't compile or it threw an exception.
Threads are a very good thing, I think. But, working with them is very hard and needs a lot of knowledge and training. The main problem is when we want to access shared resources from two other threads which can cause undesirable effects.
Consider classic example: you have a two threads which get some items from a shared list and after doing something they remove the item from the list.
The thread method that is called periodically could look like this:
void Thread()
{
if (list.Count > 0)
{
/// Do stuff
list.RemoveAt(0);
}
}
Remember that the threads, in theory, can switch at any line of your code that is not synchronized. So if the list contains only one item, one thread could pass the list.Count condition, just before list.Remove the threads switch and another thread passes the list.Count (list still contains one item). Now the first thread continues to list.Remove and after that second thread continues to list.Remove, but the last item already has been removed by the first thread, so the second one crashes. That's why it would have to be synchronized using lock statement, so that there can't be a situation where two threads are inside the if statement.
So that is the reason why UI which is not synchronized must always run in a single thread and no other thread should interfere with UI.
In previous versions of .NET if you wanted to update UI in another thread, you would have to synchronize using Invoke methods, but as it was hard enough to implement, new versions of .NET come with BackgroundWorker class which simplifies a thing by wrapping all the stuff and letting you do the asynchronous stuff in a DoWork event and updating UI in ProgressChanged event.
A couple of things are important to note when updating the UI from a non-UI thread:
If you use "Invoke" frequently, the performance of your non-UI thread may be severely adversely affected if other stuff makes the UI thread run sluggishly. I prefer to avoid using "Invoke" unless the non-UI thread needs to wait for the UI-thread action to be performed before it continues.
If you use "BeginInvoke" recklessly for things like control updates, an excessive number of invocation delegates may get queued, some of which may well be pretty useless by the time they actually occur.
My preferred style in many cases is to have each control's state encapsulated in an immutable class, and then have a flag which indicates whether an update is not needed, pending, or needed but not pending (the latter situation may occur if a request is made to update a control before it is fully created). The control's update routine should, if an update is needed, start by clearing the update flag, grabbing the state, and drawing the control. If the update flag is set, it should re-loop. To request another thread, a routine should use Interlocked.Exchange to set the update flag to update pending and--if it wasn't pending--try to BeginInvoke the update routine; if the BeginInvoke fails, set the update flag to "needed but not pending".
If an attempt to control occurs just after the control's update routine checks and clears its update flag, it may well happen that the first update will reflect the new value but the update flag will have been set anyway, forcing an extra screen redraw. On the occasions when this happens, it will be relatively harmless. The important thing is that the control will end up being drawn in the correct state, without there ever having been more than one BeginInvoke pending.

what type of bug causes a program to slowly use more processor power and all of a sudden go to 100%?

I was hoping to get some good ideas as to what might be causing a really nasty bug.
This is a program which is transmitting data over a socket, and also receives messages back.
I could explain lots more, but I don't think this will help here.
I'm just searching for hypothetical problems which can cause the following behaviour:
program runs
processor time slowly accumulates (till around 60%)
all of a sudden (could be after 30 but also after 60 seconds) the processor time shoots to 100%. the program halts completely
In my syslog it always ends on one line with a memory allocation (something similar to: myArray = new byte[16384]) in the same thread.
now here is the weird part: if I set the debugger anywhere...it immediately stops on that line. So just the act of setting a breakpoint, made the thread continue (it wasn't running since I saw no log output anymore)
I was thinking 'deadlock' but that would not cause 100% processor power. If anything, the opposite. Also, setting a breakpoint would not cause a deadlock to end.
anyone else a theoretical suggestion as to what kind of 'construct' might cause this effect?
(apart from 'bad programming') ;^)
thanks
EDIT:
I just noticed.... by setting the sendspeed slower, the problem shows itself much later than expected. I would think around the same amount of packets send...but no the amount of packets send is much higher this way before it has the same problem.
I can only guess, but the opposite of a deadlock would be a livelock. This means two threads who react to each other in an infinite loop. This could also be possibly interrupted by setting a break point, as livelocks generally depend on the right timing.
Other than this I had once a similar issue with the Java nio classes which are non-blocking which caused the main thread to busy wait for input. Although the CPU usage rose instantaneously, not just after a few seconds.
Maybe if you can provide a bit more information like the programming language or even a code sample there might be more ideas.
Anything that involves repetitive processing (looping, recursion, etc) can cause this.
What's interesting is that if the program is doing anything that normally slows down performance (such as disk IO or network access), then the processor is less likely to peg . The processor pegs at 100% only if the program is using the processor. If you have to wait for disk or network IO, then the processor thread has to wait.
So in the code, I'd check for loops where a lot of work is going on, but little IO.
Also, if you're debugging in Visual Studio, you can hit the pause button to stop the app at the current point and see what your code is doing when it locks.
I'm guessing an infinite loop in the socket receiving end. It keeps trying to allocate a buffer to receive the data that is coming in, but the buffer is never big enough so it keeps allocating. But it is really hard to say without code. I'd advise you to add more logging and/or single step the code if you don't want to share it.
Without seeing code, I only can say your program is probably infinite looping and the call that should block is not blocking correctly as you're expecting
You can also try profiling (EQUATEC free profiler, for example). If will show you how much of your processor time was spent in each method.
I found the answer... quite silly actually (it always is). The thread which is sending/receiving messages is doing this via asynchronous methods. However, the asynchronous callbacks never seem to be able to come through while the thread is also pumping messages in the sendqueue. I notice when I put a thread.sleep every second, all asynchronous callbacks are pumped through. So the solution it turns out is to have a separate thread for sending/receiving, done purely on async, and another one for filling the sendqueue.
why this would have resulted in 100% processor power is beyond me. But it does actually explain why setting a breakpoint allowed the async callbacks to catch up.
Because the program fails while allocating memory I would guess that the incoming message rate is too high for it to handle.
I imagine that your program has some thread that it's only job is to listen to the socket and send the incoming messages to some other threads to handle (maybe you have some thread pool there). Imagine a situation where the incoming message rate is too high so all the worker threads are busy handling previous messages and the thread that listen to the socket have to put the new messages into some kind of queue until one of the worker threads will be free to handle them. this queue will grow and grow until you won't have additional memory. so that could be the reason for your program's termination.
now, about the 100% CPU. I guess that the thread the uses the CPU must be one of the worker threads. this will explain why the listening thread is queuing the messages. the reason can be a corrupted message or something else that causes it to run into an infinite loop. "frenetisch applaudierend" suggested in his answer that two or more of the worker threads can cause "livelock" on each other which could also be the reason for your problem.

Is there any way to stop an application shutting down with work still in the system Threadpool

Final question for today :) thanks for your input on the previous ones.
BTW: Already searched the forum on this and nothing quite answers this one.
We use some 3rd party libraries which pop work onto the Threadpool and we don't want to shutdown while there are outstanding activities.
When shutting down an application, the application will shutdown even if there is work outstanding in the System.Threading.ThreadPool because these threads are back ground threads.
Other than doing some form of reference counting (and enforcing every item Queued into the pool to use this mechanism, which we don't have control over) is there some way to stop the application shutting down while there is outstanding work to be done.
I've looked at using the GetAvailableThreads() vs GetMaxThreads(), but these can be the same because we may have caught the Threadpool as a thread was freed up but not allocated a queued workitem.
All help appreciated?
Kind Regards
Noel
Unless you can get some kind of callback into the 3rd-party code (maybe a completion event), you are going to struggle to know when they have stopped using the ThreadPool. Sorry. Such code should typically be on a non-background thread. I realise you don't have control over this - I'm just not sure that there is an easy way out of the hole, either.
Since I'm real programmer (that's C, boys ;-) I've not used ThreadPool.
However, a suggestion comes to mind - have a semaphore increment on each task you issue into threadpool, and that that semaphore be released on completion.
When your app comes to exit, wait on the semaphore.
Any use?
I think Marc Gravell is right. Though if you could get access to the threads being used in the thread pool somehow then you could change the Thread.IsBackground property to false to achieve what you want. But I am not sure how you would obtain a reference to a managed thread within the thread pool.
Thanks for your replies. I've found a work-around outside of the thread pool using AOP and Castle Windsors dynamic proxy to inject referencing counting around the class. Luckily for me the class exposed a clean interface that I could wrap and it works a treat.

Categories

Resources