I have multiple startup projects in my client/server solution. The server is a Console app and the client is a WinForms app. The server/console is launched first in case that matters.
Now there is a deadlock caused by some synchronization client-side code that blocks the server. Thread synchronization is done using simple lock statements.
When the deadlock occurs, both apps freeze of course and hitting pause/break in VS only breaks the server app, not the client. There are two questions here:
How could I choose which project to break out of multiple start up projects?
If a lock statement is stuck in a deadlock, is there a way to find out which line of code has a current lock on that object?
I think your best solution would be to debug your client and server in separate instances of visual studio and setting startup projects accordingly.
As for the second question, I normally set a guid and output on create and release of a lock. to see if this is happening. If it is, I set breakpoints and debug and look at the stack to see where on the calls are coming from. You may be able to output System.Environment.StackTrace to a log to get this information, but I've ever attempted it.
You can use 2 Visual studios. One starting the console, one starting the server
I Would check if you really need a lock statement.
What do you want to lock ?
Do you always need a exclusive lock ?
Or are there some operations which can happen in parallel and only some which are exclusive ?
In this case you could use the ReaderWriterLockSlim
This can reduce the risk of deadlocks.
Related
I am currently health checking an application which experiences UI stuttering during heavy usage.
Using Microsoft Concurrency Visualizer extension for Visual Studio 2015, showed that quite a lot of short-lived threads are created and stopped after ~100ms of execution.
Unfortunately, their displayed callstack is like clr.dll!0x98071 ntdll.dll!0x634fb and I am not quite sure how to extract useful information out of it.
I have no clue what is the purpose of those threads and which part of the code in the application is creating them.
How can I better identify where each one of them gets started?
In the code, I was able to grep a handful of Tasks, another of QueueUserWorkItems, several dozens of plain Thread instantiations, some System.Threading.Timer & System.Timers.Timer, no Reactive Extensions. I put breakpoints for all of them but it seems I am missing some...
I don't think those are from the threadpool because they would be displayed in synchronisation state in concurrency visualizer, instead they just end, and another one with another Id gets created later. But maybe I am misleading.
We also use a few third-party libs and a bunch of JuggerNET generated code, so maybe the origin is not even in the application itself.
I was finally able to find the culprit of those short-lived threads by looking closely at some of the cryptic callstacks, which included for instance:
mmdevapi.dll
wdmaud.drv
avrt.dll
audioses.dll
It led me to thinking that I should double check the sound alert system. It was indeed this one which spawned those threads.
Note:
I will not accept my answer however because I would like someone to share a better process or any kind of tips and tricks for diagnosing unwanted threads origin.
I have a complex system with several threads. sometimes i see the application in 100% cpu and force to restart the system. I have no idea which thread caused it and which code caused it.
I need something that will give me the state of each thread in the system (i.e. in which line the thread is now) so i can find which code causes the 100% CPU
(in java you have the thread dump kill -3 which gives you the state of each thread)
Can you help please?
Tess's blog has some great debugging tutorials, including:
.NET Hang Debugging Walkthrough
People have suggested Process Explorer to me before.
You could use the debugger to break and then find out what all threads are doing. (Add the Debug Location toolbar to Visual Studio)
Another option is to remove all thread one by one and find the guilty one.
I have found that in cases like this one of the best tools around these days in Microsoft's intellitrace. This allows for historical debugging & will give you the state of all threads etc. when you break execution.
Unfortunatly its only available in Visual Studio 2010 Ultimate edition, but if this is a really critical issue and you don't have this edition you could always download a 30 day evaluation.
Use VS debugger to attach to your process, and then press the "break" (pause symbol) to break execution. In this state, you can open the Debug window called "Threads" which should give you the state of each thread, and which line they're currently executing. It also helps at this point to give explicit names to your threads when debugging them.
I find that 99% of the time (at least for me) its because I accidentally make a loop infinite when I don't mean to make it so or there should be at least a few milliseconds of sleep before the the loop continues.
You can use profiler like visual vm to better debug the behavior of threads in your application.
I'm using .Net 2008 and Oracle 10g as my database. The problem I'm getting is after deploying the application in IIS, when multiple users access the same page at a time i'm getting the error. Can't get the output.
Note: Both the users accessing the same page, same menu at a time.
How can I resolve this?
My guess would be a standard thread-safety / synchronization bug, most likely due to some static resource such as a static connection. Obviously this is pure speculation without some more code, but it (=web-sites being highly threaded) is a surprisingly common oversight.
If it is a static resource, then... well, it probably shouldn't be static. Either per-request, or (specifically in the case of connections) scoped to the local code (and let the connection-pooling worry about re-use).
Does it "Work on your machine"? ;)
If not, try to deploy a version locally and attach the debugger to iis. Point two browsers at the site. Whenever your browsers are/seem stuck, open the debugger's threads window and see where the threads are blocked/blocking. You can also ask the debugger to stop on exception throwing
You mean that nothing appears in the Browser?
Look in your program's logs. Any error messages?
Put some trace statements into your code so that you can figure out where it's going.
So the error is saying there's a failure to create a table. Would you expect to create a table for each user? Have a look at the code around the table creation. Consider what the correct behaviour should be when two copies of that code run at the same time.
Again add trace into the code at these points so you can see what happening. Often it's easier to see this than to debug because when mutiple threads are running the debugger gets in the way of reality.,
When I am working on multithreads how can I debug to know which thread causes an abnormal behavior?
Can I use permonitor for debugging, or are there any other tools or debugging facilities that are available?
Tips for Debugging Threads From MSDN
Neat New Multithreaded Debugging Features in VS 2008 by John Robbin
As an alternative to debugging, you could do thread-related testing. The book The Art of Unit Testing has a section on this in Appendix B. The author mentions three tools (two of which he has a personal interest in):
Typemock Racer
Microsoft CHESS
Osherove.ThreadTester
You can use visual studio to set up breakpoints on certain threads. See here and here for how to do it.
It depends what do you mean by "abnormal behavior"...
for most of the time, the visual studio debugger should be enough. the Threads and CallStack windows will give you a lot of information about what is going on.
for the heavy duty stuff you can use WinDbg+SOS. read about the !threads, !threadpool and !runaway commands.
If you have several threads of the same type* you could modify your code to only run one of each type of thread (or perhaps put it in the application's configuration file so you can change it quickly while debugging).
If the application still misbehaves then you know that it's an interaction between the different types of thread that's causing the problem. If it doesn't then it could be that there's some resource you've not thread locked properly (for example).
What I'm trying to say is simplify you application to the point where it's using the minimum number of threads to still be your original design.
* Not the best word to use, but for example if you spawn 10 threads to deal with file i/o only spawn 1.
How do you define abnormal behavior? Would that be an exception thrown? Not sure if this will help you but What I often do is name the thread object when I create it, then if I catch an exception or if certain criteria exists, I write to the event log. I include the time, the application name, the name of the thread and exception information. I don't just use it for debugging, I use it if a user complains about odd behavior or reports an error. Then I can go back and get information about it.
Other than that I don't know if I can reproduce it now that it's happened (I've been using this particular application for a week or two now without issue), assuming that I'm running my application in the VS debugger, how should I go about debugging a deadlock after it's happened? I thought I might be able to get at call stacks if I paused the program and hence see where the different threads were when it happened, but clicking pause just threw Visual Studio into a deadlock too till I killed my application.
Is there some way other than browsing through my source tree to find potential problems? Is there a way to get at the call stacks once the problem has occured to see where the problem is? Any other tools/tips/tricks that might help?
What you did was the correct way. If Visual Studio also deadlocks, that happens now and then. It's just bad luck, unless there's some other issue.
You don't have to run the application in the debugger in order to debug it. Run the application normally, and if the deadlock happens, you can attach VS later. Ctrl+Alt+P, select the process, choose debugger type and click attach. Using a different set of debugger types might reduce the risk of VS crashing (especially if you don't debug native code)
A deadlock involves 2 or more threads. You probably know the first one (probably your UI thread) since you noticed the deadlock in your application. Now you only need to find the other one. With knowledge of the architecture, it should be easy to find (e.g. what other threads use the same locks, interact with the UI etc)
If VS doesn't work at all, you can always use windbg. Download here: http://www.microsoft.com/whdc/devtools/debugging/default.mspx
I'd try different approaches in the following order:
First, inspect the code to look for thread-safety violations, making sure that your critical regions don't call other functions that will in turn try to lock a critical region.
Use whatever tool you can get your hands on to visualize thread activity, I use an in-house perl script that parses an OS log we made and graphs all the context switches and shows when a thread gets pre-empted.
If you can't find a good tool, do some logging to see the last threads that were running before the deadlock occurred. This will give you a clue as to where the issue might be caused, it helps if the locking mechanisms have unique names, like if an object has it's own thread, create a dedicated semaphore or mutex just to manage that thread.
I hope this helps. Good luck!
You can use different programs like Intel(R) Parallel Inspector:
http://software.intel.com/en-us/intel-parallel-inspector/
Such programs can show you places in your code with potential deadlocks. However you should pay for it, or use it only evaluation period. Don't know if there is any free tools like this.
Just like anywhere, there're no "Silver bullet" tools to catch all the deadlocks. It is all about the sequence in which different threads aquire resources so your job is to find out where the order was violated. Usually Visual Studio or other debugger will provide stack traces and you will be able to find out where the discrepancy is. DevPartner Studio does provide deadlock analysis but last time I've checked there were too many false positives. Some static analysis tools will find some potential deadlocks too.
Other than that it helps to get the architecture straight to enforce resource aquisition order. For example, layering helps to make sure upper level locks are taken before lower ones but beware of callbacks.