When I am working on multithreads how can I debug to know which thread causes an abnormal behavior?
Can I use permonitor for debugging, or are there any other tools or debugging facilities that are available?
Tips for Debugging Threads From MSDN
Neat New Multithreaded Debugging Features in VS 2008 by John Robbin
As an alternative to debugging, you could do thread-related testing. The book The Art of Unit Testing has a section on this in Appendix B. The author mentions three tools (two of which he has a personal interest in):
Typemock Racer
Microsoft CHESS
Osherove.ThreadTester
You can use visual studio to set up breakpoints on certain threads. See here and here for how to do it.
It depends what do you mean by "abnormal behavior"...
for most of the time, the visual studio debugger should be enough. the Threads and CallStack windows will give you a lot of information about what is going on.
for the heavy duty stuff you can use WinDbg+SOS. read about the !threads, !threadpool and !runaway commands.
If you have several threads of the same type* you could modify your code to only run one of each type of thread (or perhaps put it in the application's configuration file so you can change it quickly while debugging).
If the application still misbehaves then you know that it's an interaction between the different types of thread that's causing the problem. If it doesn't then it could be that there's some resource you've not thread locked properly (for example).
What I'm trying to say is simplify you application to the point where it's using the minimum number of threads to still be your original design.
* Not the best word to use, but for example if you spawn 10 threads to deal with file i/o only spawn 1.
How do you define abnormal behavior? Would that be an exception thrown? Not sure if this will help you but What I often do is name the thread object when I create it, then if I catch an exception or if certain criteria exists, I write to the event log. I include the time, the application name, the name of the thread and exception information. I don't just use it for debugging, I use it if a user complains about odd behavior or reports an error. Then I can go back and get information about it.
Related
I am currently health checking an application which experiences UI stuttering during heavy usage.
Using Microsoft Concurrency Visualizer extension for Visual Studio 2015, showed that quite a lot of short-lived threads are created and stopped after ~100ms of execution.
Unfortunately, their displayed callstack is like clr.dll!0x98071 ntdll.dll!0x634fb and I am not quite sure how to extract useful information out of it.
I have no clue what is the purpose of those threads and which part of the code in the application is creating them.
How can I better identify where each one of them gets started?
In the code, I was able to grep a handful of Tasks, another of QueueUserWorkItems, several dozens of plain Thread instantiations, some System.Threading.Timer & System.Timers.Timer, no Reactive Extensions. I put breakpoints for all of them but it seems I am missing some...
I don't think those are from the threadpool because they would be displayed in synchronisation state in concurrency visualizer, instead they just end, and another one with another Id gets created later. But maybe I am misleading.
We also use a few third-party libs and a bunch of JuggerNET generated code, so maybe the origin is not even in the application itself.
I was finally able to find the culprit of those short-lived threads by looking closely at some of the cryptic callstacks, which included for instance:
mmdevapi.dll
wdmaud.drv
avrt.dll
audioses.dll
It led me to thinking that I should double check the sound alert system. It was indeed this one which spawned those threads.
Note:
I will not accept my answer however because I would like someone to share a better process or any kind of tips and tricks for diagnosing unwanted threads origin.
I have a complex system with several threads. sometimes i see the application in 100% cpu and force to restart the system. I have no idea which thread caused it and which code caused it.
I need something that will give me the state of each thread in the system (i.e. in which line the thread is now) so i can find which code causes the 100% CPU
(in java you have the thread dump kill -3 which gives you the state of each thread)
Can you help please?
Tess's blog has some great debugging tutorials, including:
.NET Hang Debugging Walkthrough
People have suggested Process Explorer to me before.
You could use the debugger to break and then find out what all threads are doing. (Add the Debug Location toolbar to Visual Studio)
Another option is to remove all thread one by one and find the guilty one.
I have found that in cases like this one of the best tools around these days in Microsoft's intellitrace. This allows for historical debugging & will give you the state of all threads etc. when you break execution.
Unfortunatly its only available in Visual Studio 2010 Ultimate edition, but if this is a really critical issue and you don't have this edition you could always download a 30 day evaluation.
Use VS debugger to attach to your process, and then press the "break" (pause symbol) to break execution. In this state, you can open the Debug window called "Threads" which should give you the state of each thread, and which line they're currently executing. It also helps at this point to give explicit names to your threads when debugging them.
I find that 99% of the time (at least for me) its because I accidentally make a loop infinite when I don't mean to make it so or there should be at least a few milliseconds of sleep before the the loop continues.
You can use profiler like visual vm to better debug the behavior of threads in your application.
Where to begin....
I've inherited a application that searches for strings within files from a previous programmer (that had no documenation) its using EPocalipse.IFilter namespace. It has a few issues, the first of which is the VS Project is missing FilterReader.cs, FilterLoader.cs, among others I believe are required for EPocalipse IFilters (based on my research). The second is that the app (when built) is hanging on ReadToEnd() when run against .
I found this thread here:
TextReader Read and ReadToEnd hangs without throwing exception
Which was awesome...except no posted solution was given =(
Since I have this issue and others, I figured I'd start a new thread since I first want to ensure IFilter is installed properly. The project builds, but still hangs on certain files (usually MS Excel).
For example, if I try to "Go to Definition" in Visual Studio for my instantiation of FilterReader, it simply shows the tab "FilterReader [from metadata]". So I'm assuming the FilterReader.cs file is simply missing (its nowhere in the projects solution explorer either), which may be the cause of the hanging problem as well?
Any help is greatly appreciated.
SK
For detailed info on the subject, take a look at this article [CodeProject]
As for hanging issue, it cannot be easily solved. Basically, there are 2 possible solutions:
Apply infinite cycle checks like those in the thread you've found. However, some extremely complex docs may still hang inside of IFilter, and you can do nothing about it (IFilters are COM components, usually closed-source).
Make your extraction two-threaded: one thread to monitor the extraction process and stop document extraction when it times out and another thread to do the actual extraction. Should you choose this path, remember that you'll likely run into access violation exceptions, as EPocalypse implementation hasn't COM protection for multi-threaded access to ifilters.
Other than that I don't know if I can reproduce it now that it's happened (I've been using this particular application for a week or two now without issue), assuming that I'm running my application in the VS debugger, how should I go about debugging a deadlock after it's happened? I thought I might be able to get at call stacks if I paused the program and hence see where the different threads were when it happened, but clicking pause just threw Visual Studio into a deadlock too till I killed my application.
Is there some way other than browsing through my source tree to find potential problems? Is there a way to get at the call stacks once the problem has occured to see where the problem is? Any other tools/tips/tricks that might help?
What you did was the correct way. If Visual Studio also deadlocks, that happens now and then. It's just bad luck, unless there's some other issue.
You don't have to run the application in the debugger in order to debug it. Run the application normally, and if the deadlock happens, you can attach VS later. Ctrl+Alt+P, select the process, choose debugger type and click attach. Using a different set of debugger types might reduce the risk of VS crashing (especially if you don't debug native code)
A deadlock involves 2 or more threads. You probably know the first one (probably your UI thread) since you noticed the deadlock in your application. Now you only need to find the other one. With knowledge of the architecture, it should be easy to find (e.g. what other threads use the same locks, interact with the UI etc)
If VS doesn't work at all, you can always use windbg. Download here: http://www.microsoft.com/whdc/devtools/debugging/default.mspx
I'd try different approaches in the following order:
First, inspect the code to look for thread-safety violations, making sure that your critical regions don't call other functions that will in turn try to lock a critical region.
Use whatever tool you can get your hands on to visualize thread activity, I use an in-house perl script that parses an OS log we made and graphs all the context switches and shows when a thread gets pre-empted.
If you can't find a good tool, do some logging to see the last threads that were running before the deadlock occurred. This will give you a clue as to where the issue might be caused, it helps if the locking mechanisms have unique names, like if an object has it's own thread, create a dedicated semaphore or mutex just to manage that thread.
I hope this helps. Good luck!
You can use different programs like Intel(R) Parallel Inspector:
http://software.intel.com/en-us/intel-parallel-inspector/
Such programs can show you places in your code with potential deadlocks. However you should pay for it, or use it only evaluation period. Don't know if there is any free tools like this.
Just like anywhere, there're no "Silver bullet" tools to catch all the deadlocks. It is all about the sequence in which different threads aquire resources so your job is to find out where the order was violated. Usually Visual Studio or other debugger will provide stack traces and you will be able to find out where the discrepancy is. DevPartner Studio does provide deadlock analysis but last time I've checked there were too many false positives. Some static analysis tools will find some potential deadlocks too.
Other than that it helps to get the architecture straight to enforce resource aquisition order. For example, layering helps to make sure upper level locks are taken before lower ones but beware of callbacks.
From C#, is it possible to detect the number of context switches that occurred while executing a block of code on a particular thread? Ideally, I'd like to know how many times and what CPU my thread code was scheduled on.
I know I can use tools like Event Tracing for Windows and the associated viewers, but this seemed a bit complicated to get the data I wanted.
Also, tools like Process Explorer make it too hard to tell how many switches occurred as a result of a specific block of code.
Background: I'm trying to test the actual performance of a low level lock primitive in .NET (as a result of some comments on a recent blog post I made.
It sounds like you may be looking for a programmatic solution, but if not, Microsoft's Process Explorer tool will tell you very easily the number of context switches for a particular thread.
Once in the tool, double-click your process, select the Threads tab, and select your thread.
The .NET tab has more specific .NET-related perf data.
I've never done this, but here are a few leads that might help:
The .NET profiler APIs might allow you to hook in? The ICorProfilerCallback interface has RuntimeThreadSuspended and RuntimeThreadResumed callbacks. But a comment on this blog post seems to indicate that they won't get you what you are looking for: "RuntimeThreadSuspended is issued when a thread is being suspended by the runtime, typically in preparation for doing a GC."
There is a "Context Switches/sec" perfmon counter that might help. I haven't looked at this counter specifically, but I'm guessing it operates on Win32 threads and not managed threads. You can use the profiling APIs to get the Win32 thread ID for any given managed thread ID.
Good luck! ;)
It looks like procexp might be using the kernel thread (KTHREAD) or executive thread (ETHREAD) data structures that have a ContextSwitches field on them. It might be possible to get this from managed code.