Hight CPU usage perf .net application - c#

I'm trying to profile my .net application written in C# which uses 100% of cpu. Application is very big, contains tons of code, so it is impossible to provide whole project code. I tried to get threads stack for application threads that uses 25% CPU (1 core), and often i got this:
ntoskrnl.exe!KeSynchronizeExecution+0x2246
ntoskrnl.exe!KeWaitForMultipleObjects+0x135e
ntoskrnl.exe!KeWaitForMultipleObjects+0xdd9
ntoskrnl.exe!KeWaitForMutexObject+0x373
ntoskrnl.exe!KeStallWhileFrozen+0x1977
ntoskrnl.exe!_misaligned_access+0x13f9
ntoskrnl.exe!KeWaitForMultipleObjects+0x152f
ntoskrnl.exe!KeWaitForMultipleObjects+0xdd9
ntoskrnl.exe!KeWaitForMutexObject+0x373
ntoskrnl.exe!NtWaitForSingleObject+0xb2
ntoskrnl.exe!setjmpex+0x34a3
ntdll.dll!ZwWaitForSingleObject+0xa
KERNELBASE.dll!WaitForSingleObjectEx+0x98
clr.dll!GetMetaDataInternalInterface+0x25b1f
clr.dll!GetMetaDataInternalInterface+0x25ad3
clr.dll!GetMetaDataInternalInterface+0x25a92
clr.dll!GetMetaDataInternalInterface+0x39106
clr.dll!GetMetaDataInternalInterface+0x39a81
clr.dll!GetMetaDataInternalInterface+0x394ad
clr.dll!GetMetaDataInternalInterface+0x39979
clr.dll!GetMetaDataInternalInterface+0x398c1
clr.dll!GetMetaDataInternalInterface+0x3539a
clr.dll!ClrCreateManagedInstance+0x2747
KERNEL32.dll!BaseThreadInitThunk+0x22
ntdll.dll!RtlUserThreadStart+0x34
Can anyone explain to me why thread with this call stack consumes 1 core
of my CPU?
What does this 'KeSynchronizeExecution'?
How to avoid hight CPU usage in such situations?

Just trying to help you here, I am not an expert.
The ntoskrnl.exe!KeSynchronizeExecution routine synchronizes the execution of the specified routine with the interrupt service routine (ISR) that is assigned to a set of one or more interrupt objects.
The ntoskrnl.exe!KeWaitForMultipleObjects routine puts the current thread into an alertable or nonalertable wait state until any or all of a number of dispatcher objects are set to a signaled state or (optionally) until the wait times out.
The ntoskrnl.exe!KeWaitForMutexObject routine puts the current thread into an alertable or nonalertable wait state until the given mutex object is set to a signaled state or (optionally) until the wait times out.
I think ntoskrnl.exe!KEStallWhileFrozen routine is called when waits for multiple objects routines are not resolved.
ntoskrnl.exe!_misaligned_access routine is an alert when cpu cannot read misaligned data. Seems Misaligned memory accesses can incur enormous performance losses on targets that do not support them in hardware. Ref: https://msdn.microsoft.com/en-us/library/ms253949(v=vs.80).aspx. Also check the Avoiding Alignment Errors section.
ntoskrnl.exe!NtWaitForSingleObject waits until the specified object attains a state of signaled.
A call to the setjmp function saves the current instruction address as well as other CPU registers. A subsequent call to the longjmp function restores the instruction pointer and registers, and execution resumes at the point just after the setjmp call.
ntdll.dll!ZwWaitForSingleObject routine waits until the specified object attains a state of Signaled. An optional time-out can also be specified.
KERNELBASE.dll!WaitForSingleObjectEx waits until the specified object is in the signaled state, an I/O completion routine or asynchronous procedure call (APC) is queued to the thread, or the time-out interval elapses.
clr.dll!GetMetaDataInternalInterface gets a pointer to an internal interface instance that is used to read and write metadata in memory.

I used Jetbrains solution before for this. And it could really find optimization points easy. My advice use : https://www.jetbrains.com/profiler/ and find which process and methods high usage cpu. also you can find memory etc.
I know you can install it trial. Install it trial and solve your problem.

I have just saw a similar problem in my application.
I was able to profile it using PerfView application.
In my case it was lock inside Dictionary.Insert method. Trying to access dictionary from multiple threads at the same time causes infinit operations in these threads.
Looks like one CPU Core goes to 25% usage and no chance to unlock it.

Related

Value not preserved on thread Named Data Slot

This is pretty simple. We have code like this:
var slot = Thread.GetNamedDataSlot("myslot");
Thread.SetData(slot, value);
The current code exits the thread. Eventually the thread is re-allocated for more work. We expect (according to doc and many assertions in SO) that the value will still be there in the slot. And yet, at least sometimes, it isn't. It comes up null. The ManagedThreadId is the same as the one we set the value for, but the value has gone null.
We do call some opaque third-party assemblies, but I don't think that there's any way that other code could clear that slot without knowing its name.
Any thoughts on how this could go happen? Could it be that .net destroys the thread, and later creates another one with the same id? Does a thread live for the duration of the app domain?
The answer is that threads are not forever. A thread returned to the pool might be reused, or might be discarded. Take care when leaving something in TLS on a thread, if you don't code a destructor you could have a resource leak.
Here's a post that describes the same issue: http://rocksolid.gibraltarsoftware.com/development/logging/managed-thread-ids-unique-ids-that-arent-unique
Threadpool threads do not belong to you. You're not supposed to rely on their context at all, and that includes stuff like ThreadStatic data and LocalDataStoreSlot. There's so many things the runtime can do with threadpool threads that will break your code, it's not even funny. This gets even crazier when you start using await, for example (the same method can easily execute on multiple different threads, some from the thread pool, some not...).
As an implementation detail (nothing you should rely on), the .NET runtime manages the thread pool to be as big as required. On a properly asynchronous application, this means it will only have about 1-2x the amount of CPU cores. However, if those threads become tied up, it will start creating new ones to accomodate the queued work items (unless, of course, the pool threads are actually saturating the CPU - new threads will not help in that case). When the peak load is done, it will similarly start releasing the threads.
ManagedThreadId is not unique over the scope of the AppDomain over its life-time - it is only unique in any given moment. You shouldn't rely on it being unique, especially when dealing with threadpool threads. The ID will stay the same for a given thread over it's lifetime (even if the underlying system thread changes - assuming of course the managed thread is actually implemented on top of a system thread) - when you're working with threadpool threads, though, you are not working with actual threads - you're just posting work items on the thread-pool.

.Net Profiling - Knowing the managed thread begin and end

I am developing a .Net profiler.. I use ILRewriting for this..
I need to trace the managed thread creations and destroys.
Need to know the threading related function that will be called at the beginning of the thread and during the end of the thread , i can inject my code to it and record whenever the event happens.
Any ideas about the default function that will be called at the time of thread creation and ends..??
OR else is there any other way to capture the managed thread creation and destroying events??
I know that we can trace by setting the threading event mask.. but i need to capture particular managed threads not all the threads..
As Hans pointed out, the CLR notifies the profiler of thread creation/destruction using ThreadCreated and ThreadDestroyed callbacks. Note: If the runtime shuts down before the thread terminates, then you won't get the ThreadDestroyed callback ... but I think the more likely reason you don't get the ThreadDestroyed callback is that IIS (I assume by 'page load' you are referring to asp .NET pages) decided to keep the thread around for future requests as an optimization, it may decide to terminate it later if it thinks it has enough other threads.
Also, regarding your second comment on the question, there is no relation between ThreadID and ManagedThreadID. I believe the ThreadID is a reference to an internal data structure (treat it as an opaque value, don't try to interpret it) and the ManagedThreadID appears to be a simple number sequentially allocated as threads first enter managed code. If you want identify which ThreadID corresponds with which managed thread, I can think of 3 options:
Check the thread name using the ThreadNameChanged callback (Note: If the thread name is set before the thread starts, then this will be raised before the ThreadCreated callback)
Check the OS thread id using the ThreadAssignedToOSThread callback
Have the profiled code call into the profiler to provide it with context (either using pinvoke or by calling a method that is instrumented for this purpose)

C# : Thread Workload and Memory useage

the Windows Taskmanger is fine to check the CPU and memory useage of an application but in our programm we have different threads and we want to know how much of the total ammount is on each thread.
We would like to check this with an external programm and also from the application itself during runtime. It would be great if a thread could tell about his memory and cpu useage.
Here's is the example:
You have threadA and ThreadB.
ThreadA creats an object X.
ThreadB uses this object.
So what do you want to see in thread's information? Who created the object or who is using it?
The only thing you can see is how much CPU time is using thread AFAIK
And all the same the only program that I know that shows MAX info on process is Process Explorer. http://technet.microsoft.com/en-us/sysinternals/bb896653
You can use performance monitor to see how much memory is allocated to a process, but you cannot see the same for single threads inside it.
However, you could create custom performance counters to display any value you want to monitor from within your code.
SysInternals Process Explorer has this feature, check this Server fault thread.
There is an open source project on CodeProject, the screenshot looks promising: How to get CPU usage of processes and threads, but the demo project seems to be crashing on Win7 (probably missing some privileges).
[Edit] If you want to write it yourself, you can P/Invoke Thread32First and Thread32Next functions to enumerate threads within a single process, and then use QueryThreadCycleTime to query CPU time for each thread.
Objects are shared between threads, threads do not own objects.
Memory for an object is allocated on the heap, which lives in the realm of the application. Any thread can access any of this memory at any time during the lifetime of the application.
There is no way to determine which thread is or may be using any arbitrary blocks of memory.
Threads perform units of work. Unless you know which thread is going to be running which unit of work you will be able to get no reliable metrics out of CPU usage. If you do know which thread will be performing which tasks, then Process Explorer by SysInternals has this metric.

Thread Execution Timing

I have written a program which depends on threads heavily. In addition, there is a requirement to measure the total time taken by each thread, and also the execution time (kernel time plus user time).
There can be an arbitrary number of threads and many may run at once. This is down to user activity. I need them to run as quickly as possible, so using something which has some overhead like WMI/Performance Monitor to measure thread times is not ideal.
At the moment, I'm using GetThreadTimes, as shown in this article: http://www.codeproject.com/KB/dotnet/ExecutionStopwatch.aspx
My question is simple: I understand .NET threads may not correspond on a one-to-one basis with system threads (though in all my testing so far, it seems to have been one to one). That being the case, if .NET decides to put two or more of my threads into one system thread, am I going to get strange results from my timing code? If so (or even if not), is there another way to measure the kernel and user time of a .NET thread?
as it stated: Multithreading is managed internally by a thread scheduler, a function the CLR typically delegates to the operating system. A thread scheduler ensures all active threads are allocated appropriate execution time, and that threads that are waiting or blocked (for instance, on an exclusive lock or on user input) do not consume CPU time.
Theoretically NET team may implement their own scheduler, but i doubt this.So i think the GetThreadTimes function is what you need.

How do I abort a .NET task?

Here's the situation, I am writing the framework for a code war contest. As the code runs, for each turn, it calls a method in the library provided by each contestant. The rules of the contest is the method must return in 1 second or we kill the task calling them. We then use a default result for that turn.
The method has no support for a cancel because we cannot trust the called code to respond to a cancel. And we need to kill the thread because if we have 10 or 20 ignored background tasks then all calls going forward will provide fewer clock cycles on each call and methods that before took less than 1 second now take more.
On the plus side, the method we're killing should have no resources open, etc. so an abort should not leave anything hanging.
Update: Two things to keep in mind here. First, this is like a game - so performance is important. Second, the worker thread is unlikely to have any resources open. If one of the called methods goes overlong, I need to abort it and move on quickly.
You should run each contestant in his own AppDomain with low privileges. This has several advantages:
It's sandboxed
It can't interact with any other code in the process
Force unloading an AppDomain is relatively clean.
Even if you prefer killing the thread over unloading the AppDomain I'd still put each contestant into an AppDomain to get the isolation.
Unfortunately Thread.Abort is not enough. It still executes finally clauses which can take as long as they want.
I would recommend that you run the code in a second process and carefully define the interface for communicating with it to ensure that it can handle not receiving a response. Most operating systems are designed to clean up fairly well after a killing a process.
For communication, you should probably avoid .NET remoting, as that seems likely to be left in an inconsistent state on the server side. Some other choices: sockets, named pipes, web service.
Thread.Interrupt() method is maybe what you are looking for.
As the MSDN documentation says, "If this thread is not currently blocked in a wait, sleep, or join state, it will be interrupted when it next begins to block."
It is not an abort, it forces the running thread to throws ThreadInterruptedException when the thread enters in a wait state.
You can then use a timer in another thread with a timeout to check if the thread don't really want to terminate, if the thread refuses to terminate in, for example, 30 seconds, you can abort it.

Categories

Resources