I have an asp.net (3.5) web application.
In one page i use an unmanaged C library (a simulator) that produces a set of results based on some input.
When i've set the debug="false" in web.config (Release mode) that call will result in a System.AccessViolationException. If i am in the debugger, or at least set debug="true" in web.config, that exception will never appear.
Because of the nature of the bug i placed the call inside a try-catch to log the exception and then the exception does not appear!
Is there some magical work on protected memory space when in debug mode and/or in a try-catch?
AccessViolationException in this context probably means your unmanaged C code is screwing up. It's overwriting the heap or something.
Keep in mind that many pieces of code were never written to be run in a multi-threaded environment like that of ASP.NET. They were never even tested in such an environment, or were only tested using some "desktop" form of multi-threading that does not apply to ASP.NET.
I had a problem like this once. A third-party DLL which claimed to be thread-safe was very unsafe. By attaching an unmanaged debugger to the ASP.NET worker process, it was possible to see that the application was writing all over the C runtime library heap, and the heap was complaining bitterly about it. And nothing was listening to the complaints.
It was necessary to wrap calls to this code in a lock block - to ensure that only one thread could call it at a time. This was sufficient to prevent all of the crashes, even under load.
This still might not suffice for all unmanaged code. A piece of code that assumes all calls will occur on the same thread, or that all calls from the same "session" occur on the same thread, will be disappointed, and may very well crash or do something worse. In that case, you might need to move that code into a separate Windows Service which only allows one single thread to access that library, ever.
Related
In our application, we have a plugin system that takes compilers in the form of plugins, written specifically for our software. The plugins are assembly files, imported using MEF. The user is then able to select a compiler, and use it to compile a piece of source code.
A problem arises, if there is a faulty plugin added to the system. Upon execution, it could potentially hang the whole service, requiring a complete restart. I am looking for a way to stop the execution of a compiler after a certain timeout, without the need for TaskCancellation handling in the plugins, since you can never be sure that it will implemented properly, if at all. Of course, I also need to be sure that proper clean up is done after aborting, since failure to do that, could also crash or slow down the system.
I do understand that forcefully terminating managed code could potentially lead to a lot of problems and that is the reason I am asking for a good way to do it.
I have a computationally-expensive multi-threaded C# app that seems to crash consistently after 30-90 minutes of running. The error it gives is
The runtime has encountered a fatal error. The address of the error was at 0xec37ebae, on thread 0xbcc. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
(0xc0000005 is the error-code for Access Violation)
My app does not invoke any native code, or use any unsafe blocks, or even any non-CLS compliant types like uint. In fact, the line of code that the debugger says caused the crash is
overallLength += distanceTravelled;
Where both values are of type double
Given all this, I believe the crash must be due to a bug in the compiler or CLR or JIT. I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin. I've never had to view the CIL-binary, or the compiled JIT output, or the native stacktrace (there is no managed stacktrace at the time of the crash), so I'm not sure how. I can't even figure out how to view the state of all the variables at the time of the crash (VS unfortunately won't tell me like it does after managed-exceptions, and outputting them to console/a file would slow down the app 1000-fold, which is obviously not an option).
So, how do I go about debugging this?
[Edit] Compiled under VS 2010 SP1, running latest version of .Net 4.0 Client Profile. Apparently it's ".Net 4.0C/.Net 4.0E, .Net CLR 1.1.4322"
I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin.
"Smaller reproduction" definitely sounds like a great idea here... even if "smaller" won't mean "quicker to reproduce".
Before you even start, try to reproduce the error on another machine. If you can't reproduce it on another machine, that suggests a whole different set of tests to do - hardware, installation etc.
Also, check you're on the latest version of everything. It would be annoying to spend days debugging this (which is likely, I'm afraid) and then end up with a response of "Yes, we know about this - it was a bug in .NET 4 which was fixed in .NET 4.5" for example. If you can reproduce it on a variety of framework versions, that would be even better :)
Next, cut out everything you can in the program:
Does it have a user interface at all? If possible, remove that.
Does it use a database? See if you can remove all database access: definitely any output which isn't used later, and ideally input too. If you can hard code the input within the app, that would be ideal - but if not, files are simpler for reproductions than database access.
Is it data-sensitive? Again, without knowing much about the app it's hard to know whether this is useful, but assuming it's processing a lot of data, can you use a binary search to find a relatively small amount of data which causes the problem?
Does it have to be multi-threaded? If you can remove all the threading, obviously that may well then take much longer to reproduce the problem - but does it still happen at all?
Try removing bits of business logic: if your app is componentized appropriately, you can probably fake out whole significant components by first creating a stub implementation, and then simply removing the calls.
All of this will gradually reduce the size of the app until it's more manageable. At each step, you'll need to run the app again until it either crashes or you're convinced it won't crash. If you have a lot of machines available to you, that should help...
tl;dr Make sure you're compiling to .Net 4.5
This sounds suspiciously like the same error found here. From the MSDN page:
This bug can be encountered when the Garbage Collector is freeing and compacting memory. The error can happen when the Concurrent Garbage Collection is enabled and a certain combination of foreground Garbage Collection and background Garbage Collection occurs. When this situation happens you will see the same call stack over and over. On the heap you will see one free object and before it ends you will see another free object corrupting the heap.
The fix is to compile to .Net 4.5. If for some reason you can't do this, you can also disable concurrent garbage collection by disabling gcConcurrent in the app.config file:
<configuration>
<runtime>
<gcConcurrent enabled="false"/>
</runtime>
</configuration>
Or just compile to x86.
WinDbg is your friend:
http://blogs.msdn.com/b/tess/archive/2006/02/09/net-crash-managed-heap-corruption-calling-unmanaged-code.aspx
http://www.codeproject.com/Articles/23589/Get-Started-Debugging-Memory-Related-Issues-in-Net
http://www.codeproject.com/Articles/22245/Quick-start-to-using-WinDbg
Download Debug Diagnostic Tool v1.2
Run program
Add Rule "Crash"
Select "Specific Process"
on page Advanced Configuration set your exception if you know on which exception it fails or just leave this page as is
Set userdump location
Now wait for process to crash, log file is created by DebugDiag. Now activate tab Advanced Analysis, select Crash/Hang Analyzers in top list and dump file in lower list and hit Start Analysis. This will generate html report for you. Hopes you found usefull info in that report. If you have problem with analyze, upload html report somewhere and place url here so we can focus on it.
My app does not invoke any native code, or use any unsafe blocks, or
even any non-CLS compliant types like uint
You may think this, but threading, synchronization via semaphore, mutex it any handles all are native. .net is a layer over operating system, .net itself does not support pure clr code for multithreading apps, this is because OS already does it.
Most likely this is thread synchronization error. Probably multiple threads are trying to access shared resource like file etc that is outside clr boundary.
You may think you aren't accessing com etc, but when you call certain API like get desktop folder path etc it is called through shell com API.
You have following two options,
Publish your code so that we can review the bottleneck
Redesign your app using .net parallel threading framework, which includes variety of algorithms requiring CPU intensive operations.
Most likely programs fail after certain period of time as collections grow up and operations fail to execute before other thread interfere. For example, producer consumer problem, you will not notice any problem till producer will become slower or fail to finish its operation before consumer kicks in.
Bug in clr is rare, because clr is very stable. But poorly written code may lead error to appear as bug in clr. Clr can not and will never detect whether the bug is in your code or in clr itself.
Did you run a memory test for your machine as the one time I had comparable symptoms one of my dimms turned out to be faulty (a very good memorytester is included in Win7; http://www.tomstricks.com/how-to-test-your-ram-or-memory-with-windows-memory-diagnostic-tool-in-windows-7/)
It might also be a heating/throttling issue if your CPU gets too hot after this period of time. Although that would happen sooner imho.
There should be a dumpfile that you can analyze. If you never did this find someone who did, or send that to microsoft
I will suggest you open a support case via http://support.microsoft.com immediately, as the support guys can show you how to collect the necessary information.
Generally speaking, like #paulsm4 and #psulek said, you can utilize WinDbg or Debug Diag to capture crash dumps of the process, and within it, all necessary information is embedded. However, if this is the very first time you use those tools, you might be puzzled. Microsoft support team can provide you step by step guidance on them, or they can even set up a Live Meeting session with you to capture the data, as the program crashes so often.
Once you are familiar with the tools, in the future you can perform similar troubleshooting more easily,
http://blogs.msdn.com/b/lexli/archive/2009/08/23/when-the-application-program-crashes-on-windows.aspx
BTW, it is too early to say "I've found a bug". Though you cannot obviously find in your program a dependency on native code, it might still have a dependency on native code. We should not draw a conclusion before debugging further into the issue.
For some background info, .NET 4.0 no longer catches CSEs by default: http://msdn.microsoft.com/en-us/magazine/dd419661.aspx
I'm working on an app that executes code in a new AppDomain. If that code throws a CSE, the exception bubbles up to the main code if it's not handled. My question is, can I safely assume that a CSE on the second AppDomain won't corrupt the state in the main AppDomain, and thus exit the second AppDomain and continue running the main AppDomain?
In the context of a corrupted state exception, in general, you cannot assume anything to be true anymore. The point of these exceptions is that something has happened, usually due to buggy unmanaged code, that has violated some core assumption that Windows or the CLR makes about the structure of memory. That means that, in theory, the very structures that the CLR uses to track which app domains exist in memory could be corrupted. The kinds of things that cause CSEs are generally indicative that things have gone catastrophically wrong.
Having said all that, off-the-record, in some cases, you may be able to make a determination that it is safe to continue from a particular exception. An EXCEPTION_STACK_OVERFLOW, for example, is probably recoverable, and an EXCEPTION_ACCESS_VIOLATION usually indicates that Windows caught a potential bug before it had a chance to screw anything up. It's up to you if you're willing to risk it, depending on how much you know about the code that is throwing the CSEs in the first place.
I think I have a curly one here... I have an WinForms application that crashes fairly regularly every hour or so when running as an x64 process. I suspect this is due to stack corruption and would like to know if anyone has seen a similar issue or has some advice for diagnosing and detecting the issue.
The program in question has no visible UI. It's just a message window that sits in the background and acts as a sort of 'middleware' between our other client programs and a server.
It dies in different ways on different machines. Sometimes it's an 'APPCRASH' dialog that reports a fault in ntdll.dll. Sometimes it's an 'APPCRASH' that reports our own dll as the culprit. Sometimes it's just a silent death. Sometimes our unhandled exception hook logs the error, sometimes it doesn't.
In the cases where Windows Error Reporting kicks in, I've examined memory dumps from several different crash scenarios and found the same Managed exception in memory each time. This is the same exception I see reported as an unhandled exception in the cases where we it logs before it dies.
I've also been lucky (?) enough to have the application crash while I was actively debugging with Visual Studio - and saw that same exception take down the program.
Now here's the kicker. This particular exception was thrown, caught and swallowed in the first few seconds of the program's life. I have verified this with additional trace logging and I have taken memory dumps of the application a couple of minutes after application startup and verified that exception is still sitting there in the heap somewhere. I've also run a memory profiler over the application and used that to verify that no other .NET object had a reference to it.
The code in question looks a bit like this (vastly simplified, but maintains the key points of flow control)
public class AClass
{
public object FindAThing(string key)
{
object retVal = null;
Collection<Place> places= GetPlaces();
foreach (Place place in places)
{
try
{
retval = place.FindThing(key);
break;
}
catch {} // Guaranteed to only be a 'NotFound' exception
}
return retval;
}
}
public class Place
{
public object FindThing(string key)
{
bool found = InternalContains(key); // <snip> some complex if/else logic
if (code == success)
return InternalFetch(key);
throw new NotFoundException(/*UsefulInfo*/);
}
}
The stack trace I see, both in the event log and when looking at the heap with windbg looks a bit like this.
Company.NotFoundException:
Place.FindThing()
AClass.FindAThing()
Now... to me that reeks of something like stack corruption. The exception is thrown and caught while the application is starting up. But the pointer to it survives on the stack for an hour or more, like a bullet in the brain, and then suddenly breaches a crucial artery, and the application dies in a puddle.
Extra clues:
The code within 'InternalFetch' uses some Marshal.[Alloc/Free]CoTask and pinvoke code. I have run FxCop over it looking for portability issues, and found nothing.
This particular manifestation of the issue is only affecting x64 code built in release mode (with code optimization on). The code I listed for the 'Place.Find' method reflects the optimized .NET code. The unoptimized code returns the found object as the last statement, not 'throw exception'.
We make some COM calls during startup before the above code is run... and in a scenario where the above problem is going to manifest, the very first COM call fails. (Exception is caught and swallowed). I have commented out that particular COM call, and it does not stop the exception sticking around on the heap.
The problem might also affect 32 bit systems, but if it does - then the problem does not manifest in the same spot. I was only sent (typical users!) a few pixels worth of a screen shot of an 'APP CRASH' dialog, but the one thing I could make out was 'StackHash_2264' in the faulting module field.
EDIT:
Breakthrough!
I have narrowed down the problem to a particular call to SetTimer.
The pInvoke looks like this:
[DllImport("user32")]
internal static extern IntPtr SetTimer(IntPtr hwnd, IntPtr nIDEvent, int uElapse, TimerProc CB);
internal delegate void TimerProc(IntPtr hWnd, uint nMsg, IntPtr nIDEvent, int dwTime);
There is a particular class that starts a timer in its constructor. Any timers set before that object is constructed work. Any timers set after that object is constructed work. Any timer set during that constructor causes the application to crash, more often than not. (I have a laptop that crashes maybe 95% of the time, but my desktop only crashes 10% of the time).
Whether the interval is set to 1 hour, or 1 second, seems to make no different. The application dies when the timer is due - usually by throwing some previously handled exception as described above. The callback does not actually get executed. If I set the same timer on the very next line of managed code after the constructor returns - all is fine and happy.
I have had a debugger attached when the bad timer was about to fire, and it caused an access violation in 'DispatchMessage'. The timer callback was never called. I have enabled the MDAs that relate to managed callbacks being garbage collected, and it isn't triggering. I have examined the objects with sos and verified that the callback still existed in memory, and that the address it pointed to was the correct callback function.
If I run '!analyze -v' at this point, it usually (but not always) reports something along the lines of 'ERROR_SXS_CORRUPT_ACTIVATION_STACK'
Replacing the call to SetTimer with Microsoft's 'System.Windows.Forms.Timer' class also stops the crash. I've used a Reflector on the class and can see internally it still calls SetTimer - but does not register a procedure. Instead it has a native window that receives the callback. It's pInvoke definition actually looks wrong... it uses 'ints' for the eventId, where MSDN documentation says it should be a UIntPtr.
Our own code originally also used 'int' for nIDEvent rather than IntPtr - I changed it during the course of this investigation - but the crash continued both before and after this declaration change. So the only real difference that I can see is that we are registering a callback, and the Windows class is not.
So... at this stage I can 'fix' the problem by shuffing one particular call to SetTimer to a slightly different spot. But I am still no closer to actually understanding what is so special about starting the timer within that constructor that causes this error. And I dearly would like to understand the root cause of this issue.
Just briefly thinking about it it sounds like an x64 interop issue (i.e., calling x32 native functions from x64 managed code is fraught with danger). Does the problem go away if you force your application to compile as x32 platform from within project properties?
You can read suggestions on forcing x32 compile during x32/x64 development on Dotnetrocks. Richard Campbell's suggestion is that Visual Studio should default to x32 platform and not AnyCPU.
http://www.dotnetrocks.com/default.aspx?showNum=341 (transcript).
With regard to advanced debugging, I have not had a chance to debug x64 interop code, but i hear that this book is an great resource: Advanced .NET Debugging.
Finally, one thing you might try is force Visual Studio to break when an exception is thrown.
Use something like DebugDiag for x64 or Windbg to write a dump on Kernel32!TerminateProcess and second chance exception on .NET which should give you the actual .excr context frame of the exception that occurred.
This should help you in identifying the call-stack for the process terminate.
IMO it could be mostly because of PInvoke calls. You could use Managed Debugging Assistants to debug these issues.
If MDA is used along with Windbg it would give out messages that would be helpful in debugging
Also I have found tools from the http://clrinterop.codeplex.com/ team are extremely handy when dealing with interop
EDIT
This should give an answer why it is not working in 64 bit Issue with callback method in SetTimer Windows API called from C# code .
This does sound like a corruption issue. I would go through all of your interop calls and ensure that all of the parameters to the DllImport'ed functions are the correct types. For exmaple, using an int in place of an IntPtr will work in 32 bit code but can crash 64 bit.
I would use a site like PInvoke.net to verify all of the signatures.
My application (base application is MFC interop with C++/CLI but it also contains a lot of C#, Windows Forms, WPF) has has a handle leak. Shortly after application start I can see the handle count in the task manager grow continuously (at a rate of 10 new handles per second). So I used handles.exe to see what kind of handles they are. I found out that the leaking handles are process handles. And they are process handles to the process of my application.
So I wonder what operations would typically create a handle to the process it runs in. Any idea? Have you ever seen something like this? What else could I do to track down the leak, considering that I can't use debug DLLs and that I can only use tools that can be xcopy deployed.
Update:
I was able to throw windbg and !handle, !htrace at it and found out that the process handles are all created using the following stack traces (ordered by frequency):
0x79f7570b: mscorwks!CorExitProcess+0x00022055
0x79f03edd: mscorwks!GetPrivateContextsPerfCounters+0x0000b6fe
0x79f04b87: mscorwks!GetPrivateContextsPerfCounters+0x0000c3a8
0x79f04b03: mscorwks!GetPrivateContextsPerfCounters+0x0000c324
0x79f919bf: mscorwks!CorExitProcess+0x0003e309
0x79f91b28: mscorwks!CorExitProcess+0x0003e472
0x792d6b4c: mscorlib_ni+0x00216b4c
0x1391a663: +0x1391a663
0x1391a0b1: +0x1391a0b1
0x7a9ea544: System_ni+0x005aa544
0x792a842f: mscorlib_ni+0x001e842f
or
0x7c8106f5: kernel32!CreateThread+0x0000001e
0x79f04bb2: mscorwks!GetPrivateContextsPerfCounters+0x0000c3d3
0x79f04b03: mscorwks!GetPrivateContextsPerfCounters+0x0000c324
0x79f919bf: mscorwks!CorExitProcess+0x0003e309
0x79f91b28: mscorwks!CorExitProcess+0x0003e472
0x792d6b4c: mscorlib_ni+0x00216b4c
0x1391a663: +0x1391a663
0x1391a0b1: +0x1391a0b1
0x7a9ea544: System_ni+0x005aa544
0x792a842f: mscorlib_ni+0x001e842f
or
0x08ec2eba: +0x08ec2eba
0x792b8277: mscorlib_ni+0x001f8277
0x792b8190: mscorlib_ni+0x001f8190
0x792b8040: mscorlib_ni+0x001f8040
0x792b7ff2: mscorlib_ni+0x001f7ff2
0x677e48f3: System_Runtime_Remoting_ni+0x000748f3
0x677e44be: System_Runtime_Remoting_ni+0x000744be
0x677e46ec: System_Runtime_Remoting_ni+0x000746ec
0x677e8408: System_Runtime_Remoting_ni+0x00078408
0x7926eb8d: mscorlib_ni+0x001aeb8d
Now what does that tell me?
The call stacks look wrong. Did you setup the symbol server correctly? .symfix should do the trick in Windbg. Afterwards you should get a better stacktrace.
It looks like part of the code that has this problem is managed so it would make sense to break on DuplicateHandle and OpenProcess and dump the managed call stack there. These two methods are the only ones which could produce a real process handle.
You can declare a breakpoint like this and execute commands when the breakpoint is hit.
In this case the managed stack is printed and then the execution does continue.
bp kernel32!OpenProcess "!ClrStack;g"
Had same issues with a webservice calling COM objects through interop.
I solved this by explicitely calling Marshal.ReleaseComObject against the interop objects I created. No issues after that moment for me.
Hope it helps.
So... are you doing performance counters explicitely (if so, try disabling them to narrow down on the source of the leaks).