This may be a long shot but I'm out of ideas.
I've got a VS C# solution with three projects in it. There's a class library project and then two application projects that depend on that class library. The class library in turn depends on a few other DLLs including the avalonedit dll from the sharpdevelop project.
One of the applications is building and running fine, including a use of my own control that wraps the avalonedit control. The other application is failing to run and it seems to be failing at the point when the avalonedit control is initialised via the XAML in my wrapping control.
The problem is that I don't see any errors in the debug output at all, all I see is the dll loaded message and then nothing. If I step into the constructor of my control the step never completes. The debugger says the app is running, but it is apparently spinning somewhere in the avalonedit dll when the underlying edit control is constructed by the XAML side.
I have to assume that there's some issue with difference in environment between the two projects but I'm kind of stumped as to how to proceed in tracking the problem down. Am I going to have to somehow get arrange matters so that I can put a break in the avalonedit source?
Edit: If I pause/break all it just drops back to the line calling my control constructor.
Sounds like a deadlock. Take a close look at all threads, their stack traces and synchronization primitives (locks, semaphores, etc.). Keep in mind: contended resources may not be explicit (for example, when you are inside static constructor waiting on something that tries to get access to a static field of the type being constructed you get a deadlock).
There are many ways to introduce a deadlock but no simple advice to handle it. You could also enable break on all exceptions in Visual Studio (Debug -> Exceptions... and tick CLR Exceptions).
If this does not help you could provide stack traces here and maybe somebody could spot the problem.
Related
I have a computationally-expensive multi-threaded C# app that seems to crash consistently after 30-90 minutes of running. The error it gives is
The runtime has encountered a fatal error. The address of the error was at 0xec37ebae, on thread 0xbcc. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.
(0xc0000005 is the error-code for Access Violation)
My app does not invoke any native code, or use any unsafe blocks, or even any non-CLS compliant types like uint. In fact, the line of code that the debugger says caused the crash is
overallLength += distanceTravelled;
Where both values are of type double
Given all this, I believe the crash must be due to a bug in the compiler or CLR or JIT. I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin. I've never had to view the CIL-binary, or the compiled JIT output, or the native stacktrace (there is no managed stacktrace at the time of the crash), so I'm not sure how. I can't even figure out how to view the state of all the variables at the time of the crash (VS unfortunately won't tell me like it does after managed-exceptions, and outputting them to console/a file would slow down the app 1000-fold, which is obviously not an option).
So, how do I go about debugging this?
[Edit] Compiled under VS 2010 SP1, running latest version of .Net 4.0 Client Profile. Apparently it's ".Net 4.0C/.Net 4.0E, .Net CLR 1.1.4322"
I'd like to figure out what causes it, or at the very least write a smaller reproduction to send into Microsoft, but I have no idea where to even begin.
"Smaller reproduction" definitely sounds like a great idea here... even if "smaller" won't mean "quicker to reproduce".
Before you even start, try to reproduce the error on another machine. If you can't reproduce it on another machine, that suggests a whole different set of tests to do - hardware, installation etc.
Also, check you're on the latest version of everything. It would be annoying to spend days debugging this (which is likely, I'm afraid) and then end up with a response of "Yes, we know about this - it was a bug in .NET 4 which was fixed in .NET 4.5" for example. If you can reproduce it on a variety of framework versions, that would be even better :)
Next, cut out everything you can in the program:
Does it have a user interface at all? If possible, remove that.
Does it use a database? See if you can remove all database access: definitely any output which isn't used later, and ideally input too. If you can hard code the input within the app, that would be ideal - but if not, files are simpler for reproductions than database access.
Is it data-sensitive? Again, without knowing much about the app it's hard to know whether this is useful, but assuming it's processing a lot of data, can you use a binary search to find a relatively small amount of data which causes the problem?
Does it have to be multi-threaded? If you can remove all the threading, obviously that may well then take much longer to reproduce the problem - but does it still happen at all?
Try removing bits of business logic: if your app is componentized appropriately, you can probably fake out whole significant components by first creating a stub implementation, and then simply removing the calls.
All of this will gradually reduce the size of the app until it's more manageable. At each step, you'll need to run the app again until it either crashes or you're convinced it won't crash. If you have a lot of machines available to you, that should help...
tl;dr Make sure you're compiling to .Net 4.5
This sounds suspiciously like the same error found here. From the MSDN page:
This bug can be encountered when the Garbage Collector is freeing and compacting memory. The error can happen when the Concurrent Garbage Collection is enabled and a certain combination of foreground Garbage Collection and background Garbage Collection occurs. When this situation happens you will see the same call stack over and over. On the heap you will see one free object and before it ends you will see another free object corrupting the heap.
The fix is to compile to .Net 4.5. If for some reason you can't do this, you can also disable concurrent garbage collection by disabling gcConcurrent in the app.config file:
<configuration>
<runtime>
<gcConcurrent enabled="false"/>
</runtime>
</configuration>
Or just compile to x86.
WinDbg is your friend:
http://blogs.msdn.com/b/tess/archive/2006/02/09/net-crash-managed-heap-corruption-calling-unmanaged-code.aspx
http://www.codeproject.com/Articles/23589/Get-Started-Debugging-Memory-Related-Issues-in-Net
http://www.codeproject.com/Articles/22245/Quick-start-to-using-WinDbg
Download Debug Diagnostic Tool v1.2
Run program
Add Rule "Crash"
Select "Specific Process"
on page Advanced Configuration set your exception if you know on which exception it fails or just leave this page as is
Set userdump location
Now wait for process to crash, log file is created by DebugDiag. Now activate tab Advanced Analysis, select Crash/Hang Analyzers in top list and dump file in lower list and hit Start Analysis. This will generate html report for you. Hopes you found usefull info in that report. If you have problem with analyze, upload html report somewhere and place url here so we can focus on it.
My app does not invoke any native code, or use any unsafe blocks, or
even any non-CLS compliant types like uint
You may think this, but threading, synchronization via semaphore, mutex it any handles all are native. .net is a layer over operating system, .net itself does not support pure clr code for multithreading apps, this is because OS already does it.
Most likely this is thread synchronization error. Probably multiple threads are trying to access shared resource like file etc that is outside clr boundary.
You may think you aren't accessing com etc, but when you call certain API like get desktop folder path etc it is called through shell com API.
You have following two options,
Publish your code so that we can review the bottleneck
Redesign your app using .net parallel threading framework, which includes variety of algorithms requiring CPU intensive operations.
Most likely programs fail after certain period of time as collections grow up and operations fail to execute before other thread interfere. For example, producer consumer problem, you will not notice any problem till producer will become slower or fail to finish its operation before consumer kicks in.
Bug in clr is rare, because clr is very stable. But poorly written code may lead error to appear as bug in clr. Clr can not and will never detect whether the bug is in your code or in clr itself.
Did you run a memory test for your machine as the one time I had comparable symptoms one of my dimms turned out to be faulty (a very good memorytester is included in Win7; http://www.tomstricks.com/how-to-test-your-ram-or-memory-with-windows-memory-diagnostic-tool-in-windows-7/)
It might also be a heating/throttling issue if your CPU gets too hot after this period of time. Although that would happen sooner imho.
There should be a dumpfile that you can analyze. If you never did this find someone who did, or send that to microsoft
I will suggest you open a support case via http://support.microsoft.com immediately, as the support guys can show you how to collect the necessary information.
Generally speaking, like #paulsm4 and #psulek said, you can utilize WinDbg or Debug Diag to capture crash dumps of the process, and within it, all necessary information is embedded. However, if this is the very first time you use those tools, you might be puzzled. Microsoft support team can provide you step by step guidance on them, or they can even set up a Live Meeting session with you to capture the data, as the program crashes so often.
Once you are familiar with the tools, in the future you can perform similar troubleshooting more easily,
http://blogs.msdn.com/b/lexli/archive/2009/08/23/when-the-application-program-crashes-on-windows.aspx
BTW, it is too early to say "I've found a bug". Though you cannot obviously find in your program a dependency on native code, it might still have a dependency on native code. We should not draw a conclusion before debugging further into the issue.
I have the a service, that loads some dlls and starts a function in each dll. Each dll contains some rules, that can be also developed by our clients (something like plugin system). The problem is, that clients can theoretically add forms to be called inside dlls. So the goal is to disallow that, or, at least block such dlls.
The only method I can imagine now is call each dll in a separate thread and kill it after some timeout.
But I think it is not so nice.
Please advice me a better method. Thankx.
The best way to deal with plug-ins is to "sandbox" each one of them in an individual app domain. This way you can safely react to their execution errors, unload them if you need to, and manage them in whatever ways you like. But most importantly for this question, you can monitor their loading of assemblies using this event hook. If you see them loading a DLL that you do not want to allow, you can simply throw an exception. Your code would catch the exception, clean up the app domain, and optionally send the clients a warning for trying to do something that is not allowed.
The only downside to this approach is that it is rather non-trivial to implement.
It is VERY hard problem to protect server from third party code that you need to execute.
I would recommend reading on SharePoint sandbox approach (i.e. http://msdn.microsoft.com/en-us/library/ff798382.aspx) which tries to solve this and related issues.
As SLaks said - you implicitly trust code by simply executing it. Unless you expect code to be outright evil you may be better of by simply logging how long calls take (and maybe time out if possible) and provide your client with this information. Since it seems like client creates the code for themselves it is unlikely that code will be explicitly made non-functional.
Other interesting issues outside showing a Form:
stack overflow exception (easy to create, hard to handle)
while(true); code that never returns control
access to native code if full trust enabled.
You could always use reflection to inspect their code and ensure that certain namespaces and classes (e.g. System.Windows.Forms.*) are not referenced or used.
SQLCLR restricts what is allowed to be used/referenced in assemblies installed as SQLCLR extensions, and that appears to be done that way: http://msdn.microsoft.com/en-us/library/ms403273.aspx
Where to begin....
I've inherited a application that searches for strings within files from a previous programmer (that had no documenation) its using EPocalipse.IFilter namespace. It has a few issues, the first of which is the VS Project is missing FilterReader.cs, FilterLoader.cs, among others I believe are required for EPocalipse IFilters (based on my research). The second is that the app (when built) is hanging on ReadToEnd() when run against .
I found this thread here:
TextReader Read and ReadToEnd hangs without throwing exception
Which was awesome...except no posted solution was given =(
Since I have this issue and others, I figured I'd start a new thread since I first want to ensure IFilter is installed properly. The project builds, but still hangs on certain files (usually MS Excel).
For example, if I try to "Go to Definition" in Visual Studio for my instantiation of FilterReader, it simply shows the tab "FilterReader [from metadata]". So I'm assuming the FilterReader.cs file is simply missing (its nowhere in the projects solution explorer either), which may be the cause of the hanging problem as well?
Any help is greatly appreciated.
SK
For detailed info on the subject, take a look at this article [CodeProject]
As for hanging issue, it cannot be easily solved. Basically, there are 2 possible solutions:
Apply infinite cycle checks like those in the thread you've found. However, some extremely complex docs may still hang inside of IFilter, and you can do nothing about it (IFilters are COM components, usually closed-source).
Make your extraction two-threaded: one thread to monitor the extraction process and stop document extraction when it times out and another thread to do the actual extraction. Should you choose this path, remember that you'll likely run into access violation exceptions, as EPocalypse implementation hasn't COM protection for multi-threaded access to ifilters.
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 7 years ago.
Improve this question
I'm running a (mostly) single threaded program (there's a main thread that does everything, the others only read stuff). I can get the application to run fine in VS2008 after a minor change (I changed the text of a form, and tab order of another form), but I can no longer get it to work outside of the debugger. Does anyone know what would cause this?
Clarification: Release mode, launched with debugger (F5) works. Debug mode, lanuched with debugger (F5) works. Debug executable, or release executable launched outside of VS or with Ctrl+F5 fail.
It uses Microsoft's Virtual Earth 3D, and it seems to crash just when the 'ring of hope' (loading ring) is about to complete.
Event log says: ".NET Runtime version 2.0.50727.3053 - Fatal Execution Engine Error (000006427F44AA6E) (80131506)"
Culprit: this line:
this.loader = PlugInLoader.CreateLoader(this.globeControl.Host);
Causes it to fail. However, the form that was working uses the exact same line without an issue. This line is nesseccary for the program to function. I have no idea what it's doing.
Another Lead the error seems to be inside the .NET framework. Application worked on another machine, attempting reinstall. Update: didn't make a difference, although when I repaired VS it kept telling me Visual Studio was crashing even though I wasn't running it.
Error
When I launch the program after a couple minutes I get:
Application has generated an exception that could not be handled.
Proccess ID=0x9CC (2508), Thread ID =0xF0C(3852).
Click OK to terminate the application.
Click CANCEL to debug the application.
The disassembly is bizarre:
0000000077EF2A90 int 3
0000000077EF2A91 int 3
0000000077EF2A92 int 3
0000000077EF2A93 int 3
0000000077EF2A94 int 3
0000000077EF2A95 int 3
0000000077EF2A96 xchg ax,ax
0000000077EF2A9A xchg ax,ax
0000000077EF2A9E xchg ax,ax
0000000077EF2AA0 int 3 <-- Crashes here
0000000077EF2AA1 ret
It repeats that same code block several times (minus on ax exchanging with itself)
Besides my computer, it has worked on every machine I've tested it on, except for a VM on my machine which won't install the .NET framework because setup downloads 0 bytes out of 0 bytes for the framework)...lovely windows.
I've had similar issues where timing conflicts were causing the failure, and my debugging (breakpoints and stepping through the code) forced the code to run in the correct order.
Try take off optimizations from the Release build (in the project settings) and see if that helps.
I fixed it, the .NET 2.0 Framework was corrupt and when I reinstalled it, everything magically started working again.
I cannot tell you what exactly the problem is, but here's what you could do to get a clue what's really happening. I assume you're using VS2008 or 2005.
Switch to release mode
Go to Debug\Exceptions, and mark all "Thrown" exceptions, like illustrated here: http://vvcap.net/db/JbWS_tzy2IpBoI7R7amm.htp
Run executable in debugger, ignore the warnings from VS that there's no debug info
It does seem that there's a win32 exception thrown some time during execution, but this way or another, you will get one or more messages from debugger explaining what kind of exception happened and where. In most cases those messages make it pretty clear what exactly went wrong
EDIT: One thing I forgot to mention is that unmanaged debugging must also be turned on, such like here (when you start program directly from IDE) or here (when you attach to running process)
Here is a support article with that error. Does that apply?
Perhaps the debugger is eating an excaption the VE3D API is throwing. In VS, do a ctrl+alt+e and change it to break whenever any exception is thrown. This can be tedius b/c it will break on all your try catch blocks, but it might give you some information.
Here is some info. about that PlugInLoader. It seems to imply it must be called from the FirstFrameRendered eventhandler. Perhaps one of your forms is doing that and one not?
Search for #if(DEBUG) directives?
Search for Debug.Assert(?
Have you googled the error? I found this thread (admittedly not horribly helpful)
I once had a similar problem with exactly the same behaviour using a plug-in-system. When loading a plug-in from a MarshalByRef-object (see example code below), it seems as if .NET creates a new AppDomain or Context for the loaded assembly. (Can anyone confirm this? I've not found any source regarding this.)
public class ProxyAssemblyLoader : MarshalByRefObject {
public Assembly GetAssembly(string path) {
return Assembly.LoadFrom(path);
}
}
Furthermore in my case the plug-in loads a different version of the mscorlib. (My app is CLR2 and the loaded is CLR4) Afterwards I used the plug-in by reflection and tried to access a value of the new mscorlib, which was loaded from the other application domain. Usually both should be usable because the mscorlib is a commonly used assembly and only loaded once (see Global Assembly Cache). But it seems as if this is not the case. But in general Microsoft advises to avoid that.
I've not exactly figured out what the problem was, but I figured out the call that causes the application to crash without any hint. Why without any hint? It crashed without any hint, because the thrown exception was only available in "the other" appdomain and not available for the main/default app domain.
The taken action was just implicit copying the value of another appdomains assembly to a local untyped value (object) in the default appdomain. This was enough to get a type identity mismatch error due different versions of the assembly. It seems as if Visual Studio could handle it, but if the application runs standalone it crashes.
This may also explains why you needed to reinstall your .NET. Maybe your installed .NET-Framework was a beta or something like that, which contained a minor difference.
In conclusion some general solutions for the problem could be:
Avoid using different versions of assemblies with different versions of the same type.
In other cases may try to load both assemblies inside the same appdomain. (As far as possible regarding the probing context.)
A solution for cross appdomain communication could be serialization of the values.
(Make sure that the correct .NET-Framework Non-Beta Version is installed.)
we found and fixed this issue with MSFT, we faced this problem with VSTO development.
Apply the following patch from MSFT.
http://support.microsoft.com/kb/975954
http://support.microsoft.com/kb/974372
One thing left I think is to use WinDbg to try and debug it. Here are some links on how to use it:
http://www.codeproject.com/KB/debug/windbg_part1.aspx
http://blogs.msdn.com/johan/archive/2007/11/13/getting-started-with-windbg-part-i.aspx
http://blogs.msdn.com/tess/ (good blog about debugging in general in windbg)
Thinking about it, it could also be some service or something that's clashing. Try stopping all unneeded services and closing unneeded programs (including startup ones) and see what happens then.
I had the exact same issue with one of my console applications. I determined that it was my antivirus (Avast) that was causing the issue.
Add the BIN folder to the exclusion list and disable "DeepScreen".
Then rebuild the project and try again!
Other than that I don't know if I can reproduce it now that it's happened (I've been using this particular application for a week or two now without issue), assuming that I'm running my application in the VS debugger, how should I go about debugging a deadlock after it's happened? I thought I might be able to get at call stacks if I paused the program and hence see where the different threads were when it happened, but clicking pause just threw Visual Studio into a deadlock too till I killed my application.
Is there some way other than browsing through my source tree to find potential problems? Is there a way to get at the call stacks once the problem has occured to see where the problem is? Any other tools/tips/tricks that might help?
What you did was the correct way. If Visual Studio also deadlocks, that happens now and then. It's just bad luck, unless there's some other issue.
You don't have to run the application in the debugger in order to debug it. Run the application normally, and if the deadlock happens, you can attach VS later. Ctrl+Alt+P, select the process, choose debugger type and click attach. Using a different set of debugger types might reduce the risk of VS crashing (especially if you don't debug native code)
A deadlock involves 2 or more threads. You probably know the first one (probably your UI thread) since you noticed the deadlock in your application. Now you only need to find the other one. With knowledge of the architecture, it should be easy to find (e.g. what other threads use the same locks, interact with the UI etc)
If VS doesn't work at all, you can always use windbg. Download here: http://www.microsoft.com/whdc/devtools/debugging/default.mspx
I'd try different approaches in the following order:
First, inspect the code to look for thread-safety violations, making sure that your critical regions don't call other functions that will in turn try to lock a critical region.
Use whatever tool you can get your hands on to visualize thread activity, I use an in-house perl script that parses an OS log we made and graphs all the context switches and shows when a thread gets pre-empted.
If you can't find a good tool, do some logging to see the last threads that were running before the deadlock occurred. This will give you a clue as to where the issue might be caused, it helps if the locking mechanisms have unique names, like if an object has it's own thread, create a dedicated semaphore or mutex just to manage that thread.
I hope this helps. Good luck!
You can use different programs like Intel(R) Parallel Inspector:
http://software.intel.com/en-us/intel-parallel-inspector/
Such programs can show you places in your code with potential deadlocks. However you should pay for it, or use it only evaluation period. Don't know if there is any free tools like this.
Just like anywhere, there're no "Silver bullet" tools to catch all the deadlocks. It is all about the sequence in which different threads aquire resources so your job is to find out where the order was violated. Usually Visual Studio or other debugger will provide stack traces and you will be able to find out where the discrepancy is. DevPartner Studio does provide deadlock analysis but last time I've checked there were too many false positives. Some static analysis tools will find some potential deadlocks too.
Other than that it helps to get the architecture straight to enforce resource aquisition order. For example, layering helps to make sure upper level locks are taken before lower ones but beware of callbacks.