Develop Reference

Application shutting down without notice

I am currently managing a complicated application. It's written in C# and .Net 4.7.2.
Sometimes this program shuts down without notice. No error message even with a try/catch block and MessageBox.Show() in the Main method (I know it's probably not the best way but should work).
There are several threads running at different points, calling external DLLs and sometimes even drivers. So in order to log whether it's another thread that crashes the whole thing, I do this at the beginning :
AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException;
Application.ThreadException += Application_ThreadException;
Because I'm not sure which one is the correct one. In the methods, I log the Exception (after performing null checks) into a file (using File.AppendText and a timestamped based file).
Still nothing. The application keeps crashing after some random amount of time (between 2 and 6 hours) and I have no log information, no error message and I'm getting kind of lost here.
The app is running in Release mode and I cannot use Visual Studio to run the debugger into it (because that would make it easy). Maybe there's another way to run an external debugger ?
Can someone give me a hint on how to catch up for an exception that would cause an application to crash silently ?

Based on your explanations the only thing that brings to my mind is that you have some fire and forget threads in your application that throw exception sometimes but your application can't keep track of them to log or catch their exceptions.
Make sure all your tasks are awaited correctly and you don't have any async void method.
If you really need some fire and forget actions in your app, at least keep them alive with something like private Task fireAndForgetTaskAliver in your classes.
Another probability could be memory leak in your app that causes stack overflow exception.

The only way to catch an exception that is not caught anywhere in the code is indeed to look it the Windows Event Log, under Applications.
Thanks to Pavel Anikhouski for his comment.

Isolate exceptions thrown in an AppDomain to not Crash the Application

TL;DR: How do you isolate add-in exceptions from killing the main process?
I want to have a very stable .Net application that runs less stable code in an AppDomain. This would appear to be one of the prime purposes of the AppDomain in the first place (well, that and security sandboxing) but it doesn't appear to work.
For instance in AddIn.exe:
public static class Program
{
public static void Main(string[] args)
{
throw new Exception("test")
}
}
Called in my 'stable' code with:
var domain = AppDomain.CreateDomain("sandbox");
domain.UnhandledException += (sender, e) => {
Console.WriteLine("\r\n ## Unhandled: " + ((Exception) e.ExceptionObject).Message);
};
domain.ExecuteAssemblyByName("AddIn.exe", "arg A", "arg B")
The exception thrown in the AppDomain gets passed straight to the application that created the domain. I can log these with domain.UnhandledException and catch them in the wrapper application.
However, there are more problematic exceptions thrown, for instance:
public static class Program
{
public static void Main(string[] args)
{
Stackoverflow(1);
}
static int Stackoverflow(int x)
{
return Stackoverflow(++x);
}
}
This will throw a stackoverflow exception that kills the entire application every time. It doesn't even fire domain.UnhandledException - it just goes straight to killing the entire application.
In addition calling things like Environment.Exit() from inside the AppDomain also kill the parent application, do not pass GO, do not collect £200 and don't run any ~Finialiser or Dispose().
It seems from this that AppDomain fundamentally doesn't do what it claims (or at lease what it appears to claim) to do, as it just passes all exceptions straight to the parent domain, making it useless for isolation and pretty weak for any kind of security (if I can take out the parent process I can probably compromise the machine). That would be a pretty fundamental failure in .Net, so I must be missing something in my code.
Am I missing something? Is there some way to make AppDomain actually isolate the code that it's running and unload when something bad happens? Am I using the wrong thing and is there some other .Net feature that does provide exception isolation?

I'll throw on some random thoughts, but what #Will has said is correct regarding permissions, CAS, security transparency, and sandboxing. AppDomains are not quite superman. Regarding exceptions though, an AppDomain is capable of handling most unhandled exceptions. The category of exceptions that they are not is called an asynchronous exception. Finding documentation on such exceptions is a little more difficult now that we have async/await, but it exists, and they come in three common forms:
StackOverflowException
OutOfMemoryException
ThreadAbortException
These exceptions are said to be asynchronous because they can be thrown anywhere, even between CIL opcodes. The first two are about the whole environment dying. The CLR lacks the powers of a Phoenix, it cannot handle these exceptions because the means of doing so are already dead. Note that these rules only exist when the CLR throws them. If you just new-up and instance and throw it yourself, they behave like normal exceptions.
Sidenote: If you ever peek at a memory dump of a process that is hosting the CLR, you will see there are always OutOfMemoryException, ThreadAbortException, and StackOverflowException on the heap, but they have no roots you can see, and they never get GCed. What gives? The reason they are there is because the CLR preallocates them - it wouldn't be able to allocate them at the time they are needed. It wouldn't be able to allocate an OutOfMemoryException when we're out of memory.
There is a piece of software that is able to handle all of these exceptions. Starting in 2005, SQL has had the ability to run .NET assemblies with a feature called SQLCLR. SQL server is a rather important process, and having a .NET assembly throw an OutOfMemoryException and it bringing down the entire SQL process seemed tremendously undesirable, so the SQL team doesn't let that happen.
They do this using a .NET 2.0 feature called constrained execution and critical regions. This is where things like ExecuteCodeWithGuaranteedCleanup come into play. If you are able to host the CLR yourself, start with native code and spin up the CLR yourself, you are then able to change the escalation policy: from native code you are able to handle those managed exceptions. This is how SQL CLR handles those situations.

You can't do anything about Environment.Exit(), just like you can't prevent a user from killing your process in Task Manager. Static analysis for this could be circumvented, as well. I wouldn't worry too much about that. There are things you can do, and things you really can't.
The AppDomain does do what it claims to do. However, what it actually claims to do and what you believe it claims to do are two different things.
Unhandled exceptions anywhere will take down your application. AppDomains don't protect against these. But you can prevent unhandled exceptions from crossing AppDomain boundaries by the following (sorry, no code)
Create your AppDomain
Load and unwrap your plugin controller in this AppDomain
Control plugins through this controller, which
Isolates calls to 3rd party plugins by wrapping them in try/catch blocks.
Really, the only thing an AppDomain gives you is the ability to load, isolate and unload assemblies that you do not fully trust during runtime. You cannot do this within the executing AppDomain. All loaded assemblies stay until execution halts, and they enjoy the same permission set as all other code in the AppDomain.
To be a touch clearer, here's some pseudocode that looks like c# that prevents 3rd-party code from throwing exceptions across the AppDomain boundary.
public class PluginHost : IPluginHost, IPlugin
{
private IPlugin _wrapped;
void IPluginHost.Load(string filename, string typename)
{
// load the assembly (filename) into the AppDomain.
// Activator.CreateInstance the typename to create 3rd party plugin
// _wrapped = the plugin instance
}
void IPlugin.DoWork()
{
try
{
_wrapped.DoWork();
}catch(Exception ex)
// log
// unload plugin whatevs
}
}
This type would be created in your Plugin AppDomain, and its proxy unwrapped in the application AppDomain. You use it to puppet the plugin within the Plugin AppDomain. It prevents exceptions from crossing AppDomain boundaries, performs loading tasks, etc etc. Pulling a proxy of the plugin type into the application AppDomain is very risky, as any object types that are NOT MarshalByRefObject that the proxy can somehow get into your hands (e.g., Throw new MyCustomException()) will result in the plugin assembly being loaded in the application AppDomain, thus rendering your isolation efforts null and void.
(this is a bit oversimplified)

Windows service / A new guard page for the stack cannot be created

I have a windows service that does some intensive work every one minute (actually it is starting a new thread each time in which it syncs to different systems over http). The problem is, that after a few days it suddenly stops without no error message.
I have NLog in place and I have registered for AppDomain.CurrentDomain.UnhandledException. The last entry in the textfile-log is just a normal entry without any problems. Looking in the EventLog, I also can't find any message in the application log, however, there are two entries in the system log.
One basically says that the service has been terminated unexpectedly. Nothing more. The second event (at the same time as the first one) says: "...A new guard page for the stack cannot be created..."
From what I've read, this is probably a stack overflow exception. I'm not parsing any XML and I don't do recursive work. I host a webserver using Gate, Nancy and SignalR and have RavenDB running in embedded mode. Every minute a new task is started using the Taskfactory from .NET 4.0 and I also have a ContinueWith where I re-start a System.Timers.Timer to fire again in one minute.
How can I start investigating this issue? What could be possible reasons for such an error?

Based on the information that you provided, I would at least, at the minimum, do the following:
Pay extra attention to any third party calls, and add additional info logging around those points.
There are some circumstances in which AppDomain.CurrentDomain.UnhandledException won't help you - a StackOverflowException being one of them. I believe the CLR will simply just give you a string in this case instead of a stack trace.
Pay extra attention around areas where more than one thread is introduced.
An example of an often overlooked StackOverflowException is:
private string myString;
public string MyString { get { return MyString; } } //should be myString

I got this on a particular computer and traced it to a c# object referencing itself from within an initializer

Just as a 'for what it is worth' - in my case this error was reported when the code was attempting to write to the Windows Event Log and the interactive user did not have sufficient permission. This was a small console app that logged exceptions to a text file and the event log (if desired). On exception, the text file was being updated but then this error was thrown and not caught by the error handling. Disabling the Event Logging stopped the error occurring.

Just in case any other person is having the same problem, in my case I found that my windows service was trapped in an endless recursive loop accidentally. So If anyone else have this problem, take in consideration method calls that may be causing huge recursive loops.

I think why you might all be stumped is because this MAY BE a SSD hardware fault. I get this error consistently while playing games about every 3-5 hours and its my computers page file failing somehow.. I know it isnt RAM because i replaced my CPU/RAM/MOBO combo trying to battle this. And its not programming because different games and different apps all fail at the same time, unless its windows corruption?
I could be wrong but just an idea.
I have two samsung evo's in raid

Keep an Application Running even if an unhandled Exception occurs

What ?
I am developing a Console Application that needs to keep running 24/7 No Matter WhatIs there any way to stop a Multi-Threaded Application from getting blown up by some unhandled exception happening in "some thread somewhere" ?
Why ?
Please refrain from giving lessons like "you should manage all your exceptions", "this should never happen" etc. I have my reasons : We are in test deployment and we need to keep this running, log exceptions, and restart all threads again. If anything unplanned happens and causes an unhandled exception to be thrown, it needs to be detected and some method called to restart all threads(atomicity is impossible due due the tier design)
This being said, I am aware it might no be possible to restart an application from "within" if it has blown because of and UnhandledException (which I already implemented).
So far, I have used Quartz.net's FileScan Job and a Flag File to detect stuff like that and restart the application from outwards. But this sounds hacky to me. I would like to find something cleaner and less quick and dirty.
DownVoting / Sniping Warning : I KNOW this might NOT be possible "as is'". Please be creative/helpful rather than abruptly critic and think of this more as an "Open question"

If it is going to run 24/7 anyway, why not just write it as a Windows service and take advantage of the recovery options built right into windows?
This approach has the additional advantage of being able to survive a machine reboot, and it will log failures/restarts in the system event logs.

You need to attach an event handler to UnhandledException event on the Current AppDomain:
AppDomain.CurrentDomain.UnhandledException += UnhandledExceptionHandler
Inside the handler, you will have to somehow save enough state (to a file, database, etc.) for the restarted application to pass along to the new threads. Then instantiate a new instance of your Console application with a call to System.Diagnostics.Process.Start("MyConsoleApp.exe").
Be very careful to introduce logic to avoid a continuous loop of crash/restart/crash/restart.

You can't keep a process running "no matter what". What if the process is killed?
You don't want to keep a process running "no matter what". What if the process state is corrupted in such a way that "bad things" happen if it keeps running?

Well I can think of a few drawbacks in the following solution, but it is good enough to me for the moment :
static void Main()
{
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);
CS = new ConsoleServer();
CS.Run();
}
public static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
Exception exception = (Exception)e.ExceptionObject;
Logger.Log("UNHANDLED EXCEPTION : " + e.ExceptionObject.ToString());
Process.Start(#"C:\xxxx\bin\x86\Release\MySelf.exe");
}

If your using .Net 2.0 and above, the answer is you can't.
In the .NET Framework versions 1.0 and 1.1, an unhandled exception
that occurs in a thread other than the main application thread is
caught by the runtime and therefore does not cause the application to
terminate. Thus, it is possible for the UnhandledException event to be
raised without the application terminating. Starting with the .NET
Framework version 2.0, this backstop for unhandled exceptions in child
threads was removed, because the cumulative effect of such silent
failures included performance degradation, corrupted data, and
lockups, all of which were difficult to debug. For more information,
including a list of cases in which the runtime does not terminate, see
Exceptions in Managed Threads.
Taken from here:
http://msdn.microsoft.com/en-us/library/system.appdomain.unhandledexception.aspx
If you want your application to survive, then you will need very aggressive try/catch around your methods so nothing escapes.
I would advise using a windows service as mentioned by others. It's the same as a console application, but with an extra bit of service layer code on top. You could take your console app and covert it to a service application easily. Just need to override service.start/pause/stop methods.

How to host Plug-ins safely with .NET 2.0

I am writing a chess game which allows two programs compete, the player needs to write a DLL and expose a function to tell the main application where his player will move next, suppose the function looks like this
public static void MoveNext(out int x, out int y, out int discKind);
The player's DLL can be written using C# or C++.
In the chess game application, I start a new thread to call the function that the player's DLL exposed to get where he will move in a turn, and I start a timer to prevent the player timeouts, if a player timesout i will kill the corresponding thread by following APIs
thread.Abort();
thread.Join();
I have the following issues as described below:
The thread cannot be killed with 100% assurance (it depends on the player's code)
During test I found that, if the player uses a deep recursions (and if there is memory leak in the player's program), the memory usage of the host application will increase and then the host application will be terminated without any exceptions.
Are there any techniques, ideas or methods that can handle the above issues?
From this CodeInChaos suggested to load player's DLL into separate domain and then unload it when necessary, I am not sure if it still works for the unmanaged DLL (C++) and if it will cause a low efficiency?

An unhandled exception in their AppDomain will still cause your program to terminate in .Net 2.0. You get a chance to respond to the exception through an event handler but not the ability to handle it.
Your best bet is to use processes for the kind of isolation you're looking for.

If you can ensure your plugin DLL's are always managed code, then you have the option of createing a new application domain in your main application logic and loading the assembly containing the plugin into that domain.
This then gives you the option of trapping unhandled excpetions in that specific app domain and you then have the option of Unloading that whole app domain. That way you can cope with other peoples application plugins misbehaving and throwing exceptions. you also gain the option of specifying partial trust to further restrict what a plugin can do.
However this will not help if you cannot enforce the use of managed code plugins, and the earlier option of a set of seperate processes would be more apropriate.
Reading your post agin it seems you have some quality issues with the plugins you have to use. If you must cope with such buggy plugins I would take the previous advice and go with seperate processes.