I wrote a program that uses Linq to talk to an microsoft sql server. It runs nonstop and from time to time does some changes to the db, mostly at midnight. This works. But after a few days the process randomly just disappears. There is no excpetion window or an entry in the system event logs.
Now, I "fixed" it somehow. What I did: I just reconnect to the sql server every time it does some changes.
The sql server uns on the same machine btw. and there are other programs running using that sql server. So it can't be down or something like that. Besides, I'ld expect an exception in that case.
Just in case it's important: There are other clients using that same database.
How is it possible a .net app can just disappear? Shouldn't it throw exceptions? And even if it uses some native code, which this process does not, wouldn't there be a message like "windows terminated this process because of xxxxx"?
How is it possible a .net app can just disappear?
One of three things happened: The program terminated normally, the program terminated itself abnormally (via an exception or a failfast), or some other process terminated the process.
Shouldn't it throw exceptions?
I don't understand the question.
And even if it uses some native code, which this process does not, wouldn't there be a message like "windows terminated this process because of xxxxx"?
Well, first off, it might not be Windows terminating the process. For example, perhaps someone attached a debugger to the process and then instructed the debugger to terminate the process.
Some applications are noisy when they terminate abnormally, and some are not -- in particular, applications which terminate with a failfast by definition do not spend time terminating slowly -- writing to logs and letting you know what happened, and so on. That's because they're terminating as fast as possible.
Now, I "fixed" it somehow. What I did: I just reconnect to the sql server every time it does some changes
Were I in that situation I'd prefer to fix the problem by understanding the problem before I try to fix it.
Related
Here is the thing. I'm connecting via COM to some devices at KNX/EIB. But sometimes - and I want to be ready for worst-case anyways - my application crashes leaving all objects and libraries exposed somewhere, somehow. I noticed when I restart the app I have trouble to get a connection again. I get an error for a connection procedure that is actually working well normally. Sometimes this connect procedure is working sometimes it is not, randomly. That is bad! After some time (several minutes) it seems to work again after a series of complete fails. But I think I see a pattern now. It doesn't work after a crash with no clean disconnect. My guess is there are objects that hold a connection to the device that us why I can't get a new connection. This is why I ask this question.
Question:
How do I unload those unused objects to kill undead connections?
How do I make Windows to check for unused libraries to be unloaded?
I just want to tell Windows, "I messed up badly and I need to continue my work. Please clean up my mess for me, so I can start fresh! Do I deserve a 2nd chance?"
Edit:
The scenario is the app has crashed and closed. I have no references to anything anymore. No finally clause or anything. The app can only be started again. What can I do to clean up the mess that has been made before, programmatically?
Edit 2:
Hans gave me the hint of killing the responsible server. So for now I solve that with calling taskkill on startup (at least as long I'm in dev). And it works!
C:\Windows\System32\taskkill.exe /F /IM Falcon.exe
This is the failure mode of an out-of-process COM server. If the client program crashes to the desktop without releasing the interface pointers then the server is completely unaware that the client isn't around anymore. And tends to get balky when you try to reconnect, many servers just permit one client.
By far the most common way that programmers induce this failure mode is by using a debugger. They'll click the Red Button or use the Stop Debugging command. Bam, no cleanup of course.
COM garbage-collects unused servers automatically. But that isn't particularly fast, takes an easy 10 minutes before it decides it needs to step in. And doesn't always work for every server, Office programs notoriously don't get cleaned-up for example.
Not much you can do about this when your app keels over in regular usage. Otherwise the kind of problem that killed middle-ware. Still, having such a mishap in a C# program is pretty unusual, the CLR releases interface pointers at program termination even when the app crashed with an exception. You'd have to have the very nasty kind of mishaps to bypass this, critical exceptions like ExecutionEngineException or the one this site is named after.
Don't focus too much on the Stop Debugging induced failures, it is normal and using Task Manager to kill the server is expected and required. Otherwise just be sure to get the nasty bugs out of your code and you won't have a problem. If you need more help then be sure to contact the owner of the server, be sure to have a small repro project available that demonstrates the issue.
We're developing a video game that has literally no bugs ever has, like any application, bugs that can on occasion cause hard crashes. Unfortunately a number of the crashes we've cataloged so far are out of our control in terms be being able to solve them or work around them due to the closed source middleware we're using (Unity 3D).
Whilst we can hope and wait for the middleware developer to fix the problem we'd like to see if its possible to at least make the crashes more informative and user friendly. For example - One of the rare crashes our users can have is that certain AV products cause some kind of thread context race condition and cause the game to explode. We'd like to be able to detect the crash and error signature, and provide to the user a link to our wiki or forums on how to resolve it (If possible).
Is it possible to write a lightweight watchdog process or parent process that can respond to crash events on the Windows platforms?
Collecting crash dumps outside the crashing process is essential. You never know whether your unhandled exception handler is affected or not. But there are other options:
Enable WER LocalDumps and write a watchdog (FileSystemWatcher) for the directory where the dumps are stored.
Configure AeDebug and attach your own debugger at the time of the crash.
Supopse I have a unhandled exception (or a known serious, unrecoverable error). The scariest situation is a security breach, but it could apply to anything that means my state is so badly hosed I can't expect to continue safely.
What do I do?
In a traditional application, the usual technique is to end my process, quickly. as soon as possible. I'm calling Process.Exit, TerminateProcess, die, or whatever other tool the environment has that means "END. NOW". Eric Lippert's post expresses the reasoning for this attitude well.
In a production ASP.NET application running on IIS, it's not so simple. I can certainly end the current process and cough an error to the event log or wherever. That's essentially what happens with any unhandled exception. But the next time a request comes in, IIS is just going to spin up a new worker process. If my fatal error was a transient problem that's great.
But if my problem persists past the lifetime of my process, the new one won't be any better. It could even be compounded by the intialization code or a reattempt. Plus, if IIS is running multiple worker processes within the same application pool, even killing my process doesn't kill the application. Logically speaking all those other workers may be hosed too and just not know it yet.
So far I've only come up with two options.
End the process and hope for the best. Knowing that the app will just be restarted, this is pretty much the same as "catch(Exception) {}". Hardly satisfying.
"Reaching out" to tell IIS to disable the app, stop IIS, the machine, etc. This seems like a brutal hack. Moreover I'd guess it's likely to require elevated security credentials. During termination of a possibly-compromised process seems like a poor time to have those.
What I can think of are something as following:
You can go ahead use the advanced setting of an Application Pool in IIS named "Rapid-Fail protection", set the Failure Interval long enough as you like, and make the Maximum Failures as 1, then go ahead thrown the exception and make the IIS think this application pool can't work correctly so that it will send back Service Unavailable to client side or even reset the connection(depend on your setting). For more detail please check it here: Failure Settings for an Application Pool . However you need to be very careful to not overkill, I mean you need to write a very good application that all exception been handled properly and only the one you want to terminate application can really been detected by IIS, otherwise maybe a single user click just brought down your site.
Another solution is just go ahead make it your own code, I mean you can record such an error in some certain way like creating a file named SystemCrashed, and then terminate the Application, then check if file exist on Application_Startup and do nothing but terminate the Application if file been found. Something like a lock. This need more code but maybe safer than IIS settings, I mean there can't be too much overkill as long as you get it right to remove the lock.
In my Server/Client setup, I have the Client applications managing other applications on the machine (start/stop/restart/query processes). Right now, I just have a very basic setup using the Process ID, but it occured to me before it goes live, I need to improve this.
If the process stops and another starts using the same ID in between the times that it I issue a query for it, this will whack the system out. None of the processes that I start will ever come from the same file path, but will often times have the same executable name.
I am not having much luck finding it, but can I find the executable path for a running service? I imagine my best bet when querying the running state would be first to look for the stored ProcessID it should be at, if that is running, check the filepath/executable name to make sure it matches as well.
Would there be a better way to do this, or is this the best possible scenario?
You can use the Process.Exited event to be notified when a process you are monitoring exits. This way there will be no chance of things like that happening "while you aren't looking".
Note: for the Exited event to be raised, first you have to explicitly set Process.EnableRaisingEvents to true.
We have very strange problem, one of our applications is continually querying server by using .net remoting, and every 100 seconds the application stops querying for a short duration and then resumes the operation. The problem is on a client and not on the server because applications actually queries several servers in the same time and stops receiving data from all of them in the same time.
100 Seconds is a give away number as it's the default timeout for a webrequest in .Net.
I've seen in the past that the PSI (Project Server Interface within Microsoft Project) didn't override the timeout and so the default of 100 seconds was applied and would terminate anything talking to it for longer than that time.
Do you have access to all of the code and are you sure you have set timeouts where applicable so that any defaults are not being applied unbeknownst to you?
I've never seen that behavior before and unfortunately it's a vague enough scenario I think you're going to have a hard time finding someone on this board who's encountered the problem. It's likely specific to your application.
I think there are a few investigations you can do to help you narrow down the problem.
Determine whether it's the client or server that is actually stalling. If you have problems determining this, try installing a packet filter and monitor the traffic to see who sent the last data. You likely won't be able to read the binary data but at least you will get a sense of who is lagging behind.
Once you figure out whether it's the client or server causing the lag, attempt to debug into the application and get a breakpoint where the hang occurs. This should give you enough details to help track down the problem. Or at least ask a more defined question on SO.
How is the application coded to implement the continuous querying? Is it in a continuous loop? or a loop with a Thread.Sleep? or is it on a timer ?,
It would first be useful to determine if your system is executing this "trigger" in your code when you expect it to, or if it is, and the remoting server is not responding... so, ...
if you cannot reproduce this issue in a development environment where you can debug it, then, if you can, I suggest you add code to this Loop to write out to a log file (or some other persistence mechanism) each time it "should" be examining whatever conditions it uses to decide whether to query the remoting server or not, and then review those logs when the problem reoccurs...
If you can do the same in your remoting server, to record when the server receives a remoting request, this would help as well...
... and oh yes, just a thought, (I don;t know how you have coded this... ) but if you are using a separate thread in client to issue the remoting request, and the channel is being registered, and unregistered on that separate thread, make sure you are deconflicting the requests, cause you can't register the same port twice on the same machine at the same time...
(although this should probably have raised an exception in your client if this was the issue)