I am using .Net 2.0 and my site seems to reach the deadlock state at certain period. It stops working until I recycle the application pool or change something in web.config file. I think deadlock is causing this issue.
I am wondering if there is any tool to debug/check the site to find the code that could be causing the deadlock.
Right now I had to set recycling interval to 10 minutes which is really bad but it is the only way to solve the problem and there is a lot of codes on the site and I need to find the problem. If I use DOS attack tool, can I find the page/code block that is causing this issue? If I can, what is the best tool to test it?
Cheers!
EDIT
I tried to check the Event Logs and found the following warning. I don't know if it is issue will keep digging now.
Exception information:
Exception type: HttpException
Exception message: Request timed out.
Check the event log
Turn on Health Monitoring
If you use the 'Failed Request Tracing' and it'll produce a nice output which will then tell you what is causing the error, down to the module level. This will then give you the first step into where it's breaking down.
Have a read of this article on iis.net → Troubleshooting Failed Requests Using Tracing in IIS 7
I would attach visual studio to IIS and break the debugger when a deadlock occurs. You can then inspect the call stack of the running threads.
Code Project has a nice article on how to do IIS remote debugging.
Of course, you can very well set up up a test machine with a local IIS and local Visual Studio .NET and do this without the need to remotely debug.
Related
My case is that the job will run for about twelve hours every 60 seconds and then it will just stop and I have to manually start it again. My web job is just a plain CLI that is not using the SDK. It seems as if my web job is being disabled if I have an unhandled exception and then I have to manually start it again. Is this a correct behavior?
As we found in the comments, the root issue was that the Web App did not have the Always On option enabled. After enabling it, the issue went away.
We have a new ASP.NET website running on a pair of load balanced Azure VMs. The website is fairly simple and uses Kentico CMS. Twice in the 24 hours since going live the application pool on both web servers has suddenly stopped (within 5-10 minutes of each other) causing 503: Service unavailable errors.
Looking at Windows system logs I see the error which caused the problem:
Application pool '[[NAME]]' is being automatically disabled due to a
series of failures in the process(es) serving that application pool.
Leading up to this are a series of warnings:
A process serving application pool '[[NAME]]' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '[[PROCESS ID]]'. The data field contains the error
number.
Evidently this is IIS's rapid-fail protection kicking in. What's not clear is how to find the cause of this "fatal communication error".
After some web searching I've installed the Debug Diagnostics Tool which has helped me identify that in every case the relevant process was the IIS worker process (w3wp.exe). This tool is new to me and unfortunately the only time the problem occurred since I installed it, no dumps were generated. However, its logs contain a lot of messages like this:
First chance exception - 0xe0434352 caused by thread with System ID:
[[ID]]
The frustrating thing is that I don't know what steps to take to replicate the error conditions. It never occurred in UAT in a very similar environment, even under load test. Here are some facts about my setup:
ASP.NET version = 4.5.2
Application pool running with identity set to a domain account with modify permission on the website directory
Application set with max one worker process
Any advice much appreciated.
* UPDATE 1 *
I now have DebugDiag dump generated by the "fatal communication error" warning event. Dump summary reads:
Dump Summary
------------
Process Name: w3wp.exe : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Process Architecture: x86
Exception Code: 0xC00000FD
Exception Information: The thread used up its stack.
Heap Information: Present
In the end I tracked this down to a bug in my code. Under very edge-case circumstances the CMS was returning an empty Guid instead of an actual ID which was causing a stack overflow in a recursive method.
The 0xC00000FD exception code I posted above is actually a stack overflow exception, so once I knew that and downloaded the Debug Diagnostcs dump file I was able to replicate the crash scenario locally. That tool, by the way, is incredibly powerful and was able to demonstrate the exact conditions of the crash.
All I can say to people who arrive here with similar issue is - firstly, don't assume the issue is not with your code! And secondly, use Debug Diagnostcs.
First of all, what is your app pool regular recycle time interval setting & overlapping setting in IIS? - If these incidents occur when the recycling is scheduled and overlapping is disabled, this behavior is to be expected. Even when overlapping is enabled, I'd guess that it is somewhat connected to automatic recycling of app pool since both instances are impacted in cca the same time & it occurs twice a day and it can cause logging the warning you mentioned (Here you might find how to disable logging this warning in case it is caused by automatic recycling)
If that leads nowhere, you can find more details about the warning event here:
IIS Application Pool Availability
And about the Debug Diagnostcs tools here:
How to use the Debug Diagnostics tool to troubleshoot an IIS process that stops unexpectedly
We have a C# web application, and the latest deploy doesn't work on our Windows Small Business Server 2008 (IIS7). The exact copy of that site runs fine on my Windows 7 machine (IIS7.5). The previous version and other builds still work on the Server 2008 R2 machine, but this itteration doesn't.
I've checked the W3SVC logs, but no requests are logged. I've checked the eventlog for errors, but no errors are logged. I also checked in fiddler, but the request just doesn't get a response as far as I can tell (Result column remains -)
When you open the url, the browser will just keep loading (no timeout).
Is there anything else I can check or enable to debug this IIS7 behaviour?
Thanks in advance,
Nick.
UPDATE
I published the application again & created a new site in IIS, and this new version works. While my the immediate problem is solved at this time, I would still like to know how to debug IIS7, see how it works & why it would keep loading infinitely.
First, I would drop a regular .html file into the sites directory. Then I would have a browser request that specific static file. This would bypass the .net engine and should be logged.
If for some reason it doesn't work and/or isn't logged then there are other things to check, let us know.
Assuming that it does serve the file and you are pointing to the correct machine then inspect your global.asax file and remove any type of error handling you might have. Also turn off the custom errors section of your web.config. Both of which could result in the server essentially spinning off into nothingness if improperly coded. If you have any type of additional threads you are spinning up on access, then see if you can turn those off or add additional logging.
Next, look in the HTTPERR logs to see if you can identify what's going on. These are located at
%SystemRoot%\system32\LogFiles\HTTPERR\httperr*.log
Info about this log file is at: http://support.microsoft.com/default.aspx?scid=kb;en-us;820729
If your app uses ADO then there is chance that depending where the build occurred on Windows 7 or not and whether SP1 is installed or not (at the time of the build) that your build is broken by some Micorsoft ADO-update contained in SP1 (see http://www.codeproject.com/Articles/225491/Your-ADO-is-broken.aspx).
If no requests are logged in the W3SVC logs then it probably means that IIS is not recieving the request at all - likely due to firewall configuration or similar.
You should diagnose why IIS is unavailable (for example by attempting to serve some static content) and then try again.
Try these:
re-register asp.net runtime with your IIS7
make sure the asp.net extension for the correct version is set to Allowed in 'ISAPI and CGI restrictions' in your IIS
Does this ever happen to you?
You are sitting at your development machine and you are made aware of an unhandled exception in a deployed asp.net application. You visit the deployed web app. You can't see the exception detail in your browser, because custom errors is set to remote only. So you have to login to the web server and instigate the exception.
Is there a built in way to turn custom errors off for certain remote clients?
This only happens to me for trivial applications where I haven't implemented a better solution, like ELMAH. But, it's still annoying when it happens.
2 things. One, if you dont have a sophisticated Exception\Logging Policy already implemented, check out the Microsoft Patterns and Practices Enterprise Library - http://entlib.codeplex.com/ - this may be helpful in tracking down bugs in your software.
Secondly, at the very least, put some logging in your global.asax code behind's Application_Error event, you can capture the last unhandled exception by using something like:
Dim lastError As Exception = Server.GetLastError.GetBaseException
Then you can add custom error pages to your web.config and not worry about debugging from a yellow screen, but still capture any error details.
HTH
You can use remote debugging.
This MSDN Article discusses debugging strategies for ASP.NET. If you scroll down to the "Local and Remote Debugging" heading there's some information for you and a link to the remote debugging article.
Basically you can debug a remote server in visual studio. Not reccomended for production servers, but staging servers for sure.
I developed a simple windows service in C# as per this article.
http://www.codeproject.com/KB/dotnet/simplewindowsservice.aspx
I was successfully able to start the service for the first time and stop it. During the following attempts, I was not able to start the service. I got the following information.
The MyNewService service on Local
Computer started and then stopped.
Some services stop automatically if
they have no work to do, for example,
the Performance Logs and Alerts
service.
Please help.
I outlined here a method we're using to debug our Windows services. Maybe this will help you trace the error. Basically this sounds like some error is occurring while trying to execute the OnStart method.
Basically this means the main thread of your service has crashed for some reason. The most common I've seen is filesystem access to it's own log files.
Sometimes you can find the reason in the event viewer, but unfortunately a lot of the time the user you're running the service as won't actually have access to log it's error. A simple thing to do if you're in a dev environment is to just give the service an administrator account temporarily, firstly cause it'll tell you whether the crash is being caused by lack of access (cause it'll work) and secondly if it's not it'll allow it to write to the event viewer. Make sure to take the admin access of f once you fix it though, cause long-term that can be very dangerous.
Did you look in the event log? You can usually get more detailed error information there about the service error. Also, are you writing out to a log with your service? That's another way you could figure out what's going wrong.
You can get to the event log by right clicking on Computer and selecting "Manage". Under System Tools, look under Event Viewer->Application. This is on Windows XP, but other Windows OS's should be similar.
If the service is on your development machine, you should be able get Visual Studio's debugger to attach to it as it starts so that you could identify if anything is causing it to crash. It involves a bit of registry editing as described here: http://blogs.msdn.com/greggm/archive/2005/02/21/377663.aspx
It sounds like your main thread is dying for some reason. Put a call to System.Diagnostics.Debugger.Break() in your service's startup code, e.g., the Main entry point, the service constructor, or the OnStart() method. When you start your service from the Services MMC, you'll be prompted to enter a debug session. Once you're in Visual Studio, open the Exceptions dialog (from the Debug menu) and check the boxes in the Thrown column. Then debug from there to find the problem.