A test WCF webservice that I have hosted using IIS 7.5 is consistently slow to respond to calls made after a period of inactivity (i.e. the first call of each day).
From researching this topic I gather that the problem of "application warmup" is commonly encountered when using IIS (e.g. see here).
I have taken the usual steps that are recommended to try and mitigate this problem:
Installed the Application Initialization Module.
Disabled the application pool Idle Time-out, and the Regular Recycling Time Interval (i.e. set them to '0').
Edited the applicationhost.config file so that autoStart=True and startMode="alwaysRunning" for the necessary app pool, and preloadEnabled="true" for my application.
With these settings, I expect the application pool to immediately spin up a worker process when IIS is started, and spin up a new worker process when the existing one exits. Also, I expect the application to be loaded within the worker process.
However, for the first call of each day, the logs show the difference in time between the client making a call, and the webservice receiving the call, can be as much as 10 seconds. Subsequent calls are typically handled in well under 2 seconds.
Curiously, the long response time is not reproduced by making a call following an iisreset command. I would expect that such a heavy-handed operation would put the webservice in a similarly "cold" situation, however this does not seem to be the case.
I would like to know:
What could still be causing the delay in the application "warming up"?
What is the difference in the state of the webservice following iisreset and a long period of inactivity?
Should I resort to a "heart beat" solution to regularly ping the service to keep it alive?
Thanks in advance for any tips or insight.
I'll try to help with you questions:
What could still be causing the delay in the application "warming up"?
Warm up an application does not mean warm up its resources. For instance, if you configure Auto-start with Application Fabric in your WCF application (https://msdn.microsoft.com/en-us/library/ee677260(v=azure.10).aspx), and this application access database using EF, it will not initiate your DBContext.
If you want these resources initialized after your application warmed up, you need to implement a method to initialize your resources, like cache, DBContext, etc.
What is the difference in the state of the webservice following iisreset and a long period of inactivity?
When the application spend long time of inactivity, probably the application pool goes down and it is restarted when it receives any request, like a recycle does.
This link has interest information about the difference between iisreset and application pool recycle, and it can help to answer your question: https://fullsocrates.wordpress.com/2012/07/25/iisreset-vs-recycling-application-pools/
Should I resort to a "heart beat" solution to regularly ping the service to keep it alive?
If you keep on accessing your service, it will probably keep its resources initialized in memory, so can be a good approach.
Anyway, if your Application Pool is configured to recycle in some interval time, it will be recycled and your resources in memory lost.
If it looks problem to you, just turn off this feature going to IIS -> Application Pool -> Advanced settings and set Regular Time Interval=0
For this issue, it's just some suggestions, you need to make some tests and find out the better solution.
Related
How to host a Windows Service in IIS and keep that service runing like it is running on Windows?
Could I use some feature from WCF service?
I've not access to the Windows itself, only to IIS. Inside that service I'll create a thread which at scheduled time will process some data.
In short, you can't.
A more detailed answer is that there are 2 problems:
IIS worker processes are launched only when a HTTP request comes in. This means you can't start your service with the system.
IIS worker processes are recycled (i.e. restarted) on several conditions. For example, a worker process is restarted if no HTTP request comes in for a long time. This means you can't control when your service is shut down, unless you have access to application pool recycling configuration. Keep in mind that the recycling logic only ensures that all pending HTTP requests are complete, but does not await all background threads to complete.
You can come with a partial solution this way:
Create a WCF service method that checks if your long-running thread is alive and if not, starts it.
Create a very simple windows service that periodically (once in 5 seconds) calls that method. Deploy the service somewhere, e.g. on your own machine.
The only question that remains is: do you really need to avoid windows services? Could you find a place to host the service? There are some use cases when a windows service is the best or even the only way.
You cant, in a nut shell.
However you can make use of the health monitoring API specifically the heartbeat functionality. see:
http://msdn.microsoft.com/en-us/library/system.web.management.webheartbeatevent.aspx
for details on the class you will need to implement to be called when there is work to do
also this answer on SO might help
Understanding heartbeat in ASP.NET health monitoring
Once you have implemented a webheartbeatevent derived class you can check your db or what ever you want to check if there is work to do.
A better solution IMHO is to scrap the service entirely and redesign the system to be 100% web based, as services become a deployment and maintenance nightmare. as i assume you are now finding out...
I have read the MSDN article (link) which says something about the IIS process recycling and it turns out that
The new process is started before the old worker process stops, and requests are then directed to the new process.
But, when I read next part of that article i saw this:
Occurrence of multi-instancing: In multi-instancing, two or more instances of a process run simultaneously. Depending on how the application pool is configured, it is possible for multiple instances of a worker process to run, each possibly loading and running the same application code. The occurrence of an overlapped recycle is an example of multi-instancing, as is a Web garden in which two or more processes serve the application pool regardless of the recycling settings.
If your application cannot run in a multi-instance environment, you must configure only one worker process for an application pool (which is the default value), and disable the overlapped recycling feature if application pool recycling is being used.
So, I am a little confused.
On the one hand, I am okay with the thing when all the web requests immediately and permanently are become directed to the new process instance. On the other hand, this article recommends me not to use recycling at all in case if my application, quoting above "cannot run in a multi-instance environment". Which means, in my opinion, that requests may be processes as one or the other process instance, which is not allowed by my app logic.
All I want to know is, if there is any chance, any opportunity, that IIS can go such bad way and collapse my application, or shouldn't I worry about it?
IIS by default will recycle periodically; this is normal and healthy behavior (frees up system resources). IIS will only run multiple processes for the same app temporarily. On a recycle the new process will start and the original will only run until current activities have completed or the timeout period has expired. If you're concerned about session state, look into State Server (SQL being the preferred method). I've got a couple posts regarding this very issue, but check out this link.
I am not too clear about the IIS lifecycle, but my general understanding is:
Every couple of hours IIS resets itself. This is apparently done so as to fix up any memory leaks, resource deadlocks etc. etc. ie. It seems to be a cleanup operation.
Every couple more hours (I think I read 23 hours) the server just stops listening to inbound requests and runs Application_End. An external page request will restart the app.
Can I get a bit more reasoning to why these behaviors occur? Especially with regards to item #2... My server runs internal scheduling behaviors which completely died last night. The reason was that Application_End occurs and no customer requests were happening to start the IIS server again. This seems weird. Why not just clean up memory leaks etc. and then keep IIS running exactly as it was? The only reason I can think of is that it lets the server reclaim memory/cpu used by IIS, but that seems nonsensical and the cause of bugs, such as my scheduler issue!
Each website in IIS sits in an application pool and you have three different sections that influence when an application pool recycles it's worker process; Recycling, Performance and Health. When the process recycles, a new worker process (w3p.exe) is created first to handle any new requests. Any existing requests are completed on the old process before that is then closed. Application_Start and Application_End will run on each process so you can setup and teardown resources appropriately.
The Recycling settings have the most direct impact on when the worker process will recycle and you can choose to restart after a specific number of minutes running, number of requests processed or at a specific time each day. In a web farm using a specific time can ensure that you never have all the severs in the farm recycle at the same time. You can turn all these off so your worker process doesn't recycle but as you stated in your question this leaves the server vulnerable to memory leaks and threads hanging which will stop IIS serving any requests for websites in that application pool.
The Performance settings can shutdown a worker process if it has been idle for an specified number of minutes or if the CPU reaches a specified threshold. You can also increase the number of worker processes for an application pool and create a web garden.
The Health settings monitor worker processes and will shut them down if they fail repeatedly and will check that they start and stop within a specified time.
Technically, IIS is not stopping or resetting. It's the application pool that is being recycled which ensures that the application domain in which your web applications run does not bog down over time due to bugs/inefficiencies in your code, bugs in the framework, etc.
The IIS model is actually very good for the health of a long-running application. Windows Services for example don't get these benefits. If the process crashes, it's done. But because IIS can measure various aspects of your web application like response time, memory consumption, inactivity, etc, it can offer to reset your application under certain circumstances. They're all configurable but you should always strive to develop web applications in such a way that one request does not depend upon a prior request.
You should also not rely on things happening in the web application that are not directly in response to a web request. So if you are starting up a background thread to do some background tasks then I'd recommend moving that out into a separate process (such as a Windows Service or Scheduled Task.) Although if you really don't want to do that, there is an IIS 7 Application Warm-Up Module that will periodically ping your web application in order to start it.
If you are using in-process session state and the resets are causing problems, you may want to consider using a SQL-based session state provider.
In any event, you can read more about configuring the IIS 7 application pool recycling behavior here. http://technet.microsoft.com/en-us/library/cc753179(WS.10).aspx
I think the other posters have already answered your main question sufficiently well however I'd like to address the final part of your question.
Why not just clean up memory leaks etc. and then keep IIS running exactly as it was? The only reason I can think of is that it lets the server reclaim memory/cpu used by IIS, but that seems nonsensical and the cause of bugs, such as my scheduler issue!
and
Why do I need to wait for a web page request to start up my pool, rather then having the server automatically running and gung-ho about receiving client web requests?
Let's think about the following scenario and what would happen if IIS behaved in this way. If we have a machine hosting several thousand websites (i.e. your typical shared hosting environment) each website has it's own application pool (w3p.exe) running. Say that IIS started up a worker pool for each website regardless of if a request to that site had been made, you'd have a few thousand processes starting up at once each idling at say 2MB of RAM. If you've got 2000 websites you've just allocated 4GB of RAM to sit and essentially do nothing and the OS might start eating into the page file without any real need.
Is this desirable? I think you'd agree that the answer is no.
These behaviors can be controlled by changing the app pool recycle settings of your website. Our production website at work recycles its pool every night at 3am, but our QA environment recycles several times a day.
I have several important Quartz events that MUST go off at specific times of the night. Lately I have been noticing that not all the events are run. I have a feeling that overnight our server load is very light (ie. zero users) and that the web server kind of goes to sleep, and hence so does Quartz. Does this seem plausible? I am using Quartz.net within the web server, and not as a separate service.
Yes, that is plausible. In general, it is considered a bad practice to have IIS run scheduled tasks, that is a job better left for a Windows Service, or the built-in Windows Scheduled tasks (which has been much improved for Windows Server 2008).
Your worker process might have been shut down because there are no load. By default, IIS shuts down worker processes after 20 minutes without ingoing requests (you can alter this in the Application Pool settings). Also, worker processes are likely to be restarted after a certain amount of time or requests, or if they consume too much memory.
A quick-fix for your specific problem might be to use the Windows Task Scheduler to request the site periodically to keep it alive - or have it request a URL that triggers your task at the predefined time.
ASP.Net, by default, will shut down AppDomain's after a period of inactivity.
The recommended course of action is to implement timed events either in a Windows Service or as an executable launched from Window's Scheduled Tasks.
It is also possible to change the IIS configuration so that it will not shut down your AppDomain. How exactly this is accomplished varies between versions of IIS, but instructions can easily be found by searching.
I am using C# to write a program that uses a web service from http://msrmaps.com, the problem is sometimes (seemingly at random) the site won't work properly and will return a few different exceptions. Then on subsequent attempts to use the service I get the error over and over, then after a while (sometimes 30 minutes) the service starts working properly again. In order to avoid waiting for the service to work properly again, I usually just close my program and start it back up again. Usually that fixes the problem and I can continue to use the web service.
My question: Is it possible to restart my program within the program or better yet is there a way to somehow re-connect to the web service like the program does when I first run it?
What is the program actually doing that depends on the web service? Does it really need to be re-started? It sounds like you should be able to just have some UI element in the application that attempts to connect to the service. Wrap that connection in some exception handling and somewhere in the application's UI display that the service connection is currently unavailable.
Or am I way off here?
Warning: This is a bit of a hack, but it works.
I'm assuming your application is unattended, so a UI change alerting the user to the need to shut down and restart is not an option.
We had a similar situation where the simplest resolution to a problem was to shut down and restart my app. (We'll call this "App A").
What I wound up doing was writing a second executable (We'll call it "App B") as a console app that did two things.
It used the System.Diagnostics.Process to kill any instance App A.
Use System.Diagnostics.Process to re-launch App A after all instances were killed.
Then in App A, I had a try/catch around the offending code, and when the error I was looking for came up, it would call App B.
This was the only way I could find of killing a program and relaunching it. If anyone has a better solution, PLEASE post it and I'll change my hack to use a better solution.
I would check that you don't have too many simultaneous requests happening at the same time due to each waiting for a long time before returning (and relatedly, that the total allowed connections as per machine.config is high enough). Throttling code can be very useful with unattended use of web-services (or any other remote service), if you've had a few failures in a row then wait a while before allowing another request to be made (I like to roll forward the wait period each time, starting a around a minute, maxing the wait at around 10minutes).
Then while you can't guarantee the other service won't go down, you can help prevent it causing a permanent problem, or reducing the performance of other processes.