I have a ASP.NET MVC 4 application which is running on a Windows 2012 R2 with IIS 8.5 machine. There is a wierd behavior which I cannot figure it out. The cache seems is clearing itself. By cache, I mean MemoryCache.Default, HttpContext.Current.Cache, and also OutputCache. I'm googling the issue for hours and seems nothing is wrong. Can you list cause of clearing cache? I mean is there a checklist which I can test the server against it? Thanks in advance.
Those caches are held in memory, and as such are volatile. They are held against the W3wp process that IIS spawns to handle those requests.
After a period of inactivity, IIS closes down the processes, so these caches will be cleared.
IIS also closes down the processes (recycles the app pool)
By default after 1746 mins (IIRC)
After the memory used by a thread reaches a set threshold
Caches are also isolated by thread, so if you have a web farm/web garden, these caches are not shared, again IIRC
If you need to persist these items longer, then you will need to look at caching the objects in a persisted storage area, such as a db or application state managed server.
Related
My IIS / Apppool are behaving very weird. They keep recycling for almost every minute or even less. I suppose there is a memory leak somewhere.
The private byte: 137.756,00 (137 MB)
Virtual byte: 9.394.648,00 (9.39 GB????)
I only run about 2 websites. This is definitely not normal.
How can I fix this? Do I need to reinstall? I have tried to defrag my PC, but it did not help. I suppose that was because the memories are still being pointed by the application.
I tried to delete all my websites then deployed my application again, but it was not helping.
I remember that I force quit the "W something" process which was downloading some files from external source (trusted source). My web application was downloading through that "W something process" (I cant remember the name).
I force quit about 4-5 times, then this happens. What should I do to fix this?
Update
I tried reinstalled the IIS, but the problem is still around. When the request comes from my browser, I check the process worker:
Virtual Byte = 9.394.648,00
After just starting the app pool. If I doing some debugging in my Visual Studio using IIS Express, there is nothing wrong. There is no recycling and the application runs well.
Is there a way to reset IIS / App Pool process which hold the Virtual Byte (9 GB) to 0? I read a lot about to find some memory leak and something, but in my case it is not. I did something before which made the App Pool keep holding the 9GB virtual bytes even after the app pool recycle, it still hold 9GB virtual bytes!
So I need to know a way like clearing / cleaning / uninstalling / freeing those data hold by the Application Pool.
Update
I found this error in event viewer:
A process serving application pool 'DefaultAppPool' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '5832'. The data field contains the error number.
A test WCF webservice that I have hosted using IIS 7.5 is consistently slow to respond to calls made after a period of inactivity (i.e. the first call of each day).
From researching this topic I gather that the problem of "application warmup" is commonly encountered when using IIS (e.g. see here).
I have taken the usual steps that are recommended to try and mitigate this problem:
Installed the Application Initialization Module.
Disabled the application pool Idle Time-out, and the Regular Recycling Time Interval (i.e. set them to '0').
Edited the applicationhost.config file so that autoStart=True and startMode="alwaysRunning" for the necessary app pool, and preloadEnabled="true" for my application.
With these settings, I expect the application pool to immediately spin up a worker process when IIS is started, and spin up a new worker process when the existing one exits. Also, I expect the application to be loaded within the worker process.
However, for the first call of each day, the logs show the difference in time between the client making a call, and the webservice receiving the call, can be as much as 10 seconds. Subsequent calls are typically handled in well under 2 seconds.
Curiously, the long response time is not reproduced by making a call following an iisreset command. I would expect that such a heavy-handed operation would put the webservice in a similarly "cold" situation, however this does not seem to be the case.
I would like to know:
What could still be causing the delay in the application "warming up"?
What is the difference in the state of the webservice following iisreset and a long period of inactivity?
Should I resort to a "heart beat" solution to regularly ping the service to keep it alive?
Thanks in advance for any tips or insight.
I'll try to help with you questions:
What could still be causing the delay in the application "warming up"?
Warm up an application does not mean warm up its resources. For instance, if you configure Auto-start with Application Fabric in your WCF application (https://msdn.microsoft.com/en-us/library/ee677260(v=azure.10).aspx), and this application access database using EF, it will not initiate your DBContext.
If you want these resources initialized after your application warmed up, you need to implement a method to initialize your resources, like cache, DBContext, etc.
What is the difference in the state of the webservice following iisreset and a long period of inactivity?
When the application spend long time of inactivity, probably the application pool goes down and it is restarted when it receives any request, like a recycle does.
This link has interest information about the difference between iisreset and application pool recycle, and it can help to answer your question: https://fullsocrates.wordpress.com/2012/07/25/iisreset-vs-recycling-application-pools/
Should I resort to a "heart beat" solution to regularly ping the service to keep it alive?
If you keep on accessing your service, it will probably keep its resources initialized in memory, so can be a good approach.
Anyway, if your Application Pool is configured to recycle in some interval time, it will be recycled and your resources in memory lost.
If it looks problem to you, just turn off this feature going to IIS -> Application Pool -> Advanced settings and set Regular Time Interval=0
For this issue, it's just some suggestions, you need to make some tests and find out the better solution.
I work with Session[""] objects in ASP .NET/C# MVC. I do a lot of checking on them, but I ask myself, what can destroy a session object without I specify it ?
I want to be sure that these Session[""] will be intact during some processes and can be only cleared by my instructions or if the user quit the application.
If you're using InProc, then a number of things can kill a sessions state. e.g.
Timeout (this can be set in config)
AppPool recycle
Web.Config change
Server crash
If you need a more durable solution, have a read of this article, and there's a few alternatives in this article too.
It's dependent on the type of session state you use.
InProc - sessions are destroyed if the web server process shuts down or crashes. The default shutdown time is 20 minutes after the IIS Application Pool is in active. If the IIS Application pool hits its memory limits it will also shutdown, so this is the least recommended approach for session state management.
StateServer - sessions are stored in a separate service on a server. The session will stay active until this service shuts down or is unavailable. It's more reliable than InProc, but if the service shuts down you loose any session data.
SQLServer - sessions are stored in a SQL database. This is the most robust solution and sessions are essentially guaranteed until you destroy them, but it's also the most complex to set up, requiring SQL server and a database.
Custom - Some other method of storing sessions that you would have to specify.
More information on the subject can be found in the links below.
http://msdn.microsoft.com/en-us/library/vstudio/ms178581%28v=vs.100%29.aspx
http://msdn.microsoft.com/en-us/library/vstudio/system.web.sessionstate.sessionstatemode%28v=vs.100%29.aspx
At work we are discussing the possibility for a dedicated in-memory cache server. The two choices are Windows AppFabric Cache Server and Memcached. So I setup two tests, one for AppFabric and one for Memcache.
Memcache tests
OS/Appliance
Memcache VMWare Appliance
Virtualized in VMWare Workstation, Quad Core with 8gb of ram with the MemCachedDOTnet_2.0 C# API.
After I pushed a few objects into cache I started my timings and got on average.
Put 0-1MS (accuracy was only to milliseconds)
Get 14-15MS
AppFabric Cache Server
No VM because its all windows. Quad Core with 8gb of ram.
Same thing I pushed a few objects to cache before I started working. Local Cache option was disabled in the client settings.
Put 0-1MS (accuracy was only to milliseconds)
Get 0-1ms (accuracy was only to milliseconds)
Now the question is I have heard MemCached is super fast, but I assumed it to be faster then AppFabric, but not 15ms slower on Gets vs AppFabric. What is everyone else's performance for both AppFabric and/or MemCache. I'm just looking for raw numbers of MS and object size or things I should look at to see if I can make the numbers line up more to my expectations or options to set or check on.
I see that you have enabled local cache on Appfabric. When local cache is enabled, the cache client stores a reference to the object locally. This keeps the object active in the memory of the client application. When the application requests the object, the cache client first checks whether the object resides in the local cache. If so, the reference to the object is returned immediately without contacting the server. If it does not exist, the object is retrieved from the server. After objects are stored in the local cache, your application continues to use those objects until they are invalidated, regardless of whether those objects are updated by another client on the cache cluster. For this reason, it is best to use local cache for data that changes infrequently.
Since you had local cache enabled, the object was returned immediately without contacting the cluster. If your data changes frequently and you have low tolerance for stale data, you should have local cache disabled. Try the same tests on Windows Server Appfabric 1.1 with local cache disabled.
Why don't you test the windows version of memcached so that you're comparing apples with apples? When you're running one of them in a vm you'll also get the overhead of the extra OS..
I am not too clear about the IIS lifecycle, but my general understanding is:
Every couple of hours IIS resets itself. This is apparently done so as to fix up any memory leaks, resource deadlocks etc. etc. ie. It seems to be a cleanup operation.
Every couple more hours (I think I read 23 hours) the server just stops listening to inbound requests and runs Application_End. An external page request will restart the app.
Can I get a bit more reasoning to why these behaviors occur? Especially with regards to item #2... My server runs internal scheduling behaviors which completely died last night. The reason was that Application_End occurs and no customer requests were happening to start the IIS server again. This seems weird. Why not just clean up memory leaks etc. and then keep IIS running exactly as it was? The only reason I can think of is that it lets the server reclaim memory/cpu used by IIS, but that seems nonsensical and the cause of bugs, such as my scheduler issue!
Each website in IIS sits in an application pool and you have three different sections that influence when an application pool recycles it's worker process; Recycling, Performance and Health. When the process recycles, a new worker process (w3p.exe) is created first to handle any new requests. Any existing requests are completed on the old process before that is then closed. Application_Start and Application_End will run on each process so you can setup and teardown resources appropriately.
The Recycling settings have the most direct impact on when the worker process will recycle and you can choose to restart after a specific number of minutes running, number of requests processed or at a specific time each day. In a web farm using a specific time can ensure that you never have all the severs in the farm recycle at the same time. You can turn all these off so your worker process doesn't recycle but as you stated in your question this leaves the server vulnerable to memory leaks and threads hanging which will stop IIS serving any requests for websites in that application pool.
The Performance settings can shutdown a worker process if it has been idle for an specified number of minutes or if the CPU reaches a specified threshold. You can also increase the number of worker processes for an application pool and create a web garden.
The Health settings monitor worker processes and will shut them down if they fail repeatedly and will check that they start and stop within a specified time.
Technically, IIS is not stopping or resetting. It's the application pool that is being recycled which ensures that the application domain in which your web applications run does not bog down over time due to bugs/inefficiencies in your code, bugs in the framework, etc.
The IIS model is actually very good for the health of a long-running application. Windows Services for example don't get these benefits. If the process crashes, it's done. But because IIS can measure various aspects of your web application like response time, memory consumption, inactivity, etc, it can offer to reset your application under certain circumstances. They're all configurable but you should always strive to develop web applications in such a way that one request does not depend upon a prior request.
You should also not rely on things happening in the web application that are not directly in response to a web request. So if you are starting up a background thread to do some background tasks then I'd recommend moving that out into a separate process (such as a Windows Service or Scheduled Task.) Although if you really don't want to do that, there is an IIS 7 Application Warm-Up Module that will periodically ping your web application in order to start it.
If you are using in-process session state and the resets are causing problems, you may want to consider using a SQL-based session state provider.
In any event, you can read more about configuring the IIS 7 application pool recycling behavior here. http://technet.microsoft.com/en-us/library/cc753179(WS.10).aspx
I think the other posters have already answered your main question sufficiently well however I'd like to address the final part of your question.
Why not just clean up memory leaks etc. and then keep IIS running exactly as it was? The only reason I can think of is that it lets the server reclaim memory/cpu used by IIS, but that seems nonsensical and the cause of bugs, such as my scheduler issue!
and
Why do I need to wait for a web page request to start up my pool, rather then having the server automatically running and gung-ho about receiving client web requests?
Let's think about the following scenario and what would happen if IIS behaved in this way. If we have a machine hosting several thousand websites (i.e. your typical shared hosting environment) each website has it's own application pool (w3p.exe) running. Say that IIS started up a worker pool for each website regardless of if a request to that site had been made, you'd have a few thousand processes starting up at once each idling at say 2MB of RAM. If you've got 2000 websites you've just allocated 4GB of RAM to sit and essentially do nothing and the OS might start eating into the page file without any real need.
Is this desirable? I think you'd agree that the answer is no.
These behaviors can be controlled by changing the app pool recycle settings of your website. Our production website at work recycles its pool every night at 3am, but our QA environment recycles several times a day.