My azure web application sometimes reacts very slowly. He waits a few seconds before executing the request.
Of course I have the setting "always on" turned on.
It's running on a S2 service plan.
Avg users online 3
No vertical or horizontal scaling configured.
Application
Asp.net MVC
.net Framework 4.6.1
C#
Does anyone have an idea why this problem occasionally occurs?
Ok i see based on your picture that there is a wait time of 98.71% and lots of wait time from the compiler, so i would recommend you to consider to use precompiled views on your mvc app, to avoid the runtime compilation of the views. If you are using Azure DevOps, you should be able to change your task to build the solution and add the following options on the MSBuild arguments.
/p:PrecompileBeforePublish=true /p:UseMerge=true /p:SingleAssemblyName=AppCode
When you see the WebApp being slow it is important to understand what HTTP requests are slow and whether those HTTP requests are slow all the time or it is an intermittent issue? How are the CPU and memory metrics and what is the pattern of slowness? If you have application Insights enabled please navigate to the "Performance" tab to see the requests were are slow and whether they are dependent on an external component.
Collecting CLR profiler in the context of slowness will reveal where the time is spent.
You can navigate to Azure Portal-->WebApp-->Diagnose and solve problem blade-->Diagnostic tools-->Autoheal and enable the rule to collect the CLR profiler traces on slowness.
Once the rule triggers it will collect the profiler traces and build a report for your review.
Related
With this post, I try to help my team and hopefully other .NET teams around the world to help them troubleshoot slow requests in the Azure Web application running as an API for an Angular frontend. The goal is to create a small guideline together with the community. There is a lot of chaotic information out there, but each application is different. Maybe we can list up some checks based on the experience of other teams.
Context:
We have an API running in a .NETCore 2.1.x Azure web application on:
p3v2 app service plan.
Separate S1 Web application for Angular frontend (out of scope of this post)
Azure SQL Database Standard S6
Azure Redis cache Standard 6GB
Application Insights
Important Note: This post is about troubleshooting a n in general fast-running API application which is suddenly slowing down. So no specific API calls, but all requests are suddenly slow which means they cannot directly be linked to specific code.
The API application has an AVG load of 35k requests during 5 minutes (see application insights - requests) and 300-400 simultaneous users.
Usually the requests have an avg response of a few milliseconds.
When the application is running slow it can be caused by several factors. Please find below our first standard troubleshooting guideline:
Check the Memory and CPU usage of the Azure service plan. Is it too high?
Ex: Once we had an issue of Out-of-memory exceptions caused by the Dotnet.exe app still running in 32bit. Make sure you deploy your .net application as 64bit if a lot of memory is needed
Check the Database DTU usage. If it's too high this can also cause slow API requests as they are waiting for the database to reply
Check the load on your Redis cache instance and see if it's not too high.
Now, my question to the community: Are there other things that can slow down the application significantly but are not seen in the three bullets above?
The API was suddenly responding very slow yesterday, however, there were:
no high CPU usage, nor high Memory usage of the app service plan
no high DTU usage of the database
no high usage of Redis cache
Maybe some examples I can think of:
Can sudden application exceptions on one particular request slow down other API requests without influencing the three bullets above?
Can too many requests, sent by one external client, slow down the application without influencing the three bullets above? If yes how can this be noticed?
Other things that can be checked?
I've implemented health checks in one of our web apps:
services.AddHealthChecks()
.AddSqlServer(connectionString.ConnectionString, null, HealthCheckName); // Sql HealthCheck
What I've noticed is that we're getting this at least once a day, then the app will restart.
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. (directaccess-.....****) An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
Then the app will restart and function again. Did anyone run into this issue before?
Option -- 1
It appears that application insights was integrated for this app so review Application Insights data to identify why custom exceptions were thrown by application code or why app was taking a long time to load.
Please follow these instructions to view Application Insights data.
Go to Application Insights blade for this App.
Click on View Application Insights Data.
Option -- 2
If the issue is happening right now, collect .NET Profiler trace to troubleshoot the issue. A profiler trace helps you easily identify the ExceptionType, message and callstack for a .NET exception without installing any additional tools and without changing the state of the problem. Profiler trace helps you identify exceptions in both ASP.NET and ASP.NET Core applications.
Please follow these instructions to collect a profiler trace.
Go to App Service Diagnostics
Choose Diagnostic Tools .
Click on Collect .NET Profiler Trace tile and follow the instructions.
(Collect .NET Profiler Trace tile is enabled only for ASP.NET and ASP.NET Core applications. If your app is an ASP.NET app and don't this tile, choose the application stack from the top right)
Option -- 3
If the issue is not reproducible or intermittent, you can configure AutoHealing's custom action to collect some data (like profiler trace or memory dump) that will help you debug the issue further. The triggers and actions allow you to define various conditions based on request count, slow requests, memory limit on which you can take specific actions like restarting the process, logging an event, or starting another executable.
Please follow these instructions to configure an autohealing rule.
Go to App Service Diagnostics
Choose Diagnostic Tools .
Click on Auto Healing tile under Proactive Tools category
Configure a rule based on your scenario.
Knowing that Entity Framework is slow on a cold query (first query after model compilation), I am doing some of the standard work around methods to speed it up. Mainly pre-compiled views as well as making a dummy http request on the client side as soon as the application loads to trigger a query to start the model process.
My question here is specifically around how this works for a deployed application. For example, if I deploy this on Azure, is it the first cold query for the entire application that will trigger the model compilation, or will this slow cold query happen for each individual user that uses the application? In simple terms, does it happen once and only once, or every time a user hits the site for a new session?
The EF slow start is triggered from the first request/s coming into the web server that requires database services.
A couple points to note,
If you deploy to an Azure web app, ensure that the 'AlwaysOn' application setting is enabled. If not, after a given time period the web app will be suspended and the next request will trigger another cold start.
Similarly if you deploy to a VM with IIS you'll need to check the application recycling settings.
When you deploy a new version of the application code, the process will need to be restarted which will cause another slow start.
A good approach to mitigate such slow starts is by using deployment slots and pre-warming slots before sending actual user traffic to it. This is straightforward to achieve using Azure Web App deployment slots.
Right so after upgrading to Sitecore 8.2 from 8.1 with split environment i.e CD and CMS. I'm seeing few performance issues, The CMS works fine but the number of threads is around 200 in local! whereas CD just freezes by just consuming all the memory just after starting the site, there no error shown in the log as well.
Any idea what might be wrong ?
Please check if you are not getting any error continuously in log files on CD server.
Check for redirects if you have and see if they are causing problems. We found this one as an issue in our instance in the past.
You can watch for Sitecore performance counter for more diagnosis.
For CPU high usage process, i think you need to stop some agent processes which considered one of the memory consuming process, try Stop this cleanup agent in sitecore.config file be set the interval to zero
<agent type="Sitecore.Tasks.CleanupAgent" method="Run" interval="06:00:00">
A possible reason may be the size of the Sitecore event queue.
Check the record count from all the Event queues especially the web database:
SELECT count(*) FROM [EventQueue]
If the count is high like 100K you need to clean up for better performance. Work best when there are at most a few thousand records.
See:
Publish Queue, History and Event Queue too big
sitecore-event-queue-how-to-clean-it-and-why
We had a similar issue where the CPU usage was spiking but could not find any error in sitecore log files.Also the web traffic was normal as observed from the IIS logs. This is how we resolved the issue.
In IIS user interface and in the application pool where the site is hosted,please check the currently running IIS worker Processes.Here we can see that each request is in different part of ASP.NET pipeline and currently executing HTTP modules.
Now please check whether any requests are getting stuck at any stage.If there are multiple requests coming from the same URL are getting stuck then it means that some module in that URL is getting hung up or going into an infinite loop.Now we can investigate the modules used in this URL and find the actual issue.
I have one application which is developed in ASP .NET MVC 3 which using a SQL server database.
Apart from this, I have one console application which calls an external web service and update the same database with the information and business rules. (Basically we iterate the records from Web service and process the business rule and update the same database), we have configured the console application with Windows scheduler to process it periodically.
The problem is, when my Console application runs periodically, it uses the 100% CPU usage (because we're getting more than 2000 records from web service), and because of that my current MVC application is gets haging OR sometime works very very slow because both application are configured on same windows server.
Could anybody please do let me know that How would I resolve this problem where I want both the things on same server because I have central database used by both application.
Thanks in advance.
You haven't given any detail that anyone can really provide resolution, so I'll simply suggest how I would approach it.
First, I would review the database schema with a DBA to make sure there aren't things like table locks (or if there are, come up with strategies to compensate for them). I would then use the SQL Server profiler to see where (or if) there are any bottle necks in SQL server while these things are running. I would then profile the console application to make sure it's not doing something it doesn't need to be doing. I might even consider profiling the web site to see if there's anything in there that might be contributing to slowness.
After that, I would figure out how to get rid of the Console application and work its functionality into the site. Spawning another application on a given web request is not scalable. More than a couple of those come in at once and you've got the potential to bog the server down very easily.