I am facing some major issues in case of multiple concurrent users for my ERP web application. When concurrent user count getting more than 30 my application getting slow. We have developed the web application using ASP.NET ,C#, SQL Server 2016.
What point should I check to getting out from this issue?
Is there any tool/script available using which I can get the overhead in IIS or per page ?
I want to know bandwidth overhead as well.
There can several issue that can cause this issue,
1. You might want to check the connections are closed after the SQL call.
2. You might have long running queries which can cause dead lock.
You should be able to do the memory dump to see what are the command running when the slow down is happening also you can use sql profiler to see how long it takes to run the queries.
Related
Right so after upgrading to Sitecore 8.2 from 8.1 with split environment i.e CD and CMS. I'm seeing few performance issues, The CMS works fine but the number of threads is around 200 in local! whereas CD just freezes by just consuming all the memory just after starting the site, there no error shown in the log as well.
Any idea what might be wrong ?
Please check if you are not getting any error continuously in log files on CD server.
Check for redirects if you have and see if they are causing problems. We found this one as an issue in our instance in the past.
You can watch for Sitecore performance counter for more diagnosis.
For CPU high usage process, i think you need to stop some agent processes which considered one of the memory consuming process, try Stop this cleanup agent in sitecore.config file be set the interval to zero
<agent type="Sitecore.Tasks.CleanupAgent" method="Run" interval="06:00:00">
A possible reason may be the size of the Sitecore event queue.
Check the record count from all the Event queues especially the web database:
SELECT count(*) FROM [EventQueue]
If the count is high like 100K you need to clean up for better performance. Work best when there are at most a few thousand records.
See:
Publish Queue, History and Event Queue too big
sitecore-event-queue-how-to-clean-it-and-why
We had a similar issue where the CPU usage was spiking but could not find any error in sitecore log files.Also the web traffic was normal as observed from the IIS logs. This is how we resolved the issue.
In IIS user interface and in the application pool where the site is hosted,please check the currently running IIS worker Processes.Here we can see that each request is in different part of ASP.NET pipeline and currently executing HTTP modules.
Now please check whether any requests are getting stuck at any stage.If there are multiple requests coming from the same URL are getting stuck then it means that some module in that URL is getting hung up or going into an infinite loop.Now we can investigate the modules used in this URL and find the actual issue.
Our team has an application in Android, with a .NET c# backend, hosted in IIS.
Recently, we have observed sudden and unexplainable latencies in our customers with the following scenario:
Without any warning, users are enable to change the channel (Zapping) , since the product has to do with Live Media Streaming, and they can not even log out of the application
The mobile application connected to another backend (still a c# backend) , is working properly, without any problem
After some time (which varies from 6 hours of the first incident, to 5 minutes of the last one), it all turns back to normal.
I have enabled Failed Request Tracing logs, to see if I can get anything from there, and I have results as follows:
<failedRequest url="https://ourDNS.com:443/servertime.aspx"
siteId="1"
appPoolId="DefaultAppPool"
processId="22232"
verb="POST"
remoteUserName=""
userName=""
tokenUserName="NT AUTHORITY\IUSR"
authenticationType="anonymous"
activityId="{80013C53-0802-B500-B63F-84710C7967BB}"
failureReason="TIME_TAKEN"
statusCode="200"
triggerStatusCode="0"
timeTaken="45141"
xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
>
The page described above is a simple page, that first gets the server's timezone, and then after getting the customer's timezone (that can be set manually from the client), returns the exact date and time of the device where the application is hosted, for further calculations of stream program, what is playing now etc. However, for this page, that returns a simple JSON with a string in it, it requires some times more than 45 seconds (to me this is insane).
Another log from Client side in the moment is one Exception as above:
java.net.SocketTimeoutException
at java.net.PlainSocketImpl.read(PlainSocketImpl.java:491)
at java.net.PlainSocketImpl.access$000(PlainSocketImpl.java:46)
at java.net.PlainSocketImpl$PlainSocketInputStream.read(PlainSocketImpl.java:240)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:103)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:191)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:82)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:174)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:180)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:235)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:259)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:279)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:428)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
at com.framework.utilityframe.webhelper.HttpRequest.getHttpResponse(HttpRequest.java:316)
at com.framework.utilityframe.webhelper.HttpRequest.httpRequest(HttpRequest.java:393)
at com.tibo.webtv.web.TiboLog.logBufferingError(TiboLog.java:319)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:324)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:307)
at android.os.AsyncTask$2.call(AsyncTask.java:287)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
Reading through different forums, I have seen different causes of performance leaks, starting from database to IIS and even a misconfiguration of the application. I have discarded database as a cause because:
At the moment of the problem, database parameters were absolutely fine, no changes in queries time execution, no waiting tasks, no locking
Secondly, the mobile and Decoder application connect to the same database, and the mobile application is running just fine with the same queries
Now, if I think of IIS, every Application hosted at that AppPool, was running fine and without delays, but still there may be something I am missing over there
And at least, something that makes me suspicious is the fact that the mobile application differs in two ways with the Decoder application:
First, the mobile application takes the responses from the Backend in XML format, the Decoder uses JSON.
Second,the mobile application uses http requests, and the Decoder uses https (SSL)
If anyone has experienced similar issues, their help would be greatly appreciated. And for any other detail you need, just ask and I will provide.
So,
Today, our team made another test, which included :
Application hosted in one server and database in another
Application and database hosted in a completely different server (Azure environment)
In both cases, the result was the same: Latencies and problem at the service.
The problem was neither at the backend nor the server. First, the Java application by mistake executed Sync Tasks when saving the logs to another server(dedicated, with full potential to keep as much data as you can give). Second, the log server had a full HDD, with more than 1 TB of only DB Logs, so when the application executed those Sync Tasks (which came as the first call, before any interaction with the channels), they received the Socket exceptions. So, maybe for someone else who may see this post: PLEASE,ALWAYS CHECK YOUR TASKS IN YOUR APPLICATION,AND ALWAYS CHECK ANY SERVER RELATED TO YOUR APPLICATION!!! Thank you very much :D
Developing within SharePoint 2010 - all the latest updates for it are installed (SP2 etc.)
Standard farm with 2 application servers, 2 front-end servers, Active Directory server and 2 SQL servers. All this stuff is hosted by Windows Azure Virtual machines, within Virtual network.
While performing simple SPWebApplication.Lookup() noticed that it takes very-very long to complete - about 16 seconds. To compare - locally it takes about 1 second. And on another very similar farm, also hosted in Azure - about 2 seconds.
What attempts were made to fix performance degradation:
Checked configs and network settings, pings etc. - looks 100% OK.
Profiled with SQL Profiler - no bottlenecks found - there is no hard SQL for this request actually.
Double-check that all the servers and DBs are upgraded and up to date.
Kick off all the possible errors that were found in ULS and Windows logs - now it's clear there.
Investigation of metrics with Metalogix Diagnostics manager - as result nothing critical was found. It only sometimes showed that processor queue length is big. But as I know, normal number for it is #of cores +1. So 4-5 in my case is fine. Also it's needed to note that from my perspective - for VM it's also normal to have such number.
Wrote very simple console app that performs lookup of web app. Profiled with Ants profiler. Noticed that call tree differs from the result received locally. Maybe that's OK, cause locally I have standalone installation.
The result at farm is not optimistic - several calls have a huge Hits count. Though, it's clear where the bottleneck in the call tree - all the ideas about the source have already finished. Profiling result as follows: http://1drv.ms/1kYT3rT
It would be great if you could advice.
Thanks in advance.
How many disks do you have on your VM?
Azure has limited IOPS, 500 per disk... organise your databases so they are on different disks to get more IOPS. You can have 16 disks per VM.
http://msdn.microsoft.com/library/azure/dn248436.aspx
I have been using SignalR in my mvc4 asp.net application in order to update Views and have real Time aspect without having to get back to database each time to check states.
The code I integrated in my web api controller and views is working perfectly while debugging. However, I usually get cases where no refresh take place.
I can't seem to figure out the reasons behind such behaviour ( sometimes it works instantly and others not ). I have been relating it to internet connection problems but it seems that it isn't the only fact.
Any ideas or reflexions to keep in mind while using SignalR ?
check for following
Maximum concurrent requests per CPU
Request queue limit
Maximum requests per application
DefaultMessageBufferSize
one of these may be blocking the request.
Reference: http://www.asp.net/signalr/overview/signalr-20/performance-and-scaling/signalr-performance
I have one application which is developed in ASP .NET MVC 3 which using a SQL server database.
Apart from this, I have one console application which calls an external web service and update the same database with the information and business rules. (Basically we iterate the records from Web service and process the business rule and update the same database), we have configured the console application with Windows scheduler to process it periodically.
The problem is, when my Console application runs periodically, it uses the 100% CPU usage (because we're getting more than 2000 records from web service), and because of that my current MVC application is gets haging OR sometime works very very slow because both application are configured on same windows server.
Could anybody please do let me know that How would I resolve this problem where I want both the things on same server because I have central database used by both application.
Thanks in advance.
You haven't given any detail that anyone can really provide resolution, so I'll simply suggest how I would approach it.
First, I would review the database schema with a DBA to make sure there aren't things like table locks (or if there are, come up with strategies to compensate for them). I would then use the SQL Server profiler to see where (or if) there are any bottle necks in SQL server while these things are running. I would then profile the console application to make sure it's not doing something it doesn't need to be doing. I might even consider profiling the web site to see if there's anything in there that might be contributing to slowness.
After that, I would figure out how to get rid of the Console application and work its functionality into the site. Spawning another application on a given web request is not scalable. More than a couple of those come in at once and you've got the potential to bog the server down very easily.