Basically, I have a Windows service which performs a batch job.
I have two collections that are related, customerAccounts and events. The events collection logs actions that customers performed on a site, containing the timestamp, the name of the event, the page it occurred on and the username.
The service runs through each account and works out their journey phase and risk of account closure based on what events they have in the Events collection and a set of user-defined rules.
There are about 3,500 accounts and around 100,000 events in my database at present. The service takes just over 1 minute to run on my development PC, but takes seemingly forever on the server (I've estimated it takes roughly 2.5 hours based on modifying the service so it only performs the job on a single customer account.
My machine is a Core i7 with 16GB of RAM, the server is an Intel Xeon E5-2609 (64bit, Win 2008 R2) with 24GB of RAM. I put the database on a much older server (32bit, Windows 2003) and the service took about 2 minutes to run. So, I know that on my dev machine it takes just over a minute, and on older server hardware it takes just over 2 minutes, yet on a modern server it's taking a matter of hours.
Originally, the Mongo Shell was warning that NUMA was enabled on the server and should be switched off to avoid performance problems. This has since been turned off but hasn't seemed to have an affect on performance.
When I run db.currentOp(); on the server, I've noticed that it's always got a query "createIndexes" of some nature (the indexes were created ages ago), yet when I mongodump/restore the database to my dev machine and run the service/currentOp, the "createIndexes" query isn't there. Apart from that, nothing else seems to jump out at me.
Does anyone have any ideas / help on this mysterious performance issue? I'll post currentOp/mongostats if/when required.
Quick answer: I re-installed Mongo. No fancy configuration, just ran the setup and it fixed the issue.
I never worked out why Mongo was constantly creating indexes. The log file is 0.25GB for a single day full of event logs for "creating index".
Related
I'm running a .NET C# (4.5) web forms application and on our production server, the application gets slower and slower for everyone over time until about an hour where it becomes pretty much unusable and we just restart the server.
It works perfectly fine on my local machine. Never gets slower at all. So I'm trying to think of what would be different.
My local machine connect to the same database. There's also some active directory things being used for logins but that also connects to the same place on both my local and production environments.
I have debug off in the web.config. I've also looked up pretty much any other solutions and haven't had luck.
I did see some stuff about viewstates building up with ajax requests. I do have a page with updatepanels that are refreshing every few seconds, but I'm confused as to why the production server would be getting slower over time with this and not my local machine testing.
There's only about 10 people using the application at the moment also. Any ideas?
Refreshing a bunch of updatepanels every few seconds is not a good idea pretty much. I was able to optimize things here and there but in the end it was the auto refreshing that was slowing things down. Completely reprogrammed using signalr to grab updates and now it's very fast.
I developed a medium sized asp.net website that accepts several hundred pdf documents a day. I have a very simple insert SP that inserts the documents into an Image field in SQL Server 2008 R2.
A few times a week I am starting to have an issue where my website seems to be timing out on this insert. Its very strange cause my drop down lists still load and authentication still works. Most of the time I can recycle the application pool or restart IIS and that fixes everything.
That is a very simplified version of course but thats the long and short of it. Has anyone else had an issue like this ??
Thanks!!
Specify the maximum pool size:-
If this doesn't work, set the command timeout to 0. Please send us the exactly error message if the two above solutions don't work
Our team has an application in Android, with a .NET c# backend, hosted in IIS.
Recently, we have observed sudden and unexplainable latencies in our customers with the following scenario:
Without any warning, users are enable to change the channel (Zapping) , since the product has to do with Live Media Streaming, and they can not even log out of the application
The mobile application connected to another backend (still a c# backend) , is working properly, without any problem
After some time (which varies from 6 hours of the first incident, to 5 minutes of the last one), it all turns back to normal.
I have enabled Failed Request Tracing logs, to see if I can get anything from there, and I have results as follows:
<failedRequest url="https://ourDNS.com:443/servertime.aspx"
siteId="1"
appPoolId="DefaultAppPool"
processId="22232"
verb="POST"
remoteUserName=""
userName=""
tokenUserName="NT AUTHORITY\IUSR"
authenticationType="anonymous"
activityId="{80013C53-0802-B500-B63F-84710C7967BB}"
failureReason="TIME_TAKEN"
statusCode="200"
triggerStatusCode="0"
timeTaken="45141"
xmlns:freb="http://schemas.microsoft.com/win/2006/06/iis/freb"
>
The page described above is a simple page, that first gets the server's timezone, and then after getting the customer's timezone (that can be set manually from the client), returns the exact date and time of the device where the application is hosted, for further calculations of stream program, what is playing now etc. However, for this page, that returns a simple JSON with a string in it, it requires some times more than 45 seconds (to me this is insane).
Another log from Client side in the moment is one Exception as above:
java.net.SocketTimeoutException
at java.net.PlainSocketImpl.read(PlainSocketImpl.java:491)
at java.net.PlainSocketImpl.access$000(PlainSocketImpl.java:46)
at java.net.PlainSocketImpl$PlainSocketInputStream.read(PlainSocketImpl.java:240)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:103)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:191)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:82)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:174)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:180)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:235)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:259)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:279)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:428)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
at com.framework.utilityframe.webhelper.HttpRequest.getHttpResponse(HttpRequest.java:316)
at com.framework.utilityframe.webhelper.HttpRequest.httpRequest(HttpRequest.java:393)
at com.tibo.webtv.web.TiboLog.logBufferingError(TiboLog.java:319)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:324)
at com.tibo.webtv.CustomVideoView$Buffering_Problem.doInBackground(CustomVideoView.java:307)
at android.os.AsyncTask$2.call(AsyncTask.java:287)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:856)
Reading through different forums, I have seen different causes of performance leaks, starting from database to IIS and even a misconfiguration of the application. I have discarded database as a cause because:
At the moment of the problem, database parameters were absolutely fine, no changes in queries time execution, no waiting tasks, no locking
Secondly, the mobile and Decoder application connect to the same database, and the mobile application is running just fine with the same queries
Now, if I think of IIS, every Application hosted at that AppPool, was running fine and without delays, but still there may be something I am missing over there
And at least, something that makes me suspicious is the fact that the mobile application differs in two ways with the Decoder application:
First, the mobile application takes the responses from the Backend in XML format, the Decoder uses JSON.
Second,the mobile application uses http requests, and the Decoder uses https (SSL)
If anyone has experienced similar issues, their help would be greatly appreciated. And for any other detail you need, just ask and I will provide.
So,
Today, our team made another test, which included :
Application hosted in one server and database in another
Application and database hosted in a completely different server (Azure environment)
In both cases, the result was the same: Latencies and problem at the service.
The problem was neither at the backend nor the server. First, the Java application by mistake executed Sync Tasks when saving the logs to another server(dedicated, with full potential to keep as much data as you can give). Second, the log server had a full HDD, with more than 1 TB of only DB Logs, so when the application executed those Sync Tasks (which came as the first call, before any interaction with the channels), they received the Socket exceptions. So, maybe for someone else who may see this post: PLEASE,ALWAYS CHECK YOUR TASKS IN YOUR APPLICATION,AND ALWAYS CHECK ANY SERVER RELATED TO YOUR APPLICATION!!! Thank you very much :D
My code is in ASP.NET MVC Razor C# and Database is SQL Server 2012. Right now the code is on localhost.
ok. Before moving to the server, I want to test the website design should able to scale/support 100,000 concurrent users.
Question : I am a .NET Developer only. Is there any way to do the testing for 100,000 concurrent users on a single machine ?
Short answer: Yes, you can create load tests in visual studio (using VS Ultimate/Enterprise test tools) no problems.
Some basic info here: https://msdn.microsoft.com/en-us/library/vstudio/dd293540(v=vs.110).aspx
But...
Your machine will not be able to handle creating 100,000 simultaneous requests, let alone the site/application servicing those requests on a single machine.
You really need to setup a staging environment that will mimic your production implementation, then deploy and load test on that with load balancing and all the bells and whistles. Otherwise the load/stress test will be a waste of time, the stats you will get back from the test will show 100% timeouts over ~1,000 concurrent users (which is not at all a representation of the speed of your app, just the speed of your machine).
Then once you have said staging environment setup. I would suggest spreading the load test over 5-10 PC/VM's as well. This will give the best "real-world" results.
As per your question, you cannot test an application (either web or desktop application) with 100,000 users without the help of testing software's. The type of testing you are willing to perform is known as volume testing or stress testing in which number of users access the application at the same time and we observe the behavior of the application. This can be done by using a software called HP-Performance Center developed by HP. But it is a licensed software. You wont get it for free.
Developing within SharePoint 2010 - all the latest updates for it are installed (SP2 etc.)
Standard farm with 2 application servers, 2 front-end servers, Active Directory server and 2 SQL servers. All this stuff is hosted by Windows Azure Virtual machines, within Virtual network.
While performing simple SPWebApplication.Lookup() noticed that it takes very-very long to complete - about 16 seconds. To compare - locally it takes about 1 second. And on another very similar farm, also hosted in Azure - about 2 seconds.
What attempts were made to fix performance degradation:
Checked configs and network settings, pings etc. - looks 100% OK.
Profiled with SQL Profiler - no bottlenecks found - there is no hard SQL for this request actually.
Double-check that all the servers and DBs are upgraded and up to date.
Kick off all the possible errors that were found in ULS and Windows logs - now it's clear there.
Investigation of metrics with Metalogix Diagnostics manager - as result nothing critical was found. It only sometimes showed that processor queue length is big. But as I know, normal number for it is #of cores +1. So 4-5 in my case is fine. Also it's needed to note that from my perspective - for VM it's also normal to have such number.
Wrote very simple console app that performs lookup of web app. Profiled with Ants profiler. Noticed that call tree differs from the result received locally. Maybe that's OK, cause locally I have standalone installation.
The result at farm is not optimistic - several calls have a huge Hits count. Though, it's clear where the bottleneck in the call tree - all the ideas about the source have already finished. Profiling result as follows: http://1drv.ms/1kYT3rT
It would be great if you could advice.
Thanks in advance.
How many disks do you have on your VM?
Azure has limited IOPS, 500 per disk... organise your databases so they are on different disks to get more IOPS. You can have 16 disks per VM.
http://msdn.microsoft.com/library/azure/dn248436.aspx