I have a website doing some things that I've never seen before. My server is Win 2003 w/ IIS6 I'm using C# and .Net 4.0.
The site is a real-estate website that stores the data directly in my db. The site will run great for a little while and then just die. What I mean is you'll try to view a property's details and it will take the site 2-3 minutes to load, if it loads at all. If I simply resave the web.config file and reupload it to restart the app, it runs just fine for a little while and then will die again. This continues over and over. I've gone to the local copy while the live site has "died" and the local copy will run just fine and then it will die after so long as well. The time frame that it takes varies from 5 minutes to 30 minutes, i believe it has something to do with the number of requests.
Anyone have any clue as to what might be happening? The only the data query on the page is to pull the main data which is the LINQ query below:
public Listing GetListingByMLNumber(string MLNumber)
{
try
{
DatabaseDataContext db = new DatabaseDataContext();
var item = (from a in db.Listings
where a.ML_.ToLower() == MLNumber.ToLower()
select a).FirstOrDefault();
return item;
}
catch (Exception ex)
{
Message = ex.Message;
return null;
}
}
Not closing the database context stands out as the obvious error in the code you provided. Wrap it in a using statement to be sure it gets disposed correctly.
As long as the context lives, you will hold on to a sql connection, which is a limited resource. You will also waste memory by change-tracking the entities you returned. Given your code the context should be garbage collected at some point, but it might still be the problem (And, whether or not this is the problem, you should dispose your database contexts).
Try load testing locally to see if you can reproduce the problem. If you can, then use the debugger to figure out the problem. If not, you probably need to add logging to narrow down the problem.
You could also look at the IIS process to see if it uses absurd amounts of memory, handles, etc. Also check IIS settings for performance and application pool recyling as suggested in another answer here.
I would take a look at the application pool settings to see how the worker processes are being recycled, and I would also look under the Performance tab in IIS to see if there's a bandwidth threshold specified.
If you ever hit this type of problem again then your should add DebugDiag/ADPlus and WinDBG to your diagnostic toolbelt.
When your application hangs again or is taking an exceedingly long time to respond to requests then grab a dump of the worker process using DebugDiag or ADPlus. Load this up into WinDBG, load up SOS (Son of Strike) which is a WinDBG extension for managed code debugging and start digging around.
Tess Ferrandez has a great set of tutorials and labs on how to use these tools effectively:
.NET Debugging Demos - Information and setup instructions
They've gotten me out of a few pickles several times and it's well worth spending the time familiarising yourself with them.
Related
I've written a C#/WPF/Windows 10 application for a client that, when started, runs beautifully - it connects to another app over the network, begins data acquisition, as the data comes in it processes it and displays charts and tables to the user. All is well.
And then Something Bad Happens.
At some point, for no reason I've been able to determine, there's a spike in CPU usage and then the program becomes totally GC-bound. The client sent me this screenshot:
The client is running the app in a completely different environment than what I have (he has several hundred thousand dollars of actual data acquisition hardware, I have a simulator) and is several thousand miles away.
When the problem happens, the client is willing to let me take remote control over WebEx and run the debugger, but since the problem is so infrequent I only get called in once the problem has happened - and by then it's too late. Often times Visual Studio is so overwhelmed by the memory situation that it gives up when asked to do a memory snapshot or even a stack trace (the client has full source so can do full debugging).
Initially I thought this was a memory leak, so for key classes I have code that keeps WeakReferences for each object that is created, and a regular process that scans those lists every 2 minutes, reports back how many objects of each are in existence, and writes that data to a log file. The log file usually starts getting pretty wonky when the problem starts happening, so the log file data I have doesn't show anything abnormal.
I have also run the ANTS Performance Profiler and Memory Profiler on the application - on my computer. Neither one show anything unexpected happening.
QUESTION
Since this only happens on the client machine, and only happens infrequently, is there something I can put in my code to (1) detect the GC becoming unhappy, and (2) log what it is that is making the GC unhappy?
We have a WCF service developed in C# running in a production environment where it crashes every few hours with no observable pattern. Memory usage will hover at ~250mb for a while, then all of a sudden memory usage starts going up until it crashes with an OutOfMemoryException at 4gb (it's a 32bit process).
We have a hard time identifying the problem, our exceptions logged are from different places in the code, presumably from another request trying to use some memory and it receive the exception.
We have taken a memory dump when the process is at 4gb and a list of ~750k database objects is in memory when the crash occurs. We have looked up the queries of those said objects but can't pinpoint the one that loads up the entire table. The service make calls to the database using EF6.
Another thing to note, this problem never occured in our preproduction environment. The data in the database is sufficient in our preproduction environment for this to occur, if it were to load the entire table also. It's probably a specific call with a specific parameter that triggers this issue, but we can't pinpoint it.
I am out of ideas what to try next to solve our issues. Is there a tool that can help us in this situation ?
Thanks
If you want to capture all your SQL and are using Entity, you can print out queries like this
Context.Database.Log = s => Debug.Print(s);
If you mess around with that a bit you can get it to output to a variable and save the result to text file or Db. You would have to wrap it around all Db calls-not sure how big your project is?
Context.Database.Log = null;
turns it off
I have a user reporting this error when they're using my application.
The application is a .NET Winforms application running on Windows XP Embedded, using SQL Server CE 3.5 sp1, and Linq-To-SQL as the ORM. The database itself is located in a subdirectory my application creates in the My Documents folder. The user account is an adminstrator account on the system. There are no other applications or processes connecting to the database.
For the most part, the application seems to run fine. It starts up, can load data from and save data to the database. The user is using the application to access the database maybe a couple hundred times a day. They get this error, but only intermittently. Maybe 3-4 times a day.
In the code itself, all of the calls to the database are using a Linq-To-SQL data context that's wrapped in a using clause. So in other words:
using(MyDataContext db = new MyDataContext(ConnectionString))
{
List<blah> someList = db.SomeTable.Where(//selection criteria).ToList();
return(someList);
}
That's what pretty much all of the calls to the database look like (with the exception that the ones that save data obviously aren't selecting and returning anything). As I mentioned before, they have no issue 99% of the time but only get the shared memory error a few times a day.
My current "fix" is on application startup I simply read all of the data out of the database (there's not a lot) and cache it in memory and converted my database calls to read from the in-memory lists. So far, this seems to have fixed the problem. For a day and a half now they've reported no problems. But this is still bugging me, because I don't know what would cause the error in the first place.
While the application is accessing the database a few hundred times a day, it's typically not in rapid-fire succession. It's usually once every few minutes at the least. However, there is one use-case where there might be two calls one right after the other, as fast as possible. In other words, something like:
//user makes a selectio n on the screen
DatabaseCall1();
DatabaseCall2();
Both of those would follow the pattern in the code block above where they create a new context, do work, and then return. But these calls aren't asynchronous, so I would expect the connection would be closed and disposed of before DatabaseCall2 is invoked. However, could it be that something on the SQL Server CE end isn't closing the connection fast enough? It might explain why it's intermittent since maybe most of the time it doesn't have a problem? I should also mention that this exact program without the fix is installed on a few other systems with the exact same hardware and software (they're clones of each other), and users of the other systems have not reported any errors.
I'm stuck scratching my head because I can't reproduce this error on my development machine or a test machine, and answers to questions about this exception here and other places typically revolve around insufficient user permissions or the database on a shared network folder.
Check this previous post,I think you will find your answer :-
SQL Server CE - Internal error: Cannot open the shared memory region
I'm trying to determine the cause of a very long (imho) initial start up of an ASP.NET application.
The application uses various third party libraries, and lots of references that I'm sure could be consolidated, however, I'm trying to identify (and apportion blame) the dlls and how much they contribute to the extended startup process.
So far, the start up times vary from 2-5 minutes depending on usage of other things on the box. This is unacceptable in my opinion based on the complexity of the site, and I need to reduce this to something in the region of 30 seconds maximum.
To be clear on the scope of the performance I'm looking for, it's the time from first request to the initial Application_Start method being hit.
So where would I start with getting information on which DLL's are loaded, and how long they take to load so I can try to put a cost/benefit together on which we need to tackle/consolidate.
From an ability perspective, I've been using JetBrains dotTrace for a while, and I'm clear on how benchmark the application once we're in the application, but it appears this is outside of the application code, and therefore outside of what I currently know.
What I'm looking for is methodologies on how to get visibility of what is happening before the first entry point into my code.
Note: I know that I can call the default page on recycle/upgrade to do an initial load, but I'd rather solve the actual problem rather than papering over it.
Note2: the hardware is more than sufficiently scaled and separated in terms of functionality, therefore I'm fairly sure that this isn't the issue.
Separate answer on profiling/debugging start up code:
w3wp is just a process that runs .Net code. So you can use all profiling and debugging tools you would use for normal .Net application.
One tricky point is that w3wp process starts automatically on first request to an application and if your tools do not support attaching to process whenever it start it makes problematic to investigate startup code of your application.
Trick to solve it is to add another application to the same Application Pool. This way you can trigger w3wp creation by navigating to another application, than attach/configure your tools against already running process. When you finally trigger your original application tools will see loading happening in existing w3wp process.
With 2-5 minutes delay you may not even need profiler - simply attach Visual Studio debugger the way suggested above and randomly trigger "break all" several times during load of your site. There is a good chance that slowest portion of the code will be on the stack of one of many threads. Also watch out for debug output - may give you some clues what is going on.
You may also use WinDbg to capture stacks of all threads in similar way (could be more light way than VS).
Your DLL references are loaded as needed, not all at once.
Do external references slow down my ASP.NET application? (VS: Add Reference dialog)
If startup is taking 2-5 minutes, I would look at what happens in Application_Start, and at what the DLLs do once loaded. Are they trying to connect to a remote service that is very slow? Is the machine far too small for what it's doing (e.g. running a DB with large amounts of data plus the web server on an AWS micro instance or similar)?
Since the load time is probably not the IIS worker process resolving references, I would turn to traditional application profilers (e.g. Jetbrains, Antz, dotTrace) to see where the time is being spent as the DLLs initialize, and in your Application_Start method.
Entertainment options check along with profiling:
profile everything, add time tracing to everything and log the information
if you have many ASPX views that need to be compiled on startup (I think it is default for release configuration) than it will take some time
references to Web services or other XML serialization related code will need to compile serialization assemblies if none are present yet
access to remote services (including local SQL) may require the services start up too
aggressive caching in application/remote services may require per-population of caches
Production:
What is the goal for start up time? Figure out it first, otherwise you will not be able to reach it.
What is price you are willing to pay to decrease start up time. Adding 1-10 more servers may be cheaper than spending months of development/test time and delaying the product.
Consider multiple servers, rolling restarts with warm up calls, web gardens
If caching of DB objects or in general is an issue consider existing distributed in-memory caches...
Despite a large number of dlls I'm almost sure that for a reasonable application it cannot be a cause of problem. Most of the time it is static objects initialization is causing slow startup.
In C# static variables are initialized when a type is first time accessed. I would recommend to use a sql profiler and see what are the queries that are performed during the start time of the application and from there see what are the objects that are expensive to initialized.
I'm using .Net 2008 and Oracle 10g as my database. The problem I'm getting is after deploying the application in IIS, when multiple users access the same page at a time i'm getting the error. Can't get the output.
Note: Both the users accessing the same page, same menu at a time.
How can I resolve this?
My guess would be a standard thread-safety / synchronization bug, most likely due to some static resource such as a static connection. Obviously this is pure speculation without some more code, but it (=web-sites being highly threaded) is a surprisingly common oversight.
If it is a static resource, then... well, it probably shouldn't be static. Either per-request, or (specifically in the case of connections) scoped to the local code (and let the connection-pooling worry about re-use).
Does it "Work on your machine"? ;)
If not, try to deploy a version locally and attach the debugger to iis. Point two browsers at the site. Whenever your browsers are/seem stuck, open the debugger's threads window and see where the threads are blocked/blocking. You can also ask the debugger to stop on exception throwing
You mean that nothing appears in the Browser?
Look in your program's logs. Any error messages?
Put some trace statements into your code so that you can figure out where it's going.
So the error is saying there's a failure to create a table. Would you expect to create a table for each user? Have a look at the code around the table creation. Consider what the correct behaviour should be when two copies of that code run at the same time.
Again add trace into the code at these points so you can see what happening. Often it's easier to see this than to debug because when mutiple threads are running the debugger gets in the way of reality.,