I'm new to profiling. I'm trying to profile a C# application which connects to an SQLite database and retrieve data. The database contains 146856400 rows and the select query retrieves 428800 rows after execution.
On the first execution the main thread takes 246686 ms
On second execution of the same code the main thread takes only 4296 ms
After restarting the system
On the first execution the main thread takes 244533 ms
On the second execution of the same code the main thread takes only 4053 ms
Questions:
1) Why is there a big difference between the first execution timing and the second execution timing
2) After restarting the system why I'm not getting the same results.
Pls help
You experience the difference between cold and warm execution of your query. Cold means the first time and warm all subsequent invocations of your db query.
The first time everything is "cold"
OS file system cache is empty.
SQLLite cache is empty.
ORM dynamic query compilation is not done and cached yet.
ORM Mapper cache is empty.
Garbage Collector needs to tune your working set
....
When you execute your query a second time all these first time initializations (caching) are done and you are measuring the effects of different cache levels as long as there is enough memory available to cache a substantial amount of your requested data.
A performance difference between 4 minutes and 4s is impressive. Both numbers are valid. Measuring something is easy. Telling someone else what exactly you have measured and how the performance can be improved by changing this or that is much harder.
The performance game goes often like this:
Customer: It is slow
Dev: I cannot repro your issue.
Customer: Here is my scenario ....
Dev: I still cannot repro it. Can you give me data set you use and the exact steps you did perform?
Customer: Sure. Here is the data and the test steps.
Dev: Ahh I see. I can make it 10 times faster.
Customer: That is great. Can I have the fix?
Dev: Sure here it is.
Customer: **Very Angry** It has become faster yes. But I cannot read my old data!
Dev: Ups. We need to migrate all your old data to the new much more efficient format.
We need to develop a a conversion tool which will take 3 weeks and your site will
have 3 days downtime while the conversion tool is running.
Or
We keep the old inefficient data format. But then we can make it only 9 times faster.
Customer: I want to access my data faster without data conversion!
Dev: Here is the fix which is 10% slower with no schema changes.
Customer: Finally. The fix does not break anything but it has not become faster?
Dev: I have measured your use case. It is only slow for the first time.
All later data retrievals are 9 times faster than before.
Customer: Did I mention that in my use case I read always different data?
Dev: No you did not.
Customer: Fix it!
Dev: That is not really possible without a major rewrite of large portions of our software.
Customer: The data I want to access is stored in a list. I want to process it sequentially.
Dev: In that case we can preload the data in the background while you are working the current data set. You will only experience a delay for the first data set on each working day.
Customer: Can I have the fix?
Dev: Sure here it is.
Customer: Perfect. It works!
Performance is hard to grasp since most of the time you deal with perceived performance which is subjective. Bringing it down to quantitative measurements is a good start but you need to tune your metrics to reflect actual customer use cases or you will likely optimize at the wrong places like above. A complete understanding of customer requirements and use cases is a must. On the other hand you need to understand your complete system (profile it as hell) to be able to tell the difference between cold and warm query execution and where you can tune the whole thing. These caches become useless if you query for different data all of the time (not likely). Perhaps you need a different index to speed up queries or you buy a SSD or you keep all of the data in memory and do all subsequent queries in memory....
Related
I have a large enterprise web application that is starting to be heavily used. Recently I've noticed that we are making many database calls for things like user permissions, access, general bits of profile information.
From what I can see on Azure we are looking at an average of 50,000 db queries per hour.
We are using Linq to query via the DevExpress XPO ORM. Now some of these are joins, but the majority are simple 1 table queries.
Is constantly hitting the database the best way to be accessing this kind of information? Are there ways for us to offload the database work as some of this information will never change?
Thanks in advance.
Let's start putting this into perspective. With 3600 seconds in an hour you have less than 20 operations per second. Pathetically low in any measurement.
That said, there is nothing wrong with for example caching user permissions for let's say 30 seconds or a minute.
Generally try to cache not in your code, but IN FRONT - the ASP.NET output cache and donut caching are concepts mostly ignored but still most efficient.
http://www.dotnettricks.com/learn/mvc/donut-caching-and-donut-hole-caching-with-aspnet-mvc-4
has more information. Then ignore all the large numbers and run a profiler - see what your real heavy hitters are (likely around permissions as those are used on every page). Put that into a subsystem and cache this. Given that you can preload that into user identity object in the asp.net subsystem - your code should not hit the database in the pages anyway, so the cache is isolated in some filter in asp.net.
Measure. Make sure your SQL is smart - EF and LINQ lead to extremely idiotic SQL because people are too lazy. Avoid instantiating complete objects just to throw them away, ask only for the fields you need. Make sure your indices are efficient. Come back when you start having a real problem (measured).
But the old rule is: cache early. And LINQ optimization is quite far in the back.
For getting user specific information like profile, access etc. from database, instead of fetching it for every request it is better to get information once at the time of login and keep it session. This should reduce your transactions with database
I'm currently getting into PerfView for performance analysis for my (C#) apps.
But typically those apps use a lot of database calls.
So I asked myself questions like:
- How much time is spent in Repositories?
- (How much time is spent waiting for SQL Queries to return?) -> I don't know if this is even possible to discover with PerfView
But from my traces I get barely any useful results. In the "Any Stacks" View it tells me (when I use grouping on my Repository) that 1,5 seconds are spent in my Repsoitory (the whole call is about 45 seconds). And i know this is not really true, because the repositories calls the database A LOT.
Is it just that CPU metric is not captured when waiting for SQL Queries to complete because CPU has nothing to do in this period of time and therefore my times are just including data transformation times etc in the repository?
Thanks for any help!
EDIT:
What i missed is turning on thread times option to get times of blocked code (which is what's happening during database calls i suppose). I got all the stacks now, just have filter out the uninteresting things. But i don't seem to get anywhere.
What's especially interesting for me when using "Thread Time" is the BLOCKED_TIME. But with it the times are off i think. When you look at the screenshot, it tells me that CPU_TIME is 28,384. Which is milliseconds (afaik), but BLOCKED_TIME is 2,314,732, which can't be milliseconds. So percentage for CPU_TIME is very low with 1.2% but 28 out of 70 seconds are still a lot. So the Inclusive Percentage time is comparing apples and oranges here. Can somebody explain?
So, i succeeded.
What I missed (and Vance Morrison was explaining it in his video tutorial actually) is: When doing a wall clock time analysis with perfview, you get accumulated time from all the threads that have been "waiting around" in what is called "BLOCKED_TIME". Which means for a 70 seconds time, alone the finalizer thread adds 70 seconds to this "BLOCKED_TIME" because it was sitting there not doing anything (at least almost anything in my case).
So when doing wall clock time analysis it is important to filter out what you're interested in. For example search for the thread that was taking the most CPU-time and just include this one in your analysis and go further down the stack to find pieces of your code that are expensive (and also might lead to DB or Service Calls). As soon as you a analysis from the point of view of a method you are really getting the times that were spent in this method and the accumulated "BLOCK_TIME" is out of the picture.
What I found most useful is searching for methods in my own code that "seemed time consuming", i switched to the callers view for this method. Which shed some light from where it's called and in the callees view what is responsible for the consuming time further down the stack (a DB call in a repository or service calls for getting some data).
Little bit hard to explain, but as soon as i had understand really the basics of wall clock time analysis, it all made sense at some point.
I recommend this video tutorial: http://channel9.msdn.com/Series/PerfView-Tutorial
Again, great and very powerful tool!
I want to compute the shortest path by computing the distance between school and student locations, here is my code, It works only for the first row in the database but I do not Know why! and there is no error while running but takes a lot of time!It seems like there is an infinity loop !!but I do not Know where is the error??!!
the value of counter is romper of students
Oh man, beginners ;) Funny.
but takes a lot of time!It seems like there is an infinity loop
You do realize that common sense says that an infinite loop does not take a LOT of time, but INFINITE time? SO if that thing comes back after 15 minutes it is likely atotally crappy algorithm, but it is NOT an infinite loop. Infinite is not slow, it is STOP HERE NEVER EXIT.
Given that you run 2 loops on data we do not know and that you run statements that make zero sense, I Would suggest you get a tutorial on how to use the debugger and start - debugging. That is developer baseline info and if you do not learn it now, rather give up programming - there is no chance to write more complex code without regularly using a debuger.
For pure performane there are profilers. Again, unless you hate programming and do not want to make anyhting more than small samples - you WILL need to use a profiler more or less regularly. Sp ypi cam start dpomg so now.
I think you will find out that you do a LOT of SQL stuff. THat is a select / update every time - and as such it is very inefficient (unless it happens ot often, but hey, without the data, how should I know?). I doubt you have so many busses (like: more than a couple of millions), it may make more sense to pull the data into memory once and then not hit the database until your loop is ready. Any select will take some time, and if you do that a lot - welcome to performance hell.
So I am troubleshooting some performance problems on a legacy application, and I have uncovered a pretty specific problem (there may be others).
Essentially, the application is using an object relational mapper to fetch data, but it is doing so in a very inefficient/incorrect way. In effect, it is performing a series of entity graph fetches to fill a datagrid in the UI, and on databinding the grid (it is ASP.Net Webforms) it is doing additional fetches, which lead to other fetches, etc.
The net effect of this is that many, many tiny queries are being performed. Using SQL Profiler shows that a certain page performs over 10,000 queries (to fill a single grid. No query takes over 10ms to complete, and most of them register as 0ms in Profiler. Each query will use and release one connection, and the series of queries would be single-threaded (per http request).
I am very familiar with the ORM, and know exactly how to fix the problem.
My question is: what is the exact effect of having many, many small queries being executed in an application? In what ways does it/can it stress the different components of the system?
For example, what is the effect on the webserver's CPU and memory? Would it flood the connection pool and cause blocking? What would be the impact on the database server's memory, CPU and I/O?
I am looking for relatively general answers, mainly because I want to start monitoring the areas that are likely to be the most affected (I need to measure => fix => re-measure). Concurrent use of the system at peak would likely be around 100-200 users.
It will depend on the database but generally there is a parse phase for each query. If the query has used bind variables it will probably be cached. If not, you wear the hit of a parse and that often means short locks on resources. i.e. BAD. In Oracle, CPU and blocking are much more prevelant at the parse than the execute. SQL Server less so but it's worse at the execute. Obviously doing 10K of anything over a network is going to be a terrible solution, especially x 200 users. Volume I'm sure is fine but that frequency will really highlight all the overhead in comms latency and stuff like that. Connection pools generally are in the hundreds, not tens of thousands, and now you have 10s of thousands of objects all being created, queued, managed, destroyed, garbage collected etc.
But I'm sure you already know all this deep down. Ditch the ORM for this part and write a stored procedure to execute the single query to return your result set. Then put it on the grid.
Does anyone have any experience with receiving and updating a large volume of data, storing it, sorting it, and visualizing it very quickly?
Preferably, I'm looking for a .NET solution, but that may not be practical.
Now for the details...
I will receive roughly 1000 updates per second, some updates, some new rows of data records. But, it can also be very burst driven, with sometimes 5000 updates and new rows.
By the end of the day, I could have 4 to 5 million rows of data.
I have to both store them and also show the user updates in the UI. The UI allows the user to apply a number of filters to the data to just show what they want. I need to update all the records plus show the user these updates.
I have an visual update rate of 1 fps.
Anyone have any guidance or direction on this problem? I can't imagine I'm the first one to have to deal with something like this...
At first though, some sort of in memory database I would think, but will it be fast enough for querying for updates near the end of the day once I get a large enough data set? Or is that all dependent on smart indexing and queries?
Thanks in advance.
It's a very interesting and also challenging problem.
I would approach a pipeline design with processors implementing sorting, filtering, aggregation etc. The pipeline needs an async (threadsafe) input buffer that is processed in a timely manner (according to your 1fps req. under a second). If you can't do it, you need to queue the data somewhere, on disk or in memory depending on the nature of your problem.
Consequently, the UI needs to be implemented in a pull style rather than push, you only want to update it every second.
For datastore you have several options. Using a database is not a bad idea, since you need the data persisted (and I guess also queryable) anyway. If you are using an ORM, you may find NHibernate in combination with its superior second level cache a decent choice.
Many of the considerations might also be similar to those Ayende made when designing NHProf, a realtime profiler for NHibernate. He has written a series of posts about them on his blog.
May be Oracle is more appropriate RDBMS solution fo you. The problem with your question is that at this "critical" levels there are too much variables and condition you need to deal with. Not only software, but hardware that you can have (It costs :)), connection speed, your expected common user system setup and more and more and more...
Good Luck.