I work on a big project in company. We collect data which we get via API methods of the CMS.
ex.
DataSet users = CMS.UserHelper.GetLoggedUser(); // returns dataset with users
Now on some pages we need many different data, not just users, also Nodes of the tree of the CMS or specific data of subtreee.
So we thought of write an own "helper class" in which we later can get different data easy.
ex:
MyHelperClass.GetUsers();
MyHelperClass.Objects.GetSingleObject( ID );
Now the problem is our "Helper Class" is really big and now we like to collect different data from the "Helper Class" and write them into a typed dataset . Later we can give a repeater that typed dataset which contains data from different tables. (which even comes from the methods I wrote before via API)
Problem is: It is so slow now, even at loading the page! Does it load or init the whole class??
By the way CMS is Kentico if anyone works with it.
I'm tired. Tried whole night..but it's soooo slow. Please give a look to that architecture.
May be you find some crimes which are not allowed :S
I hope we get it work faster. Thank you.
alt text http://img705.imageshack.us/img705/3087/classj.jpg
Bottlenecks usually come in a few forms:
Slow or flakey network.
Heavy reading/writing to disk, as disk IO is 1000s of times slower than reading or writing to memory.
CPU throttle caused by long-running or inefficiently implemented algorithm.
Lots of things could affect this, including your database queries and indexes, the number of people accessing your site, lack of memory on your web server, lots of reflection in your code, just plain slow hardware etc. No one here can tell you why your site is slow, you need to profile it.
For what its worth, you asked a question about your API architecture -- from a code point of view, it looks fine. There's nothing wrong with copying fields from one class to another, and the performance penalty incurred by wrapper class casting from object to Guid or bool is likely to be so tiny that its negligible.
Since you asked about performance, its not very clear why you're connecting class architecture to performance. There are really really tiny micro-optimizations you could apply to your classes which may or may not affect performance -- but the four or five nanoseconds you'll gain with those micro-optimizations have already been lost simply by reading this answer. Network latency and DB queries will absolutely dwarf the performance subtleties of your API.
In a comment, you stated "so there is no problem with static classes or a basic mistake of me". Performance-wise, no. From a web-app point of view, probably. In particular, static fields are global and initialized once per AppDomain, not per session -- the variables mCurrentCultureCode and mcurrentSiteName sound session-specific, not global to the AppDomain. I'd double-check those to see your site renders correctly when users with different culture settings access the site at the same time.
Are you already using Caching and Session state?
The basic idea being to defer as much of the data loading to these storage mediums as possible and not do it on individual page loads. Caching especially can be useful if you only need to get the data once and want to share it between users and over time.
If you are already doing these things, ore cant directly implement them try deferring as much of this data gathering as possible, opting to short-circuit it and not do the loading up front. If the data is only occasionally used this can also save you a lot of time in page loads.
I suggest you try to profile your application and see where the bottlenecks are:
Slow load from the DB?
Slow network traffic?
Slow rendering?
Too much traffic for the client?
The profiling world should be part of almost every senior programmer. It's part of the general toolbox. Learn it, and you'll have the answers yourself.
Cheers!
First thing first... Enable Trace for your application and try to optimize Response size, caching and work with some Application and DB Profilers... By just looking at the code I am afraid no one can be able to help you better.
Related
I have looked around for a simple answer but |I haven't found it (Though if I am being blind or impatient, I be happy for someone to post me the link)
I have the following code in my repository
get
{
if (context.entity.Local.Count == 0)
{
return context.entity;
}
return context.entity.Local;
}
I know from common sense that the word local is not querying the database and getting the result set from memory. However, what I would like to know is, how much faster is fetching the result set from local than it is from the database? It is a huge difference?
I am asking as I would like to speed up my web application so I am trying to find weaknesses in the code.
Thanks
First, your common sense makes no sense at all. Local is nothing that is defined at all in EF so it depends on whoever made the repository and could as well refer to something else.
Second - a lot. Easily factor of 1000. THe database is a separate process, involves generating and then parsing SQL. 2x network transfer (or network alyer transfer). Compare that to just reading out a property. 1000 is likely conservative. Not that it may be a lot of time in the database to start with.
It depends on what you DO - but caching in memory and avoiding the database is a valid strategy that can make a lot of difference, performance wise. At the cost of more memory consumption and change synchronization issues. The later is not really relevant for some (never changing) data.
I understand that a static member will be shared by all users of an ASP.NET website; but in this particular case - that's exactly what I want.
It's a private-use webpage I threw together to facilitate web-based chatting between two users. I wanted to avoid persisting data to a database or a datafile, and thought I could store the last X messages in a static concurrent queue. This seems to work great on my development machine.
I'm very inexperienced with ASP.NET, but in all of the examples I've found, none use this approach. Is this a bad-practice, are there 'gotchas' I should be aware of? The alternative, that I can see, is to use a database. But I felt like it would be more effort and, my guess, is more resources (I figure my 'buffer' of messages will take about 40kb of memory and save quite a few trips to the database).
Assuming that you make sure that the entire thing is thread-safe, that will work.
However, IIS can recycle your AppDomain at any time, so your queue may get blow away when you don't expect it.
Even if IIS wouldn't flush and restart your AppDomain every now and then, using static variables for this purpose sounds like a smelly hack to me.
The HttpApplicationState class provides access to an application-wide cache you can use to store information.
ASP.NET Application State Overview
This is perfectly fine as long as your requirements don't change and you are OK with randomly loosing all messages on server side.
I would slightly refactor code to provide "message storage" interface to simplify testing of the code (with potential benefit in the future if you decide to make it more complicated/persisted/multi-user).
Pro of the static storage approach (or HttpApplicationState):
no issues with server side storage of the messages - less privacy concerns. Nothing is stored forever so you can say whatever you want.
extremely simple implementation.
perfect for IM / phone conversation.
unlikely to have performance problems in single server case
Cons:
messages can be lost. Can be mitigated by storing history on the client (i.e. retrieving message with AJAX queries on the same web page)
require more care if data is sensitive when more users are involved/or application is shared with some other code as static data is visible to everyone. Also not much different from any other storage.
Can't be directly migrated to multiple servers/web garden scenario. Really unlikely issue for 2 person chat server.
Sure, one gotcha I've seen in the past has been the use of static variables with Web Gardens.
See this SO question:
Web Garden and Static Objects difficult to understand
Note a key point from the discussion:
Static objects are not shared in web gardens/web farms.
I have an application used to import a large dataset (millions of records) from one database to another, doing a diff in the process (IE removing things that were deleted, updating things, etc). Due to many foreign key constraints and such and to try to speed up the processing of the application, it loads up the entire destination database into memory and then tries to load up parts of the source database and does an in-memory compare, updating the destination in memory as it goes. In the end it writes these changes back to the destination. The databases do not match one to one, so a single table in one may be multiple tables in the other, etc.
So to my question: it currently takes hours to run this process (sometimes close to a day depending on the amount of data added/changed) and this makes it very difficult to debug. Historically, when we encounter a bug, we have made a change, and then rerun the app which has to load all of the data into memory again (taking quite some time) and then run the import process until we get to the part we were at and then we cross our fingers and hope our change worked. This isn't fun :(
To speed up the debugging process I am making an architectural change by moving the import code into a separate dll that is loaded into a separate appdomain so that we can unload it, make changes, and reload it and try to run a section of the import again, picking up where we left off, and seeing if we get better results. I thought that I was a genius when I came up with this plan :) But it has a problem. I either have to load up all the data from the destination database into the second appdomain and then, before unloading, copy it all to the first using the [Serializable] deal (this is really really slow when unloading and reloading the dll) or load the data in the host appdomain and reference it in the second using MarshalByRefObject (which has turned out to make the whole process slow it seems)
So my question is: How can I do this quickly? Like, a minute max! I would love to just copy the data as if it was just passed by reference and not have to actually do a full copy.
I was wondering if there was a better way to implement this so that the data could better be shared between the two or at least quickly passed between them. I have searched and found things recommending the use of a database (we are loading the data in memory to AVOID the database) or things just saying to use MarshalByRefObject. I'd love to do something that easy but it hasn't really worked yet.
I read somewhere that loading a C++ dll or unmanaged dll will cause it to ignore app domains and could introduce some problems. Is there anyway I could use this to my advantage, IE, load an unmanaged dll that holds my list for me or something, and use it to trick my application into using the same memory are for both appdomains so that the lists just stick around when I unload the other dll by unloading the app domain?
I hope this makes sense. It's my first question on here so if I've done a terrible job do help me out. This has frustrated me for a few days now.
App domains approach is a good way of separating for the sake of loading/unloading only part of your application. Unfortunately, as you discovered, exchanging data between two app domains is not easy/fast. It is just like two different system processes trying to communicate which will always be slower than the same process communication. So the way to go is to use quickest possible inter process communication mechanism. Skip WCF as it ads overhead you do not need here. Use named pipes through which you can stream data very fast. I have used it before with good results. To go even faster you can try MemoryMappedFile (link) but that's more difficult to implement. Start with named pipes and if that is too slow go for memory mapped files.
Even when using fast way of sending, you may hit another bottleneck - data serialization. For large amounts of data, standard serialization (even binary) is very slow. You may want to look at Google's protocol buffers.
One word of caution on AppDomain - any uncaught exception in one of the app domains brings the whole process down. They are not that separated, unfortunately.
On the side note. I do not know what your application does but millions of records does not seem that excessive. Maybe there is a room for optimization?
You didn't say if it were SQL Server, but did you look at using SSIS for doing this? There are evidently some techniques that can make it fast with big data.
Work on C#.In my application several time need to select\collect datafrom DB.Fro this task I do the bellow step
1)Write SP
2)Execute the Sp
3)Fill result to Generic collection(ORM)
4)By the collection Bind the control
I want to know is there any mechanism or technique \Advanced technique available help to collect data from database.Thanks in advance
When i again and again rapidly hit the db.then it's performance become bottleneck .What to do?
It sounds like you should be caching some results. In a high load application, caching even for a few seconds can have a big impact on performance. There are a myriad of cache solutions out there; if this is a web app, the inbuilt http-context .Cache should be fine (.NET 4.0 adds MemoryCache to do the same more conveniently in non-web applications).
Re loading the data; you mention ORM - in our experience here, we find most ORMs indeed are a bottleneck for "hot" code paths - a subject I'm talking on in a few hours as it happens. Because we faced this problem, we wrote an intentionally simple but really really fast micro-ORM, dapper-dot-net. It isn't as feature rich as some full ORMs, but if you are trying to load data quick for display, it is ideal.
The other thing, of course, is to look at your query and improve the performance. Look in particular at the logical IO reads, and where they are coming from. It could well be that an extra index or a little denormalization could make a really big difference to your query performance.
Ye, but the only exception is to use a DataReader or a DataTable.
For ex. datareader is usefull for limited view of rows from a large collection being retrieved.
However Datatable is important, if to apply functions on a complete collection of Data.
Plus there are different methods like connection pooling, localviews, indexes that will matter most when Data fetched is more than available Server resources.
TL;DR: Which is likely faster: accessing static local variable, accessing variable stored in HttpRuntime.Cache, or accessing variable stored in memcached?
At work, we get about 200,000 page views/day. On our homepage, we display a promotion. This promotion is different for different users, based on their country of origin and language.
All the different promotions are defined in an XML file on each web server. We have 12 web servers all serving the same site with the same XML file. There are about 50 different promotion combinations based on country/language. We imagine we'll never have more than 200 or so (if ever) promotions (combinations) total.
The XML file may be changed at any time, out of release cycle. When it's changed, the new definitions of promotions should immediately change on the live site. Implementing the functionality for this requirement is the responsibility of another developer and I.
Originally, I wrote the code so that the contents of the XML file were parsed and then stored in a static member of a class. A FileSystemWatcher monitored changes to the file, and whenever the file was changed, the XML would be reloaded/reparsed and the static member would be updated with the new contents. Seemed like a solid, simple solution to keeping the in-memory dictionary of promotions current with the XML file. (Each server doing this indepedently with its local copy of the XML file; all XML files are the same and change at the same time.)
The other developer I was working holds a Sr. position and decided that this was no good. Instead, we should store all the promotions in each server's HttpContext.Current.Cache with a CacheDependency file dependency that automatically monitored file changes, expunging the cached promotions when the file changed. While I liked that we no longer had to use a FileSystemWatcher, I worried a little that grabbing the promotions from the volitile cache instead of a static class member would be less performant.
(Care to comment on this concern? I already gave up trying to advocate not switching to HttpRuntime.Cache.)
Later, after we began using HttpRuntime.Cache, we adopted memcached with Enyim as our .NET interface for other business problems (e.g. search results). When we did that, this Sr. Developer decided we should be using memcached instead of the HttpRuntime (HttpContext) Cache for storing promotions. Higher-ups said "yeah, sounds good", and gave him a dedicated server with memcached just for these promotions. Now he's currently implementing the changes to use memcached instead.
I'm skeptical that this is a good decision. Instead of staying in-process and grabbing this promotion data from the HttpRuntime.Cache, we're now opening a socket to a network memcached server and transmitting its value to our web server.
This has to be less performant, right? Even if the cache is memcached. (I haven't had the chance to compile any performance metrics yet.)
On top of that, he's going to have to engineer his own file dependency solution over memcached since it doesn't provide such a facility.
Wouldn't my original design be best? Does this strike you as overengineering? Is HttpRuntime.Cache caching or memcached caching even necessary?
Not knowing exactly how much data you are talking about (assuming it's not a lot), I tend to somewhat agree with you; raw-speed wise, a static member should be the 'fastest', then Cache. That doesn't necessarily mean it's the best option, of course. Scalability is not always about speed. In fact, the things we do for scalability often negatively (marginally) affect the speed of an application.
More specifically; I do tend to start with the Cache object myself, unless a bit of 'static' data is pretty darn small and is pretty much guaranteed to be needed constantly (in which case I go for static members. Don't forget thread synch too, of course!)
With a modest amount of data that won't change often at all, and can easily be modified when you need to, by altering the files as you note, the Cache object is probably a good solution. memcached may be overkill, and overly complex... but it should work, too.
I think the major possible 'negative' to the memcached solution is the single-point-of-failure issue; Using the local server's Cache keeps each server isolated.
It sounds like there may not really be any choice in your case, politically speaking. But I think your reasoning isn't necessarily all that bad, given what you've shared here.
Very much agree with Andrew here. Few additions/deviations:
For small amount of rarely changing data, static fields would offer best performance. When your caching happens at no UI layer, it avoids taking dependency on System.Web assembly (of course, you can achieve this by other means as well as). However, in general, ASP.NET Cache would also be a good bet (especially when data is large, the cached data can expire if there is memory pressure etc.)
From both speed & scalability, output caching (including browser & down level caching) would be the best option and you should evaluate it. Even if data is changing frequently, output caching for 30-60 seconds can give significant performance boost for very large number of requests. If needed, you can do partial caching (user controls) and/or substitutions. Of course, this needs to be done with combination with data caching.