I have a scenario and i want to know bestpractices to reduce database hits. Scenario is I have a dictionary table in my application where i put all the words/keywords for translation purpose because my system is multilingual.
Keywords are placed all over the page they can be 10 to 20 in one page and on each word it fetches the translation from database if user in not viewing english version of website.
My application in on Asp.Net MVC 2 with C# and LINQ2SQL.
Caching is a good way to reduce database queries. There are 2 levels of cache you could use:
Cache objects (for example results of database queries)
Cache HTML output of entire controller actions or partials
The translation typically don't change very often and the amount of data is limited. Read up all translated strings when the web app is started and put them in a globally accessible Dictionary. Whenever you need the translated strings, look them up in the dictionary.
linq will lazy load, which means the queries won't hit the database unless you access a property returned by the query, so make sure you avoid accessing property before they are really needed.
you could also try to combine linq queries into one and have a look at your loops to make sure there isn't a better way to cycle through your queries.
you should also be able to remove database access altogether and use translation files in xml rather than on a database.
Before you can do things like caching and lazy loading, etc... it's best to figure out WHAT is going wrong.
Enter LinqToSql Profiler. Yes, it's a commercial product .. but it's worth it. Also, it has a DEMO period ..
This can show you the crap performing queries .. and which queries are doing N+1, etc....
Related
I have some SQL Server Store Procs that generates statistical data for charting in a C# web application.
Right now the user in the web app has to wait about 5 minutes to see these charts with updated data and this is a pain in the neck for the user and for me.
Some of the Store procs takes more than 5 minutes to generate the data but the web user don't need to see the info on the fly. Maybe update the chart every 2-3 hours.
So, I dont know what is the best practice to solve this.
I was thinking on creating a windows service that every 2-3 hours will call the SP's and then store the data in different tables.
Any clue on how to deal with this?
Appreciate the help
As I said in the comments, indexed views (kind of like materialized views) can increase performance of certain common queries without having to make temporary tables and things like that.
The benefits of indexed views are performance and that it doesn't require much extra coding and effort. When you create an indexed view as opposed to a temp table, the query navigator will (should) know when to take advantage of this view, without the end user needing to specify a temp or aggregate table.
Examples of the benefits of indexed views and how to implement them can be found here http://msdn.microsoft.com/en-us/library/dd171921(v=sql.100).aspx
here are some links to indexed views. Like the comments said, views allow you to quickly get information rather then always doing a select every time using a stored proc. Read the second link for a very good explanation about views.
MSDN
http://msdn.microsoft.com/en-ca/library/ms187864%28v=sql.105%29.aspx
Very well explained here
http://www.codeproject.com/Articles/199058/SQL-Server-Indexed-Views-Speed-Up-Your-Select-Quer
Okay, so I have a LINQ to SQL system set up on a WCF service. My application contains a reference to this service which it uses to collect data from an SQL database. I use a DataContext object which was generated by SQLMetal.exe.
I have two entity collections in my DataContext object, Clients and Groups. Each Client contains a field which says how many Groups it belongs to (a comma separated list of Group IDs).
In the application, I have a table of Clients. If I select one and click a button, a second table shows details of the Groups that Client is a part of.
Here's the question: When I click this button, do I go to the database for the Groups each time, or should I load the Groups when the application starts and sift through those? The latter would be quicker, but I want a concurrent solution.
The second question (I know there shouldn't be two really, but I just realised I might be confused on this issue): when I run a LINQ query on a collection in my DataContext object, am I getting the latest database data?
Thanks.
For your second question - Yes, each query against LINQ to SQL results in a SQL statement issued to the backing database. And to clarify further this is each time to attempt to enumerate a LINQ statement. I don't mean to imply that every LINQ statement is sent to the database immediately, which of course it isn't.
The first part on caching vs. querying each time is dependent upon other factors. Is it necessary? Meaning, is there a performance hit you're trying to correct? Also how "stale" can the data afford to be before it becomes a concern to your users? Those are question you'd need to take back to the application owners to decide.
As with most real world performance questions... it depends. The best way to tell is to write your application with what 'feels right' and then if performance is a concern, measure and change accordingly.
Yes, you'll be getting the latest database information.
Unless loading the groups takes a significant amount of time, don't cache.
Premature optimisation is never a good idea.
Just bear in mind that you might want to and make sure your collection of groups is nicely decoupled, optimising if you have to will be relatively simple.
I wonder if somebody could point me in the right direction. I've recently started playing with LinqToSQL and love the strongly typed data objects etc.
I'm just struggling to understand the impact on database performance etc. For example, say I was developing a simple user profile page. The page shows basic information about the user, some information on their recent activity, and a list of unread notifications.
If I was developing a stored procedure for this page, I could create a single SP which returns multiple datatables covering all of the required information - resulting in a single db call.
However, using LinqToSQL, this could results in many calls - one for user info, atleast one for activity, atleast one for notifications, if I then want further info on notifications this may result in further calls - multiple db calls.
Should I be worried about the number of db calls happenning as a result of using this design pattern? Ie, are the multiple db handshakes etc going to degrade my db etc?
I'd appreciate your thoughts on this!
Thanks
David
LINQ to SQL can consume multiple results from a stored proc if you need to go that route. Unfortnately the designer has problems mapping them correctly, so you will probably need to create your mapping manually. See http://www.thinqlinq.com/Default/Using-LINQ-to-SQL-to-return-Multiple-Results.aspx.
You can configure LINQ to SQL to eagerly load the child records if you know that you're going to need them for every parent record. Use the DataLoadOptions and .LoadWith to configure it.
You can also project an object graph with multiple child collections in the Select clause of a LINQ query to reduce the number of DB hits that you make.
Ultimately, you need to check a number of options to determine which route is the best performance for your situation. It's not a one size fits all scenario.
Is it worst from a performance standpoint ? Yes, it should be. Multiple roundtrips are usually worse than single.
The real question is, do you mind? Is your application going to receive enough visits to warrant the added complexity of a stored procedure? Or do you value the simplicity of future modifications over raw performance?
In any case, if you need the performance, you can create a stored procedure and map it on your context. This will give you one single call, but return the data as objects
Here is an article explaining a bit about that option:
linq-to-sql-returning-multiple-result-sets
I've been implementing MS Search Server 2010 and so far its really good. Im doing the search queries via their web service, but due to the inconsistent results, im thinking about caching the result instead.
The site is a small intranet (500 employees), so it shouldnt be any problems, but im curious what approach you would take if it was a bigger site.
I've googled abit, but havent really come over anything specific. So, a few questions:
What other approaches are there? And why are they better?
How much does it cost to store a dataview of 400-500 rows? What sizes are feasible?
Other points you should take into consideration.
Any input is welcome :)
You need to employ many techniques to pull this off successfully.
First, you need some sort of persistence layer. If you are using a plain old website, then the user's session would be the most logical layer to use. If you are using web services (meaning session-less) and just making calls through a client, well then you still need some sort of application layer (sort of a shared session) for your services. Why? This layer will be home to your database result cache.
Second, you need a way of caching your results in whatever container you are using (session or the application layer of web services). You can do this a couple of ways... If the query is something that any user can do, then a simple hash of the query will work, and you can share this stored result among other users. You probably still want some sort of GUID for the result, so that you can pass this around in your client application, but having a hash lookup from the queries to the results will be useful. If these queries are unique then you can just use the unique GUID for the query result and pass this along to the client application. This is so you can perform your caching functionality...
The caching mechanism can incorporate some sort of fixed length buffer or queue... so that old results will automatically get cleaned out/removed as new ones are added. Then, if a query comes in that is a cache miss, it will get executed normally and added to the cache.
Third, you are going to want some way to page your result object... the Iterator pattern works well here, though probably something simpler might work... like fetch X amount of results starting at point Y. However the Iterator pattern would be better as you could then remove your caching mechanism later and page directly from the database if you so desired.
Fourth, you need some sort of pre-fetch mechanism (as others suggested). You should launch a thread that will do the full search, and in your main thread just do a quick search with the top X number of items. Hopefully by the time the user tries paging, the second thread will be finished and your full result will now be in the cache. If the result isn't ready, you can just incorporate some simple loading screen logic.
This should get you some of the way... let me know if you want clarification/more details about any particular part.
I'll leave you with some more tips...
You don't want to be sending the entire result to the client app (if you are using Ajax or something like an IPhone app). Why? Well because that is a huge waste. The user likely isn't going to page through all of the results... now you just sent over 2MB of result fields for nothing.
Javascript is an awesome language but remember it is still a client side scripting language... you don't want to be slowing the user experience down too much by sending massive amounts of data for your Ajax client to handle. Just send the prefetched result your client and additional page results as the user pages.
Abstraction abstraction abstraction... you want to abstract away the cache, the querying, the paging, the prefetching... as much of it as you can. Why? Well lets say you want to switch databases or you want to page directly from the database instead of using a result object in cache... well if you do it right this is much easier to change later on. Also, if using web services, many many other applications can make use of this logic later on.
Now, I probably suggested an over-engineered solution for what you need :). But, if you can pull this off using all the right techniques, you will learn a ton and have a very good base in case you want to extend functionality or reuse this code.
Let me know if you have questions.
It sounds like the slow part of the search is the full-text searching, not the result retrieval. How about caching the resulting resource record IDs? Also, since it might be true that search queries are often duplicated, store a hash of the search query, the query, and the matching resources. Then you can retrieve the next page of results by ID. Works with AJAX too.
Since it's an intranet and you may control the searched resources, you could even pre-compute a new or updated resource's match to popular queries during idle time.
I have to admit that I am not terribly familiar with MS Search Server so this may not apply. I have often had situations where an application had to search through hundreds of millions of records for result sets that needed to be sorted, paginated and sub-searched in a SQL Server though. Generally what I do is take a two step approach. First I grab the first "x" results which need to be displayed and send them to the browser for a quick display. Second, on another thread, I finish the full query and move the results to a temp table where they can be stored and retrieved quicker. Any given query may have thousands or tens of thousands of results but in comparison to the hundreds of millions or even billions of total records, this smaller subset can be manipulated very easily from the temp table. It also puts less stress on the other tables as queries happen. If the user needs a second page of records, or needs to sort them, or just wants a subset of the original query, this is all pulled from the temp table.
Logic then needs to be put into place to check for outdated temp tables and remove them. This is simple enough and I let the SQL Server handle that functionality. Finally logic has to be put into place for when the original query changes (significant perimeter changes) so that a new data set can be pulled and placed into a new temp table for further querying. All of this is relatively simple.
Users are so used to split second return times from places like google and this model gives me enough flexibility to actually achieve that without needing the specialized software and hardware that they use.
Hope this helps a little.
Tim's answer is a great way to handle things if you have the ability to run the initial query in a second thread and the logic (paging / sorting / filtering) to be applied to the results requires action on the server ..... otherwise ....
If you can use AJAX, a 500 row result set could be called into the page and paged or sorted on the client. This can lead to some really interesting features .... check out the datagrid solutions from jQueryUI and Dojo for inspiration!
And for really intensive features like arbitrary regex filters and drag-and-drop column re-ordering you can totally free the server.
Loading the data to the browser all at once also lets you call in supporting data (page previews etc) as the user "requests" them ....
The main issue is limiting the data you return per result to what you'll actually use for your sorts and filters.
The possibilities are endless :)
I have an ASP.Net MVC web app that includes a set of Forums. In order to maintain flexible security, I have chosen an access-control-list style of security.
However, this is getting to be a pretty heavy chunk of data to retrieve every time somebody views the forum index.
I am using the EnterpriseLibrary.Caching functionality to cache various non-LINQ items on the site. (The BBCode interpreter, the Skins, and etc.)
My question is this:
What is the safest and most elegant way to cache a LINQ result?
Essentially, I would like to keep a copy of the ACL for each forum in memory to prevent the database hit. That way, for each person that hits the site, at most I would have to fetch group membership information.
All-in-all I'm really looking for a way to cache large abouts of LINQ data effectively, not just these specific rows.
If you've already got a caching system for general objects, all you should need is this:
var whatever = linkQuery.ToList();