Say I have some 10 "categories" that I need to reference in a web app. Currently I'm storing these categories in the DB and am retrieving them (category name and id) during pageload in my basepage, storing them in a hashtable and then using the hashtable to reference the categories. I realized that the DB call is being made during each pageload. I only need to pull these categories once as they never change during a given session. What's the best way to do this?
Do whatever I am doing, but store it as a Application variable?
Hard code the categories in the code, and do away with the database?
Store the categories in a session variable? And call the DB only if the session is empty?
Or something else?
I'm just curious to know what the best-practice is for doing such things.
If the categories are user dependent then I'd store them in the Session variable; otherwise I'd use the ASP.NET Caching functionality (or preferred distributed caching). This allows you to avoid hitting the database whilst being able to control how long the categories should be cached for.
Calling the database is not always as expensive as it seems, a typical website makes dozens of calls to the DB on each pageload. But if it becomes troublesome in terms of performance, consider using an ORM solution. The excellent open source NHibernate comes to mind, which is the "de facto" standard for mapping databases to classes and objects. Beyond the mapping, it automatically provides two levels of caching and connection pooling. Without too much trouble, your website outperforms any other's by a landslide.
The downside of using an ORM? To many people, it is considered a rather steep learning curve. If you want to read into this, make sure to pay a visit to the NHibernate Best Practices. A tough read at first, but definitely worthwhile.
If you combine NHibernate with FluentNHibernate, it becomes a breeze to use.
Related
i have a similar requirement to stackoverflow to show a number of metrics on a page in my asp.net-mvc site that are very expensive to calculate. Stackoverflow has a lot of metrics on the page (like user accept rate, etc) which clearly is not being calculated on the fly on page request, given that it would be too slow.
What is a recommended practice for serving up calculated data really fast without the performance penalty (assuming we can accept that this data maybe a little out of date.
is this stored in some caching layer or stored in some other "results" database table so every day there is a job to calculate this data and store the results so they can be queries directly?
assuming that i am happy to deal with the delayed of having this data as a snapshot,what is the best solution for this type of problem.
Probably they may be relying on the Redis data store for such calculations and caching. This post from marcgravell may help.
yes, the answer is caching, how you do it is (can be) the complicated part, if you are using NHibernate adding caching is really easy, is part of your configuration and on the queries you just add .Cacheable and it manages it for you. Caching also depends on the type of environment, if you're using a single worker, web farm or web garden, you would have to build a caching layer to accomodate for your scenario
Although this is a somewhat-recent technique, one really great way to structure your system to make stuff like this possible is by using Command and Query Responsibility Segregation, more often referred to by CQRS.
Imagine the following:
I have a table of 57,000 items that i regularly use in my application to figure out things like targeting groups etc.
instead of querying the database 300,000 times a day, for a table that hardly ever changes it's data, is there a way to store its information in my application and poll data in memory directly? Or, do I have to create some sort of custom datatype for each row and iterate through testing each row, to check for the results i want?
After some googling, the closest thing i could find is in-memory database
thank you,
- theo
SQLite supports in-memory tables.
For 57,000 items that you will be querying against (and want immediately available) I would not recommend implementing just simple caching. For that many items I'd either recommend a distributed memory cache (even if it's only one machine) such as Memcache, Velocity etc or to go with your initial idea of using an in memory database.
Also if you use any full fledged ORM such as NHibernate you can implement it to use clients for the distributed caching tools with almost no work. Many of the major clients have NHibernate implementations for them including Memcache, Velocity and some others. This might be a better solution as you can have it where it's only caching data it truly is using and not all the data it might need.
Read up on Caching
It sounds like this is application level data rather than user level, so you should look into "Caching Application Data"
Here are some samples of caching datatables
If you only need to find rows using the same key all the time, a simple Dictionary<Key, Value> could very well be all that you need. To me, 57,000 items doesn't really sound that much unless each row contains a huge amount of data. However, if you need to search by different columns, an in-memory database is most likely the way to go.
Currently, my entire website does updating from SQL parameterized queries. It works, we've had no problems with it, but it can occasionally be very slow.
I was wondering if it makes sense to refactor some of these SQL commands into classes so that we would not have to hit the database so often. I understand hitting the database is generally the slowest part of any web application For example, say we have a class structure like this:
Project (comprised of) Tasks (comprised of) Assignments
Where Project, Task, and Assignment are classes.
At certain points in the site you are only working on one project at a time, and so creating a Project class and passing it among pages (using Session, Profile, something else) might make sense. I imagine this class would have a Save() method to save value changes.
Does it make sense to invest the time into doing this? Under what conditions might it be worth it?
If your site is slow, you need to figure out what the bottleneck is before you randomly start optimizing things.
Caching is certainly a good idea, but you shouldn't assume that this will solve the problem.
Caching is almost always underutilized in ASP .NET applications. Any time you hit your database, you should look for ways to cache the results.
Serializing objects into the session can be costly in itself, but most likley faster than just hitting the database every single time. You are benefiting now from Execution Plan caching in SQL Server so it's very likely that what you're getting is optimal performance out of your stored procedure.
One option you might consider doing to increase performance is to astract your data into objects via LINQ to SQL (against your sprocs) and then use AppFabric to cache the objects.
http://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx
As for your updates, you should do that directly against the sprocs, but you will also need to clear our the Cache in AppFabric for objects that are affected by the Insert/Update/Delete.
You could also do the same thing simply using the standard Cache as well, but AppFabric has some added benefits.
Use the SQL Profiler to identify your slowest queries, and see if you can improve them with some simple index changes (removing unused indexes, adding missing indexes).
You could very easily improve your application performance by an order of magnitude without changing your front-end app at all.
See http://sqlserverpedia.com/wiki/Find_Missing_Indexes
If you have look up data only, you can store it in Cache object. This will avoid the hits to DB. Only data that can be used globally should be stored in Cache.
If this data requires filtering, you can restore it from Cache, and filter the data before rendering.
Session can be used to store user specific data. But care must be taken that too much of session variables can easily cause performance problems.
This book may be helpful.
http://www.amazon.com/Ultra-Fast-ASP-NET-Build-Ultra-Scalable-Server/dp/1430223839
I have an ASP.Net MVC web app that includes a set of Forums. In order to maintain flexible security, I have chosen an access-control-list style of security.
However, this is getting to be a pretty heavy chunk of data to retrieve every time somebody views the forum index.
I am using the EnterpriseLibrary.Caching functionality to cache various non-LINQ items on the site. (The BBCode interpreter, the Skins, and etc.)
My question is this:
What is the safest and most elegant way to cache a LINQ result?
Essentially, I would like to keep a copy of the ACL for each forum in memory to prevent the database hit. That way, for each person that hits the site, at most I would have to fetch group membership information.
All-in-all I'm really looking for a way to cache large abouts of LINQ data effectively, not just these specific rows.
If you've already got a caching system for general objects, all you should need is this:
var whatever = linkQuery.ToList();
I am building a web-store with many departments and categories. They are stored in our database and accessed often.
We are using URL rewriting so almost every request within the store generates a lookup. We also need to iterate over the data frequently to generate menus for the main store and the department pages.
This information will not change often so I'm thinking that I should load the database into a dictionary to speed up the information retrieval.
I know the standard practice is to load data into the application cache, however i assume that there is some level of serialization that occurs during caching, and for a large data-structure I'm thinking the overhead would be significant.
My impulse on this is to put the dictionary in a static variable in one of the related classes. I would however like to get some input input on this. Am I right in thinking that this method would be faster? Is it horrible practice? Is there a better way that I'm missing?
I can't seem to find much information on this and I'd really appreciate any information that you can share. Thanks!
The Application and Cache collections do not serialize the objects you pass into them, they store the actual reference. Retrieving an object from Cache will not be an expensive operation, no matter how large the object is. Always stick with the Cache objects unless you have a very good reason not to, its just good practice.
The only other thing worth mentioning is to make sure you think about multithreaded access to this collection. You're going to end up with some serious issues very quickly if you don't lock properly
Well, I don't think it's so much work to rewrite code to use a static field instead of application cache if there's need to do so. I'd personally use the cache first. There's no need for premature optimization, have you measured the performance? It may behave just right with the application cache object. Maybe it even works well with db queries? :)
So, my answer is - use the cache and see how it works.
memcached is your friend! (but could be overkill if you're not scaling out)
Any idea how large your dictionary would be in application cache? I'd be tempted to recommend that as a good first option.
IMHO, generally speaking, if you have control on updates on underlying object, you should use static storage. Otherwise, if you are dependent on a 3rd party API for data retrievals, use Caching technology.