In an ASP.NET or MVC website project (or any other) where and how a collection of users taken from the database should be placed?
For example, I have a table of users in the database and I want to load it once to the memory as a dictionary of <UserId,User> and perform all the operations on it (and from it to the database).
The collection should be accessible from all of the pages/controllers.
What will be the "Best practices" way to do that?
Should I create a static object called Users that will contained the dictionary and some methods (add, remove, etc.) also as static?
Or should it be a non static object with a static dictionary inside it? and if so, where should it be placed?
Or maybe I am thinking of it in a totally wrong way?
Sorry if my question is not 100% clear, I just gave an example that I think can illustrate the scenario.
It seems to me like a basic issue but I am really confused about the right way of designing it.
For our WCF server, we used a static object that contained a table of users and their authorizations. This worked well and prevented frequent database round-trips on every connection.
The real challenge was ensuring this table was up-to-date when user accounts change. We implemented a state refresh mechanism. When someone saves a change to user accounts, the web service detects this change and refreshes its state information.
Note that the .NET Framework 4.0 and higher a MemoryCache class built in.
First of all, using static objects (static properties) in a web application is a horrible idea. Concurrency becomes an issue and weird things with user values changing due to other user input (since a static object is shared across the whole app domain) become apparent.
A static read-only object is an exception to the above.
Probably the best way to handle the scenario in your question is using caching. Cache the list, and then rebuild the cache each time after any updates.
If using .net 4.0 or above, take a look at the System.Runtime.Caching namespace. It is similar to the old System.Web.Caching namespace from earlier versions, but now available to the entire .net framework, and also extensible if needed.
This will take care of "where to put the data".
Then you can implement a Business Logic Layer that handles pulling data from the cache and sending to the UI, communicate with data layer, update the cache after any database updates are performed, etc.
That's how I'd do something like this.
Related
I have done a lot of searching and experimenting and have been unable to find a workable resolution to this problem.
Environment/Tools
Visual Studio 2013
C#
Three tier web application:
Database tier: SQL Server 2012
Middle tier: Entity Framework 6.* using Database First, Web API 2.*
Presentation tier: MVC 5 w/Razor, Bootstrap, jQuery, etc.
Background
I am building a web application for a client that requires a strict three-tier architecture. Specifically, the presentation layer must perform all data access through a web service. The presentation layer cannot access a database directly. The application allows a small group of paid staff members to manage people, waiting lists, and the resources they are waiting for. Based on the requirements the data model/database design is entirely centered around the people (User table).
Problem
When the presentation layer requests something, say a Resource, it is related to at least one User, which in turn is related to some other table, say Roles, which are related to many more Users, which are related to many more Roles and other things. The point being that, when I query for just about anything EF wants to bring in almost the entire database.
Normally this would be okay because of EF's default lazy-load behavior, but when serializing just about any object to JSON for returning to the presentation layer, the Newtonsoft.Json serializer hangs for a long time then blows a stack error.
What I Have Tried
Here is what I have attempted so far:
Set Newtonsoft's JSON serialier ReferenceLoopHandling setting to Ignore. No luck. This is not cyclic graph issue, it is just the sheer volume of data that gets brought in (there are over 20,000 Users).
Clear/reset unneeded collections and set reference properties to null. This showed some promise, but I could not get around Entity Framework's desire to track everything.
Just setting nav properties to null/clear causes those changes to be saved back to the database on the next .SaveChanges() (NOTE: This is an assumption here, but seemed pretty sound. If anyone knows different, please speak up).
Detaching the entities causes EF to automatically clear ALL collections and set ALL reference properties to null, whether I wanted it to or not.
Using .AsNotTracking() on everything threw some exception about not allowing non-tracked entities to have navigation properties (I don't recall the exact details).
Use AutoMapper to make copies of the object graph, only including related objects I specify. This approach is basically working, but in the process of (I believe) performing the auto-mapping, all of the navigation properties are accessed, causing EF to query and resolve them. In one case this leads to almost 300,000 database calls during a single request to the web service.
What I am Looking For
In short, has anyone had to tackle this problem before and come up with a working and performant solution?
Lacking that, any pointers for at least where to look for how to handle this would be greatly appreciated.
Additional Note: It occurred to me as I wrote this that I could possibly combine the second and third items above. In other words, set/clear nav properties, then automap the graph to new objects, then detach everything so it won't get saved (or perhaps wrap it in a transaction and roll it back at the end). However, if there is a more elegant solution I would rather use that.
Thanks,
Dave
It is true that doing what you are asking for is very difficult and it's an architectural trap I see a lot of projects get stuck in.
Even if this problem were solveable, you'd basically end up just having a data layer which just wraps the database and destroys performance because you can't leverage SQL properly.
Instead, consider building your data access service in such a way that it returns meaningful objects containing meaningful data; that is, only the data required to perform a specific task outlined in the requirements documentation. It is true that an post is related to an account, which has many achievements, etc, etc. But usually all I want is the text and the name of the poster. And I don't want it for one post. I want it for each post in a page. Instead, write data services and methods which do things which are relevant to your application.
To clarify, it's the difference between returning a Page object containing a list of Posts which contain only a poster name and message and returning entire EF objects containing large amounts of irrelevant data such as IDs, auditing data such as creation time.
Consider the Twitter API. If it were implemented as above, performance would be abysmal with the amount of traffic Twitter gets. And most of the information returned (costing CPU time, disk activity, DB connections as they're held open longer, network bandwidth) would be completely irrelevant to what developers want to do.
Instead, the API exposes what would be useful to a developer looking to make a Twitter app. Get me the posts by this user. Get me the bio for this user. This is probably implemented as very nicely tuned SQL queries for someone as big as Twitter, but for a smaller client, EF is great as long as you don't attempt to defeat its performance features.
This additionally makes testing much easier as the smaller, more relevant data objects are far easier to mock.
For three tier applications, especially if you are going to expose your entities "raw" in services, I would recommend that you disable Lazy Load and Proxy generation in EF. Your alternative would be to use DTO's instead of entities, so that the web services are returning a model object tailored to the service instead of the entity (as suggested by jameswilddev)
Either way will work, and has a variety of trade-offs.
If you are using EF in a multi-tier environment, I would highly recommend Julia Lerman's DbContext book (I have no affiliation): http://www.amazon.com/Programming-Entity-Framework-Julia-Lerman-ebook/dp/B007ECU7IC
There is a chapter in the book dedicated to working with DbContext in multi-tier environments (you will see the same recommendations about Lazy Load and Proxy). It also talks about how to manage inserts and updates in a multi-tier environment.
i had such a project which was the stressful one .... and also i needed to load large amount of data and process them from different angles and pass it to complex dashboard for charts and tables.
my optimization was :
1-instead of using ef to load data i called old-school stored procedure (and for more optimization grouping stuff to reduce table as much as possible for charts. eg query returns a table that multiple charts datasets can be extracted from it)
2-more important ,instead of Newtonsoft's JSON i used fastJSON which performance was mentionable( it is really fast but not compatible with complex object. simple example may be view models that have list of models inside and may so on and on or )
better to read pros and cons of fastJSON before
https://www.codeproject.com/Articles/159450/fastJSON
3-in relational database design who is The prime suspect of this problem it might be good to create those tables which have raw data to process in (most probably for analytics) denormalized schema which save performance on querying data.
also be ware of using model class from EF designer from database for reading or selecting data especially when u want serialize it(some times i think separating same schema model to two section of identical classes/models for writing and reading data in such a way that the write models has benefit of virtual collections came from foreign key and read models ignore it...i am not sure for this).
NOTE: in case of very very huge data its better go deeper and set up in-memory table OLTP for the certain table contains facts or raw data how ever in that case your table acts as none relational table like noSQL.
NOTE: for example in mssql you can use benefits of sqlCLR which let you write scripts in c#,vb..etc and call them by t-sql in other words handle data processing from database level.
4-for interactive view which needs load data i think its better to consider which information might be processed in server side and which ones can be handled by client side(some times its better to query data from client-side ... how ever you should consider that those data in client side can be accessed by user) how ever it is situation-wise.
5-in case of large raw data table in view using datatables.min.js is a good idea and also every one suggest using serverside-paging on tables.
6- in case of importing and exporting data from big files oledb is a best choice i think.
how ever still i doubt them to be exact solutions. if any body have practical solutions please mention it ;) .
I have fiddled with a similar problem using EF model first, and found the following solution satisfying for "One to Many" relations:
Include "Foreign key properties" in the sub-entities and use this for later look-up.
Define the get/set modifiers of any "Navigation Properties" (sub-collections) in your EF entity to private.
This will give you an object not exposing the sub-collections, and you will only get the main properties serialized. This workaround will require some restructuring of your LINQ queries, asking directly from your table of SubItems with the foreign key property as your filtering option like this:
var myFitnessClubs = context.FitnessClubs
?.Where(f => f.FitnessClubChainID == myFitnessClubChain.ID);
Note 1:
You may off-cause choose to implement this solution partly, hence only affecting the sub-collections that you strongly do not want to serialize.
Note 2:
For "Many to Many" relations, at least one of the entities needs to have a public representation of the collection. Since the relation cannot be retrieved using a single ID property.
I have a question that is more related to how ASP.NET works and will try my best to explain so here goes.
I have a four tiered architecture in my app
Web (ASP.NET Web Application)
Business (Class Library)
Generic CRUD layer ** (Class Library)
Data (Class Library)
This generic CRUD layer of mine uses reflection to set and read properties of an object, however I have read that using PropertyInfo is fairly expensive and thus want to cache these items.
Here is the scenario:
Two people access the site, lets call them Fred and Jim. Fred creates a Customer which in turn called the Generic CRUD layer and caches the property info of the Customer class within System.RuntimeCache. Jim, then seconds later also creates a Customer.
My question is will the two requests from both Fred and Jim cause the obtaining of propery info to be triggered twice? Or will ASP.NET retrieve it from cache the second time, i.e. Jim's request is quicker as property info is obtained via the cache?
My thinking is that because my CRUD is a class library and not having access to System.Web.Cache, the property info won't be cached across all sessions / users?
No, it will issue new queries for each request (unless you've coded otherwise).
There are multiple layers of caching that could happen in ASP.Net application (browser, proxies, server side response caching, intermediate objects caching, DA layer caching) which all can be configured/used. But nothing is ever cached in ASP.Net (or in any application) unless one specifically wrote code/rules/configuration to do so.
As Alexei Levenkov points out, you have to configure caching to happen explicitly: you're not going to get automatic caching of specific property values.
But let me point out that while using PropertyInfo is expensive compared to writing code to directly access properties, it pales in comparison to the cost of a database round-trip or the latency between your server and your end user. Tools like Entity Framework, WebForms, and MVC all make extensive use of reflection because the performance cost it incurs is totally worth the reduced maintenance costs.
Avoid premature optimization.
Ok guys, another my question is seems to be very widely asked and generic. For instance, I have some accounts table in my db, let say it would be accounts table. On client (desktop winforms app) I have appropriate functionality to add new account. Let say in UI it's a couple of textboxes and one button.
Another one requirement is account uniqueness. So I can't add two same accounts. My question is should I check this account existence on client (making some query and looking at result) or make a stored procedure for adding new account and check account existence there. As it for me, it's better to make just a stored proc, there I can make any needed checks and after all checks add new account. But there is pros and cons of that way. For example, it will be very difficult to manage languagw of messages that stored proc should produce.
POST EDIT
I already have any database constraints, etc. The issue is how to process situation there user is being add an existence account.
POST EDIT 2
The account uniqueness is exposed as just a simple tiny example of business logic. My question is more abour handling complicated business logic on that accounts domain.
So, how can I manage this misunderstanding?
I belive that my question is basic and has proven solution. My tools are C#, .NET Framework 2.0. Thanks in advance, guys!
If the application is to be multi-user ( i.e. not just a single desktop app with a single user, but a centralised DB with the app acting as clients maybe on many workstations), then it is not safe to rely on the client (app) to check for such as uniqueness, existance, free numbers etc as there is a distinct possibility of change happening between calls (unless read locking is used, but this often become more of an issue than a help!).
There is the ability of course to precheck and then recheck (pre at app level, re at DB), but of course this would give extra DB traffic, so depends on whether it is a problem for you.
When I write SPROCs that will return to an app, I always use the same framework - I include parameters for a return code and message and always populate them. Then I can use standard routines to call them and even add in the parameters automatically. I can then either display the message directly on failure, or use the return code to localize it as required (or automate a response). I know some DBs (like SQL Svr) will return Return_Code parameters, but I impliment my own so I can leave inbuilt ones for serious system based errors and unexpected failures. Also allows me to have my own numbering systems for return codes (i.e. grouping them to match Enums in the code and/or grouping by severity)
On web apps I have also used a different concept at times. For example, sometimes a request is made for a new account but multiple pages are required (profile for example). Here I often use a header table that generates a hidden user ID against the requested unique username, a timestamp and someway of recognising them (IP Address etc). If after x hours it is not used, the header table deletes the row freeing up the number (depending on DB the number may never become useable again - this doesn;t really matter as it is just used to keep the user data unique until application is submitted) and the username. If completed correctly, then the records are simply copied across to the proper active tables.
//Edit - To Add:
Good point. But account uniqueness is just a very tiny simple sample.
What about more complex requirements for accounts in business logic?
For example, if I implement in just in client code (in winforms app) I
will go ok, but if I want another (say console version of my app or a
website) kind of my app work with this accounts I should do all this
logic again in new app! So, I'm looking some method to hold data right
from two sides (server db site and client side). – kseen yesterday
If the requirement is ever for mutiuse, then it is best to separate it. Putting it into a separate Class Library Project allows the DLL to be used by your WinForm, Console program, Service, etc. Although I would still prefer rock-face validation (DB level) as it is closest point in time to any action and least likely to be gazzumped.
The usual way is to separate into three projects. A display layer [DL] (your winform project/console/Service/etc) and Business Application Layer [BAL] (which holds all the business rules and calls to the DAL - it knows nothing about the diplay medium nor about the database thechnology) and finally the Data Access Layer [DAL] (this has all the database calls - this can be very basic with a method for insert/update/select/delete at SQL and SPROC level and maybe some classes for passing data back and forth). The DL references only the BAL which references the DAL. The DAL can be swapped for each technology (say change from SQL Server to MySQL) without affecting the rest of the application and business rules can be changed and set in the BAL with no affect to the DAL (DL may be affected if new methods are added or display requirement change due to data change etc). This framework can then be used again and again across all your apps and is easy to make quite drastic changes to (like DB topology).
This type of logic is usually kept in code for easier maintenance (which includes testing). However, if this is just a personal throwaway application, do what is most simple for you. If it's something that is going to grow, it's better to put things practices in place now, to ease maintenance/change later.
I'd have a AccountsRepository class (for example) with a AddAcount method that did the insert/called the stored procedure. Using database constraints (as HaLaBi mentioned), it would fail on trying to insert a duplicate. You would then determine how to handle this issue (passing a message back to the ui that it couldn't add) in the code. This would allow you to put tests around all of this. The only change you made in the db is to add the constraint.
Just my 2 cents on a Thrusday morning (before my cup of green tea). :)
i think the answer - like many - is 'it depends'
for sure it is a good thing to push logic as deeply as possible towards the database. This prevent bad data no matter how the user tries to get it in there.
this, in simple terms, results in applications that TRY - FAIL - RECOVER when attempting an invalid transaction. you need to check each call(stored proc, or triggered insert etc) and IF something bad happens, recover from that condition. Usually something like tell the user an issue occurred, reset the form or something, and let them try again.
i think at a minimum, this needs to happen.
but, in addition, to make a really nice experience for the user, the app should also preemptively check on certain data conditions ahead of time, and simply prevent the user from making bad inserts in the first place.
this is of course harder, and sometimes means double coding of business rules (one in the app, and one in the DB constraints) but it can make for a dramatically better user experience.
The solution is more of being methodical than technical:
Implement - "Defensive Programming" & "Design by Contract"
If the chances of a business-rule being changed over time is very less, then apply the constraint at database-level
Create a "validation or rules & aggregation layer (or class)" that will manage such conditions/constraints for entity and/or specific property
A much smarter way to do this would be to make a user-control for the entity and/or specific property (in your case the "Account-Code"), which would internally use the "validation or rules & aggregation layer (or class)"
This will allow you to ensure a "systematic-way-of-development" or a more "scalable & maintainable" application-architecture
If your application is a website then along with placing the validation on the client-side it is always better to have validation even in the business-layer or C# code as well
When ever a validation would fail you could implement & use a "custom-error-message" library, to ensure message-content is standard across the application
If the errors are raised from database itself (i.e., from stored-procedures), you could use the same "custom-error-message" class for converting the SQL Exception to the fixed or standardized message format
I know that this is all a bit too much, but is will always good for future.
Hope this helps.
As you should not depend on a specific Storage Provider (DB [mysql, mssql, ...], flat file, xml, binary, cloud, ...) in a professional project all constraint should be checked in the business logic (model).
The model shouldn't have to know anything about the storage provider.
Uncle Bob said something about architecture and databases: http://blog.8thlight.com/uncle-bob/2011/11/22/Clean-Architecture.html
I have an asp classic and asp.net application. I want to share the session from asp classic to asp.net back and forth. I made a solution to share the session by iterating the session names and put it in the hidden objects. This works fine , but I don't know if it is a better approach to share the session. iread some articles some of them using xml and pass it through form posting, some using cookies, and the other are using database.From this approach, what is the best to impleemnt with, or is there any other best solution?
Unfortunately your description of what you've already done is not very clear:
I made a solution to share the session by iterating the session names and put it in the hidden objects.
So, I'm going to go back one step and approach it from there. Before that, however, I should say that I don't think integrating two systems on this level is a good idea at all; you are trying to make two systems work together by replacing some of the piping that they run on, in such a way that they don't really notice. This might open you up for errors later on, when something changes and the relevant change is not made in the other system. I also would advise against the heavy use of sessions in general, although many ASP systems did abuse it.
Basically, both the classic ASP and ASP.Net's session works similarly, they use a session identifier (either stored automatically as a cookie - usually the default, or passed via the URL), and allow you to store key / value pairs of objects against this session identifier.
If you want to share the objects in session, you can either pass the information directly between the two applications when they interact (this would be the "form posting" approach), or store it in an external place, such as the database. I have some issues with the form posting approach, namely:
It is passing data which should be strictly on the server to the browser and back again, which has both security and bandwidth implications (you would probably want to at least encrypt the data);
It assumes that you have a clear handover or boundary between the two applications. This may or may not be the case.
I don't really see too many benefits to this approach, apart from the fact that it's probably the simplest, as long as you have limited interactions (say, a few fields, and only one or two pages that interact). The same argument goes for any other approach that passes all of the session info to the browser and back to the server (e.g. storing everything in cookies).
Other approaches will assume that you have a place to store your session, let's assume a database for now. With a database, we have a single place where the data exists, and each application can simply read and write to this single place. If you have a large number of different interactions, and / or the systems don't have a clear handover to each other, this is clearly an easier approach.
What remains is a question of how to encode the different types of data and how to store it. This is essentially a question of encoding, and you would need to be fairly clear on the different data types being used on each side (you should make sure that the ASP.Net code uses types that are at least logically equivalent to the ASP data types).
To go about implementing this, I would create a database implementation which supports how I want to store my session data. It could be a simple key / value pair table, along with data type information, or it could be more normalised than that, it's your decision. After this, I would create a new implementation of Session for each application, which makes use of this database. All that is left is to replace all references to the standard Session objects with the new Session that you've created, and everything should work. This is probably similar to what you've done, but I am specifically mentioning the use of a database as a data store.
Performance issues and the cleaning of old sessions should be relatively easy to work out, although this implementation will definitely be slower than an in-memory implementation.
I know its not exactly an answer to your question but I have worked on and integrated my self a few .net and classic asp applications, just wanted to give you some comments from my experience:
it can be confusing if its not well documented..
on some of the systems that were not properly documented when we re-deployed we missed out a .net app that handled part of the functionality
the most solid and best to work with (in my experience) were the ones where we picked one application to be the "core" with other parts making calls to this core to get work done.
for example creating .net applications (desktop, web, handheld) that called classic asp web pages that were effectively providing a simple xml web service - seeing as the classic asp apps tended to be the legacy "core" part of the system. (xml web services are fun to write in classic asp - seriously they are fast and easy to code - no type casting issues ;-) )
I don't know what you are doing here specifically but i found that this approach made the structure of the code more intuitive and actually simplified many odd associated issues beyond sessions e.g. on one app had an access DB used as part of it and only having classic asp access it removed concurrency issues.
May not be of any use but its based experience working with around 10 operational management systems that integrated .net and classic asp in various ways.
If I wanted to access a database in Delphi, I could add a datamodule to a project, configure it from my mainform and then access it anywhere in the application; a reference would be stored in a global variable.
I know that in C# and other more modern OO languages global variables are frowned upon. So how can I access my database from where I need it? The biggest problem I have is the configuration: location, user, password, etc. are unknown at design time.
I now have a db-class and make a new instance when I need it, but then I would have to store those settings in some globally accessible thing, and I have simply moved the problem.
What's the standard solution?
Thanks, regards, Miel.
I always use the singleton pattern. As for configuration, look at the System.Configuration.ConfigurationManager class which allows you to read settings from your project's app.config/web.config file.
It's a bit tricky to define the absolute best practice for database access in OOP.
You've hit the nail on the head that there are a lot of factors to consider:
how are configuration parameters handled?
is the app multi-threaded? do you need database connection pools?
do you need database portability (ie: do you use different DBs in dev versus production? are you concerned about vendor lock-in with one DB? Are you distributing the app the other users who may be using a different db?)
are you concerned with securing your SQL statements, or centrally enforcing other access permissions?
is there common logic involved when performing some inserts and updates that you'd rather not duplicate everywhere a particular table is touched?
Because of this, many OOP folks gravitate to an ORM framework for anything but the simplest cases. The general idea is that your application logic shouldn't need to talk to the database directly at any point: isolate your business code from the actual persistence mechanism for as long as possible.
Instead, try to design your application so that your business logic talks to a model layer. In other words, have model objects in the system that encapsulate and describe your business data. These model objects then expose methods for obtaining and saving their state into the database, but your logic doesn't need to care about that.
For example, say you have a concept called "Person" in your system. You'd probably model this as a class with some properties. In pseudo-code:
Person:
- first_name
- last_name
Your actual code in the system is then only concerned with instantiating and using Person objects, not with obtaining DB handles or writing SQL:
p = Person.get(first_name='Joe')
p.last_name = 'Bloggs'
p.save()
In an object-oriented world, you'll find that your business logic code becomes cleaner (and shorter!), easier to maintain, and much more testable.
Of course, you're right in that this means you need to now go off and build a database back-end that translates that Person class to one or more tables in your relational database. This is where using an ORM framework comes in handy. In Python, people use Django and SQLAlchemy. I'll let others indicate what folks use in C# (I'm not a C# developer, but you did tag your question OOP, so I'm going for the generic answer here, rather than C# specific).
The point, though, is that the ORM framework puts all the DB access in a single set of classes in the code, so that the DB access, configuration and pools are handled in one place... no need to instantiate them all over the application. What you use "where you need it" is the model object.
Of course, if your app is very simple and you want just a raw DB handle, then I do recommend the dependency injection approach others have listed.
Hope that helps.
It seems to me that you need to create an appropriate object (containing the connection or similar), and pass that instance to each object requiring access (see dependency injection)
This is different from using singletons. By using this mechanism, it'll free you from the dependency on one object and (perhaps a more compelling reason in this instance) allow you to perform testing by injecting mock objects or similar in place of the originally-injected database accessor object. I would definitely shy away from the singleton mechanism in this scenario.
I actually use a repository class that takes in the db information in its constructor and have the classes that need it get it passed in. I actually use an Inversion of Control (IOC) tool to inject that values in.
You could store the user information in a flat file somewhere, then read / write to it from your db-class
This way you won't duplicate the settings in your code, but the user can still modify the settings.
SubSonic is the "Swiss Army knife" for object relational mapping, and offers the ability to execute stored procedures and return results to List. You can have it up and running within a half hour.