Develop Reference

Using a Data Access Layer in an OOP C# application using SQL

I have come from an environment where I was taught to use objects and employ OOP techniques where possible and I guess it has guided me down a particular road and influenced my product designs for quite some time.
Typically my data access layer will have a number of classes which map onto database tables, so if I need to display a list of Companies, I will have a 'Company' object and a database table called 'company'. The company object knows how to instantiate itself from a DataRow from the database read using a 'SELECT * FROM company WHERE id = x' type query. So when-ever I display a companies list I will populate a list of company objects and display them. If I need to display attributes of the company I already have the data loaded.
It has been mentioned that 'select *' is frowned on and my object approach can be inefficient, but I am having issues identifying another way of working with a database table and objects which would work if you only read specific fields - the object just wouldn't be populated.
Yes I could change the list to directly query just the required fields from the database and display them but that means my UI code would need to be more closely linked to the data access code - I personally like the degree of separation of having the object separating the layers.
Always willing to learn though - I work on my own so don't always get up to speed with the latest technologies or methodologies so any comments welcome.

I don't think I can show you a definitive solution to that, but I'll try pointing you on the right direction (since it's more of a theoretical question).
Depending on the design pattern you follow on your application, you could decouple the data access layer from the UI and still follow this rule of not fetching all columns when they are not necessary. I mean, chosing the right design pattern for an application can bring you this sort of easyness.
For example, maybe you could interpret the less detailed version of an object as an object itself (which honestly I don't think it would be a good approach).
Also, I'll comment that the very popular rails ORM ActiveRecord only fetches from DB when the data is used. Maybe you could use a similar logic to track not only when but which columns will be used so you could limit the query.

Serializing complex EF model over JSON

I have done a lot of searching and experimenting and have been unable to find a workable resolution to this problem.
Environment/Tools
Visual Studio 2013
C#
Three tier web application:
Database tier: SQL Server 2012
Middle tier: Entity Framework 6.* using Database First, Web API 2.*
Presentation tier: MVC 5 w/Razor, Bootstrap, jQuery, etc.
Background
I am building a web application for a client that requires a strict three-tier architecture. Specifically, the presentation layer must perform all data access through a web service. The presentation layer cannot access a database directly. The application allows a small group of paid staff members to manage people, waiting lists, and the resources they are waiting for. Based on the requirements the data model/database design is entirely centered around the people (User table).
Problem
When the presentation layer requests something, say a Resource, it is related to at least one User, which in turn is related to some other table, say Roles, which are related to many more Users, which are related to many more Roles and other things. The point being that, when I query for just about anything EF wants to bring in almost the entire database.
Normally this would be okay because of EF's default lazy-load behavior, but when serializing just about any object to JSON for returning to the presentation layer, the Newtonsoft.Json serializer hangs for a long time then blows a stack error.
What I Have Tried
Here is what I have attempted so far:
Set Newtonsoft's JSON serialier ReferenceLoopHandling setting to Ignore. No luck. This is not cyclic graph issue, it is just the sheer volume of data that gets brought in (there are over 20,000 Users).
Clear/reset unneeded collections and set reference properties to null. This showed some promise, but I could not get around Entity Framework's desire to track everything.
Just setting nav properties to null/clear causes those changes to be saved back to the database on the next .SaveChanges() (NOTE: This is an assumption here, but seemed pretty sound. If anyone knows different, please speak up).
Detaching the entities causes EF to automatically clear ALL collections and set ALL reference properties to null, whether I wanted it to or not.
Using .AsNotTracking() on everything threw some exception about not allowing non-tracked entities to have navigation properties (I don't recall the exact details).
Use AutoMapper to make copies of the object graph, only including related objects I specify. This approach is basically working, but in the process of (I believe) performing the auto-mapping, all of the navigation properties are accessed, causing EF to query and resolve them. In one case this leads to almost 300,000 database calls during a single request to the web service.
What I am Looking For
In short, has anyone had to tackle this problem before and come up with a working and performant solution?
Lacking that, any pointers for at least where to look for how to handle this would be greatly appreciated.
Additional Note: It occurred to me as I wrote this that I could possibly combine the second and third items above. In other words, set/clear nav properties, then automap the graph to new objects, then detach everything so it won't get saved (or perhaps wrap it in a transaction and roll it back at the end). However, if there is a more elegant solution I would rather use that.
Thanks,
Dave

It is true that doing what you are asking for is very difficult and it's an architectural trap I see a lot of projects get stuck in.
Even if this problem were solveable, you'd basically end up just having a data layer which just wraps the database and destroys performance because you can't leverage SQL properly.
Instead, consider building your data access service in such a way that it returns meaningful objects containing meaningful data; that is, only the data required to perform a specific task outlined in the requirements documentation. It is true that an post is related to an account, which has many achievements, etc, etc. But usually all I want is the text and the name of the poster. And I don't want it for one post. I want it for each post in a page. Instead, write data services and methods which do things which are relevant to your application.
To clarify, it's the difference between returning a Page object containing a list of Posts which contain only a poster name and message and returning entire EF objects containing large amounts of irrelevant data such as IDs, auditing data such as creation time.
Consider the Twitter API. If it were implemented as above, performance would be abysmal with the amount of traffic Twitter gets. And most of the information returned (costing CPU time, disk activity, DB connections as they're held open longer, network bandwidth) would be completely irrelevant to what developers want to do.
Instead, the API exposes what would be useful to a developer looking to make a Twitter app. Get me the posts by this user. Get me the bio for this user. This is probably implemented as very nicely tuned SQL queries for someone as big as Twitter, but for a smaller client, EF is great as long as you don't attempt to defeat its performance features.
This additionally makes testing much easier as the smaller, more relevant data objects are far easier to mock.

For three tier applications, especially if you are going to expose your entities "raw" in services, I would recommend that you disable Lazy Load and Proxy generation in EF. Your alternative would be to use DTO's instead of entities, so that the web services are returning a model object tailored to the service instead of the entity (as suggested by jameswilddev)
Either way will work, and has a variety of trade-offs.
If you are using EF in a multi-tier environment, I would highly recommend Julia Lerman's DbContext book (I have no affiliation): http://www.amazon.com/Programming-Entity-Framework-Julia-Lerman-ebook/dp/B007ECU7IC
There is a chapter in the book dedicated to working with DbContext in multi-tier environments (you will see the same recommendations about Lazy Load and Proxy). It also talks about how to manage inserts and updates in a multi-tier environment.

i had such a project which was the stressful one .... and also i needed to load large amount of data and process them from different angles and pass it to complex dashboard for charts and tables.
my optimization was :
1-instead of using ef to load data i called old-school stored procedure (and for more optimization grouping stuff to reduce table as much as possible for charts. eg query returns a table that multiple charts datasets can be extracted from it)
2-more important ,instead of Newtonsoft's JSON i used fastJSON which performance was mentionable( it is really fast but not compatible with complex object. simple example may be view models that have list of models inside and may so on and on or )
better to read pros and cons of fastJSON before
https://www.codeproject.com/Articles/159450/fastJSON
3-in relational database design who is The prime suspect of this problem it might be good to create those tables which have raw data to process in (most probably for analytics) denormalized schema which save performance on querying data.
also be ware of using model class from EF designer from database for reading or selecting data especially when u want serialize it(some times i think separating same schema model to two section of identical classes/models for writing and reading data in such a way that the write models has benefit of virtual collections came from foreign key and read models ignore it...i am not sure for this).
NOTE: in case of very very huge data its better go deeper and set up in-memory table OLTP for the certain table contains facts or raw data how ever in that case your table acts as none relational table like noSQL.
NOTE: for example in mssql you can use benefits of sqlCLR which let you write scripts in c#,vb..etc and call them by t-sql in other words handle data processing from database level.
4-for interactive view which needs load data i think its better to consider which information might be processed in server side and which ones can be handled by client side(some times its better to query data from client-side ... how ever you should consider that those data in client side can be accessed by user) how ever it is situation-wise.
5-in case of large raw data table in view using datatables.min.js is a good idea and also every one suggest using serverside-paging on tables.
6- in case of importing and exporting data from big files oledb is a best choice i think.
how ever still i doubt them to be exact solutions. if any body have practical solutions please mention it ;) .

I have fiddled with a similar problem using EF model first, and found the following solution satisfying for "One to Many" relations:
Include "Foreign key properties" in the sub-entities and use this for later look-up.
Define the get/set modifiers of any "Navigation Properties" (sub-collections) in your EF entity to private.
This will give you an object not exposing the sub-collections, and you will only get the main properties serialized. This workaround will require some restructuring of your LINQ queries, asking directly from your table of SubItems with the foreign key property as your filtering option like this:
var myFitnessClubs = context.FitnessClubs
?.Where(f => f.FitnessClubChainID == myFitnessClubChain.ID);
Note 1:
You may off-cause choose to implement this solution partly, hence only affecting the sub-collections that you strongly do not want to serialize.
Note 2:
For "Many to Many" relations, at least one of the entities needs to have a public representation of the collection. Since the relation cannot be retrieved using a single ID property.

What is a better practice ? Working with Dataset or Database

I have been developing many application and have been into confusion about using dataset.
Till date i dont use dataset and works into my application directly from my database using queries and procedures that runs on Database Engine.
But I would like to know, what is the good practice
Using Dataset ?
or
Working direclty on Database.
Plz try to give me certain cases also when to use dataset along with operation (Insert/Update)
can we set read/write lock on dataset with respect to our database

You should either embrace stored procedures, or make your database dumb. That means that you have no logic whatsoever in your db, only CRUD operations. If you go with the dumb database model, Datasets are bad. You are better off working with real objects so you can add business logic to them. This approach is more complicated than just operating directly on your database with stored procs, but you can manage complexity better as your system grows. If you have large system with lots of little rules, stored procedures become very difficult to manage.

In ye olde times before MVC was a mere twinkle in Haack's eye, it was jolly handy to have DataSet handle sorting, multiple relations and caching and whatnot.
Us real developers didn't care about such trivia as locks on the database. No, we had conflict resolution strategies that generally just stamped all over the most recent edits. User friendliness? < Pshaw >.
But in these days of decent generic collections, a plethora of ORMs and an awareness of separation of concerns they really don't have much place any more. It would be fair to say that whenever I've seen a DataSet recently I've replaced it. And not missed it.

As a rule of thumb, I would put logic that refers to data consistency, integrity etc. as close to that data as possible - i.e. in the database. Also, if I am having to fetch my data in a way that is interdependent (i.e. fetch from tables A, B and C where the relationship between A, B and C's contribution is known at request time), then it makes sense to save on callout overhead and do it one go, via a database object such as a function, procedure (as already pointed out by OMGPonies). For logic that is a level or two removed, it makes sense to have it where dealing with it "procedurally" is a bit more intuitive, such as in a dataset. Having said all that, rules of thumb are sometimes what their acronym infers...ROT!
In past .Net projects I've often done data imports/transformations (e.g. for bank transaction data files) in the database (one callout, all logic is encapsulated in in procedure and is transaction protected), but have "parsed" items from that same data in a second stage, in my .net code using datatables and the like (although these days I would most likely skip the dataset stage and work on them from a higher lever of abstraction, using class objects).

I have seen datasets used in one application very well, but that is in 7 years development on quite a few different applications (at least double figures).
There are so many best practices around these days that point twords developing with Objects rather than datasets for enterprise development. Objects along with an ORM like NHibernate or Entity Framework can be very powerfull and take a lot of the grunt work out of creating CRUD stored procedures. This is the way I favour developing applications as I can seperate business logic nicely this way in a domain layer.
That is not to say that datasets don't have their place, I am sure in certain circumstances they may be a better fit than Objects but for me I would need to be very sure of this before going with them.

I have also been wondering this when I never needed DataSets in my source code for months.
Actually, if your objects are O/R-mapped, and use serialization and generics, you would never need DataSets.
But DataSet has a great use in generating reports.
This is because, reports have no specific structure that can be or should be O/R-mapped.
I only use DataSets in tandem with reporting tools.

Caching calculated values (Sums/Totals) in the Database

Consider the Following object model (->> indicates collection):
Customer->Orders
Orders->>OrderLineItems->Product{Price}
The app is focused on processing orders, so most of the time tables showing all the orders that match certain criteria are used in the UI. 99% of the time i am only interested in displaying the Sum of LineTotals, not the individual LineTotals.
Thinking about it further, there also might be multiple payments (wire transfers,cheque, credit card etc.) associated with each order, again, im only interested in the sum of money that i received.
When querying the database for an order, I dont want to select all orders and then, for each order, its payments and LineItems.
My idea was to store the associate each order with a "status" object, caching all the sums and status of an order, improving query performance by orders of magnitude and also supporting query scenarios for unpaid orders, paid orders, orders due etc.
This prevents domain logic (e.g. when an order is considered to be paid) from leaking into database queries. However, it puts the responsibility for keeping the sums up to date. The system usually has well defined points where that needs to happen, e.g. entering or integrating payments, creating/modifying an order.
So far i have used Observable Collections, that trigger recalculations of Status when items are added or removed, or certain properties on the items are updated. I ask myself where the logic for all that should be put from a ddd perspective. It seems strange to me to force all the event wiring and calculation logic in the aggregate root.

You need to express the intent of a request in an intention-revealing interface, so that your repositories can understand what exactly you want to do and react accordingly. In this case the interface reveals intent, not to other developers, but to other code. So if you want a status or total, create an interface that reveals this intent and request an object of that type from your repository. The repository can then create and return a domain object which encapsulates doing exactly the work required to calculate the total and no more than that.
In addition, your DAL can intelligently choose which fetching strategy to apply from the interface you request, i.e. lazy loading for situations where you don't need to access child objects and eager loading where you do.
Udi Dahan has some great blog posts about this. He has written and talked on applying intention-revealing interfaces to this problem, which he calls making roles explicit.

I highly recommend looking into OR (object relational) mappers that support LINQ. To name the two primary ones, LINQ to SQL and Entity Framework, both from Microsoft. I believe LLBLGen also supports LINQ now, and nHibernate has a few half-baked LINQ solutions you could try. My prime recommendation is Entity Framework v4.0, which is available through .NET 4.0 betas or the Visual Studio 2010 Beta.
With a LINQ enabled OR mapper, you can easily query for the aggregate information you need dynamically, real-time, using only your domain model. There is no need for business logic to leak into your data layer, because you generally will not use stored procedures. OR mappers generate parameterized SQL for you on the fly. LINQ combined with OR mappers is an extremely powerful tool that allows you to not only query for and retrieve entities and entity graphs, but also query for data projections on your domain model...allowing the retrieval of custom data sets, aggregations, etc. via a single conceptual model.

"It seems strange to me to force all the event wiring and calculation logic in the aggregate root."
That is usually a call for a «Service».

Should I use one LINQ DataContext or many?

For a large project, would it make sense to have one datacontext which maps out your database, which you can interact with from your classes?
Or would it make more sense to split this up into small datacontexts which are focused on the specific tasks within the database that will be required.
I'm curious as to the performance. It's my understanding that the datacontext itself is a very lightweight object, which only initializes its internal collections as they're required etc. Therfore dealing with a datacontext with many defintions but only two tables of data should be as fast as dealing with a special datacontext with only those two tables in it.
I also think you would benefit at the JIT time, as the first class to do data access will compile your dc which is now available to all classes.

I'm assuming you're asking about design versus runtime pattern. I general I'd say no:
While it may be possible to partition your database into multiple data contexts, then that would be desirable if and only if there was zero overlap between the two contexts.
Overlap is Bad
e.g. you have a WebsiteContext and an AdminContext. The WebsiteContext is for displaying Product and fulfilling Orders. A WebsiteUser is attached to an Order. The AdminContext is for your Staff members to process refunds for cancelled Orders, which also reference WebsiteUser. The AdminContext also needs to reset passwords and update other details for the WebsiteUser.
You're thinking of doing this because you don't want the website to process or even know about Returns
WebsiteContext
Product -- Order -- WebsiteUser
AdminContext
Staff -- Returns -- Order -- WebsiteUser
In the above, we can see we're duplicating many objects in the different Data Contexts. This smells bad, and it really indicates that artificially dividing the database into different data contexts is a the wrong decision. Do you have >2 databases after all, or just the one? The duplication violates the DRY principle (Dont Repeat Yourself) because WebsiteContext.WebsiteUser is not the same as AdminContext.WebsiteUser and in all likelihood the code is going to be messy when something needs to care which one they're referencing.
The Linq Data Context is just an OR mapper, and needs to be treated as a fancy black box that makes writing some of the data access code easier. Some linq demos make it look like you no longer need the other layers, but a program of any complexity still benefits from a layered design.
You're probably better off treating the Linq objects as just objects for easily transferring data and create a Domain layer that hides them as an implementation detail. Have a read of DDD - Domain Driven Design.
On their own, just using the Linq objects from the UI most resembles the Transaction Script pattern. In such a case you'd still benefit from having a logic layer that takes care of the details.
While you may not want the responsibility of a context to be so broad, the data context is just a representation of the database. It's not a security mechanism, and it can't prevent you from corrupting the data.

From a "repository pattern" angle, there is something to be said for having separate aggregates and minimizing the navigation properties between aggregates. I'm not 100% sure what that something is though... at the moment I'm considering using a moderately-sized dbml, with multiple repositories using it (the UI doesn't use the datacontexts directly - only the repository classes), with the navigation properties marked internal so the DAL can use them... Maybe...