Develop Reference

Dapper and n-tier application (navigation properties)

I'm struggling to get my head around how I should implement dapper in my application. I have a n-tier mvc application and have some experience with EF. Even that I think EF is good, I have not passed the learning curve to make it flow easy and not struggle with performance. In the new project we decided to give dapper a go, mostly to get control over the sql and hopefully get good performance.
Background
I created a layered aplication (core) with these layer
Web - mvc
Service - Business layer to handle the business logic
Data - datalayer to access the ms sql server
I went ahead and started implementing a UnitOfWork and generic Repositories in the datalayer.
A normal structure in the Database would be
Order
ref to User
ref to Address
OrderLine
ref to Product
And in many cases I want to retrieve multiple orders with all lines and products.
So what I did was to have navigation properties on the entity models as you would in EF and populate them with dapper using either multiquery or split the result into different entities and mapping them to the graph.
The problem
The problem I run into is when I do an insert. I have an sqlextension that maps the properties to table columns. But the navigation would also be mapped by default. I realize that I can decorate with attributes and read them on the mapping, but as I google I'm getting aware that maybe I should drop the UnitOfWork pattern and also repository, making the data-layer "super thin" and just expose the connection.
Then the service-layer would call the Dapper with correct sql, kind of what I would do today but with repositories.
I would also drop the navigation properties and fetch each entity on it's own, and combine them in the ViewModel.
My problem with this is if we take the order table above I would have to do something like this to get a full list (normally paged, also I removed the User/address)
var listModel = new OrderListViewModel();
var orders = orderService.GetAll();
foreach(var order in orders) {
var orderModel = new OrderViewModel(); // also map fields
var orderLines = orderService.GetOrderLinesForOrder(order.OrderId);
foreach(var orderline in orderLines) {
var orderLineModel = new OrderLineViewModel(); // also map fields
var product = productService.GetProduct(orderline.ProductId);
orderLineModel.Product = new ProductViewModel(); // also map fields
orderModel.OrderLines(orderLineModel);
}
listModel.Orders.Add(orderModel);
}
This will generate ALOT of queries (almost like EF lazy loading). So I could do a mapping thing
var orders = orderService.GetAll();
var orderLines = orderService.GetOrderLinesForOrders(orders.Select(o => o.OrderId).ToArray() ); // get all orderlines for all orders
var products = productService.GetProductsForOrderLines(orderLines.Select(p => p.OrderLineId).ToArray() ); // get all products for all orderlines
foreach(var order in orders) {
var orderModel = new OrderViewModel(); // also map fields
var orderLines = orderLines.Where( o => o.OrderId == order.OrderId );
foreach(var orderline in orderLines) {
var orderLineModel = new OrderLineViewModel(); // also map fields
var product = products.First(p => p.ProductId == orderline.ProductId);
orderLineModel.Product = new ProductViewModel(); // also map fields
orderModel.OrderLines(orderLineModel);
}
listModel.Orders.Add(orderModel);
}
This will generate alot less sql queries and is optimal in performance I think. I know there can be a problem with more than 2100 (?) parameters, but I think that will not be a problem in my case.
The problem is that many of out tables have different status, and many relations to other tables. I would have to do alot of these queries all the time.
When I first did repository and navigation I would do it like
repo.Get<Order, OrderLines, Product, Order>(sqlThatWouldJoinAllTables);
// split and map the structure into order Entity and just return that
That way I could just call orderService.GetAll() and retrieve a graph of order, orderlines and products.
I don't know which of the solutions is "best practice". I've tried to find a good open source project using layers and dapper to get some real world usage, but without success.
The approach of removing navigation properties also remove some of the purpose of the service layer, since i'm in kind of a way moving some of the business logic to the mvc controller.
I can't find a good practice how I would go forward, please advice.

If the RDBMS you're using supports JSON, I would suggest to wrap everything you need to insert into a JSON and send it to stored procedure with just one call. Same technique can be used to return a graph of related object with just one call. The Unit-Of-Work, a transaction really, will be taken care in the stored procedure itself, which is also the right place where to deal with transactions that operation on data IMHO.
This helps enormously to reduce round-trips at the expense of more CPU used on the database. This is usually not a problem unless you expect a really huge number (= more then several thousands of concurrent queries per second.
I have wrote extensively about this here:
https://medium.com/dapper-net/one-to-many-mapping-with-dapper-55ae6a65cfd4
and more specifically the "Complex Custom Handling" sample shows exactly what I mentioned.

Passing a GetWhere query (Func<entityDTO,bool>) to a data layer method which needs a (Func<entity,bool>) parameter to work

I have the following method in a data access class which uses entity framework:
public static IEnumerable<entityType> GetWhere(Func<entityType, bool> wherePredicate)
{
using (DataEntities db = new DataEntities())
{
var query = (wherePredicate != null)
? db.Set<entityType>().Where(wherePredicate).ToList()
: db.Set<entityType>().ToList();
return query;
}
}
This works fine when I use the entities across all layers... however I am trying to move to using a DTO class and I would like to do something like the following:
public static IEnumerable<EntityTypeDTO> GetWhere(Func<EntityTypeDTO, bool> wherePredicate)
{
//call a method here which will convert Func<EntityTypeDTO,bool> to
// Func<EntityType,bool>
using (DataEntities db = new DataEntities())
{
var query = new List<EntityType>();
if (wherePredicate == null)
{
query = db.Set<EntityType>().ToList();
}
else
{
query = (wherePredicate != null)
? db.Set<EntityType>().Where(wherePredicate).AsQueryable<EntityType>().ToList()
: db.Set<EntityType>().ToList();
}
List<EntityTypeDTO> result = new List<EntityTypeDTO>();
foreach(EntityType item in query)
{
result.Add(item.ToDTO());
}
return result;
}
}
Essentially I want a method which will convert Func to Func.
I think I have to break down the Func into an expression tree and then rebuild it somehow in the entityType?
I want to do this to allow the Presentation Layer to just pass the Expression queries?
Am I missing something basic or is there an easier design pattern that can pass a query from a DTO to a data access class without knowing the details of the query?
I have tried making the DTO inherit from the entity which doesn't seem to work either?
If there is a better design pattern that I am missing I would love a pointer and I can investigate from there...

Firstly I would suggest that you put a querying layer of your own in front of Entity Framework rather than allowing any arbitrary Func to be passed in because it will be very easy in the future to pass a Func that Entity Framework can not translate into a SQL statement (it can only translate some expressions - the basics are fine but if your expression calls a C# method, for example, then Entity Framework will probably fail).
So your search layer could have classes that you build up as criteria (eg. a "ContainsName" search class or a "ProductHasId" class) that are then translated into expressions in your search layer. This separates your app entirely from the ORM, which means that ORM details (like the entities or like the limitations of what Funcs can and can't be translated) don't leak out. There's lots out there that's been written about this some of arrangement.
One final note, though, if you are working close to the ORM layer, Entity Framework is very clever and you could probably get a long way without trying to translate your Func<dto, bool> to a Func<entity, bool>. For example, in the below code, accessing "context.Products" returns a "DbSet" and calling Select on it returns an IQueryable and calling Where on that also returns an IQueryable. Entity Framework will translate all of that into a single SQL statement so it won't pull all other Products into memory and then filter the ID on that memory set, it will actually perform the filtering in SQL even though the filter is operating on a projected type (which is equivalent to the DTO in your case) and not the Entity Framework entity -
var results = context.Products
.Select(p => new { ID = p.ProductID, Name = p.ProductName })
.Where(p => p.ID < 10)
.ToList();
The SQL executed is:
SELECT
[Extent1].[ProductID] AS [ProductID],
[Extent1].[ProductName] AS [ProductName]
FROM [dbo].[Products] AS [Extent1]
WHERE [Extent1].[ProductID] < 10
So, if you changed your code to get something like..
return context.Products
.Map<Product, ProductDTO()>()
.Where(productDtoWherePredicate)
.ToList();
.. then you might be just fine with the Funcs that you already have. I presume that you already have some sort of mapping functions to get from EF entities to DTOs (but if not then you might want to look into AutoMapper to help you out - which has support for "projections", which are basically IQueryable maps).

I am going to put this up as an answer.Thanks to Dan for the quick answer. Looking at what you are saying I can write a query/filter set of classes. for example, take the following code:
GetProducts().GetProductsInCategory().GetProductsWithinPriceRange(minPrice, maxPrice);
This code would run like so: Get Products would get all products in the table and the remaining functions would filter the results. if all queries run like this it may put a significant load on the Data Access Layer/ DB Server Connections... not sure.
or
An Alternate I will work on also is:
If each function creates a Linq expression, I could combine them like this: How do I combine multiple linq queries into one results set?
this may allow me to do this in a manner where I can return the filtered results set from the database.
Either way I am marking this as answered. I will update when I have more details.

Improving efficiency with Entity Framework

I have been using the Entity Framework with the POCO First approach. I have pretty much followed the pattern described by Steve Sanderson in his book 'Pro ASP.NET MVC 3 Framework', using a DI container and DbContext class to connect to SQL Server.
The underlying tables in SQL server contain very large datasets used by different applications. Because of this I have had to create views for the entities I need in my application:
class RemoteServerContext : DbContext
{
public DbSet<Customer> Customers { get; set; }
public DbSet<Order> Orders { get; set; }
public DbSet<Contact> Contacts { get; set; }
...
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<Customer>().ToTable("vw_Customers");
modelBuilder.Entity<Order>().ToTable("vw_Orders");
...
}
}
and this seems to work fine for most of my needs.
The problem I have is that some of these views have a great deal of data in them so that when I call something like:
var customers = _repository.Customers().Where(c => c.Location == location).Where(...);
it appears to be bringing back the entire data set, which can take some time before the LINQ query reduces the set to those which I need. This seems very inefficient when the criteria is only applicable to a few records and I am getting the entire data set back from SQL server.
I have tried to work around this by using stored procedures, such as
public IEnumerable<Customer> CustomersThatMatchACriteria(string criteria1, string criteria2, ...) //or an object passed in!
{
return Database.SqlQuery<Customer>("Exec pp_GetCustomersForCriteria #crit1 = {0}, #crit2 = {1}...", criteria1, criteria2,...);
}
whilst this is much quicker, the problem here is that it doesn't return a DbSet and so I lose all of the connectivity between my objects, e.g. I can't reference any associated objects such as orders or contacts even if I include their IDs because the return type is a collection of 'Customers' rather than a DbSet of them.
Does anyone have a better way of getting SQL server to do the querying so that I am not passing loads of unused data around?

var customers = _repository.Customers().Where(c => c.Location == location).Where(...
If Customers() returns IQueryable, this statement alone won't actually be 'bringing back' anything at all - calling Where on an IQueryable gives you another IQueryable, and it's not until you do something that causes query execution (such as ToList, or FirstOrDefault) that anything will actually be executed and results returned.
If however this Customers method returns a collection of instantiated objects, then yes, since you are asking for all the objects you're getting them all.
I've never used either code-first or indeed even then repository pattern, so I don't know what to advise, other than staying in the realm of IQueryable for as long as possible, and only executing the query once you've applied all relevant filters.

What I would have done to return just a set of data would have been the following:
var customers = (from x in Repository.Customers where <boolean statement> &&/|| <boolean statement select new {variableName = x.Name , ...).Take(<integer amount for amount of records you need>);
so for instance:
var customers = (from x in _repository.Customers where x.ID == id select new {variableName = x.Name} ).take(1000);
then Iterate through the results to get the data: (remember, the linq statement returns an IQueryable)...
foreach (var data in customers)
{
string doSomething = data.variableName; //to get data from your query.
}
hope this helps, not exactly the same methods, but I find this handy in my code

Probably it's because your Cusomters() method in your repository is doing a GetAll() kind of thing and fetching the entire list first. This prohibits LINQ and your SQL Server from creating smart queries.
I don't know if there's a good workaround for your repository, but if you would do something like:
using(var db = new RemoteServerContext())
{
var custs = db.Customers.Where(...);
}
I think that will be a lot quicker. If your project is small enough, you can do without a repository. Sure, you'll lose an abstraction layer, but with small projects this may not be a big problem.
On the other hand, you could load all Customers in your repository once and use the resulting collection directly (instead of the method-call that fills the list). Beware of adding, removing and modifying Customers though.

You need the LINQ query to return less data like sql paging like top function in sql or do manual querying using stored procedures. In either cases, you need to rewrite your querying mechanism. This is one of the reasons why I didn't use EF, because you don't have a lot of control over the code it seems.

C# Generics Question

I have a couple of areas in an application I am building where it looks like I may have to violate the living daylights out of the DRY (Don't Repeat Yourself) principle. I'd really like to stay dry and not get hosed and wondered if someone might be able to offer me a poncho. For background, I am using C#/.NET 3.51 SP1, Sql Server 2008, and Linq-to-Sql.
Basically, my situations revolve around the following scenario. I need to be able to retrieve either a filtered list of items from virtually any table in my database or I need to be able to retrieve a single item from any table in my database given the id of the primary key. I am pretty sure that the best solutions to these problems will involve a good dose of generics and/or reflection.
Here are the two challenges in a little more depth. (Please forgive the verbosity.)
Given a table name (or perhaps a pluralized table name), I would like to be able to retrieve a filtered list of elements in the table. Specifically, this functionality will be used with lookup tables. (There are approximately 50 lookup tables in this database. Additional tables will frequently be added and/or removed.) The current lookup tables all implement an interface (mine) called IReferenceData and have fields of ID (PK), Title, Description, and IsActive.
For each of these lookup tables, I need to sometimes return a list of all records. Other times I need to only return the active records. Any Linq-to-Sql data context automatically contains a List property for each and every TableName. Unfortunately, I don't believe I can use this in it's raw form because it is unfiltered, and I need to apply a filter on the IsActive property.
One option is to write code similar to the following for all 50 tables. Yuk!!!
public List<AAA> GetListAAA(bool activeOnly)
{
return AAAs.Where(b => b.IsActive == true || b.IsActive == activeOnly).OrderBy(c => c.Title).ToList();
}
This would not be terribly hard, but it does add a burden to maintenance.
Note: It is important that when the list is returned that I maintain the underlying data type. The records in these lookup tables may be modified, and I have to apply the updates appropriately.
For each of my 150 tables, I need to be able to retrieve an individual record (FirstOrDefault or SingleOrDefault) by its primary key id. Again, I would prefer not to write this same code many times. I would prefer to have one method that could be used for all of my tables.
I am not really sure what the best approach would be here. Some possibilities that crossed my mind included the following. (I don't have specific ideas for their implementation. I am simply listing them as food for thought.)
A. Have a method like GetTableNameItemByID (Guid id) on the data context. (Good)
B. Have an extension method like GetItem(this, string tableName, Guid id) on the data context. (Better)
C. Have a Generic method or extension method like GetItem (this, Table, Guid id). (I don't even know if this possible but it would be the cleanest to use.) (Best)
Additional Notes
For a variety of reasons, I have already created a partial class for my data context. It would certainly be acceptable if the methods were included in that partial class either as normal methods or in a separate static class for extension methods.

Since you already have a partial implementation of your data context, you could add:
public IQueryable<T> GetList<T>( bool activeOnly ) where T : class, IReferenceData
{
return this.GetTable<T>()
.Where( b => !activeOnly || b.isActive )
.OrderBy( c => c.Title );
}
Retaining the IQueryable character of the data will defer the execution of the query until you are ready to materialize it. Note that you may want to omit the default ordering or have separate methods with and without ordering to allow you to apply different orderings if you desire. If you leave it as an IQueryable, this is probably more valuable since you can use it with paging to reduce the amount of data actually returned (per query) if you desire.

There's a design pattern for your needs called "Generic Repository" .Using this pattern you'll get an IQueryable instead of a real list of your entities which lets you do some other stuff with your query as you go.The point is to let the business layer gets whatever it needs whenever it needs it in a generic approach.
You can find an example here.

Have you considered using a code generation tool? Have a look at CodeSmith. Using a tool like that or T4 will allow you to generate your filter functions automatically and should make them fairly easy to maintain.
I'm not sure the best link to provide for T4, but you could start with this video.

Would this meet your needs?
public static IEnumerable<T> GetList<T>(this IEnumerable<IReferenceData> items, bool activeOnly)
{
return items.Where(b => b.IsActive == true || b.IsActive == activeOnly).OrderBy(c => c.Title).Cast<T>().ToList();
}
You could use it like this:
IEnumerable<IReferenceData> yourList;
List<DerivedClass> filtered = yourList.GetList<DerivedClass>(true);

To do something like this without demanding interfaces etc, you can use dynamic Expressions; something like:
public static IList<T> GetList<T>(
this DataContext context, bool activeOnly )
where T : class
{
IQueryable<T> query = context.GetTable<T>();
var param = Expression.Parameter(typeof(T), "row");
if(activeOnly)
{
var predicate = Expression.Lambda<Func<T, bool>>(
Expression.Equal(
Expression.PropertyOrField(param, "IsActive"),
Expression.Constant(true,typeof(bool))
), param);
query = query.Where(predicate);
}
var selector = Expression.Lambda<Func<T, string>>(
Expression.PropertyOrField(param, "Title"), param);
return query.OrderBy(selector).ToList();
}

Ado.Net Entity : Object doesn't display linked members (foreign keys)

I have a simple databasescheme: User, Account. User has 1-to-many relationship with Account.
I have generated a ado.net entity data model, and I can create users and accounts, and even link them together. In the database the account.user_id is correctly filled, so theoretically I should be able to acces User.Account.ToList() in C# through entity.
However, When I try to acces User.Account.ToList() I get zero results.
User user = db.User.First(U => U.id == 1);
List<Account> accounts = user.Account.ToList(); ##count = 0...
When I add the following code before the previous code it suddenly gives me the correct count 2.
Account account1 = db.Account.First(A => A.id == 1);
Account account2 = db.Account.First(A => A.id == 2);
User user = db.User.First(U => U.id == 1);
List<Account> accounts = user.Account.ToList(); ##count = 2...??
What am I missing here??

You should use the ObjectQuery.Include method for this. Your method works also but results in an additional query.
In your example you would get
User user = db.User.Include("Account").First(u => u.id == 1);
You have to figure out whether the string "Account" is correct. Usually it should be prefixed with something like MyEntities. This depends on the namespace of your entities but with a little trial and error you should be able to figure this out.

Yes, that's a common problem when starting to use the Entity framework - neither parent nor child relationships are lazy loaded so you have to load them explicitly. If you are going to share the object context around between classes / methods you might want to make a check to see if the relationship is already loaded:
e.g.
if(!user.Account.IsLoaded)
user.Account.Load();
You can make this easier with a simple extension method:
public static class EntityExtensions
{
public static void EnsureLoaded(this RelatedEnd relatedEnd)
{
if (!relatedEnd.IsLoaded)
relatedEnd.Load();
}
}
using this makes your load call shorter again:
user.Account.EnsureLoaded();
And as it uses a RelatedEnd, which is common to parent and child relationships in the entity framework, you can use this for parent reference relationships too - e.g.
account.UserReference.EnsureLoaded();
As rwwilden says, if you are always going to load the child objects with the parent in this case, you might want to use an Include to make the call more efficient and avoid an extra roundtrip to the database.

I guess my knowledge is a bit small of the framework. :)
You need to explicitly load the related accounts first.
user.Account.Load();
Now It does display correctly.