Develop Reference

Passing a GetWhere query (Func<entityDTO,bool>) to a data layer method which needs a (Func<entity,bool>) parameter to work

I have the following method in a data access class which uses entity framework:
public static IEnumerable<entityType> GetWhere(Func<entityType, bool> wherePredicate)
{
using (DataEntities db = new DataEntities())
{
var query = (wherePredicate != null)
? db.Set<entityType>().Where(wherePredicate).ToList()
: db.Set<entityType>().ToList();
return query;
}
}
This works fine when I use the entities across all layers... however I am trying to move to using a DTO class and I would like to do something like the following:
public static IEnumerable<EntityTypeDTO> GetWhere(Func<EntityTypeDTO, bool> wherePredicate)
{
//call a method here which will convert Func<EntityTypeDTO,bool> to
// Func<EntityType,bool>
using (DataEntities db = new DataEntities())
{
var query = new List<EntityType>();
if (wherePredicate == null)
{
query = db.Set<EntityType>().ToList();
}
else
{
query = (wherePredicate != null)
? db.Set<EntityType>().Where(wherePredicate).AsQueryable<EntityType>().ToList()
: db.Set<EntityType>().ToList();
}
List<EntityTypeDTO> result = new List<EntityTypeDTO>();
foreach(EntityType item in query)
{
result.Add(item.ToDTO());
}
return result;
}
}
Essentially I want a method which will convert Func to Func.
I think I have to break down the Func into an expression tree and then rebuild it somehow in the entityType?
I want to do this to allow the Presentation Layer to just pass the Expression queries?
Am I missing something basic or is there an easier design pattern that can pass a query from a DTO to a data access class without knowing the details of the query?
I have tried making the DTO inherit from the entity which doesn't seem to work either?
If there is a better design pattern that I am missing I would love a pointer and I can investigate from there...

Firstly I would suggest that you put a querying layer of your own in front of Entity Framework rather than allowing any arbitrary Func to be passed in because it will be very easy in the future to pass a Func that Entity Framework can not translate into a SQL statement (it can only translate some expressions - the basics are fine but if your expression calls a C# method, for example, then Entity Framework will probably fail).
So your search layer could have classes that you build up as criteria (eg. a "ContainsName" search class or a "ProductHasId" class) that are then translated into expressions in your search layer. This separates your app entirely from the ORM, which means that ORM details (like the entities or like the limitations of what Funcs can and can't be translated) don't leak out. There's lots out there that's been written about this some of arrangement.
One final note, though, if you are working close to the ORM layer, Entity Framework is very clever and you could probably get a long way without trying to translate your Func<dto, bool> to a Func<entity, bool>. For example, in the below code, accessing "context.Products" returns a "DbSet" and calling Select on it returns an IQueryable and calling Where on that also returns an IQueryable. Entity Framework will translate all of that into a single SQL statement so it won't pull all other Products into memory and then filter the ID on that memory set, it will actually perform the filtering in SQL even though the filter is operating on a projected type (which is equivalent to the DTO in your case) and not the Entity Framework entity -
var results = context.Products
.Select(p => new { ID = p.ProductID, Name = p.ProductName })
.Where(p => p.ID < 10)
.ToList();
The SQL executed is:
SELECT
[Extent1].[ProductID] AS [ProductID],
[Extent1].[ProductName] AS [ProductName]
FROM [dbo].[Products] AS [Extent1]
WHERE [Extent1].[ProductID] < 10
So, if you changed your code to get something like..
return context.Products
.Map<Product, ProductDTO()>()
.Where(productDtoWherePredicate)
.ToList();
.. then you might be just fine with the Funcs that you already have. I presume that you already have some sort of mapping functions to get from EF entities to DTOs (but if not then you might want to look into AutoMapper to help you out - which has support for "projections", which are basically IQueryable maps).

I am going to put this up as an answer.Thanks to Dan for the quick answer. Looking at what you are saying I can write a query/filter set of classes. for example, take the following code:
GetProducts().GetProductsInCategory().GetProductsWithinPriceRange(minPrice, maxPrice);
This code would run like so: Get Products would get all products in the table and the remaining functions would filter the results. if all queries run like this it may put a significant load on the Data Access Layer/ DB Server Connections... not sure.
or
An Alternate I will work on also is:
If each function creates a Linq expression, I could combine them like this: How do I combine multiple linq queries into one results set?
this may allow me to do this in a manner where I can return the filtered results set from the database.
Either way I am marking this as answered. I will update when I have more details.

Converting this to an aggregate function

I am using the repository pattern + entity framework for a small project I'm working on. I needed to cut out lazy loading for performance reasons, but now I also need to include child entities in my db fetches. My current (working) solution is this:
protected MediaDbEntities MediaDb { get { return db ?? (db = DatabaseFactory.Get()); } }
public virtual IEnumerable<T> All(string[] childEntities = null)
{
var query = MediaDb.Set<T>();
foreach (var childEntity in childEntities)
{
query.Include(childEntity);
}
return query.ToList();
}
I would really like to explore the use of an aggregate in this case, but don't really know how to apply. I have only used aggregates for sums and arithmetic operations. Anybody have an answer I can learn from?

In this case, you're not actually aggregating any data so trying to create any sort of aggregate is the wrong answer.
There's nothing wrong with your current solution. It makes perfect sense the way it is and I, personally, would leave it the way it is.

Improving efficiency with Entity Framework

I have been using the Entity Framework with the POCO First approach. I have pretty much followed the pattern described by Steve Sanderson in his book 'Pro ASP.NET MVC 3 Framework', using a DI container and DbContext class to connect to SQL Server.
The underlying tables in SQL server contain very large datasets used by different applications. Because of this I have had to create views for the entities I need in my application:
class RemoteServerContext : DbContext
{
public DbSet<Customer> Customers { get; set; }
public DbSet<Order> Orders { get; set; }
public DbSet<Contact> Contacts { get; set; }
...
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<Customer>().ToTable("vw_Customers");
modelBuilder.Entity<Order>().ToTable("vw_Orders");
...
}
}
and this seems to work fine for most of my needs.
The problem I have is that some of these views have a great deal of data in them so that when I call something like:
var customers = _repository.Customers().Where(c => c.Location == location).Where(...);
it appears to be bringing back the entire data set, which can take some time before the LINQ query reduces the set to those which I need. This seems very inefficient when the criteria is only applicable to a few records and I am getting the entire data set back from SQL server.
I have tried to work around this by using stored procedures, such as
public IEnumerable<Customer> CustomersThatMatchACriteria(string criteria1, string criteria2, ...) //or an object passed in!
{
return Database.SqlQuery<Customer>("Exec pp_GetCustomersForCriteria #crit1 = {0}, #crit2 = {1}...", criteria1, criteria2,...);
}
whilst this is much quicker, the problem here is that it doesn't return a DbSet and so I lose all of the connectivity between my objects, e.g. I can't reference any associated objects such as orders or contacts even if I include their IDs because the return type is a collection of 'Customers' rather than a DbSet of them.
Does anyone have a better way of getting SQL server to do the querying so that I am not passing loads of unused data around?

var customers = _repository.Customers().Where(c => c.Location == location).Where(...
If Customers() returns IQueryable, this statement alone won't actually be 'bringing back' anything at all - calling Where on an IQueryable gives you another IQueryable, and it's not until you do something that causes query execution (such as ToList, or FirstOrDefault) that anything will actually be executed and results returned.
If however this Customers method returns a collection of instantiated objects, then yes, since you are asking for all the objects you're getting them all.
I've never used either code-first or indeed even then repository pattern, so I don't know what to advise, other than staying in the realm of IQueryable for as long as possible, and only executing the query once you've applied all relevant filters.

What I would have done to return just a set of data would have been the following:
var customers = (from x in Repository.Customers where <boolean statement> &&/|| <boolean statement select new {variableName = x.Name , ...).Take(<integer amount for amount of records you need>);
so for instance:
var customers = (from x in _repository.Customers where x.ID == id select new {variableName = x.Name} ).take(1000);
then Iterate through the results to get the data: (remember, the linq statement returns an IQueryable)...
foreach (var data in customers)
{
string doSomething = data.variableName; //to get data from your query.
}
hope this helps, not exactly the same methods, but I find this handy in my code

Probably it's because your Cusomters() method in your repository is doing a GetAll() kind of thing and fetching the entire list first. This prohibits LINQ and your SQL Server from creating smart queries.
I don't know if there's a good workaround for your repository, but if you would do something like:
using(var db = new RemoteServerContext())
{
var custs = db.Customers.Where(...);
}
I think that will be a lot quicker. If your project is small enough, you can do without a repository. Sure, you'll lose an abstraction layer, but with small projects this may not be a big problem.
On the other hand, you could load all Customers in your repository once and use the resulting collection directly (instead of the method-call that fills the list). Beware of adding, removing and modifying Customers though.

You need the LINQ query to return less data like sql paging like top function in sql or do manual querying using stored procedures. In either cases, you need to rewrite your querying mechanism. This is one of the reasons why I didn't use EF, because you don't have a lot of control over the code it seems.

Linq to SQL Expression Properties That are Translatable to SQL

I have a LINQ to SQL class, we'll call it Test, and I want to be able to access properties with LINQ queries but I get the famed "No Supported Translation to SQL" runtime error. I'm interested in the conceptual problem. Here is my simplified class:
public class Test
{
public int ID {get; set;} // Stored in Database
public int NonForeignKeyValue {get; set;} // Stored in Database
}
Here is sort of an example of what I'm trying to accomplish, but I don't want the overhead of always explicitly writing the join in LINQ:
var db = (new DataContext()).GetTable<Test>();
var q = (from t in db.GetTable<Test>()
join o in db.GetTable<OtherTable>() on o.ID equals t.ID
where t.OtherStuff
select t)
I'd like to be able to add a property to Test that tells me if there are any rows in OtherTable that could be joined with Test:
public bool IsInOtherTable
{
get
{
return (new DataContext())
.GetTable<OtherTabke>()
.Any(x => x.NonForeignKeyValue == this.NonForeignKeyValue));
}
}
Ultimately this is what I want my code to look like, but it errors. I basically want to return all entries that contain some database computed value:
using (DataContext db = new DataContext())
{
var q = db.GetTable<Test>()
.Where(x => x.IsInOtherTable && x.OtherStuff); //Error
}
I'm basically trying to save myself from writing this code every single time I want to check if Test has certain information in another table. I'm not that interested in the exact problem I described, I'm more interested in how to conceptually add the join part to the SQL and still use LINQ. I'm guessing I use Linq.Expression, but I really don't know and I'm not aware of how to do it.
As an aside, I could just write the actual SQL, as its not that complicated, but I'd like to know how to get around this and still use LINQ.
Edit: I tried this property, but I get the same error. Its more complicated that just changing the return type to Expression...
public System.Linq.Expressions.Expression<Func<Article3, bool>> Exists
{
get
{
using (DataContext db = new DataContext())
{
return i => db.GetTable<OtherTable>()
.Any(x => x.NonForeignKeyValue == i.NonForeignKeyValue));
}
}
}

Each time the linq generator is to translate a code into a query, it has to process an expression tree.
In your examples, you are not passing around expression but rather - properties, delegates, i.e. stuff which the expression visitor is unable to "step into".
In general, try to rethink your conditions so that instead of bool you have Expression<T, bool> etc.
http://netpl.blogspot.com/2008/02/linq-to-object-vs-linq-to-sql-and.html

Firstly, I think you may be overestimating "the overhead of always explicitly writing the join in LINQ". It's an extra line of code which has the advantage of being relatively self-documenting as to just what you are doing (always a nice thing), and any other approach is going to be turned first into SQL and then into a query plan that will be at least as expensive to execute, possibly more expensive (SQLServer is good a joins!)
Still, there are two alternatives I can thinkof.
One is to have an EntityRef property on the class that defines this relationship with the other table. You can then test if it is null in your query (or EntitySet if it's on the other side of a one-to-many relationship).
The other is to define a SQL function that returns a bit result indicating whether an id refers to a row that does or doesn't relate to the other table.
Then define a protected method on your DataContext-derived class that matches the signature in C# terms, and has a Function attribute mapping it to that SQL function. (Since this isn't something that you can give a sensible non-db-using version of in the C# version, you can implement the C# function by calling ExecuteMethodCall).
Then you can use that method instead of the join.
Still, this is likely less self-explanatory in the code and at risk of being less efficient than just keeping the join.

Adding behavior to LINQ to Entities models

What's the preferred approach when using L2E to add behavior to the objects in the data model?
Having a wrapper class that implements the behavior you need with only the data you need
using (var dbh = new ffEntities())
{
var query = from feed in dbh.feeds select
new FFFeed(feed.name, new Uri(feed.uri), feed.refresh);
return query.ToList();
}
//Later in a separate place, not even in the same class
foreach (FFeed feed in feedList) { feed.doX(); }
Using directly the data model instances and have a method that operates over the IEnumerable of those instances
using (var dbh = new ffEntities())
{
var query = from feed in dbh.feeds select feed;
return query.ToList();
}
//Later in a separate place, not even in the same class
foreach (feeds feed in feedList) { doX(feed); }
Using extension methods on the data model class so it ends up having the extra methods the wrapper would have.
public static class dataModelExtensions {
public static void doX(this feeds source) {
//do X
}
}
//Later in a separate place, not even in the same class
foreach (feeds feed in feedList) { feed.doX(); }
Which one is best? I tend to favor the last approach as it's clean, doesn't interfere with the CRUD facilities (i can just use it to insert/update/delete directly, no need to wrap things back), but I wonder if there's a downside I haven't seen.
Is there a fourth approach? I fail at grasping LINQ's philosophy a bit, especially regarding LINQ to Entities.

The Entity classes are partial classes as far as i know, so you can add another file extending them directly using the partial keyword.
Else, i usually have a wrapper class, i.e. my ViewModel (i'm using WPF with MVVM). I also have some generic Helper classes with fluent interfaces that i use to add specific query filters to my ViewModel.

I think it's a mistake to put behaviors on entity types at all.
The Entity Framework is based around the Entity Data Model, described by one of its architects as "very close to the object data model of .NET, modulo the behaviors." Put another way, your entity model is designed to map relational data into object space, but it should not be extended with methods. Save your methods for business types.
Unlike some other ORMs, you are not stuck with whatever object type comes out of the black box. You can project to nearly any type with LINQ, even if it is shaped differently than your entity types. So use entity types for mapping only, not for business code, data transfer, or presentation models.
Entity types are declared partial when code is generated. This leads some developers to attempt to extend them into business types. This is a mistake. Indeed, it is rarely a good idea to extend entity types. The properties created within your entity model can be queried in LINQ to Entities; properties or methods you add to the partial class cannot be included in a query.
Consider these examples of a business method:
public Decimal CalculateEarnings(Guid id)
{
var timeRecord = (from tr in Context.TimeRecords
.Include(“Employee.Person”)
.Include(“Job.Steps”)
.Include(“TheWorld.And.ItsDog”)
where tr.Id = id
select tr).First();
// Calculate has deep knowledge of entity model
return EarningsHelpers.Calculate(timeRecord);
}
What's wrong with this method? The generated SQL is going to be ferociously complex, because we have asked the Entity Framework to materialize instances of entire objects merely to get at the minority of properties required by the Calculate method. The code is also fragile. Changing the model will not only break the eager loading (via the Include calls), but will also break the Calculate method.
The Single Responsibility Principle states that a class should have only one reason to change. In the example shown on the screen, the EarningsHelpers type has the responsibility both of actually calculating earnings and of keeping up-to-date with changes to the entity model. The first responsibility seems correct, the second doesn't sound right. Let's see if we can fix that.
public Decimal CalculateEarnings(Guid id)
{
var timeData = from tr in Context.TimeRecords
where tr.Id = id
select new EarningsCalculationContext
{
Salary = tr.Employee.Salary,
StepRates = from s in tr.Job.Steps
select s.Rate,
TotalHours = tr.Stop – tr.Start
}.First();
// Calculate has no knowledge of entity model
return EarningsHelpers.Calculate(timeData);
}
In the next example, I have rewritten the LINQ query to pick out only the bits of information required by the Calculate method, and project that information onto a type which rolls up the arguments for the Calculate method. If writing a new type just to pass arguments to a method seemed like too much work, I could have also projected onto an anonymous type, and passed Salary, StepRates, and TotalHours as individual arguments. But either way, we have fixed the dependency of EarningsHelpers on the entity model, and as a free bonus we've gotten more efficient SQL, as well.
You might look at this code and wonder what would happen if the Job property of TimeRecord where nullable. Wouldn't I get a null reference exception?
No, I would not. This code will not be compiled and executed as IL; it will be translated to SQL. LINQ to Entities coalesces null references. In the example query shown on the screen, StepRates would simply return null if Job was null. You can think of this as being identical to lazy loading, except without the extra database queries. The code says, "If there is a job, then load the rates from its steps."
An additional benefit of this kind of architecture is that it makes unit testing of the Web assembly very easy. Unit tests should not access a database, generally speaking (put another way, tests which do access a database are integration tests rather than unit tests). It's quite easy to write a mock repository which returns arrays of objects as Queryables rather than actually going to the Entity Framework.