Develop Reference

How to create a reusable where clause for EF6

I have recently moved from coding in Java to c# and I am still learning the various elements of c#.
To access an existing database, which I cannot redesign, I am using Entity Frameworks 6 and 'Code First from database' to generate contexts and types representing the database tables. I am using Ling-To-SQL to retrieve the data from the database which is heavily denormalized.
My current task is create a report where each section is read from various tables, which all have a relationship to one base table.
This is my working example:
using(var db = new PaymentContext())
{
var out = from pay in db.Payment
join typ in db.Type on new { pay.ID, pay.TypeID } equals
new { typ.ID, typ.TypeID }
join base in db.BaseTable on
new { pay.Key1, pay.Key2, pay.Key3, pay.Key4, pay.Key5 } equals
new { base.Key1, base.Key2, base.Key3, base.Key4, base.Key5 }
where
base.Cancelled.Equals("0") &&
base.TimeStamp.CompareTo(startTime) > 0 &&
base.TimeStamp.CompareTo(endTime) < 1 &&
.
(other conditions)
.
group new { pay, typ } by new { typ.PaymentType } into grp
select new
{
name = grp.Key,
count = grp.Count(),
total = grp.Sum(x => x.pay.Amount)
};
}
There will be a large number of sections in the report and each section will generate a where clause which will contain the conditions shown. In some sections, the required data will be extracted from tables up to five levels below the BaseTable.
What I want to do is create a resuable where clause for each report section, to avoid a lot of duplicated code.
After a lot of searching, I tried to use the solution suggested here , but this has been superseded in Entity Framework 6.
How do I avoid duplicating code unnecessarily?

I did try to use the extension clauses you suggested, but my generated classes do not extend the BaseTable, so I had to explicitly define the link through the navigation property. As only a small number of tables will be common in the queries, I decided to apply the filters directly to each table as required. I will define these as required.
krillgar suggested moving to straight LINQ syntax, which seems like good advice. We intend to redesign our database in the near future and this will remove some of the SQL dependency. I merged the suggested filters and full LINQ syntax to access my data.
// A class to hold all the possible conditions applied for the report
// Can be applied at various levels within the select
public class WhereConditions
{
public string CancelledFlag { get; set; } = "0"; // <= default setting
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
}
// Class to define all the filters to be applied to any level of table
public static class QueryExtensions
{
public static IQueryable<BaseTable> ApplyCancellationFilter(this IQueryable<BaseTable> base, WhereConditions clause)
{
return base.Where(bse => bse.CancelFlag.Equals(clause.CancelledFlag));
}
public static IQueryable<BaseTable> ApplyTimeFilter(this IQueryable<BaseTable> base, WhereConditions clause)
{
return base.Where(bse => bse.TimeStamp.CompareTo(clause.StartTime) > 0 &&
bse.TimeStamp.CompareTo(clause.EndTime) < 1);
}
}
And the query is composed as follows:
using (var db = new PaymentContext())
{
IEnumerable<BaseTable> filter = db.BaseTable.ApplyCancellationFilter(clause).ApplyTimeFilter(clause);
var result = db.Payment.
Join(
filter,
pay => new { pay.Key1, pay.Key2, pay.Key3, pay.Key4, pay.Key5 },
bse => new { bse.Key1, bse.Key2, bse.Key3, bse.Key4, bse.Key5 },
(pay, bse) => new { Payment = pay, BaseTable = bse }).
Join(
db.Type,
pay => new { pay.Payment.TypeKey1, pay.Payment.TypeKey2 },
typ => new { typ.TypeKey1, typ.TypeKey2 },
(pay, typ) => new { name = typ.Description, amount = pay.Amount }).
GroupBy(x => x.name).
Select(y => new { name = y.Key,
count = y.Count(),
amount = y.Sum(z => z.amount)});
}
And then to finally execute composed query.
var reportDetail = result.ToArray(); // <= Access database here
As this query is the simplest I will have to apply, future queries will become much more complicated.

The nice thing about LINQ is that methods like Where() return an IEnumerable<T> that you can feed into the next method.
You could refactor the where clauses into extension methods akin to:
public static class PaymentQueryExtensions {
public static IQueryable<T> ApplyNotCancelledFilter(
this IQueryable<T> payments)
where T : BaseTable {
// no explicit 'join' needed to access properties of base class in EF Model
return payments.Where(p => p.Cancelled.Equals("0"));
}
public static IQueryable<T> ApplyTimeFilter(
this IQueryable<T> payments, DateTime startTime, DateTime endTime)
where T: BaseTable {
return payments.Where(p => p.TimeStamp.CompareTo(startTime) > 0
&& p.TimeStamp.CompareTo(endTime) < 1);
}
public static IGrouping<Typ, T> GroupByType(
this IQueryable<T> payments)
where T: BaseTable {
// assuming the relationship Payment -> Typ has been set up with a backlink property Payment.Typ
// e.g. for EF fluent API:
// ModelBuilder.Entity<Typ>().HasMany(t => t.Payment).WithRequired(p => p.Typ);
return payments.GroupBy(p => p.Typ);
}
}
And then compose your queries using these building blocks:
IEnumerable<Payment> payments = db.Payment
.ApplyNotCancelledFilter()
.ApplyTimeFilter(startTime, endTime);
if (renderSectionOne) {
payments = payments.ApplySectionOneFilter();
}
var paymentsByType = payments.GroupByType();
var result = paymentsByType.Select(new
{
name = grp.Key,
count = grp.Count(),
total = grp.Sum(x => x.pay.Amount)
}
);
Now that you have composed the query, execute it by enumerating. No DB access has happened until now.
var output = result.ToArray(); // <- DB access happens here
Edit After the suggestion of Ivan, I looked at our codebase. As he mentioned, the Extension methods should work on IQueryable instead of IEnumerable. Just take care that you only use expressions that can be translated to SQL, i.e. do not call any custom code like an overriden ToString() method.
Edit 2 If Payment and other model classes inherit BaseTable, the filter methods can be written as generic methods that accept any child type of BaseTable. Also added example for grouping method.

How do I simplify the access of a has-many relationship with the entity framework?

Here is what I want to do:
var user = db.User.First(conditions);
user.Book.First();
Here is currently how I have to do that.
var user = db.User.Include("Book").First(conditionsForUser);
user.Book.First();
The reason why I want to simplify this, is because I don't want to have to specify what is included every time I want to access a relationship. Seems very cumbersome.
e.g.: I would like to just be able to do the following, given I have previously retrieved a user:
user.Book.First()
user.Blog.First()
user.SomeOtherHasManyRelationship.Where(conditions)
Here is what I have so far:
public object RelationshipFor(string relationship)
{
using (var db = User.DbContext())
{
var relationshipType = TypeRepresentedBy(relationship); // unused for now, not sure if I need the type of the relationship
var myTable = ((ICollection)db.Send(RelationshipName)); // RelationshipName is "User" in this instance.
var meWithRelationship = myTable.Where(i => i.Send(IdColumn) == Id).Include(relationship); // currently, myTable doesn't know about 'Where' for some reason.
return meWithRelationship.Send(relationship);
}
}
And then how that would be used would be the following:
user.RelationshipFor("Book") // returns a list of books
I have some other logic in my code which abstracts that further which would allow me to do user.Book.First().
Hopefully I can get permission to open source a lot of this, as I'm modelling a lot of the api after ActiveRecord-style crud.
Note, that I'm using I set of extensions I made to help dealing with dynamicness less painful: https://github.com/NullVoxPopuli/csharp-extensions
UPDATE 1:
public object RelationshipFor(string relationship)
{
using (var db = User.DbContext())
{
var myTable = (DbSet<DatabaseModels.User>)db.Send(RelationshipName);
var myInclude = myTable.Include(i => i.Send(relationship));
var meWithRelationship = myInclude.First(i => (long)i.Send(IdColumn) == Id);
return meWithRelationship.Send(relationship);
}
}
For now, I've hard coded the cast of the user in an attempt to just get something working.
My error now is:
Unable to cast object of type 'System.Linq.Expressions.MethodCallExpressionN' to type 'System.Linq.Expressions.MemberExpression'.

This is not a trivial problem, and there's no "one size fits all" approach. What you actually seem to be after is lazy loading, which was not included in EF7 for many reasons.
I don't know what the code you show is supposed to do, but one option would be to introduce a repository pattern, where you specify the "entities to include" at the collection level:
public class UserRepository
{
private readonly IQueryable<User> _dataSet;
public UserRepository(IQueryable<User> userDataSet)
{
_dataSet = userDataSet;
}
public IQueryable<User> Include()
{
return _dataSet.Include(u => u.Book)
.Include(u => u.Blog);
}
}
And you can move lots of the logic to a generic base class, leaving you with just the Include() method. You can for example work with strings as you show (or enums, or ...), to only select related entities to include:
public class GenericRepository
{
// ...
public IQueryable<User> Include(string includeGroup = null)
{
return IncludeGroup(includeGroup);
}
protected virtual IncludeGroup(string includeGroup)
{
return _dataSet;
}
}
And then in UserRepository:
protected override IQueryable<User> IncludeGroup(string includeGroup)
{
switch (includeGroup.ToUpperInvariant())
{
case "BOOK":
return _dataSet.Include(u => u.Book)
.Include(u => u.Book.Author);
case "BLOG":
return _dataSet.Include(u => u.Blog);
default:
return base.Include(includeGroup);
}
}
And then use it like this:
var userRepo = new UserRepository(db.User);
var userWithBooks = userRepo.Include("Book");
var firstUser = userWithBooks.FirstOrDefault(u => u.Name == "Foo");
var firstUserFirstBook = firstUser.Book.FirstOrDefault();
One alternative would be to always include all navigation properties (recursively), but that would be a horrible approach in terms of query efficiency, as every query will be one massive join to all related tables, whether that is necessary or not.

LINQ to Entities. How to create "Reusable" methods that "wrap" entities?

SETUP
I have an "Order" class and an "OrderDetail" class.
The Order class has a property called "OrderDetails" of type IQueryable.
I have a method in each class called "GetData" and this method returns IQueryable of the class it is in.
The "GetData" methods are wrappers around the Entities that are used to transform the Entities into friendlier .NET classes / properties.
For example: The OrderDetail Entity has a "PullDate" in the database as Char(10) but I want the OrderDetail Class to have a property called "PullDate" that is of type DateTime in the class.
CODE
public class Order
{
[Key]
public int ID { get; set; }
public IQueryable<OrderDetail> OrderDetails { get; set; }
public static IQueryable<Order> GetData()
{
IQueryable<Order> orders;
var db = new OrderEntities();
// NOTE: This code will work
try
{
var data =
(from O in db.Orders
where O.OrderID == 1
select new Order
{
ID = O.OrderID,
OrderDetails =
(from OD in db.OrderDetails
where OD.OrderID == O.OrderID
select new OrderDetail
{
ID = OD.OrderDetailID,
OrderID = OD.OrderID,
PullDate = OD.PullDate == "00000000" ?
(DateTime?)null : db.yyyyMMddToDate(OD.PullDate),
Description = OD.Desc
})
});
orders = data.AsQueryable();
var orderList = orders.ToList();
}
catch (Exception ex)
{
throw;
}
// NOTE: This code will NOT work
try
{
var data = (from O in db.Orders
where O.OrderID == 1
select new Order
{
ID = O.OrderID,
OrderDetails = (from OD in OrderDetail.GetData(db)
where OD.OrderID == O.OrderID
select OD)
});
orders = data.AsQueryable();
var orderList = orders.ToList();
}
catch (Exception ex)
{
throw;
}
return orders;
}
}
public class OrderDetail
{
[Key]
public int ID { get; set; }
public int OrderID { get; set; }
public DateTime? PullDate { get; set; }
public string Description { get; set; }
public static IQueryable<OrderDetail> GetData(OrderEntities db)
{
IQueryable<OrderDetail> orderDetails;
var data = (from OD in db.OrderDetails
select new OrderDetail
{
ID = OD.OrderDetailID,
OrderID = OD.OrderID,
PullDate = OD.PullDate == "00000000" ?
(DateTime?)null : db.yyyyMMddToDate(OD.PullDate),
Description = OD.Desc
});
orderDetails = data.AsQueryable();
var orderList = orderDetails.ToList();
return orderDetails;
}
}
ERROR
LINQ to Entities does not recognize the method 'System.Linq.IQueryable`1[Models.OrderDetail] GetData(Models.OrderEntities)' method, and this method cannot be translated into a store expression.
Request
I would like the Order.GetData method to call the OrderDetail.GetData method using LINQ.
I need to "join" the GetData methods together or "sub select" the OrderDetail.GetData while inside the Order.GetData class.
Both of the classes are querying EntityFramework inside of their GetData methods.
Projection is a requirement.
My goal is to create "Reusable" methods like "GetData" in my DTO classes that will contain specific SQL / Entity logic.
For example, I am using a lot of custom SQL functions like "db.yyyyMMddToDate" in my DTO classes to transform the Entities into something more object / .NET friendly and I don't want to "retype" all that logic each time I need to "join / sub select" data from entities.
In LINQ to Objects this would be the same as joining two different lists from different classes.
But it seems that LINQ to Entity does not know how to join methods from other classes even if the method is marked as Queryable.
I understand that LINQ to Entity is treating the "GetData" methods as if they were SQL functions but I need a way to tell LINQ to Entity that these are just more Entities to join together to get the results.

You get this error because, when dealing with LINQ to Entities, you are not passing in lambdas to the LINQ operators like you do in LINQ to Object. You are passing expression trees. The code that you write in those queries will never be executed. It will be interpreted by Entity Framework as a series of instruction to translate into an other language like SQL.
The GetData method doesn't mean anything in SQL. In fact, it doesn't mean anything outside of your own program. This is why LINQ to Entities do not recognize your method as a valid construct.
To access the details of an order in a scenario like yours, you need to build your query so that EF knows to include the OrderDetails entities inside the primary query using the Include operator. When doing this, EF expects your object to have a property similar to this one:
public virtual ICollection<OrderDetail> OrderDetails { get; set; }
This is how EF will know where to put the resulting objects from queries that includes those elements.
var ctx = new DatabaseContext();
var query = ctx.Orders
.Where(o => o.OrderId == 1)
.Include(o => o.OrderDetails);
This query will asks the remote server for all Orders, with an OrderId of 1, with all its matching OrderDetails. EF will then automatically create all of the related client side entities for you.
Here's the official documentation on EF about this topic: http://msdn.microsoft.com/en-ca/data/jj574232.aspx

Global WHERE clause for items that are marked disabled

Recently I had to add a new column for a shop I've made, determining whether item should be available for sale or not.
Now the thing. Is it possible to do something like global where clause or I must to add it separately for each query, that relates to certain column (e.g. Products)? It would be really hard to correct every query and not miss anything.
Example query I use looks like this, but it's just a very basic one. Normally these where clasues are multiline, including selects from another tables.
DataBaseContext db = new DataBaseContext();
// ...
protected bool SomeFunction() {
// ...
var Products = db.Products.
Where(k => k.Active == true).
OrderByDescending(k => k.Date);
// ...
}
Normally I would do
var Products = db.Products.Where(k => k.Active == true);
Products = Products.
Where( ... ).
Select( ... ).
OrderBy( ... ).
...
Take( ... );
But there are mulptiple functions (db is common for every function in class), and I was thinking about writing the condition on the SQL server side, but I have no knowledge about that, sadly.

A simple solution would be to change your products implementation:
Old:
class DataBaseContext
{
//...
public DbSet<Product> Products { get; set; }
}
New:
class DataBaseContext
{
//...
public IQueryable<Product> Products
{
get
{
return this.Set<Product>().Where(pr => pr.IsActive == true);
}
}
}
However, this is not very robust and maintenance friendly, since you would have to do this for every type of item that can be activated. Also, you would need to create a second property of type DbSet called AllProducts and then vary between if you want to get active or allitems by checking all the points where the query is used.
Alternatively, you could create a wrapper for your DbContext:
interface IMyContext {
void SaveChanges();
IQueryable<T> Set<T>() where T: class
IQUeryable<T> GetActiveItems<T>() where T : SomeBaseClassWithActiveProperty
}
public class MyContext : IMyContext {
DataBaseContext _underylingContext = new DataBaseContext();
//... save changes implementation etc
public IQueryable<T> Set<T>()
where T : class
{
return _underlyingContext.Set<T>();
}
public IQueryable<T> GetActiveItems<T>()
where T : SomeBaseClassWithActiveProperty
{
return this.Set<T>().Where(item => item.IsActive == true);
}
}
then, when using it:
MyContext context = new MyContext();
var activeProducts = from p in context.GetActiveItems<Product>()
order p by p.Date //... or whatever;
var allProducts = from p in context.Set<Product>() //....
Either way, you should go by and check all calls to the Product DbSet and validate if you only need active items or all items.

You can do this inside the database with the following two steps:
(1) Rename your existing table to something else.
(2) Create a view with the name of the existing table:
create view <tablename> as
select *
from <newtablename>
where <your condition is true>;
(You probably want to list all the column instead of using *.)
Now all queries will use the view instead of the base table.
By the way, when designing an API for a database, it is a good idea to have all access be through views. This allows changes like this after the API is in place.

What is the right return value for a LINQ join method?

I am working on a 3-Tier application. Also I am using LINQ to SQL for data access.
The DataLayer has a function which returns a table of customers:
public Table<Customer> getCustomers()
{
DataContext context = new DataContext();
Table<Customer> customerTable = context.GetTable<Customer>();
return customerTable;
}
It is provided to the Business-Layer where the results are being passed to the Presentation-Layer as IEnumerable<Customer>:
public IEnumerable<Customer> getCustomers()
{
CustomerDAL customerDAL = new CustomerDAL();
return from c in customerDAL.getCustomers() select c;
}
In the Presentation-Layer I am simply using the IEnumerable for a DatagridView's DataSource.
What if I had another table like "Information" and the according customerDAL.getInfo() table? Now I want to make a join query in a method in the Business-Layer. I imagined it something like this:
public IEnumerable<Customer> getCustomerInfo()
{
CustomerDAL customerDAL = new CustomerDAL ();
return from c in customerDAL.getCustomers()
join i in customerDAL.getInfo() on c.id equals i.InfoID
select new { c.id, c.Name, i.DateTime, i.Project };
}
The problem is IEnumerable needs the object's type. My return value isn't a customer-table anymore, but a combination of a customer- and a info-table. Did I get it right? What would be the right choice for a return value here?
After your advice I created a custom class, CustomerInfo.cs:
public class CustomerInfo
{
string name { get; set; }
long id { get; set; }
string dateTime { get; set; }
string project { get; set; }
public CustomerInfo(long _id, string _name, string _date, string _project)
{
name = _name;
id = _id;
dateTime = _date;
project = _project;
}
}
Then I am calling the exact same method as described by Reed. But in the Presentation-Layer when I set the DataSource I get the exception:
The query contains references to items defined on a different data context.
Actually this is not true all Entity classes are in the same .dbml file. What could be wrong?

Regarding the second error:
Then I am calling the exact same method as described by Reed. But in the Presentation-Layer when I set the DataSource I get the exception:
The query contains references to items defined on a different data context.
Most likely your DAL is instantiating a separate instance of the context for each table that you are returning (Typical in a ActiveRecord patterned implementation). In order for the join to work, both tables need to be retrieved by the same context object. You may want to modify your DAL so that you inject the context in the constructor of the DALs so that you can centralize the lifetime of the context.

What would be the right choice for a return value here?
If you want to return a strongly typed class, you will need a custom class to represent this type, such as a CustomerInfo class. You'd need to define this class, and include the appropriate properties and constructor. You could then do:
public IEnumerable<CustomerInfo> GetCustomerInfo()
{
CustomerDAL customerDAL = new CustomerDAL();
return from c in customerDAL.getCustomers()
join i in customerDAL.getInfo() on c.id equals i.InfoID
select new CustomerInfo(c.id, c.Name, i.DateTime, i.Project);
}
This would allow you to return the exact information you need, in a strongly typed manner. I would argue that creating a custom class is especially important in this case, as you have this defined as part of your public API.

So far everything looks good. If you want to just use the query within the scope of a single method, then using an anonymous class (which is what you're currently doing) is just fine. Since you aren't, you need to create a concrete class for your projection so that you can return it.
The query will become:
select new SomeClassYouAreAboutToCreate { c.id, c.Name, i.DateTime, i.Project };
That class can probably just have a bunch of properties, id, Name, etc. You'll just set those properties in the Select.

When you create an anonymous object like that, its type is dynamic. So to return a list of anonymous object like that, you would use the following signature:
public IEnumerable<dynamic> getCustomerInfo() {
CustomerDAL customerDAL = new CustomerDAL ();
return from c in customerDAL.getCustomers()
join i in customerDAL.getInfo() on c.id equals i.InfoID
select new { c.id, c.Name, i.DateTime, i.Project };
}
Keep in mind, though, that the dynamic type has its drawbacks. In particular, you lose strong typing, which can introduce hard-to-find bugs. You should consider Servy's answer for a more robust solution.