My goal is to query and map complex objects with as little overhead as possible. I am working with a large database with lots of related tables. I am trying to use LINQ select and projection to select only the necessary information i need to make the object.
This is the original query I had which was fast and worked great.
List<ClientDTO> clientList = dbClients.Select(client =>
new ClientDTO
{
ID = client.ClientID,
FirstName = client.FirstName,
LastName = client.LastName,
//etc....
Products = client.Products
.Select(prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date,
//etc...
}).ToList(),
Items = client.Items
.Select(item => new ItemDTO
{
ID = item.ID,
Date = item.Date,
//etc...
}
});
Keep in mind the Client table has over 50 related tables, so this query worked great in that it only selected the fields I needed to make the object.
Now what I needed to do is make mappers for these Objects and try to build the same query statement but using the mappers this time. Here is what I ended up with.
List<ClientDTO> clients = dbClients.ProjectToClientDTO();
Using these Mappers
public static List<ClientDTO> ProjectToClientDTO(this IQueryable<Clients> query)
{
var clientList = query.Select(client => new
{
ID = client.ClientID,
FirstName = client.FirstName,
LastName = client.LastName,
//etc...
Products = client.Products.AsQueryable().ProjectToProductDTO().ToList(),
Items = client.Items.AsQueryable().ProjectToItemDTO().ToList()
}
List<ClientDTO> dtoClientList = new List<ClientDTO>();
foreach (var client in clientList)
{
ClientDTO clientDTO = new ClientDTO();
clientDTO.EncryptedID = EncryptID(client.ID, client.FirstName, client.LastName);
//etc...
clientDTO.Products = client.Products;
clientDTO.Items = client.Items;
}
return dtoClientList;
}
public static IQueryable<ProductDTO> ProjectToProductDTO(this IQueryable<Products> query)
{
return query.Select(prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date,
//etc...
});
}
public static IQueryable<ItemDTO> ProjectToItemDTO(this IQueryable<Items> query)
{
return query.Select(item => new ItemDTO
{
ID = item.ID,
Date = item.Date,
//etc...
});
}
After trying to run this I get the following error.
LINQ to Entities does not recognize the method 'ProjectToProductDTO(IQueryable[Products])', and this method cannot be translated into a store expression."}
Can I make LINQ invoke these methods to build the query?
Or is there a better way to query and map these objects without grabbing 50+ tables of unnecessary data for hundreds of clients?
UPDATE
User Tuco mentioned that I could try looking into expression trees. After reading up on them for a bit I came up with this.
public static Expression<Func<Product, ProductDTO>> test = prod =>
new ProductDTO()
{
ID= prod.ID,
Date= prod.Date,
//etc...
};
And use it as such.
Products = client.Products.Select(prod => test.Compile()(prod)),
But running this I receive this error.
The LINQ expression node type 'Invoke' is not supported in LINQ to Entities
You are very close with your 2nd approach!
Let's say you define the projection of the product entity to the DTO (the mapper as you call it) like you did:
Expression<Func<Product, ProductDTO>> productProjection = prod => new ProductDTO
{
ID = prod.ID,
DateOfTransaction = prod.Date
// ...
};
and the projection of the client entity to it's DTO like this (slightly simpler, but logically equivalent to what you did):
Expression<Func<Client, ClientDTO>> clientProjection = client => new ClientDTO
{
ID = client.ClientID,
FirstName = client.FirstName,
// ...
Products = client.Products.Select(productProjection.Compile()).ToList(),
// ...
};
The compiler let's you do so, but the queryable will not understand that. However what you have achieved is that the productProjection is somehow contained in the expression tree. All you have to do is some expression manipulation.
If you look at the subtree the compiler builds for the argument to .Select you'll find a MethodCallExpression - the call to .Compile(). It's .Object expression - the thing that is to be compiled - is a MemberExpression accessing a field named productProjection(!) on an ConstantExpression containing an instance of an oddly named compiler generated closure class.
So: Find .Compile() calls and replace them with what would be compiled, ending up with the very expression tree you had in your original version.
I'm maintaining a helper class for expression stuff called Express. (See another answer that deals with .Compile().Invoke(...) for a similar situation).
clientProjection = Express.Uncompile(clientProjection);
var clientList = dbClients.Select(clientProjection).ToList();
Here's the relevant snipped of the Express class.
public static class Express
{
/// <summary>
/// Replace .Compile() calls to lambdas with the lambdas themselves.
/// </summary>
public static Expression<TDelegate> Uncompile<TDelegate>(Expression<TDelegate> lambda)
=> (Expression<TDelegate>)UncompileVisitor.Singleton.Visit(lambda);
/// <summary>
/// Evaluate an expression to a value.
/// </summary>
private static object GetValue(Expression x)
{
switch (x.NodeType)
{
case ExpressionType.Constant:
return ((ConstantExpression)x).Value;
case ExpressionType.MemberAccess:
var xMember = (MemberExpression)x;
var instance = xMember.Expression == null ? null : GetValue(xMember.Expression);
switch (xMember.Member.MemberType)
{
case MemberTypes.Field:
return ((FieldInfo)xMember.Member).GetValue(instance);
case MemberTypes.Property:
return ((PropertyInfo)xMember.Member).GetValue(instance);
default:
throw new Exception(xMember.Member.MemberType + "???");
}
default:
// NOTE: it would be easy to compile and invoke the expression, but it's intentionally not done. Callers can always pre-evaluate and pass a member of a closure.
throw new NotSupportedException("Only constant, field or property supported.");
}
}
private sealed class UncompileVisitor : ExpressionVisitor
{
public static UncompileVisitor Singleton { get; } = new UncompileVisitor();
private UncompileVisitor() { }
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.Name != "Compile" || node.Arguments.Count != 0 || node.Object == null || !typeof(LambdaExpression).IsAssignableFrom(node.Object.Type))
return base.VisitMethodCall(node);
var lambda = (LambdaExpression)GetValue(node.Object);
return lambda;
// alternatively recurse on the lambda if it possibly could contain .Compile()s
// return Visit(lambda); // recurse on the lambda
}
}
}
Use LINQKit to expand user-defined lambda functions into the lambdas needed in the query:
https://github.com/scottksmith95/LINQKit
Related
Tried to search a lot for this but couldn't come up with an answer that works. Here's what I am trying to do:
I have Entity ObjectA that has a Navigation property ObjectB (not a Collection type, it's a virtual property that gets loaded via lazy loading). When I do a Where query for A, I want to also expand property B in the expression using another Expression that maps B.
Here's some code to demonstrate, question is in the ToObjectADto() function
public static Expression<Func<ObjectB, ObjectBDto>> ToObjectBDto()
{
return b => new ObjectBDto
{
Prop1 = b.Prop1,
Prop2 = b.Prop2;
};
}
public static Expression<Func<ObjectA, ObjectADto>> ToObjectADto()
{
return a => new ObjectADto
{
Name = a.Name,
SomeProperty = a.SomeProperty,
ObjectB = /* How can I call the ToObjectBDto Expression here without re-writing it? */
};
}
var aDto = _dbContext.ObjectAs.Where(q => q.SomeProperty > 0).Select(ToObjectADto());
I tried to create a compiled Expression:
private static _toBDtoCompiled = ToObjectBDto().Compile();
and then call it in ToObjectADto() like below but I get the API Error There is already an open DataReader associated error because it's doing it client side.
public static Expression<Func<ObjectA, ObjectADto>> ToObjectADto()
{
return a => new ObjectADto
{
Name = a.Name,
SomeProperty = a.SomeProperty,
ObjectB = _toBDto().Invoke(a.ObjectB)
};
}
My recommendation would be to save yourself the work and leverage AutoMapper. The benefit here is that Automapper can feed off EF's IQueryable implementation via ProjectTo to build queries and populate DTO graphs.
var config = new MapperConfiguration(cfg =>
{
cfg.CreateMap<ObjectA, ObjectADto>();
cfg.CreateMap<ObjectB, ObjectBDto>();
});
var aDto = _dbContext.ObjectAs.Where(q => q.SomeProperty > 0).ProjectTo<ObjectADto>(config);
Any particular mapping that cannot be inferred between Object and DTO can be set up in the mapping. The benefit of ProjectTo vs. custom mapping is that it will build a related query without tripping out lazy load hits or risking triggering code that EF cannot translate into SQL. (One query to populate all relevant DTOs)
Automapper can assist with copying values from a DTO back into either a new entity or update an existing entity:
var config = new MapperConfiguration(cfg =>
{
cfg.CreateMap<NewObjectADto, ObjectA>();
cfg.CreateMap<UpdateObjectADto, ObjectA>();
});
var mapper = config.CreateMapper();
New..
var objectA = mapper.Map<ObjectA>(dto);
_dbContext.ObjectAs.Add(objectA);
... or Update existing.
var objectA = _dbContext.ObjectAs.Single(x => x.ObjectAId == dto.ObjectAId);
mapper.Map(objectA, dto);
Where the DTO reflects either the data needed to create a new object, or the data that the client is allowed to update for updating the existing object. The goal being to keep update/add/delete operations as atomic as possible vs. passing large complex objects /w relatives to be updated all at once. I.e. actions like "AddObjectBToA" "RemoveObjectBFromA" etc. rather than resolving all operations via a single "UpdateObjectA".
It's a pity that C# doesn't handle compiling a lambda into an expression, where one expression calls another. Particularly since an expression tree can represent this case. But then EF Core 3 or higher won't look through invoke expressions anyway.
Automapper is probably easier. But if you don't want to use 3rd party code, you'll have to inline the expression yourself. Including replacing any ParameterExpression with the argument to the method.
public static R Invoke<T, R>(this Expression<Func<T, R>> expression, T argument) => throw new NotImplementedException();
// etc for expressions with more parameters
public class InlineVisitor : ExpressionVisitor {
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.Name == "Invoke"
&& node.Object == null
&& node.Arguments.Count >= 1
&& node.Arguments[0] is LambdaExpression expr)
return Visit(
new ReplacingExpressionVisitor(
expr.Parameters.ToArray(),
node.Arguments.Skip(1).ToArray())
.Visit(expr.Body)
);
return base.VisitMethodCall(node);
}
}
// usage;
public static Expression<Func<ObjectA, ObjectADto>> ToObjectADto()
{
var ToBorNotToB = ToObjectBDto();
Expression<Func<ObjectA, ObjectADto>> expr = a => new ObjectADto
{
Name = a.Name,
SomeProperty = a.SomeProperty,
ObjectB = ToBorNotToB.Invoke(a.ObjectB)
};
return new InlineVisitor().VisitAndConvert(expr), "");
}
I would like to selectively ignore a property from a table.
I have an API which exposes the following methods.
public interface IReadService
{
FullDTO Get();
HeaderDTO[] GetList();
}
My data structure looks like so:
public ServiceDTO : ServiceHeaderDTO
{
public string LargeXMLData { get; set; }
}
public ServiceHeaderDTO
{
public int Id { get; set; }
public string Description { get; set; }
//.... Other properties
}
I have a few services which have similar issues, So I would like to be able to ignore the XML property in some cases, so I'm not using extra time to send a large string property which will be ignored.
Normally you might write something like this to hide a property
var entities = context.Services.Select(x =>
new Service { Id = Id, Description = Description, LargeXMLData = "" }).ToArray();
var dtos = this.AdaptToDTO(entities);
Now this would be fine if I had to do this in a single service, but when you have 20 services duplicating the logic it gets annoying.
I would like the be able to just say:
var entities = context.Services.Excluding(x => x.LargeXMLData).ToArray();
var dtos = this.AdaptToHeaderDTO(entities);
Edit: I'm not using automapper. Alot of our code has mappings which cannot translate to expressions. I do not want to have to specify maps
Is there a simple way I can exclude a property from a query? Without having to manually build maps.
Preferably a way which uses the existing mappings internal to EF which maps the entity to the db object
Normally you might write something like this to hide a property
var entities = context.Services.Select(x =>
new Service { Id = Id, Description = Description, LargeXMLData = "" })
If you can do that manually, it should be doable automatically using the exact same concept, with little reflection and Expression APIs.
But note that this woult work only for EF Core, since EF6 does not support projecting to entity types, like new Service { ... } here, and projecting to dynamic types at runtime is not trivial and also will break the DTO mapping.
With that being said, following is a sample implementation of the aforementioned concept:
public static partial class QueryableExtensions
{
public static IQueryable<T> Excluding<T>(this IQueryable<T> source, params Expression<Func<T, object>>[] excludeProperties)
{
var excludeMembers = excludeProperties
.Select(p => ExtractMember(p.Body).Name)
.ToList();
if (excludeMembers.Count == 0) return source;
// Build selector like (T e) => new T { Prop1 = e.Prop1, Prop2 = e.Prop2, ... }
// for each public property of T not included in the excludeMembers list,
// which then will be used as argument for LINQ Select
var parameter = Expression.Parameter(typeof(T), "e");
var bindings = typeof(T).GetProperties()
.Where(p => p.CanWrite && !excludeMembers.Contains(p.Name))
.Select(p => Expression.Bind(p, Expression.MakeMemberAccess(parameter, p)));
var body = Expression.MemberInit(Expression.New(typeof(T)), bindings);
var selector = Expression.Lambda<Func<T, T>>(body, parameter);
return source.Select(selector);
}
static MemberInfo ExtractMember(Expression source)
{
// Remove Convert if present (for value type properties cast to object)
if (source.NodeType == ExpressionType.Convert)
source = ((UnaryExpression)source).Operand;
return ((MemberExpression)source).Member;
}
}
The usage would be exactly as desired:
var entities = context.Services.Excluding(x => x.LargeXMLData).ToArray();
The problem with this though is that it will automatically "include" navigation properties and/or unmapped properties.
So it would be better to use EF model metadata instead of reflection. The problem is that currently EF Core does not provide a good public way of plugging into their infrastructure, or to get access to DbContext (thus Model) from IQueryble, so it has to be passed as argument to the custom method:
public static IQueryable<T> Excluding<T>(this IQueryable<T> source, DbContext context, params Expression<Func<T, object>>[] excludeProperties)
{
var excludeMembers = excludeProperties
.Select(p => ExtractMember(p.Body).Name)
.ToList();
if (excludeMembers.Count == 0) return source;
// Build selector like (T e) => new T { Prop1 = e.Prop1, Prop2 = e.Prop2, ... }
// for each property of T not included in the excludeMembers list,
// which then will be used as argument for LINQ Select
var parameter = Expression.Parameter(typeof(T), "e");
var bindings = context.Model.FindEntityType(typeof(T)).GetProperties()
.Where(p => p.PropertyInfo != null && !excludeMembers.Contains(p.Name))
.Select(p => Expression.Bind(p.PropertyInfo, Expression.MakeMemberAccess(parameter, p.PropertyInfo)));
var body = Expression.MemberInit(Expression.New(typeof(T)), bindings);
var selector = Expression.Lambda<Func<T, T>>(body, parameter);
return source.Select(selector);
}
which makes the usage not so elegant (but doing the job):
var entities = context.Services.Excluding(context, x => x.LargeXMLData).ToArray();
Now the only remaining potential problem are shadow properties, but they cannot be handled with projection, so this technique simply cannot be used for entities with shadow properties.
Finally, the pure EF Core alternative of the above is to put the LargeXMLData into separate single property "entity" and use table splitting to map it to the same table. Then you can use the regular Include method to include it where needed (by default it would be excluded).
I needed to double-check this before answering, but are you using Automapper or some other mapping provider for the ProjectTo implementation? Automapper's ProjectTo extension method requires a mapper configuration, so it may be that your mapping implementation is materializing the entities prematurely.
With Automapper, your example projecting to a DTO that does not contain the large XML field would result in a query to the database that does not return the large XML without needing any new "Exclude" method.
For instance, if I were to use:
var config = new MappingConfiguration<Service, ServiceHeaderDTO>();
var services = context.Services
.ProjectTo<ServiceHeaderDTO>(config)
.ToList();
The resulting SQL would not return the XMLData because ServiceHeaderDTO does not request it.
It is equivalent to doing:
var services = context.Services
.Select(x => new ServiceHeaderDTO
{
ServiceId = x.ServiceId,
// ... remaining fields, excluding the XML Data
}).ToList();
As long as I don't reference x.LargeXMLData, it will not be returned by my resulting query. Where you can run into big data coming back is if something like the following happens behind the scenes:
var services = context.Services
.ToList()
.Select(x => new ServiceHeaderDTO
{
ServiceId = x.ServiceId,
// ... remaining fields, excluding the XML Data
}).ToList();
That extra .ToList() call will materialize the complete Service entity to memory including the XMLData field. Now Automapper's ProjectTo does not work against IEnumerable, only IQueryable so it is unlikely that any query fed to it was doing this, but if you are using a home-grown mapping implementation where ProjectTo is materializing the entities before mapping, then I would strongly recommend using Automapper as it's IQueryable implementation avoids this problem for you automatically.
Edit: Tested with EF Core and Automapper just in case the behaviour changed, but it also excludes anything not referenced in the mapped DTO.
When using LINQ to Entities, is there a way to reuse a block of a select statment in other select stements?
For example, in the below code, I'm using LINQ to select a customer device object from my database. I also select a related Model object from a table that is joined to the customer devices table.
list = context.PTT_CUSTOMER_DEVICES
.Include(...)
.Select(d => new CustomerDevice
{
customerAssetTag = d.CustomerAssetTag,
customerDeviceID = d.CustomerDeviceID,
//This section is used in several LINQ statements throughout the application.
Model = new Model()
{
ModelID = d.PTS_MODELS.ModelID,
Name = d.PTS_MODELS.Name,
Make = new Make()
{
MakeID = d.PTS_MODELS.PTS_MAKES.MakeID,
Name = d.PTS_MODELS.PTS_MAKES.Name
}
}
})...
There are, however, other objects in the database that also reference the Model table. In my select statements for those other tables, I basically copied the same Model = new Model() code to a select statement for those different tables.
What I'm wondering is, is it possible to store and reuse that block of code in multiple selects? Possibly using an extension method?
As an alternative to Func<>s, you can also make use of extension methods to reuse the conversions between your entity types to and other POCOs.
public static IQueryable<CustomerDevice> ToCustomerDevice(this IQueryable<PTT_CUSTOMER_DEVICES> devices)
{
return devices.Select(d => new CustomerDevice
{
customerAssetTag = d.CustomerAssetTag,
customerDeviceID = d.CustomerDeviceID
}
}
However, EF will not allow you to nest these, and will complain that it cannot convert the nested extension method to SQL.
A work around for this can be to perform the transform in memory, rather than in SQL:
public static Model ToModel(this PTS_MODELS model)
{
return new Model()
{
ModelID = model.ModelID,
Name = model.Name,
Make = new Make()
{
MakeID = model.PTS_MAKES.MakeID,
Name = model.PTS_MAKES.Name
}
};
}
public static IEnumerable<CustomerDevice> ToCustomerDevice(this IQueryable<PTT_CUSTOMER_DEVICES> devices)
{
return devices
.Include(d => d.PTS_MODELS.PTS_MAKES)
.AsEnumerable() // Evaulate everything that follows in memory
.Select(d => new CustomerDevice
{
customerAssetTag = d.CustomerAssetTag,
customerDeviceID = d.CustomerDeviceID,
Model = d.PTS_MODELS.ToModel()
});
}
Since you are now returning an IEnumerable<> any further Where() conditions will be evaluated in memory, rather than in SQL, so its important that ToCustomerDevice() is your last call.
If those tables have a common base class or implement the same interface you can do it.
To make it simple assume I have a IQueryable<string> names. I can sort it like this:
IQueryable<string> sorted = names.OrderBy(name => name);
But if I want to keep this particular sort to use later I can do this:
Func<IQueryable<string>, IQueryable<string>> orderBy = q => q.OrderBy(x => x);
Now to call it I just pass any IQueryable<string> in:
IQueryable<string> sorted = orderBy(names);
You would need to create a constructor for Model that takes a CustomerDevice? (what ever the entity is for PTT_CUSTOMER_DEVICES:
public Model(CustomerDevice d) {
ModelID = d.PTS_MODELS.ModelID;
Name = d.PTS_MODELS.Name;
Make = new Make() {
MakeID = d.PTS_MODELS.PTS_MAKES.MakeID,
Name = d.PTS_MODELS.PTS_MAKES.Name
};
}
Then you could just call that constructor in the LINQ:
Model = new Model(d),
You can use Expression:
public static Expression<Func<PTT_CUSTOMER_DEVICES, CustomerDevice>>
CustomerDeviceExpression = d =>
new CustomerDevice
{
customerAssetTag = d.CustomerAssetTag,
customerDeviceID = d.CustomerDeviceID,
Model = new Model()
{
ModelID = d.PTS_MODELS.ModelID,
Name = d.PTS_MODELS.Name,
Make = new Make()
{
MakeID = d.PTS_MODELS.PTS_MAKES.MakeID,
Name = d.PTS_MODELS.PTS_MAKES.Name
}
}
};
Then:
list = context.PTT_CUSTOMER_DEVICES
.Include(...)
.Select(CustomerDeviceExpression)...
I have recently moved from coding in Java to c# and I am still learning the various elements of c#.
To access an existing database, which I cannot redesign, I am using Entity Frameworks 6 and 'Code First from database' to generate contexts and types representing the database tables. I am using Ling-To-SQL to retrieve the data from the database which is heavily denormalized.
My current task is create a report where each section is read from various tables, which all have a relationship to one base table.
This is my working example:
using(var db = new PaymentContext())
{
var out = from pay in db.Payment
join typ in db.Type on new { pay.ID, pay.TypeID } equals
new { typ.ID, typ.TypeID }
join base in db.BaseTable on
new { pay.Key1, pay.Key2, pay.Key3, pay.Key4, pay.Key5 } equals
new { base.Key1, base.Key2, base.Key3, base.Key4, base.Key5 }
where
base.Cancelled.Equals("0") &&
base.TimeStamp.CompareTo(startTime) > 0 &&
base.TimeStamp.CompareTo(endTime) < 1 &&
.
(other conditions)
.
group new { pay, typ } by new { typ.PaymentType } into grp
select new
{
name = grp.Key,
count = grp.Count(),
total = grp.Sum(x => x.pay.Amount)
};
}
There will be a large number of sections in the report and each section will generate a where clause which will contain the conditions shown. In some sections, the required data will be extracted from tables up to five levels below the BaseTable.
What I want to do is create a resuable where clause for each report section, to avoid a lot of duplicated code.
After a lot of searching, I tried to use the solution suggested here , but this has been superseded in Entity Framework 6.
How do I avoid duplicating code unnecessarily?
I did try to use the extension clauses you suggested, but my generated classes do not extend the BaseTable, so I had to explicitly define the link through the navigation property. As only a small number of tables will be common in the queries, I decided to apply the filters directly to each table as required. I will define these as required.
krillgar suggested moving to straight LINQ syntax, which seems like good advice. We intend to redesign our database in the near future and this will remove some of the SQL dependency. I merged the suggested filters and full LINQ syntax to access my data.
// A class to hold all the possible conditions applied for the report
// Can be applied at various levels within the select
public class WhereConditions
{
public string CancelledFlag { get; set; } = "0"; // <= default setting
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
}
// Class to define all the filters to be applied to any level of table
public static class QueryExtensions
{
public static IQueryable<BaseTable> ApplyCancellationFilter(this IQueryable<BaseTable> base, WhereConditions clause)
{
return base.Where(bse => bse.CancelFlag.Equals(clause.CancelledFlag));
}
public static IQueryable<BaseTable> ApplyTimeFilter(this IQueryable<BaseTable> base, WhereConditions clause)
{
return base.Where(bse => bse.TimeStamp.CompareTo(clause.StartTime) > 0 &&
bse.TimeStamp.CompareTo(clause.EndTime) < 1);
}
}
And the query is composed as follows:
using (var db = new PaymentContext())
{
IEnumerable<BaseTable> filter = db.BaseTable.ApplyCancellationFilter(clause).ApplyTimeFilter(clause);
var result = db.Payment.
Join(
filter,
pay => new { pay.Key1, pay.Key2, pay.Key3, pay.Key4, pay.Key5 },
bse => new { bse.Key1, bse.Key2, bse.Key3, bse.Key4, bse.Key5 },
(pay, bse) => new { Payment = pay, BaseTable = bse }).
Join(
db.Type,
pay => new { pay.Payment.TypeKey1, pay.Payment.TypeKey2 },
typ => new { typ.TypeKey1, typ.TypeKey2 },
(pay, typ) => new { name = typ.Description, amount = pay.Amount }).
GroupBy(x => x.name).
Select(y => new { name = y.Key,
count = y.Count(),
amount = y.Sum(z => z.amount)});
}
And then to finally execute composed query.
var reportDetail = result.ToArray(); // <= Access database here
As this query is the simplest I will have to apply, future queries will become much more complicated.
The nice thing about LINQ is that methods like Where() return an IEnumerable<T> that you can feed into the next method.
You could refactor the where clauses into extension methods akin to:
public static class PaymentQueryExtensions {
public static IQueryable<T> ApplyNotCancelledFilter(
this IQueryable<T> payments)
where T : BaseTable {
// no explicit 'join' needed to access properties of base class in EF Model
return payments.Where(p => p.Cancelled.Equals("0"));
}
public static IQueryable<T> ApplyTimeFilter(
this IQueryable<T> payments, DateTime startTime, DateTime endTime)
where T: BaseTable {
return payments.Where(p => p.TimeStamp.CompareTo(startTime) > 0
&& p.TimeStamp.CompareTo(endTime) < 1);
}
public static IGrouping<Typ, T> GroupByType(
this IQueryable<T> payments)
where T: BaseTable {
// assuming the relationship Payment -> Typ has been set up with a backlink property Payment.Typ
// e.g. for EF fluent API:
// ModelBuilder.Entity<Typ>().HasMany(t => t.Payment).WithRequired(p => p.Typ);
return payments.GroupBy(p => p.Typ);
}
}
And then compose your queries using these building blocks:
IEnumerable<Payment> payments = db.Payment
.ApplyNotCancelledFilter()
.ApplyTimeFilter(startTime, endTime);
if (renderSectionOne) {
payments = payments.ApplySectionOneFilter();
}
var paymentsByType = payments.GroupByType();
var result = paymentsByType.Select(new
{
name = grp.Key,
count = grp.Count(),
total = grp.Sum(x => x.pay.Amount)
}
);
Now that you have composed the query, execute it by enumerating. No DB access has happened until now.
var output = result.ToArray(); // <- DB access happens here
Edit After the suggestion of Ivan, I looked at our codebase. As he mentioned, the Extension methods should work on IQueryable instead of IEnumerable. Just take care that you only use expressions that can be translated to SQL, i.e. do not call any custom code like an overriden ToString() method.
Edit 2 If Payment and other model classes inherit BaseTable, the filter methods can be written as generic methods that accept any child type of BaseTable. Also added example for grouping method.
I face a problem, which is new to me
I have the following entity (I use the fluent nhibernate but it doesn't matter here)
public class SomeEntity
{
public virtual string Name { get; set; }
}
filter classes:
public class FilterOptions
{
public string logic { get; set; } // "and", "or"
public FilterItems[] filters { get; set; }
}
public class FilterItems
{
public string #operator { get; set; }
public string value { get; set; } //string value provided by user
}
the #operator property can have the following values
EndsWith
DoesNotContain
Contains
StartsWith
NotEqual
IsEqualTo
all I wanna do is to do some filter operations basing on the 2 filters:
private IQueryable<SomeEntity> BuildQuery(FilterOptions opts)
{
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
var firstFilter = opts.filters[0];
var secondFilter = opts.filters[1];
}
because the fact the #operator property can have so many options, I wonder if there's a posibility to have the external method with the swich operator, and use that method inside of the .Where method.
Something like
var query = query.Where(firstSwitchFilterMethod && secondFilterMethod)
pseudo code:
firstSwitchFilterMethod:
if (firstFilter.#operator == "Contains")
return SomeEntity.Name.Contains(firstFilter.value);
and so on...
Any ideas ? I'm thinking about the Expression<Func<>> - is it good direction ? If so, how to use it in my case ?
Or, maybe build my own the extension method for SomeEntity which will be using that filter class ?
You can not make arbitrary function call expression and expect that it can be translated into SQL. But there are some functions, like StartsWith, that can. Here is one example, how You can build Your own expression:
protected IQueryable<T> GetFiltered<T>(IQueryable<T> query, string filterOnProperty, string startsWithString, string endsWithString)
{
LambdaExpression startsWithLambda = (Expression<Func<string, string, bool>>)((x, s) => x.StartsWith(s));
MethodInfo startsWithMI = (startsWithLambda.Body as MethodCallExpression).Method;
LambdaExpression endsWithLambda = (Expression<Func<string, string, bool>>)((x, s) => x.EndsWith(s));
MethodInfo endsWithMI = (endsWithLambda.Body as MethodCallExpression).Method;
ParameterExpression param = Expression.Parameter(typeof(T));
Expression nameProp = Expression.Property(param, filterOnProperty);
Expression filteredOk = Expression.Constant(true);
Expression startsWithStringExpr = Expression.Constant(startsWithString);
Expression startsWithCondition = Expression.Call(nameProp, startsWithMI, startsWithStringExpr);
filteredOk = Expression.AndAlso(filteredOk, startsWithCondition);
Expression endsWithStringExpr = Expression.Constant(endsWithString);
Expression endsWithCondition = Expression.Call(nameProp, endsWithMI, endsWithStringExpr);
filteredOk = Expression.AndAlso(filteredOk, endsWithCondition);
Expression<Func<T, bool>> where = Expression.Lambda<Func<T, bool>>(filteredOk, new ParameterExpression[] { param });
return query.Where(where);
}
usage is simple
DCDataContext dc = new DCDataContext();
var query = dc.testtables.AsQueryable();
query = GetFiltered(query, "name", "aaa", "2");
NHibernate or any other LINQ provider doesn't really care how you build the expression - the only thing that matters is that the final expression only contain constructs that the LINQ provider understands and knows what to do with.
Yes, you can have a method return an Expression<Func<>> and then add that expression to the LINQ query using the Where() method.
However, you cannot ask NHibernate to analyze your compiled code and convert the if statement to SQL. You method will need to analyze the options and return a suitable expression that gets inserted into the full LINQ query.
You can write it in a single method like this:
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
if (isEqOp)
query = query.Where(e => e.Name == options.Value)
if (isContainsOp)
query = query.Where(e => e.Name.Contains(options.Value))
query.ToList();
Or if you want the filter logic in a separate method as you said you can also split it like this:
public Expression<Func<Entity, bool>> GetExpression(options)
{
if (isEqOp)
return (Entity e) => e.Name == options.Value;
if (isContainsOp)
return (Entity e) => e.Name.Contains(options.Value);
}
{
IQueryable<SomeEntity> query = Session.Query<SomeEntity>();
query = query.Where(GetExpression(options))
...
}