How to dynamically create the below linq expression.
IQueryable abc = QueryData.Select(a => new { a, TempData = a.customer.Select(b => b.OtherAddress).ToList()[0] }).OrderBy(a => a.TempData).Select(a => a.a);
public class Orders
{
public long OrderID { get; set; }
public string CustomerID { get; set; }
public int EmployeeID { get; set; }
public double Freight { get; set; }
public string ShipCountry { get; set; }
public string ShipCity { get; set; }
public Customer[] customer {get; set;}
}
public class Customer
{
public string OtherAddress { get; set; }
public int CustNum { get; set; }
}
Actual data:
List<Orders> order = new List<Orders>();
Customer[] cs = { new Customer { CustNum = 5, OtherAddress = "Hello" }, new
Customer { CustNum = 986, OtherAddress = "Other" } };
Customer[] cso = { new Customer { OtherAddress = "T", CustNum = 5 }, new
Customer { CustNum = 777, OtherAddress = "other" } };
order.Add(new Orders(code + 1, "ALFKI", i + 0, 2.3 * i, "Mumbari", "Berlin", cs));
order.Add(new Orders(code + 2, "ANATR", i + 2, 3.3 * i, "Sydney", "Madrid", cso));
order.Add(new Orders(code + 3, "ANTON", i + 1, 4.3 * i, "NY", "Cholchester", cs));
order.Add(new Orders(code + 4, "BLONP", i + 3, 5.3 * i, "LA", "Marseille", cso));
order.Add(new Orders(code + 5, "BOLID", i + 4, 6.3 * i, "Cochin", "Tsawassen", cs));
public Orders(long OrderId, string CustomerId, int EmployeeId, double Freight, string ShipCountry, string ShipCity, Customer[] Customer = null)
{
this.OrderID = OrderId;
this.CustomerID = CustomerId;
this.EmployeeID = EmployeeId;
this.Freight = Freight;
this.ShipCountry = ShipCountry;
this.ShipCity = ShipCity;
this.customer = Customer;
}
If i sort the OtherAddress field 0th index means Customer field only sorted. I need to sort the whole order data based on OtherAddress field.
I have tried the below way:
private static IQueryable PerformComplexDataOperation<T>(this IQueryable<T> dataSource, string select)
{
string[] selectArr = select.Split('.');
ParameterExpression param = Expression.Parameter(typeof(T), "a");
Expression property = param;
for (int i = 0; i < selectArr.Length; i++)
{
int n;
if (int.TryParse(selectArr[i + 1], out n))
{
int index = Convert.ToInt16(selectArr[i + 1]);
property = Expression.PropertyOrField(Expression.ArrayIndex(Expression.PropertyOrField(property, selectArr[i]), Expression.Constant(index)), selectArr[i + 2]);
i = i + 2;
}
else property = Expression.PropertyOrField(property, selectArr[i]);
}
var TempData = dataSource.Select(Expression.Lambda<Func<T, object>>(property, param));
IQueryable<object> data = dataSource.Select(a => new { a, TempData = property});// Expression.Lambda<Func<T, object>>(property, param) });
return data;
}
Method call : PerformComplexDataOperation(datasource, "customer.0.OtherAddress")
I can get the value from this line : var TempData = dataSource.Select(Expression.Lambda>(property, param));
But i can't get the values in dataSource.Select(a => new { a, TempData = property});
It is working when we use the below code :
var TempData = dataSource.Select(Expression.Lambda<Func<T, object>>(property, param)).ToList();
IQueryable<object> data = dataSource.Select((a, i) => new { a, TempData = TempData[i] });
Is it proper solution ?
XY problem?
This feels like it's a case of the XY problem. Your solution is contrived (no offense intended), and the problem you're trying to solve is not apparent by observing your proposed solution.
However, I do think there is technical merit to your question when I read the intention of your code as opposed to your described intention.
Redundant steps
IQueryable abc = QueryData
.Select(a => new {
a,
TempData = a.customer.Select(b => b.OtherAddress).ToList()[0] })
.OrderBy(a => a.TempData)
.Select(a => a.a);
First of all, when you inline this into a single chained command, TempData becomes a redundant step. You could simply shift the first TempData logic (from the first Select) directly into the OrderBy lambda:
IQueryable abc = QueryData
.OrderBy(a => a.customer.Select(b => b.OtherAddress).ToList()[0])
.AsQueryable();
As you can see, this also means that you no longer need the second Select (since it existed only to undo the earlier Select)
Parametrization and method abstraction
You mentioned you're looking for a usage similar to:
PerformComplexDataOperation(datasource, "customer.0.OtherAddress")
However, this doesn't quite make sense, since you've defined an extension method:
private static IQueryable PerformComplexDataOperation<T>(this IQueryable<T> dataSource, string select)
I think you need to reconsider your intended usage, and also the method as it is currently defined.
Minor note, the return type of the method should be IQueryable<T> instead of IQueryable. Otherwise, you lose the generic type definition that LINQ tends to rely on.
Based on the method signature, your expected usage should be myData = myData.PerformComplexDataOperation("customer.0.OtherAddress").
Strings are easy hacks to allow you to circumvent an otherwise strongly typed system. While your strign usage is technically functional, it is non-idiomatic and it opens the door to unreadable and/or bad code.
Using strings leads to a contrived string parsing logic. Look at your method definition, and count how many lines are there simply to parse the string and translate that into actual code again.
Strings also mean that you get no Intellisense, which can cause unseen bugs further down the line.
So let's not use strings. Let's look back at how I initially rewrote the `OrderBy:
.OrderBy(a => a.customer.Select(b => b.OtherAddress).ToList()[0])
When you consider OrderBy as an ordinary method, no different from any custom method you and I can develop, then you should understand that a => a.customer.Select(b => b.OtherAddress).ToList()[0] is nothing more than a parameter that's being passed.
The type of this parameter is Func<A,B>, where:
A equals the type of your entity. So in this case, A is the same as T in your existing method.
B equals the type of your sorting value.
OrderBy(x => x.MyIntProp) means that B is of type int.
OrderBy(x => x.MyStringProp) means that B is of type string.
OrderBy(x => x.Customer) means that B is of type Customer.
Generally speaking, the type of B doesn't matter for you (since it will only be used by LINQ's internal ordering method).
Let's look at a very simple extension method that uses a parameter for its OrderBy:
public static IQueryable<A> OrderData<A, B>(this IQueryable<A> data, Func<A, B> orderbyClause)
{
return data
.OrderBy(orderbyClause)
.AsQueryable();
}
Using the method looks like:
IQueryable<MyEntity> myData = GetData(); //assume this returns a correct value
myData = myData.OrderData(x => x.MyIntProperty);
Notice how I did not need to specify either of the generic type arguments when calling the method.
A is already known to be MyEntity, because we're calling the method on an object of type IQueryable<MyEntity>.
B is already known to be an int, since the used lambda method returns a value of type int (from MyIntProperty)
As it stands, my example method is just a boring wrapper that does nothing different from the existing OrderBy method. But you can change the method's logic to suit your needs, and actually make it meaningfully different from the existing OrderBy method.
Your expectations
Your description of your goals makes me think that you're expecting too much.
I need to sort "customer.0.OtherAddress" nested file compared to whole base data. But it sorted only for that field. For this case, I find that field value and stored it to TempData. Then Sorting the TempData field.
i need to sort the parent nodes not an sibling alone. QueryData.Select(a => new { a, TempData = a.customer.Select(b => b.OtherAddress).ToList()[0] }).OrderBy(a => a.TempData).Select(a => a.a); I sorting a original data based on temp data. Then i split the original data alone.
It's not possible to sort an entire nested data structure based on a single OrderBy call. OrderBy only sorts the collection on which you call Orderby, nothing else.
If you have a list of Customer entities, who each have a list of Adress entities, then you are working with many lists (a list of customer and several lists of adresses). OrderBy will only sort the list that you ask it to sort, it will not look for any nested lists.
You mention that your TempData solution works. I actually wrote an entire answer contesting that notion (it should be functionally similar to my suggested alternatives, and it should always order the original list, not any nested list), until I noticed that you've made it work for a very insidious and non-obvious reason:
.Select(a => new {
a,
TempData = a.customer.Select(b => b.OtherAddress).ToList()[0]
})
You are calling .ToList(), which changes how the code behaves. You started off with an IQueryable<>, which means that LINQ was preparing an SQL command to retrieve the data when you enumerate it.
This is the goal of an IQueryable<>. Instead of pulling all the data into memory and then filtering it according to your specifications, it instead constructs a complex SQL query, and will then only need to execute a single (constructed) query.
The execution of that constructed query occurs when you try to access the data (obviously, the data needs to be fetched if you want to access it). A common method of doing so is by enumerating the IQueryable<> into an IEnumerable<>.
This is what you've done in the Select lambda. Instead of asking LINQ to enumerate your list of orders, you've asked it to enumerate every list of addresses from every customer from every order in the list of orders.
But in order to know which adresses need to be enumerated, LINQ must first know which customers it's supposed to get the adresses from. And to find out which customers it needs, it must first figure out which orders you're working with. The only way it can figure all of that out is by enumerating everything.
My initial suggestion, that you should avoid using the TempData solution, is still valid. It's a redundant step that serves no functional purpose. However, the enumeration that also takes place may actually be of use to you here, because it changes LINQ's behavior slightly. You claim that it fixes your problem, so I'm going to take your statement at face value and assume that the slightly different behavior between LINQ-to-SQL and LINQ-to-Entities solves your problem.
You can keep the enumeration and still omit the TempData workaround:
IQueryable abc = QueryData
.OrderBy(a => a.customer.Select(b => b.OtherAddress).ToList()[0])
.AsEnumerable()
.AsQueryable();
Some footnotes:
You can use ToList() instead of AsEnumerable(), the result is the same.
When you use First() or Single(), enumeration will inherently take place, so you don't need to call AsEnumerable() beforehand.
Notice that I cast the result to an IEnumerable<>, but then I immediately re-cast it to IQueryable<>. Once a collection has been enumerated, any further operation on it will occur in-memory. Casting it back to an IQueryable<> does not change the fact that the collection has already been enumerated.
But does it work?
Now, I think that this still doesn't sort all of your nested lists with a single call. However, you claim it does. If you still believe that it does, then you don't need to read on (because your problem is solved). Otherwise, the following may be useful to you.
SQL, and by extension LINQ, has made it possible to sort a list based on information that is not found in the list. This is essentially what you're doing, you're asking LINQ to sort a list of orders based on a related address (regardless of whether you want the adresses to be retrieved from the database or not!) You're not asking it to sort the customers, or the addresses. You're only asking it to sort the orders.
Your sort logic feels a bit dirty to me. You are supplying an Address entity to your OrderBy method, without specifiying any of its (value type) properties. But how are you expecting your addresses to be sorted? By alphabetical street name? By database id? ...
I would expect you to be more explicit about what you want, e.g. OrderBy(x => x.Address.Street).ThenBy(x => x.Address.HouseNumber) (this is a simplified example).
After enumeration, since all the (relevant) data is in-memory, you can start ordering all the nested lists. For example:
foreach(var order in myOrders)
{
order.Customer.Addresses = order.Customer.Addresses.OrderBy(x => x.Street).ToList();
}
This orders all the lists of addresses. It does not change the order of the list of orders.
Do keep in mind that if you want to order data in-memory, that you do in fact need the data to be present in-memory. If you never loaded the customer's addresses, you can't use addresses as a sorting argument.
Ordering the list of orders should be done before enumeration. It's generally faster to have it handled by your SQL database, which is what happens when you're working with LINQ-to-SQL.
Ordering nested lists should be done after enumeration, because the order of these lists is unrelated to the original IQueryable<Order>, which only focused on sorting the orders, not its nested related entities (during enumeration, the included entities such as Customer and Address are retrieved without ordering them).
You can transform your OrderBy so you don't need an anonymous type (though I like the Perl/Lisp Schwartzian Transform) and then it is straightforward to create dynamically (though I am not sure how dynamically you mean).
Using the new expression:
var abc = QueryData.OrderBy(a => a.customer[0].OtherAddress);
Not being sure what you mean by dynamic, you can create the lambda
x => x.OrderBy(a => a.customer[0].Otheraddress)
using Expression as follows:
var parmx = Expression.Parameter(QueryData.GetType(), "x");
var parma = Expression.Parameter(QueryData[0].GetType(), "a");
var abc2 = Expression.Lambda(Expression.Call(MyExtensions.GetMethodInfo((IEnumerable<Orders> x)=>x.OrderBy(a => a.customer[0].OtherAddress)),
new Expression[] { parmx,
Expression.Lambda(Expression.Property(Expression.ArrayIndex(Expression.Property(parma, "customer"), Expression.Constant(0)), "OtherAddress"), parma) }),
parmx);
Before try to use a variable (this has sql server do the calculation):
IQueryable<MyEntity> query = _dbSet; // IQueryable<TEntity>
var results = query.Select(m => new MyViewModel
{
MyCalculation = m.Column1 * m.Column2
}).ToList();
What I want to do (dynamically create part of my select statement from a Func variable or some other kind of variable to allow this):
IQueryable<MyEntity> query = _dbSet; // IQueryable<TEntity>
Func<MyEntity, decimal> funcVariableAttempt = m => m.Column1 * m.Column2;
var results = query.Select(m => new MyViewModel
{
MyCalculation = funcVariableAttempt.Invoke(m) // My foolish attempt does not work.
}).ToList();
The error I get when I try what I want (aka my foolish attempt):
LINQ to Entities does not recognize the method 'System.Decimal Invoke(MyProject.Repository.Models.MyEntity)' method, and this method cannot be translated into a store expression.
How do I define and utilize a variable (maybe a Func variable) to define part of a Select statement?
It's a completely valid question that I stumbeled across earlier and found a solution that works well for me.
The problem with your Func<MyEntity, decimal> is that it is a delegate and that the O/R mapper has to have access to the internal expression (the multiplication of two properties in your case). But this information is compiled into the delegate and hidden forever.
If you however start off with a Expression<Func<MyEntity, decimal>> customCalculation, things look more promising as you have the internal logic as an expression tree.
Problem with this is: you cannot invoke an expression. customCalculation(m) doesn't compile.
The compiler would let you write
m => new MyViewModel { MyCalculation = customCalculation.Compile()(m) }
, but this wouldn't be understood by most O/R mappers.
But you see that it at least somehow contains the customCalculation lambda expression and also how it relates to its surrounding expressions.
Getting from here to the expression tree as in your original working version involves some expression manipulation:
We have to replace customCalculation.Compile()(m) with the body of the lambda that is to be Compile()d, but with the lambda's parameter(s) replaced with the respective expression(s) of the delegate invocation. So if customCalculation were x => x.Column1 * x.Column2, customCalculation.Compile()(m) would have to be replaced with m.Column1 * m.Column2
Doing so is not trivial, since the lambda itself has to be dug out of a field inside an instance of a compiler generated closure class.
I've posted my implementation of this expression manipulator in another similar question . Hope that helps.
With that, you should be able to:
var customCalculation = (Expression<Func<MyEntity, decimal>>)(x => x.Column1 * x.Column2);
var selector = Express.Prepare((Expression<Func<MyEntity, MyViewModel>>)(m => new MyViewModel { MyCalculation = customCalculation.Compile()(m) }));
var result = query.Select(selector).ToList();
As you already know, your funcVariableAttempt makes no sense to your database, so you have to call your method in the linq-to-object context. i.e. for instance first fetch the data as an Enumerable, then call your method:
var results = query.Select(m => new {
Column1= col1,
Column2= col2
}).AsEnumerable()
.Select(m => new MyViewModel
{
MyCalculation = Foo(m.Column1, m.Column2)
});
*Note: Code is not tested.
You should call ToList() first and perform the Select() on the result in memory.
var results = query.ToList()
.Select(m => new MyViewModel {
MyCalculation = Foo(m.Column1, m.Column2)
});
You're trying to perform the Select as part of the query. You should just use a regular function for the mapping calculation. Lambda functions are useful with LINQ but in this case they're not needed.
Origin
I have a LINQ-to-SQL query that will vary slightly based on external criteria. I wrote a switch statement for the three different cases, but the series of queries in the different cases are nearly identical. The only thing that differs is the delegate used for the join condition.
So, in order to make my code DRYer, I decided to just use the switch statement to select from three different delegates. I would then save the chosen delagate in a variable, and use that variable in the .Join call.
Problem
However, when I use the variable instead of explicitly typing out the delegate, the .Join no longer returns an IQueryable but instead returns an IEnumerable. I should clarify that I'm getting the return value from Intellisense on the 'var' keyword.
So this one returns an IQueryable:
var dc = new CloudDataContext();
var manifests = dc.ShippingManifests.Join(
dc.Locations,
man => man.OriginId,
loc => loc.Id,
(man, loc) => man
);
But this one returns an IEnumerable:
protected static int? ManifestOriginId( ShippingManifest manifest ) {
return manifest.OriginId;
}
Func<ShippingManifest, int?> originJoiner = GridModelManifests.ManifestOriginId;
var dc = new CloudDataContext();
var manifests = dc.ShippingManifests.Join(
dc.Locations,
originJoiner,
loc => loc.Id,
(man, loc) => man
);
Question
I'm curious why this happens, but I'm more curious how I can accomplish what I'm trying to accomplish. I need to get an IQueryable, as the resulting query is going to be passed on down the line for further manipulation.
Try it as an Expression.
When it isn't, the query will probably be hydrated and pulled into memory prior to the join. What you're after is an Expression<Func<ShippingManifest, int?>>:
Expression<Func<ShippingManifest, int?>> originJoiner = man => man.OriginId;
This allows the expression visitor in Linq to SQL to continue processing it before it sends it off to the database.
I have a function like that :
public int? calculateContractPrice(int? comid)
{
int? sum = 0;
var q = from i in dbconnect.tblMaterialGroups
where i.tenderId == _tenderId
select i.id;
foreach (int i in q )
{
var q2 = from g in dbconnect.tblMaterialTenderAnnouncePrices
where g.MaterialGroupId == i && g.companyId == comid
select g;
sum = q2.First().amount*q2.First().price + q2.First().amount*q2.First().PriceForElse + sum;
}
return sum ;
}
When i try to execute this :
List<presentationcontract> q = (from i in dbconnect.tblContracts
where i.tender == _tenderId
select new presentationcontract()
{
tax =(calculateContractPrice(i.companyId)*(6/100)).ToString()
}).ToList();
Tax is string .after executing i got this error :
couldn't translate expression calculateContractPrice(i.companyId)*(6/100),invoke(value(system.Func1[system.nullable1[system.Int32]]))).ToString() into SQL and could not treat it as a local expression
Your edit makes clear the issue. You're trying to do
tax =(calculateContractPrice(i.companyId)*(6/100)).ToString()
in a sql statement but calculateContractPrice is in c#! To understand what's going on you really need to understand a bit how LINQ works.
First of all, stop using the silly sql-style syntax for LINQ. It is less powerful than the lambda syntax and hides what is going on under the hood in a way that makes it hard to understand.
Second consider a LINQ statement
users.Where(u => u.Name == "George").ToList();
where users is IEnumerable<User>. What happens here is that the lambda part is of type Func<User, bool> and gets compiled to a method that gets run against every single instance of User.
Now consider this LINQ statement
db.Users.Where(u => u.Name == "George").ToList();
where db.Users is IQueryable<T>. This looks the same but what happens is VERY different. What happens is that lambda is actually of type Expression<Func<User, bool>> this doesn't get compiled to a method, instead it gets compiled into something called an expression tree. This gets passed to the LINQ provider (in your case Entity Framework I'm guessing) which examines it and converts that into a SQL statement
SELECT Id, Name, Address FROM users WHERE Name = 'George'
What is happening in your case is that it sees the call to calculateContractPrice and simply has no way of converting that to SQL.
What you should therefore do is ensure the query runs first, then use the IEnumerable<T> form of LINQ that runs in c# to call your method.
var contracts = dbconnect.tblContracts.Where(i => i.tender == _tenderId)
.ToList() //Query executes here, now you have IEnumerable<T>
.Select(i => new PresentationContract {
Tax = ...
}).ToList(); //this ToList is only necessary if you want to prevent multiple iteration
You will want to solve all the other problems everyone else pointed out as well of course.
A few other notes - you will want to read up on .Net naming conventions. Usually, anything public,protected, or internal (classes, fields, properties, etc.) is recommended to be PascalCase. Also you probably want to move the division portion into the PresentationContract class. This class that has a Tax property should probably be the one that knows how to generate it.
Try this:
int? ret = calculateContractPrice(i.companyId);
if(ret.HasValue)
{
tax =(ret.Value*(6/100)).ToString();
}
You should make sure that the function indeed returned a value and then you use that integer value in calculation.
I would like to create a compiled query which uses reusable where predicates. An example to make this clear:
ObjectContext.Employees.Where(EmployeePredicates.CustomerPredicate)
EmployeePredicates is a static class with a property CustomerPredicate that looks like this:
public static Expression<Func<Employee, bool>> CustomerPredicate
{
get
{
return t => t.CustomerId == 1;
}
}
This works as expected.
In most cases however, you would like to pass a parameter to Expression. To achieve this I have to change the property to a static function:
public static Expression<Func<Employee, bool>> CustomerPredicate(int id)
{
return t => t.CustomerId == id;
}
And I can use this like this:
ObjectContext.Employees.Where(EmployeePredicates.CustomerPredicate(id))
This works, but now comes the tricky part. I would like to compile this query... Visual studio doesn't give me any compile errors, but when I run this example the following exception is thrown at runtime:
Internal .NET Framework Data Provider error 1025
Just so we're on the same page here is the full code that gives me the exception:
var _compiledQuery = CompiledQuery.Compile<AdventureWorksEntities, int, IQueryable<Employee>>(
(ctx, id) =>
(ctx.Employee.Where(EmployeePredicates.CustomerPredicate(id))
));
Does anyone have a clue why this exception is being thrown? I've taken this way of working from http://www.albahari.com/nutshell/linqkit.aspx. Any help would be much appreciated.
Thanks Jon,
The problem with the approach you describe, is that chaining predicates together will become very hard. What if I need to combine this predicate with another predicate that filters the employees with a certain name? Then you end up with a lot of selects to pass in the parameters.
(ctx, id, name) =>
(ctx.Employee.Select(emp => new {emp, id})
.Where(EmployeePredicates.CustomerPredicate(id))
.Select(emp => new {emp, name})
.Where(EmployeePredicates.NamePredicate(name))
It gets even worse when you're working on joined tables.
(ctx, id, name) =>
(ctx.Employee
.Join(ctx.Contact, e=> e.ContactId, c => c.Id), (emp, cont) => new Container<Employee, Customer> {Employee = emp, Contact = cont})
.Where(EmployeePredicates.CustomerPredicate(id))
.Where(EmployeePredicates.NamePredicate(name))
.Select(t => new EmployeeDTO {Name = t.cont.Name, Customer = e.emp.Customer})
Because each Where() operates on something of type T and returns something of type T, the WherePredicates in the code above must work on the type Container. This makes it very hard to reuse the Predicates. And reuse was the initial goal of this approach...
The problem is that the Entity Framework is trying to examine the expression tree represented by
(ctx, id) => (ctx.Employee.Where(EmployeePredicates.CustomerPredicate(id))
It can't do that, because it doesn't know what EmployeePredicates.CustomerPredicate does.
As for the best fix... I'm not sure. Basically it's got to know at query compile time what the full query looks like, just with the placeholders for parameters.
I suspect the best solution will involve something like this:
public static Expression<Func<Employee, int, bool>> CustomerPredicate()
{
return (t, id) => t.CustomerId == id;
}
... as that raises the abstraction by one level; it gives you an expression tree which uses id as a ParameterExpression, which is something you'll need in order to build the appropriate expression tree to call CompileQuery. It gets a little hard to think about, unfortunately :(