I have an app written with C#. I have a collection of objects and I need to perform tasks like sorting and jumping to a page of the objects. Currently, I have something that looks like this:
this.Items = model.Items
.Where(x => x.Id > 0)
.ToList();
if (SortBy == "Name")
{
if (Direction == "Descending")
this.Items = (this.Items.OrderByDescending(x => x.Name)).ToList();
else
this.Items = (this.Items.OrderBy(x => x.Name)).ToList();
}
else if (SortBy == "Description")
{
if (Direction == "Descending")
this.Items = (this.Items.OrderByDescending(x => x.Description)).ToList();
else
this.Items = (this.Items.OrderBy(x => x.Description)).ToList();
}
// I want to jump to the second page
this.Items = this.Items.Skip(26).ToList();
Is there a better way to do this? All of the ToList calls seem like that would be performance draining. Maybe I'm wrong though.
Thanks!
You can defer calling .ToList until your last line since based on your logic it always runs. Interim logic builds upon an IQueryable. Whether or not ToList has a significant impact largerly depends on whether this is Linq2Objects or it is Linq2SQL/Entitity Framework/Some other Database ORM. With database ORMs typically each .ToList hydrates the list and results in a database query executed. For Linq2Objects there is still a performance cost, but no where near the magnitude involved in a database call.
// itemsQuery will be of type IQueryable or IEnumerable depending on type of Items
var itemsQuery = model.Items.Where(x => x.Id > 0);
if (SortBy == "Description")
{
// we'll chain additional criteria or sorts to the IQueryable,
// and update the query reference by assigning over the previous itemsQuery variable
if (Direction == "Descending")
itemsQuery = itemsQuery.OrderByDescending(x => x.Description);
else
itemsQuery = itemsQuery.OrderBy(x => x.Description);
}
// I want to jump to the second page
this.Items = itemsQuery.Skip(26).ToList();
Essentially you are dynamically building up a query, but not actually running the query until the end where you call ToList, which will essentially build an SQL query from the expression tree and execute the entire chained sequence of .Where(...).OrderBy(...).Skip(26).ToList
You're quite right that all of the ToList calls are a major problem. Not only are they forcing all of the work to be done in the client if the underlying data source represents a database query provider, but even if it is inherently a LINQ to objects query you're needlessly creating, populating, and discarding lists. You're also removing any ability for deferred execution. Since you're not actually getting anything out of those calls, you can simply remove them all.
Next, you can write a method to create a selector, so that you don't need to duplicate all of that code for each column you sort on:
public static Expression<Func<T, TResult>> CreateSelector<T, TResult>(
string memberName)
{
var param = Expression.Parameter(typeof(T));
var body = Expression.PropertyOrField(param, memberName);
return Expression.Lambda<Func<T, TResult>>(body, param);
}
This allows you to write:
IQueryable<Item> items = model.Items;
if (Direction == "Descending")
items = items.OrderByDescending(CreateSelector<Item, string>(SortBy));
else
items = items.OrderBy(CreateSelector<Item, string>(SortBy));
items = items.Skip(26);
If the underlying data source that you have is an IEnumerable, not an IQueryable, just change the type of Items and Compile the selector expression.
Related
At work I have to make a console application which can contact a WCF service at Sharepoint, pull out some data from a list and compare it to some data from our Microsoft SQL Server Database.
Now, I almost got it down but I have run into some issues when trying to pull out some values that I need. I need two different values that are put in the same object representation. To do this I have to make the same query twice but on two different Title field criteria:
context.Values.Where(i => i.Title.Equals("Flight number:") && surveyIds.Contains(i.Survey.Id) == true).ToList();
context.Values.Where(i => i.Title.Equals("Date") && surveyIds.Contains(i.Survey.Id) == true).ToList();
The problem is that I can't call this portion of the code without getting the NotSupportedException
surveyIds.Contains(i.Survey.Id)
The expression (([10007].Title == "Flight number:") And (value(System.Collections.Generic.List`1[System.Int32]).Contains([10007].Survey.Id) == True)) is not supported.
Further up in the code I've made another list called surveyIds which is full of integers and to limit the list search that I do, I wanted to compare the Survey attached to the Value's ID (since it's a lookup) with the ones in my surveyIds list.
List<FlightSurveysDataContext.SurveysItem> reports = context.Surveys.Where(i => i.Title.Equals("Quality report - Cleaning") && i.UploadComplete == true).ToList();
List<int> surveyIds = new List<int>();
foreach (SurveysItem item in reports) { surveyIds.Add(item.Id); }
Can I do this in some other fashion? The reason I wanna do it in one go is that if I don't limit the search, the Collection will only get the first 1,000 values it finds that matches the title and the Sharepoint list have a little over 200,000 items currently so I am sure to get items I don't want.
As I mentioned in the comments, looks like the SharePoint LINQ query provider does not support constant Contains expression.
You can try replacing it with the equivalent || based condition build with the following helper:
public static partial class QueryableExtensions
{
public static IQueryable<T> WhereIn<T, V>(this IQueryable<T> source, Expression<Func<T, V>> valueSelector, IEnumerable<V> values)
{
var condition = values
.Select(value => Expression.Equal(valueSelector.Body, Expression.Constant(value)))
.DefaultIfEmpty()
.Aggregate(Expression.OrElse);
if (condition == null) return source;
var predicate = Expression.Lambda<Func<T, bool>>(condition, valueSelector.Parameters);
return source.Where(predicate);
}
}
The usage would be something like:
var result = context.Values
.Where(i => i.Title.Equals("Flight number:"))
.WhereIn(i => i.Survey.Id, surveyIds)
.ToList();
I am using JQuery widgets's datatable to have a table control in an MVC application with server-side paging, sorting and filtering. When doing the filtering, I have the following method:
private IQueryable<ImportRfqViewModel> BuildLinqQuery(System.Collections.Specialized.NameValueCollection query)
{
var result = query.GetValues("filterslength");
var filtersCount = int.Parse(query.GetValues("filterslength")[0]);
if (result == null || filtersCount == 0)
{
return AllImportRfq();
}
Predicate<ImportRfqViewModel> orResultPredicate = PredicateExtensions.False<ImportRfqViewModel>();
for (var i = 0; i < filtersCount; i += 1)
{
var filterValue = query.GetValues("filtervalue" + i)[0].ToUpper();
var filterCondition = query.GetValues("filtercondition" + i)[0];
var filterDataField = query.GetValues("filterdatafield" + i)[0];
var filterOperator = query.GetValues("filteroperator" + i)[0];
if (filterDataField == "ImportRfqId")
{
Predicate<ImportRfqViewModel> predicate = p => p.ImportRfqId.ToString().Contains(filterValue);
orResultPredicate = orResultPredicate.Or(predicate);
}
else if (filterDataField == "DateCreated")
{
Predicate<ImportRfqViewModel> predicate = p => p.DateCreated.ToString("yyyy/MM/dd hh:mm:ss").Contains(filterValue);
orResultPredicate = orResultPredicate.Or(predicate);
}
...
}
Func<ImportRfqViewModel, bool> funcOr = l => orResultPredicate(l);
var allResearch = AllImportRfq().Where(funcOr).AsQueryable();
return allResearch;
}
I'm using the predicates in order to chain Or conditions. I obviously want to return an IQueryable so that the query is not run before I get to the part:
dbResult = dbResult.Skip(pagesize * pagenum).Take(pagesize);
The result of the predicate though, is an IEnumerable. This is why I call the .AsQueryable();
What I'm concerned about and don't know is whether the thing potentially is enumerating the query and then returning a IQueryable afterwards and throwing away the enumeration, if I'm making sense.
In all this I'm assuming that if I return an IEnumerable, it would execute the query.
Returning an IEnumerable by itself does not mean you are actually executing the query. It depends on the IEnumerable's creator, but normally IEnumerable is considered lazy and the underlying data source is touched only when enumerating an IEnumerator. Linq operators and extension methods behave nicely, so your AsQueryable() call does not enumerate the target, and you should be safe.
BTW, did you try changing your code from this:
Func<ImportRfqViewModel, bool> funcOr = l => orResultPredicate(l);
to this (as long as your query provider fully supports Where and the predicate you are building, of course)?
Expression<Func<ImportRfqViewModel, bool>> funcOr = ...;
This way you should be able to avoid going back and forth across IQueryable and IEnumerable, you would not need calling AsQueryable() at all and therefore you'd have no more doubts. If your query provider supports it, it would be much more efficient than what you are doing, which would execute the filtering after querying the underlying data source.
EDIT: just to make it clearer, the move from Func<> to Expression<> is not just a matter of changing the type of your variable, it actually means you'd have to review the whole process of building your predicates so that you always work by composing Expression instances, and not just Func ones, otherwise your resulting expression tree will hardly be understandable to your query provider.
EDIT: I managed to reduce my code to the least-common-denominator with the assistance of LinqPad.
I have a method that returns a complex LinqKit predicate. The desired result is to be able to reuse this method whenever I need to perform this query. It works fine in a Where clause against an IQueryable collection of these entities, but I can't for the life of me figure out how to use this predicate when I have this entity as a single property of an object, and want to use predicates to query the other properties of that entity.
I'll give you a simplified example of what I mean.
// Note: .Includes removed for brevity's sake.
var task = Tasks.First(t => t.QaConfigTaskId == 2);
var predicate = PredicateBuilder.True<Task>();
// Ensure that the task's own properties match.
predicate = predicate.And(t => t.TaskType == task.TaskType &&
t.Description == task.Description &&
t.PreTreatment == task.PreTreatment &&
t.Treatment == task.Treatment);
var structureAnalysis = task.StructureAnalysis;
var query = PredicateBuilder.True<StructureAnalysis>();
query = query.And(analysis =>
// The names match
analysis.Name == structureAnalysis.Name &&
// We have the same # of goals so they must all match.
analysis.Goals.Count == structureAnalysis.Goals.Count
);
predicate = predicate.And(t => query.Invoke(t.StructureAnalysis));
This will work fine:
StructureAnalyses.AsExpandable().Where(query).Dump();
...assuming StructureAnalyses is an IQueryable of my StructureAnalysis objects from Entity Framework.
But let's say I want to get the Tasks, filtering with the related StructureAnalyses. How can I do such a thing? Keep in mind that in reality, query is built out by a reusable function, and predicate is built out in a separate function that calls it, so I can't simply merge the two queries.
It compiles but fails when I try and execute the query:
Tasks.AsExpandable().Where(predicate).Dump();
...with "The parameter 't' was not bound in the specified LINQ to Entities query expression.", or the like. In this example, Tasks is an IQueryable that contains all of the Task type entities in my DB.
I've also tried altering this line:
predicate = predicate.And(t => query.Invoke(t.StructureAnalysis));
...to:
compiled = query.Compile();
predicate = predicate.And(t => compiled(t.StructureAnalysis));
That also compiles but fails with, "Unable to cast object of type 'System.Linq.Expressions.FieldExpression' to type 'System.Linq.Expressions.LambdaExpression'." and understandably so.
I've also tried calling Expand on both query and compiled, which had no effect, as well as the following "solutions":
This one seems incomplete (Utility is undefined):
https://stackoverflow.com/questions/26105570/linqkit-the-parameter-was-not-bound-in-the-specified-linq-to-entities-query-e
This one simply says, "it can't be done" which I refuse to believe:
Entity Framework Code First 4.3 / LINQKit predicate for related table
I ended up finding the solution on my own. The trick is to wrap the invocation of query in another predicate that accesses the property, then expand that predicate when calling it. So for my example where I'm attempting to invoke a query on a single StructureAnalysis, my code now looks like this:
// Note: .Includes removed for brevity's sake.
var task = Tasks.First(t => t.QaConfigTaskId == 2);
var predicate = PredicateBuilder.True<Task>();
// Ensure that the task's own properties match.
predicate = predicate.And(t => t.TaskType == task.TaskType &&
t.Description == task.Description &&
t.PreTreatment == task.PreTreatment &&
t.Treatment == task.Treatment);
var structureAnalysis = task.StructureAnalysis;
var query = PredicateBuilder.True<StructureAnalysis>();
query = query.And(analysis =>
// The names match
analysis.Name == structureAnalysis.Name &&
// We have the same # of goals so they must all match.
analysis.Goals.Count == structureAnalysis.Goals.Count
);
//// HERE'S WHAT'S NEW ////
Expression<Func<Task, bool>> subPredicate = t => query.Invoke(t.StructureAnalysis);
predicate = predicate.And(subPredicate.Expand());
Tasks.AsExpandable().Where(predicate).Dump();
I've currently got a piece of Linq that looks something like this ;
List<dynamic> childrenToBeRemoved = this.ItemsSource.Where(o => o.ParentID == "1234").ToList();
where ItemsSource is an ObservableCollection of dynamics.
This works fine, but the problem I've got is that the ParentID is a property that can vary. E.g. it could be named ParentPkey or ParentKey etc.
Can I create an expression where I can specify the property that I want to use in my comparison?
I've tried using dynamic linq but it doesn't work using a collection of dynamics, works fine with a collection of pocos.
Thanks...
it should not matter if query is dynamic linq or not
Expression<Func<Entity, int>> predicate = x => x.Id == myvalue;
from entity in _context.Entities.Where(predicate)
select entity;
Check out PredicateBuilder of LinkKit # http://www.albahari.com/nutshell/linqkit.aspx
there are enough examples there as well
Responsibility of translation of an expression to corresponding sql lies with the linq provider, so make sure the provider you are using supports the relevant aspects
why make the implementation itself dynamic? you could simply do a dynamic invocation!
IEnumerable<MyItem> result;
if (condition1)
{
result = this.Items.Where(arg => arg.ProductId == "123");
}
else if (condition2)
{
result = this.Items.Where(arg => arg.ProductPK == "123");
}
or
Func<Item, bool> predicate;
if (condition1)
{
predicate = item => item.ProductId == "123";
}
else if (condition2)
{
predicate = item => item.ProductPK == "123";
}
var result = this.Items.Where(predicate);
Sooo ... I believe you should tell us more about your actual problem - I do not see any current need to implement sth - so, I believe your requirement is ill-defined!
Put you linq expression into a function, and pass in this property as a parameter.
If you know the type of your items, you can use reflection :
PropertyInfo parentProp = itemType.GetProperty("ParentKey or whatever");
List<dynamic> childrenToBeRemoved = this.ItemsSource.Where(o => Equals("1234", parentProp.GetValue(o, null))).ToList();
This is not another question about 'How Can I Sort Dynamically (based on an arbitrary user provided field)?'
The question is -- how can I change sort order when I know the potential sorts in advance? (And thus avoid reflection / custom Expression building typically associated with truly dynamic sorting.)
Take for instance this subquery (shortened for this example) of a larger query:
(from solutionIds in context.csExtendedQAIncident_Docs
where solutionIds.tiRecordStatus == 1
&& (from solutionProductAssocation in context.csProductDocs
where solutionProductAssocation.iSiteId == Settings.Current.WebUtility().Onyx.SiteId
&& (from allowedProduct in context.KB_User_Allowed_Products
where allowedProduct.UserId == userId
select allowedProduct.ModelCode
).Contains(solutionProductAssocation.chModelCd)
select solutionProductAssocation.chIdNo).Distinct().Contains(solutionIds.chIdNo)
).OrderByDescending(s => s.dtUpdateDate)
.Select(s => s.chIdNo)
.Take(count ?? Settings.Current.WCFServices().Output.HomePage.MaxRows)
The OrderByDescending portion works as I would expect.
Now -- I want to factor that out like the following:
Expression<Func<csExtendedQAIncident_Doc, IComparable>> ordering = (s) => s.dtUpdateDate;
if (viewType == HomepageViewType.MostViewed)
ordering = (s) => s.vchUserField8;
else if (viewType == HomepageViewType.MostEffective)
ordering = (s) => s.vchUserField4;
and then use:
OrderByDescending(ordering)
This does compile, but blows up at run-time.
Unsupported overload used for query operator 'OrderByDescending'.
This of course comes from deep in the bowels of System.Data.Linq.SqlClient.QueryConverter -- in particular VisitSequenceOperatorCall. Reflectoring that code reveals that the following conditions must be met for OrderByDescending to properly evaluate. 'mc' is the MethodCallExpression passed into the method.
if (((mc.Arguments.Count != 2) || !this.IsLambda(mc.Arguments[1]))
|| (this.GetLambda(mc.Arguments[1]).Parameters.Count != 1))
{
break;
}
So essentially that MethodCallExpression has to have 2 arguments, the second of which has to be a Expressions.LambdaExpression with a single parameter (presumably the sort field). If that code breaks out, the exception that I got is thrown.
So clearly I have not constructed the expression correctly. Without digging in any further here, does anyone know how to correctly construct the sorting Expression?
I think the unsupported part of your code is the use of IComparable as a general return type for your ordering expression. If you consider the plain use of OrderByDescending, the compiler-generated lambda expression has a return type of the type of the property that you're ordering by: for example, an Expression<Func<csExtendedQAIncident_doc, string>> for a string property.
One possible answer, although I'm not sure whether it works in your case, is to first create an unordered query:
IQueryable<Foo> unorderedQuery = from f in db.Foo select f;
And then, depending on the sort:
IOrderedQueryable<Foo> orderedQuery = unorderedQuery
.OrderBy(f => f.DefaultSortKey);
if (sortBy == SortByName)
orderedQuery = unorderedQuery.OrderBy(f => f.Name);
else if (sortBy == SortByDate)
orderedQuery = unorderedQuery.OrderBy(f => f.Date);
// etc.
I believe that this will not work unless the two possible fields have the identical type.
Then the linq to sql will (if possible) correctly create the relevant sql.
so for example if both of those fields were DateTimes:
Expression<Func<csExtendedQAIncident_Doc, DateTime>> ordering =
s => s.dtUpdateDate;
if (viewType == HomepageViewType.MostViewed)
ordering = (s) => s.vchUserField8; // a DateTime
else if (viewType == HomepageViewType.MostEffective)
ordering = (s) => s.vchUserField4; // another DateTime
Then this would work just fine (I tested it and it worked)
You could instead do a per type order by either a series of nested switch/if statements of by constructing a dictionary or similar structure to get them.
For the linq to sql to work without explicit dynamic creation of the query I believe it must know the precise type of the query as opposed to just it being an IComparable...