I have a paging API that returns rows a user requests, but only so many at one time, not the entire collection. The API works as designed, but I do have to calculate the total number of records that are available (for proper page calculations). Within the API, I use Linq2Sql and I work a lot with the IQueryable before i finally make my requests. When I go to get the count, I call something like: totalRecordCount = queryable.Count();
The resulting SQL is interesting none the less, but it also adds an unnecessary Order By which makes the query very expensive.
exec sp_executesql N'SELECT COUNT(*) AS [value]
FROM (
SELECT TOP (1) NULL AS [EMPTY]
FROM [dbo].[JournalEventsView] AS [t0]
WHERE [t0].[DataOwnerID] = #p0
ORDER BY [t0].[DataTimeStamp] DESC
) AS [t1]',N'#p0 int',#p0=1
Because I am using the IQueryable, I can manipulate the IQueryable prior to it making it to the SQL server.
My question is, if I already have an IQueryable with a OrderBy in it, is it possible to remove that OrderBy before I call the Count()?
like: totalRecordCount = queryable.NoOrder.Count();
If not, no biggie. I see many questions how to OrderBy, but not any involving removing an OrderBy from the Linq expression.
Thanks!
So, the below code is a spike against an in-memory array. There may be some hurdles to get this working with Entity Framework (or some other arbitrary IQueryProvider implementation). Basically, what we are going to do is visit the expression tree and look for any Ordering method call and simply remove it from the tree. Hope this points you in the right direction.
class Program
{
static void Main(string[] args)
{
var seq = new[] { 1, 3, 5, 7, 9, 2, 4, 6, 8 };
var query = seq.OrderBy(x => x);
Console.WriteLine("Print out in reverse order.");
foreach (var item in query)
{
Console.WriteLine(item);
}
Console.WriteLine("Prints out in original order");
var queryExpression = seq.AsQueryable().OrderBy(x => x).ThenByDescending(x => x).Expression;
var queryDelegate = Expression.Lambda<Func<IEnumerable<int>>>(new OrderByRemover().Visit(queryExpression)).Compile();
foreach (var item in queryDelegate())
{
Console.WriteLine(item);
}
Console.ReadLine();
}
}
public class OrderByRemover : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType != typeof(Enumerable) && node.Method.DeclaringType != typeof(Queryable))
return base.VisitMethodCall(node);
if (node.Method.Name != "OrderBy" && node.Method.Name != "OrderByDescending" && node.Method.Name != "ThenBy" && node.Method.Name != "ThenByDescending")
return base.VisitMethodCall(node);
//eliminate the method call from the expression tree by returning the object of the call.
return base.Visit(node.Arguments[0]);
}
}
There isn't just an unneeded ORDER BY, there's also a spurious TOP(1).
SELECT TOP (1) NULL AS [EMPTY] ...
That subselect will only return 0 or 1 rows. In fact without the TOP there it wouldn't be legal to have an ORDER BY in a subselect.
The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.: SELECT COUNT(*) FROM ( SELECT * FROM Table1 ORDER BY foo )
sqlfiddle
I think you have probably done something wrong in your LINQ. Are you sure you haven't written .Take(1) or similar somewhere in your query, before calling .Count()?
This is wrong:
IQueryable<Foo> foo = (...).OrderBy(x => x.Foo).Take(1);
int count = foo.Count();
You should do this instead:
IQueryable<Foo> foo = (...);
Iqueryable<Foo> topOne = foo.OrderBy(x => x.Foo).Take(1);
int count = foo.Count();
I am afraid there is no easy way to remove the OrderBy operator from queryable.
What you can do, however, is to re-create the IQueryable based on the new expression obtained from rewriting queryable.Expression(see here) omitting the OrderBy call.
If you can't eliminate the root cause, here is a workaround:
totalRecordCount = queryable.OrderBy(x => 0).Count();
SQL Server's query optimizer will remove this useless ordering. It won't have runtime cost.
I think you have implemented you paging code wrongly. You actually need to query the database twice, once for the paged datasource and once for the total row count. This is how the setup should look.
public IList<MyObj> GetPagedData(string filter, string sort, int skip, int take)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
if(!String.IsNullOrEmpty(sort))
q = q.OrderBy(sort); //And here
return q.Skip(skip).Take(take).ToList();
}
}
public int GetTotalCount(string filter)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
return q.Count(); //Without ordering and paging.
}
}
private static IQuerable<MyObj> GetDataInternal(DataContext db)
{
return
from x in db.JournalEventsView
where ...
select new ...;
}
The filtering and sorting is done using the Dynamic linq library
I know it is not quite what you are looking for, but index on [DataOwnerID] with inclusion of DataTimeStamp could make your query less expensive.
Related
I am trying to remove duplicate code throughout my project and I am at a standstill trying to figure this out. What I am trying to do is create a base linq query that will be reused to add things like Where, Take...etc in multiple different methods.
public IQueryable<Object> FooLinq(int id)
{
using (var ctx = new dbEntities())
{
var results =
(from account in ctx.account
join memberProducts in ctx.tblMemberProducts on account.Id equals memberProducts.AccountId
orderby account.date descending
select new{account,memberProducts}).ToList();
return results;
}
}
So that would be by base query above and I would have a seperate method that would reuse VioLinq but this time would use a where clause in it.
public List<IncomingViolations> Foo1(int id)
{
//Linq query FooLinq() where Name == "Bob"
}
You'll need to do two things:
Return the query prior to materializing it.
Make sure the context is still in scope when the final query is materialized.
These two requirements will play off each other somewhat, and there are a number of approaches you can take to meet them.
For example, you could make your method take the context as a parameter, forcing the caller to provide it and manage its lifecycle.
public IQueryable<AccountInfo> FooLinq(DbEntities ctx, int id)
{
return
from account in ctx.account
orderby account.date descending
select new AccountInfo()
{
Name = account.Name,
Mid = account.MemberID,
Date = account.Date,
Address = account.Address,
};
}
public List<IncomingViolations> Foo1(int id)
{
using(var ctx = new dbEntities())
{
//Linq query FooLinq() where Name == "Bob"
return FooLinq(ctx).Where(v => v.Name == "Bob").ToList();
}
}
You could alternatively inject the context as a constructor-injected dependency, and use a DI framework to manage the context's lifecycle.
You can do it as Queryable then add conditions to it.
For example:
public List<account> GetAccountsByName(string name, bool usePaging, int offset = 0, int take = 0) {
var query = GetMyQuery();
query = query.Where(x => x.Name == name);
query = query.OrderBy(x => x.Name);
if(usePaging) {
query = query.Take(take).Skip(offset);
}
query = PrepareSelectForAccount(query);
return query.ToList(); .
}
public IQueryable<account> GetMyQuery(){
return ctx.account.AsQueryable();
}
public IQueryable<account> PrepareSelectForAccount(IQueryAble<account> query){
return query.Select(select new AccountInfo()
{
Name = account.Name,
Mid = account.MemberID,
Date = account.Date,
Address = account.Address,
}
);
}
Sure, but don't call .ToList(), and return IQueryable<T> instead of List<T>. LINQ is based on the concept of deferred execution which means the query is not actually performed until the enumerable is iterated over. Until then, all you have done is built an object which knows how to do the query when the time comes.
By returning an IQueryable<T> from a function that sets up the "basic query," you are then free to tack on additional LINQ methods (such as .Where() or .Take()) to produce a modified query. At this point you are still simply setting up the query; it is actually performed only when you iterate over the enumerable, or call something like .ToList() which does that for you.
I have a Linq statement which has two optional Where clauses but I cannot work out how to implement this linq (at the moment I have 4 linq statements in this method which off course is not good!):
//Params passed into my method:
int StoreId = 0;
bool ShowEnabledTills = true;
var query = (from Transactions in db.Transactions
join Tills in db.Tills on new { TillId = Convert.ToInt32(Transactions.TillId) } equals new { TillId = Tills.TillId }
join CompanyAddresses in db.CompanyAddresses on new { CompanyId = Convert.ToInt32(Transactions.StoreID) } equals new { CompanyId = CompanyAddresses.CompanyId }
where
CompanyAddresses.CompanyId == StoreId <===== OPTIONAL
&& Tills.Active == ShowEnabledTills <===== OPTIONAL
select new
{
Tills.TillId,
Tills.ComputerName,
Tills.Description,
CompanyAddresses.CompDescription
}).Distinct();
I took a look at PredicateBuilder but I couldn't quite get my head around this but if I can create some re-usable code this would be great!
// get these from params to decide optional or not
var validateStoredId = false;
var validateActiveStatus = false;
And in your where clause do something like:
where
(!validateStoreId || CompanyAddresses.CompanyId == StoreId)
&& (!validateActiveStatus || Tills.Active == ShowEnabledTills)
I have a little to contribute here I think. I came across the same problem, and implemented a similar solution. HOWEVER! This wont short-circuit as you would hope (or at least it didn't in mine!)
This led to me looping through a list of 10k items checking if true == false effectively, which whilst not an expensive check is a check you don't want to do thousands of times nevertheless.
For anyone else coming to this, I'd recommend splitting your queries down into subqueries wrapped in if checks, that is far more performant when dealing with large datasets :)
Edit : I built this little helper for doing exactly this, hope someone else finds it useful, I enjoyed not having to break my method chains :)
public static IEnumerable<T> OptionalWhere<T>(this IEnumerable<T> source, Func<T, bool> predicate, bool excecuteFilter)
{
if (!excecuteFilter)
{
return source;
}
else
{
return source.Where(predicate);
}
}
Try this:
var query = db.Transactions.Join(
db.Tills, //Join to this table
transaction => transaction.TillId //Outer key
, till => till.TillId, //Inner key
(transaction, till) => new {transaction, till}) //Result of join
.Join(db.CompanyAddresses, //Join to this table
r => r.till.StoreId, //Outer key
a => a.CompanyId, //Inner key
(r, address) => new {r.transaction, r.till, address}); //Result of join
if(StoreId != null)
query = query.Where(d => d.address.StoreId == StoreId);
if(ShowEnabledTills)
query = query.Where(d => d.till.Active == true);
var items = query.Select(d => new {
d.Till.TillId,
d.Till.ComputerName,
d.Till.Description,
d.address.CompDescription
}).Distinct();
I did it without knowing much about your schema, but this should give you a good idea.
I disliked the "SQL style" syntax for this very reason (confusing at times). I started using method calls like this and never had a problem since.
Is there a way to clear an IQueryable from its ordeyby clauses?
For example, we have a function that returns an ordered query:
public IQueryable<SomeType> GetOrderedQuery()
{
var query = from item in db.itemsTable
where item.x != null
orderby item.y, item.z
select item;
return query;
}
And we have another function that needs to use the same query, but it needs to have it unordered:
public IQueryable<SomeType> GetUnorderedQuery()
{
var query = GetOrderedQuery();
query.RemoveOrders(); // How to implement a RemoveOrders function?
return query;
}
How can a RemoveOrders function be implemented? (Doesn't matter if as an extension method or not)
If you don't want it ordered; don't order it. There's no robust way to walk back through an IQueryable<T> to get earlier states, let alone remove individual bits out of the middle. I suspect you want two queries:
public IQueryable<SomeType> GetUnorderedQuery()
=> db.itemsTable.Where(item => item.x != null);
public IOrderedQueryable<SomeType> GetOrderedQuery()
=> GetUnorderedQuery().OrderBy(item => item.y).ThenBy(item => item.z);
I have a basic IQueryable,
private static IQueryable<TestObject> GetFilteredQuery(Guid componentId, Guid productId)
{
IQueryable<TestObject> query = from t in ModelQuery.Query<TestObject>()
where t.ComponentId == componentId && t.ProductId == productId
select t;
return query;
}
This is trivial if I have to compare single componentId and productId.
My problem is how can I handle when I have a list of value pairs,
Guid[] componentIds, Guid[] productIds
where, its kind of a keyValue pair.
something like,
private static IQueryable<TestObject> GetFilteredQuery(Guid[] componentIds, Guid[] productIds)
{
IQueryable<TestObject> query = from t in ModelQuery.Query<TestObject>()
where (t.ComponentId must be present in componentIds[] && t.ProductId must be present in productIds)
select t;
return query;
}
Use Contains:
private static IQueryable<TestObject> GetFilteredQuery(Guid[] componentIds, Guid[] productIds)
{
IQueryable<TestObject> query =
from t in ModelQuery.Query<TestObject>()
where (componentIds.Contains(t.ComponentId)
&& productIds.Contains(t.ProductId))
select t;
return query;
}
Edit
AFAIK there is no way Linq2Sql is going to map a sequence of Guid tuples to native Sql (you would likely need an #Table parameter for this)
So here's one approach, viz to run a query the same contains as above, but using OR on the 2 filter lists. Sql will hopefully be able to filter a significant amount of data out at the database level.
The results (candidates) then need to be materialized, and then filtered in memory against the component and product pairs. I've done this by zipping the 2 guid arrays together (assuming similar length - possibly you want to remodel the arrays as an array of Pairs to express the intention more explicitly?)
private static IQueryable<TestObject> GetFilteredQuery(Guid[] componentIds,
Guid[] productIds)
{
var candidates = ModelQuery
.Query<TestObject>()
.Where(componentIds.Contains(
t.ComponentId) || productIds.Contains(t.ProductId))
.ToList();// Need to materialize
var guidPairs = componentIds.Zip(productIds,
(c, p) => new {ComponentId = c, ProductId = p});
return candidates
.Join(guidPairs,
c => new {ComponentId = c.ComponentId, ProductId = c.ProductId},
gp => gp,
(c, gp) => c)
.AsQueryable();
}
Note that the resultant queryable isn't really suitable for further composition, given that it has already been materialized. Also, if you can do additional filtering before hitting this, it would be beneficial. And I'm afraid I haven't actually tested this.
Use Contains:
where componentIds.Contains(t.ComponentId) && productIds.Contains(t.ProductId)
I would like to return a set of entities who has and ID that is contained in a list or array of IDs using LINQ and Data Services. I know how to this using LinqToEF but I am at a loss how to this with Data Services or using OData query conventions for that matter.
My thought is that I would do something like:
int[] intArray = {321456, 321355, 218994, 189232};
var query = (from data in context.Entity
where intArray.contains(data.ID)
select data);
Is there any way to accomplish using Data Services / OData? I know I could probably hack it with a Service Operation but I would prefer not to do that.
Cheers.
Currently OData (the underlying protocol) doesn't support the Contains operation. So that's why the client library does not translate the above query.
People are basically using two ways to overcome this limitation:
1) Use service operations as you noted.
2) Construct a where clause dynamically which uses simple comparisons to compare the value to each item from the array. So if the array contains 1, 2, 3, the where would be data.ID == 1 || data.ID == 2 || data.ID == 3
The #2 solution is nice because it's a client side only change. The downside is, that it only works for small arrays. If the array contains too many items the expression gets too long and that leads to all kinds of troubles.
The #1 solution doesn't have the size problem, but you need to provide the operation on the server.
Here is my realization of WhereIn() Method, to filter IQueryable collection by a set of selected entities:
public static IQueryable<T> WhereIn<T,TProp>(this IQueryable<T> source, Expression<Func<T,TProp>> memberExpr, IEnumerable<TProp> values) where T : class
{
Expression predicate = null;
ParameterExpression param = Expression.Parameter(typeof(T), "t");
bool IsFirst = true;
// Create a comparison for each value eg:
// IN: t => t.Id == 1 | t.Id == 2
MemberExpression me = (MemberExpression) memberExpr.Body;
foreach (TProp val in values)
{
ConstantExpression ce = Expression.Constant(val);
Expression comparison = Expression.Equal(me, ce);
if (IsFirst)
{
predicate = comparison;
IsFirst = false;
}
else
{
predicate = Expression.Or(predicate, comparison);
}
}
return predicate != null
? source.Where(Expression.Lambda<Func<T, bool>>(predicate, param)).AsQueryable<T>()
: source;
}
And calling of this method looks like:
IQueryable<Product> q = context.Products.ToList();
var SelectedProducts = new List<Product>
{
new Product{Id=23},
new Product{Id=56}
};
...
// Collecting set of product id's
var selectedProductsIds = SelectedProducts.Select(p => p.Id).ToList();
// Filtering products
q = q.WhereIn(c => c.Product.Id, selectedProductsIds);
Thank you men you really helped me :) :)
I did it like Vitek Karas said.
1) Download the Dynamic query library
Check this link
No need to read it just download the Dynamic query library
2)Check the project named DynamicQuery. In it you will find a class named Dynamic.cs . Copy It to your project
3)Generate your project( If you are using silverlight an error that say ReaderWriterLock is not found will appear. Don't be affraid. Just comment or delete the lines that make errors( there is just 6 or 7 lines that make errors) )
4) All done you just need now to write your query
Example: ordersContext.CLIENTS.Where(" NUMCLI > 200 || NUMCLI < 20");
All done. If you have to use the 'Contains' method you just to write a method that iterate over your array and return the string that your request will use.
private string MyFilter()
{ string st = "";
foreach(var element in myTab)
{
st = st + "ThePropertyInTheTable =" + element + "||";
}
return st;
}
I hope you understand me and that i helped someone :)