Develop Reference

Reusing Base Linq Query from one method to another

I am trying to remove duplicate code throughout my project and I am at a standstill trying to figure this out. What I am trying to do is create a base linq query that will be reused to add things like Where, Take...etc in multiple different methods.
public IQueryable<Object> FooLinq(int id)
{
using (var ctx = new dbEntities())
{
var results =
(from account in ctx.account
join memberProducts in ctx.tblMemberProducts on account.Id equals memberProducts.AccountId
orderby account.date descending
select new{account,memberProducts}).ToList();
return results;
}
}
So that would be by base query above and I would have a seperate method that would reuse VioLinq but this time would use a where clause in it.
public List<IncomingViolations> Foo1(int id)
{
//Linq query FooLinq() where Name == "Bob"
}

You'll need to do two things:
Return the query prior to materializing it.
Make sure the context is still in scope when the final query is materialized.
These two requirements will play off each other somewhat, and there are a number of approaches you can take to meet them.
For example, you could make your method take the context as a parameter, forcing the caller to provide it and manage its lifecycle.
public IQueryable<AccountInfo> FooLinq(DbEntities ctx, int id)
{
return
from account in ctx.account
orderby account.date descending
select new AccountInfo()
{
Name = account.Name,
Mid = account.MemberID,
Date = account.Date,
Address = account.Address,
};
}
public List<IncomingViolations> Foo1(int id)
{
using(var ctx = new dbEntities())
{
//Linq query FooLinq() where Name == "Bob"
return FooLinq(ctx).Where(v => v.Name == "Bob").ToList();
}
}
You could alternatively inject the context as a constructor-injected dependency, and use a DI framework to manage the context's lifecycle.

You can do it as Queryable then add conditions to it.
For example:
public List<account> GetAccountsByName(string name, bool usePaging, int offset = 0, int take = 0) {
var query = GetMyQuery();
query = query.Where(x => x.Name == name);
query = query.OrderBy(x => x.Name);
if(usePaging) {
query = query.Take(take).Skip(offset);
}
query = PrepareSelectForAccount(query);
return query.ToList(); .
}
public IQueryable<account> GetMyQuery(){
return ctx.account.AsQueryable();
}
public IQueryable<account> PrepareSelectForAccount(IQueryAble<account> query){
return query.Select(select new AccountInfo()
{
Name = account.Name,
Mid = account.MemberID,
Date = account.Date,
Address = account.Address,
}
);
}

Sure, but don't call .ToList(), and return IQueryable<T> instead of List<T>. LINQ is based on the concept of deferred execution which means the query is not actually performed until the enumerable is iterated over. Until then, all you have done is built an object which knows how to do the query when the time comes.
By returning an IQueryable<T> from a function that sets up the "basic query," you are then free to tack on additional LINQ methods (such as .Where() or .Take()) to produce a modified query. At this point you are still simply setting up the query; it is actually performed only when you iterate over the enumerable, or call something like .ToList() which does that for you.

How can I make this LINQ query?

I have a Dropdownlist that I fill with all consultants.
This is my entities I am using for this LINQ query:
What I want to do is to get all consultants ids that have a goalcard with Complete_date set. I have no idea how to do this.
Right now I have this following LINQ query to get all consultants ids.
public List<Consultant> GetAllConsultantsByID()
{
var allConsultants = from id in db.Consultant
select id;
return allConsultants.ToList();
}
Any kind of help is appreciated
Thanks in advance
Update:
This is how I have to use my Linq method in my Get Action Method:
var consultants = repository.GetAllConsultantsByID();
model.Consultants = new SelectList(consultants, "Id", "Name");

You can use the Consultant.GoalCard navigation property and the Any extension method:
var query = from con in db.Consultant
where con.GoalCard.Any(card => card.Completed_Date != null)
select con;
return query.ToList();
Consultant.Goalcard exposes all GoalCards of the Consultant as a queryable property. So you can perform queries on that, too. (This example assumes Completed_Date is nullable)
Note: Seeing that a Consultant can have several GoalCards, you might want to rename the Consultant's GoalCard navigation property to GoalCards (to make it clear there can be several).

Now assuming the Complete_Date is of type DateTime?, you could do it like that:
public IEnumerable<Consultant> GetConsultantIds()
{
return db.Consultant.Where(c => c.GoalCard != null && c.GoalCard.Completed_Date.HasValue).Select(c => c.Id).AsEnumerable();
}
[EDIT]
Since GoalCard is a collection (misleading name :) ), you can do something like that to get the IDs of Consultants who have at least one completed date set on any of the cards:
public IEnumerable<int> GetConsultantIds()
{
return db.Consultant.Where(c => c.GoalCard != null && c.GoalCard.Any(card => card.Completed_Date.HasValue)).Select(c => c.Id).AsEnumerable();
}
That's for the list of IDs only, for the list of Consultant objects meeting the criteria:
public IEnumerable<Consultant> GetConsultantIds()
{
return db.Consultant.Where(c => c.GoalCard != null && c.GoalCard.Any(card => card.Completed_Date.HasValue)).AsEnumerable();
}

Remove OrderBy from an IQueryable<T>

I have a paging API that returns rows a user requests, but only so many at one time, not the entire collection. The API works as designed, but I do have to calculate the total number of records that are available (for proper page calculations). Within the API, I use Linq2Sql and I work a lot with the IQueryable before i finally make my requests. When I go to get the count, I call something like: totalRecordCount = queryable.Count();
The resulting SQL is interesting none the less, but it also adds an unnecessary Order By which makes the query very expensive.
exec sp_executesql N'SELECT COUNT(*) AS [value]
FROM (
SELECT TOP (1) NULL AS [EMPTY]
FROM [dbo].[JournalEventsView] AS [t0]
WHERE [t0].[DataOwnerID] = #p0
ORDER BY [t0].[DataTimeStamp] DESC
) AS [t1]',N'#p0 int',#p0=1
Because I am using the IQueryable, I can manipulate the IQueryable prior to it making it to the SQL server.
My question is, if I already have an IQueryable with a OrderBy in it, is it possible to remove that OrderBy before I call the Count()?
like: totalRecordCount = queryable.NoOrder.Count();
If not, no biggie. I see many questions how to OrderBy, but not any involving removing an OrderBy from the Linq expression.
Thanks!

So, the below code is a spike against an in-memory array. There may be some hurdles to get this working with Entity Framework (or some other arbitrary IQueryProvider implementation). Basically, what we are going to do is visit the expression tree and look for any Ordering method call and simply remove it from the tree. Hope this points you in the right direction.
class Program
{
static void Main(string[] args)
{
var seq = new[] { 1, 3, 5, 7, 9, 2, 4, 6, 8 };
var query = seq.OrderBy(x => x);
Console.WriteLine("Print out in reverse order.");
foreach (var item in query)
{
Console.WriteLine(item);
}
Console.WriteLine("Prints out in original order");
var queryExpression = seq.AsQueryable().OrderBy(x => x).ThenByDescending(x => x).Expression;
var queryDelegate = Expression.Lambda<Func<IEnumerable<int>>>(new OrderByRemover().Visit(queryExpression)).Compile();
foreach (var item in queryDelegate())
{
Console.WriteLine(item);
}
Console.ReadLine();
}
}
public class OrderByRemover : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType != typeof(Enumerable) && node.Method.DeclaringType != typeof(Queryable))
return base.VisitMethodCall(node);
if (node.Method.Name != "OrderBy" && node.Method.Name != "OrderByDescending" && node.Method.Name != "ThenBy" && node.Method.Name != "ThenByDescending")
return base.VisitMethodCall(node);
//eliminate the method call from the expression tree by returning the object of the call.
return base.Visit(node.Arguments[0]);
}
}

There isn't just an unneeded ORDER BY, there's also a spurious TOP(1).
SELECT TOP (1) NULL AS [EMPTY] ...
That subselect will only return 0 or 1 rows. In fact without the TOP there it wouldn't be legal to have an ORDER BY in a subselect.
The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.: SELECT COUNT(*) FROM ( SELECT * FROM Table1 ORDER BY foo )
sqlfiddle
I think you have probably done something wrong in your LINQ. Are you sure you haven't written .Take(1) or similar somewhere in your query, before calling .Count()?
This is wrong:
IQueryable<Foo> foo = (...).OrderBy(x => x.Foo).Take(1);
int count = foo.Count();
You should do this instead:
IQueryable<Foo> foo = (...);
Iqueryable<Foo> topOne = foo.OrderBy(x => x.Foo).Take(1);
int count = foo.Count();

I am afraid there is no easy way to remove the OrderBy operator from queryable.
What you can do, however, is to re-create the IQueryable based on the new expression obtained from rewriting queryable.Expression(see here) omitting the OrderBy call.

If you can't eliminate the root cause, here is a workaround:
totalRecordCount = queryable.OrderBy(x => 0).Count();
SQL Server's query optimizer will remove this useless ordering. It won't have runtime cost.

I think you have implemented you paging code wrongly. You actually need to query the database twice, once for the paged datasource and once for the total row count. This is how the setup should look.
public IList<MyObj> GetPagedData(string filter, string sort, int skip, int take)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
if(!String.IsNullOrEmpty(sort))
q = q.OrderBy(sort); //And here
return q.Skip(skip).Take(take).ToList();
}
}
public int GetTotalCount(string filter)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
return q.Count(); //Without ordering and paging.
}
}
private static IQuerable<MyObj> GetDataInternal(DataContext db)
{
return
from x in db.JournalEventsView
where ...
select new ...;
}
The filtering and sorting is done using the Dynamic linq library

I know it is not quite what you are looking for, but index on [DataOwnerID] with inclusion of DataTimeStamp could make your query less expensive.

Include OrderBy delegate in Method parameters

I have a method;
public List<Task> GetTasksByAssignedTo(Guid contactId)
{
List<Task> tasks = dc.Tasks.Where(x => x.ContactId == contactId).ToList();
return tasks;
}
which returns a list of items. Say I now wanted to specify the sort order I want to return the list in.
So I might sort by Name, DueDate, Completed etc, etc.
How could I include that in the method as a parameter? I don't want to use a switch statement rather I'd like to use a lambda if possible.
So;
List<Task> list = GetTasksByAssignedTo("guid", ??????);
Or is this the wrong approach.

I think that your approach is the wrong way to use LINQ.
LINQ uses a deferred execution model for a reason. It allows you to chain together a series of operations that get executed only when you tell it to compute the result - often with .ToList(), .ToArray(), .First() - but you can also force the computation by filtering with a OrderBy clause that uses a Func<T, ?> as its parameter.
Now you're returning a List<Task> which means that you've forced the execution - which is the right thing to do when you're ready to use the results - but if you then go on to do further operations you are potentially loading many more records into memory than you need to.
You could certainly do this:
public List<Task> GetTasksByAssignedTo<P>(Guid contactId, Func<Task, P> orderBy)
{
return dc.Tasks
.Where(x => x.ContactId == contactId)
.OrderBy(orderBy) // this forces evaluation - sort happens in memory
.ToList();
}
To make the execution happen in the database you need to change it like this:
public List<Task> GetTasksByAssignedTo<P>(
Guid contactId,
Expression<Func<Task, P>> orderBy)
{
return dc.Tasks
.Where(x => x.ContactId == contactId)
.OrderBy(orderBy)
.ToList(); // Now execution happens here
}
But the issue is what if you did this:
var query =
from t1 in GetTasksByAssignedTo(contactId, t => t.Name)
join t2 in GetTasksByAssignedTo(contactId, t => t.Name)
on t1.Name equals t2.Name
select new { t1, t2 };
Because your GetTasksByAssignedTo brings records into memory you are doing the join in memory. (Yes, the query is a bit contrived, but the principle is solid though.)
It's often much better to do it in the database.
Here's how to fix it:
public IQueryable<Task> GetTasksByAssignedTo<P>(
Guid contactId,
Expression<Func<Task, P>> orderBy)
{
return dc.Tasks
.Where(x => x.ContactId == contactId)
.OrderBy(orderBy);
}
Now the above query won't execute until you do query.ToList() and all will happen at the database.
But I have an even bigger issue.
You're hiding a lot of information in the GetTasksByAssignedTo. Someone using the code doesn't know that they're actually getting a list when they read the code and they really don't know if the actual implementation is doing the right thing. I think, for these kinds of queries, it's often better to leave it as plain LINQ.
Compare these:
var tasks1 = GetTasksByAssignedTo(contactId);
var tasks2 = GetTasksByAssignedTo(contactId, t => t.Name);
var tasks3 = GetTasksByAssignedToDescending(contactId, t => t.Name);
var tasks4 = (
from t in dc.Tasks
where t.ContactId == contactId
orderby t.Name descending
select t
).ToList();
The first query, tasks1 isn't too bad, but it doesn't tell you what the return type is;
The second query, tasks2 does something with some t and the property Name, but doesn't tell you what.
The third query, tasks3 give you a hint that it is sorting descending, but doesn't tell you if it's by the mysterious Name property or something else.
The fourth query, tasks4 tells you everything that you need to know - it's filtering tasks by ContactId, reverse ordering the results by Name, and finally returning a list.
Now, take a look at this query:
var query2 =
from t1 in dc.Tasks
where t1.ContactId == contactId
join t2 in dc.Tasks on t1.Name equals t2.Name
where t2.ContactId != contactId
orderby t2.Name descending
select t2;
I can read that quite easily and see what it is doing. Just imagine what the helper method name would be for this one! Or what insane nesting of helper methods would be required.
The bottom-line is that LINQ is the API for querying.
If you desperately want to create helper methods then use extension methods.
public static class TaskEx
{
public static IQueryable<Task> WhereAssignedTo(this IQueryable<Task> tasks,
Guid contactId)
{
return tasks.Where(t => t.ContactId == contactId);
}
public static IQueryable<Task> OrderByName(this IQueryable<Task> tasks)
{
return tasks.OrderBy(t => t.Name);
}
}
This then allows you to write this:
var tasks = dc.Tasks
.WhereAssignedTo(contactId)
.OrderByName()
.ToList();
And that is clear, concise, extensible, composable, re-usable, and you control when execution occurs.

You could pass a Func<Task, object> to your method for ordering:
public List<Task> GetTasksByAssignedTo(Guid contactId, Func<Task, object> someOrder)
{
List<Task> tasks = dc.Tasks.Where(x => x.ContactId == contactId)
.OrderBy(someOrder)
.ToList();
return tasks;
}
Now you can call your method like
Func<Task, object> someOrder = (Task t) => t.DueDate;
List<Task> list = GetTasksByAssignedTo(someGuid, someOrder);
Generally I agree with the comments though - it does not seem that ordering is required for a method named GetTasksByAssignedTo.

#BrokenGlass beat me to the punch.
The other option is to use an extension method which hides away the switch, and represent the different ordering options as an enumeration.
public static IEnumerable<Task> WithOrdering(this IEnumerable<Task> source, TaskOrdering order)
{
switch (order)
{
case TaskOrdering.Name:
return source.OrderBy(task => task.Name);
case TaskOrdering.DueDate:
return source.OrderByDescending(task => task.DueDate);
}
}
And then:
public List<Task> GetTasksByAssignedTo(Guid contactId, TaskOrdering order)
{
List<Task> tasks = dc.Tasks.Where(x => x.ContactId == contactId)
.WithOrdering(order)
.ToList();
return tasks;
}
I do this all the time. Allowing the predicate as a method parameter can be tricky because what happens if you want to do ascending/descending? You'll need another parameter for that (bool), then do a if/else check to do OrderBy or OrderByDescending.
Hide the logic in a filter, then you can re-use it anywhere in your app.

Try this... input parameters are in string. This is according modified solution from StackOverflow
/// <summary>
/// Sort List<typeparam name="T"></typeparam> objects base on string options
/// </summary>
/// <param name="SortDirection">Ascending or Descending</param>
/// <param name="ColumnName">Column name in complex object (object.ColumnName)</param>
public static class ListExtension
{
public static List<T> SortList<T>(this List<T> data, string sortDirection, string sortExpression)
{
try
{
switch (sortDirection)
{
case "Ascending":
data = (from n in data
orderby GetDynamicSortProperty(n, sortExpression) ascending
select n).ToList();
break;
case "Descending":
data = (from n in data
orderby GetDynamicSortProperty(n, sortExpression) descending
select n).ToList();
break;
default:
data = null; //NUL IF IS NO OPTION FOUND (Ascending or Descending)
break;
}
return data;
} catch(Exception ex){
throw new Exception("Unable to sort data", ex);
}
}
private static object GetDynamicSortProperty(object item, string propName)
{
//Use reflection to get order type
return item.GetType().GetProperty(propName).GetValue(item, null);
}
}

Performing part of a IQueryable query and deferring the rest to Linq for Objects

I have a Linq provider that sucessfully goes and gets data from my chosen datasource, but what I would like to do now that I have my filtered resultset, is allow Linq to Objects to process the rest of the Expression tree (for things like Joins, projection etc)
My thought was that I could just replace the expression constant that contains my IQueryProvider with the result-sets IEnumerable via an ExpressionVisitor and then return that new expression. Also return the IEnumerable's provider from my IQueryable...but this does not seem to work :-(
Any idea's?
Edit:
Some good answers here, but given the form...
var qry = from c in MyProv.Table<Customer>()
Join o in MyProv.Table<Order>() on c.OrderID equals o.ID
select new
{
CustID = c.ID,
OrderID = o.ID
}
In my provider I can easily get back the 2 resultsets from Customers and Orders, if the data was from a SQL source I would just construct and pass on the SQL Join syntax, but it this case the data is not from a SQL source so I need to do the join in code...but as I said I have the 2 result sets, and Linq to Objects can do a join...(and later the projection) it would be real nice to just substitute the Expression constants MyProv.Table<Customer> and MyProv.Table<Order> with List<Customer> and List<Order> and let a List<> provider process the expression...is that possible? how?

Both of the previous answers work, but it reads better if you use AsEnumerable() to cast the IQueryable to IEnumerable:
// Using Bob's code...
var result = datacontext.Table
.Where(x => x.Prop == val)
.OrderBy(x => x.Prop2)
.AsEnumerable() // <---- anything after this is done by LINQ to Objects
.Select(x => new { CoolProperty = x.Prop, OtherProperty = x.Prop2 });
EDIT:
// ... or MichaelGG's
var res = dc.Foos
.Where(x => x.Bla > 0) // uses IQueryable provider
.AsEnumerable()
.Where(y => y.Snag > 0); // IEnumerable, uses LINQ to Objects

The kind of thing that I was after was replacing the Queryable<> constant in the expression tree with a concrete IEnumerable (or IQueryable via .AsQueryable()) result set...this is a complex topic that probably only makes any sense to Linq Provider writers who are knee deep in expression tree visitors etc.
I found a snippet on the msdn walkthrough that does something like what I am after, this gives me a way forward...
using System;
using System.Linq;
using System.Linq.Expressions;
namespace LinqToTerraServerProvider
{
internal class ExpressionTreeModifier : ExpressionVisitor
{
private IQueryable<Place> queryablePlaces;
internal ExpressionTreeModifier(IQueryable<Place> places)
{
this.queryablePlaces = places;
}
internal Expression CopyAndModify(Expression expression)
{
return this.Visit(expression);
}
protected override Expression VisitConstant(ConstantExpression c)
{
// Replace the constant QueryableTerraServerData arg with the queryable Place collection.
if (c.Type == typeof(QueryableTerraServerData<Place>))
return Expression.Constant(this.queryablePlaces);
else
return c;
}
}
}

If you implemented a Repository Pattern you could get away with just providing an IQueryable back and abstract away the table.
Example:
var qry = from c in MyProv.Repository<Customer>()
Join o in MyProv.Repository<Order>() on c.OrderID equals o.ID
select new
{
CustID = c.ID,
OrderID = o.ID
}
and then just build your provider to model the IQueryable pattern in your Repository method just like this article illustrates.
This way, you can write all kinds of providers to use for whatever you need. You can have a LINQ 2 SQL provider, or write an in memory provider for your unit tests.
The Repository method for the LINQ 2 SQL provider would look something like this:
public IQueryable<T> Repository<T>() where T : class
{
ITable table = _context.GetTable(typeof(T));
return table.Cast<T>();
}

Unless I'm misunderstanding, I generally just add .ToArray() in the chain of linq methods at the point where I want the linq provider to execute.
For example (think Linq to SQL)
var result = datacontext.Table
.Where(x => x.Prop == val)
.OrderBy(x => x.Prop2)
.ToArray()
.Select(x => new {CoolProperty = x.Prop, OtherProperty = x.Prop2});
So through OrderBy() gets translated into SQL, but the Select() is LINQ to Objects.

Rob's answer is good, but forces full enumeration. You could cast to keep extension method syntax and lazy evaluation:
var res = ((IEnumerable<Foo>)dc.Foos
.Where(x => x.Bla > 0)) // IQueryable
.Where(y => y.Snag > 0) // IEnumerable