Develop Reference

LINQ querying using an entity object not in the database - Error: LINQ to Entities does not recognize the method

I have an Entity Framework v5 model created from a database. The table Season has a corresponding entity called Season. I need to calculate the Season's minimum start date and maximum end date for each year for a Project_Group. I then need to be able to JOIN those yearly min/max Season values in other LINQ queries. To do so, I have created a SeasonLimits class in my Data Access Layer project. (A SeasonLimits table does not exist in the database.)
public partial class SeasonLimits : EntityObject
{
public int Year { get; set; }
public DateTime Min_Start_Date { get; set; }
public int Min_Start_Date_ID { get; set; }
public DateTime Max_End_Date { get; set; }
public int Max_End_Date_ID { get; set; }
public static IQueryable<SeasonLimits> QuerySeasonLimits(MyEntities context, int project_Group_ID)
{
return context
.Season
.Where(s => s.Locations.Project.Project_Group.Any(pg => pg.Project_Group_ID == project_Group_ID))
.GroupBy(x => x.Year)
.Select(sl => new SeasonLimits
{
Year = sl.Key,
Min_Start_Date = sl.Min(d => d.Start_Date),
Min_Start_Date_ID = sl.Min(d => d.Start_Date_ID),
Max_End_Date = sl.Max(d => d.End_Date),
Max_End_Date_ID = sl.Max(d => d.End_Date_ID)
});
}
}
// MVC Project
var seasonHoursByYear =
from d in context.AuxiliaryDateHours
from sl in SeasonLimits.QuerySeasonLimits(context, pg.Project_Group_ID)
where d.Date_ID >= sl.Min_Start_Date_ID
&& d.Date_ID < sl.Max_End_Date_ID
group d by new
{
d.Year
} into grp4
orderby grp4.Key.Year
select new
{
Year = grp4.Key.Year,
HoursInYear = grp4.Count()
};
In my MVC project, whenever I attempt to use the QuerySeasonLimits method in a LINQ query JOIN, I receive the message,
"LINQ to Entities does not recognize the method
'System.Linq.IQueryable`1[MyDAL.SeasonLimits]
QuerySeasonLimits(MyDAL.MyEntities, MyDAL.Project_Group)' method, and
this method cannot be translated into a store expression."
Is this error being generated because SeasonLimits is not an entity that exists in the database? If this can't be done this way, is there another way to reference the logic so that it can be used in other LINQ queries?

EF is trying to translate your query to SQL and as there is no direct mapping between your method and the generated SQL you're getting the error.
First option would be not to use the method and instead write the contents of the method directly in the original query (I'm not sure at the moment if this would work, as I don't have a VS running). In the case this would work, you'll most likely end up with a very complicated SQL with a poor performance.
So here comes the second option: don't be afraid to use multiple queries to get what you need. Sometimes it also makes sense to send a simpler query to the DB and continue with modifications (aggregation, selection, ...) in the C# code. The query gets translated to SQL everytime you try to enumerate over it or if you use one of the ToList, ToDictionary, ToArray, ToLookup methods or if you're using a First, FirstOrDefault, Single or SingleOrDefault calls (see the LINQ documentation for the specifics).
One possible example that could fix your query (but most likely is not the best solution) is to start your query with:
var seasonHoursByYear =
from d in context.AuxiliaryDateHours.ToList()
[...]
and continue with all the rest. This minor change has fundamental impact:
by calling ToList the DB will be immediately queried and the whole
AuxiliaryDateHours table will be loaded into the application (this will be a performance problem if the table has too many rows)
a second query will be generated when calling your QuerySeasonLimits method (you could/should also include a ToList call for that)
the rest of the seasonHoursByYear query: where, grouping, ... will happen in memory
There are a couple of other points that might be unrelated at this point.
I haven't really investigated the intent of your code - as this could lead to further optimizations - even total reworks that could bring you more gains in the end...

I eliminated the SeasonLimits object and the QuerySeasonLimits method, and wrote the contents of the method directly in the original query.
// MVC Project
var seasonLimits =
from s in context.Season
.Where(s => s.Locations.Project.Project_Group.Any(pg => pg.Project_Group_ID == Project_Group_ID))
group s by new
{
s.Year
} into grp
select new
{
grp.Key.Year,
Min_Start_Date = grp.Min(x => x.Start_Date),
Min_Start_Date_ID = grp.Min(x => x.Start_Date_ID),
Max_End_Date = grp.Max(x => x.End_Date),
Max_End_Date_ID = grp.Max(x => x.End_Date_ID)
};
var seasonHoursByYear =
from d in context.AuxiliaryDateHours
from sl in seasonLimits
where d.Date_ID >= sl.Min_Start_Date_ID
&& d.Date_ID < sl.Max_End_Date_ID
group d by new
{
d.Year
} into grp4
orderby grp4.Key.Year
select new
{
Year = grp4.Key.Year,
HoursInYear = grp4.Count()
};

More Elegant LINQ Alternative to Foreach Extension

This is purely to improve my skill. My solution works fine for the primary task, but it's not "neat". I'm currently working on a .NET MVC with Entity framework project. I know only basic singular LINQ functions which have sufficed over the years. Now I'd like to learn how to fancy.
So I have two models
public class Server
{
[Key]
public int Id { get; set; }
public string InstanceCode { get; set; }
public string ServerName { get; set; }
}
public class Users
{
[Key]
public int Id { get; set; }
public string Name { get; set; }
public int ServerId { get; set; } //foreign key relationship
}
In one of my view models I was asked to provide a dropdown list for selecting a server when creating a new user. The drop down list populated with text and value Id as an IEnumerable
Here's my original property for dropdown list of servers
public IEnumerable<SelectListItem> ServerItems
{
get { Servers.ToList().Select(s => new selectListItem { Value = x.Id.ToString(), Text = $"{s.InstanceCode}#{s.ServerName}" }); }
}
Update on requirements, now I need to display how many users are related to each server selection. Ok no problem. Here's what I wrote off the top of my head.
public IEnumerable<SelectListItem> ServerItems
{
get
{
var items = new List<SelectListItem>();
Servers.ToList().ForEach(x => {
var count = Users.ToList().Where(t => t.ServerId == x.Id).Count();
items.Add(new SelectListItem { Value = x.Id.ToString(), Text = $"{x.InstanceCode}#{x.ServerName} ({count} users on)" });
});
return items;
}
}
This gets my result lets say "localhost#rvrmt1u (8 Users)" but thats it..
What if I want to sort this dropdown list by user count. All I'm doing is another variable in the string.
TLDR ... I'm sure that someone somewhere can teach me a thing or two about converting this to a LINQ Query and making it look nicer. Also bonus points for knowing how I could sort the list to show servers with the most users on it first.

OK, we have this mess:
var items = new List<SelectListItem>();
Servers.ToList().ForEach(x => {
var count = Users.ToList().Where(t => t.ServerId == x.Id).Count();
items.Add(new SelectListItem { Value = x.Id.ToString(), Text = $"{x.InstanceCode}#{x.ServerName} ({count} users on)" });
});
return items;
Make a series of small, careful, obviously-correct refactorings that gradually improve the code.
Start with: Let's abstract those complicated operations to their own methods.
Note that I've replaced the unhelpful x with the helpful server.
int UserCount(Server server) =>
Users.ToList().Where(t => t.ServerId == server.Id).Count();
Why on earth is there a ToList on Users? That looks wrong.
int UserCount(Server server) =>
Users.Where(t => t.ServerId == server.Id).Count();
We notice that there is a built-in method that does these two operations together:
int UserCount(Server server) =>
Users.Count(t => t.ServerId == server.Id);
And similarly for creating an item:
SelectListItem CreateItem(Server server, int count) =>
new SelectListItem
{
Value = server.Id.ToString(),
Text = $"{server.InstanceCode}#{server.ServerName} ({count} users on)"
};
And now our property body is:
var items = new List<SelectListItem>();
Servers.ToList().ForEach(server =>
{
var count = UserCount(server);
items.Add(CreateItem(server, count);
});
return items;
Already much nicer.
Never use ForEach as a method if you're just going to pass a lambda body! There's already a built-in mechanism in the language that does it better! There is no reason to write items.Foreach(item => {...}); when you could simply write foreach(var item in items) { ... }. It's simpler and easier to understand and debug, and the compiler can optimize it better.
var items = new List<SelectListItem>();
foreach (var server in Servers.ToList())
{
var count = UserCount(server);
items.Add(CreateItem(server, count);
}
return items;
Much nicer.
Why is there a ToList on Servers? Completely unnecessary!
var items = new List<SelectListItem>();
foreach(var server in Servers)
{
var count = UserCount(server);
items.Add(CreateItem(server, count);
}
return items;
Getting better. We can eliminate the unnecessary variable.
var items = new List<SelectListItem>();
foreach(var server in Servers)
items.Add(CreateItem(server, UserCount(server));
return items;
Hmm. This gives us an insight that CreateItem could be doing the count itself. Let's rewrite it.
SelectListItem CreateItem(Server server) =>
new SelectListItem
{
Value = server.Id.ToString(),
Text = $"{server.InstanceCode}#{server.ServerName} ({UserCount(server)} users on)"
};
Now our prop body is
var items = new List<SelectListItem>();
foreach(var server in Servers)
items.Add(CreateItem(server);
return items;
And this should look familiar. We have re-invented Select and ToList:
var items = Servers.Select(server => CreateItem(server)).ToList();
Now we notice that the lambda can be replaced with the method group:
var items = Servers.Select(CreateItem).ToList();
And we have reduced that whole mess to a single line that clearly and unambiguously looks like what it does. What does it do? It creates an item for every server and puts them in a list. The code should read like what it does, not how it does it.
Study the techniques I used here carefully.
Extract complex code to helper methods
Replace ForEach with real loops
Eliminate unnecessary ToLists
Revisit earlier decisions when you realize there's an improvement to be made
Recognize when you are re-implementing simple helper methods
Don't stop with one improvement! Each improvement makes it possible to do another.
What if I want to sort this dropdown list by user count?
Then sort it by user count! We abstracted that away into a helper method, so we can use it:
var items = Servers
.OrderBy(UserCount)
.Select(CreateItem)
.ToList();
We now notice that we're calling UserCount twice. Do we care? Maybe. It could be a perf problem to call it twice, or, horrors, it might not be idempotent! If either are a problem then we need to undo a decision we made before. It's easier to deal with this situation in comprehension mode rather than fluent mode, so let's rewrite as a comprehension:
var query = from server in Servers
orderby UserCount(server)
select CreateItem(server);
var items = query.ToList();
Now we go back to our earlier:
SelectListItem CreateItem(Server server, int count) => ...
and now we can say
var query = from server in Servers
let count = UserCount(server)
orderby count
select CreateItem(server, count);
var items = query.ToList();
and we are only calling UserCount once per server.
Why go back to comprehension mode? Because to do this in fluent mode makes a mess:
var query = Servers
.Select(server => new { server, count = UserCount(server) })
.OrderBy(pair => pair.count)
.Select(pair => CreateItem(pair.server, pair.count))
.ToList();
And it looks a little ugly. (In C# 7 you could use a tuple instead of an anonymous type, but the idea is the same.)

The trick with LINQ is just to type return and go from there. Don't create a list and add items to it; there is usually a way to select it all in one go.
public IEnumerable<SelectListItem> ServerItems
{
get
{
return Servers.Select
(
server =>
new
{
Server = server,
UserCount = Users.Count( u => u.ServerId = server.Id )
}
)
.Select
(
item =>
new SelectListItem
{
Value = item.Server.Id.ToString(),
Text = string.Format
(
#"{0}{1} ({2} users on)" ,
item.Server.InstanceCode,
item.Server.ServerName,
item.UserCount
)
}
);
}
}
In this example there are actually two Select statements-- one to extract the data, and one to do the formatting. In an ideal situation the logic for those two tasks would be separated into different layers, but this is an OK compromise.

Sorting data issue

So I have a little issue in sorting some data I have. In a Telerik Grid, I have a column called Requestor that displays the name of a person or Unit (group of people). The problem is, Requestor has two sources it can get it's data from. Here are the two sources.
1.) RequestorId: This is a foreign key to a table called Customer. Here, I store all the data for the user, including their full name. This field can be null btw.
2.) UnitId: This is another foreign key to a table called Units. Here, I store all the data for the Units, particularlly their names. This field can be null btw.
Here is the logic:
//Entity class that contains all the data for my grid
var purchaseOrders = _purchaseOrders.GetPurchaseOrders();
//Key is Id of PurchaseOrders, Value is name of requestor
var dictionary = new Dictionary<int, string>();
foreach (var purchaseOrder in purchaseOrders)
{
if (purchaseOrder.requestorId != null)
dictionary.add(purchaseOrder.Requestor.Fullname);
else
dictionary.add(purchaseOrder.unit.Fullname);
}
dictionary.orderby(x => x.value).ToDictionary(x => x.Key, x.Value);
var tempPurchaseOrders = new List<PurchaseOrder>();
foreach (var item in dictionary)
{
tempPurchaseOrders.Add(purchaseOrders.Where(x => x.Id == item.Key).FirstOrDefault());
}
purchaseOrders = tempPurchaseOrders.AsQueryable();
return purchaseOrders;
This logic returns an ordered list based on what I want to do, however, the problem is the amount of time it takes to process. It takes 1 minute to process. That's horrible obviously. Is there anyway to optimize this? I cut down the source after I return for the grid because there is no logical way to really cut it down beforehand.
Any help would be appreciated. Thanks.
Edit: I found out I no longer am required to use the RequestName field. That limits the data to two areas now. Still a minute to process though.

Did you try something like this:
return _purchaseOrders.GetPurchaseOrders().Select(i => new
{
OrderColumn = i.requestorId != null ? purchaseOrder.Requestor.Fullname : purchaseOrder.unit.Fullname,
// map other columns
})
.OrderBy(i => i.OrderColumn);

A bit like Sławomir Rosiek's solution (but entity framework won't accept that statement):
return _purchaseOrders.GetPurchaseOrders()
.OrderBy(o => o.unit.Fullname).ToList();
(since you don't use RequestName anymore).
Especially when GetPurchaseOrders() is an IQueryable from EF you delegate the sorting to the database engine because the sort expression becomes part of the SQL statement.

So I came up with my own solution. I first tried what both Sławomir Rosiek and Gert Arnold did. Unfortunately, like Gert mentioned, the first answer would not go through. The second one had similar issues.
In the end, I created a class to store the data from both Requestors and Units. It consisted of the following:
internal class RequestorData
{
public int entityId { get; set; }
public string Name { get; set; }
public bool isRequestorId { get; set; }
}
Next, I did the following.
//Entity class that contains all the data for my grid
var purchaseOrders = _purchaseOrders.GetPurchaseOrders();
var tempPurchaseOrders = new List<PurchaseOrder>();
var requestors = new List<RequestorData>();
var customers = purchaseOrders.Select(po => po.Requestor).Distinct().ToList();
var units = purchaseOrders.Select(po => po.Unit).Distinct().ToList();
foreach (var customer in customers)
{
if (customer != null)
requestors.Add(new RequestorData { entityId = customer.Id, Name = customer.FullName, isRequestorId = true });
}
foreach (var unit in units)
{
if (unit != null)
requestors.Add(new RequestorData { entityId = unit.Id, Name = unit.FullName, isRequestorId = false });
}
requestors = requestors.OrderBy(r => r.Name).ToList();
foreach (var requestor in requestors)
{
var id = requestor.entityId;
if (requestor.isRequestorId)
tempPurchaseOrders.AddRange(purchaseOrders.Where(po => po.RequestorId == id).ToList());
else
tempPurchaseOrders.AddRange(purchaseOrders.Where(po => po.UnitId == id).ToList());
}
purchaseOrders = tempPurchaseOrders.AsQueryable();
return purchaseOrders;
I ran this new rendition and have a 5-6 second time of wait. That's not perfect but much better than before. Thanks for all the help.

Object mapping with LINQ and SubSonic

I'm building a small project with SubSonic 3.0.0.3 ActiveRecord and I'm running into an issue I can't seem to get past.
Here is the LINQ query:
var result = from r in Release.All()
let i = Install.All().Count(x => x.ReleaseId == r.Id)
where r.ProductId == productId
select new ReleaseInfo
{
NumberOfInstalls = i,
Release = new Release
{
Id = r.Id,
ProductId = r.ProductId,
ReleaseNumber = r.ReleaseNumber,
RevisionNumber = r.RevisionNumber,
ReleaseDate = r.ReleaseDate,
ReleasedBy = r.ReleasedBy
}
};
The ReleaseInfo object is a custom class and looks like this:
public class ReleaseInfo
{
public Release Release { get; set; }
public int NumberOfInstalls { get; set; }
}
Release and Install are classes generated by SubSonic.
When I do a watch on result, the Release property is null.
If I make this a simpler query and watch result, the value is not null.
var result = from r in Release.All()
let i = Install.All().Count(x => x.ReleaseId == r.Id)
where r.ProductId == productId
select new Release
{
Id = r.Id,
ProductId = r.ProductId,
ReleaseNumber = r.ReleaseNumber,
RevisionNumber = r.RevisionNumber,
ReleaseDate = r.ReleaseDate,
ReleasedBy = r.ReleasedBy
};
Is this an issue with my LINQ query or a limitation of SubSonic?

I think the issue might be that you're essentially duplicating the functionality of the ORM. The key thing to understand is this line:
from r in Release.All()
This line returns a list of fully-populated Release records for every item in your database. There should never be a need to new up a release anywhere else in your query - just return the ones that SubSonic has already populated for you!
Using this logic, you should be able to do the following:
var result = from r in Release.All()
select new ReleaseInfo {
Release = r,
NumberOfInstalls = Install.All().Count(x => x.ReleaseId == r.Id)
};
That being said, you should look at the Install.All() call, because that's likely to be tremendously inefficient. What that will do is pull every install from the database, hydrate those installs into objects, and then compare the id of every record in .NET to check if the record satisfies that condition. You can use the .Find method in SubSonic to only return certain records at the database tier, which should help performance significantly. Even still, inflating objects may still be expensive and you might want to consider a view or stored procedure here. But as a simple first step, the following should work:
var result = from r in Release.All()
select new ReleaseInfo {
Release = r,
NumberOfInstalls = Install.Find(x => x.ReleaseId == r.Id).Count()
};

I think I've found the actual answer to this problem. I've been rummaging around in the SubSonic source and found that there are two types of object projection that are used when mapping the datareader to objects: one for anonymous types and groupings and one for everything else:
Here is a snippet: Line 269 - 298 of SubSonic.Linq.Structure.DbQueryProvider
IEnumerable<T> result;
Type type = typeof (T);
//this is so hacky - the issue is that the Projector below uses Expression.Convert, which is a bottleneck
//it's about 10x slower than our ToEnumerable. Our ToEnumerable, however, stumbles on Anon types and groupings
//since it doesn't know how to instantiate them (I tried - not smart enough). So we do some trickery here.
if (type.Name.Contains("AnonymousType") || type.Name.StartsWith("Grouping`") || type.FullName.StartsWith("System.")) {
var reader = _provider.ExecuteReader(cmd);
result = Project(reader, query.Projector);
} else
{
using (var reader = _provider.ExecuteReader(cmd))
{
//use our reader stuff
//thanks to Pascal LaCroix for the help here...
var resultType = typeof (T);
if (resultType.IsValueType)
{
result = reader.ToEnumerableValueType<T>();
}
else
{
result = reader.ToEnumerable<T>();
}
}
}
return result;
Turns out that the SubSonic ToEnumerable tries to match the column names in the datareader to the properties in the object you're trying to project to. The SQL Query from my Linq looks like this:
SELECT [t0].[Id], [t0].[ProductId], [t0].[ReleaseDate], [t0].[ReleasedBy], [t0].[ReleaseNumber], [t0].[RevisionNumber], [t0].[c0]
FROM (
SELECT [t1].[Id], [t1].[ProductId], [t1].[ReleaseDate], [t1].[ReleasedBy], [t1].[ReleaseNumber], [t1].[RevisionNumber], (
SELECT COUNT(*)
FROM [dbo].[Install] AS t2
WHERE ([t2].[ReleaseId] = [t1].[Id])
) AS c0
FROM [dbo].[Release] AS t1
) AS t0
WHERE ([t0].[ProductId] = 2)
Notice the [t0].[c0] is not the same as my property name NumberOfInstalls. So the value of c0 never gets projected into my object.
THE FIX:
You can simply take out the if statement and use the 10x slower projection and everything will work.

We have a bug with projections that trips on certain occassions - I think it's been patched but I need to test it more. I invite you to try the latest bits - I think we've fixed it... sorry to be so vague but a bug worked it's way in between 3.0.0.1 and 3.0.0.3 and I haven't been able to find it.

Has this been fixed in 3.0.0.4? I was so peeved to find this post. After 2 days of trying to figure out why my projections were not working - except when the property names matched the query exactly - I ended up here.
I am so dependant on SS SimpleRepository that it is too late to turn back now. A bug like this is crippling. Any chance it is sorted out?
I went the 10x slower route for now so I can at least release to my client. Would much prefer the faster method to work correctly :)

Programmatically load a LINQ2SQL partial class

I am working on project that allows a user to add Time to a Task. On the Task I have a field for EstimatedDuration, and my thoughts are I can get ActualDuration from the Time added to the Task.
I have a LINQ2SQL class for the Task, as well as and additional Task class (using partials).
I have the following for my query so far:
public IQueryable<Task> GetTasks(TaskCriteria criteria)
{
// set option to eager load child object(s)
var opts = new System.Data.Linq.DataLoadOptions();
opts.LoadWith<Task>(row => row.Project);
opts.LoadWith<Task>(row => row.AssignedToUser);
opts.LoadWith<Task>(row => row.Customer);
opts.LoadWith<Task>(row => row.Stage);
db.LoadOptions = opts;
IQueryable<Task> query = db.Tasks;
if (criteria.ProjectId.HasValue())
query = query.Where(row => row.ProjectId == criteria.ProjectId);
if (criteria.StageId.HasValue())
query = query.Where(row => row.StageId == criteria.StageId);
if (criteria.Status.HasValue)
query = query.Where(row => row.Status == (int)criteria.Status);
var result = query.Select(row => row);
return result;
}
What would be the best way to get at the ActualDuration, which is just a sum of the Units in the TaskTime table?

Add a property to your Task partial class, similiar to this:
public int ActualDuration
{
get {
YourDataContext db = new YourDataContext();
return
db.TaskDurations.Where(t => t.task_id == this.id).
Sum (t => t.duration);
}
}
Then you can reference the actual duration as Task.ActualDuration.
Update: You asked about how to do this with a partial class. Of course it hits the database again. The only way to get data from the database that you don't know yet is to hit the database. If you need to avoid this for performance reasons, write a subquery or SQL function that calculates the actual duration, and use a Tasks view that includes the calculated value. Now, the function/query will still have to aggregate the entered durations for every task row in the result set, so it will still be performance-intensive. If you have a very large task table and performance issues, keep a running tally on the task table. I think that even the partial class solution is fine for several 100,000's of tasks. You rarely retrieve large numbers at once, I would assume. Grid controls with paging only get a page at the time.