Develop Reference

EF 6 - Performance of GroupBy

I don't have a problem currently, but I want to make sure, that the performance is not too shabby for my issue. My search on Microsofts documentation was without any success.
I have a Entity of the name Reservation. I now want to add some statistics to the program, where I can see some metrics about the reservations (reservations per month and favorite spot/seat in particular).
Therefore, my first approach was the following:
public async Task<ICollection<StatisticElement<Seat>>> GetSeatUsage(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return await this.FetchGroupedSeatData(allReservations, company);
}
public async Task<ICollection<StatisticElement<DateTime>>> GetMonthlyReservations(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return this.FetchGroupedReservationData(allReservations);
}
private async Task<ICollection<StatisticElement<Seat>>> FetchGroupedSeatData(
IEnumerable<Reservation> reservations,
Company company)
{
var groupedReservations = reservations.GroupBy(r => r.SeatId).ToList();
var companySeats = await this.seatService.GetAll(company);
return (from companySeat in companySeats
let groupedReservation = groupedReservations.FirstOrDefault(s => s.Key == companySeat.Id)
select new StatisticElement<Seat>()
{
Value = companySeat,
StatisticalCount = groupedReservation?.Count() ?? 0,
}).OrderByDescending(s => s.StatisticalCount).ToList();
}
private ICollection<StatisticElement<DateTime>> FetchGroupedReservationData(IEnumerable<Reservation> reservations)
{
var groupedReservations = reservations.GroupBy(r => new { Month = r.Date.Month, Year = r.Date.Year }).ToList();
return groupedReservations.Select(
groupedReservation => new StatisticElement<DateTime>()
{
Value = new DateTime(groupedReservation.Key.Year, groupedReservation.Key.Month, 1),
StatisticalCount = groupedReservation.Count(),
}).
OrderBy(s => s.Value).
ToList();
}
To explain the code a little bit: With GetSeatUsage and GetMonthlyReservations I can get the above mentioned data of a company. Therefore, I fetch ALL reservations at first (with reservationService.GetAll) - this is the point, where I think the performance will be a problem in the future.
Afterwards, I call either FetchGroupedSeatData or FetchGroupedReservationData, which first groups the reservations I previously fetched from the database and then converts them in a, for me, usable format.
As I said, I think the group by after I have read ALL the data from the database MIGHT be a problem, but I cannot find any information regarding performance in the documentation.
My other idea was, that I create a new method in my ReservationService, which then already returns the grouped list. But, again, I can't find the information, that the EF adds the GroupBy to the DB Query or basically does it after all of the data has been read from the database. This method would look something like this:
return await this.Context.Set<Reservation>.Where(r => r.User.CompanyId == company.Id).GroupBy(r => r.SeatId).ToListAsync();
Is this already the solution? Where can I check that? Am I missing something completely obvious?

LINQ querying using an entity object not in the database - Error: LINQ to Entities does not recognize the method

I have an Entity Framework v5 model created from a database. The table Season has a corresponding entity called Season. I need to calculate the Season's minimum start date and maximum end date for each year for a Project_Group. I then need to be able to JOIN those yearly min/max Season values in other LINQ queries. To do so, I have created a SeasonLimits class in my Data Access Layer project. (A SeasonLimits table does not exist in the database.)
public partial class SeasonLimits : EntityObject
{
public int Year { get; set; }
public DateTime Min_Start_Date { get; set; }
public int Min_Start_Date_ID { get; set; }
public DateTime Max_End_Date { get; set; }
public int Max_End_Date_ID { get; set; }
public static IQueryable<SeasonLimits> QuerySeasonLimits(MyEntities context, int project_Group_ID)
{
return context
.Season
.Where(s => s.Locations.Project.Project_Group.Any(pg => pg.Project_Group_ID == project_Group_ID))
.GroupBy(x => x.Year)
.Select(sl => new SeasonLimits
{
Year = sl.Key,
Min_Start_Date = sl.Min(d => d.Start_Date),
Min_Start_Date_ID = sl.Min(d => d.Start_Date_ID),
Max_End_Date = sl.Max(d => d.End_Date),
Max_End_Date_ID = sl.Max(d => d.End_Date_ID)
});
}
}
// MVC Project
var seasonHoursByYear =
from d in context.AuxiliaryDateHours
from sl in SeasonLimits.QuerySeasonLimits(context, pg.Project_Group_ID)
where d.Date_ID >= sl.Min_Start_Date_ID
&& d.Date_ID < sl.Max_End_Date_ID
group d by new
{
d.Year
} into grp4
orderby grp4.Key.Year
select new
{
Year = grp4.Key.Year,
HoursInYear = grp4.Count()
};
In my MVC project, whenever I attempt to use the QuerySeasonLimits method in a LINQ query JOIN, I receive the message,
"LINQ to Entities does not recognize the method
'System.Linq.IQueryable`1[MyDAL.SeasonLimits]
QuerySeasonLimits(MyDAL.MyEntities, MyDAL.Project_Group)' method, and
this method cannot be translated into a store expression."
Is this error being generated because SeasonLimits is not an entity that exists in the database? If this can't be done this way, is there another way to reference the logic so that it can be used in other LINQ queries?

EF is trying to translate your query to SQL and as there is no direct mapping between your method and the generated SQL you're getting the error.
First option would be not to use the method and instead write the contents of the method directly in the original query (I'm not sure at the moment if this would work, as I don't have a VS running). In the case this would work, you'll most likely end up with a very complicated SQL with a poor performance.
So here comes the second option: don't be afraid to use multiple queries to get what you need. Sometimes it also makes sense to send a simpler query to the DB and continue with modifications (aggregation, selection, ...) in the C# code. The query gets translated to SQL everytime you try to enumerate over it or if you use one of the ToList, ToDictionary, ToArray, ToLookup methods or if you're using a First, FirstOrDefault, Single or SingleOrDefault calls (see the LINQ documentation for the specifics).
One possible example that could fix your query (but most likely is not the best solution) is to start your query with:
var seasonHoursByYear =
from d in context.AuxiliaryDateHours.ToList()
[...]
and continue with all the rest. This minor change has fundamental impact:
by calling ToList the DB will be immediately queried and the whole
AuxiliaryDateHours table will be loaded into the application (this will be a performance problem if the table has too many rows)
a second query will be generated when calling your QuerySeasonLimits method (you could/should also include a ToList call for that)
the rest of the seasonHoursByYear query: where, grouping, ... will happen in memory
There are a couple of other points that might be unrelated at this point.
I haven't really investigated the intent of your code - as this could lead to further optimizations - even total reworks that could bring you more gains in the end...

I eliminated the SeasonLimits object and the QuerySeasonLimits method, and wrote the contents of the method directly in the original query.
// MVC Project
var seasonLimits =
from s in context.Season
.Where(s => s.Locations.Project.Project_Group.Any(pg => pg.Project_Group_ID == Project_Group_ID))
group s by new
{
s.Year
} into grp
select new
{
grp.Key.Year,
Min_Start_Date = grp.Min(x => x.Start_Date),
Min_Start_Date_ID = grp.Min(x => x.Start_Date_ID),
Max_End_Date = grp.Max(x => x.End_Date),
Max_End_Date_ID = grp.Max(x => x.End_Date_ID)
};
var seasonHoursByYear =
from d in context.AuxiliaryDateHours
from sl in seasonLimits
where d.Date_ID >= sl.Min_Start_Date_ID
&& d.Date_ID < sl.Max_End_Date_ID
group d by new
{
d.Year
} into grp4
orderby grp4.Key.Year
select new
{
Year = grp4.Key.Year,
HoursInYear = grp4.Count()
};

error in calling and calculate function with int? as input

I have a function like that :
public int? calculateContractPrice(int? comid)
{
int? sum = 0;
var q = from i in dbconnect.tblMaterialGroups
where i.tenderId == _tenderId
select i.id;
foreach (int i in q )
{
var q2 = from g in dbconnect.tblMaterialTenderAnnouncePrices
where g.MaterialGroupId == i && g.companyId == comid
select g;
sum = q2.First().amount*q2.First().price + q2.First().amount*q2.First().PriceForElse + sum;
}
return sum ;
}
When i try to execute this :
List<presentationcontract> q = (from i in dbconnect.tblContracts
where i.tender == _tenderId
select new presentationcontract()
{
tax =(calculateContractPrice(i.companyId)*(6/100)).ToString()
}).ToList();
Tax is string .after executing i got this error :
couldn't translate expression calculateContractPrice(i.companyId)*(6/100),invoke(value(system.Func1[system.nullable1[system.Int32]]))).ToString() into SQL and could not treat it as a local expression

Your edit makes clear the issue. You're trying to do
tax =(calculateContractPrice(i.companyId)*(6/100)).ToString()
in a sql statement but calculateContractPrice is in c#! To understand what's going on you really need to understand a bit how LINQ works.
First of all, stop using the silly sql-style syntax for LINQ. It is less powerful than the lambda syntax and hides what is going on under the hood in a way that makes it hard to understand.
Second consider a LINQ statement
users.Where(u => u.Name == "George").ToList();
where users is IEnumerable<User>. What happens here is that the lambda part is of type Func<User, bool> and gets compiled to a method that gets run against every single instance of User.
Now consider this LINQ statement
db.Users.Where(u => u.Name == "George").ToList();
where db.Users is IQueryable<T>. This looks the same but what happens is VERY different. What happens is that lambda is actually of type Expression<Func<User, bool>> this doesn't get compiled to a method, instead it gets compiled into something called an expression tree. This gets passed to the LINQ provider (in your case Entity Framework I'm guessing) which examines it and converts that into a SQL statement
SELECT Id, Name, Address FROM users WHERE Name = 'George'
What is happening in your case is that it sees the call to calculateContractPrice and simply has no way of converting that to SQL.
What you should therefore do is ensure the query runs first, then use the IEnumerable<T> form of LINQ that runs in c# to call your method.
var contracts = dbconnect.tblContracts.Where(i => i.tender == _tenderId)
.ToList() //Query executes here, now you have IEnumerable<T>
.Select(i => new PresentationContract {
Tax = ...
}).ToList(); //this ToList is only necessary if you want to prevent multiple iteration
You will want to solve all the other problems everyone else pointed out as well of course.
A few other notes - you will want to read up on .Net naming conventions. Usually, anything public,protected, or internal (classes, fields, properties, etc.) is recommended to be PascalCase. Also you probably want to move the division portion into the PresentationContract class. This class that has a Tax property should probably be the one that knows how to generate it.

Try this:
int? ret = calculateContractPrice(i.companyId);
if(ret.HasValue)
{
tax =(ret.Value*(6/100)).ToString();
}
You should make sure that the function indeed returned a value and then you use that integer value in calculation.

Return best fit item from collection in C# 3.5 in just a line or two

Here is some sample code I have basically written thousands of times in my life:
// find bestest thingy
Thing bestThing;
float bestGoodness = FLOAT_MIN;
foreach( Thing x in arrayOfThings )
{
float goodness = somefunction( x.property, localvariable );
if( goodness > bestGoodness )
{
bestGoodness = goodness;
bestThing = x;
}
}
return bestThing;
And it seems to me C# should already have something that does this in just a line. Something like:
return arrayOfThings.Max( delegate(x)
{ return somefunction( x.property, localvariable ); });
But that doesn't return the thing (or an index to the thing, which would be fine), that returns the goodness-of-fit value.
So maybe something like:
var sortedByGoodness = from x in arrayOfThings
orderby somefunction( x.property, localvariable ) ascending
select x;
return x.first;
But that's doing a whole sort of the entire array and could be too slow.
Does this exist?

This is what you can do using System.Linq:
var value = arrayOfThings
.OrderByDescending(x => somefunction(x.property, localvariable))
.First();
If the array can be empty, use .FirstOrDefault(); to avoid exceptions.
You really don't know how this is implemented internally, so you can't assure this will sort the whole array to get the first element. For example, if it was linq to sql, the server would receive a query including the sort and the condition. It wouldn't get the array, then sort it, then get the first element.
In fact, until you don't call First, the first part of the query isn't evaluated. I mean this isn't a two steps evaluation, but a one step evaluation.
var sortedValues =arrayOfThings
.OrderByDescending(x => somefunction(x.property, localvariable));
// values isn't still evaluated
var value = sortedvalues.First();
// the whole expression is evaluated at this point.

I don't think this is possible in standard LINQ without sorting the enuermable (which is slow in the general case), but you can use the MaxBy() method from the MoreLinq library to achieve this. I always include this library in my projects as it is so useful.
http://code.google.com/p/morelinq/source/browse/trunk/MoreLinq/MaxBy.cs
(The code actually looks very similar to what you have, but generalized.)

I would implement IComparable<Thing> and just use arrayOfThings.Max().
Example here:
http://msdn.microsoft.com/en-us/library/bb347632.aspx
I think this is the cleanest approach and IComparable may be of use in other places.
UPDATE
There is also an overloaded Max method that takes a projection function, so you can provide different logic for obtaining height, age, etc.
http://msdn.microsoft.com/en-us/library/bb534962.aspx

I followed the link Porges listed in the comment, How to use LINQ to select object with minimum or maximum property value and ran the following code in LINQPad and verified that both LINQ expressions returned the correct answers.
void Main()
{
var things = new Thing [] {
new Thing { Value = 100 },
new Thing { Value = 22 },
new Thing { Value = 10 },
new Thing { Value = 303 },
new Thing { Value = 223}
};
var query1 = (from t in things
orderby GetGoodness(t) descending
select t).First();
var query2 = things.Aggregate((curMax, x) =>
(curMax == null || (GetGoodness(x) > GetGoodness(curMax)) ? x : curMax));
}
int GetGoodness(Thing thing)
{
return thing.Value * 2;
}
public class Thing
{
public int Value {get; set;}
}
Result from LinqPad

Generic parsing of Expressiontree to SQL

So, im having this very simple linq provider that does stuff with objects in a small constrained domain. No need for a full blown Linq2Sql implementation, but some of the stuff i need to do involves sql queries. My thougt was that, if you look away from table and column names, most expression tree => sql must be very (very) generic.
Expression<Func<DateTime, bool>> expr = d => d.DateTime.Month == 1
will always translate to
DATEPART({0}, m) = 1
or
Expression<Func<DateTime?, bool>> expr = d => d.HasValue || d == DateTime.Now
into
{0} IS NOT NULL OR {0} = NOW()
hope you get the point. Is there any generic sql generators like this out there?

hehe, dont we just looove hacking around? So, its not FULLY working yet, i need to figure out how to dynamically inject some mapping information, but so far it looks promising.
I've been able to produce this sql
SELECT t0.[Born], t0.[Yo]
FROM [Heies] AS t0
WHERE ((MONTH(t0.[Born]) = 1) OR (NOT (t0.[Yo] IS NULL OR t0.[Yo] = '') AND (t0.
[Born] = #p0)))
with this code
var provider = new DbEntityProvider(new SqlConnection(), new TSqlLanguage(), new ImplicitMapping(), new QueryPolicy());
var exp = provider.GetTable<Hey>().Where(d => d.Born.Month == 1 || (!String.IsNullOrEmpty(d.Yo) && d.Born == DateTime.Now)).Expression;
var sql = ((QueryProvider)provider).GetQueryText(exp);
using the IQToolkit, so thanks to Damian for reminding me about it again! If anyone has a similar way to produce sql via Linq2Sql i would love to hear from you so i could loose the dependency on IQToolkit.
Update
After a lot of fooling around i have hit the wall at extracting the values for each of the parameters. After a lot of source reading it seems like its hidden far away for the actual execution of the query. Well, guess what, i don't want the query to be executed :/
After looking into Linq2Sql i came up with this code that does roughly the same, so maybe i should just stick with that
var xml = Generate<NewsProperty>();
var mapping = XmlMappingSource.FromUrl(xml);
var ctx = new DataContext(new SqlConnection("..."), mapping);
var query = ctx.GetTable<NewsProperty>().Where(n => n.Date.Year == 2010).OrderBy(n => n.Date).Take(5);
var cmd = ctx.GetCommand(query);
string sql = cmd.CommandText;
foreach (DbParameter param in cmd.Parameters)
{
sql = sql.Replace(param.ParameterName, param.Value.ToString());
}

You should take a look at the IQueryable Toolkit and its related articles by Matt Warren http://blogs.msdn.com/b/mattwar/archive/2008/11/18/linq-links.aspx.