Develop Reference

EF 6 - Performance of GroupBy

I don't have a problem currently, but I want to make sure, that the performance is not too shabby for my issue. My search on Microsofts documentation was without any success.
I have a Entity of the name Reservation. I now want to add some statistics to the program, where I can see some metrics about the reservations (reservations per month and favorite spot/seat in particular).
Therefore, my first approach was the following:
public async Task<ICollection<StatisticElement<Seat>>> GetSeatUsage(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return await this.FetchGroupedSeatData(allReservations, company);
}
public async Task<ICollection<StatisticElement<DateTime>>> GetMonthlyReservations(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return this.FetchGroupedReservationData(allReservations);
}
private async Task<ICollection<StatisticElement<Seat>>> FetchGroupedSeatData(
IEnumerable<Reservation> reservations,
Company company)
{
var groupedReservations = reservations.GroupBy(r => r.SeatId).ToList();
var companySeats = await this.seatService.GetAll(company);
return (from companySeat in companySeats
let groupedReservation = groupedReservations.FirstOrDefault(s => s.Key == companySeat.Id)
select new StatisticElement<Seat>()
{
Value = companySeat,
StatisticalCount = groupedReservation?.Count() ?? 0,
}).OrderByDescending(s => s.StatisticalCount).ToList();
}
private ICollection<StatisticElement<DateTime>> FetchGroupedReservationData(IEnumerable<Reservation> reservations)
{
var groupedReservations = reservations.GroupBy(r => new { Month = r.Date.Month, Year = r.Date.Year }).ToList();
return groupedReservations.Select(
groupedReservation => new StatisticElement<DateTime>()
{
Value = new DateTime(groupedReservation.Key.Year, groupedReservation.Key.Month, 1),
StatisticalCount = groupedReservation.Count(),
}).
OrderBy(s => s.Value).
ToList();
}
To explain the code a little bit: With GetSeatUsage and GetMonthlyReservations I can get the above mentioned data of a company. Therefore, I fetch ALL reservations at first (with reservationService.GetAll) - this is the point, where I think the performance will be a problem in the future.
Afterwards, I call either FetchGroupedSeatData or FetchGroupedReservationData, which first groups the reservations I previously fetched from the database and then converts them in a, for me, usable format.
As I said, I think the group by after I have read ALL the data from the database MIGHT be a problem, but I cannot find any information regarding performance in the documentation.
My other idea was, that I create a new method in my ReservationService, which then already returns the grouped list. But, again, I can't find the information, that the EF adds the GroupBy to the DB Query or basically does it after all of the data has been read from the database. This method would look something like this:
return await this.Context.Set<Reservation>.Where(r => r.User.CompanyId == company.Id).GroupBy(r => r.SeatId).ToListAsync();
Is this already the solution? Where can I check that? Am I missing something completely obvious?

Appending to the Select part of a query

We have a duplicate part of our LINQ METHOD syntax query. Here is a contrived example.
IQueryable<orders> query = _context.Set<orders>();
var result = query.Select(p => new{
REMAINING = p.qtyOrdered + p.alreadysent,
AWATING = p.qtyOrdered + p.alreadysent
}).ToList();
So we are trying to resolve the duplicate part by putting something in a method and then calling that and getting some sort of result. So something like this....
private IQueryable WhatsLeft()
{
IQueryable<orders> query = _context.Set<orders>();
return query.Select(p => new{p.qtyOrdered + p.alreadysent});
}
IQueryable<orders> query = _context.Set<orders>();
var result = query.Select(p => new{
REMAINING = WhatsLeft(),
AWATING = WhatsLeft()
}).ToList();
Is this at all possible and if so can anyone give me some brief advise on how I would achieve this.

Wouldn't you just simply pass the Order object to the new function directly?
private int Total(Order order)
{
return order.qtyOrdered + order.alreadySent;
}
IQueryable<orders> query = _context.Set<orders>();
var result = query.Select(p => new{
REMAINING = Total(p),
AWATING = Total(p)
}).ToList();
If I understand what you're after correctly. I can't remember off the top of my head how well Linq to sql etc can handle functions, interpreting them into SQL functions. Maybe you could give it a try.
Alternatively, to reduce the complexity of the function (to facilitate L2S conversion) you can make the parameters granular on the function such as:
private int Total(int left, int right)
{
return left + right;
}
Then make the call more like:
var result = query.Select(p => new{
REMAINING = Total(p.qtyOrdered, p.alreadysent),
AWATING = Total(p.qtyOrdered, p.alreadysent)
}).ToList();
UPDATE:
Have you thought about querying the calculation up front?
var result = query.Select(c => c.qtyOrdered + c.alreadysent).Select(p => new {
REMAINING = p,
AWAITING = p
}).ToList();

Find and count from array in another array using linq C#

I have the following scenario:
I get a collection of records from a data source and add them to a List
List<Entity> e = new List<Entity>();
foreach (var elm in datasource)
{
e.Add(new Entity { First = elm.First, Second = elm.Second, Third = elm.Third });
}
This gives me something like:
// -> First = John, Second = Sally, Third = Ingrid
// -> First = Sally, Second = Ingrid, Third = James
// -> First = Ingrid, Second = Sally, Third = James
I have an array of possible values that can be true and possible values that can be negative, in a List
List<string> positives = { "John", "Sally" }
List<string> negatives = { "Ingrid", "James" }
I am looking for an elegant why of matching the amount of positives and the amount of negatives I have in the generic list e, as per the above? Would something like this be possible with LINQ?
As I am not familiar with LINQ have no idea where to start. I tried something like:
var pos = e.SelectMany(s => s in positives).Count()
But this throws an exception.
Any help would be greatly appreciated

SQL's in clause in LINQ is Contains method.
Test:
var pos = e.SelectMany(s => positives.Contains(s)).Count();
Seems that I haven't understood your question. Reading again.
To be honest, your question in not well-phrased.
But based on what I understood, I think the correct response is:
var pos = e.SelectMany(i => positives.Contains(i.First)
|| positives.Contains(i.Second)
|| positives.Contains(i.Third)).Count();

You are using in incorrectly
e.SelectMany(s => positives.Contains(s)).Count();
but what I assue you are wanting is this
var pos = e.Count(s => positives.Contains(s.First) || positives.Contains(s.Second) ||positives.Contains(s.Third));
var neg= e.Count(s => negatives.Contains(s.First) || negatives.Contains(s.Second) ||negatives.Contains(s.Third));

or you can use an any
var pos = e.SelectMany(i =>
positives.Any(p=> p==i.First || p==i.Second || p==i.Third))
.Count();

How to select multiple properties properly?

MyObject have two property named p1 and p2 in int type ;now I want for each of MyObject take p1 and p2 and add those up. I tried this:
int p1Sum = 0, p2Sum = 0;
foreach (int[] ps in new MyEntity().MyObject.Select(o => new { o.p1, o.p2 }))
{
p1Sum += ps[0];
p2Sum += ps[1];
}
but says:
cannot convert AnonymousType#1 to int[]
on foreach.
How can I fix this?

foreach (var ps in new MyEntity().MyObject.Select(o => new { o.p1, o.p2 }))
{
p1Sum += ps.p1;
p2Sum += ps.p2;
}

jyparask's answer will definitely work, but it's worth considering using Sum twice instead - it will involve two database calls, but it may (check!) avoid fetching all the individual values locally:
var entities = new MyEntity().MyObject;
var p1Sum = entities.Sum(x => x.p1);
var p2Sum = entities.Sum(x => x.p2);
Now there's at least logically the possibility of inconsistency here - some entities may be removed or added between the two Sum calls. However, it's possible that EF will ensure that doesn't happen (e.g. via caching) or it may not be relevant in your situation. It's definitely something you should think consider.

In addition to Jon Skeet and jyparask answer you can also try :
var result = (new MyEntity().MyObject
.GroupBy(_=> 0)
.Select(r=> new
{
p1Sum = r.Sum(x=> x.p1)
p2Sum = r.Sum(x=> x.p2)
})
.FirstOrDefault();
The above would result in a single query fetching only Sum for both columns, You may look at the query generated and its execution plan if you are concerned about the performance.
if(result != null)
{
Console.WriteLine("p1Sum = " + result.p1Sum);
Console.WriteLine("p2Sum = " + result.p2Sum);
}

NHibernate get first x distinct results does not get me x results?

I have an ICriteria query like so:
var contentCriteria = DetachedCriteria.For<InvoiceItem>();
var countCriteria = DetachedCriteria.For<InvoiceItem>();
if (model.CurrentPage <= 0) model.CurrentPage = 1;
if (model.OnlyShowErrors)
{
contentCriteria.Add(Restrictions.Not(Restrictions.Eq("TroubleClass", TroubleClasses.Success)));
countCriteria.Add(Restrictions.Not(Restrictions.Eq("TroubleClass", TroubleClasses.Success)));
}
if (!string.IsNullOrEmpty(model.BatchId))
{
contentCriteria.Add(Restrictions.Eq("BatchId", model.BatchId));
countCriteria.Add(Restrictions.Eq("BatchId", model.BatchId));
}
if (model.DocumentStartDate != null)
{
contentCriteria.Add(Restrictions.Ge("DocumentDate", model.DocumentStartDate));
countCriteria.Add(Restrictions.Ge("DocumentDate", model.DocumentStartDate));
}
if (model.DocumentEndDate != null)
{
contentCriteria.Add(Restrictions.Le("DocumentDate", model.DocumentEndDate));
countCriteria.Add(Restrictions.Le("DocumentDate", model.DocumentEndDate));
}
if (!string.IsNullOrEmpty(model.VendorId))
{
contentCriteria.Add(Restrictions.Eq("VendorId", model.VendorId));
countCriteria.Add(Restrictions.Eq("VendorId", model.VendorId));
}
using (var session = GetSession())
{
var countC = countCriteria.GetExecutableCriteria(session)
.SetProjection(Projections.CountDistinct("RecordId"));
var contentC = contentCriteria
.AddOrder(Order.Desc("PersistedTimeStamp"))
.GetExecutableCriteria(session)
.SetResultTransformer(Transformers.DistinctRootEntity)
.SetFirstResult((model.CurrentPage * model.ItemsPerPage) - model.ItemsPerPage)
.SetMaxResults(model.ItemsPerPage);
var mq = session.CreateMultiCriteria()
.Add("total", countC)
.Add<InvoiceItem>("paged", contentC);
model.Invoices = ((IEnumerable<InvoiceItem>)mq.GetResult("paged"));
model.Invoices = model.Invoices
.OrderBy(x => x.PersistedTimeStamp);
model.TotalItems = (int)(mq.GetResult("total") as System.Collections.ArrayList)[0];
}
return model;
This returns results, but where I would expect the results to be in groups of model.ItemsPerPage, it rarely is. I think that the .SetResultTransformer(Transformers.DistinctRootEntity) transform is being run after the .SetMaxResults(model.ItemsPerPage) limit, and I don't know why or how to fix it. Can someone please enlighten me?

You need to see the SQL generated by NHibernate as this is not essentially NHibernate bug but behavior of SQL queries when ROWNUM and DISTINCT are applied together. This has been an issue in our known issues list from long.
Following URLs might enlighten you...
ROW NUMBER vs DISTINCT
How ROWNUM works

So this is directly related to what was written in this blog post. Additionally, I had the platform-specific complication of PostgreSQL not allowing a DISTINCT ordered set ordered by something not in the SELECT list. Ultimately, I had to make two calls to the database, like so:
using (var session = GetSession())
{
//I honestly hope I never have to reverse engineer this mess. Pagination in NHibernate
//when ordering by an additional column is a nightmare.
var countC = countCriteria.GetExecutableCriteria(session)
.SetProjection(Projections.CountDistinct("RecordId"));
var contentOrdered = contentCriteria
.SetProjection(Projections.Distinct(
Projections.ProjectionList()
.Add(Projections.Id())
.Add(Projections.Property("PersistedTimeStamp"))
))
.AddOrder(Order.Desc("PersistedTimeStamp"))
.SetFirstResult((model.CurrentPage * model.ItemsPerPage) - model.ItemsPerPage)
.SetMaxResults(model.ItemsPerPage);
var contentIds = contentOrdered.GetExecutableCriteria(session)
.List().OfType<IEnumerable<object>>()
.Select(s => (Guid)s.First())
.ToList();
var contentC = DetachedCriteria.For<InvoiceItem>()
.Add(Restrictions.In("RecordId", contentIds))
.SetResultTransformer(Transformers.DistinctRootEntity);
var mq = session.CreateMultiCriteria()
.Add("total", countC)
.Add("paged", contentC);
model.Invoices = (mq.GetResult("paged") as System.Collections.ArrayList)
.OfType<InvoiceItem>()
.OrderBy(x => x.PersistedTimeStamp);
model.TotalItems = (int)(mq.GetResult("total") as System.Collections.ArrayList)[0];
}
return model;
This is not pretty, but it worked; I think the folks over at NHibernate need to work on this and make it a tad bit easier.