I have a list of items in a SharePoint which has Title, Date and Priority. I would like to sort this collection on my client application while displaying it ,sorting need to first by Date Descending and then within a set of objects for a date, it should sort objects by Priority.
I am using below LINQ code, which is not providing the expected result. Means the below code not sorting object by priority with in a date.
var results = spObjects.OrderByDescending(n => n.Date).ThenBy(t => t.Priority);
Unsorted collection is as below
Sorted Collection - Expectation
I am using SharePoint CAML query, since I am fetching these object from SP and I found it difficult to do it in CAML, but I assume this can be done using LINQ on C# side, Isnt it ? Could anyone please help ?
I'll bet you that n.Date is a DateTime object, with hours, minutes, and seconds. Your Linq looks good, so you probably have different times that are causing items later in the day with lower priorities to be sorted before items earlier in the day.
Try something like:
var results = spObjects.OrderByDescending(n => new DateTime(n.Date.Year, n.Date.Month, n.Date.Day)).ThenBy(t => t.Priority);
If I understand you correctly you want a nested collection under a date heading, you might do something like this...
var results = from so in spObjects
orderby so.Date descending
select new {
Date = so.Date,
Items = (from so2 in spObjects
where so2.Date == so.Date
orderby so2.Priority
select new {
so2.Title,
so2.Priority
})
};
Related
I have a database table with over 200K+ records and a column containing a Date (NOT NULL). I am struggling to do a GroupBy Date since the database is massive the query takes soooo long to process (like 1 minute or so).
My Theory:
Get the list of all records from that table
From that list find the end date and the start date (basically the oldest date and the newest)
Then taking say like 20 dates to do the GroupBy on so the query will be done in a shorter set of records..
Here is my Model that I have to get the list:
registration.Select(c => new RegistrationViewModel()
{
DateReference = c.DateReference,
MinuteWorked = c.MinuteWorked,
});
The DateReferenceis the database column that I have to work with...
I am not pretty sure how to cycle through my list getting the dates start and end without taking too long.
Any idea on how to do that?
EDIT:
var registrationList = await context.Registration
.Where(c => c.Status == StatusRegistration.Active) // getting all active registrations
.ToRegistrationViewModel() // this is simply a select method
.OrderBy(d => d.DateReference.Date) // this takes long
.ToListAsync();
The GroupBy:
var grpList = registrationList.GroupBy(x => x.DateReference.Date).ToList();
var tempList = new List<List<RegistrationViewModel>>();
foreach (var item in grpList)
{
var selList = item.Select(c => new RegistrationViewModel()
{
RegistrationId = c.RegistrationId,
DateReference = c.DateReference,
MinuteWorked = c.MinuteWorked,
}).ToList();
tempList.Add(selList);
}
This is my SQL table:
This is the ToRegistrationViewModel() function:
return registration.Select(c => new RegistrationViewModel()
{
RegistrationId = c.RegistrationId,
PeopleId = c.PeopleId,
DateReference = c.DateReference,
DateChange = c.DateChange,
UserRef = c.UserRef,
CommissionId = c.CommissionId,
ActivityId = c.ActivityId,
MinuteWorked = c.MinuteWorked,
Activity = new ActivityViewModel()
{
Code = c.Activity.Code,
Description = c.Activity.Description,
},
Commission = new CommissionViewModel()
{
Code = c.Commission.Code,
Description = c.Commission.Description
},
People = new PeopleViewModel()
{
UserId = c.People.UserId,
Code = c.People.Code,
Name = c.People.Name,
Surname = c.People.Surname,
Active = c.People.Active
}
});
There are multiple potential problems here
Lack of indexes
Your query uses the Status and DateReference, and neither looks to have an index. If there are only a few active statuses a index on that column might suffice, otherwise you need a index on the date to speedup sorting. You might also consider a composite index that includes both columns. An appropriate index should solve the sorting issue.
Materializing the query
ToListAsync will trigger the execution of the sql query, making every subsequent operation run on the client. I would also be highly suspicious of ToRegistrationViewModel, I would try changing this to an anonymous type, and only convert to an actual type after the query has been materialized. Running things like sorting and grouping on the client is generally considered a bad idea, but you need to consider where the actual bottleneck is, optimizing the grouping will not help if the transfer of data takes most time.
Transferring data
Fetching a large number of rows will be slow, no matter what. The goal is usually to do as much filtering in the database as possible so you do not need to fetch so many rows. If you have to fetch a large amount of records you might use Pagination, i.e. combine OrderBy with Skip and Take to fetch smaller chunks of data. This will not save time overall, but can allow for things like progress and showing data continuously.
I know this could be a possible duplicate question, pardon me if it is.
Is there a way to GroupBy all the records from the database by date?
So:
say i have multiple records for this date 22/05/2022
and say i have multiple records from this date: 23/05/2022
Can i group all the records based on date parameter 22/05 and 23/05?
So that i would end up with a list containing n list for each day.
Here is what i did:
var grpQuery = await ctx.Registration.GroupBy(c => c.DateReference.Day).ToListAsync();
Where:
Registration is my table from where i am pulling the data
DateReference is a Date object containing the date
But i am getting this error "the linq expession could not be translated".
Can somone give me some advice on this?
EDIT
I tried this but it seems not to load any data, even setting break a break point will not return anything:
var grpQuery = await query.GroupBy(d => new { DateReference = d.DateReference.Date }).Select(c => new RegistrationViewModel()
{
RegistrationId = c.FirstOrDefault().RegistrationId,
PeopleId = c.FirstOrDefault().PeopleId,
DateReference = c.Key.DateReference,
DateChange = c.FirstOrDefault().DateChange,
UserRef = c.FirstOrDefault().UserRef,
CommissionId = c.FirstOrDefault().CommissionId,
ActivityId = c.FirstOrDefault().ActivityId,
MinuteWorked = c.FirstOrDefault().MinuteWorked,
}).OrderBy(d => d.DateReference).ToListAsync();
Where:
RegistrationViewModel contains all those properties including DateReference
If i call the method using the API is stuck at "pending"
First, don't. Even if the query was fixed, the equivalent query in the database would be GROUP BY DATEPART(day,registration.Date) which can't use indexes and therefore is slow.
According to the docs the equivalent of DATEPART(day, #dateTime) is dateTime.Day. The query still needs to have a proper Select though.
A correct query would be :
counts = ctx.Registrations.GroupBy(r=>r.RegistrationDate.Day)
.Select(g=>new {Day=g.Key,Count=g.Count())
.ToList();
The equivalent, slow query would be
SELECT DATEPART(day,registration.Date) as Day,count(*)
FROM Registrations
GROUP BY DATEPART(day,registration.Date)
Things get worse if we have to eg filter by date. The query would have to scan the entire table because it wouldn't be able to use any indexes covering the Date column
SELECT DATEPART(day,registration.Date) as Day,count(*)
FROM Registrations
WHERE Date >'20220901'
GROUP BY DATEPART(day,registration.Date)
Imagine having to scan 10 years of registrations only to get the current month.
This is a reporting query. For date related reports, using a prepopulated Calendar table can make the query infinitely easier.
SELECT Calendar.Day,COUNT(*)
FROM Registrations r
INNER JOIN Calendar on r.RegistrationDate=Calendar.Date
GROUP BY Calendar.Day
or
SELECT Calendar.Year, Calendar.Semester, Calendar.Day,COUNT(*)
FROM Registrations r
INNER JOIN Calendar on r.RegistrationDate=Calendar.Date
WHERE Calendar.Year = #someYear
GROUP BY Calendar.Year, Calendar.Semester,Calendar.Day
A Calendar table or Date dimension is a table with prepopulated dates, years, months, semesters or any other reporting period along with their names or anything needed to make reporting easier. Such a table can contain eg 10 or 20 years of data without taking a lot of space. To speed up queries, the columns can be aggressively indexed without taking too much extra space.
Doing the same in EF Core requires mapping Calendar as an entity and performing the JOIN in LINQ. This is one of the cases where it makes no sense to add a relation between entities :
var query=from registration in ctx.Registrations
join date in Calendar
on registration.Date equals Calendar.Date
group registration by date.Day into g
select new { Day=g.Key, Count=g.Count()};
var counts = query.ToList();
If you are using EF Core Please try this:
var grpQuery = await ctx.Registration.Select(a=>new {Re = a, G = (EF.Functions.DateDiffDay(a.End,DateTime.Today))}).ToListAsync().ContinueWith(d=>d.Result.GroupBy(a=>a.G));
I am querying using EF linq as below for Movie records can have many Reviews:
var movieSummaries = _db.Movies.AsNoTracking().Select(l => new MovieSummary
{
Name = l.Name,
Rating = l.Reviews.OrderByDescending(r => r.Date).First().ReviewText
});
However, I don't need the dates to be ordered first, I just need the review with the latest date. Is there a way to do it in a more optimized manner?
You can try use a linq aux variable, with compare with max date reviews.
Something like this:
var moviesSumaries = from m in Movies
let Rating = m.Reviews.FirstOrDefault(d => .Date==m.Reviews.Max(s=>s.Date))?.ReviewText
select new {m.Name, Rating};
You could use lambda expressions too if you want.
Personaly, I think your propousal of use orderbydescending it's correct. Always depend on the scenary, row amount in table, db index, etc...
I have a linq query which seems to be reversing one column of several in some rows of an earlier query:
var dataSet = from fb in ds.Feedback_Answers
where fb.Feedback_Questions.Feedback_Questionnaires.QuestionnaireID == criteriaType
&& fb.UpdatedDate >= dateFeedbackFrom && fb.UpdatedDate <= dateFeedbackTo
select new
{
fb.Feedback_Questions.Feedback_Questionnaires.QuestionnaireID,
fb.QuestionID,
fb.Feedback_Questions.Text,
fb.Answer,
fb.UpdatedBy
};
Gets the first dataset and is confirmed working.
This is then grouped like this:
var groupedSet = from row in dataSet
group row by row.UpdatedBy
into grp
select new
{
Survey = grp.Key,
QuestionID = grp.Select(i => i.QuestionID),
Question = grp.Select(q => q.Text),
Answer = grp.Select(a => a.Answer)
};
While grouping, the resulting returnset (of type: string, list int, list string, list int) sometimes, but not always, turns the question order back to front, without inverting answer or questionID, which throws it off.
i.e. if the set is questionID 1,2,3 and question A,B,C it sometimes returns 1,2,3 and C,B,A
Can anyone advise why it may be doing this? Why only on the one column? Thanks!
edit: Got it thanks all! In case it helps anyone in future, here is the solution used:
var groupedSet = from row in dataSet
group row by row.UpdatedBy
into grp
select new
{
Survey = grp.Key,
QuestionID = grp.OrderBy(x=>x.QuestionID).Select(i => i.QuestionID),
Question = grp.OrderBy(x=>x.QuestionID).Select(q => q.Text),
Answer = grp.OrderBy(x=>x.QuestionID).Select(a => a.Answer)
};
Reversal of a grouped order is a coincidence: IQueryable<T>'s GroupBy returns groups in no particular order. Unlike in-memory GroupBy, which specifies the order of its groups, queries performed in RDBMS depend on implementation:
The query behavior that occurs as a result of executing an expression tree that represents calling GroupBy<TSource,TKey,TElement>(IQueryable<TSource>, Expression<Func<TSource,TKey>>, Expression<Func<TSource,TElement>>) depends on the implementation of the type of the source parameter.`
If you would like to have your rows in a specific order, you need to add OrderBy to your query to force it.
How I do it and maintain the relative list order, rather than apply an order to the resulting set?
One approach is to apply grouping to your data after bringing it into memory. Apply ToList() to dataSet at the end to bring data into memory. After that, the order of subsequent GrouBy query will be consistent with dataSet. A drawback is that the grouping is no longer done in RDBMS.
I'm using Entity Framework code-first and I want to query the appointments table in my database selecting:
All the appointments on given dates
public IEnumerable<Appointment> GetAppointments(IEnumerable<DateTime> datesWithEverything)
{
using (OneClickContext context = new OneClickContext())
{
var query = from e in context.AppointmentSet
where datesWithEverything.Contains(e.StartDate)
select e;
return query;
}
}
I also need only one appointment from any other day, the first found for each day not included in the given dates.
The reason for this, is that my calendar component will be able to show in bold the days containing at least one appointment, but I don't want to load thousands of appointments since I will be looking few days (with all the details) per request.
Is this possible hitting the database once?
Any help is appreciated.
If I understand your needs correctly, you are getting all appointment data for a range of dates, and then you simply need to know all other dates that have at least one appointment in order to bold them. You aren't actually using the appointment data on those other dates just yet. Is that correct?
You are right to try to limit the number of trips to the database, but you need to balance that with not returning more data than you actually need. If I understand right and all you need are dates with appointments, you should use a second query:
public IEnumerable<DateTime> GetDatesWithAppointments()
{
using (OneClickContext context = new OneClickContext())
{
var query = (from e in context.AppointmentSet
order by e.StartDate
select e.StartDate).Distinct();
return query.ToList();
}
}
Also, it's a matter of style but the variable "query" is not needed, and you can simply:
return (from e in context.AppointmentSet
order by e.StartDate
select e.StartDate).Distinct().ToList();
Note the ToList(). Linq query execution is deferred until the first time it is enumerated and your query context is declared in a using block. You must execute the query before the using block exits in this case.
On edit, since you require an actual appointment, you should be able to get what you want in a single trip using linq's union and grouping. Union will automatically remove any duplicate entries.
public IEnumerable<Appointment> GetAppointments(IEnumerable<DateTime> datesWithEverything)
{
using (OneClickContext context = new OneClickContext())
{
var query = from e in context.AppointmentSet
where datesWithEverything.Contains(e.StartDate)
select e;
query = query.Union(from e in context.AppointmentSet
group e by DbFunctions.TruncateTime(e.StartDate) into grp
select grp.FirstOrDefault());
return query.ToList();
}
}