Develop Reference

Adjust .NET Linq results list so it shows ASC future then DESC past list which can be skipped for a single page

I've read the many related questions but can't find this exactly.
I'm trying to adjust the ordering on an OLD .NET (4.0) web page so it shows events that are upcoming ASC (showing closest in the future first), followed by events in the past showing them DESC in a single list that is skipped to take a 'page' of results.
So like this:
Event 1 - tomorrow
Event 2 - in a week
Event 3 - in a month
Event 4 - yesterday
Event 5 - a week ago
Event 6 - a month ago
The current function grabs the list and does a sort, skip and take (a single page):
// Currently this creates a single list order by date desc
var numToLoad = 20;
var list = _context.AllItems.Where(i => i.typeTitle == source);
var items = list.OrderByDescending(i => i.dateCreated).Skip((pageNum - 1) * numToLoad).Take(numToLoad);
I have tried making two lists, ordering each appropriately, and then concatenating them, but then I can't do a skip, as that requires a sorted list.
// We need a single list but with upcoming dates first, ascending, then past dates descending
var dateNow = DateTime.Now;
var listFuture = _context.AllItems.Where(i => i.typeTitle == source && i.dateCreated >= dateNow).OrderBy(i => i.dateCreated);
var listPast = _context.AllItems.Where(i => i.typeTitle == source && i.dateCreated < dateNow).OrderByDescending(i => i.dateCreated);
var listAll = listFuture.Concat(listPast);
var itemsAll = listAll.Skip((pageNum - 1) * numToLoad).Take(numToLoad); // <-- this gives an error as it's not sorted
So I don't have to rewrite all the code that handles the returned list (pagination etc) I'd really like to be able to return a single list from the function!
I did see that it might be possible to do conditional sorting, then do the skip and take in a single linq but I just can't get anything like that to work.
Any ideas?

The problem here is that .Concat() of two queries loose ordering when translated to SQL (you cannot to UNION two queries with different order each).
If you are using MSSQL, you can use ordering like that:
var dateNow = DateTime.Now;
var query = _context.AllItems
.Where(i => i.typeTitle == source)
// future items first
.OrderBy(i => i.dateCreated >= dateNow ? 0 : 1)
// items which closer to now first
.ThenBy(i => Math.Abs(System.Data.Entity.SqlServer.SqlFunctions.DateDiff("day", dateNow, i.dateCreated).Value));
var list = query.Skip((page - 1) * pageSize).Take(pageSize).ToList();

How to filter list that has date and retrieve only data within 7 days

So I have this model:
Student Model
public int StudentId {get; set;}
public string StudentName {get; set;}
public DateTime EnrollDate {get; set;}
I also have a list of student Model which is something like
List<Student> listOfStudents = new List<Student>();
and inside that list there are 100 students detail and the enroll date.
What I do next is to sort the list into showing from the latest one to the oldest one.
listOfStudents.Sort((x, y) => DateTime.Compare(y.EnrollDate, x.EnrollDate));
and it's working. However, I am currently struggling in showing only the EnrollDate within 7 days from Today.

Conceptually, I think of LINQ a lot like SQL. You have the SELECT portion, which is your projection (i.e. what am I pulling out of this set of data?). If you omit the Select() clause from LINQ, you'll get the whole record vs. only a portion if you wanted to pluck out only pieces of it. You have your WHERE portion which is a limiter, or filter condition that when applied to the set pulls back only the records that satisfy said condition. And lastly, there are operations you can apply that affect the order of the returned set. That's where the OrderBy() and OrderByDescending() come into play. So lets map those concepts to the examples below
No Select(), but we do have a Where() and an OrderBy()
var then = DateTime.Now.AddDays(-7); //One portion of our Where. More below
var sortedStudents = listOfStudents
//Our predicate. 's' = the Student passed to the function. Give me only the students
//where s.EnrollDate is greater or equal to the variable 'then' (defined above)
.Where(s => s.EnrollDate >= then)
//We have no Select statement, so return whole students
//And order them by their enrollment date in ascending order
.OrderBy(s => s.EnrollDate);
When run, sortedStudents will be loaded up only with students (entire Student objects, not a projection) that meet our Where() criteria. The Where() function takes predicate that specifies our criteria. A predicate is simply a function that accepts a record from the set that we're filtering, and returns a bool indicating whether or not it should be included.
Let's change the filter by adjusting the Where()
//Notice we've changed 'then' from 7 days ago to a fixed point in time: 26 June 2018
var then = new DateTime.Parse("26 June 2018");
var sortedStudents = listOfStudents
.Where(s => s.EnrollDate >= then)
//Still no Select(). We'll do that next
.OrderBy(s => s.EnrollDate);
Just like before sortedStudents will have whole Student records, but this time it will only contain those enrolled after or on 26 June 2018, as specified by our predicate.
Let's add a Select()
var then = new DateTime.Parse("26 June 2018");
var dates = listOfStudents
.Where(s => s.EnrollDate >= then)
.Select(s => s.EnrollDate);
Now we've changed it so that instead of pulling back a whole Student we're only plucking out the EnrollDate. Notice I've changed the name of the receiving variable from sortedStudents to dates reflecting the fact that it now only contains a list of DateTime objects.
You could still replace .OrderBy() with .OrderByDescending() to change the order.

How about breaking down the problem into 2 sub-problems?
Sub-problem #1
showing only the EnrollDate within 7 days from Today
We only need Students whose EnrollDate property is within 7 days from today:
var today = DateTime.UtcNow;
sevenDaysOldList = listOfStudents.Where(x => (today - x.EnrollDate).TotalDays < 7);
The subtraction of the two dates results in a TimeSpan with a TotalDays property, which we can use to determine the number of days elapsed between the two dates.
Sub-problem #2
sort the list into showing from the latest one to the oldest one.
We need to sort sevenDaysOldList by EnrollDate in descending order:
sevenDaysOldList.Sort((x, y) => y.EnrollDate.CompareTo(x.EnrollDate));
..which will sort the list in place. OrderByDescending is a good candidate for this (it returns a new ordered list implementing IOrderedEnumerable<T>):
sevenDaysOldList.OrderByDescending(x => x.EnrollDate);
// and of course .OrderBy(x => x.EnrollDate) for ascending order
Combine #1 & #2
You can now combine the solutions of the two sub-problems into one. How you do it is at your own discretion. This is how I would do it:
var sevenDaysOldList = listOfStudents.Where(x => (today - x.EnrollDate).TotalDays < 7)
.OrderByDescending(x => x.EnrollDate);
Update: question in comment
How do I modify/sort the list that remove all the list less than "26 June 2018" ? So the list will only have data date greater than 26 June 2018. Any data with date before 26 June will be removed
You can initialize that date in a DateTime variable, and use it with List<T>.RemoveAll(Predicate<T>), to remove items in sevenDaysOldList which are smaller than that date:
var filterDate = new DateTime(2018, 06, 26);
sevenDaysOldList.RemoveAll(x => x.EnrollDate < filterDate);

LINQ: Group by date including the year

I have dataset looks like this:
FileName Date
ABC - 01/10/16
DBC - 01/11/16
ZYX - 03/10/16
ABX2 - 01/10/17
IOS - 01/09/17
How can I group them into a list of groups of months while ensuring that the year is taken into account in the clause?
I'm currently using a LINQ Query is creating groups by month but not including the year, so I have a group of ABC, ZYX and ABX2. even though ABX2 was a 2017 report but the same month so should be in a different group.
I've been trying different ways of doing this but none of have been successful as of yet.
var newList = from x in list
group x
by x.Properties.LastModified.Value.Month into lastMod
where lastMod.Count() > 1
select lastMod;
Once I have them in separate groups, I will find out which one was written last and save that and remove the rest. I'm quite stuck and been on this issue for half day. would appreciate fresh eyes on it.

You can group by a composite year-month key, like this:
var newList = list
.Where(x => x.Properties.LastModified.HasValue)
.GroupBy(x => new {
x.Properties.LastModified.Value.Month
, x.Properties.LastModified.Value.Year
})
.Where(g => g.Count() > 1);
You need to ensure that LastModified has non-null value before accessing its Value property.

I can't test this at the moment but I think grouping by an anonymous type that contains both the month and year should do it.
var newList = from x in list
group x
by new {x.Properties.LastModified.Value.Month, x.Properties.LastModified.Value.Year} into lastMod
where lastMod.Count() > 1
select lastMod;

Fetch every nth row with LINQ

We have a table in our SQL database with historical raw data I need to create charts from. We access the DB via Entity Framework and LINQ.
For smaller datetime intervals, I can simply read the data and generate the charts:
var mydata = entity.DataLogSet.Where(dt => dt.DateTime > dateLimit);
But we want to implement a feature where you can quickly "zoom out" from the charts to include larger date intervals (last 5 days, last month, last 6 months, last 10 years and so on and so forth.)
We don't want to chart every single data point for this. We want to use a sample of the data, by which I mean something like this --
Last 5 days: chart every data point in the table
Last month: chart every 10th data point in the table
Last 6 months: chart every 100th data point
The number of data points and chart names are only examples. What I need is a way to pick only the "nth" row from the database.

You can use the Select overload that includes the item index of enumerations. Something like this should do the trick --
var data = myDataLogEnumeration.
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
If you need to limit the query with a Where or sort with OrderBy, you must do it before the first Select, otherwise the indexes will be all wrong --
var data = myDataLogEnumeration.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Unfortunately, as juharr commented, this overload is not supported in Entity Framework. One way to deal with this is to do something like this --
var data = entity.DataLogSet.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
ToArray().
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Note the addition of a ToArray(). This isn't ideal though as it will force loading all the data that matches the initial query before selecting only every nth row.

There might be a trick that is supported by ef that might work for this.
if (step != 0)
query = query.Where(_ => Convert.ToInt32(_.Time.ToString().Substring(14, 2)) % step == 0);
this code converts the date into string then cuts the minutes out converts the minutes into an int and then gets every x'th minute for example if the variable step is 5 it's every 5 minutes.
For Postgresql this converts to:
WHERE ((substring(c.time::text, 15, 2)::INT % #__step_1) = 0)
this works best with fixed meassure points such as once a minute.
However, you can use the same method to group up things by cutting up to the hour or the minutes or the first part of the minute (10 minutes grouped) and use aggregation functions such as max() average() sum(), what might even is more desirable.
For example, this groups up in hours and takes the max of most but the average of CPU load:
using var ef = new DbCdr.Context();
IQueryable<DbCdr.CallStatsLog> query;
query = from calls in ef.Set<DbCdr.CallStatsLog>()
group calls by calls.Time.ToString().Substring(0, 13)
into g
orderby g.Max(_ => _.Time) descending
select new DbCdr.CallStatsLog()
{
Time = g.Min(_ => _.Time),
ConcurrentCalls = g.Max(_ => _.ConcurrentCalls),
CpuUsage = (short)g.Average(_ => _.CpuUsage),
ServerId = 0
};
var res = query.ToList();
translates to:
SELECT MAX(c.time) AS "Time",
MAX(c.concurrent_calls) AS "ConcurrentCalls",
AVG(c.cpu_usage::INT::double precision)::smallint AS "CpuUsage",
0 AS "ServerId"
FROM call_stats_log AS c
GROUP BY substring(c.time::text, 1, 13)
ORDER BY MAX(c.time) DESC
note: the examples work with postgres and iso datestyle.

LINQ to SQL exception: Local sequence cannot be used in LINQ to SQL implementations of query operators except the Contains operator

I know this is a duplicate on SO, but I can't figure out how to use the contains operator in my specific code:
I have 5 bookings in the database:
ID, Booking, Pren, ReservationCode
1, VisitHere, 1, 1000A
2, VisitHere, 1, 1000A
3, VisitHere, 1, 1000A
4, VisitThere, 2, 2000A
5, VisitThere, 2, 2000A
public int SpecialDelete(DataContext db, IEnumerable<BookingType> bookings) {
var rescodes = (from b in bookings
select b).Distinct().ToArray();
// Code Breaks here
IEnumerable<BookingType> bookingsToDelete = db.GetTable<BookingType>().Where(b => bookings.Any(p => p.Pren == b.Pren && p.ReservationCode == b.ReservationCode));
int deleted = bookingsToDelete.Count();
db.GetTable<BookingType>().DeleteAllOnSubmit(bookingsToDelete);
db.SubmitChanges();
return deleted;
}
When I pass the first record into this method (1, VisitHere, 1, 1000A), I want it to retrieve ids 1,2 and 3, but not 4 and 5.
I can do this by matching Pren and ReservationCode.
How can I do this as the .Any and .All operators are throwing the above exception?
Note: The method must accept a list of bookings because the argument will always be multiple bookings passed into the method, I just used a single booking as an example.
Edit: I basically need LINQ2SQL to generate a bunch of SQL statements like so (let's say I want to delete all records in my DB):
DELETE
FROM Bookings b
WHERE b.ReservationCode = '1000A' AND b.Pren = 1
DELETE
FROM Bookings b
WHERE b.ReservationCode = '2000A' AND b.Pren = 2

The error you are getting is trying to direct you to use the .Contains method passing in a simple array. By default it translates that array into an In clause in the format:
Where foo In ("b1", "B2", "B3")
Notice here that you can't do a multi-dimentional array in the In clause (as you would need to do). Since you can't join server side to a local array, your options become limited as long as you have a composite key relationship.
If you don't need to fetch the rows in order to delete them, it will probably be faster anyway to just use Context's ExecuteCommand to issue your deletes. Just make sure to parameterize your query (see http://www.thinqlinq.com/Post.aspx/Title/Does-LINQ-to-SQL-eliminate-the-possibility-of-SQL-Injection)
string deleteQuery = "DELETE FROM Bookings b WHERE b.ReservationCode = {0} AND b.Pren = {1}";
foreach (var bookingType in bookings)
{
db.ExecuteCommand(deleteQuery, bookingType.ReservationCode, bookingType.Preen);
}

What if you have a quasi temp table on the server. You can put the list values in there.
This is a real problem with ORMs. You have a lot is mismatch between local and remote capabilities.
I have tried even using .Range to generated a remote list to join against, but it doesn't work either.
Essentially you have to rearrange your data islands somehow ( i.e. where does the lists of pren and rs come from? Is it on the server somewhere ? ) or upload one of your local collections to a staging area on the server.

The error message says "except the contains operator." Have you considered using the Contains operator? It should do the same thing.
So from
IEnumerable<BookingType> bookingsToDelete = db.GetTable<BookingType>().Where(b => bookings.Any(p => p.Pren == b.Pren && p.ReservationCode == b.ReservationCode));
to
IEnumerable<BookingType> bookingsToDelete = db.GetTable<BookingType>().Where(b => bookings.Contains(p => p.Pren == b.Pren && p.ReservationCode == b.ReservationCode));
I realise that the list wont contain the same objects so you may need to do something like:
bookings.Select(booking => booking.PrimaryKeyOfAwesome).Contains(b => b.PrimaryKeyOfAwesome) etc etc.
Edited for clarity
Edit for humility
Ok, so after actually recreating the entire setup I realised that my solution doesnt work because of the two parameter, not just one. Apologies. This is what I came up with in the end, which works, but is genuinely a terrible solution and should not be used. I include it here only for closure ;)
public static int SpecialDelete(DataContext db, IEnumerable<BookingType> bookings)
{
var compositeKeys = bookings.Select(b => b.Pren.ToString() + b.ReservationCode).Distinct();
IEnumerable<BookingType> bookingsToDelete = db.GetTable<BookingType>().Where(b => compositeKeys.Contains(b.Pren.ToString() + b.ReservationCode));
int deleted = bookingsToDelete.Count();
db.GetTable<BookingType>().DeleteAllOnSubmit(bookingsToDelete);
db.SubmitChanges();
return deleted;
}