Lets say I have a datatable with three columns: timestamp, curveID and price. I would like to give a time and then select for each day the timestamp, curveID and price but only if all curveIDs are present.
The problem is, not for every time all the data is present, so at 10:00:00 there might be only data for curveID 1 but nothing for ID =2, and so forth.
I thought I could do the following to select the first dataset where all curveIDs are there and time is greater or equal to my criteria:
dataSet.ReadXml(#"C:\temp\Prices.xml", XmlReadMode.InferTypedSchema);
ds = dataSet.Tables[0];
var dt = ds.Clone();
int criteria = 10;
var list = ds.AsEnumerable().Where(x => x.Field<DateTime>("Timestamp").Hour >= criteria)
.GroupBy(x => new{Date = x.Field<DateTime>("Timestamp").Date, Curve = x.Field<object>("CurveID")})
.First().ToList();
However, this returns multiple records on the same day (at different times) for the same curve ID.
I would like to return only a single record for each curveID on each day at a time close to the criteria time where all curveIDs are present.
For clarity, lets say I m looking for curveID 1 & 2, if at 10:00:00 on day 1 only curveID 1 is present but curveID 2 is missing I would need to check whether at 10:01:00 both are there, if yes I take for that day the two record sets from that time. This I would have to check for every day in the database
// criteria is your integer Hour representation
var criteria = 10;
// array of curveIds to look for
var curveIds = new int[] {1, 2};
var result =
// grouping by date first
ds.GroupBy(x => x.Field<DateTime>("Timestamp").Date,
(date, items) => new { date, items = items
// items with the same timestamp go to one group
.GroupBy(i => i.Field<DateTime>("Timestamp"), (datetime, timestampItems) => new { datetime, timestampItems })
// filter by criteria
.Where(dti => dti.datetime.Hour >= criteria)
// filter by curveIds
.Where(dti => curveIds.All(cid => dti.timestampItems.Any(tsi => tsi.Field<int>("curveID") == cid)))
.OrderBy(dti => dti.datetime)
.FirstOrDefault() });
In the end you will receive a "per day" result fitting all your mentioned requirements: occurs after some criteria, have all curveIds, be earliest one.
You may want to group by Date first and then by hour using something like
group thing by new {
firstThing = x.Field<DateTime>("TimeStamp").Date,
secondThing = x.Field<DateTime>("TimeStamp").Date.Hour,
}
My syntax is probably off by a little, but that should get you moving in the right direction
Related
I have a List which contains the order details of several users such as StoreId, OrderTotal, dateModified etc. I want to add the order total which fall under same date but different time.
Below is the code which fetches data from the database. Also I have attached the output which I am getting.
An example of what I am trying to do is: if there are 2 orders for 08/09/2020 then I want to add the orderValue. The third column in the image refers to order value.
public async Task<IActionResult> OnGet()
{
try
{
StoreOrderDetails = new List<Store_Order>();
StoreOrderDetails2 = new List<Store_Order>();
var res = await UOW.CurrentContext.StoreOrder
.OrderBy(x => x.OrderModified)
.ToListAsync();
foreach (var item in res)
{
if(item.Status.ToString() == "PaymentComplete" &&
item.StoreId == defaultStoreId)
{
StoreOrderDetails.Add(item);
}
}
totalCount = StoreOrderDetails.Count();
StoreId = defaultStoreId;
StoreOrderDetails.GroupBy(x => x.OrderModified).Select(grpData =>
new Store_Order
{
OrderModified = grpData.Key,
OrderTotal = grpData.Sum(item => item.OrderTotal),
});
}
catch
{
}
return Page();
}
If that first column in your image is the date you refer to in code when you say x => x.OrderModified, you'll need to group by only the date element, excluding the time:
StoreOrderDetails.GroupBy(x => x.OrderModified.Date).Select(grpData =>
^^^^^
When you group a date that has a time too, then events will go into different groups if they are even only a nanosecond different. If you want dates to group according to the day all different things happened on, then you have to group by the date without the time. The easiest way to do this is to ask for a DateTime's Date property - it's the date, but with the time of midnight. If you fake it as if all orders occurred at midnight, then the Sum will work out
Oh, and you'll actually need to store/use the results of the group operation...
So I have this model:
Student Model
public int StudentId {get; set;}
public string StudentName {get; set;}
public DateTime EnrollDate {get; set;}
I also have a list of student Model which is something like
List<Student> listOfStudents = new List<Student>();
and inside that list there are 100 students detail and the enroll date.
What I do next is to sort the list into showing from the latest one to the oldest one.
listOfStudents.Sort((x, y) => DateTime.Compare(y.EnrollDate, x.EnrollDate));
and it's working. However, I am currently struggling in showing only the EnrollDate within 7 days from Today.
Conceptually, I think of LINQ a lot like SQL. You have the SELECT portion, which is your projection (i.e. what am I pulling out of this set of data?). If you omit the Select() clause from LINQ, you'll get the whole record vs. only a portion if you wanted to pluck out only pieces of it. You have your WHERE portion which is a limiter, or filter condition that when applied to the set pulls back only the records that satisfy said condition. And lastly, there are operations you can apply that affect the order of the returned set. That's where the OrderBy() and OrderByDescending() come into play. So lets map those concepts to the examples below
No Select(), but we do have a Where() and an OrderBy()
var then = DateTime.Now.AddDays(-7); //One portion of our Where. More below
var sortedStudents = listOfStudents
//Our predicate. 's' = the Student passed to the function. Give me only the students
//where s.EnrollDate is greater or equal to the variable 'then' (defined above)
.Where(s => s.EnrollDate >= then)
//We have no Select statement, so return whole students
//And order them by their enrollment date in ascending order
.OrderBy(s => s.EnrollDate);
When run, sortedStudents will be loaded up only with students (entire Student objects, not a projection) that meet our Where() criteria. The Where() function takes predicate that specifies our criteria. A predicate is simply a function that accepts a record from the set that we're filtering, and returns a bool indicating whether or not it should be included.
Let's change the filter by adjusting the Where()
//Notice we've changed 'then' from 7 days ago to a fixed point in time: 26 June 2018
var then = new DateTime.Parse("26 June 2018");
var sortedStudents = listOfStudents
.Where(s => s.EnrollDate >= then)
//Still no Select(). We'll do that next
.OrderBy(s => s.EnrollDate);
Just like before sortedStudents will have whole Student records, but this time it will only contain those enrolled after or on 26 June 2018, as specified by our predicate.
Let's add a Select()
var then = new DateTime.Parse("26 June 2018");
var dates = listOfStudents
.Where(s => s.EnrollDate >= then)
.Select(s => s.EnrollDate);
Now we've changed it so that instead of pulling back a whole Student we're only plucking out the EnrollDate. Notice I've changed the name of the receiving variable from sortedStudents to dates reflecting the fact that it now only contains a list of DateTime objects.
You could still replace .OrderBy() with .OrderByDescending() to change the order.
How about breaking down the problem into 2 sub-problems?
Sub-problem #1
showing only the EnrollDate within 7 days from Today
We only need Students whose EnrollDate property is within 7 days from today:
var today = DateTime.UtcNow;
sevenDaysOldList = listOfStudents.Where(x => (today - x.EnrollDate).TotalDays < 7);
The subtraction of the two dates results in a TimeSpan with a TotalDays property, which we can use to determine the number of days elapsed between the two dates.
Sub-problem #2
sort the list into showing from the latest one to the oldest one.
We need to sort sevenDaysOldList by EnrollDate in descending order:
sevenDaysOldList.Sort((x, y) => y.EnrollDate.CompareTo(x.EnrollDate));
..which will sort the list in place. OrderByDescending is a good candidate for this (it returns a new ordered list implementing IOrderedEnumerable<T>):
sevenDaysOldList.OrderByDescending(x => x.EnrollDate);
// and of course .OrderBy(x => x.EnrollDate) for ascending order
Combine #1 & #2
You can now combine the solutions of the two sub-problems into one. How you do it is at your own discretion. This is how I would do it:
var sevenDaysOldList = listOfStudents.Where(x => (today - x.EnrollDate).TotalDays < 7)
.OrderByDescending(x => x.EnrollDate);
Update: question in comment
How do I modify/sort the list that remove all the list less than "26 June 2018" ? So the list will only have data date greater than 26 June 2018. Any data with date before 26 June will be removed
You can initialize that date in a DateTime variable, and use it with List<T>.RemoveAll(Predicate<T>), to remove items in sevenDaysOldList which are smaller than that date:
var filterDate = new DateTime(2018, 06, 26);
sevenDaysOldList.RemoveAll(x => x.EnrollDate < filterDate);
We have a table in our SQL database with historical raw data I need to create charts from. We access the DB via Entity Framework and LINQ.
For smaller datetime intervals, I can simply read the data and generate the charts:
var mydata = entity.DataLogSet.Where(dt => dt.DateTime > dateLimit);
But we want to implement a feature where you can quickly "zoom out" from the charts to include larger date intervals (last 5 days, last month, last 6 months, last 10 years and so on and so forth.)
We don't want to chart every single data point for this. We want to use a sample of the data, by which I mean something like this --
Last 5 days: chart every data point in the table
Last month: chart every 10th data point in the table
Last 6 months: chart every 100th data point
The number of data points and chart names are only examples. What I need is a way to pick only the "nth" row from the database.
You can use the Select overload that includes the item index of enumerations. Something like this should do the trick --
var data = myDataLogEnumeration.
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
If you need to limit the query with a Where or sort with OrderBy, you must do it before the first Select, otherwise the indexes will be all wrong --
var data = myDataLogEnumeration.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Unfortunately, as juharr commented, this overload is not supported in Entity Framework. One way to deal with this is to do something like this --
var data = entity.DataLogSet.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
ToArray().
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Note the addition of a ToArray(). This isn't ideal though as it will force loading all the data that matches the initial query before selecting only every nth row.
There might be a trick that is supported by ef that might work for this.
if (step != 0)
query = query.Where(_ => Convert.ToInt32(_.Time.ToString().Substring(14, 2)) % step == 0);
this code converts the date into string then cuts the minutes out converts the minutes into an int and then gets every x'th minute for example if the variable step is 5 it's every 5 minutes.
For Postgresql this converts to:
WHERE ((substring(c.time::text, 15, 2)::INT % #__step_1) = 0)
this works best with fixed meassure points such as once a minute.
However, you can use the same method to group up things by cutting up to the hour or the minutes or the first part of the minute (10 minutes grouped) and use aggregation functions such as max() average() sum(), what might even is more desirable.
For example, this groups up in hours and takes the max of most but the average of CPU load:
using var ef = new DbCdr.Context();
IQueryable<DbCdr.CallStatsLog> query;
query = from calls in ef.Set<DbCdr.CallStatsLog>()
group calls by calls.Time.ToString().Substring(0, 13)
into g
orderby g.Max(_ => _.Time) descending
select new DbCdr.CallStatsLog()
{
Time = g.Min(_ => _.Time),
ConcurrentCalls = g.Max(_ => _.ConcurrentCalls),
CpuUsage = (short)g.Average(_ => _.CpuUsage),
ServerId = 0
};
var res = query.ToList();
translates to:
SELECT MAX(c.time) AS "Time",
MAX(c.concurrent_calls) AS "ConcurrentCalls",
AVG(c.cpu_usage::INT::double precision)::smallint AS "CpuUsage",
0 AS "ServerId"
FROM call_stats_log AS c
GROUP BY substring(c.time::text, 1, 13)
ORDER BY MAX(c.time) DESC
note: the examples work with postgres and iso datestyle.
Here is an example Data Table:
ID Amount Date
--------------------------
1 3.000 2016/1/1
2 4.000 2016/1/1
1 6.000 2017/1/1
1 3.000 2018/1/1
3 2.000 2019/1/1
I need to count Dates which a specific Customers does not have a Shop in it.
for example ID 2 does not have a shop in 2017/1/1 and 2018/1/1, so the count will be 2. and the count for customer ID 3 will be 3 because He does not have a shop in 2016/1/1 and 2017/1/1 and 2018/1/1
I think I should use grouping but do not know how to count which I want for a specific ID!
orders.GroupBy(x => x.Date)......???
Assuming you have list of objects:
// number of all different dates
var datesCount = list.Select(i => i.Date).Distinct().Count();
int customerId = 2;
//number of dates with customer shopping
var customerDatesCount = list.Where(i => i.ID == customerId).Select(i => i.Date).Distinct().Count();
var nonShoppingDatesCount = datesCount - customerDatesCount;
You need two steps here.
Get all distinct order dates
var allDates = orders.Select(o => o.Date).Distinct();
Find those dates which customer don't have. Except operation will do that
var missingDates = allDates.Except(orders.Where(o => o.ID == 2).Select(o => o.Date));
If you need just number of missing dates, then use missingDates.Count() or (better) use solution by #Roma
I am collecting data within every ten seconds (six records in a minute).
Using Entity Framework or LINQ, I want to obtain the average of the records within a every minute.
Simply, Date column is in (%Y.%m.%d %H:%i:%s) format, and i want to group by (%Y.%m.%d %H:%i) format in mysql using Entity Framework or LINQ
Assuming that you mean you have a date time column you can group by the DateTime properties Year, Month, Day, Hour, and Minute.
var results = from row in db.SomeTable
group by new
{
row.Date.Year,
row.Date.Month,
row.Date.Day,
row.Date.Hour,
row.Date.Minute
} into grp
select new
{
YMDHM = grp.Key
SomeAverage = grp.Average(x => x.SomeValueToAverage)
};
Then when you iterate the results you can turn the values in YMDHM back into a DateTime. Here's an example where you could turn the results into a Dictionary.
var dictionaryOfAverages = results.ToDictionary(
x => new DateTime(x.YMDHM.Year,
x.YMDHM.Month,
x.YMDHM.Day,
x.YMDHM.Hour,
x.YMDHM.Minute,
0),
x => x.SomeAverage);
On the other hand if you actually mean you are storing the date time in the DB as a formatted string then the following would be what you want
var results = from row in db.SomeTable
group by new row.Date.SubString(0, 16) into grp
select new
{
YMDHM = grp.Key
SomeAverage = grp.Average(x => x.SomeValueToAverage)
};
This is assuming that your string formatted dates look like "2013.05.08 07:25:33" If the format isn't fixed width with leading zeros for month, day, and/or hour you'd have to do something like row.Date.SubString(0, row.Date.Length - 3) instead