Develop Reference

Partition By Logic in Code to calculate value of a DataTable Column

I'm using the following SQL for calculating the value of a column named weight within a view.
I need to move this calculation logic to code.
CASE
WHEN SUM(BaseVal) OVER (PARTITION BY TEMPS.MandateCode) = 0 THEN 0
ELSE (BaseVal / (SUM(BaseVal) OVER (PARTITION BY TEMPS.MandateCode))) END AS [Weight]
Is iterating over each and grouping by MandateCode a good idea
var datatableenum = datatable.AsEnumerable();
foreach(var item in datatableenum)
{
List<DataTable> result = datatable.AsEnumerable()
.GroupBy(row => row.Field<int>("MandateCode"))
.Select(g => g.CopyToDataTable())
.ToList();
}

I'm going to say "no" because as you have it, it will perform the group operation for every mandate code, for each row then copy then to list, which adds up to a huge amount of burnt resources.. I would make a dictionary of mandatecode=>sum first and then use it when iterating the table
var d = datatable.AsEnumerable()
.GroupBy(
row => row.Field<int>("MandateCode"),
row => row.Field<double>("BaseVal")
).ToDictionary(g => g.Key, g => g.Sum());
Note I've no idea what type BaseVal is; you need to adjust this. If it's an integer remember that you'll be doing a calc of small_int/big_int eg 12/6152, which is always 0 so cast one of the operandi to eg double so the result will be like 0.1234
Then use the dictionary on each row
foreach(var item in datatableenum)
{
int sumbv = d[item.Field<int>("MandateCode"));
item["Weight"] = sumbv == 0 ? 0 : item.Field<double>("BaseVal") / sumbv;
}

Fetch every nth row with LINQ

We have a table in our SQL database with historical raw data I need to create charts from. We access the DB via Entity Framework and LINQ.
For smaller datetime intervals, I can simply read the data and generate the charts:
var mydata = entity.DataLogSet.Where(dt => dt.DateTime > dateLimit);
But we want to implement a feature where you can quickly "zoom out" from the charts to include larger date intervals (last 5 days, last month, last 6 months, last 10 years and so on and so forth.)
We don't want to chart every single data point for this. We want to use a sample of the data, by which I mean something like this --
Last 5 days: chart every data point in the table
Last month: chart every 10th data point in the table
Last 6 months: chart every 100th data point
The number of data points and chart names are only examples. What I need is a way to pick only the "nth" row from the database.

You can use the Select overload that includes the item index of enumerations. Something like this should do the trick --
var data = myDataLogEnumeration.
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
If you need to limit the query with a Where or sort with OrderBy, you must do it before the first Select, otherwise the indexes will be all wrong --
var data = myDataLogEnumeration.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Unfortunately, as juharr commented, this overload is not supported in Entity Framework. One way to deal with this is to do something like this --
var data = entity.DataLogSet.
Where(dt => dt.DateTime > dateLimit).
OrderBy(dt => dt.SomeField).
ToArray().
Select((dt,i) => new { DataLog = dt, Index = i }).
Where(x => x.Index % nth == 0).
Select(x => x.DataLog);
Note the addition of a ToArray(). This isn't ideal though as it will force loading all the data that matches the initial query before selecting only every nth row.

There might be a trick that is supported by ef that might work for this.
if (step != 0)
query = query.Where(_ => Convert.ToInt32(_.Time.ToString().Substring(14, 2)) % step == 0);
this code converts the date into string then cuts the minutes out converts the minutes into an int and then gets every x'th minute for example if the variable step is 5 it's every 5 minutes.
For Postgresql this converts to:
WHERE ((substring(c.time::text, 15, 2)::INT % #__step_1) = 0)
this works best with fixed meassure points such as once a minute.
However, you can use the same method to group up things by cutting up to the hour or the minutes or the first part of the minute (10 minutes grouped) and use aggregation functions such as max() average() sum(), what might even is more desirable.
For example, this groups up in hours and takes the max of most but the average of CPU load:
using var ef = new DbCdr.Context();
IQueryable<DbCdr.CallStatsLog> query;
query = from calls in ef.Set<DbCdr.CallStatsLog>()
group calls by calls.Time.ToString().Substring(0, 13)
into g
orderby g.Max(_ => _.Time) descending
select new DbCdr.CallStatsLog()
{
Time = g.Min(_ => _.Time),
ConcurrentCalls = g.Max(_ => _.ConcurrentCalls),
CpuUsage = (short)g.Average(_ => _.CpuUsage),
ServerId = 0
};
var res = query.ToList();
translates to:
SELECT MAX(c.time) AS "Time",
MAX(c.concurrent_calls) AS "ConcurrentCalls",
AVG(c.cpu_usage::INT::double precision)::smallint AS "CpuUsage",
0 AS "ServerId"
FROM call_stats_log AS c
GROUP BY substring(c.time::text, 1, 13)
ORDER BY MAX(c.time) DESC
note: the examples work with postgres and iso datestyle.

Count Dates which a specific Customer does not have a Shop in it

Here is an example Data Table:
ID Amount Date
--------------------------
1 3.000 2016/1/1
2 4.000 2016/1/1
1 6.000 2017/1/1
1 3.000 2018/1/1
3 2.000 2019/1/1
I need to count Dates which a specific Customers does not have a Shop in it.
for example ID 2 does not have a shop in 2017/1/1 and 2018/1/1, so the count will be 2. and the count for customer ID 3 will be 3 because He does not have a shop in 2016/1/1 and 2017/1/1 and 2018/1/1
I think I should use grouping but do not know how to count which I want for a specific ID!
orders.GroupBy(x => x.Date)......???

Assuming you have list of objects:
// number of all different dates
var datesCount = list.Select(i => i.Date).Distinct().Count();
int customerId = 2;
//number of dates with customer shopping
var customerDatesCount = list.Where(i => i.ID == customerId).Select(i => i.Date).Distinct().Count();
var nonShoppingDatesCount = datesCount - customerDatesCount;

You need two steps here.
Get all distinct order dates
var allDates = orders.Select(o => o.Date).Distinct();
Find those dates which customer don't have. Except operation will do that
var missingDates = allDates.Except(orders.Where(o => o.ID == 2).Select(o => o.Date));
If you need just number of missing dates, then use missingDates.Count() or (better) use solution by #Roma

mvc Groupby and Orderby nested

mvc beginner
I have a table of lots that contain a property Num_of_steps representing the number of completed steps toward building a house.
I currently use this to retrieve the lot information and am sorting by the lot number.
var ViewModel = new Sub_lot_VM();
ViewModel.Subdivisions = db.Subdivisions
.Include(i => i.Lots)
.ToList();
if (ViewModel.Subdivisions !=null) // if data sort by lot number
{
foreach (var item in ViewModel.Subdivisions)
item.Lots = item.Lots.OrderBy(i => i.LotName).ToList();
}
return View(ViewModel);
}
Now I want to display this information a 3 groups:
first where the count is between 1 and 114 (active),
second where the count is above 115 (or GTE 115?) (finished)( and then orderby lot name) and
third group is count = 0 (not started) also order by lotname.
I've been trying to think of how to add .where and .groupby lambda expressions to my method without luck. Such as.where(I=>i.Lot.Num_of_steps=0).
I also see that I needed a foreach where some LINQ examples did not need the foreach. Still confused on that.

Get the lots first and then use groupby with ranges to get the groups
from x in
(
db.Subdivisions.SelectMany(sd => sd.Lots)
)
group x by x.Num_of_steps == 0 ? 3 : x.Num_of_steps < 115 ? 1 : 2 into g
orderby g.Key
select g.OrderBy(g1 => g1.LotName)
You can give the groups meaningful names in stead of 1, 2 and 3, but you can also postpone that until it's display time. The numbers facilitate correct sorting.

How can I subsample data from a time series with LINQ to SQL?

I have a database table full of time points and experimental values at those time points. I need to retrieve the values for an experiment and create a thumbnail image showing an XY plot of its data. Because the actual data set for each experiment is potentially 100,000 data points and my image is only 100 pixels wide, I want to sample the data by taking every nth time point for my image and ignoring the rest.
My current query (which retrieves all the data without sampling) is something simple like this:
var points = from p in db.DataPoints
where p.ExperimentId == myExperimentId
orderby p.Time
select new {
X = p.Time,
Y = p.Value
}
So, how can I best take every nth point from my result set in a LINQ to SQL query?

This will do every nth element:
int nth = 100;
var points = db.DataPoints
.Where(p => p.ExperimentId == myExperimentId)
.OrderBy(p => p.Time)
.Where( (p, index) => index % nth == 0 )
.Select( p => new { X = p.Time, Y = p.Value } );
It works by using the Queryable.Where overload which provides an index in the sequence, so you can filter based off the index.

.Skip(n).Take(1)
Will return one sample point. Call it repeatedly to get more points.
http://msdn.microsoft.com/en-us/library/bb386988.aspx
If performance becomes an issue, and you have a primary identity key of type int containing consecutive values, you can try returning all records that will evenly divide the primary key by your n.
.Where(x => x.PK % n == 0)

You should you
.Skip(n).Take(100)
It skips how many every record you want it to skip and Takes 100 records.
HTH