Develop Reference

Resampling with Deedle duplicate keys

My code below resamples 5-minute interval to 1-day interval for the daily profit stats. The problem is that BacktestResult consists of duplicate CloseDate values, because I'm testing with multiple pairs (TRXUSDT, ETHUSDT and BTCUSDT). dailyProfit returns Series<DateTime, double>, which explains the exception. How can I make it grouped by Pair or something? It works fine when tested with one pair.
// Create series
var series = _backtestResults.ToOrdinalSeries();
// daily_profit = results.resample('1d', on = 'close_date')['profit_percent'].sum()
var dailyProfit = series.ResampleEquivalence(
index => new DateTime(series[index].CloseDate.Year, series[index].CloseDate.Month, series[index].CloseDate.Day, 0, 0, 0, DateTimeKind.Utc),
group => group.SelectValues(g => g.ProfitPercentage).Sum()).DropMissing();
// classes
public class BacktestResult
{
public string Pair { get; set; }
public decimal ProfitPercentage { get; set; }
public decimal ProfitAbs { get; set; }
public decimal OpenRate { get; set; }
public decimal CloseRate { get; set; }
public DateTime OpenDate { get; set; }
public DateTime CloseDate { get; set; }
public decimal OpenFee { get; set; }
public decimal CloseFee { get; set; }
public decimal Amount { get; set; }
public decimal TradeDuration { get; set; }
public SellType SellReason { get; set; }
}
Edit:
Example which takes the JSON data from pastebin:
using Deedle;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
namespace Resample
{
class Program
{
public class BacktestResultTest
{
public string Pair { get; set; }
public decimal ProfitPercentage { get; set; }
public decimal ProfitAbs { get; set; }
public decimal OpenRate { get; set; }
public decimal CloseRate { get; set; }
public DateTime OpenDate { get; set; }
public DateTime CloseDate { get; set; }
public decimal OpenFee { get; set; }
public decimal CloseFee { get; set; }
public decimal Amount { get; set; }
public decimal TradeDuration { get; set; }
public bool OpenAtEnd { get; set; }
public int SellReason { get; set; }
}
static void Main(string[] args)
{
// Take JSON data from pastebin
using var webClient = new WebClient();
var json = webClient.DownloadString("https://pastebin.com/raw/Dhp9202f");
// Deserialize the data
var data = JsonConvert.DeserializeObject<List<BacktestResultTest>>(json);
var ts = data.ToOrdinalSeries();
var byDateAndPair = ts.SelectKeys(kvp => Tuple.Create(kvp.Value.Value.CloseDate, kvp.Value.Value.Pair)).SortByKey();
// daily_profit = results.resample('1d', on = 'close_date')['profit_percent'].sum()
var dailyProfit2 = byDateAndPair.ResampleEquivalence(
k => Tuple.Create(new DateTime(k.Item1.Year, k.Item1.Month, k.Item1.Day), k.Item2),
g => g.Select(kvp => kvp.Value.ProfitPercentage).Sum());
// backtest_worst_day = min(daily_profit)
var worstDay2 = dailyProfit2.Min();
// backtest_best_day = max(daily_profit)
var bestDay2 = dailyProfit2.Max();
// winning_days = sum(daily_profit > 0)
var winningDays2 = dailyProfit2.SelectValues(x => x > 0).Sum();
// draw_days = sum(daily_profit == 0)
var drawDays2 = dailyProfit2.SelectValues(x => x == 0).Sum();
// losing_days = sum(daily_profit < 0)
var losingDays2 = dailyProfit2.SelectValues(x => x < 0).Sum();
Console.ReadLine();
}
}
}

You can use a custom data type as a key in Deedle. If you want to be able to use resampling on the series, then this needs to support IComparable. You can either define your own type or use built-in Tuple.
Assuming we have some very basic data:
var ts =
new[] {
KeyValue.Create(new DateTime(2020,1,1), new { Value = 1.0, Kind = "A" }),
KeyValue.Create(new DateTime(2020,1,2), new { Value = 1.0, Kind = "A" }),
KeyValue.Create(new DateTime(2020,1,3), new { Value = 1.0, Kind = "B" }),
KeyValue.Create(new DateTime(2020,1,4), new { Value = 1.0, Kind = "B" }),
}.ToSeries();
The first thing we need to do is to change the key to be the date together with a kind. (In fact, you can get into trouble earlier in your code if you had duplicate dates!)
var byDateAndKind =
ts.SelectKeys(kvp => Tuple.Create(kvp.Key, kvp.Value.Value.Kind)).SortByKey();
Now the key is Tuple<DateTime, string> consisting of the date and the kind. You can now use ResampleEquivalence on this. Here, we use year and kind as the new key and sum values in group:
var aggByYearAndKind =
byDateAndKind.ResampleEquivalence(
(k) => Tuple.Create(k.Item1.Year, k.Item2),
(g) => g.Select(kvp => kvp.Value.Value).Sum());
aggByYearAndKind.Print();
This will print a series that maps 2020, "A" to 2 and also 2020, "B" to 2.
EDIT You are right - this does not seem to work. I was able to get it to work using GroupBy instead of ResampleEquvialence:
var dailyProfit2 =
ts.GroupBy(kvp =>
new { Date = new DateTime(kvp.Value.CloseDate.Year, kvp.Value.CloseDate.Month, kvp.Value.CloseDate.Day), Kind = kvp.Value.Pair })
.SelectValues(g => g.Select(kvp => kvp.Value.ProfitPercentage).Values.Sum());
// backtest_worst_day = min(daily_profit)
var worstDay2 = dailyProfit2.Min();
// backtest_best_day = max(daily_profit)
var bestDay2 = dailyProfit2.Max();
// winning_days = sum(daily_profit > 0)
var winningDays2 = dailyProfit2.Where(x => x.Value > 0).Values.Sum();
// draw_days = sum(daily_profit == 0)
var drawDays2 = dailyProfit2.Where(x => x.Value == 0).Values.Sum();
// losing_days = sum(daily_profit < 0)
var losingDays2 = dailyProfit2.Where(x => x.Value < 0).Values.Sum();

How do I use LINQ to group & project objects, based on a member dictionary key?

I have a List<Predictions> that I would like to project onto a List<List<PredictionObj>>. These two classes are defined as follows:
public Predictions
{
public Dictionary<string, double> PredictedMetrics { get; private set; }
public DateTime PredictionTimeStamp { get; set; }
public Predictions()
{
PredictedMetrics = new Dictionary<string, double>();
}
}
public class PredictionObj
{
public string PredictedMetricName { get; set; }
public double PredictedMetricValue { get; set; }
public DateTime PredictionTimeStamp { get; set; }
}
For context, each Predictions object in the List<Predictions> list contains a collection (PredictedMetrics) of predicted values for a set of metrics, which were made at PredictionTimeStamp. I'd like to separate each of those metrics into their own list, such that there will be one list (List<PredictionObj>) for every unique PredictedMetrics key in the list. (PredictedMetricName will map to the dictionary's key, PredictedMetricValue will map to the dictionary's value). I'd like to store all of these lists in one List<List<PredictionObj>> list.
Is there a way to accomplish this using LINQ extension methods?
You can copy/paste the source below into LINQPad as an example. I'm looking for LINQ that will accomplish what GenerateSeperateMetricLists is doing:
void Main()
{
DateTime currTime = DateTime.UtcNow;
List<Predictions> records = new List<Predictions>();
Predictions record1 = new Predictions();
record1.PredictedMetrics.Add("metric1", 2.2d);
record1.PredictedMetrics.Add("metric2", 0.2d);
record1.PredictionTimeStamp = currTime;
records.Add(record1);
Predictions record2 = new Predictions();
record2.PredictedMetrics.Add("metric1", 1.2d);
record2.PredictedMetrics.Add("metric2", 0.1d);
record2.PredictionTimeStamp = currTime.AddMinutes(1);
records.Add(record2);
Predictions record3 = new Predictions();
record3.PredictedMetrics.Add("metric1", 3.2d);
record3.PredictedMetrics.Add("metric2", 0.3d);
record3.PredictionTimeStamp = currTime.AddMinutes(2);
records.Add(record3);
Predictions record4 = new Predictions();
record4.PredictedMetrics.Add("metric1", 4.2d);
record4.PredictedMetrics.Add("metric2", 0.4d);
record4.PredictionTimeStamp = currTime.AddMinutes(3);
records.Add(record4);
//What's the LINQ that could replace this method?
GenerateSeperateMetricLists(records).Dump();
}
private static List<List<PredictionObj>> GenerateSeperateMetricLists(List<Predictions> predictionRecords)
{
var predictionMetricLists = new List<List<PredictionObj>>();
foreach (Predictions forecastRecord in predictionRecords)
{
foreach (KeyValuePair<string, double> prediction in forecastRecord.PredictedMetrics)
{
PredictionObj predictionMetric = new PredictionObj
{
PredictedMetricName = prediction.Key,
PredictedMetricValue = prediction.Value,
PredictionTimeStamp = forecastRecord.PredictionTimeStamp
};
var metricList = predictionMetricLists.Where(x => x.First().PredictedMetricName == prediction.Key);
if (metricList.Count() == 0)
{
predictionMetricLists.Add(new List<PredictionObj> {predictionMetric});
}
else
{
metricList.First().Add(predictionMetric);
}
}
}
return predictionMetricLists;
}
private class Predictions
{
public Dictionary<string, double> PredictedMetrics { get; private set; }
public DateTime PredictionTimeStamp { get; set; }
public Predictions()
{
PredictedMetrics = new Dictionary<string, double>();
}
}
private class PredictionObj
{
public string PredictedMetricName { get; set; }
public double PredictedMetricValue { get; set; }
public DateTime PredictionTimeStamp { get; set; }
}

You should first flatten the data into a list of PredictionObj:
var flatList = records
.SelectMany(r => r.PredictedMetrics.Select(p => new PredictionObj
{
PredictedMetricName = p.Key,
PredictedMetricValue = p.Value,
PredictionTimeStamp = r.PredictionTimeStamp
}));
This produces a flat sequence of PredictionObj objects. Now you can group them by PredictedMetricName:
flatList.GroupBy(x => x.PredictedMetricName).Dump();
The query syntax equivalent, in one statement:
(
from r in records
from p in r.PredictedMetrics
select new PredictionObj
{
PredictedMetricName = p.Key,
PredictedMetricValue = p.Value,
PredictionTimeStamp = r.PredictionTimeStamp
} into flatList
group flatList by flatList.PredictedMetricName into fg
select fg
).Dump();

You just need to take each Prediction and project it into a List<PredictionObj> and then convert those into a List<Prediction>:
var ans = records.SelectMany(p => p.PredictedMetrics.Select(pm => new PredictionObj { PredictedMetricName = pm.Key, PredictedMetricValue = pm.Value, PredictionTimeStamp = p.PredictionTimeStamp }))
.GroupBy(p => p.PredictedMetricName)
.Select(g => g.ToList())
.ToList();
Updated for change in OP.

Empty data on List in LinQ Statement

I got this LinQ statement
var daysList = new List<int>(new int[30]);
var model_pdv = db_pdv.Pdv.GroupBy(x => new { Pdv = x.Clave_PDV, Nombre_Pdv = x.Nombre_Pdv})
.Select(x => new DishVM()
{
Clave_PDV = x.Key.Pdv,
Nombre_Pdv = x.Key.Nombre_Pdv,
Days = daysList,
Data = x
}).ToList();
However i dont know why my "Data" List inside my LinQ gets empty values the first time but then i saves the LinQ as it should
This is my DishVm Class:
public class DishVM
{
public string Clave_PDV { get; set; }
public string Nombre_Pdv { get; set; }
public IEnumerable<Pdv> Data { get; set; }
}
And my Pdv class:
public class Pdv
{
public string Clave_PDV { get; set; }
public string Nombre_Pdv { get; set; }
}
How can i avoid the empty Data List?

The type of x within your Select statement is IGrouping, change it to produce a list:
var model_pdv = db_pdv.Pdv.GroupBy(x => new { Pdv = x.Clave_PDV, Nombre_Pdv = x.Nombre_Pdv})
.Select(x => new DishVM()
{
Clave_PDV = x.Key.Pdv,
Nombre_Pdv = x.Key.Nombre_Pdv,
Days = daysList,
Data = x.ToList()
}).ToList();

Query multiple tables and send query to Dictionary

I'm trying to query multiple table and save the query as a global dictionary for further processing. I've tried the following, but instead of values I get the class name in the dictionary. Please take a look and show me what's wrong and where to read up more on todictionary queries?
public class linqtosql
{
public Dictionary<int, MC_VARIABLES> dctMC = new Dictionary<int, MC_VARIABLES>();
public class MC_VARIABLES
{
public int ID { get; set; }
public int UDLY_LAST { get; set; }
public int STRIKE { get; set; }
public decimal SKEW_A { get; set; }
public decimal SKEW_B { get; set; }
public double SKEW_C { get; set; }
}
public void GET_DATA()
{
var qryBOOK = from B in Globals.DATA.BOOKs
from O in Globals.DATA.OPTIONs
from U in Globals.DATA.UDLies
from S in Globals.DATA.SKEWs
where B.CONTRACT == O.CONTRACT
where O.UDLY_SYMBOL == U.UDLY_SYMBOL
where O.CONTRACT == S.CONTRACT
select new MC_VARIABLES
{ ID = B.ID, STRIKE = (int)B.STRIKE, SKEW_A = (decimal)S.SKEW_A };
dctMC = qryBOOK.ToDictionary(x => x.ID, x => x);
foreach (KeyValuePair<int, MC_VARIABLES> KVP in dctMC)
{
var key = KVP.Key;
var item = KVP.Value.SKEW_A;
}
}
}

it should be x => x instead of x => MC_VARIABLES, x is of type MC_VARIABLES in this case.
qryBOOK.ToDictionary(x => x.ID, x => x)

How to count parent's property with linq

I have 2 object collections looking like this
public class Meter
{
public string UID { get; set; }
public string NR { get; set; }
public List<GMSData> data { get; set; }
}
public class GSMData : Meter
{
public DateTime TimeStamp { get; set; }
public int CellID { get; set; }
}
public static List<Meter> GetMeterUIDList()
{
return meters.Values.ToList();
}
public static List<GSMData> GetGsmdataList()
{
return meters.Values.SelectMany(m => m.Gsmdata)
.OrderBy(t => t.TimeStamp)
.ToList();
}
I need to get all NR for each CellId and a count on how many NR there are on each CellID.
How can i do that?

Perhaps:
var idGroups = meters
.SelectMany(m => m.data)
.GroupBy(d => d.CellID)
.Select(g => new { CellID = g.Key, UniqueNr = g.Select(m => m.NR).Distinct() });
foreach (var g in idGroups)
Console.WriteLine("CellID: {0} Count: {1}", g.CellID, g.UniqueNr.Count());
If the NR's don't need to be unique remove the Distinct.