Develop Reference

Resampling with Deedle duplicate keys

My code below resamples 5-minute interval to 1-day interval for the daily profit stats. The problem is that BacktestResult consists of duplicate CloseDate values, because I'm testing with multiple pairs (TRXUSDT, ETHUSDT and BTCUSDT). dailyProfit returns Series<DateTime, double>, which explains the exception. How can I make it grouped by Pair or something? It works fine when tested with one pair.
// Create series
var series = _backtestResults.ToOrdinalSeries();
// daily_profit = results.resample('1d', on = 'close_date')['profit_percent'].sum()
var dailyProfit = series.ResampleEquivalence(
index => new DateTime(series[index].CloseDate.Year, series[index].CloseDate.Month, series[index].CloseDate.Day, 0, 0, 0, DateTimeKind.Utc),
group => group.SelectValues(g => g.ProfitPercentage).Sum()).DropMissing();
// classes
public class BacktestResult
{
public string Pair { get; set; }
public decimal ProfitPercentage { get; set; }
public decimal ProfitAbs { get; set; }
public decimal OpenRate { get; set; }
public decimal CloseRate { get; set; }
public DateTime OpenDate { get; set; }
public DateTime CloseDate { get; set; }
public decimal OpenFee { get; set; }
public decimal CloseFee { get; set; }
public decimal Amount { get; set; }
public decimal TradeDuration { get; set; }
public SellType SellReason { get; set; }
}
Edit:
Example which takes the JSON data from pastebin:
using Deedle;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
namespace Resample
{
class Program
{
public class BacktestResultTest
{
public string Pair { get; set; }
public decimal ProfitPercentage { get; set; }
public decimal ProfitAbs { get; set; }
public decimal OpenRate { get; set; }
public decimal CloseRate { get; set; }
public DateTime OpenDate { get; set; }
public DateTime CloseDate { get; set; }
public decimal OpenFee { get; set; }
public decimal CloseFee { get; set; }
public decimal Amount { get; set; }
public decimal TradeDuration { get; set; }
public bool OpenAtEnd { get; set; }
public int SellReason { get; set; }
}
static void Main(string[] args)
{
// Take JSON data from pastebin
using var webClient = new WebClient();
var json = webClient.DownloadString("https://pastebin.com/raw/Dhp9202f");
// Deserialize the data
var data = JsonConvert.DeserializeObject<List<BacktestResultTest>>(json);
var ts = data.ToOrdinalSeries();
var byDateAndPair = ts.SelectKeys(kvp => Tuple.Create(kvp.Value.Value.CloseDate, kvp.Value.Value.Pair)).SortByKey();
// daily_profit = results.resample('1d', on = 'close_date')['profit_percent'].sum()
var dailyProfit2 = byDateAndPair.ResampleEquivalence(
k => Tuple.Create(new DateTime(k.Item1.Year, k.Item1.Month, k.Item1.Day), k.Item2),
g => g.Select(kvp => kvp.Value.ProfitPercentage).Sum());
// backtest_worst_day = min(daily_profit)
var worstDay2 = dailyProfit2.Min();
// backtest_best_day = max(daily_profit)
var bestDay2 = dailyProfit2.Max();
// winning_days = sum(daily_profit > 0)
var winningDays2 = dailyProfit2.SelectValues(x => x > 0).Sum();
// draw_days = sum(daily_profit == 0)
var drawDays2 = dailyProfit2.SelectValues(x => x == 0).Sum();
// losing_days = sum(daily_profit < 0)
var losingDays2 = dailyProfit2.SelectValues(x => x < 0).Sum();
Console.ReadLine();
}
}
}

You can use a custom data type as a key in Deedle. If you want to be able to use resampling on the series, then this needs to support IComparable. You can either define your own type or use built-in Tuple.
Assuming we have some very basic data:
var ts =
new[] {
KeyValue.Create(new DateTime(2020,1,1), new { Value = 1.0, Kind = "A" }),
KeyValue.Create(new DateTime(2020,1,2), new { Value = 1.0, Kind = "A" }),
KeyValue.Create(new DateTime(2020,1,3), new { Value = 1.0, Kind = "B" }),
KeyValue.Create(new DateTime(2020,1,4), new { Value = 1.0, Kind = "B" }),
}.ToSeries();
The first thing we need to do is to change the key to be the date together with a kind. (In fact, you can get into trouble earlier in your code if you had duplicate dates!)
var byDateAndKind =
ts.SelectKeys(kvp => Tuple.Create(kvp.Key, kvp.Value.Value.Kind)).SortByKey();
Now the key is Tuple<DateTime, string> consisting of the date and the kind. You can now use ResampleEquivalence on this. Here, we use year and kind as the new key and sum values in group:
var aggByYearAndKind =
byDateAndKind.ResampleEquivalence(
(k) => Tuple.Create(k.Item1.Year, k.Item2),
(g) => g.Select(kvp => kvp.Value.Value).Sum());
aggByYearAndKind.Print();
This will print a series that maps 2020, "A" to 2 and also 2020, "B" to 2.
EDIT You are right - this does not seem to work. I was able to get it to work using GroupBy instead of ResampleEquvialence:
var dailyProfit2 =
ts.GroupBy(kvp =>
new { Date = new DateTime(kvp.Value.CloseDate.Year, kvp.Value.CloseDate.Month, kvp.Value.CloseDate.Day), Kind = kvp.Value.Pair })
.SelectValues(g => g.Select(kvp => kvp.Value.ProfitPercentage).Values.Sum());
// backtest_worst_day = min(daily_profit)
var worstDay2 = dailyProfit2.Min();
// backtest_best_day = max(daily_profit)
var bestDay2 = dailyProfit2.Max();
// winning_days = sum(daily_profit > 0)
var winningDays2 = dailyProfit2.Where(x => x.Value > 0).Values.Sum();
// draw_days = sum(daily_profit == 0)
var drawDays2 = dailyProfit2.Where(x => x.Value == 0).Values.Sum();
// losing_days = sum(daily_profit < 0)
var losingDays2 = dailyProfit2.Where(x => x.Value < 0).Values.Sum();

Unable to get Additional Value from Dictionary

I have a dictionary collection which consist "TextDescription" and "TextValues" (List Collection) which has two objects "Date","Text"
"TextDescription":"Your payment due date is dec 12 on every month"
0 th Item
"Date"=> "12/12/2018"
"Text"=>"A,B,C,D,E,F"
1 st Item
"Date"=> "12/12/2019"
"Text"=>"G,H,I,J,K,L"
I have to split the "Text" object with it's comma separated values and add it in a list so I achieved it from below code
var lstDictionaryItems = cache.Get(CacheKeys.AutoText) as Dictionary<string, Autotext>;
var txtItemsList = lstDictionaryItems.FirstOrDefault(i => i.Key.ToString() == _languageIndicator.ToString()).Value.
TextValues.SelectMany(st => st.Text.Split(',').Select(x => new TextValues { Date = x.Trim(), Text = st.Text }));
However I have achieved the Text and Date fields in a separate list item but I left with "TextDescription".
txtItemsList doesn't have the property "TextDescription" how can I get this in txtItemList object
Here is my class structure
public class AutoText
{
public string TextDescription{ get; set; }
public List<TextValues> TextValues{ get; set; }
}
public class TextValues
{
public string Date { get; set; }
public string Text{ get; set; }
}

See following :
class Program
{
static void Main(string[] args)
{
List<List<string>> inputs = new List<List<string>>(){
new List<string>(){"Your payment due date is dec 12 on every month","12/12/2018","A,B,C,D,E,F"},
new List<string>(){"Your payment due date is dec 12 on every month","12/12/2019","G,H,I,J,K,L"}
};
var listInput = inputs.Select(x => new {TextDescription = x[0], Date = DateTime.Parse(x[1]), Text = x[2]}).ToList();
Dictionary<DateTime, AutoText> dict = listInput.Select(x => new AutoText()
{
TextDescription = x.TextDescription,
TextValues = x.Text.Split(new char[] { ',' }).Select(y => new TextValues() { Date = x.Date, Text = y }).ToList()
}).GroupBy(x => x.TextValues.FirstOrDefault().Date, y => y)
.ToDictionary(x => x.Key, y => y.FirstOrDefault());
}
}
public class AutoText
{
public string TextDescription { get; set; }
public List<TextValues> TextValues { get; set; }
}
public class TextValues
{
public DateTime Date { get; set; }
public string Text { get; set; }
}

How do I use LINQ to group & project objects, based on a member dictionary key?

I have a List<Predictions> that I would like to project onto a List<List<PredictionObj>>. These two classes are defined as follows:
public Predictions
{
public Dictionary<string, double> PredictedMetrics { get; private set; }
public DateTime PredictionTimeStamp { get; set; }
public Predictions()
{
PredictedMetrics = new Dictionary<string, double>();
}
}
public class PredictionObj
{
public string PredictedMetricName { get; set; }
public double PredictedMetricValue { get; set; }
public DateTime PredictionTimeStamp { get; set; }
}
For context, each Predictions object in the List<Predictions> list contains a collection (PredictedMetrics) of predicted values for a set of metrics, which were made at PredictionTimeStamp. I'd like to separate each of those metrics into their own list, such that there will be one list (List<PredictionObj>) for every unique PredictedMetrics key in the list. (PredictedMetricName will map to the dictionary's key, PredictedMetricValue will map to the dictionary's value). I'd like to store all of these lists in one List<List<PredictionObj>> list.
Is there a way to accomplish this using LINQ extension methods?
You can copy/paste the source below into LINQPad as an example. I'm looking for LINQ that will accomplish what GenerateSeperateMetricLists is doing:
void Main()
{
DateTime currTime = DateTime.UtcNow;
List<Predictions> records = new List<Predictions>();
Predictions record1 = new Predictions();
record1.PredictedMetrics.Add("metric1", 2.2d);
record1.PredictedMetrics.Add("metric2", 0.2d);
record1.PredictionTimeStamp = currTime;
records.Add(record1);
Predictions record2 = new Predictions();
record2.PredictedMetrics.Add("metric1", 1.2d);
record2.PredictedMetrics.Add("metric2", 0.1d);
record2.PredictionTimeStamp = currTime.AddMinutes(1);
records.Add(record2);
Predictions record3 = new Predictions();
record3.PredictedMetrics.Add("metric1", 3.2d);
record3.PredictedMetrics.Add("metric2", 0.3d);
record3.PredictionTimeStamp = currTime.AddMinutes(2);
records.Add(record3);
Predictions record4 = new Predictions();
record4.PredictedMetrics.Add("metric1", 4.2d);
record4.PredictedMetrics.Add("metric2", 0.4d);
record4.PredictionTimeStamp = currTime.AddMinutes(3);
records.Add(record4);
//What's the LINQ that could replace this method?
GenerateSeperateMetricLists(records).Dump();
}
private static List<List<PredictionObj>> GenerateSeperateMetricLists(List<Predictions> predictionRecords)
{
var predictionMetricLists = new List<List<PredictionObj>>();
foreach (Predictions forecastRecord in predictionRecords)
{
foreach (KeyValuePair<string, double> prediction in forecastRecord.PredictedMetrics)
{
PredictionObj predictionMetric = new PredictionObj
{
PredictedMetricName = prediction.Key,
PredictedMetricValue = prediction.Value,
PredictionTimeStamp = forecastRecord.PredictionTimeStamp
};
var metricList = predictionMetricLists.Where(x => x.First().PredictedMetricName == prediction.Key);
if (metricList.Count() == 0)
{
predictionMetricLists.Add(new List<PredictionObj> {predictionMetric});
}
else
{
metricList.First().Add(predictionMetric);
}
}
}
return predictionMetricLists;
}
private class Predictions
{
public Dictionary<string, double> PredictedMetrics { get; private set; }
public DateTime PredictionTimeStamp { get; set; }
public Predictions()
{
PredictedMetrics = new Dictionary<string, double>();
}
}
private class PredictionObj
{
public string PredictedMetricName { get; set; }
public double PredictedMetricValue { get; set; }
public DateTime PredictionTimeStamp { get; set; }
}

You should first flatten the data into a list of PredictionObj:
var flatList = records
.SelectMany(r => r.PredictedMetrics.Select(p => new PredictionObj
{
PredictedMetricName = p.Key,
PredictedMetricValue = p.Value,
PredictionTimeStamp = r.PredictionTimeStamp
}));
This produces a flat sequence of PredictionObj objects. Now you can group them by PredictedMetricName:
flatList.GroupBy(x => x.PredictedMetricName).Dump();
The query syntax equivalent, in one statement:
(
from r in records
from p in r.PredictedMetrics
select new PredictionObj
{
PredictedMetricName = p.Key,
PredictedMetricValue = p.Value,
PredictionTimeStamp = r.PredictionTimeStamp
} into flatList
group flatList by flatList.PredictedMetricName into fg
select fg
).Dump();

You just need to take each Prediction and project it into a List<PredictionObj> and then convert those into a List<Prediction>:
var ans = records.SelectMany(p => p.PredictedMetrics.Select(pm => new PredictionObj { PredictedMetricName = pm.Key, PredictedMetricValue = pm.Value, PredictionTimeStamp = p.PredictionTimeStamp }))
.GroupBy(p => p.PredictedMetricName)
.Select(g => g.ToList())
.ToList();
Updated for change in OP.

How can i convert string to double ? I'm getting exception

for(int i = 0; i < Rect.rectangles.Count; i++)
{
if (Rect.rectangles[i].transform != null && Rect.rectangles[i].transform != "")
{
string firstNumber = Rect.rectangles[i].transform.Substring(18, 10);
string secondNumber = Rect.rectangles[i].transform.Substring(7, 10);
double a = Convert.ToDouble(firstNumber);
double b = Convert.ToDouble(secondNumber);
}
}
The first number is 0.49052715 and the Convert.ToDouble of a is fine.
But when it's getting to the line:
double b = Convert.ToDouble(secondNumber);
I'm getting exception:
FormatException: Unknown char: ,
But there is no any char/s at all the string in secondNumber is: 0.87142591
This is the whole string in transform: matrix(0.87142591,0.49052715,-0.49052715,0.87142591,0,0)
And i'm extracting the first two numbers: 0.49052715 and 0.87142591 and then trying to convert them to double. But getting the exception.
The code including the Rect.rectangles definition:
private void ParseSvgMap()
{
XDocument document = XDocument.Load(#"C:\Users\mysvg\Documents\my.svg");
Rect.rectangles = document.Descendants().Where(x => x.Name.LocalName == "rect").Select(x => new Rect()
{
style = Rect.GetStyle((string)x.Attribute("style")),
id = (string)x.Attribute("id"),
width = (double)x.Attribute("width"),
height = (double)x.Attribute("width"),
x = (double?)x.Attribute("width"),
y = (double?)x.Attribute("width"),
transform = x.Attribute("transform") == null ? "" : (string)x.Attribute("transform")
}).ToList();
for(int i = 0; i < Rect.rectangles.Count; i++)
{
if (Rect.rectangles[i].transform != null && Rect.rectangles[i].transform != "")
{
string firstNumber = Rect.rectangles[i].transform.Substring(18, 10);
string secondNumber = Rect.rectangles[i].transform.Substring(7, 10);
double a = Convert.ToDouble(firstNumber);
double b = Convert.ToDouble(secondNumber);
float degree = FindDegree(a, b);
}
}
}
public static float FindDegree(double a, double b)
{
float value = (float)((System.Math.Atan2(a, b) / System.Math.PI) * 180f);
if (value < 0) value += 360f;
return value;
}
public class Rect
{
public static List<Rect> rectangles { get; set; }
public Dictionary<string, string> style { get; set; }
public string id { get; set; }
public double width { get; set; }
public double height { get; set; }
public double? x { get; set; }
public double? y { get; set; }
public string transform { get; set; }
public double degree { get; set; }
public static Dictionary<string, string> GetStyle(string styles)
{
string pattern = #"(?'name'[^:]+):(?'value'.*)";
string[] splitArray = styles.Split(new char[] { ';' });
Dictionary<string, string> style = splitArray.Select(x => Regex.Match(x, pattern))
.GroupBy(x => x.Groups["name"].Value, y => y.Groups["value"].Value)
.ToDictionary(x => x.Key, y => y.FirstOrDefault());
return style;
}
}

Code like .Substring(18, 10) is hopelessly brittle and (as you have discovered) is an accident waiting to happen. Numbers aren't guaranteed to be the same length. Just look at the sample data you provided.
You should use something like a regex to extract the numbers from the strings before attempting to parse them.
var regex = new Regex(#"matrix\((?<num>-?(?:\d|\.)+)(?:,(?<num>-?(?:\d|\.)+))*\)");
var data = "matrix(0.87142591,0.49052715,-0.49052715,0.87142591,0,0)";
var values = regex
.Matches(data)
.Cast<Match>()
.SelectMany(m => m.Groups["num"]
.Captures
.Cast<Capture>()
.Select(c=>c.Value))
.Select(double.Parse)
.ToList();
Then pick out the numbers you want with values[0] and values[1].

Its doomed to fail.
I would use following regex:
string test = "matrix(0.1234,0.23233433,0.34,-3343,-3.33)";
Regex reg = new Regex("[-]?[0-9]+(\\.[0-9]+)?");
MatchCollection coll = reg.Matches(test);

You can write a method along the lines of :
public double fixString(string incoming)
{
Double a;
if(incoming.Contains(','))
{
string fixedNum = incoming.Remove(incoming.IndexOf(','));
a = Convert.ToDouble(fixedNum);
return a;
}
else
{
a = Convert.ToDouble(incoming);
return a;
}
}

Query multiple tables and send query to Dictionary

I'm trying to query multiple table and save the query as a global dictionary for further processing. I've tried the following, but instead of values I get the class name in the dictionary. Please take a look and show me what's wrong and where to read up more on todictionary queries?
public class linqtosql
{
public Dictionary<int, MC_VARIABLES> dctMC = new Dictionary<int, MC_VARIABLES>();
public class MC_VARIABLES
{
public int ID { get; set; }
public int UDLY_LAST { get; set; }
public int STRIKE { get; set; }
public decimal SKEW_A { get; set; }
public decimal SKEW_B { get; set; }
public double SKEW_C { get; set; }
}
public void GET_DATA()
{
var qryBOOK = from B in Globals.DATA.BOOKs
from O in Globals.DATA.OPTIONs
from U in Globals.DATA.UDLies
from S in Globals.DATA.SKEWs
where B.CONTRACT == O.CONTRACT
where O.UDLY_SYMBOL == U.UDLY_SYMBOL
where O.CONTRACT == S.CONTRACT
select new MC_VARIABLES
{ ID = B.ID, STRIKE = (int)B.STRIKE, SKEW_A = (decimal)S.SKEW_A };
dctMC = qryBOOK.ToDictionary(x => x.ID, x => x);
foreach (KeyValuePair<int, MC_VARIABLES> KVP in dctMC)
{
var key = KVP.Key;
var item = KVP.Value.SKEW_A;
}
}
}

it should be x => x instead of x => MC_VARIABLES, x is of type MC_VARIABLES in this case.
qryBOOK.ToDictionary(x => x.ID, x => x)