Develop Reference

Issue with ML.NET Fit: Source array was not long enough

The full error is : 'Source array was not long enough. Check the source index, length, and the array's lower bounds. (Parameter 'sourceArray')'
(IDataView trainingDataView, IDataView testDataView) = LoadData(mlContext); // Load
IDataView transformedNewData = dataPrepPipeline.Transform(trainingDataView);
var est = mlContext.Transforms.Categorical.OneHotEncoding("usernameEncoded", "username").Append(mlContext.Transforms.Categorical.OneHotEncoding("tagEncoded", "tag"));
var fitted = est.Fit(transformedNewData);
var data = fitted.Transform(transformedNewData);
var testEst = mlContext.Transforms.Categorical.OneHotEncoding("usernameEncoded", "username").Append(mlContext.Transforms.Categorical.OneHotEncoding("tagEncoded", "tag"));
var testFitted = testEst.Fit(testDataView);
var testData = testFitted.Transform(testDataView);
var options = new FieldAwareFactorizationMachineTrainer.Options
{
FeatureColumnName = "usernameEncoded",
ExtraFeatureColumns = new string[] { "tagEncoded" },
LabelColumnName = "Label",
LambdaLatent = 0.01f,
LambdaLinear = 0.001f,
LatentDimension = 16,
NumberOfIterations = 50,
LearningRate = 0.5f,
Verbose = true
};
FieldAwareFactorizationMachinePredictionTransformer retrainedModel =
mlContext.BinaryClassification.Trainers.FieldAwareFactorizationMachine(options).Fit(data, testData, originalModelParameters); // Error is here

Populating a 2D Matrix of List<string> from csv

I have a matrix of routes stored into a csv from a pandas dataframe. The content of this csv looks like:
,hA,hB,hC
hA,[],["hA","hB"],["hA","hB","hC"]
hB,["hB","hA"],[],["hB","hC"]
hC,["hC","hB","hA"],["hC","hB"],[]
From this file I would like to generate a matrix in c#, so I could get the route from hA to hC with something like:
routes["hA"]["hC"]
I can achieve this generating manually a Dictionary<string, Dictionary<string, List<string>>> like:
Dictionary<string, Dictionary<string, List<string>>> routes = new Dictionary<string, Dictionary<string, List<string>>>(){
{"hA", new Dictionary<string, List<string>>(){ { "hA", new List<string>() }, {"hB", new List<string>() { "hA", "hB" }}, { "hC", new List<string>() { "hA", "hB", "hC" }}}},
{ "hB", new Dictionary<string, List<string>>() { { "hA", new List<string>() { "hB", "hA" }}, { "hB", new List<string>() { }, { "hC", new List<string>() { "hB", "hC" }}}},
{ "hC", new Dictionary<string, List<string>>() { { "hA", new List<string>() { "hC", "hB", "hA" }}, { "hB", new List<string>() { "hC", "hB" }}, { "hC", new List<string>() { } }}}
};
But in case the size of the matrix increases or everytime a route changes it is a lot of reworking involved. Thant is why I could like to populate the routes matrix from the csv directly
Is there any way of populating this matrix from a csv? or is it a better type of collections to store this routes instead of Dictionary<string, Dictionary<string, List<string>>>?

Oof.. I think I'd read that CSV with a parser library set to use [ and ] as "quote" chars, but this will read it simplistically:
var lines = File.ReadAllLines(path);
var cols = lines[0].Split(',');
var frame = new Dictionary<string, Dictionary<string, string[]>>();
foreach(var line in lines.Skip(1)){
var bits = line.Replace("]", "").Split(",[");
var row = bits[0];
for(int i = 1; i < cols.Length; i++){
var col = cols[i];
frame.TryAdd(row, new Dictionary<string, string[]>());
frame[row][col] = bits[i].Split(',').Select(s => s.Trim('"')).ToArray();
}
}
That should deliver you your nested dictionaries so you can address them like you would a dataframe.. By the way, I don't know what happens if you ask a dataframe for something that isn't there, but c# would throw a KeyNotFoundException if you asked for eg frame["hZ"]["hello"] ..
If you want the innermost storage container to be a List you can swap the ToArray to be ToList
You perhaps don't need to nest, by the way:
var frame = new Dictionary<(string, string), string[]>();
foreach(var line in lines.Skip(1)){
var bits = line.Replace("]", "").Split(",[");
var row = bits[0];
for(int i = 1; i < cols.Length; i++){
var col = cols[i];
frame[(row, col)] = bits[i].Split(',').Select(s => s.Trim('"')).ToArray();
}
}
It could be queried like frame[("hA","hB")]

Turn your node names (ie, hA, hB, hC) into an enum:
enum Indexer {
hA = 0,
hB = 1,
hC = 2
}
Use a two-dimensional array of lists:
List<string>[,] Matrix = new List<string>[3,3];
Access the data out of the Matrix:
List<string> path = Matrix[(int)Indexer.hA, (int)Indexer.hC];
If you need to, you can convert the text-based node names back to an enum:
var n = (Indexer)Enum.Parse(typeof(Indexer), "hA");
This assumes that you are importing a csv of pre-defined node names. Let me know if the node names can't be pre-defined and I'll update the answer.

Based on #CaiusJard and #derHugo suggestions
I needed to modify a little bit the original csv file to make it easier by removing the first column (which cointaed the index) and using ";" as column separator df_routes.to_csv("routes_mtrx_headers.csv", sep = ';', index = False)
The final solution is
var route_dictionary = new Dictionary<(string, string), string[]>();
using (var reader = new StreamReader(#"C:/mypath/routes_mtrx_headers.csv"))
{
string[] locations = reader.ReadLine().Split(';');
int rowIdx = 0;
int colIdx = 0;
while (!reader.EndOfStream)
{
var row = reader.ReadLine();
var columns = row.Split(';');
colIdx = 0;
foreach (var col in columns)
{
// Removing the List wrapper
var path = col.Replace("]", "").Replace("[", "").Split(',');
route_dictionary.Add((locations[colIdx], locations[rowIdx]), path);
colIdx += 1;
}
rowIdx += 1;
}
}
// Finally to access each element in the matrix
var route = route_dictionary[("hA", "hC")];

Largest key of nested dictionary in C#

Tried searching online. But couldn't find an answer:
public class A
{
public Dictionary<int, string> Dict { get; } = new Dictionary<int, string>();
}
var list = new List<A>();
var a = new A();
a.Dict.Add(1, "A");
var b = new A();
b.Dict.Add(2, "A");
list.Add(a);
list.Add(b);
How to get the maximum and minimum key in dictionary.
max key is 2 and min key is 1.
var max = list.Max(e => e.Dict.??); //

You can use Dictionary.Keys property for this:
var max = list.Max(e => e.Dict.Keys.Max())
or
var max = list.SelectMany(e => e.Dict.Keys).Max()

IEnumerable.Select() when attribute is known only at runtime

Say I have a data class like this and a list of its objects:
public class DataSet
{
public int A { get; set; }
public string B { get; set; }
public double C { get; set; }
}
var data = new List<DataSet>
{
new DataSet() { A = 1, B = "One", C = 1.1 },
new DataSet() { A = 2, B = "Two", C = 2.2 },
new DataSet() { A = 3, B = "Three", C = 3.3 }
};
I would like to do a Select() on the list, based on different properties. For example, if I need a list of property A, I could do this easily:
var listA = data.Select(x => x.A).ToList();
All good so far.
But in my program, I need to do the above, only, I wouldn't know whether I need a list of A or B or C until runtime. This 'knowledge' of what to select is stored in a list of strings, and I need to iterate it and extract only the appropriate lists. Something like this:
// GetKeys() will return the keys that I need to extract.
// So at one time keyList could have "A" and "B", another time "B" and "C" etc.
List<string> keyList = GetKeys();
foreach (var key in keyList)
{
// What do I do here?
data.Select(x =>???).ToList();
}
Is this possible at all? I'm fine with even a non-LINQ solution, if it achieves my goal.
EDIT:
Clarifying the requirement.
The end result I want is a separate list based on each 'key' mentioned above. So, something like
List<List<object>>
The count in outer list would be the count of keyList.
The inner list would have as many items as in DataSet.

This would probably not be the most efficient solution, but you could use Reflection for a fully dynamic solution:
private static List<List<object>> SelectDynamicData<T>(IEnumerable<T> data, List<string> properties)
{
// get the properties only once per call
// this isn't fast
var wantedProperties = typeof(T)
.GetProperties()
.Where(x => properties.Contains(x.Name))
.ToArray();
var result = new Dictionary<string, List<object>>();
foreach (var item in data)
{
foreach (var wantedProperty in wantedProperties)
{
if (!result.ContainsKey(wantedProperty.Name))
{
result.Add(wantedProperty.Name, new List<object>());
}
result[wantedProperty.Name].Add(wantedProperty.GetValue(item));
}
}
return result.Select(x => x.Value).ToList();
}
And, of course, you'd need to do a double foreach or a LINQ query to print that. For example:
var data = new List<DataSet>
{
new DataSet() { A = 1, B = "One", C = 1.1 },
new DataSet() { A = 2, B = "Two", C = 2.2 },
new DataSet() { A = 3, B = "Three", C = 3.3 }
};
var selectedData = SelectDynamicData(data, new List<string> { "A", "C" });
foreach (var list in selectedData)
{
foreach (object item in list)
{
Console.Write(item + ", ");
}
Console.WriteLine();
}

Using Creating Expression Trees by Using the API you can build an expression tree to represent the linq query you were hard coding in order to make it more dynamic.
Expression<Func<TModel, object>> GetPropertyExpression<TModel>(string propertyName) {
// Manually build the expression tree for
// the lambda expression v => v.PropertyName.
// (TModel v) =>
var parameter = Expression.Parameter(typeof(TModel), "v");
// (TModel v) => v.PropertyName
var property = Expression.Property(parameter, propertyName);
// (TModel v) => (object) v.PropertyName
var cast = Expression.Convert(property, typeof(object));
var expression = Expression.Lambda<Func<TModel, object>>(cast, parameter);
return expression;
}
Review the comments to understand the building of the expression tree.
This now can be used with the data to extract the desired result.
Following similar to what was provided in another answer it would be simplified to
List<List<object>> SelectDynamicData<T>(IEnumerable<T> data, List<string> properties) {
return properties
.Select(_ => data.Select(GetPropertyExpression<T>(_).Compile()).ToList())
.ToList();
}
Both methods are displayed in the following example
[TestMethod]
public void TestMethod1() {
var data = new List<DataSet>
{
new DataSet() { A = 1, B = "One", C = 1.1 },
new DataSet() { A = 2, B = "Two", C = 2.2 },
new DataSet() { A = 3, B = "Three", C = 3.3 }
};
var propertyKnownAtRuntime = "A";
var expression = GetPropertyExpression<DataSet>(propertyKnownAtRuntime);
var listA = data.Select(expression.Compile()).ToList();
//Produces
// { 1, 2, 3}
var listAC = SelectDynamicData(data, new List<string> { "A", "C" });
//Produces
//{
// { 1, 2, 3},
// { 1.1, 2.2, 3.3 }
//}
}

You can use reflection, for example
string key = "A";
var query = data.Select(x =>
{
var prop = x.GetType().GetProperty(key); //NOTE: if key does not exist this will return null
return prop.GetValue(x);
});
foreach (var value in query)
{
Console.WriteLine(value); //will print 1, 2, 3
}

C# Replacing values from one dictionary to another

I have 2 dictionaries with the same keys but different values. I need to find in dictionary1 all values that are equal to 1 and replace them with the value in dictionary2. I could edit the first dictionary or better idea is to save the new values into another dictionary either way is fine. My idea was to use a foreach loop and then find any values in dictionary1 == 1 then go inside dictionary2 and extract that value for that key and save it into dictionary1 or another dictionary for that key.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace Practice_With_Dictionaries
{
class Program
{
static void Main(string[] args)
{
object[] keys= new object[5];
keys[0] = "1111";
keys[1] = "2222";
kesy[2] = "3333";
kesy[3] = "4444";
kesy[4] = "5555";
object[] speed = new object[5];
speed[0] = 1;
speed[1] = 2;
speed[2] = 1;
speed[3] = 1;
speed[4] = 5;
object[] speed2 = new object[5];
speed2[0] = 1;
speed2[1] = 2;
speed2[2] = 3;
speed2[3] = 4;
speed2[4] = 5;
Dictionary<object, object> dictionary1 = new Dictionary<object,object>();
Dictionary<object, object> dictionar2 = new Dictionary<object, object>();
for (int iRow = 0; iRow < interfaceKeys.Length; iRow++)
{
dictionary1.Add(keys[iRow], (double)speed[iRow]);
dictionary2.Add(keys[iRow], (double)speed2[iRow]);
}
// for each loop with if statements
}
}
}

First of all, if you have string as key, and int as value, then you should use Dictionary<string, int> instead of Dictionary<object, object>.
Dictionary<string, int> dictionary1 = new Dictionary<string, int>();
Dictionary<string, int> dictionary2 = new Dictionary<string, int>();
// fill your dictionaries
Now if you always have matching keys between two dictionaries:
var keys = dictionary1.Where(kvp => kvp.Value == 1).Select(kvp => kvp.Key);
foreach (var key in keys)
dictionary1[key] = dictionary2[key];
Otherwise
var keys = dictionary1.Where(kvp => kvp.Value == 1).Select(kvp => kvp.Key);
foreach (var key in keys)
{
int value;
if (dictionary2.TryGetValue(key, out value))
dictionary1[key] = value;
}

You can use LINQ and the ToDictionary method (I made some changes to simplify your code, I hope this is acceptable):
static void Main(string[] args)
{
var keys = new string[5] { "1111", "2222", "3333", "4444", "5555" };
var speed = new double[5] { 1, 2, 1, 1, 5 };
var speed2 = new double[5] { 1, 2, 3, 4, 5 };
var dictionary1 = new Dictionary<string, double>();
var dictionary2 = new Dictionary<string, double>();
for (int iRow = 0; iRow < keys.Length; iRow++)
{
dictionary1.Add(keys[iRow], speed[iRow]);
dictionary2.Add(keys[iRow], speed2[iRow]);
}
// the solution
var result = dictionary1.ToDictionary(x => x.Key, kv => kv.Value == 1 ? dictionary2[kv.Key] : kv.Value);
}

you could do something like this
var dictionary1 = new Dictionary<string, string>
{
{"a", "1"},
{"b", "2"},
{"c", "3"},
{"d", "3"}
};
var dictionary2 = new Dictionary<string, string>
{
{"a", "1"},
{"b", "22"},
{"c", "33"}
};
var result = dictionary1
.Where(d => dictionary2.ContainsKey(d.Key) && dictionary2[d.Key] == d.Value);
result will contain all values that have same keys in dictinary1 and dictinary2 with values in dictinary1.