I have been struggling to create the proper data structure for ML.net and get it to load into my application. Essentially, I have an application where the training data will be dynamic and the type and/or size will not be known prior to runtime. In addition, I have to convert the training data from a non-standard primitive types (ie. App_Bool, or App_Number... rather than simply using bool or double, etc.) So, this has been proving to be a problem as I try to convert my training data into a generic data type which can then be loaded from memory using the LoadFromEnumerable function.
I have four basic data type classes:
public class MLIntData
{
public MLIntData(string label, List<object> l)
{
Label = label;
foreach (App_Integer element in l)
Features.Add((int)element.Value);
}
public List<int> Features { get; set; } = new List<int>();
public string Label { get; set; } = "";
}
public class MLNumberData
{
public MLNumberData(string label, List<object> l)
{
Label = label;
foreach (App_Number element in l)
Features.Add((double)element.Value);
}
public List<double> Features { get; set; } = new List<double>();
public string Label { get; set; } = "";
}
public class MLBoolData
{
public MLBoolData(string label, List<object> l)
{
Label = label;
foreach (App_Boolean element in l)
Features.Add((bool)element.Value);
}
public List<bool> Features { get; set; } = new List<bool>();
public string Label { get; set; } = "";
}
public class MLTextData
{
public MLTextData(string label, List<object> l)
{
Label = label;
foreach (App_String element in l)
Features.Add(element.Value.ToString());
}
public List<string> Features { get; set; } = new List<string>();
public string Label { get; set; } = "";
}
So, each base class will contain a label for the data and then a list of features which will either be of type bool, double, int, or string.
Now, in my ML.net code I'm trying to load in the training data and then create an IDataView object of the data. First I loop through the input data (which is originally of the generic type object) then create the new classes of data.
List<object> data = new List<object>();
for(int i = 0; i < input.Count; i++)
{
MLCodifiedData codifiedData = input[i].Value as MLCodifiedData;
Type dataType = codifiedData.Features[0].GetType();
if (dataType == typeof(App_Boolean))
{
data.Add(new MLBoolData(codifiedData.Label, codifiedData.Features));
}
else if (dataType == typeof(App_Number))
{
data.Add(new MLNumberData(codifiedData.Label, codifiedData.Features));
}
else if (dataType == typeof(App_Integer))
{
data.Add(new MLIntData(codifiedData.Label, codifiedData.Features));
}
if (dataType == typeof(App_String))
{
data.Add(new MLTextData(codifiedData.Label, codifiedData.Features));
}
}
IDataView TrainingData = mlContext.Data.LoadFromEnumerable<object>(data);
I have tried creating a schema definition (which can be passed in as the second parameter in the LoadFromEnumerable method, but I can't seem to get that to work. I've also tried creating a schema using the schema builder to create a schema, but that doesn't seem to work either. Right now, I'm using one of the datasets that is included in one of the sample files. And to preempt questions, yes, I know I could simply load the data as file and read it in that way... However, in my app I need to first read in the CSV into memory, then create the data structure so I can't really use many of the examples which are geared toward reading in a CSV file using the LoadFromTextFile method. Can anyone provide support as to how I could setup a dynamic in-memory collection and get it converted into a IDataView object?
Related
I'm developing a 2D game app using unity c#.
I have a list of item(object) and each item contains double Price, double Multiplier and int Level.
I'm trying to load that List to the FireBase Realtime DataBase using json inorder to save the player's peronal shop.
to try and make it look like this in the DB:
This is my Item class:
public class Item
{
public double Price {get; set;}
public double Multiplier { get; set; }
public int Level { get; set; }
public Item(double price, double multiplier, int level)
{
this.Price = price;
this.Multiplier = multiplier;
this.Level = level;
}
}
And this is my Shop class:
public class Shop
{
public List<Item> ShopList { get; set; }
public Shop()
{
this.ShopList = new List<Item>();
}
public void InitShop()
{
string[] lines = System.IO.File.ReadAllLines(#"Assets\Scripts\Data\ShopData.txt");
foreach (string line in lines)
{
string[] words = line.Split(',');
double price = Convert.ToDouble(words[0]);
double mul = Convert.ToDouble(words[1]);
int level = Convert.ToInt32(words[2]);
Item i = new Item(price, mul, level);
this.ShopList.Add(i);
}
}
If you are gonna use Json to create your save system, you might aswell use the provided JsonUtitlity by Unity. With this you can automatically create a save data object in the json format and save that.
API Reference: https://docs.unity3d.com/ScriptReference/JsonUtility.html
Youtube Tutorial (Code Monkey): https://www.youtube.com/watch?v=6uMFEM-napE
If you don't mind using a third party dll, you might want to check out Newtonsoft
https://www.newtonsoft.com/json
When you read from realtime db to Unity, it keeps the type data as keywords which is problematic to JSON.parse()
these are most likely causing issues for your code, there is a JSON.parser that handles this, at the current time I am unable to find the script in mind.
an alternative from the community is to convert it to a dictionary
Dictionary<string, object> dict = new Dictionary<string, object>();
foreach (DataSnapshot child in snapshot.Children)
{
dict.Add(child.Key, child.Value);
}
I would recommend using NewtonSoft /Json.Net for Unity. Pakcage can be found here:
https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347
Works like this:
string ConvertToJson(object obj)
{
return JsonConvert.SerializeObject(obj);
}
string ConvertFromJson<T>(string json)
{
return JsonConvert.DeserializeObject<T>(json);
}
void Usage()
{
List<object> myList = populateListWithSomeData();
// Save it
var json = ConvertToJson(myList);
SaveInDatabase(json);
// Load it
var json = LoadFromDatabase();
myList = ConvertFromJson<List<object>>(json);
}
I have an sql query that provides me my data where I sometimes have lines that should be clustered (the data is aligned with an order by). The data is grouped by the field CAPName. Going through those rows line by line, I need to decide whether a new list should be initiated (content of CAPName differs to previous itteration), or whether the (already) initated list (from the previous iteration) should be added, too.
My pain lays with the location of the declaration of the relatedCapabilitySystem list.
I wanted to declare it within the if statement (Because, as I stated I need to decide whether the list from the previous iteration should be added too, or whether it should start a new list), but I can't as the compiler throws an exception, as the RLCapSys.Add(rCs); is non-existing in this content (which is only theoretically true). I understand why the compiler throws this exception. But if I declare the list on a "higher" level, than I always have a new list, which I don't want in the case that the item should be added to the list defined in the iteration(s) (1 or more) before
So what I want to achieve is, generate the list RLCapSys and add to it, in case the previous iteration contains the same CAPName (for clustering), otherwise create a new list.
SqlCommand cmdDetail = new SqlCommand(SQL_SubSytemsToCapability, DBConDetail);
SqlDataReader rdrDetail = cmdDetail.ExecuteReader();
List<relatedCapility> RLCaps = new List<relatedCapility>();
string lastCapShown = null;
while (rdrDetail.Read())
{
List<relatedCapabilitySystem> RLCapSys = new List<relatedCapabilitySystem>();
if (lastCapShown != rdrDetail["CAPName"].ToString())
{
//List<relatedCapabilitySystem> RLCapSys2 = new List<relatedCapabilitySystem>();
relatedCapility rC = new relatedCapility
{
Capability = rdrDetail["CAPName"].ToString(),
systemsRelated = RLCapSys,
};
RLCaps.Add(rC);
}
relatedCapabilitySystem rCs = new relatedCapabilitySystem
{
system = rdrDetail["name"].ToString(),
start = rdrDetail["SysStart"].ToString(),
end = rdrDetail["SysEnd"].ToString(),
};
RLCapSys.Add(rCs);
// method to compare the last related Capability shown create a new related Capabilty entry or add to the existing releated Capabilty related system list
lastCapShown = rdrDetail["CAPName"].ToString();
}
DBConDetail.Close();
and for reason of completness (but I think it is not needed here):
internal class CapabilitiesC
{
public List<Capability>Capabilities{ get;set;}
}
public class Capability
{
public string name { get; internal set; }
public string tower { get; internal set; }
public string color { get; internal set; }
public List<relatedCapility> related { get; set; }
}
public class relatedCapility
{
public string Capability { get; set; }
public List<relatedCapabilitySystem> systemsRelated { get; set; }
}
public class relatedCapabilitySystem
{
public string system { get; set; }
public string start { get; set; }
public string end { get; set; }
}
The purpose of your code is to take the input data and group it by capability. However, that is not immediately obvious. You can change your code to use LINQ so it becomes easier to understand and in the process solving your problem.
First you need a type to represent a record in your database. For lack of better name I will use Record:
class Record
{
public string System { get; set; }
public string Start { get; set; }
public string End { get; set; }
public string Capabilty { get; set; }
}
You can then create an iterator block to return all the records from the database (using an OR mapper like Entity Framework avoids most of this code and you can even shift some of the work from your computer to the database server):
IEnumerable<Record> GetRecords()
{
// Code to create connection and command (preferably in a using statement)
SqlDataReader rdrDetail = cmdDetail.ExecuteReader();
while (rdrDetail.Read())
{
yield return new Record {
System = rdrDetail["name"].ToString(),
Start = rdrDetail["SysStart"].ToString(),
End = rdrDetail["SysEnd"].ToString(),
Capability = rdrDetail["CAPName"].ToString()
};
}
// Close connection (proper using statement will do this)
}
Finally, you can use LINQ to perform the grouping:
var RLCaps = GetRecords()
.GroupBy(
record => record.Capability,
(capability, records) => new relatedCapility
{
Capability = capability ,
systemsRelated = records
.Select(record => new relatedCapabilitySystem
{
system = record.System,
start = record.Start,
end = record.End
})
.ToList()
})
.ToList();
Why not just assign it as NULL. The pattern would be
List<> myList = null;
if(condition)
{
myList = new List<>();
}
else
{
myList = previousList;
}
myList.Add();
previousList = myList;
I've got it working now. Thx everyone for your help. #martin, thx for your solution, you have put quite some effort into this, but that would have required for me to completely re-write my code. I am sure your approach would work and will be my next approach should I have a similar problem again.
It was a combination of the other answers that helped me figure it out. Let me show you what I ended up with:
SqlCommand cmdDetail = new SqlCommand(SQL_SubSytemsToCapability, DBConDetail);
SqlDataReader rdrDetail = cmdDetail.ExecuteReader();
List<relatedCapility> RLCaps = new List<relatedCapility>();
List<relatedCapabilitySystem> RLCapSys = new List<relatedCapabilitySystem>();
string lastCapShown = null;
while (rdrDetail.Read())
{
if (lastCapShown != rdrDetail["CAPName"].ToString())
{
RLCapSys = relatedCapabilitySystemList();
relatedCapility rC = new relatedCapility
{
Capability = rdrDetail["CAPName"].ToString(),
systemsRelated = RLCapSys,
};
RLCaps.Add(rC);
}
relatedCapabilitySystem rCs = new relatedCapabilitySystem
{
system = rdrDetail["name"].ToString(),
start = rdrDetail["SysStart"].ToString(),
end = rdrDetail["SysEnd"].ToString(),
};
RLCapSys.Add(rCs);
// method to compare the last related Capability shown create a new related Capabilty entry or add to the existing releated Capabilty related system list
lastCapShown = rdrDetail["CAPName"].ToString();
}
DBConDetail.Close();
So that's the section already shown bevor including my changes. Plus I added this:
private List<relatedCapabilitySystem> relatedCapabilitySystemList()
{
List<relatedCapabilitySystem> RLCapSys = new List<relatedCapabilitySystem>();
return RLCapSys;
}
Now I have new list reference everytime the CapName changes that is then added to the "higher" list. Before I had the issue of the very same list repeatedly assigned rather than a fresh one started. So thx again for your effort.
So I'm making a game, and it saves users' progress on the computer in a binary file. The User class stores a few things:
Integers for stat values (Serializable)
Strings for the Username and the skin assets
Lists of both the Achievement class and the InventoryItem class, which I have created myself.
Here are the User fields:
public string Username = "";
// ID is used for local identification, as usernames can be changed.
public int ID;
public int Coins = 0;
public List<Achievement> AchievementsCompleted = new List<Achievement>();
public List<InventoryItem> Inventory = new List<InventoryItem>();
public List<string> Skins = new List<string>();
public string CurrentSkinAsset { get; set; }
The Achievement class stores ints, bools, and strings, which are all serializable. The InventoryItem class stores its name (a string) and an InventoryAction, which is a delegate that is called when the item is used.
These are the Achievement class's fields:
public int ID = 0;
public string Name = "";
public bool Earned = false;
public string Description = "";
public string Image;
public AchievmentDifficulty Difficulty;
public int CoinsOnCompletion = 0;
public AchievementMethod OnCompletion;
public AchievementCriteria CompletionCriteria;
public bool Completed = false;
And here are the fields for the InventoryItem class:
InventoryAction actionWhenUsed;
public string Name;
public string AssetName;
The source of the InventoryAction variables are in my XNAGame class. What I mean by this is that the XNAGame class has a method called "UseSword()" or whatever, which it passes into the InventoryItem class. Previously, the methods were stored in the Game1 class, but the Game class, which Game1 inherits from, is not serializable, and there's no way for me to control that. This is why I have an XNAGame class.
I get an error when trying to serialize: "The 'SpriteFont' class is not marked as serializable", or something like that. Well, there is a SpriteFont object in my XNAGame class, and some quick tests showed that this is the source of the issue. Well, I have no control over whether or not the SpriteFont class is Serializable.
Why is the game doing this? Why must all the fields in the XNAGame class be serializable, when all I need is a few methods?
Keep in mind when answering that I'm 13, and may not understand all the terms you're using. If you need any code samples, I'll be glad to provide them for you. Thanks in advance!
EDIT: One solution I have thought of is to store the InventoryAction delegates in a Dictionary, except that this will be a pain and isn't very good programming practice. If this is the only way, I'll accept it, though (Honestly at this point I think this is the best solution).
EDIT 2: Here's the code for the User.Serialize method (I know what I'm doing in inefficient, and I should use a database, blah, blah, blah. I'm fine with what I'm doing now, so bear with me.):
FileStream fileStream = null;
List<User> users;
BinaryFormatter binaryFormatter = new BinaryFormatter();
try
{
if (File.Exists(FILE_PATH) && !IsFileLocked(FILE_PATH))
{
fileStream = File.Open(FILE_PATH, FileMode.Open);
users = (List<User>)binaryFormatter.Deserialize(fileStream);
}
else
{
fileStream = File.Create(FILE_PATH);
users = new List<User>();
}
for (int i = 0; i < users.Count; i++)
{
if (users[i].ID == this.ID)
{
users.Remove(users[i]);
}
}
foreach (Achievement a in AchievementsCompleted)
{
if (a.CompletionCriteria != null)
{
a.CompletionCriteria = null;
}
if (a.OnCompletion != null)
{
a.OnCompletion = null;
}
}
users.Add(this);
fileStream.Position = 0;
binaryFormatter.Serialize(fileStream, users);
You cannot serialize a SpriteFont by design, actually this is possible (.XNB file) but it hasn't been made public.
Solution:
Strip it off your serialized class.
Alternatives:
If for some reasons you must serialize some font, the first thing that comes to my mind would be to roll-out your own font system such as BMFont but that's a daunting task since you'll have to use it everywhere else where you might already do ...
Generate a pre-defined amount of fonts (i.e. Arial/Times/Courier at size 10/11/12 etc ...) using XNA Content app (can't recall its exact name); then store this user preference as two strings. With a string.Format(...) you should be able to load the right font back quite easily.
Alternative 2 is certainly the easiest and won't take more than a few minutes to roll-out.
EDIT
Basically, instead of saving a delegate I do the following:
inventory items have their own type
each type name is de/serialized accordingly
their logic does not happen in the main game class anymore
you don't have to manually match item type / action method
So while you'll end up with more classes, you have concerns separated and you can keep your main loop clean and relatively generic.
Code:
public static class Demo
{
public static void DemoCode()
{
// create new profile
var profile = new UserProfile
{
Name = "Bill",
Gold = 1000000,
Achievements = new List<Achievement>(new[]
{
Achievement.Warrior
}),
Inventory = new Inventory(new[]
{
new FireSpell()
})
};
// save it
using (var stream = File.Create("profile.bin"))
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, profile);
}
// load it
using (var stream = File.OpenRead("profile.bin"))
{
var formatter = new BinaryFormatter();
var deserialize = formatter.Deserialize(stream);
var userProfile = (UserProfile) deserialize;
// set everything on fire :)
var fireSpell = userProfile.Inventory.Items.OfType<FireSpell>().FirstOrDefault();
if (fireSpell != null) fireSpell.Execute("whatever");
}
}
}
[Serializable]
public sealed class UserProfile
{
public string Name { get; set; }
public int Gold { get; set; }
public List<Achievement> Achievements { get; set; }
public Inventory Inventory { get; set; }
}
public enum Achievement
{
Warrior
}
[Serializable]
public sealed class Inventory : ISerializable
{
public Inventory() // for serialization
{
}
public Inventory(SerializationInfo info, StreamingContext context) // for serialization
{
var value = (string) info.GetValue("Items", typeof(string));
var strings = value.Split(';');
var items = strings.Select(s =>
{
var type = Type.GetType(s);
if (type == null) throw new ArgumentNullException(nameof(type));
var instance = Activator.CreateInstance(type);
var item = instance as InventoryItem;
return item;
}).ToArray();
Items = new List<InventoryItem>(items);
}
public Inventory(IEnumerable<InventoryItem> items)
{
if (items == null) throw new ArgumentNullException(nameof(items));
Items = new List<InventoryItem>(items);
}
public List<InventoryItem> Items { get; }
#region ISerializable Members
public void GetObjectData(SerializationInfo info, StreamingContext context)
{
var strings = Items.Select(s => s.GetType().AssemblyQualifiedName).ToArray();
var value = string.Join(";", strings);
info.AddValue("Items", value);
}
#endregion
}
public abstract class InventoryItem
{
public abstract void Execute(params object[] objects);
}
public abstract class Spell : InventoryItem
{
}
public sealed class FireSpell : Spell
{
public override void Execute(params object[] objects)
{
// using 'params object[]' a simple and generic way to pass things if any, i.e.
// var world = objects[0];
// var strength = objects[1];
// now do something with these !
}
}
Okay, so I figured it out.
The best solution was to use a Dictionary in the XNAGame class, which stores two things: an ItemType (an enumeration), and an InventoryAction. Basically, when I use an item, I check it's type and then look up it's method. Thanks to everyone who tried, and I'm sorry if the question was confusing.
I am trying to add elements into a list, order them and then output them, there a number of "columns" if you like, per list
List<Info> infoList = new List<Info>();
while (dr.Read())
{
meeting_id = dr.GetValue(0).ToString();
try
{
Appointment appointment = Appointment.Bind(service, new ItemId(meeting_id));
Info data = new Info();
data.Start = appointment.Start;
data.Fruit = Convert.ToInt32(dr.GetValue(1));
data.Nuts = Convert.ToInt32(dr.GetValue(2));
infoList.Add(data);
}
Then to output it I want to order it by Start and then display all associated columns
for (int i = 0; i < infoList.Count; i++)
{
meet = meet + infoList[i];
}
First question: is the way I am inputting the data right?
Second question: How to I output all the columns to display all the associated columns? Is this possible? Is there a better practice?
Thanks
EDIT:
The class if you are interested:
public class Info
{
public DateTime Start { get; set; }
public int Fruit { get; set; }
public int Nuts { get; set; }
}
You can use Enumerable.OrderBy extension for enumerating your collection in some particular order (e.g. ordered by Start property value):
foreach(var info in infoList.OrderBy(i => i.Start))
{
// use info object here
// info.Fruits
// info.Nuts
}
BTW consider to add sorting on database side - that will be more efficient
I am fairly new to arrays in C# and am used to storing a mass of data in a string and in INI files and then breaking it down into basic arrays using delimiters...so yeh, my knowledge is almost none existent.
My main form class begin this definition:
public CAirportData[] _AirportData; //size not known
This is the method I am using to create the array:
...string[] airports = possibleAirports.Split(','); //size is known
foreach (string airport in airports)
{
string[] rwys = inif.Read(airport, "rwys").Split(':'); //size is known (2)
_AirportData = new CAirportData[] { new CAirportData() { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] } };
}
I know this just boils down to my limited knowledge of objects and arrays. But I can't seem to find anything on the internet that uses this sort of thing. I have tried to combine other peoples code with little success.
I need the _AirportData array to be available outside of the form hence public and declared outside of any methods. I supose the main problem is that I am overwriting array and foreach airport I am creating a new array hence loosing the previous. I had tried moving the ..= new CAirportData[] to all sorts of places but Visual Studio doesn't like it.
Below is the class definition for CAirportData:
public class CAirportData
{
public string icao { get; set; }
public string depRwy { get; set; }
public string arrRwy { get; set; }
public override string ToString()
{
string result = string.Format("ICAO: {0}, Dep: {1}, Arr: {2}", this.icao, this.depRwy, this.arrRwy);
return result;
}
}
public class CMRunways
{
public string icao { get; set; }
public string depRwy { get; set; }
public string arrRwy { get; set; }
}
Many thanks in advance for any help!
What you're looking for is generic List. Change the definition to:
public List<CAirportData> _AirportData = new List<CAirportData>();
Then the code in the loop to:
_AirportData.Add(new CAirportData { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] });
This is what I would do...Create a static class, with a static property (airports) and add a static constructor to load the airports from file at the begining.
public static class Session
{
public static CAirportData[] _AirportData;
static Session()
{
string airports = possibleAirports.Split(",");
foreach (string airport in airports)
{
string[] rwys = inif.Read(airport, "rwys").Split(':'); //size is known (2)
_AirportData = new CAirportData[] { new CAirportData() { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] } };
}
}
}
Now you can access the array anywhere in the project like
MessageBox.Show(Session.CAirportData[0].depRwy);