I have a tricky model I'm trying to normalize for binary classification. Here's an example of the model structure. I renamed a few things just to simplify.
public class Review
{
public bool Label { get; set; }
public ReviewItem ReviewItem { get; set; }
public List<OtherItem> OtherItems { get; set; }
}
public class ReviewItem
{
public string SomeText { get; set; }
public float SomeNumber { get; set; }
public bool SomeBool { get; set; }
}
public class OtherItem
{
public string SomeDifferentText { get; set; }
public float SomeDifferentNumber { get; set; }
public bool SomeDifferentBool { get; set; }
}
There can be any number of OtherItem in the List. This is what I tried to flatten the model a bit.
public class ReviewMlModel
{
public bool Label { get; set; }
public string ReviewItem_SomeText { get; set; }
public float ReviewItem_SomeNumber { get; set; }
public bool ReviewItem_SomeBool { get; set; }
public string[] OtherItem_SomeDifferentText { get; set; }
public float[] OtherItem_SomeDifferentNumber { get; set; }
public bool[] OtherItem_SomeDifferentBool { get; set; }
}
From there I tried this to normalize it:
var data = mlContext.Data.LoadFromEnumerable(allReviews);
var dataPrepEstimator = mlContext.Transforms.Text.FeaturizeText("ReviewItem_SomeText")
.Append(mlContext.Transforms.Text.FeaturizeText("OtherItem_SomeDifferentText"))
.Append(mlContext.Transforms.Conversion.ConvertType("ReviewItem_SomeBool"))
.Append(mlContext.Transforms.Conversion.ConvertType("OtherItem_SomeDifferentBool"))
.Append(mlContext.Transforms.Concatenate("Features",
"ReviewItem_SomeText", "OtherItem_SomeDifferentText", "ReviewItem_SomeBool",
"OtherItem_SomeDifferentBool", "ReviewItem_SomeNumber", "OtherItem_SomeDifferentNumber"));
var transformedData = dataPrepEstimator.Fit(data).Transform(data);
var model = mlContext.BinaryClassification.Trainers.AveragedPerceptron()
.Fit(transformedData);
This gives me the exception on the line where I try to create the model:
Schema mismatch for feature column 'Features': expected Vector<Single>, got VarVector<Single> (Parameter 'inputSchema')
I'm guessing this is due to the fact these arrays all have variable lengths, but I don't see a way to transform the VarVector. Do I need to go make the original allReviews enumerable have the same length array for every array? Or am I way off track with how I flattened the original model?
Looks like it was a combination of changing the arrays to all have the same size, as well as adding [VectorType(size)] attribute to all the array properties.
I've been trying to deserialize a Json string that contains a list of ints representing colors to then be inserted into a sql database using Entity Framework. I'm pretty new to Entity Framework and read that it doesn't support collections of primitive types, I thought to get around it I could add the class
public class Colors
{
public int Id { get; set; }
public int Color { get; set; }
}
and then create a list in the CharacterColor class to hold the ints
public List<Colors> Colors { get; set; }
However I get the error when trying to deserialize the Json.
Newtonsoft.Json.JsonSerializationException: 'Error converting value
255 to type 'ORAC.Data.Entities.Colors'. Path
'characterColors[0].colors[0]', line 1, position 1200.'
ArgumentException: Could not cast or convert from System.Int64 to ORAC.Data.Entities.Colors.
Would anyone with more experience with Entity Framework be able to see where I'm going wrong.
Character character = JsonConvert.DeserializeObject<Character>(jsonString);
Json string
"{\"packedRecipeType\":\"DynamicCharacterAvatar\",\"name\":\"Character\",\"race\":\"HumanMaleHighPoly\",\"dna\":[{\"dnaType\":\"UMADnaHumanoid\",\"dnaTypeHash\":-212795365,\"packedDna\":\"{\\\"height\\\":128,\\\"headSize\\\":128,\\\"headWidth\\\":93,\\\"neckThickness\\\":108,\\\"armLength\\\":135,\\\"forearmLength\\\":128,\\\"armWidth\\\":116,\\\"forearmWidth\\\":128,\\\"handsSize\\\":118,\\\"feetSize\\\":109,\\\"legSeparation\\\":128,\\\"upperMuscle\\\":129,\\\"lowerMuscle\\\":152,\\\"upperWeight\\\":128,\\\"lowerWeight\\\":81,\\\"legsSize\\\":134,\\\"belly\\\":66,\\\"waist\\\":108,\\\"gluteusSize\\\":38,\\\"earsSize\\\":121,\\\"earsPosition\\\":233,\\\"earsRotation\\\":61,\\\"noseSize\\\":115,\\\"noseCurve\\\":128,\\\"noseWidth\\\":124,\\\"noseInclination\\\":128,\\\"nosePosition\\\":128,\\\"nosePronounced\\\":128,\\\"noseFlatten\\\":118,\\\"chinSize\\\":128,\\\"chinPronounced\\\":128,\\\"chinPosition\\\":128,\\\"mandibleSize\\\":128,\\\"jawsSize\\\":128,\\\"jawsPosition\\\":128,\\\"cheekSize\\\":128,\\\"cheekPosition\\\":128,\\\"lowCheekPronounced\\\":128,\\\"lowCheekPosition\\\":195,\\\"foreheadSize\\\":128,\\\"foreheadPosition\\\":128,\\\"lipsSize\\\":128,\\\"mouthSize\\\":128,\\\"eyeRotation\\\":128,\\\"eyeSize\\\":69,\\\"breastSize\\\":128}\"},{\"dnaType\":\"UMADnaTutorial\",\"dnaTypeHash\":-1679007774,\"packedDna\":\"{\\\"eyeSpacing\\\":128}\"}],\"characterColors\":[{\"name\":\"Skin\",\"colors\":[255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0]},{\"name\":\"Hair\",\"colors\":[255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0]},{\"name\":\"Eyes\",\"colors\":[255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0]},{\"name\":\"Undies\",\"colors\":[255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0]}],\"wardrobeSet\":[{\"slot\":\"Underwear\",\"recipe\":\"MaleUnderwear\"}],\"raceAnimatorController\":\"Locomotion\"}"
Entity
public class Character
{
public int UserId { get; set; }
public int CharacterId { get; set; }
public string CharacterName { get; set; }
public string PackedRecipeType { get; set; }
public string Name { get; set; }
public string Race { get; set; }
public List<Dna> Dna { get; set; }
public List<CharacterColor> CharacterColors { get; set; }
public List<WardrobeSet> WardrobeSet { get; set; }
public string RaceAnimatorController { get; set; }
public User User { get; set; }
}
public class Dna
{
public int Id { get; set; }
public string DnaType { get; set; }
public int DnaTypeHash { get; set; }
public string PackedDna { get; set; }
}
public class CharacterColor
{
public int Id { get; set; }
public string Name { get; set; }
public List<Colors> Colors { get; set; }
}
public class WardrobeSet
{
public int Id { get; set; }
public string Slot { get; set; }
public string Recipe { get; set; }
}
public class Colors
{
public int Id { get; set; }
public int Color { get; set; }
}
Solution
I updated the Json and replaced the colors array with a new array of objects
JObject jsonToParse = JObject.Parse(jsonString);
JArray characterColors = (JArray)jsonToParse["characterColors"];
foreach(var item in characterColors)
{
JArray colors = (JArray)item["colors"];
JArray newColorsArray = new JArray();
var i = 0;
foreach (var col in colors)
{
var color = new Color
{
ColorId = i,
Value = (int)col
};
newColorsArray.Add(JToken.FromObject(color));
i++;
}
colors.Replace(newColorsArray);
}
In your Json you have colors defined as an array of ints, not as an object array of your new Colors class. Should be more like colors: [{Id:0, Color:255},{Id:2, Color:255}, .......]
So the JSON is incorrect, in the json the colors array is sent as a List basically
colors: [255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0,255,255,255,255,0,0,0,0]
but the .net is expecting something more like the following which is more like a List
colors: [{Id:0, Color:255},{Id:1, Color:255}, ...]
So you can do any of the following:
Change whatever is sending your JSON to send an array of objects of {int, int} instead of an array of ints.
Change your List to be a List and then update all your .net code to adjust to that.
Write a custom Json converter to convert from the json you have to the .net that you have.
You should do 1 or 2, as your data doesn't seem complex enough to go through the effort of 3.
I have no previous experience with JSON or using web services, however I'm trying to consume a web service that returns meteorological information.
Here's the documentation on the API that I'm trying to use.
This API gives me data serialized with JSON. I did some reading into JSON, and from what I understand, the best way to access this serialized data after I download it would be to de-serialize it into an object with matching properties and types. Did I get this part right? I don't understand however in this case how am I supposed to accurately know the types of the data returned via JSON.
In the API that I mentioned before I got this example of a response from the API in JSON:
{"coord":
{"lon":145.77,"lat":-16.92},
"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],
"base":"cmc stations",
"main":{"temp":293.25,"pressure":1019,"humidity":83,"temp_min":289.82,"temp_max":295.37},
"wind":{"speed":5.1,"deg":150},
"clouds":{"all":75},
"rain":{"3h":3},
"dt":1435658272,
"sys":{"type":1,"id":8166,"message":0.0166,"country":"AU","sunrise":1435610796,"sunset":1435650870},
"id":2172797,
"name":"Cairns",
"cod":200}
What I did was, on Visual Studio I used the "Paste Special" > "Paste as JSON classes" option, which created these classes for me:
public class Rootobject
{
public Coord coord { get; set; }
public Weather[] weather { get; set; }
public string _base { get; set; }
public Main main { get; set; }
public Wind wind { get; set; }
public Clouds clouds { get; set; }
public Rain rain { get; set; }
public int dt { get; set; }
public Sys sys { get; set; }
public int id { get; set; }
public string name { get; set; }
public int cod { get; set; }
}
public class Coord
{
public float lon { get; set; }
public float lat { get; set; }
}
public class Main
{
public float temp { get; set; }
public int pressure { get; set; }
public int humidity { get; set; }
public float temp_min { get; set; }
public float temp_max { get; set; }
}
public class Wind
{
public float speed { get; set; }
public int deg { get; set; }
}
public class Clouds
{
public int all { get; set; }
}
public class Rain
{
public int _3h { get; set; }
}
public class Sys
{
public int type { get; set; }
public int id { get; set; }
public float message { get; set; }
public string country { get; set; }
public int sunrise { get; set; }
public int sunset { get; set; }
}
public class Weather
{
public int id { get; set; }
public string main { get; set; }
public string description { get; set; }
public string icon { get; set; }
}
The problem is that when I request data using the HttpClient, when I try to de-serialize the response I get I few errors regarding mismatching data types, like for example, float data being assign to properties of the type int.
Here's a snippet of my code:
string json = await client.GetStringAsync("weather?q=London,uk&appid=010101010101010101101");
Rootobject currentWeather = new Rootobject();
currentWeather = JsonConvert.DeserializeObject<Rootobject>(json);
MessageBox.Show(currentWeather.name);
I understand that in this case I could change the types of the properties in my classes to match what is being returned by the API, however that doesn't feel right to me, mainly because seems like it could be a source of troubles and unpredictable behavior. Am I doing this right? Maybe I'm missing something in the API documentation, shouldn't they provide the type of the data being returned?
Correct : De-serialize it into an object with matching properties and types.
First check the API documentation for the types, and if that is not comprehensiive enough I would consider changing your types to match what you infer from the JSON.
A value of 289.9 is a float.
A value of 1435650870 can be stored an int.
A value of AU can be a string/enum.
Edit:
I checked the API documentation you linked to and don't see anywhere were it explicitly states the types of data returned.
Edit 2:
Answering your question more directly, "how am I supposed to accurately know the types of the data returned via JSON?" (thanks #CodeCaster), without finding that information in documentation I don't think you can.
But I feel you can get 99.999% close by just looking at historical data returned.
If it is satisfied for you to use dynamic you can try this snippet
string json = await client.GetStringAsync("weather?q=London,uk&appid=010101010101010101101");
dynamic currentWeather= JObject.Parse(json);
MessageBox.Show(currentWeather.name);
You will find more details in documentation