I am trying to read an onnx file, and input my parameters as an array, and make a prediction.
this is what my code looks like
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.Transforms.Onnx;
using System.Collections.Generic;
namespace MachineLearning
{
public class MLPrediction
{
public static double PredictOutput(string onnxFilePath, float[] inputParameterValues)
{
// Set up the MLContext
var mlContext = new MLContext();
// Load the ONNX model
OnnxScoringEstimator estimator = mlContext.Transforms.ApplyOnnxModel(onnxFilePath);
// Create an instance of InputData
InputData inputData = new InputData
{
FeatureVector = inputParameterValues
};
// Create an IEnumerable with a single element: the InputData instance
IEnumerable<InputData> inputEnumerable = new InputData[] { inputData };
// Load the data from the IEnumerable
IDataView prediction = mlContext.Data.LoadFromEnumerable<InputData>(inputEnumerable);
var model = estimator.Fit(prediction);
// Return the requested output parameter
double output = (double)model.GetType().GetProperty("Target").GetValue(model);
return output;
}
class InputData
{
[VectorType]
public float[] FeatureVector { get; set; }
}
class OutputData
{
public double Target { get; set; }
}
}
}
however i get the error on "System.ArgumentOutOfRangeException: 'Could not find input column 'Target' '" when I try and create the model.
Would anyone be able to point me where I am going wrong?
Have tried adding a Target to the inputs, but I don't believe it should include this.
Related
In question YamlDotNet !!binary type the answer provided by Antoine Aubry and the example in A fully working example can be tried here do not work with Version="11.2.1".
Is there any way to add a decorator in object properties or an alternative procedure like the former referenced?.
Our trouble is that byte array properties byte[] serialize in a very long (too many lines) file, and base64 encoding will improve that substantially.
We inspect the source repository at GitHub to find out the new way to do it.
And found in the release log the SerializerBuilder and DeserializerBuilder, so we make changes in the example referenced above (A fully working example can be tried here) and include a fully running project example, but the core changes are these:
To serialize
change:
var serializer = new Serializer();
serializer.RegisterTypeConverter(new ByteArrayConverter());
for:
var serializer = new SerializerBuilder()
.WithTypeConverter(new ByteArrayConverter())
.Build();
To deserialize
change:
var deserializer = new Deserializer();
deserializer.RegisterTagMapping("tag:yaml.org,2002:binary", typeof(byte[]));
deserializer.RegisterTypeConverter(new ByteArrayConverter());
for:
var deserializer = new DeserializerBuilder()
.WithTypeConverter(new ByteArrayConverter())
.WithTagMapping("tag:yaml.org,2002:binary", typeof(byte[]))
.Build();
Full demo example
Main program Program.cs (using a Demo.cs class for testing)
using System;
using System.Linq;
using System.Text.Json;
using YamlDotNet.Serialization;
namespace SerializeByteArrayAsBinary
{
class Program
{
static void Main()
{
var demoToSerialize = new Demo { Data = new byte[40] };
// Serialization
var serializer = new SerializerBuilder()
.WithTypeConverter(new ByteArrayConverter())
.Build();
var demoSerialized = serializer.Serialize(demoToSerialize);
Console.WriteLine($"--- After serialization the {nameof(Demo)} object is:\n{demoSerialized}\n");
// Deserialization
var deserializer = new DeserializerBuilder()
.WithTypeConverter(new ByteArrayConverter())
.WithTagMapping("tag:yaml.org,2002:binary", typeof(byte[]))
.Build();
var demoDeserialized = deserializer.Deserialize<Demo>(demoToDeserialize);
var demoDeserializedJson = JsonSerializer.Serialize(demoDeserialized, new JsonSerializerOptions { WriteIndented = true });
Console.WriteLine($"--- After deserialization the {nameof(Demo)} object viewed as JSON is:\n{demoDeserializedJson}");
// Verifications
if (demoToSerialize.Name != demoDeserialized.Name ||
demoToSerialize.Address != demoDeserialized.Address ||
!demoToSerialize.Data.SequenceEqual(demoDeserialized.Data))
throw new Exception($"Fail, {nameof(demoToSerialize)} is different from {nameof(demoDeserialized)}");
if (demoSerialized != demoToDeserialize)
throw new Exception($"Fail, {nameof(demoSerialized)} is different from {nameof(demoToDeserialize)}");
}
const string demoToDeserialize = #"Name: Jhon Doe
Address: 1200 Somewhere St
Data: !!binary AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==
";
}
}
Demostration class Demo.cs
using System;
namespace SerializeByteArrayAsBinary
{
/// <summary>
/// Demo class to demonstrate how to serialize and deserialize with YamlDotNet.
/// </summary>
public class Demo
{
public string Name { get; set; } = "Jhon Doe";
public string Address { get; set; } = "1200 Somewhere St";
public byte[] Data { get; set; } = Array.Empty<byte>();
}
}
Original ByteArrayConverter.cs (untouched)
using System;
using YamlDotNet.Core;
using YamlDotNet.Serialization;
namespace SerializeByteArrayAsBinary
{
public class ByteArrayConverter : IYamlTypeConverter
{
public bool Accepts(Type type)
{
return type == typeof(byte[]);
}
public object ReadYaml(IParser parser, Type type)
{
var scalar = (YamlDotNet.Core.Events.Scalar)parser.Current;
var bytes = Convert.FromBase64String(scalar.Value);
parser.MoveNext();
return bytes;
}
public void WriteYaml(IEmitter emitter, object value, Type type)
{
var bytes = (byte[])value;
emitter.Emit(new YamlDotNet.Core.Events.Scalar(
null,
"tag:yaml.org,2002:binary",
Convert.ToBase64String(bytes),
ScalarStyle.Plain,
false,
false
));
}
}
}
I'm using csvhelper 18.0 and am trying to update some code that used 2.8.4.
I have a file that I'm trying to read that has no headers.
I've defined a ClassMap to map by index.
I've created the configuration of the CsvReader so HasHeaderRecord = false.
When I try to import this file, I get an error that states There is no header record to determine the index by name. I'm confused as to why an error is being thrown regarding the header record. The header record does not exist, which is why I am mapping with an index.
Would anyone know how I can read a headerless file and still map to a class?
Here is the class and mapping class:
public class TFile
{
public int Wn { get; set; }
public string Hiwn { get; set; }
public string Sync { get; set; }
}
public sealed class TFileMap : ClassMap<TFile>
{
public TFileMap()
{
Map(m => m.Wn).Index(0);
Map(m => m.Hiwn).Index(1);
Map(m => m.Sync).Index(2);
}
}
Here is the piece of code that throughs the error:
using (TextReader textReader = new StringReader(data))
{
var csvT = new CsvReader(textReader, CultureInfo.InvariantCulture);
csvT.Configuration.HasHeaderRecord = false;
csvT.Configuration.RegisterClassMap<TFileMap>();
csvT.Configuration.CultureInfo = new CultureInfo("en-AU");
// error occurs on this line
tData1 = csvT.GetRecords<TFile>().ToList();
}
Here is a small sample file:
37,1R,Y
38,1L,Y
39,2R,Y
40,2L,Y
Any help would be greatly appreciated. Thanks
I pulled CsvHelper version 18.0 and tried with your code and sample data and it worked just fine for me.
var data = #"37,1R,Y
38,1L,Y
39,2R,Y
40,2L,Y";
using (TextReader textReader = new StringReader(data))
{
var csvT = new CsvReader(textReader, CultureInfo.InvariantCulture);
csvT.Configuration.HasHeaderRecord = false;
csvT.Configuration.RegisterClassMap<TFileMap>();
csvT.Configuration.CultureInfo = new CultureInfo("en-AU");
var tData1 = csvT.GetRecords<TFile>().ToList();
tData1.Dump();
}
Questions.
1. How to add, delete, update, read only one entry in a JSON file?
2. Is it possible to programmatically add, delete an entry from the file "SettingJson.json"?
Description.
I suggest using a JSON file to store application settings.
How to read and write the whole file, I figured it out.
If I understand correctly, then:
Delete:
1. Remove the property from the "Settings.cs" class;
2. Delete the property from the file "SettingJson.json";
Add:
1. Add a property from the "Settings.cs" class;
2. Add a property from the file "SettingJson.json";
Update:
???
Read:
???
Settings.cs
namespace ConsoleApp
{
public class Settings
{
public string NameSetting1 { get; set; }
public string NameSetting2 { get; set; }
public string NameSetting3 { get; set; }
}
}
Program.cs
//
using Newtonsoft.Json;
using System.IO;
namespace ConsoleApp
{
class Program
{
static void Main(string[] args)
{
Settings[] settings = new Settings[] { };
settings = FillArrayAll();
SaveDataArray(settings);
}
static Settings[] FillArrayAll()
{
Settings[] settings = new Settings[]
{
new Settings()
{
NameSetting1 = "ValueSetting_1",
NameSetting2 = "ValueSetting_2",
NameSetting3 = "ValueSetting_3"
}
};
return settings;
}
static Settings[] FillArrayOneItem()
{
Settings[] settings = new Settings[]
{
new Settings()
{
NameSetting2 = "ValueSetting_2",
}
};
return settings;
}
static void SaveDataArray(Settings[] settings)
{
string path = $"{Environment.CurrentDirectory}\\SettingsFolder\\SettingJson.json";
using (StreamWriter writer = File.CreateText(path))
{
string output = JsonConvert.SerializeObject(settings);
writer.Write(output);
}
}
public Settings[] ReadData()
{
string path = $"{Environment.CurrentDirectory}\\SettingsFolder\\SettingJson.json";
using (var reader = File.OpenText(path))
{
var fileText = reader.ReadToEnd();
return JsonConvert.DeserializeObject<Settings[]> (fileText);
}
}
}
}
Update-1
Based on materials from the answer Pavel Anikhouski - 2020-02-28 16: 16
How to make separate methods?
The "Write" method.
Gets:
key-value (one key or a collection of keys);
path to the file SettingJson.json.
Performs:
- writes data to SettingJson.json.
The Read Method.
Gets:
- key (one key or collection of keys);
- path to the file SettingJson.json.
Performs:
- reads data from SettingJson.json.
Returns:
- key value (values or a collection of values).
You can use Json.Linq for manipulation with json, e.g. create JObject from Settings using FromObject method and manipulate with it
var settings = FillArrayAll();
//create JObject from settings
var created = JObject.FromObject(new { settings });
//convert to json
var json = created.ToString();
//write json to a file
//…
//read it back
var updated = JObject.Parse(json);
//get value by key
var key = updated["settings"]?.FirstOrDefault()?["NameSetting1"];
//add new empty Settings to an array
(updated["settings"] as JArray)?.Add(JObject.FromObject(new Settings()));
//convert the updated object back to json
var resultJson = updated.ToString();
The comments near code should be pretty explanatory.
You can also find some details at Querying JSON with LINQ, Parsing JSON and Creating JSON
My Situation
I am attempting to create a neural network that classifies two types of signals (yes or no essentially) using ML.net. I have one set of data that maps to no and another that will map to yes. I hope to train the network with this data.
My Problem
Since my training data is supervised (I know the desired output), how do I "tell" the LoadFromTextFile function that all that data should map to "yes" (or 1 it doesn't matter)
My Question
In short, how do you train a network with supervised data(I know the desired output of my training data) in ML.Net?
My Data Model:
public class Analog
{
[LoadColumn(0, Global.SAMPLE_SIZE - 1)]
[VectorType(Global.SAMPLE_SIZE)]
public float[] DiscreteSignal { get; set; }
}
Loading code:
//Create MLContext
static MLContext mCont = new MLContext();
//Load Data
IDataView data = mCont.Data.LoadFromTextFile<Analog>("myYesSignalData.csv", separatorChar: ',', hasHeader: false);
ML.NET has support for loading multiple datasets into one IDataView, by using the MultiFileSource class:
var loader = mCont.Data.LoadFromTextFile<Analog>(separatorChar: ',', hasHeader: false);
IDataView data = loader.Load(new MultiFileSource("myYesSignalData.csv", "myNoSignalData.csv"));
However, I currently see no way to let the trainer know which examples are positive and which are negative other than to add a label column to both files: in the "yes" file add an all-ones column and in the "no" file add an all-zeros column. Then define the Analog class this way:
public class Analog
{
[LoadColumn(0, Global.SAMPLE_SIZE - 1)]
[VectorType(Global.SAMPLE_SIZE)]
public float[] DiscreteSignal { get; set; }
[LoadColumn(Global.SAMPLE_SIZE)]
public float Label { get; set; }
}
Adding the label column can be done with a simple C# program, such as this:
public class AnalogNoLabel
{
[LoadColumn(0, Global.SAMPLE_SIZE - 1)]
[VectorType(Global.SAMPLE_SIZE)]
public float[] DiscreteSignal { get; set; }
}
public void AddLabel(MLContext mCont)
{
IDataView data = mCont.Data.LoadFromTextFile<AnalogNoLabel>("myYesSignalData.csv", separatorChar: ',', hasHeader: false);
var pipeline = mCont.Transforms.CustomMapping<AnalogNoLabel, Analog>((input, output) => {
output.DiscreteSignal = input.DiscreteSignal;
output.Label = 1;
}, contractName: null);
IDataView dataWithLabel = pipeline.Fit(data).Transform(data);
using (var stream = new FileStream("myNewYesSignalData.txt", FileMode.Create))
mCont.Data.SaveAsText(dataWithLabel, stream);
}
and a similar script for "myNoSignalData.csv" with output.Label = 0 instead of output.Label = 1.
I'd like to add a custom column after loading my IDataView from file.
In each row, the column value should be the sum of previous 2 values. A sort of Fibonacci series.
I was wondering to create a custom transformer but I wasn't able to find something that could help me to understand how to proceed.
I also tried to clone ML.Net Git repository in order to see how other transformers were implemented but I saw many classes are marked as internal so I cannot re-use them in my project.
There is a way to create a custom transform with CustomMapping
Here's an example I used for this answer.
The input and output classes:
class InputData
{
public int Age { get; set; }
}
class CustomMappingOutput
{
public string AgeName { get; set; }
}
class TransformedData
{
public int Age { get; set; }
public string AgeName { get; set; }
}
Then, in the ML.NET program:
MLContext mlContext = new MLContext();
var samples = new List<InputData>
{
new InputData { Age = 16 },
new InputData { Age = 35 },
new InputData { Age = 60 },
new InputData { Age = 28 },
};
var data = mlContext.Data.LoadFromEnumerable(samples);
Action<InputData, CustomMappingOutput> mapping =
(input, output) =>
{
if (input.Age < 18)
{
output.AgeName = "Child";
}
else if (input.Age < 55)
{
output.AgeName = "Man";
}
else
{
output.AgeName = "Grandpa";
}
};
var pipeline = mlContext.Transforms.CustomMapping(mapping, contractName: null);
var transformer = pipeline.Fit(data);
var transformedData = transformer.Transform(data);
var dataEnumerable = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: true);
foreach (var row in dataEnumerable)
{
Console.WriteLine($"{row.Age}\t {row.AgeName}");
}
Easy thing. I am assuming, you know how to use pipelines.
This is a part of my project, where I merge two columns together:
IEstimator<ITransformer> pipeline = mlContext.Transforms.CustomMapping(mapping, contractName: null)
.Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question1", outputColumnName: "question1Featurized"))
.Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question2", outputColumnName: "question2Featurized"))
.Append(mlContext.Transforms.Concatenate("Features", "question1Featurized", "question2Featurized"))
//.Append(mlContext.Transforms.NormalizeMinMax("Features"))
//.AppendCacheCheckpoint(mlContext)
.Append(mlContext.BinaryClassification.Trainers.SdcaLogisticRegression(labelColumnName: nameof(customTransform.Label), featureColumnName: "Features"));
As you can see the two columns question1Featurized and question2Featurized are combined into Features which will be created and can be used as any other column of IDataView. The Features column does not need to be declared in a separate class.
So in your case you should transform the columns firs in their data type, if strings you can do what I did and in case of numeric values use a custom Transformer/customMapping.
The documentation of the Concatenate function might help as well!